How to upgrade to IHP v1.1.0 on Shipnix

The IHP tooling has been greatly improved. It's time to upgrade.

22 July 2023

The new version of IHP, v1.1.0, is packed with new features and improvements.

The most notable change is the transition to Nix Flakes and devenv, a big overhaul to the overall IHP experience.

As IHP now uses Nix flakes directly, deployments are also faster and less error prone.

To keep using Shipnix with the new IHP version, you need to do some changes as described in this article.

If you are provisioning your IHP app to Shipnix for the first time, you don't need to follow this guide as Shipnix is all set for IHP v1.1.0.

Backup your data #

Do not skip this step

The upgrade should be painless if you closely follow the steps, but since we are doing a big upgrade, it's best to be on the safe side.

Take care to backup your production data and store it on your machine before you deploy the update.

To download a database dump on your local machine, run this command to have a backup.sql just in case:

ssh ship@myhostname "pg_dump postgres://shipadmin@127.0.0.1:5432/defaultdb  -a --inserts --column-inserts --disable-triggers | sed -e '/^--/d'" > backup.sql

First step: Upgrade IHP #

The first thing you need to do is follow the upgrade guide to IHP 1.1.0.

You will notice that is has a conflict with your current Shipnix setup: IHP now uses a flake.nix, and you already have a one that looks different.

Not to worry. You can overwrite the current flake.nix with what the IHP upgrade guide instructs and return to this guide when you are done.

Go to the UPGRADE.md for IHP

flake.nix #

Now that you have upgraded IHP and verified that everything works fine in your development environment, there are just a couple of changes you need to do in your new flake.nix.

Remember to replace flake.nixosConfigurations."test-server-one" with your Shipnix server name.

{
  inputs = {
    ihp.url = "github:digitallyinduced/ihp/v1.1";
    nixpkgs.follows = "ihp/nixpkgs";
    flake-parts.follows = "ihp/flake-parts";
    devenv.follows = "ihp/devenv";
    systems.follows = "ihp/systems";
  };

-  outputs = inputs@{ ihp, flake-parts, systems, ... }:
+  outputs = inputs@{ ihp, flake-parts, systems, nixpkgs, self, ... }:
    flake-parts.lib.mkFlake { inherit inputs; } {

      systems = import systems;
      imports = [ ihp.flakeModules.default ];

      perSystem = { pkgs, ... }: {
        ihp = {
          enable = true;
          projectPath = ./.;
          packages = with pkgs; [
            # Native dependencies, e.g. imagemagick
          ];
          haskellPackages = p: with p; [
            # Haskell dependencies go here
            p.ihp
            cabal-install
            base
            wai
            text
            hlint
          ];
        };
      };
+     flake.nixosConfigurations."test-server-one" = nixpkgs.lib.nixosSystem {
+       system = "x86_64-linux";
+       specialArgs = inputs // {
+         environment = "production";
+         ihp-migrate = self.packages.x86_64-linux.migrate;
+         ihpApp = self.packages.x86_64-linux.default;
+       };
+       modules = [
+         ./nixos/configuration.nix
+       ];
+     };

    };
}

nixos/scripts/provision #

The provision script won't be used on your running server, so this is for keeping your server reproducible.

# Server provisioning script
set -uo pipefail

# Check if the script is being run as root
if [ "$EUID" -ne 0 ]
  then echo "This script must be run as root" >&2
  exit 1
fi

cd /home/ship/server &&
set -o allexport; source /etc/shipnix/.env; set +o allexport &&

echo "Performing initial database operations..." &&
psql postgres://postgres@127.0.0.1:5432/defaultdb -c "ALTER ROLE shipadmin CREATEDB CREATEROLE SUPERUSER" &&
psql postgres://postgres@127.0.0.1:5432/defaultdb -c "GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO shipadmin;" &&
psql postgres://postgres@127.0.0.1:5432/defaultdb -c "GRANT ALL PRIVILEGES ON SCHEMA public TO shipadmin;" &&

echo "Dropping database schema public..." &&
psql $DATABASE_URL -c "drop schema public cascade; create schema public;" &&
echo "Creating schema migrations table..." &&
psql $DATABASE_URL -c "CREATE TABLE IF NOT EXISTS schema_migrations (revision BIGINT NOT NULL UNIQUE);" &&
echo "Importing IHPSchema.sql..." &&
nix build .#ihp-schema
psql $DATABASE_URL < result/IHPSchema.sql
echo "Importing app schema..." &&
psql $DATABASE_URL < Application/Schema.sql &&

if [[ -f Application/Fixtures.sql ]]; then
  echo "Importing Fixtures.sql..." &&
  psql $DATABASE_URL < Application/Fixtures.sql
else 
  echo "No Fixtures.sql found, skipping..."
fi

If you have any custom provisioning actions in your previous provision script, make sure to re-add these in the new ones.

Do not execute the provision manually unless you are deliberately resetting your server. Doing this will wipe your data.

nixos/scripts/before-rebuild #

The new before-rebuild script will have your migrations run a lot quicker as we later will add the migrate script as a system package.

# This script is run before rebuilding the server
set -euo pipefail

# Loads production environment variables
set -o allexport; source /etc/shipnix/.env; set +o allexport
cd /home/ship/server

# Checks if the `migrate` binary is installed as a system package and if the `Application/Migration` directory exists
if command -v migrate >/dev/null 2>&1 && [ -d "Application/Migration" ]; then
    echo "Running database migrations..."
    migrate
else 
    echo "Skipping..."
fi

If you added your own custom commands that needs to be run outside of the nixos-rebuild, remember to re-add these where appropriate.

nixos/scripts/after-rebuild #

Your after-rebuild script will likely be unchanged, unless you run a very old Shipnix configuration, but to be safe, it should look like this:

# Actions to be run after a server is rebuilt, if needed
set -e
echo "Running after-rebuild script"
# Restarting systemctl services makes sure environment variables are properly reloaded in cases where deploys are running and code hasn't changed
sudo systemctl restart ship.service
# Enable if you start using jobs
# sudo systemctl restart ship_jobs.service

And if you are using IHP jobs, make sure to uncomment the command that restarts the ship_jobs.service after a successful deployment.

nixos/ship.nix #

Your new ship.nix can be replaced with the snippet below, although the changes are minimal.

Only make sure to put your unique credential public key into users.users.ship.openssh.authorizedKeys.keys and users.users.root.openssh.authorizedKeys.keys . It's just a small safety measure in case you remove the key from your authorized_keys file by accident so Shipnix can still reach your server.

You can find this public key by going to Server providers in the Shipnix UI and then to your DigitalOcean credentials.

# Shipnix recommended settings
# IMPORTANT: These settings are here for ship-nix to function properly on your server
# Modify with care

{ config, pkgs, modulesPath, lib, ... }:
{
  nix = {
    package = pkgs.nixUnstable;
    extraOptions = ''
      experimental-features = nix-command flakes ca-derivations
    '';
    settings = {
      trusted-users = [ "root" "ship" "nix-ssh" ];
    };
  };

  programs.git.enable = true;
  programs.git.config = {
    advice.detachedHead = false;
  };

  services.openssh = {
    enable = true;
    # ship-nix uses SSH keys to gain access to the server
    # Manage permitted public keys in the `authorized_keys` file
    passwordAuthentication = false;
    #  permitRootLogin = "no";
  };


  users.users.ship = {
    isNormalUser = true;
    extraGroups = [ "wheel" "nginx" ];
    # If you don't want public keys to live in the repo, you can remove the line below
    # ~/.ssh will be used instead and will not be checked into version control. 
    # Note that this requires you to manage SSH keys manually via SSH,
    # and your will need to manage authorized keys for root and ship user separately
    openssh.authorizedKeys.keyFiles = [ ./authorized_keys ];
    openssh.authorizedKeys.keys = [
      # Replace with your unique credential public key found at https://shipnix.io/Credentials 
      "ssh-rsa YOUR UNIQUE CREDENTIAL PUBLIC KEY ship@tite-ship"
    ];
  };

  # Can be removed if you want authorized keys to only live on server, not in repository
  # Se note above for users.users.ship.openssh.authorizedKeys.keyFiles
  users.users.root.openssh.authorizedKeys.keyFiles = [ ./authorized_keys ];
  users.users.root.openssh.authorizedKeys.keys = [
    # Replace with your unique credential public key found at https://shipnix.io/Credentials 
    "ssh-rsa YOUR UNIQUE CREDENTIAL PUBLIC KEY ship@tite-ship"
  ];

  security.sudo.extraRules = [
    {
      users = [ "ship" ];
      commands = [
        {
          command = "ALL";
          options = [ "NOPASSWD" "SETENV" ];
        }
      ];
    }
  ];
}

nixos/configuration.nix #

In your configuration.nix, you need to add the ihp-migrate package that is passed from the flake.nix into the arguments at the top, and add it to environment.systemPackages.

The IHP binary cache is also added to trusted and extra-trusted to have a declarative rule to trust the IHP binary cache whenever needed.

direnv is no longer needed, as it was admittedly a bad practice to use this. Now with the improved IHP tooling we no longer need hacks like this.

You might also see if there are other settings you want to take from this. We recommend looking at nix.gc and nix.settings.auto-optimise-store to preserve disk space, but otherwise, you should be good to go with the highlighted changes.

-{ config, pkgs, modulesPath, lib, environment, ... }:
+{ config, pkgs, modulesPath, lib, environment, ihp-migrate, ... }:
{

  imports = lib.optional (builtins.pathExists ./do-userdata.nix) ./do-userdata.nix ++ [
    (modulesPath + "/virtualisation/digital-ocean-config.nix")
    ./ship.nix
    ./site.nix
  ];

+  nix.settings.substituters = [ "https://digitallyinduced.cachix.org" ];
+  nix.settings.trusted-substituters = [ "https://digitallyinduced.cachix.org" ];
+  nix.settings.extra-trusted-substituters = [ "https://digitallyinduced.cachix.org" ];
+  nix.settings.extra-trusted-public-keys = [ "digitallyinduced.cachix.org-1:y+wQvrnxQ+PdEsCt91rmvv39qRCYzEgGQaldK26hCKE=" ];
+  nix.settings.trusted-public-keys = [ "digitallyinduced.cachix.org-1:y+wQvrnxQ+PdEsCt91rmvv39qRCYzEgGQaldK26hCKE=" ];

  swapDevices = [{ device = "/swapfile"; size = 2048; }];


  # Add system-level packages for your server here
  environment.systemPackages = with pkgs; [
    bash
    jc
-   direnv
+   ihp-migrate
  ];

  # Loads all environment variables into shell. Remove this if you don't want this enabled
  environment.shellInit = "set -o allexport; source /etc/shipnix/.env; set +o allexport";

  nix.settings.sandbox = false;

  # Automatic garbage collection. Enabling this frees up space by removing unused builds periodically
  nix.gc = {
    automatic = true;
    dates = "weekly";
    options = "--delete-older-than 30d";
  };

  # Saves disk space by detecting and handling identical contents in the Nix Store
  nix.settings.auto-optimise-store = true;

  networking.firewall.enable = true;
  networking.firewall.allowedTCPPorts = [ 80 443 22 ];

  programs.vim.defaultEditor = true;

  services.fail2ban.enable = true;

  # system.stateVersion must coincide with the NixOS version used when your first provisioned the server, and then never change even if you upgrade your NixOS version.
  # If you you run this configuration on multiple servers, provisioned with different NixOS versions, you can
  # use the `environment` value that is passed through `specialArgs` from the flake.nix and differentiate so it's correct for each server
  # for example `system.stateVersion = if environment == "production" then "23.05" else "22.11";` or whatever fits your case 

  system.stateVersion = "22.11"; # Keep as is from your original configuration!

}

Do not change system.stateVersion. It should be constant at the value it had when you provisioned your server.

nixos/site.nix #

The site.nix is largely similar to before.

The notable change is that we don't import the ihpApp directly, but it's passed in from the flake.nix.

PostgresSQL notes #

We also recommend to update the services.postgresql declaration and especially that you should explicitly set the services.postgresql.package to to be the same as what's currently on your server.

To find out what your current PostgresSQL version is, request it from your server with ssh:

$ ssh ship@yourserverhost.com "postgres --version"
> postgres (PostgreSQL) 14.8

So if you get version 14 something like above, set it to pkgs.postgresql_14.

This helps with upgrading your postgres version later, and keeps duplicate servers consistent with the same postgres version.

Upgrading to another major Postgres version will require you to do some manual steps as outlined in the NixOS manual.

If you prefer not to deal with databases yourself, consider switching to a managed database provider. This has potential additional cost, but outsources the concern to a specialized service.

So with all that out of the way, here's the recommended changes for the site.nix:

-{ config, lib, pkgs, environment, ... }:
+{ config, lib, pkgs, environment, ihpApp, ... }:
let
  # TODO: Enable SSL/HTTPS when your domain records are hooked up
  # By enabling SSL, you accept the terms and conditions of LetsEncrypt
- ihpApp = import ../.;
  httpsEnabled = true;
  jobsEnabled = true;
in
{
  services.cron = {
    enable = true;
    systemCronJobs = [
      # "*/30 * * * *      root    ${ihpApp}/bin/SomeScript"
    ];
  };

  security.acme.defaults.email = "yourname@email.com";
  security.acme.acceptTerms = httpsEnabled;

  services.nginx = {
    enable = true;
    enableReload = true;
    recommendedProxySettings = true;
    recommendedGzipSettings = true;
    recommendedOptimisation = true;
    recommendedTlsSettings = true;
  };
  services.nginx.virtualHosts = {
    # you can switch out "localhost" with a custom domain name
    "localhost" = {
      serverAliases = [ ];
      enableACME = httpsEnabled;
      forceSSL = httpsEnabled;
      locations = {
        "/" = {
          proxyPass = "http://localhost:8000";
          proxyWebsockets = true;
          extraConfig =
            # required when the target is also TLS server with multiple hosts
            "proxy_ssl_server_name on;" +
            # required when the server wants to use HTTP Authentication
            "proxy_pass_header Authorization;";
        };
      };
    };
  };


  services.postgresql = {
    enable = true;
+   package = pkgs.postgresql_14;
    ensureDatabases = [ "defaultdb" ];
    ensureUsers = [
      {
        name = "shipadmin";
        ensurePermissions = {
          "ALL TABLES IN SCHEMA public" = "ALL PRIVILEGES";
        };
      }
    ];
    # Set to true if you want to access your database from an external database manager like Beekeeper Studio
    # It also requires port `5432` to be open on `networking.firewall.allowedTCPPorts`
    enableTCPIP = false;
+   authentication = ''
+     local all all trust
+     host all all 127.0.0.1/32 trust
+     host all all ::1/128 trust
+     host all all 0.0.0.0/0 reject
+   '';
  };


  systemd.services.ship = {
    description = "IHP service";
    enable = true;
    after = [
      "network.target"
      "postgresql.service"
    ];
    wantedBy = [
      "multi-user.target"
    ];
    serviceConfig = {
      Type = "simple";
      User = "ship";
      Restart = "always";
      WorkingDirectory = "${ihpApp}/lib";
      EnvironmentFile = /etc/shipnix/.env;
      ExecStart = "${ihpApp}/bin/RunProdServer";
    };
  };

  systemd.services.ship_jobs = {
    description = "IHP job watcher";
    enable = jobsEnabled;
    after = [ "ship.service" ];
    wantedBy = [
      "multi-user.target"
    ];
    serviceConfig = {
      Type = "simple";
      User = "ship";
      Restart = "always";
      WorkingDirectory = "${ihpApp}/lib";
      EnvironmentFile = /etc/shipnix/.env;
      ExecStart = '' ${ihpApp}/bin/RunJobs '';
    };
  };
}

Ensure database permissions #

This should in most cases not be necessary, but to be safe that the shipadmin user has the proper permissions to run IHP, you can SSH into your server and run these commands.

psql postgres://postgres@127.0.0.1:5432/defaultdb -c "ALTER ROLE shipadmin CREATEDB CREATEROLE SUPERUSER"
psql postgres://postgres@127.0.0.1:5432/defaultdb -c "GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO shipadmin;"
psql postgres://postgres@127.0.0.1:5432/defaultdb -c "GRANT ALL PRIVILEGES ON SCHEMA public TO shipadmin;"

IHP requires high permission levels for your database user. This ensures that the IHP database user has the permissions it needs to function properly.

Commit and deploy #

With this, you can commit your changes, and your IHP v1.1.0 app should be upgraded and working with Shipnix ✨

Next: Shipnix is sunsetting January 2025