Installer is interrupted with password prompt when target host contains pre-existing SSH key

[xchangeee]

Hey 👋

I tried to use nixos-anywhere to install NixOS on a Hetzner Cloud VM thats provisioned with a pre-existing ssh key, and ran into some issues.

To keep it simple, the VM is created via terraform without invoking any nix first (e.g. no local-exec provisioner or other stuff like that). On my local machine (darwin aarch64) I have ssh-agent running and SSH'ing after terraform apply works without any problems.

When I run nixos-anywhere on a fresh VM, it prints the following log lines right at the beginning:

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "nixos-anywhere.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: WARNING: All keys were skipped because they already exist on the remote system.
                (if you think this is a mistake, you may want to use -f option)

and then continues to reboot into the nixos installer, but after reboot I am greeted with a regular password prompt. At this point I can open the hetzner console and set a temporary root password with passwd, type that into my local password prompt, and the nixos installer continues, but that doesnt feel right.

After some investigation I found out that invoking nixos-anywhere with -i path/to/private/key resolves the problem, the installer is not interrupted by a password prompt and runs to completion. But this also doesnt feel right because I already have ssh-agent running and passing the private key path should not be necessary.

To pin down the problem, I emulated the behavior of nixos-anywhere.sh and ran ssh-copy-id in isolation.

On a fresh VM:

> ssh-copy-id -i nixos-anywhere.pub -o ConnectTimeout=10 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@host
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "nixos-anywhere.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: WARNING: All keys were skipped because they already exist on the remote system.
                (if you think this is a mistake, you may want to use -f option)

Next I tried to emulate -i by passing my private key, but to my surprise this didnt work:

> ssh-copy-id -i nixos-anywhere.pub -o ConnectTimeout=10 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentityFile=~/.ssh/id_ed25519.pub root@host
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "nixos-anywhere.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: WARNING: All keys were skipped because they already exist on the remote system.
                (if you think this is a mistake, you may want to use -f option)

I double-checked nixos-anywhere.sh and noticed that -i not only adds -o IdentityFile=path/to/private/key but also -f, and this does the trick:

> ssh-copy-id -i nixos-anywhere.pub -o ConnectTimeout=10 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentityFile=~/.ssh/id_ed25519.pub -f root@host
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "nixos-anywhere.pub"
Warning: Permanently added 'x.x.x.x' (ED25519) to the list of known hosts.

Number of key(s) added:        1

Now try logging into the machine, with:   "ssh -o 'ConnectTimeout=10' -o 'UserKnownHostsFile=/dev/null' -o 'StrictHostKeyChecking=no' -o 'IdentityFile=~/.ssh/id_ed25519.pub' 'root@host'"
and check to make sure that only the key(s) you wanted were added.

And finally it also works just with -f:

> ssh-copy-id -i nixos-anywhere.pub -o ConnectTimeout=10 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -f root@host
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "nixos-anywhere.pub"
Warning: Permanently added 'x.x.x.x' (ED25519) to the list of known hosts.

Number of key(s) added:        1

Now try logging into the machine, with:   "ssh -o 'ConnectTimeout=10' -o 'UserKnownHostsFile=/dev/null' -o 'StrictHostKeyChecking=no' 'root@host'"
and check to make sure that only the key(s) you wanted were added.

ssh-copy-id uses ssh -i nixos-anywhere root@host to check if the new public key already exists. If there is another public key present, this check succeeds even though the new public key is not installed. Not sure if this can be easily fixed upstream.

A possible fix in nixos-anywhere could be to always run ssh-copy-id with the -f option. IMHO this should be fine given that ssh-copy-id is just called once to propagate the temporary ssh public key to the nixos installer root fs.