Hey I'm in the middle of doing a very similar thing, though I didn't go quite as far as OP. I'm slowly deprecating my old RSA SSH keys and replacing them with a combination of Yubikey, Password Manager, and alternative SSH auth:
1. -sk keys on Yubikeys (with touch enabled) are my primary keys, used for signing and always installed on every host I touch/etc. I initially planned on using the PIV slots on my Yubi's to store the SSH keys instead of the built-in FIDO2 setup, but I hated having to manage multiple moving parts (like getting pscd running on all my clients). The only main benefit of using the PIV slots was being able to set the "cache" touch policy: the YK would cache a touch for up to 15 seconds so doing something like "signing multiple commits during a rebase" would only take one touch instead of num_commits. The side benefit would have been getting to re-use the same SSH key across primary and backup Yubis. The simplicity of the FIDO setup won out and I decided to treat getting annoyed at pressing my Yubikey a bunch of times as a sign I should be doing some sort of alternative setup like a dedicated key. Most prominently, I've now properly setup my Ansible configs with dedicated deploy keys kept as secrets instead of reusing my personal key.
2. I use 1Pass and Bitwarden (work/personal) to store a secondary SSH key that's only used for basic Git auth or other low-risk tasks. This SSH key is loaded from the Password manager directly into ssh-agent via stdin so it never touches my disk; I use fnox to make this backend agnostic.
3. Work uses Teleport, personal uses Tailscale so I rely on those SSH auth most of the time. Nothing much to say here except that Tailscale ssh is really convenient :) .
I've also setup PAM login and sudo on my machines, which brings me back to the old Sun Ray/Smartcard days. None of this is really to prevent security breaches/dedicated attackers or anything, though it certainly makes me harder to compromise. I just wanted to make absolutely sure I couldn't accidentally upload an SSH private key somewhere.
> You will notice though that the private keyfile held on disk uses a .tpm extension this time. This is where an important caveat of the TPM method comes in – Unlike the YubiKey approach, the private key material is actually being stored on disk here. This is because TPM devices do not natively support enrolling SSH keys. So what is happening is that the private key material is being stored on disk in encrypted form, and the TPM is being used to decrypt it.
While this is true, you can create static hierarchy keys that would not resolve into separate key files for the TPM. The issue here is that anyone with access to your TPM can recreate the keys, unless you set a owner password.
- Have NFTables checks for MSS, TCP SYN TTL, Window sizes, CIDR blocks permitted on the public ssh instances (throw away nodes) and behind a VPN on more important roles.
Great article! I discovered ssh-tpm-agent last year and was surprised that all my ThinkPads have hardware TPM, even one from 2018! I've switched over to using it for everything and there are no more private keys on my devices.
I also have a Yubikey but it's been a few years since I tried it for ssh keys. I use it as a regular security key. Maybe I should try it for ssh keys as it sounds like the tooling has improved.
My sticking point with these has always been SSH agent support: the last time I looked, I couldn't use my Yubikey to authenticate the second hop in an A->B->C sequence with a jump box in the middle. Is that resolved now? This blog doesn't address that use case.
1. -sk keys on Yubikeys (with touch enabled) are my primary keys, used for signing and always installed on every host I touch/etc. I initially planned on using the PIV slots on my Yubi's to store the SSH keys instead of the built-in FIDO2 setup, but I hated having to manage multiple moving parts (like getting pscd running on all my clients). The only main benefit of using the PIV slots was being able to set the "cache" touch policy: the YK would cache a touch for up to 15 seconds so doing something like "signing multiple commits during a rebase" would only take one touch instead of num_commits. The side benefit would have been getting to re-use the same SSH key across primary and backup Yubis. The simplicity of the FIDO setup won out and I decided to treat getting annoyed at pressing my Yubikey a bunch of times as a sign I should be doing some sort of alternative setup like a dedicated key. Most prominently, I've now properly setup my Ansible configs with dedicated deploy keys kept as secrets instead of reusing my personal key.
2. I use 1Pass and Bitwarden (work/personal) to store a secondary SSH key that's only used for basic Git auth or other low-risk tasks. This SSH key is loaded from the Password manager directly into ssh-agent via stdin so it never touches my disk; I use fnox to make this backend agnostic.
3. Work uses Teleport, personal uses Tailscale so I rely on those SSH auth most of the time. Nothing much to say here except that Tailscale ssh is really convenient :) .
I've also setup PAM login and sudo on my machines, which brings me back to the old Sun Ray/Smartcard days. None of this is really to prevent security breaches/dedicated attackers or anything, though it certainly makes me harder to compromise. I just wanted to make absolutely sure I couldn't accidentally upload an SSH private key somewhere.
While this is true, you can create static hierarchy keys that would not resolve into separate key files for the TPM. The issue here is that anyone with access to your TPM can recreate the keys, unless you set a owner password.
https://man.archlinux.org/man/ssh-tpm-agent.1#Hierarchy_keys
- Put the public authorized keys in /etc/ssh/keys/%u immutable, monitored for checksum changes.
- Have CIDR restrictions in said public keys.
- Have NFTables checks for MSS, TCP SYN TTL, Window sizes, CIDR blocks permitted on the public ssh instances (throw away nodes) and behind a VPN on more important roles.I also have a Yubikey but it's been a few years since I tried it for ssh keys. I use it as a regular security key. Maybe I should try it for ssh keys as it sounds like the tooling has improved.
I'd rather have qubes than hardware resident keys.