I push it up to an encrypted disk on my VPS behind an ssh connection.

the only people with access to the host are trusted people.

The only way ssh access works without a key is from certain trusted networks. (over a vpn only)