This article is a work in-progres
So, I need a new backup-system.
Today I use tarsnap on my servers, and Arq on my personal computer(s). Tarsnap uses Amazon S3, and Arq backs stuff up to my home-server. This works fairly well, but tarsnap isn’t the cheapest solution, and the way I use Arq isn’t redundant - if my home-server breaks down, my backup is gone.
I don’t have too many requirements for a backup solution, but I have some - in prioritized order:
- Files need to be encrypted locally before they are uploaded, no matter where they are uploaded
- If you “p0wn” one of my servers, you don’t automatically get access to all my backups
- Be able to resume an interrupted backup - I have some external disks for my laptop, and it can take some time to make a complete backup of these
- Be able to backup different folders on different schedules - I want to backup
chatlogs and mail hourly on servers, but I don’t need hourly backups of
- Redundancy - I need to be able to access backups even if my home burns down. Basically I need to be able to lose any two devices and still be able to make a full recovery
Looking at the list of different backup-solutions listed in the Arch-wiki, it seems that Borg might fit most of my requirements. I’ve tried Borg previously, through work, and it seemed fine. Also, paired with rclone or git-annex, I can store a redundant backup in the cloud. It appears that several others have done similar things.
According to the post at opensource.com, I can use rclone to just mount up my redundant cloud-backup if I lose my home-server. This seems really easy, so I’m going with this. As such, the solution will be borg+rclone.
mrslave is my home-server, and will serve as my main backup repository.
laptop is a laptop with macOS
stella is one of my servers (it’s the one serving you this page, actually)
All clients (stella and laptop are clients) back up to mrslave, over ssh.
Since we’re using ssh, we’ll need a user-account on mrslave. All clients will
have their own directory, and they will be locked to this directory. We’ll use
the restriction that the
recommends, which is to restrict what commands a user will be able to use via
the ssh-key in
Installing Borg is pretty trivial.
dennis@mrslave ~ % sudo apt install borgbackup borgbackup-doc
(you don’t really need the doc, but it’s nice to have)
Then we will need a user dedicated to Borg. Let’s call that user borg. We’ll
also need a directory to store the backups. I find that
/srv/ is a good place
to put backup-data.
dennis@mrslave ~ % sudo groupadd --system borg dennis@mrslave ~ % sudo useradd --home-dir /srv/backup/borg --create-home --system --gid borg borg dennis@mrslave ~ % sudo -u borg -H mkdir ~/.ssh ~/stella ~/laptop dennis@mrslave ~ % sudo -u borg -H touch ~/.ssh/authorized_keys dennis@mrslave ~ % sudo chmod 700 /srv/backup/borg /srv/backup/borg/.ssh
We need to install Borg, create a ssh-key, create a keyfile (and a place to put
it). For a keyfile, we’ll just use a file generated by
dennis@stella ~ % sudo apt install borgbackup *snip* dennis@stella ~ % sudo su - root@stella ~ # ssh-keygen -t ed25519 -f ~/.ssh/mrslave.borg.id_ed25519 -P "" *snip* root@stella ~ # cat << EOF >> ~/.ssh/config root@stella ~ # heredoc> Host borgbackup HostName mrslave.example IdentityFile ~/.ssh/mrslave.borg.id_ed25519 User borg root@stella ~ # heredoc> EOF
If you’re on macOS, the installation is pretty much the same, except for the first line:
dennis@laptop ~% brew cask install borgbackup *snip*
Then we need to copy the public-key to the server. The public-key goes into
/srv/backup/borg/.ssh/authorized_keys, and should be in the following format,
with the following restrictions (IN ONE LINE):
command="cd /srv/backup/borg/<client>; borg serve --restrict-to-path /srv/backup/borg/<client>" <keytype> <key> <host>
command="cd /srv/backup/borg/stella; borg serve --restrict-to-path /srv/backup/borg/stella" ssh-ed25519 AAAAv4aTaC4lZOI1LTE3BBAAIPo9xS64a0p///IyU8vkl90KHck42Ole/w/I0po6FuCK firstname.lastname@example.org
Alright. Let’s initialize the backup. Since I’m planning on syncing to the cloud, I’ll use keyfile, which stores the keyfile on the client, instead of repokey, which stores the key in the repo (you still need the passphrase, though).
root@stella ~  # borg init --encryption=keyfile borg@borgbackup:stella *snip*
And then, to backup the keyfile:
root@stella ~ # borg key export borg@borgbackup:stella keyfile && cat keyfile && rm keyfile *snip*
Copy the key, and the passphrase, to somewhere safe.
Now, let’s test the backup.
root@stella ~ # echo "foo" > bar root@stella ~ # borg create borg@borgbackup:stella::testarchive test root@stella ~ # mkdir mountpoint root@stella ~ # borg mount borg@borgbackup:stella mountpoint Enter passphrase for key /root/.config/borg/keys/borgbackup__stella: root@stella ~ # cd mountpoint root@stella ~/mountpoint # ls testarchive root@stella ~/mountpoint # cd testarchive root@stella ~/mountpoint/testarchive # ls bar root@stella ~/mountpoint/testarchive # cat bar foo root@stella ~/mountpoint/testarvhice # cd root@stella ~ # borg umount mountpoint
Wohoo, it worked.
I’m working on a script for automating the backups. It will be posted when it’s finished.
The password-manager can’t be backed up in the same way as the rest of your system. If you lose access to your password-manager, how are you going to get access to your backups? You need the password-manager to access your backups, and as such, you need another way of backing it up.
I solve this by backing up my password-managers to several of my servers. That way, unless I lose them all, I’ll still have access to them.
Choosing a cloud-storage
As far as I can tell, Wasabi and Backblaze B2 offers some of the cheapest cloud-storage available today, costing respectively $0.0049 and $0.005 per gigabyte of storage per month (as of 2018-05-20. See this page for more options).
However, I’ve never been a huge fan of cloud-services. Especially not those hosted in the US. Time4VPS offers storage VPSes for a good price. Right now, you pay €5.99/TB/month, or €9.99/2TB/month. If you go with the bi-anual plan, you get 25% off. That’s €7.49/2TB/month! Converted to gigabytes and US dollars, that’s $0.00418 per gigabyte, per month. Of course, here you have to pay for the full 2TB, so it’ll probably end up being more expensive than Wasabi or Backblaze, where you only pay for what you use. Also, it means you have to manage one more server.
I’m going with Time4VPS.
Installing and setting up rclone
dennis@mrslave ~ % sudo apt install rclone *snip* dennis@mrslave ~ % sudo su - root@mrslave ~ # rclone config 2018/05/20 16:48:15 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults No remotes found - make a new one n) New remote r) Rename remote c) Copy remote s) Set configuration password q) Quit config n/r/c/s/q> n name> remote *snip*
Installing rclone was pretty easy. For the rest of the config, you can follow rclone’s documentation on setting up SFTP storage. Also, you might want to throw some encrypted storage on top of that, using rclones crypt-setup.
Recovery of client
Recovery of files on a client is easy.
root@stella ~ # borg key import borg@borgbackup:stella path-to-keyfile && rm path-to-keyfile root@stella ~ # borg mount borg@borgbackup:stella mountpoint root@stella ~ # cd mountpoint
Remember setting up ssh-keys and such first.
Recovery of server
Recovery after the server has gone should also be easy. First you set up rclone,
as described above. Then you can use
dennis@laptop ~ % rclone mount backup:repo mountpoint
Now you should have your entire borg-archive available at