Backup with Borg and rclone

June 6, 2018

This article is a work in-progres

So, I need a new backup-system.

Today I use tarsnap on my servers, and Arq on my personal computer(s). Tarsnap uses Amazon S3, and Arq backs stuff up to my home-server. This works fairly well, but tarsnap isn't the cheapest solution, and the way I use Arq isn't redundant - if my home-server breaks down, my backup is gone.

I don't have too many requirements for a backup solution, but I have some - in prioritized order:

Looking at the list of different backup-solutions listed in the Arch-wiki, it seems that Borg might fit most of my requirements. I've tried Borg previously, through work, and it seemed fine. Also, paired with rclone or git-annex, I can store a redundant backup in the cloud. It appears that several others have done similar things.

According to the post at opensource.com, I can use rclone to just mount up my redundant cloud-backup if I lose my home-server. This seems really easy, so I'm going with this. As such, the solution will be borg+rclone.

Backup-scheme

mrslave is my home-server, and will serve as my main backup repository.

laptop is a laptop with macOS

stella is one of my servers (it's the one serving you this page, actually)

All clients (stella and laptop are clients) back up to mrslave, over ssh. Since we're using ssh, we'll need a user-account on mrslave. All clients will have their own directory, and they will be locked to this directory. We'll use the restriction that the borg-doc recommends, which is to restrict what commands a user will be able to use via the ssh-key in ~/.ssh/authorized_keys.

Borg

Server-side

Installing Borg is pretty trivial.

dennis@mrslave ~ % sudo apt install borgbackup borgbackup-doc

(you don't really need the doc, but it's nice to have)

Then we will need a user dedicated to Borg. Let's call that user borg. We'll also need a directory to store the backups. I find that /srv/ is a good place to put backup-data. 

dennis@mrslave ~ % sudo groupadd --system borg
dennis@mrslave ~ % sudo useradd --home-dir /srv/backup/borg --create-home --system --gid borg borg
dennis@mrslave ~ % sudo -u borg -H mkdir ~/.ssh ~/stella ~/laptop
dennis@mrslave ~ % sudo -u borg -H touch ~/.ssh/authorized_keys
dennis@mrslave ~ % sudo chmod 700 /srv/backup/borg /srv/backup/borg/.ssh

Client-side

We need to install Borg, create a ssh-key, create a keyfile (and a place to put it). For a keyfile, we'll just use a file generated by ssh-keygen.

dennis@stella ~ % sudo apt install borgbackup
*snip*
dennis@stella ~ % sudo su -
root@stella ~ # ssh-keygen -t ed25519 -f ~/.ssh/mrslave.borg.id_ed25519 -P ""
*snip*
root@stella ~ # cat << EOF >> ~/.ssh/config
root@stella ~ # heredoc>
Host borgbackup
    HostName mrslave.example
    IdentityFile ~/.ssh/mrslave.borg.id_ed25519
    User borg
root@stella ~ # heredoc> EOF

If you're on macOS, the installation is pretty much the same, except for the first line:

dennis@laptop ~% brew cask install borgbackup
*snip*

Then we need to copy the public-key to the server. The public-key goes into /srv/backup/borg/.ssh/authorized_keys, and should be in the following format, with the following restrictions (IN ONE LINE):

command="cd /srv/backup/borg/<client>;
         borg serve --restrict-to-path /srv/backup/borg/<client>"
         <keytype> <key> <host>

For example:

command="cd /srv/backup/borg/stella; borg serve --restrict-to-path /srv/backup/borg/stella" ssh-ed25519 AAAAv4aTaC4lZOI1LTE3BBAAIPo9xS64a0p///IyU8vkl90KHck42Ole/w/I0po6FuCK root@stella.dnns.no

Alright. Let's initialize the backup. Since I'm planning on syncing to the cloud, I'll use keyfile, which stores the keyfile on the client, instead of repokey, which stores the key in the repo (you still need the passphrase, though).

root@stella ~ [2] # borg init --encryption=keyfile borg@borgbackup:stella
*snip*

And then, to backup the keyfile:

root@stella ~ # borg key export borg@borgbackup:stella keyfile && cat keyfile && rm keyfile
*snip*

Copy the key, and the passphrase, to somewhere safe.

Now, let's test the backup.

root@stella ~ # echo "foo" > bar
root@stella ~ # borg create borg@borgbackup:stella::testarchive test
root@stella ~ # mkdir mountpoint
root@stella ~ # borg mount borg@borgbackup:stella mountpoint
Enter passphrase for key /root/.config/borg/keys/borgbackup__stella:
root@stella ~ # cd mountpoint
root@stella ~/mountpoint # ls
testarchive
root@stella ~/mountpoint # cd testarchive
root@stella ~/mountpoint/testarchive # ls
bar
root@stella ~/mountpoint/testarchive # cat bar
foo
root@stella ~/mountpoint/testarvhice # cd
root@stella ~ # borg umount mountpoint

Wohoo, it worked.

Automating it

I'm working on a script for automating the backups. It will be posted when it's finished.

The password-manager

The password-manager can't be backed up in the same way as the rest of your system. If you lose access to your password-manager, how are you going to get access to your backups? You need the password-manager to access your backups, and as such, you need another way of backing it up.

I solve this by backing up my password-managers to several of my servers. That way, unless I lose them all, I'll still have access to them.

Cloud redundancy

Choosing a cloud-storage

As far as I can tell, Wasabi and Backblaze B2 offers some of the cheapest cloud-storage available today, costing respectively $0.0049 and $0.005 per gigabyte of storage per month (as of 2018-05-20. See this page for more options).

However, I've never been a huge fan of cloud-services. Especially not those hosted in the US. Time4VPS offers storage VPSes for a good price. Right now, you pay €5.99/TB/month, or €9.99/2TB/month. If you go with the bi-anual plan, you get 25% off. That's €7.49/2TB/month! Converted to gigabytes and US dollars, that's $0.00418 per gigabyte, per month. Of course, here you have to pay for the full 2TB, so it'll probably end up being more expensive than Wasabi or Backblaze, where you only pay for what you use. Also, it means you have to manage one more server.

I'm going with Time4VPS.

Installing and setting up rclone

dennis@mrslave ~ % sudo apt install rclone
*snip*
dennis@mrslave ~ % sudo su -
root@mrslave ~ # rclone config
2018/05/20 16:48:15 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
n/r/c/s/q> n
name> remote
*snip*

Installing rclone was pretty easy. For the rest of the config, you can follow rclone's documentation on setting up SFTP storage. Also, you might want to throw some encrypted storage on top of that, using rclones crypt-setup.

Data recovery

Recovery of client

Recovery of files on a client is easy.

root@stella ~ # borg key import borg@borgbackup:stella path-to-keyfile && rm path-to-keyfile
root@stella ~ # borg mount borg@borgbackup:stella mountpoint
root@stella ~ # cd mountpoint

Remember setting up ssh-keys and such first.

Recovery of server

Recovery after the server has gone should also be easy. First you set up rclone, as described above. Then you can use rclone mount.

dennis@laptop ~ % rclone mount backup:repo mountpoint

Now you should have your entire borg-archive available at mountpoint.