Automating backups on a Raspberry Pi NAS

ntlx

6 years ago

In the first part of this three-part collection utilizing a Raspberry Pi for network-attached storage (NAS), we lined the basics of the NAS setup, connected two 1TB onerous drives (one for information and one for backups), and mounted the information drive on a distant gadget through the community filesystem (NFS). In half two, we are going to take a look at automating backups. Automated backups assist you to frequently safe your information and get better from a defect or unintended file removing.

Backup technique

Let’s get began by arising with with a backup technique for our small NAS. I like to recommend creating each day backups of your information and scheduling them for a time they will not intervene with different NAS actions, together with while you have to entry or retailer your information. For instance, you may set off the backup actions every day at 2am.

You additionally have to determine how lengthy you will maintain every backup, since you’d rapidly run out of storage when you saved every each day backup indefinitely. Keeping your each day backups for one week means that you can journey again into your latest historical past when you understand one thing went fallacious over the earlier seven days. But what when you want one thing from additional up to now? Keeping every Monday backup for a month and one month-to-month backup for an extended time period ought to be enough. Let’s maintain the month-to-month backups for a yr and one backup yearly for long-distance time travels, e.g., for the final 5 years.

This leads to a bunch of backups in your backup drive over a five-year interval:

7 each day backups
four (approx.) weekly backups
12 month-to-month backups
5 annual backups

You could recall that your backup drive and your information drive are of equal dimension (1TB every). How will greater than 10 backups of 1TB out of your information drive match onto a 1TB backup disk? If you create full backups, they will not. Instead, you’ll create incremental backups, reusing the information from the final backup if it did not change and creating replicas of latest or modified information. That approach, the backup does not double each night time, however solely grows somewhat bit relying on the adjustments that occur to your information over a day.

Here is my state of affairs: My NAS has been operating since August 2016, and 20 backups are on the backup drive. Currently, I retailer 406GB of information on the information drive. The backups take up 726GB on my backup drive. Of course, this relies closely in your information’s change frequency, however as you’ll be able to see, the incremental backups do not eat as a lot house as 20 full backups would. Nevertheless, over time the 1TB disk will in all probability develop into inadequate to your backups. Once your information grows near the 1TB restrict (or no matter your backup drive capability), you must select a much bigger backup drive and transfer your information there.

Creating backups with rsync

To create a full backup, you should utilize the rsync command line software. Here is an instance command to create the preliminary full backup.

pi@raspberrypi:~ $ rsync -a /nas/information/ /nas/backup/2018-08-01

This command creates a full reproduction of all information saved on the information drive, mounted on /nas/information, on the backup drive. There, it’s going to create the folder 2018-08-01 and create the backup inside it. The -a flag begins rsync in archive-mode, which implies it preserves every kind of metadata, like modification dates, permissions, and homeowners, and copies tender hyperlinks as tender hyperlinks.

Now that you’ve got created your full, preliminary backup as of August 1, on August 2, you’ll create your first each day incremental backup.

pi@raspberrypi:~ $ rsync -a --link-dest /nas/backup/2018-08-01/ /nas/information/ /nas/backup/2018-08-02

This command tells rsync to once more create a backup of /nas/information. The goal listing this time is /nas/backup/2018-08-02. The script additionally specified the --link-dest possibility and handed the situation of the final backup as an argument. With this feature specified, rsync appears on the folder /nas/backup/2018-08-01 and checks what information information modified in comparison with that folder’s content material. Unchanged information won’t be copied, slightly they are going to be hard-linked to their counterparts in yesterday’s backup folder.

When utilizing a hard-linked file from a backup, you will not discover any distinction between the preliminary copy and the hyperlink. They behave precisely the identical, and when you delete both the hyperlink or the preliminary file, the opposite will nonetheless exist. You can think about them as two equal entry factors to the identical file. Here is an instance:

The left field displays the state shortly after the second backup. The field within the center is yesterday’s reproduction. The file2.txt did not exist yesterday, however the picture file1.jpg did and was copied to the backup drive. The field on the proper displays as we speak’s incremental backup. The incremental backup command created file2.txt, which did not exist yesterday. Since file1.jpg did not change since yesterday, as we speak a tough hyperlink is created so it does not take a lot extra house on the disk.

Automate your backups

You in all probability do not need to execute your each day backup command by hand at 2am every day. Instead, you’ll be able to automate your backup by utilizing a script like the next, which you will need to begin with a cron job.

#!/bin/bashTODAY=$(date +%Y-%m-%d)
DATADIR=/nas/information/
BACKUPDIR=/nas/backup/
SCRIPTDIR=/nas/information/backup_scripts
LASTDAYPATH=$BACKUPDIR/$(ls $BACKUPDIR | tail -n 1)
TODAYPATH=$BACKUPDIR/$TODAY
if [[ ! -e $TODAYPATH ]]; then
        mkdir -p $TODAYPATH
fi
rsync -a --link-dest $LASTDAYPATH $ $TODAYPATH $@
$SCRIPTDIR/deleteOldBackups.sh

The first block calculates the final backup’s folder title to make use of for hyperlinks and the title of as we speak’s backup folder. The second block has the rsync command (as described above). The final block executes a deleteOldBackups.sh script. It will clear up the outdated, pointless backups based mostly on the backup technique outlined above. You may additionally execute the cleanup script independently from the backup script if you’d like it to run much less continuously.

The following script is an instance implementation of the backup technique on this how-to article.

#!/bin/bash
BACKUPDIR=/nas/backup/operate listYearlyBackups()  egrep "$(date +%Y -d "$i yr in the past")-[0-9]2-[0-9]2" 
operate listMonthlyBackups()  head -n 1
        achieved
operate listWeeklyBackups()  grep "$(date +%Y-%m-%d -d "final monday -$i weeks")"
        achieved
operate listDailyBackups()  grep "$(date +%Y-%m-%d -d "-$i day")"
        achieved
operate getAllBackups() 
        listYearlyBackups
        listMonthlyBackups
        listWeeklyBackups
        listDailyBackups
operate listUniqueBackups()  type -u
operate listingBackupsToDelete()  grep -v -e "$(echo -n $(listUniqueBackups) 
cd $BACKUPDIR
listingBackupsToDelete | whereas learn file_to_delete; do
        rm -rf $
achieved

This script will first listing all of the backups to maintain (in line with our backup technique), then it’s going to delete all of the backup folders that aren’t needed anymore.

To execute the scripts each night time to create each day backups, schedule the backup script by operating crontab -e as the basis consumer. (You have to be in root to verify it has permission to learn all of the information on the information drive, regardless of who created them.) Add a line like the next, which begins the script each night time at 2am.

zero 2 * * * /nas/information/backup_scripts/each day.sh

For extra data, examine scheduling tasks with cron.

There are extra issues you are able to do to fortify your backups in opposition to unintended removing or injury, together with the next:

Unmount your backup drive or mount it as read-only when no backups are operating
Attach the backup drive to a distant server and sync the information over the web

This instance backup technique allows you to again up your helpful information to verify it will not get misplaced. You may also simply alter this system to your private wants and preferences.

In half three of this collection, we are going to discuss Nextcloud, a handy approach to retailer and entry information in your NAS system that additionally gives offline entry because it synchronizes your information to the consumer units.