Chapter 31 Backups – Linux Administration: A Beginner's Guide, Eighth Edition, 8th Edition

CHAPTER

31

Backups

Performing backups is a critical part of any server’s maintenance, because a production server that is not backed up is a disaster waiting to happen. Managing backups is also one of the responsibilities of the system administrator. Of course, simply performing backups isn’t the full extent of your responsibilities; you also have to periodically test the backups and ensure that they are accessible and restorable. These statements are true regardless of which underlying operating system the server is running.

In this chapter, we discuss the backup options that come with Linux distros—our coverage will be mostly at a high level. Many commercial backup solutions exist as well, to suit various budget sizes. The best solution for you depends on your site and its needs.

Evaluating Your Backup Needs

Developing a backup solution is no trivial task. It requires that you consider the interactions among all the parts of your network, the servers, and the resources distributed among them. And, of course, you must arrange for backups to occur regularly and to be verified regularly.

Every organization has different backup needs based on its site(s), its network, and its growth pattern. To make an educated decision, you need to consider the following:

•   The amount of data that needs to be backed up

•   Backup hardware and backup medium

•   The amount of network throughput that needs to be supported

•   The speed and ease of data recovery

•   Data deduplication issues

•   Tape management

We discuss these considerations and others in the following sections.

Amount of Data

Determining an accurate count of the data to be backed up is an important variable when estimating your backup needs. What makes this question tough to answer is that you must factor in anticipated future growth in your determination. And so when you’re planning for backup, it’s always wise to try and plan as far ahead as is financially feasible.

It is also important to consider how frequently your data changes. Data that changes often (such as data in databases) needs to be backed up frequently and quickly, whereas data that rarely changes (such as the contents of the /etc directory) doesn’t need to be backed up quite as often. For these and other reasons, multiple backup policies might be needed to fit the types of data you are backing up.

When examining your requirements, take careful stock of compressible versus noncompressible data. With local disk capacities becoming so large, many individuals have taken to storing personal music/image/video collections on their work systems, and of course a lot of this personal data may have nothing to do with the primary business functions. It may be prudent to spell out your organization’s policy on this issue so that expectations are set: If users think that all of the systems are being backed up, they may be taken aback when they find out that their personal music collection wasn’t covered in that!

Backup Hardware and Backup Medium

The type of hardware you choose should reflect the amount and type of data you need to back up, the frequency with which you’re backing it up, and whether it is necessary that backups get rotated to an offsite location.

The common choices available are tape, disk, USB drives, cloud-based solutions, storage area network (SAN), network-attached storage (NAS), and other networked storage arrays. Of these, tape has been around the longest and offers the widest available choices in media density, form factor, and mechanisms.

With all the choices for tape, selecting a specific type can be tricky. Many of the high-density options are appealing for the obvious reason that you can cram more data onto a single tape. Of course, high-capacity tapes and tape drives typically cost more. Work toward finding an optimum solution that backs up the most system data at the best price, balanced with your requirements for media capacity.

NOTE  Many advertisements for tape drives boast impressive capacities, but keep in mind that these numbers are for the compressed data on the tape, not the actual data. The amount of uncompressed data on the tape is usually about half the compressed capacity. This is important to know, because compression algorithms achieve various levels of compression, depending on the type of data being compressed. For example, textual data compresses quite well, but certain graphics/audio/video formats get little to no compression at all!

Disk-based backups are very common. The concept is simple: Back up the contents of source drives A, B, and C to destination drives X, Y, and Z. The protocol or interface for communicating with the destination drives is another issue and can be implemented in a variety of ways. If the primary goal of backups is to protect against simple accidents (file deletions, primary disk going bad, and so on), this works well. Transfers are fast, media is cheap, and building a home-brew Redundant Array of Independent Disks (RAID) system using a low-cost PC, a decent RAID controller, and a few commodity disks can be done relatively cheaply.

TIP  If you don’t mind getting your hands a little dirty and a very minimal learning curve, you should seriously consider using the free alternatives to the prepackaged commercial storage appliances. Several free and open source solutions are available. One incredibly solid example is the FreeNAS project www.freenas.org.

Using disk-based backups, you can automate scheduled file copies with no tape swapping as well as add additional capacity cheaply. The downside is that offsite storage is not so easy. While getting hot-swappable hard disks is possible, they are not nearly as robust as tape when it comes to being handled and moved. For instance, a tape can be dropped several times with practically no risk, but the same cannot be said of dropping a hard disk!

A combination between fixed (disk) and removable (tape) media is also increasing in popularity. In combination solutions, regular backups are made to disk, with periodic backups moved from the backup disk to tape.

In all of these backup options, you should always plan for possible media failure—that is, plan to move backed-up data to new media every few years. This is necessary to ensure that you’re not just using new media, but that the drive itself is still correctly calibrated and modern equipment can still read and write it. In some cases, you might need to consider the data format. After all, data that can be read but not understood doesn’t do anyone any good!

Network Throughput

Unfortunately, network throughput is easily forgotten in the planning of backup operations. A really fast backup server and storage subsystem isn’t very efficient if you feed in the data through a thin straw! Take the necessary time to understand your network infrastructure. Look at where the data is coming from and where it’s going. Use a network monitoring tool such as MRTG (www.mrtg.org) to collect statistics about your switches and routers. Implementing virtual local area networks (VLANs) dedicated to bulk data transfers is a great idea.

Gathering all this information will help you estimate the bandwidth necessary to perform backups. With your analysis done, you’ll be able to figure out which new gear or improvements will net you the best returns.

Speed and Ease of Data Recovery

When requests to restore data arrive, you’re likely to be under the gun to get the data back to the user(s) as quickly as possible. How long your users have to wait will depend on the tool used for backup. This means you need to incorporate the cost of response time into your backup evaluation. How much are you willing to spend to get the response time you need for a restore?

Disk-based restores are, of course, the fastest. They also offer the possibility of online backups, where users can visit the backup server themselves and copy the file back. Tape, by comparison, is much slower. The specific data on the tape needs to be found, the archive read, and the data extracted. This can all take a little bit of time.

So remember to factor the speed and ease of data recovery into the equation when choosing your backup solution, because there isn’t much point in having a high-end and extensive backup solution when the backed-up data is not readily available to the people when they need it most!

Data Deduplication

Besides being a bit of a mouthful to say, data deduplication deals with reducing unnecessary redundant data in a storage system. It’s another way of saying “data un-copying.”

With the quantity and types of data that most organizations and individuals deal with on a daily basis in today’s digital-data-obsessed world, it is easy to see how we could accumulate multiple copies of the same data or files. This unnecessary redundancy is not only inefficient, but it adds to the total costs of performing backups.

We will use the scenario described next to better explain how systems or solutions that are built for solving the problem of redundant data work. Let’s pretend that there exists an organization named Penguins ‘R’ US.

Penguins ‘R’ US has a standard policy that backups of all user workstations within the organization be performed automatically every weekday at 1:00 A.M.

1.   Assume an end user (user A) in the Graphics department downloads five image files from the company’s central file server onto her home folder to work on a project at her workstation (station-A). The five files are named A.jpg, B.jpg, C.jpg, D.jpg, and E.gif.

2.   Like clockwork, the next organization-wide backup of user workstations occurs on Monday at 1:00 A.M.

3.   At some point during the backup process, the deduplication system will do its best to understand and digest the bits and bytes (or the signature) that make up files A.jpg, B.jpg, C.jpg, D.jpg, and E.gif.

4.   Assume another end user (user B) in the Marketing department downloads seven image files from the company’s central file server onto his workstation (station-B) to include in some company marketing material. The seven files are named A.jpg, B.jpg, C.jpg, D.jpg, E.gif, X.bmp, and Y.gif.

5.   When the time comes to run the regular backups on Tuesday, the data deduplication system detects it already has some files with the exact same signature as the files A.jpg, B.jpg, C.jpg, D.jpg, and E.gif somewhere in the backup destination. It also notices that the files X.bmp and Y.gif are brand new on station-B. It creates a signature for the two new files.

6.   During the backup process, the deduplication mechanism copies over only the two new files, X.bmp and Y.gif, on station-B to the destination.

7.   Instead of re-copying the files A.jpg, B.jpg, C.jpg, D.jpg, and E.gif from station-B, the system instead uses some kind of a pointer or marker that points/links to the original and unchanged versions of the files already stored on the destination.

8.   The deduplication system, therefore, saves Penguins ‘R’ US on backup time and storage space. Everybody therefore wins—including you, the brave Linux system administrator!

Tape Management

As the size of your backups grows, so will your need to manage the data you back up. This is where commercial tools often come into play. When evaluating your choices, be sure to consider their indexing and tape management. It does you no good to have 50 tapes’ worth of data if you can’t find the right file. And, unfortunately, this problem only gets worse as you start needing more tapes for each night’s backups.

Managing the Tape Device

The tape device interacts with Linux just as most other devices do: as a file. The filename will depend on the type of tape drive, your chosen mode of operation (auto-rewind or non-rewind), and how many drives are attached to the system.

SCSI tape drives, for example, use the following naming scheme:

Let’s say, for example, that you have a single SCSI tape drive. You can access it using either of these filenames: /dev/st0 or /dev/nst0. If you use /dev/st0, the drive will automatically rewind the tape after each file is written to it. If you use /dev/nst0, on the other hand, you can write a single file to the tape, mark the end of file, but then stay at the tape’s current position. This lets you write multiple files to a single tape.

NOTE  Non-SCSI devices will obviously use a different naming scheme. Unfortunately, there is no standard for naming backup devices if they are not SCSI devices. The QIC-02 tape controller, for example, uses the /dev/tpqic* series of filenames. If you use a non-SCSI tape device, you will need to find its corresponding driver documentation to see what device name it will use.

You may find it handy to create a symbolic link from /dev/tape to the appropriate device name for the rewinding mode and a link from /dev/nrtape for the non-rewinding mode. You can use the following mapping as an example:

•   /dev/tape/dev/st0

•   /dev/nrtape/dev/nst0

This will make it easier to remember the name of the tape device when issuing commands.

What makes these backup device files different from disk files is that they have no file-system structure. Files are continuously written to the tape until it’s full or until an end-of-file marker is written. If a tape device is in non-rewind mode, the write head remains in the position immediately after the last end-of-file marker, ready for the next file to be written.

You can think of tape devices as being similar to a book with chapters. The book’s binding and the paper, like the tape itself, provide a place to put the words (the files). It’s the markings of the publisher (the backup application) that separate the entire book into smaller subsections or chapters (positions). If you (the reader) were an auto-rewinding tape drive, you would close the tape every time you were done with a single file and then have to search through the tape to find the next position (chapter) when you’re ready to read it. If, however, you were a non-rewinding tape drive, you would leave the tape open to the last page you read.

Using mknod and scsidev to Create the Device Files

If you don’t have the /dev/st0 or /dev/nst0 file, you can create one using the mknod command. The major number for SCSI tape drives is 9, and the minor number dictates which drive and whether it is auto-rewinding. The numbers 0 through 15 represent drive numbers 0 through 15, auto-rewinding. The numbers 128 through 143 represent drive numbers 0 through 15, non-rewinding. The tape drive is a character device.

So, to create /dev/st0, we would type this mknod command:

And to create /dev/nst0, we would use this command:

Another choice for creating device names is to use the scsidev program (provided by sg3_utils*.rpm on Red Hat–like distros and provided by scsitools*.dpkg on Debian-like distros). This will create device entries under the /dev/scsi directory that reflect the current state of your SCSI hardware, with the appropriate device type (block or character) and corresponding major and minor numbers. This method, unfortunately, has yet another naming scheme.

The naming scheme for tape devices created using scsidev is as follows:

Here, A is the host number, B is the channel number, T is the target ID, and L is the logical unit (LUN) number.

All the different naming schemes can seem frustrating. The key to all of them, however, is that they are still using the same major and minor numbers. In other words, they all refer to the same driver! In the end, you could decide to call your rewinding and non-rewinding tape devices “Omolara” and “Adere,” respectively, so long as they had the correct major and minor numbers.

Manipulating the Tape Device with mt

The mt program provides simple controls for the tape drive, such as rewinding the tape, ejecting the tape, or seeking a particular file on the tape. In the context of backups, mt is most useful as a mechanism for rewinding and seeking. The mt utility is provided as a part of the mt-st software package. You can install it on an RPM-based distro such as Fedora, CentOS, or RHEL by running the following:

Install mt-st on Debian based distros via the following:

Command-Line Backup Tools

Linux distros come with several command-line interface (CLI) tools that can be used for backing up or restoring data. Though some of them lack administrative front-ends, they are still simple to use—and most important, they get the job done! Many formal and more expansive backup packages (even some GUI-based ones) actually use and rely on these utilities in their back-ends.

dump and restore

The dump tool works by making a copy of an entire file system. The restore tool can then pull any and all files from this copy.

To support incremental backups, dump uses the concept of dump levels. A dump level of 0 means a full backup. Any dump level above 0 is an incremental relative to the last time a dump with a lower dump level occurred. For example, a dump level of 1 covers all the changes to the file system since the last level 0 dump, a dump level of 2 covers all of the changes to the file system since the last level 1 dump, and so on—all the way through dump level 9.

Consider a case in which you have three dumps: the first is a level 0, the second is a level 1, and the third is also a level 1. The first dump is, of course, a full backup. The second dump (level 1) contains all the changes made since the first dump. The third dump (also a level 1) also has all the changes since the last level 0. If a fourth dump were made at level 2, it would have all the changes since the third level 1.

The dump utility stores all the information about its dumps in the /etc/dumpdates file. This file lists each backed-up file system, when it was backed up, and at what dump level. Given this information, you can determine which backup medium to use for a restore. For example, if you perform level 0 dumps on Mondays, level 1 incrementals on Tuesday and Wednesday, and then level 2 incrementals on Thursday and Friday, a file that was last modified on Tuesday but got accidentally erased on Friday can be restored from Tuesday night’s incremental backup. A file that was last modified during the preceding week will be on Monday’s level 0 tape.

NOTE  The dump tool comes with most popular Linux distros. If it isn’t installed by default in your distro, you should be able to install it easily using the distro’s package management system (for example, via dnf install dump). This utility is file system dependent, and the version for Linux works only on Linux’s native file systems (ext2, ext3, ext4, and Btrfs). For other file systems, such as ReiserFS, JFS, and XFS, be sure to use the appropriate dump tool (for example, xfsdump or xfsrestore for XFS file systems).

Using dump

The dump utility takes many parameters, but the most relevant are shown in Table 31-1.

Table 31-1   Parameters for the dump Tool

For example, here is the command to perform a level 0 dump to /dev/st0 of the /dev/sda1 file system:

Using dump to Back Up an Entire System

The dump utility works by making an archive of one file system. If your entire system comprises multiple file systems, you need to run dump for every file system. Since dump creates its output as a single large file, you can store multiple dumps to a single tape by using a non-rewinding tape device.

Assuming we’re backing up to a SCSI tape device, /dev/nst0, we must first decide which file systems we’re backing up. This information is in the /etc/fstab file. Obviously, we don’t want to back up files such as /dev/cdrom, so we skip those. Depending on our data, we may or may not want to back up certain partitions (such as swap and /tmp).

Let’s assume this leaves us with /dev/sda1, /dev/sda3, /dev/sda5, and /dev/sda6. To back up these to /dev/nst0, compressing them along the way, we would issue the following series of commands:

The first mt command makes sure the tape is completely rewound and ready to accept data. Then come all the dump commands run on the partitions, with their outputs piped through gzip before going to the tape. To make the backups go a little faster, the --fast option is used with gzip. This results in compression that isn’t as good as normal gzip compression, but it’s much faster and takes less CPU time. The -c option on gzip tells it to send its output to the standard out. We then rewind the tape and eject it.

Using restore

The restore program reads the dumpfiles created by dump and extracts individual files and directories from them. Although restore is a command-line tool, it does offer a more intuitive interactive mode that lets you go through your directory structure from the tape. Table 31-2 shows the command-line options for the restore utility.

Table 31-2   Command-Line Options for the restore Utility

Here is a typical invocation of restore:

This will pull the dumpfile from the device /dev/st0 (the first SCSI tape device), print out each step restore takes, and then provide an interactive session for you to decide which files from the dump get restored.

Should a complete file system be lost, you can re-create the file system using the appropriate mkfs.* command and then restore to populate the file system. For example, let’s say our external SATA drive (/dev/sdb) fails, and it had a single partition on it (/dev/sdb1) formatted with the ext4 file system. After replacing the failed drive with a new drive and partitioning it, we would re-create the file system like so:

Next, we have to mount the file system at the appropriate location. We’ll assume this is the /home file system, so we type the following:

Finally, with the dump tape in the SCSI tape drive (/dev/st0), we perform the restoration using the following command:

TIP  If you used gzip to compress your dump, you’ll need to decompress it before restore can do anything with it. Simply tell gzip to uncompress the tape device and send its output to the standard out. Standard out should then be piped to restore, with the -f parameter set to read from standard in (stdin). Here’s the command:

tar

Chapter 4 discussed the use of tar for creating archives of files. What wasn’t discussed, however, is the fact that tar was originally meant to create archives of files onto tape (tar = tape archive). Because of Linux’s flexible approach of treating devices the same as files, we’ve been using tar as a means to archive and unarchive a group of files into a single disk file. Those same tar commands could be rewritten to send the files to tape instead.

The tar command can archive a subset of files much more easily than dump can. The dump utility works only with complete file systems, but tar can work on mere directories. Does this mean tar is better than dump for backups? Well, yes, sometimes.

Overall, dump turns out to be much more efficient than tar at backing up entire file systems. Furthermore, dump stores more information about the file, requiring a bit more tape space, but making recovery that much easier. On the other hand, tar is truly cross-platform—a tar file created under Linux can be read by the tar command under any other Linux platform. In fact, gzip-ed tar files can even be read by the popular Windows WinZip and WinRAR programs! Whether you are better off with tar or dump depends on your environment and needs.

rsync

No discussion of traditional open source backup solutions is complete without mentioning the rsync utility, which is used for synchronizing files, directories, or entire file systems from one location to another. The location could be from a local system to another networked system, or it could be within the local file system. It does its best to minimize the amount of data transmitted by using so-called delta encoding (differences between sequential data) when appropriate. rsync lends itself to being scriptable and, as such, is easily included in cron jobs or other scheduled tasks that run regularly/automatically on systems.

Many CLI-based and GUI front-ends rely heavily on rsync (or the librsync library) in their back-end to do the grunt work. Duplicity (http://duplicity.nongnu.org/) is one popular example that provides encrypted bandwidth-efficient backups using the rsync algorithm/library.

Miscellaneous Backup Solutions

Several open source projects exist that aim to provide enterprise-grade backup solutions. A few of them are AMANDA, Bacula, Dirvish, Mondo Rescue, and BackupPC. They all have their strengths and weaknesses, but all provide robust backup solutions nonetheless. They have nice GUI front-ends to them, which make them attractive to administrators with varying skill levels.

The Advanced Maryland Automatic Network Disk Archiver (AMANDA) is a backup system that allows the use of a single master backup server to back up multiple hosts over the network to tape drives, disks, or optical media. You can learn more about AMANDA at www.amanda.org.

Bacula is a network-based backup suite of programs that allows the backup, recovery, and verification of data across a heterogeneous network of systems. The project web site for Bacula is www.bacula.org.

Dirvish is a disk-based, rotating network backup system. Dirvish is especially interesting because it is adept at and refined for backing up to disk rather than tape. You can learn more about Dirvish at www.dirvish.org.

While not exactly a traditional backup solution, Mondo Rescue is another software product worthy of mention. It is more of a disaster-recovery suite. It supports Logical Volume Management (LVM), RAID, and other file systems. It is usually best to create Mondo archives right after a system has been built and configured. The created Mondo images or archives can be used easily and quickly to restore an OS image back to a bare-bones system. Mondo Rescue can be found at www.mondorescue.org.

BackupPC is another popular open source backup software product for backing up Linux and Microsoft Windows PCs and laptops to a server’s disk. The project is hosted at https://backuppc.github.io/backuppc/.

Summary

Backups are one of the most important aspects of system maintenance. Your system may be superbly designed and maintained, but without solid backups, the whole system could be gone in a flash. Think of backups as your site’s insurance policy.

Although tape drives are a slightly aging/unfashionable medium, we covered the fundamentals of tape drives under Linux, along with some of the command-line tools for controlling tape drives and for backing up data to tape drives. With this and other information, you should be able to perform a complete backup of your system.

Thankfully, dump, restore, tar, and rsync are not your only options for backup under Linux. Many commercial and noncommercial backup packages exist as well. Open source projects such as AMANDA, Dirvish, Mondo, and BackupPC are mature and robust choices.

Whichever way you decide to go about the task of backing up your data, just make sure that you do it!