Backups

1. Backups Chapter 10

2. Introduction • At most sites, the information stored on computers is worth more than the computers themselves. • It is also much harder to replace. • Protecting this information is one of the system administrator’s most important tasks • and, unfortunately, most tedious Chapter 10 - Backups

3. Introduction • If executed correctly, backups allow administrators to restore filesystems (or any part of a filesystem) to the condition it was in at the time of the last backup. • Backups must be done carefully and on a strict schedule • The backup system and backup media must be tested regularly to verify that they are working correctly. Chapter 10 - Backups

4. Introduction • Chapter Overview • We begin this chapter with some general backup philosophy, • Followed by a discussion of the most commonly used backup devices and media • Their strengths, weaknesses, and costs • Next, we discuss the standard UNIX backup and archiving commands • and give some suggestions as to which commands are best for which situations Chapter 10 - Backups

5. Introduction • We then talk about how to design a backup scheme and review the mechanics of the UNIX commands dump and restore. • Finally we take a look at Amanda, a free network backup package and offer some comments about its commercial alternatives Chapter 10 - Backups

6. 1. Motherhood and apple pie • Before we get into the meat and potatoes of backups, we want to pass on some general hints that we have learned over time • (usually the hard way) • None of these suggestions is an absolute rule, but you will find that the more of them you follow, the smoother your dump process will be. Chapter 10 - Backups

7. 1. Motherhood and apple pie • Perform all dumps from one machine • rdump allows you to perform dumps over the network. • Although there is some performance penalty for doing this, the ease of administration makes it worthwhile. • We have found the best method is to run a script from a central location that executes rdump (via rsh or ssh) on each machine that needs to be dumped • Or use a software package that automates this process. Chapter 10 - Backups

8. 1. Motherhood and apple pie • All dumps should go to the same backup device • If your network is too large to be backed up by a single tape drive, you should try to keep your backup system as centralized as possible. • Centralization makes administration easier and allows you to verify that all machines were dumped correctly. Chapter 10 - Backups

9. 1. Motherhood and apple pie • Label your tapes • It is essential that you label each dump tape clearly and completely. • An unlabeled tape is a scratch tape. • Detailed information such as lists of filesystems and dump dates can be written on the cases. Chapter 10 - Backups

10. 1. Motherhood and apple pie • You must be able to restore root and /usr filesystems without looking at dump scripts • label the dump tapes with their format, the exact syntax of the dump command used to create them, and any other information you would need to restore from them without referring to on-line documentation • Free and commercial labeling programs abound. • Save yourself a major headache and invest in one. Chapter 10 - Backups

11. 1. Motherhood and apple pie • Pick a reasonable backup interval • The more often backups are done, the smaller amount of data that can be lost in a crash. • However, backups use a systems resources and an operator’s time. • On busy systems it is generally appropriate to back up filesystems with home directories every workday. • The question is: How much data are your users willing to lose? Chapter 10 - Backups

12. 1. Motherhood and apple pie • Choose filesystems carefully • Filesystems that are rarely modified do not need to be backed up as frequently as user’s home directories. • If only a few files change on an otherwise static filesystem (such as /etc/passwd in the root filesystem), these files can be copied every day to another partition that is backed up regularly. Chapter 10 - Backups

13. 1. Motherhood and apple pie • Make daily dumps fit on one tape • In a perfect world, you could do daily dumps of all your filesystems onto a single tape. • You can mount a tape every day before you leave work and run the dump later at night from cron. • This way, dumps occur at a time when files are not likely to be changing, and the dumps have minimal impact on users. Chapter 10 - Backups

14. 1. Motherhood and apple pie • If you can’t fit your daily backups on tape, you have several options: • Buy a higher capacity tape device. • Buy a stacker or library and feed multiple pieces of media to one device • Change your dump sequence • Write a smarter script • Use multiple backup devices. Chapter 10 - Backups

15. 1. Motherhood and apple pie • Make filesystems smaller than your dump device • dump is perfectly capable of dumping filesystems to multiple tapes. • But if a dump spans multiple tapes, an operator must be present to change tapes. • Unless you have a good reason to create a really large filesystem, don’t do it. Chapter 10 - Backups

16. 1. Motherhood and apple pie • Keep tapes off-site • Most organizations keep backups off-site so that a disaster such as fire cannot destroy both the original data and the backups. • The speed with which tapes are moved off-site should depend on how often you need to restore files and how much latency you can accept. Chapter 10 - Backups

17. 1. Motherhood and apple pie • Protect your backups • “What does a backup do? It reliably violates file permissions at a distance.” • Not only should you keep your tapes off-site, but you should also keep them under lock and key • Some companies feel so strongly about the importance of backups that they make duplicates. Chapter 10 - Backups

18. 1. Motherhood and apple pie • Limit activity during dumps • Filesystem activity should be limited during dumps because changes can cause dump to make mistakes. • You can limit activity by • doing the dumps when few active users are around (at night or weekend) or • by making the filesystem accessible only to dump. • Most users will not like this option. They want 24/7 access. Chapter 10 - Backups

19. 1. Motherhood and apple pie • Check your tapes • There are many horror stories about system administrators that did not discover problems with their dump regime until after a serious system failure. • Checks: • Have your dump software attempt to reread the tapes immediately after it has finished dumping. • It is often useful to run restore t to generate a table of contents for each filesystem and store the results to disk. Chapter 10 - Backups

20. 1. Motherhood and apple pie • Develop a tape life cycle • Tapes have a finite life cycle. • It is great to recycle your media. • Most manufacturers quantify their recommended life in terms of the number of passes that a tape can stand. • A backup, a restore, and an mt fsf (file skip forward) each represent one pass. Chapter 10 - Backups

21. 1. Motherhood and apple pie • Prepare for the worst • After you have established a backup procedure, explore the worst case scenario: your site is completely destroyed. • Determine how much data would be lost and how long it would take to get your system back to life. • Include the time it would take to get new hardware. • Then determine if you can live with your answers. Chapter 10 - Backups

22. 2. Backup devices and media • Since many types of failure can damage several pieces of hardware at once, backups should be written to some sort of removable media • Mirroring is a great form of backup, but if the controller dies, then what…. • Many kinds of media store data using magnetic particles. These are subject to damage by electrical and magnetic fields. Chapter 10 - Backups

23. 2. Backup devices and media • Here are some specific hazards to avoid: • Audio speakers contain large electromagnets • Transformers and power supplies • including UPS boxes • Monitors use transformers and high voltages. • Many monitors retain an electrical charge even after being turned off. Color monitors are the worst. • Long Shelf life • All tapes will become unreadable over a period of years. • Most media will keep for three years. If you need longer than that, use optical media or re-record the data. Chapter 10 - Backups

24. 2. Backup devices and media • The following sub-sections describe some of the media that can be used for backups. • Although cost and media capacity are both important considerations, it is important to consider throughput as well. • Fast media are more pleasant to deal with, and they allow more flexibility in the scheduling of dumps. Chapter 10 - Backups

25. 2. Backup devices and media • Floppy disks • The most inconvenient way to store backups. • They are slow and do not hold much data. • Although the individual disks are cheap, they hold so little data that they are in fact the most expensive backup medium overall. Chapter 10 - Backups

26. 2. Backup devices and media • Super floppies • Iomega Zip drives • 100MB to 250MB • SCSI, USB, Parallel, and Serial interfaces • Imation • Drives can read and write normal floppies and 120MB media • Although these products are useful for exchanging data, their high media costs make them a poor choice for backups. Chapter 10 - Backups

27. 2. Backup devices and media • CD-R and CD-RW • Price changes make these a far more attractive medium than they were a few years ago. • CD-R’s are not quite as durable as stamped CDs. • They are good for archiving, but not a good choice for backup media. • Recordable DVD technology, with capacity around 10GB, will change how this field is viewed. Chapter 10 - Backups

28. 2. Backup devices and media • Removable hard disks • The removable drive market is becoming very competitive, and prices vary daily. • Lifetime is 5-10 years • The main advantage of these products is speed (close to normal drives) • They are attractive as backup devices for small systems and home machines, although the disks themselves are somewhat pricey Chapter 10 - Backups

29. 2. Backup devices and media • 8mm cartridge tapes • Small format videotapes - also called Exabyte drives • The original format held 2GB, and newer formats hold up to 7GB. • Hardware compression build into the drives will push this even higher. • The size of the tapes makes off-site storage convenient. • Susceptible to heat damage. Chapter 10 - Backups

30. 2. Backup devices and media • DAT (4mm) cartridge tapes • Digital Audi Tape drives - or Digital Data Storage (DDS) Drives • Original format was 2GB, DDS-4 holds up to 20GB • DDS-4 drives seek rapidly and transfer data at up to about 2.5MB/s, making the drives relatively fast. • They also do not have a history of alignment problems. Chapter 10 - Backups

31. 2. Backup devices and media • Travan tapes • The next generation of QIC tape technology • Travan drives use a linear recording technology and support media from 2.5GB to 10GB • Tapes cost about $3/GB (slightly more than others) • Marketing hype claims 1MB/s transfer rate. • UNIX drive support is sketchy. Chapter 10 - Backups

32. 2. Backup devices and media • OnStream ADR • A relative newcomer to backups. • Based upon Linear recording technology • Supports media sizes of 15GB and 25GB • Drives are cheap and fast Chapter 10 - Backups

33. 2. Backup devices and media • DLT • Popular backup device • reliable, affordable, and hold large amounts of data. • Tapes hold up to 40GB • Transfer rates at 6 MB/s • Media is expensive ($65 per tape) • Manufacturers boast that tapes will last 20 to 30 years • (will the drives still be running then?) Chapter 10 - Backups

34. 2. Backup devices and media • AIT • Sony’s 8mm product on steroids • AIT-2 claims • 6 MB/s native transfer rate and • media capacity of 50GB • Tapes • Long life cycle • Contain an EEPROM to give the media some smarts. Chapter 10 - Backups

35. 2. Backup devices and media • Mammoth • Produced by Exabyte - after falling out with Sony • Mammoth-2 drives offer blindingly fast 12MB/s native transfer rate • far faster than other tape drives in this price range. Chapter 10 - Backups

36. 2. Backup devices and media • Jukeboxes, stackers, and tape libraries • With the low cost of disks these days, most sites have so much disk space that a full backup requires multiple tapes, even at 20GB per tape. • One solution for these sites is a stacker, jukebox, or tape library. Chapter 10 - Backups

37. 2. Backup devices and media • Hard disks • We would be remiss if we did not mention the decreasing cost of hard drives as a reason to consider disk to disk backups. • Although we suggest that you not duplicate one disk to another within the same physical machine, hard disks can be a good low-cost solution for storage over a network. Chapter 10 - Backups

38. 2. Backup devices and media • Summary of media types Chapter 10 - Backups

39. 2. Backup devices and media • What to buy • DAT and Exabyte drives are excellent solutions for small workgroups • DLT, AIT, and Mammoth-2 are all roughly comparable and target University and Corporate environments. Chapter 10 - Backups

40. 3. Setting up an incremental backup regime • The dump and restore commands are the most common way to create and restore from backups. • These programs have been part of UNIX for a very long time, and their behavior is well known. • At most sites, dump and restore are the underlying commands used by automated backup software. Chapter 10 - Backups

41. 3. Setting up an incremental backup regime • Dumping filesystems • The dump command builds a list of files that have been modified since a previous dump, then packs those files into a single large file to archive on an external device. • dump has several advantages over other utilities described later in this chapter • Backups can span multiple tapes • Files of any types (even devices) can be backed up and restored • Permissions, ownerships, and modification times are preserved. • Files containing holes are handled correctly • Backups can be performed incrementally (with only recently modified files being written to tape) Chapter 10 - Backups

42. 3. Setting up an incremental backup regime • dump understands the layout of raw filesystems, and it reads the filesystem’s inode tables directly to decide which files must be backed up • This allows dump to be very efficient • It also imposes a few limitations: • Every filesystem must be dumped individually • Only filesystems on the local machine can be dumped • you can overcome this with rdump Chapter 10 - Backups

43. 3. Setting up an incremental backup regime • The most important feature of dump is its support for the concept of an “incremental” backup. • Incremental dumps make it possible to back up only filesystems that have changed since the last backup. • When you do a dump, you assign it a backup level (0-9). • A level 0 backup places the entire filesystem on the tape. • A level N dump backs up all files that have changed since the last dump of less than N Chapter 10 - Backups

44. 3. Setting up an incremental backup regime • The first argument to dump must be the incremental dump level. • dump uses the /etc/dumpdates file to determine how far back to go. • The u flag causes dump to automatically update /etc/dumpdates when the dump completes. • Why should it be after it completes? Chapter 10 - Backups

45. 3. Setting up an incremental backup regime • Dump sequences • Because dump levels are arbitrary, dumps can be performed on various schedules. • The schedule that is right for you depends on: • The activity of your filesystem • The capacity of your dump device • The amount of redundancy you want. • The number of tapes you want to buy. Chapter 10 - Backups

46. 3. Setting up an incremental backup regime • A simple schedule • If your total disk space is smaller than the capacity of your tape device, you can use a trivial dump schedule • Do level zero dumps of every filesystem each day. • Reuse a group of tapes, but every N days, keep the tape forever • Don’t reuse the exact same tape for every night’s dump. It is better to rotate among a set of tapes so that even if one night’s dump is blown, you can still fall back to the previous night. Chapter 10 - Backups

47. 3. Setting up an incremental backup regime • A moderate schedule • Assign a tape to each day of the week, and each month of the year • Every day do a level 9 dump to the daily tape. • Every week do a level 5 dump to the weekly tape. • And every month do a level 3 dump to the monthly tape. • Do a level 0 at least once per year. Chapter 10 - Backups

48. 4. Restoring from dumps • Most of the many variations of the program that extracts data from dump tapes are called restore. • We will first discuss restoring individual files (or a small set of files), then explain how to restore entire filesystems. Chapter 10 - Backups

49. 4. Restoring from dumps • Restoring individual files • The first step to take when you are notified of a lost file is to determine which tapes contain versions of the file. • Users often want the most recent version of a file, but that is not always the case. • It is helpful if you can browbeat the users into telling you not only what files are missing, but also when they were lost and when they were last modified Chapter 10 - Backups

50. 4. Restoring from dumps • If you do not keep on-line catalogs, you must mount tapes and repeatedly attempt to restore the missing files until you find the correct tape. • If the user remembers when the files were last changed, you may be able to make an educated guess about which tape the files might be on. Chapter 10 - Backups