Common Disaster Recovery Tools In Unix and Linux*

1. Common Disaster Recovery Tools In Unix and Linux*

2. Business Continuity Plans and Backups • People make IT support a complex issue! • Disaster Recovery must be based on Business Continuity Plans and meet the requirements as set in the following question: What is the cost of downtime per hour? Webinar

3. Loss of Data - Most Feared Threat Human error 35 Systems failure 31 Supply chain disruption 29 Virus, worm or other malicious attack on IT systems 28 Employee malfeasance (e.g. theft or fraud) 25 % ofrespondents Natural disasters, such as fires or floods 22 Unplanned downtime of online systems 22 Terrorism 16 Power outage 13 Pandemic 13 Application failure 12 Industrial Action 8 Webinar

4. Bootable System Images in Unix and Linux Many tools available. For the sake of brevity, the following will be discussed: AIX mksysb, Network Installation Manager (NIM) HP make_tape_recovery/make_net_recovery, Dynamic Root Disk (DRD)* Linux Mondo Rescue, Clonezilla Solaris ufsdump, fssnap+ufsdump, flash/JumpStart Tru64 btcreate Webinar

5. Tape Drives Limitations inherent with tape media: • A tape drive must be available on each system to be archived. • Must remove old tapes and insert new ones for new backups. • If an archive exceeds the capacity of a tape, you must swap tapes for both creation and extraction. • Must check log files and run dummy restores to ensure data consistency. • Tape drives are more error-prone than a local network or CD-ROM and DVD. • Cost of tapes in large environment is significant. • Cost of managing tape loading and storage is significant. • Generally slower that disk or LAN. Webinar

6. AIX – mksysb(1) Creates a backup of the operating system (root volume group). The file system image is in backup-file format. The tape format includes a boot image, a bosinstall image, and an empty table of contents followed by the system backup (root volume group) image. The root volume group image is in backup-file format, starting with data files and then any optional map files. Webinar

7. AIX – mksysb(1) Pros highlights: • For managing a single or limited number of servers. • Can be used for system cloning. • Use when servers are not networked. • Allows backup to tape drive (CD-ROM and DVD typically done through mkcd(1)). Webinar

8. AIX – mksysb(1) Cons highlights: • Identical tape drive is needed for an off-site recovery. • Cannot back up files that are mounted from a remote server. • If /usr is remote-mounted, you cannot reinstall system from backup image. • Image does not include data on raw devices or in user-defined paging spaces. • It may not restore all device configurations for special features, such as /dev/netbios and some device drives not shipped with the product. • LC_ALL environment variable should be unset (if non-C value). • Does not have built-in error checking to minimize problems when backing up an active file system. • Format specific to AIX (backup-file).* Webinar

9. AIX – mksysb(1) Examples # mksysb –i /dev/rmt0* # mksysb –m /dev/rmt1** # mksysb –i –e /dev/rmt1*** # mksysb /dev/rmt0 –V # mkcd –U –d /dev/cd0 –P –V rootvg**** # mkdvd –d /dev/cd1 Webinar

10. AIX – mksysb(1) Recovery • Boot off the tape drive. • Select option 3 “Maintenance mode for system recovery”. • Access your devices. • Restore files. • Boot. Webinar

11. AIX – NIM(1) • Requires a NIM master and the number of client instances. • NIM depends on certain protocols (NFS, bootp or DHCP, and TFTP). Older versions of AIX also required RSH and other RCMD commands, but in AIX 5.3 and above you can use basic nimsh or openssl. • NIM master must be at the highest level of AIX that it is required to support. • Now supports Linux installations too. Webinar

12. HP-UX – make_tape_recovery(1) Pros highlights: • For managing a single or limited number of servers. • Can be used for system cloning. • Use when servers are not networked. • Suitable tape drive must exist. • Allows backup to tape drive, CD-ROM, or DVD. • Interactive and non-interactive. • Supports tar (default), cpio or pax formats. • Supports LVM and VxVM. • Multi-tape CLUI only. • Configurable. • Versions 7.x and above have ability to block particular paths and protocols during inventory (see instl_adm(4)). Webinar

13. HP-UX – make_tape_recovery(1) Cons highlights: • Identical tape drive is needed for an off-site recovery. • Cannot back up files that are mounted from a remote server. • Does not have built-in error checking to minimize problems when backing up an active file system. • LVM disk mirrors not restored. • LVM physical extents allocated to a logical volume may be in a different location on a disk than before (consider extending contiguous volumes). • Cannot use remote tape drive.* Webinar

14. HP-UX – make_tape_recovery(1) checks # check_tape_recovery # copy_boot_tape –u /dev/rmt/0mn –b –d /tmp # lifls –l /tmp/bootimage # mt –f /dev/rmt/0mn rew # mt –f /dev/rmt/0mn fsf 1 # tar tvf /dev/rmt/0mn* Webinar

15. HP-UX – make_net_recovery(1) Pros highlights: • For managing a large number of servers. • Can be used for system cloning. • Use when servers are networked. • No tape, CD-ROM, or DVD drives needed. • Interactive and non-interactive. • Supports tar (default), cpio or pax formats. • Highly configurable. Webinar

16. HP-UX – make_net_recovery(1) Cons highlights: • Requires large space if many clients are backed up. • Ignite-UX bundles must be at same version on server and clients. • NFS used to save data to Ignite server (firewall issues, especially older versions of NFS)*. • Requires Ignite server for recoveries. Webinar

17. HP-UX – make_net_recovery Protocols and Ports 67 bootpd UDP2 Bootstrap Protocol Server - This service should function only if the server is a BOOTP/DHCP server 68 bootpd UDP Bootstrap Protocol Client - This service should function only if the server is a BOOTP server 69 tftpd UDP Trivial File Transfer Protocol - Found on systems that have Ignite-UX installed. This service should function only if the host is being used as a TFTP server 111 portmap/sunrpc/ rpcbind TCP/UDP SUN Remote Procedure Call (RPC) Webinar

18. HP-UX – make_net_recovery Protocols and Ports (continued) 135 rpcd/dced TCP Distributed Computing Environment (DCE)-based RPC 514 shell TCP Remote Command, No Password Used 1067 instl_boots UDP Installation Bootstrap Protocol Server 1068 instl_bootc UDP Installation Bootstrap Protocol Client 2049 nfsd TCP/UDP NFS Remote File System Webinar

19. HP-UX – make_net_recovery Protocols and Ports (continued) 2121 swagentd TCP/UDP HP Software Distributor Daemon - Used for communication between systems for software installation, listing, or other sw commands 4000 - 4009 secure swagent ports TCP/UDP The swagent firewall configurable ports 49152 - 65535 Dynamic or Private Ports TCP/UDP Dynamic and Private Ports are used by many applications for dynamic port assignments. UDP ports in this range are often RPC ports Webinar

20. HP-UX – Ignite(1) Examples # make_tape_recovery -v -a /dev/rtape/tape3 -x \ inc_entire=vg00 -d “mysrv Ignite“ # make_net_recovery -s remsrv -n 3 -P s -x inc_entire=vg00 \ -x exclude=/tmp -x exclude=/var/tmp –d “mysrv Ignite" Webinar

21. HP-UX – Dynamic Root Disk • DRD (current release A.3.1.0 - February 2008) runs on both Integrity and PA platforms running the following operating systems: HP-UX 11i v2 (11.23) September 2004 or more recent HP-UX 11i v3 (11.31) • Root group being cloned can be managed by any release of LVM on an O/S release supported by DRD. In addition, the root group can be managed by VxVM 4.1 (HP-UX 11i v2 or 11i v3) or VxVM 5.0 (HP-UX 11i v2 only). Webinar

22. HP-UX DRD Benefit: Minimizing Planned Downtime Without DRD: Software management may require extended downtime With DRD: Install/remove software on the clone while applications continue running Install patcheson the clone;applicationsremain running lvol1lvol2lvol3 lvol1lvol2lvol3 lvol1lvol2lvol3 lvol1lvol2lvol3 boot disk clone disk boot mirror clone mirror vg00(active) cloned vg00 (inactive/patched) Activate theclone to makechanges takeeffect lvol1lvol2lvol3 lvol1lvol2lvol3 lvol1lvol2lvol3 lvol1lvol2lvol3 boot disk clonedisk boot mirror clone mirror vg00 (inactive) cloned vg00 (active/patched) Webinar

23. HP-UX – Dynamic Root Disk Pros highlights: • Fully supported by HP. • Full clone. • Complements other parts of total HP solution by reducing system downtime required to install and update patches and other software. • Copy operation is currently done by fbackup and frecover. • Kctune(1) command can be used to modify kernel parameters in the clone. • The ioconfig file and the entire /dev directory are copied by the DRD clone operation, so instance numbers will not change when the clone is booted.* • Supports nPars, vPars, and Integrity VMs. Webinar

24. HP-UX – Dynamic Root Disk Pros highlights: • No tape drive is needed. • No impact on network performance. • No security issues of transferring data across the network. • All DRD processes, including drd clone and drd runcmd, can be safely interrupted issuing Control-C (SIGINT) from the controlling terminal or by issuing kill -HUP<pid> (SIGHUP). This action causes DRD to abort processing and perform any necessary clean up. Do not interrupt DRD using the kill -9 <pid> command (SIGKILL), which fails to abort safely and does not perform cleanup. Webinar

25. HP-UX – Dynamic Root Disk Cons highlights: • VxVM 5.0 not supported in HP-UX 11.31 yet. • Target disk must be a single disk. • Not easy to list all differences between active VG and the clone.** • Cloning should be done when the server’s activity is quiescent. • Cloned disk can be used to boot another system - it is possible to do this, however, factors such as machine personality (e.g., hostname, IP address and so on) make this very difficult. HP does not recommend using the cloned disk to boot another system. Webinar

26. HP-UX – Dynamic Root Disk Cons: • Only the contents of vg00 are copied. A system that has /opt (or any file system that is patched) not in vg00 is not suitable for use with DRD. • Does not provide a mechanism for resizing file systems during a drd clone operation. However, after the clone is created, you can manually change file system sizes on the inactive system without needing an immediate reboot. The whitepaper, Using the Dynamic Root Disk Toolset describes resizing file systems other than /stand. The whitepaper Using the DRD toolset to extend the /stand file system in an LVM environment describes resizing the boot (/stand) file system on an inactive system image. • Current release of DRD does not copy the Itanium service partition (s3). Webinar

27. HP-UX – Dynamic Root Disk Examples HP-UX 11.21: # drd clone -t /dev/dsk/c2t1d0 -x overwrite=true [-x mirror_disk=/dev/dsk/c3t0d1] HP-UX 11.31, use agile views: # drd clone -t /dev/disk/disk32 -x overwrite=true [-x mirror_disk=/dev/disk/disk41] Note that all partitions on Itanium disk are created and s1 and s2 are copied. Webinar

28. HP-UX – Dynamic Root Disk Examples # drd runcmd kctune maxuprc=350 # drd runcmd swlist # drd runcmd swinstall –s mysrv:/mydepot PHSS_01111 # drd runcmd swremove PHSS_01111 # drd runcmd view /var/adm/sw/swagent.log Webinar

29. Linux – Mondo Rescue Mondo Rescue is a GPL disaster recovery solution. It supports Linux (i386, x86_64, ia64) and FreeBSD (i386). Packaged for multiple distributions (RedHat, RHEL, SuSE, SLES, Mandiva, Debian, Gentoo). Uses afio* to create cpio-format archives. Webinar

30. Linux – Mondo Rescue Pros highlights: • GNU General Public License (GPL). • Supports LVM 1 and 2, RAID, ext2, ext3, JFS, XFS, ReiserFS, VFAT and UFS. • Supports tapes, disks, network and CD/DVD as backup media, multiple file systems, USB key/disks, LVM, software and hardware RAID (no more floppy support). • Can use used in interactive and non-interactive mode. • Can backup data to NFS. • Can move/resize/re-allocate partitions. • Supports GRUB and LILO boot managers. Webinar

31. Linux – Mondo Rescue Cons highlights: • Certain packages can create problems. • Watch our for number of free inodes (backups can fail). • Long backups due to huge sparse /var/log/lastlog file. • Cannot handle system and hidden attributes when archiving DOS/Windows files. • Number of bugs: http://trac.mondorescue.org/ • Does not support Red Hat GFS yet. • Does not support multipathing devices (/dev/mapper/mapthXpY). • Should have option to select NIC when archiving across network (currently it takes the first interface available). • Pre- and Post-install script support for restores. • Does not support bootable USB that backs up to DVDs. • Does not support sshfs (CloneZilla has it) and webdav. • Can hang if no floppy present. • Issues when mounting /proc in a chroot environment (for example, mount –bind /proc /var/named/chroot/proc). • Does not work with SELinux. Webinar

32. Linux – Mondo Rescue Examples # mondoarchive -OVr -d /dev/dvd -9 -I \ “/etc /home” –gF # mondoarchive -OVr -d /scd0 mydir/ -0i -qF \ –s 4480m # mondoarchive -0t –d /dev/st0 Webinar

33. Linux – Mondo Rescue Network Recovery • Uses network by default. • Boot from the Mindi mondorescue.iso and hit ENTER a few times to restore. Mindi (Mindi-Linux) makes a mini-distribution from your kernel, modules, modules, tools and libraries. It can also generate an El Torito 2.88/5.76MB boot disk image. Mondo uses Mindi to create a mini-distro, then boots from it and runs on it. • The ISO images can also be used for a PXE restore. For this to work, refer to the file README.pxe provided with Mindi package. Webinar

34. Linux – Clonezilla Clonezilla is a GPL disaster recovery solution. It supports Linux and Microsoft Windows. Clonezilla Live: Allows you to use CD/DVD or USB flash drive to boot and run clonezilla (unicast only). Clonezilla server edition: A DRBL* server must first be set up in order to use Clonezilla (Both unicast and multicast are supported). Based on Partimage, ntfsclone and dd to clone partition. However, clonezilla, containing some other programs, can save and restore not only partitions, but also a whole disk. Webinar

35. Linux – Clonezilla Pros highlights: • File system supported: ext2 2, ext 3, ReiserFS, XFS, JFS, FAT, and NTFS. For these file systems, only used blocks in partition are saved and restored. For unsupported file system, sector-to-sector copy is done by dd in Clonezilla. • LVM 2 is supported. • Multicast is supported in Clonezilla server edition, which is suitable for massive cloning. You can also remotely use it to save or restore machines if PXE and Wake-on-LAN are supported in your clients. Webinar

36. Linux – Clonezilla Cons highlights: • LVM 2 is supported but LVM 1 is not. • Multicast is supported in Clonezilla server edition, which is suitable for massive cloning. You can also remotely use it to save or restore machines if PXE and Wake-on-LAN are supported in your clients. • Due to the limitations of program mkisofs , ocs-iso can not process an image file larger than 4.5 GB. For this reason, if your image is larger than 4.5 GB, ocs-iso will refuse to process it. Webinar

37. Linux – Clonezilla Examples # /opt/drbl/sbin/ocs-live # /opt/drbl/sbin/ocs-iso myimage* # /opt/drbl/sbin/ocs-iso -g en -t -k NONE -e "-g auto -b -c restoredisk myimg2 hda" myimg2** # /opt/drbl/sbin/ocs-iso image3 image4 # /opt/drbl/sbin/ocs-iso -g en -k NONE -s -m ./cust-ocs*** # /opt/drbl/sbin/ocs-live-dev -g en -k NONE -s -c -m ./cust-ocs*** Webinar

38. Solaris – ufsdump(1) Pros highlights: • Easier to restore individual files. • Allows you to choose directories to back up. • Allows you to back up entire system. • Allows backup to tape drive (both local and remote!), CD-ROM, file, or diskette. • Supports UFS and VxFS. • Portable to other versions of dump/restore. • Interactive and non-interactive. Webinar

39. Solaris – ufsdump(1) Cons highlights: • Backs up single file system only – you must enter multiple ufsdump commands to back up data. • Possible errors when backing up open files.* • Cannot automatically calculate number of tapes need.** • Cannot back up files that are mounted from remote server. • Does not have built-in error checking to minimize problems when backing up an active file system.*** • Not efficient in tuning transfer rates to tape drive. • Ufsrestore requires some prior knowledge of disk partitioning. • Does not support ZFS. Webinar

40. Solaris – ufsdump(1) Comparison with HP-UX Ignite • make_tape_recovery creates a bootable tape. There is no need to boot of the installation CD-ROM or DVD. • make_tape_recovery does not require to partition the boot disk manually in recovery process. • make_tape_recovery is fully automated. • Solaris ufsdump resembles fbackup in HP-UX. Webinar

41. Solaris – ufsdump(1) Examples # ufsdump 0f - /dev/rdsk/c0t2d4s5 | \ ( cd /home && ufsrestore xpf -) # ufsdump 0uf /dev/rmt/1cn /dev/rdsk/c3t1d2s1 # ufsdump 0f /home/etc.dmp /etc # ufsdump 0f – /dev/md/rdsk/d33 | \ ssh remsrv “dd obs=32k ibs=32k of=/dev/rmt/0n” # ufsdump 0ucf remsrv:/dev/rmt/0 /usr Webinar

42. Solaris – ufsrestore(1) Example 1.Boot from the media at OBP prompt: ok boot -s cdrom 2. Format the new boot disk. 3. Newfs each of the partitions on the boot disk that are to be restored: # newfs /dev/rdsk/c0t0d0s0 (/) # newfs /dev/rdsk/c0t0d0s3 (/usr) # newfs /dev/rdsk/c0t0d0s4 (/var) # newfs /dev/rdsk/c0t0d0s5 (/tmp) 4. Each slice should be fsck to make sure newfs worked. Webinar

43. Solaris – ufsrestore(1) Example (continued) 5. Mount all slice to be restored (do not mount or restore swap): # mount /dev/dsk/c0t0d0s0 /a # cd /a # ufsrestore rvf /dev/rmt/0n # rm restoresymtable # mount /dev/dsk/c0t0d0s3 /a/usr # mount /dev/dsk/c0t0d0s4 /a/var # mount /dev/dsk/c0t0d0s5 /a/tmp … # cd /a/usr # ufsrestore rvf /dev/rmt/0n … Webinar

44. Solaris – ufsrestore(1) Example (continued) 6. Unmount the file systems: # cd / # umount /a/usr # umount /a/var … 7. For Solaris 2.5 and greater, run the installboot(1) program to re-install the boot block: # cd /usr/platform/`uname -i`/lib/fs/ufs # installboot bootblk /dev/rdsk/c0t0d0s0 8. Check the file systems: # fsck /dev/rdsk/c0t0d0s0 # fsck /dev/rdsk/c0t0d0s3 ... 9. Reboot the server. Webinar

45. Solaris – flash(1) Pros highlights: • Creates archives in cpio (default) or pax formats. Use “-L pax” for flarcreate(1) if individual files are larger than 4 GB. • Can create differential archives (flag “-A unchanged_master_image_dir”). • Customisable. • Flash archives can be copied to NFS, HTTP or HTTPS server, FTP server, tape, CD-ROM, DVD, diskette, and local drive of clone system. • Non-interactive. Webinar

46. Solaris – flash(1) Cons highlights: • Flash creation software removes all RAID-1 volume information from the archive to keep the integrity of the clone system. • VxVM stores configuration information in areas not available to Solaris Flash. If VxVM file systems have been configured, do not create Flash archives. • Active sockets (like /var/tmp/orbit-* directories) can cause flash failures). • The master system and the clone systems must have the same kernel architectures. • Soft partitions not handled properly. Webinar

47. Solaris – flash(1) Webinar

48. Solaris – JumpStart Protocols and Ports TCP and UDP 37 (time) UDP 67 (bootp/DHCP) UDP 69 (TFTP) TCP and UDP 2049 (NFSv4) TCP and UDP 4045 (lockd, may not be needed) UDP 111 (Sun RPC) Webinar

49. Solaris – flash(1) Examples #flarcreate -n mysrv.flar -c -S -R / -t /dev/rmt/2* # flar info -t /dev/rmt/2 files_archived_method=cpio creation_date=200802171923517 creation_master=mysrv content_name=mysrv.flar creation_node=mysrv creation_hardware_class=sun4u creation_platform=SUNW,Sun-Fire-T200 creation_processor=sparc creation_release=5.10 creation_os_name=SunOS creation_os_version=Generic_118833-36 files_compressed_method=none content_architectures=sun4u # flarcreate -n “mysrv" -S /mysrv.flar Webinar

50. Solaris – flash(1) Restore Example • If you want to install the system using a flash archive, select Initial option. Follow the prompts and answer questions. • Options offered: F2_Upgrade F3_Go Back F4_Initial F5_Exit F6_Help Select F4_Initial. • Select Solaris Interactive Installation (Menu 2). Follow the prompts and answer questions. Webinar