NDMP with Tivoli Storage Manager September 2013

1. NDMP with Tivoli Storage ManagerSeptember 2013 IBM Advanced Technical Skills Tom Hepner hep@us.ibm.com

2. TSM Advanced Technical Skills Team • Dave Canan • ddcanan@us.ibm.com • Richard Crespo • racrespo@us.ibm.com • Dave Daun • djdaun@us.ibm.com • Tom Hepner • hep@us.ibm.com

3. Agenda • Overview of NDMP • NDMP Backup Configurations • Configuring NDMP Filer to Server Backups • Common Problems and Hints for Backups/Restores • Snapmirror • Snapdiff

4. Network Attached Data Devices • A Network Attached Storage (NAS) subsystem looks, to a client Operating System (OS), as any other mounted filesystem or drive. Data is presented as files. • A SAN disk appears as a raw device to the OS. Management utilities must be used to mount, format and make that device ready. The filesystem is a function the client OS. Data is presented as blocks.

5. Network Data Management Protocol • SCSI commands enable data devices to receive, respond and execute commands to and from other devices. • The protocol has been designed to transport data between NAS devices and backup devices. This removes the need for transporting the data through the backup server, thus enhancing speed and removing load from the backup server.

6. NDMP Devices • Dedicated file server appliances (filers) run an operating system optimized for file-serving functions. • NetApp filers run DATA ONTAP with WAFL. • EMC VNX filers run DART or FLARE. • IBM SONAS uses GPFS on RHEL • Filers accept various different types of connections. • Filers typically do not run software from outside sources.

7. Overview of NDMP • The Network Data Management Protocol is an open standard protocol created by NetApp and Legato. • A NetApp filer, provides storage on a SAN through NFS, CIFS, FTP, TFTP, or HTTP. • NDMP enables backup applications to control backup activities on filers for heterogeneous file systems.

8. Overview of TSM and NDMP • TSM supports a typical NDMP setup which sends data from a filer to SCSI, 349X, and ACSLS tape or virtual libraries. • TSM supports 3-WAY NDMP backup Important: An NDMP device class can only be defined for a library where all of the drives can read and write all of the media in the library.

9. TSM NDMP backup configuration setup

10. TSM NDMP backup configuration setup • Setting Up Tape Libraries for NDMP Operations • Configuring Tivoli Storage Manager Policy for NDMP Operations • Registering NAS Nodes with the TSM Server • Defining a Data Mover for the NAS File Server • Defining a Path to a Library • Defining Tape Drives and Paths for NDMP Operations • Labeling Tapes and Checking Tapes into Library • Scheduling NDMP Operations • Defining Virtual File Spaces

11. Setting Up Tape Libraries for NDMP Operations • Set up the tape library and media. • Attach the SCSI library to the NAS file server or to the Tivoli Storage Manager server, or attach the ACSLS library or 349X library to the Tivoli Storage Manager server. • Define the library with a library type of SCSI, ACSLS, or 349X. • Define a device class for the tape drives. • Supported library types • SCSI (TSM server controls robotics directly or by passing SCSI commands through NAS device) • VTL (mimics SCSI) • ACSLS • 349x (Requires NAS vendor support in NDMP server )

12. POLICY DOMAIN POLICY SET MANAGEMENT CLASS COPY GROUP Configuring TSM Policy for NDMP Operations • DEFINE DOMAIN • DEFINE POLICY SET • DEFINE MGMTCLASS • DEFINE COPYGROUP • DEFINE STGPOOL

13. Registering NAS Nodes with the TSM Server • REGISTER NODE NASNODE NASPASS DOMAIN=NASDOMAIN TYPE=NAS Defining a Data Mover for the NAS File Server • DEFINE DATAMOVER NASNODE TYPE=NAS HLADDRESS=NETAPP2 LLADDRESS=10000 USERID=ROOT PASSWORD=ADMIN DATAFORMAT=NETAPPDUMP

14. Defining a Path to a Library and Drives • Define a path to the SCSI library from either the Tivoli Storage Manager or the NAS file server. DEFINE PATH NASNODE NASLIB SRCT=DATAMOVER DESTT=LIBRARY DEVICE=mc0 • The value of the DEVICE parameter is the special file name for the tape library as it is known to the NAS file server. See Obtaining Special File Names for Path Definitions. ( Next slide)

15. Defining a Path to a Library and Drives (cont.) • Obtaining Special File Names for Path Definitions • To display the device names for tape libraries, issue the following command to the NAS filer: sysconfig –m • To display the device names for tape drives, issue the following command to the NAS filer: sysconfig -t

16. Defining Tape Drives and Paths for NDMP • Define a drive named NASDRIVE for a library named NASLIB • DEFINE DRIVE NASLIB NASDRIVE ELEMENT=117 •  Define a path for the drive: • DEFINE PATH NASNODE NASDRIVE SRCT=DATAMOVER LIBRARY=NASLIB DEVICE=rst01

17. Labeling Tapes and Checking Tapes into Library • You must label the tapes and check the tapes into the tape library. These tasks are the same as for other libraries. Scheduling NDMP Operations • You can schedule the backup or restore of images produced by NDMP operations by using administrative schedules that process the BACKUP NODE or RESTORE NODE administrative commands.

18. Defining Virtual Filespaces Server Virtual filespace definitions are used to reference NAS directory paths. A virtual filespace definition is used just like a NAS filespace. • Example: VFS NamerepresentsFull path to Directory on NAS device /NasDir represents /vol/vol2/home

19. Virtual Filespaces Advantages • A file system backup can be partitioned among several NDMP backup operations and multiple tape drives. • You can also use different backup schedules to back up sub-trees of a file system. • Backup and restore times will be reduced • Provides more flexibility in configuring your NAS backups • Virtual filespaces provide a mechanism for the server to manage NAS directory level backups just like NAS filespaces

20. Virtual Filespaces Notes • The virtual file space name cannot be identical to any file system on the NAS node. • TSM server translates VFS names to the filer’s sub-directory so that the NDMP Component uses a full NAS directory path to drive the backup. • VFS names can be used wherever NAS filespace names are used such as in: • BACKUP NODE • RESTORE NODE • QUERY TOC • QUERY NAS BACKUP • QUERY FILESPACE

21. Virtual Filespaces Notes (cont.) • Server commands related to Virtual Filespaces and NDMP • DEFINE VIRTUALFSMAPPING • DELETE VIRTUALFSMAPPING • QUERY VIRTUALFSMAPPING • UPDATE VIRTUALFSMAPPING

22. TSM NDMP backup configuration setup (basic)

23. Configuring NDMP Filer to Server Backups (3-way)

24. Configuring NDMP Filer to Server Backups(Also known as “3-way NDMP Backup”) • Set up a native storage pool • DEFINE STGPOOL NASPOOL DISK FORMATSIZE=100G • Set the copy destination to the storage pool • UPDATE COPYGROUP STANDARD STANDARDSTANDARD DESTINATION=NASPOOL TOCDESTINATION=NASPOOL • Activate the associated policy set • ACTIVATE POLICYSET STANDARD STANDARD

25. Configuring NDMP Filer to Server Backups • Set up a node for the NAS device. • REGISTER NODE NASNODE NASPASS DOMAIN=NASDOMAIN TYPE=NAS • 5. Set up a datamover for the NAS device. • DEFINE DATAMOVER NASNODE TYPE=NAS HLADDRESS=NETAPP2 LLADDRESS=10000 USERID=ROOT PASSWORD=ADMIN DATAFORMAT=NETAPPDUMP

26. Advantages with NDMP Filer to Server Backups(NDMP backup over LAN to TSM server) • There are fewer library and device restrictions • You can also backup NAS images to disk. • Migration, Reclamation, Backup Storage Pool can be used with NAS data

27. Common problems and hints for NAS backups The filer can be backed up 2 different ways: • Backup using DSMC (the TSM command-line client). The TOC option can be used with this interface. • Backup using BACKUP NODE from DSMADMC (the TSM administrative client).

28. Common problems and hints for NAS backups (cont.) • A TOC is stored in the TOCDESTINATION and can be used later for file-level restore and file-level browsing • NAS Backups pin the recovery log until completion

29. Common problems and hints for NAS backups (cont.) • There are a few common problems that can be experienced when running NAS backups. • ANR1104 Operation process ID terminated - NDMP session errors encountered. • Investigate TSM and NDMP logs for a root cause

30. NDMP in Server to Server Configuration • TSM Server • Library Manager • (NDMP here only) • TSM Server • Library Client • Server to Server • Storage Pools • Storage Pools • Device Class • Device Class

31. SnapmirrorOverview Typical TSM Snapmirror Deployment

32. Snapmirror Support Overview Allows Snapshot Differential backup of snapmirror restricted (read-only) mirror volumes By default TSM creates base and diff snapshots on volumes being backed up which isn’t possible on read only mirror volumes Options allow using and specifying existing base and diff snapshots

33. Snapmirror Support – Backing up Mirror Volumes Toleration for renamed snapshots to support snapshot scheduler naming hierarchy Snapshot scheduler creates and retains policy defined snapshots Default policy is 0 weekly, 2 nightly, and 6 hourly Naming hierarchy uses sequence numbers to version snapshots. Older snapshots have higher sequence numbers. Per retention policy, older snapshots are deleted, sequence numbers of existing snapshots pushed up in hierarchy TSM recognizes that the name of a snapshot registered by a previous backup has changed and will use the new name as the base for the current backup

34. Snapmirror Support – options USEEXISTINGBase Specifies that an existing snapshot should be selected WHEN establishing a new base snapshot instead of creating one May be used with the BASESNAPSHOTNAME option to select a particular snapshot or match a snapshot pattern Does NOT cause a new base to be established New base snapshots are established During an initial backup When the CreateNewBase=Yes option is specified When a snapshot registered by a previous backup doesn’t exist AND an older snapshot also doesn’t exist Establishing a new base snapshot always results in a full incremental

35. Snapmirror Support – options BASESNAPSHOTName Used in conjunction with the USEEXISTINGBASE option to specify criteria for selecting base snapshots May specify a specific snapshot or a pattern containing wildcards ? matches any one character * matches zero or more characters most recent snapshot matching the pattern will be selected Examples BASESNAPSHOTNAME=nightly.1 - selects nightly.1 snapshot BASESNAPSHOTNAME=nightly.? - matches the most recent nightly snapshot BASESNAPSHOTNAME=snap* - matches the lates snapshot prefixed by snap

36. Snapmirror Support – options DIFFSNAPSHOTName Used in conjunction with the DIFFSNAPSHOT=latest option to specify criteria for selecting diff snapshots May specify a specific snapshot or a pattern containing wildcards ? matches any one character * matches zero or more characters most recent snapshot matching the pattern will be selected Examples DIFFSNAPSHOTNAME=nightly.1- selects the nightly.1 snapshot DIFFSNAPSHOTNAME=nightly.? - matches the latest nightly snapshot DIFFSNAPSHOTNAME=snap* - matches the latest snapshot prefixed by snap

37. Snapmirror Support – Backing up Source Volumes Recommend using the TSM default of creating base and diff snapshots Selecting latest existing snapshot via DIFFSNAPSHOT=LATEST or USEEXISTINGBASE may cause problems Temporary, transient snapshots created by snapmirror may be inaccessible at the snapshot or file level May cause failure of the entire backup or of individual files within backup Recommend using DIFFSNAPSHOTNAME and BASESNAPSHOTNAME options to select specific snapshots or patterns

38. Snapmirror Support – Backing up Mirror Volumes Mirror volumes are read only so snapshots must be created and managed on the source volume and replicated to the mirror User written scripts to create and manage snapshots used for backup Create and delete unique snapshots based on backup window Possible scripting choices are Perl, PowerShell, shell script, etc. PuTTY plink utility may be used to issue filer commands Snapshots created by the default scheduler (nightly and hourly) Snapshots automatically deleted per defined retention policy Default policy retains 5 hourly snapshots, 2 nightly snapshots Snapshot used for backup must be retained long enough for the next backup to successfully complete, failure to do so will result in a full incremental backup

39. Snapmirror Support – Backing up Mirror Volumes Requires USEEXISTINGBASE and DIFFSNAPSHOT=latest options Both options should always be specified since it is unpredictable when a new base snapshot may need to be established Strongly recommend using BASESNAPSHOTNAME and DIFFSNAPSHOTNAME options to avoid selecting temporary, transient snapshots created by snapmirror Base snapshot must always be older than selected diff snapshot Backup needs to take into account “lag time”, that is the time it takes snapshots to be replicated from the source to the mirror Lag time can be viewed from NetApp System Commander utility

40. Incremental/Snapdiff Backup • NetApp’s Snapshot API includes Snapshot Difference (SnapDiff) • When called, Snapdiff returns a list of “changed” files, based on successive snapshots (“base” and “diff”) • Files are deemed “changed” if they have been created, modified or deleted (as determined by Snapdiff)

41. Incremental/Snapdiff Backup - Processing (1) Initial Incremental Backup • Incremental (using Snapdiff) initiated by TSM • nSeries/NetApp machine creates snapshot to use as source of TSM incremental • Incremental of snapshot sent to TSM server • Snapshot name is sent to TSM, (stored in client’s filespace)

42. Incremental/Snapdiff Backup - Processing (2) Subsequent Incremental Backups • Name of the last snapshot retrieved from TSM server • New snapshot taken by nSeries/NetApp machine • Snapdiff compares the new and last snapshots and reports changes to TSM • Incremental of new snapshot is performed (based on #3 changes) and sent to TSM server • New snapshot name is sent to TSM server for use in next incremental

43. Questions ?

44. Where to Find More Information: • Tivoli Storage Manager Information Center • http://pic.dhe.ibm.com/infocenter/tsminfo/v6r3/index.jsp • Search on NDMP • NDMP Providers • www.netapp.com • www.emc.com