ATO Overview

1. ATO Overview ATO Downloads http://ca-center.lss.emc.com/imsg/ProductPages/ATO.html Adam Kirkpatrick Created: August, 2009 Last Update: June 12, 2010 kirkpatrick_adam@emc.com

2. ATO – Avamar Tape Out • ATO Overview • Re-Architected Hydrated Tape Out • Flexible & Precise Selection Process • Significant Performance Gains • Comprehensive & Interactive User Interface • Simple to Install, Use, Automate & Troubleshoot • Customer Installable

3. ATO – Architectural Overview Source or Target Single or Multi Node Systems Networker De-Dupe Nodes ATO Environment File Configuration Criteria ATO Client Configuration File Criteria

4. ATO – Architectural Overview • ATO managed 100% from Utility Node or Red Hat client • Client configuration file contains; • client names, tape policy name, staging FS location, staging server type, various control and selection variables • Environment file contains TO infrastructure components; • staging server name, tape backup server info, email notification list, control variables, ATO specific file locations • Supports concurrent staging activities when using multiple SS’s • Supports having multiple tape policy definitions • Supports multiple staging file system destinations • Supports several tape backup applications • Supports using multiple tape backup servers

5. ATO Operational Cycle

6. ATO Operator Menu • Provides Easy Interactive Access To All Administrative Functions • Menu Usage -ato • Batch Manager -ato -batch • Configuration Manager - ato –cfg manager • Environment File Functions • ato –env • ato –env parse • ato –env update • Backup View - ato -v • Event Log Access - ato -l • ATO Help - ato -h • Recovery Manager - ato -recover • Interactive Selection - ato –select ... • Interactive Tapeout - ato –tapeout ... • ATO Administration • - ato –grab | –backup | –backup rollback | -locate • ATO Upgrade – ato -upgrade

7. ATO – Backup Selection Process • Client Selection • By client group–gid group-id • By individual client within a group–gid group-id –client client-id • Backup Selection / Reduction Filters • By File System (default), SQL–sqlor Exchange–exch NDMP –ndmp Oracle –oracle Sharepoint –shpt Lotus Notes -lotus • By first–firstor last–lastbackup or–first_For–last_F • By day of the week–week sunor day instance of the week within a month–week sun_1 • By Avamar policy group name –gname <policy-group-name> • By Avamar retention tag type –rtype <none | daily | weekly | monthly | yearly> • By backup-ID number–buid “#’s ... “ • Automated Search Date Range • Default range adjusts from1st of current month to the current date • To override, Start date: -sdate yyyy-mm-ddEnd date:-edate yyyy-mm-dd • Fields 2 & 3 in client configuration file can be used for a more permanent override • Automated Search Relative Date Range • Override default date range, -rdate 1(adjust to the 1st of the previous month • Useful when combined with –last filter accommodating the selection of the last backup of the month without the need to ever specify a date range • IE. ato –gid group-id –select –last –rdate 1

8. ATO - Sample Selection Process • –lastfilter used limiting selected backups to most recent only • clients scanned are those defined in group gidx only • final selected backup count is 2 with 15 being excluded • used automated date range 2009-08-01 to 2009-08-16 • used ATO environment file 1 • Note: By default the selection process acts on scheduled backups only • refer to –butype option to select non-scheduled backups

9. ATO - Sample View Process • Selected Backup View • Display backup metrics per client • Display total combined backup size • Display combined size allocated per staging FS or folder location • To View: ato-vorato–view

10. ATO – Tape-Out Process • A Tape-Out Session Initiated Using –tapeout Involves 2 Steps • Staging refers to a redirected recovery of selected backups to a staging server location • Tape-Backup refers to backing up the staging area to tape • Step-1 - Staging • stage backup data selected during the last selection process • staging destination determined from Environment & Client configuration files • concurrent staging operations supported if multiple staging servers available • optionally perform stage only with –s option skipping initiation of Tape Backup • any staging platform may be used, Windows, all flavors of UNIX & Linux • a staging server must be a supported client of both Avamar & tape application • Incremental –inc or non-incremental strategies supported • incremental provides significant performance gains • Staging may be directed to any configured staging server and destination FS’s • Include individual folders or files–data “C:/Program Files | My Documents” • Exclude individual folders or files –xdata “C:/Program Files | My Documents” • Step-2 - Tape Backup • back up data staged during Step-1 to tape • initiated automatically by default or optionally using –t (tape-only) option • appropriate tape script generated and initiated on staging server automatically • user defined scripts can be used if preferred • tape backup policy definitions must be `defined to tape application • tape incremental backups can be leveraged only if incremental staging is used

11. ATO – Sample Tape-Out Execution Process

12. ATO Environment File

13. ATO - Environment File Purpose • Defines Key Components of Tape Out Environment • x1 Windows & x1 UNIX staging server per environment file • x1 tape backup server name per environment file • supported tape backup application – ARCserv/Brightstore, hpdp, networker, backupexec, netbackup, tsm • any tape backup application with a suitable CLI can be accommodated • Avamar & Tape backup agent home directories • event log, client configuration & temp file locations • ATO specific control variables • key variables established automatically • Up To20Environment Files Permitted • each defined environment file functions independently • a given ATO session can access and use any defined environment file • base environment(1) assumed by default doesn’t need to be specified • lock mechanism prevents concurrent use of a given environment • each environment maintains its own file set and operates independently

14. ATO - Sample Environment File • Site specific parameters will require modification • staging server parameters • tape backup server parameters • email notification address list • Operational parameters • adjust as required • refer to documentation for details • ATO Control Files • Caution: modification of control file names not recommended • TMP_PATH, CFG & LOG files, Pseudo Client name etc.

15. Establishing & Accessing Environment Files • Establishing Environment Files • To Establish: ato [#] -env • where # is optional defining the environment number, default of 1 is assumed • creates a environment config file /usr/local/avamar/etc/atoenv.cfg[#] • establishes temp file location /tmp/atocfg[#] • establishes event log file /usr/local/avamar/var/atoevent.log[#] • references client configuration file /usr/local/avamar/etc/atoclient.cfg • creates the Avamar pseudo client /clients/tapeoutato[#] • establish various control variables with default values • file names remain consistent with the exception of an appended numeric value • I.E. /usr/local/avamar/etc/atoenv,cfg2, /tmp/atocfg2, /usr/local/avamar/var/atoevent.log2 etc.. • Accessing Environment Files • Displayato [#] -env - displays environment file contents • Update: ato [#] -env update -opens a vi session using appropriate file name • Verify: ato [#] –env parse - verifies file contents and report on any anomalies • Note: environment # when required must be the first argument specified to ATO

16. ATO Client Configuration File

17. ATO - Client Configuration File • Client Configuration File Variables • defines client names and how they are grouped • staging server type to use UNIX/Linux or Windows • destination staging FS or folder, may vary between clients within a group • tape policy name to use • determines whether to use auto generated or a user defined tape script • several control variables • Enables Simplified Tape-Out Operations • client groups enable precise and segmented work flow • better to define several smaller groups as opposed to one large one • only defined & enabled clients are processed • clients from different domains are permitted • tape policies & staging destinations used during automated tape script generation • tape policies can vary between client groups (not between clients within a group) • a given client name can exist in any number of groups

18. ATO - Client Configuration File • Establishing Client Configuration • CSV formatted file • base file established automatically • recommended to share a single client configuration file across all environment #’s • file can be updated interactively, in bulk or manually • automated Avamar domain and client discovery process ensures accuracy • typically a one time task at install time with infrequent updates afterwards

19. ATO - Client Configuration File Contents • Viewing Configuration File Contents • To Display: ato –cfg (all groups) or ato –cfg <grp-id> (specific group only) • Content Verification: ato [#] –cfg parse – perform checks of file contents • Color Codes: Blue=predefined sample lines Green=user comments Cyan=critical field highlight Red=field separator Configuration File Display

20. ATO - Client Configuration File Usage • Key ATO Process’s Rely on This File • All tape out clients must be defined in this file • Client Groups Enable Workload Segmentation • group names cannot contain spaces or special characters • a client can be defined once per group but may exist in any number of groups • all clients in a group must share a common tape policy definition & staging server type • An Organized Group Naming Convention is Beneficial • I.E. A marketing dept has 100 clients where 25 clients is the desired workload per session. Group names might be mkt1,mkt2, mkt3, mkt4 maintaining a logical connection between them while facilitating easy work load segmentation and automation from CRON or 3rd party Scheduler using ATO’s Batch Manager described later • Several Options Available on Command Line & in Client File • date ranges, incdel include/exclude folders, policy group name, retention type and destination path • for infrequent use it may be more convenient to use these on the command line • for frequent use it may be more convenient to define these in the client file

21. ATO – Client Configuration File Update Selection Phase • Manual Update:ato –cfg update • Update Method-1 (bulk – useful at install time only) • Automated Update: ato –cfg add or ato –cfg add_v (initiate discovery & update process) • 1st blue highlighted line contains a generic potential line entry • DOMAINX & CLIENTX are keywords replaced automatically, do not modify these • green highlighted line shows a user modified line with fields Fld-1=training, Fld-10=Drive T being changed • first non configured client displayed was optionally rejected • second non configured client displayed and optionally accepted & added to configuration file • subsequent non configured clients are displayed and added or rejected as required • existing client entries requiring modification can be edited manually or using interactive update (Method-2) ► ► ► ► Method-1 Update Session ►

22. ATO – Client Configuration File Selection Phase • Update Method-2 (Interactive – Preferred Method) • Via Interactive Update: ato –cfg manager • Clients viewed by Avamar Domain or ATO Client Group • Drill down into a Avamar domain or ATO group provides options to • add, disable, enable, modify&view individual client entries Method-1 Update Session

23. ATO – Client Configuration File Cont... • Method-2 cont.... • Client status view realtive to ATO showing all available clients • Select a client to add, disable, enable, modify & view individual client entries

24. ATO – Sample Client Configuration File Update • Method-2 cont.. • Sample process to disable/enable a configured client to ATO • Client status display, Blue=Not configured, Green=Enabled, Red=Disabled to ATO enter d to disable a client enter d to disable a client enter “d” to disable a client enter the client-# to disable confirm the displayed line is okay to disable client#2 disabled confirmed by its color change

25. ATO – Sample Client Configuration File Update • Method-2 cont.. • Sample process to modify an existing client entry • Process to add a new client (not-shown) is similar enter “m” to modify a client entry enter the client-# to modify enter the group name for client entry to be modified verification screen can be looped through as often as necessary modifying field contents as required. When ready to accept the change enter “c” to commit the change

26. How to Locate/Track ATO Data on Tape System • All Staged Data Belongs to Staging Server Clients Used • tape backup effectively owned by staging server client • Data Staged to Structured Directory • <user-defined>/<ATO-defined>/Orig-Client-Name/<client-data> • Tape Solution Media Reports Relative to Staging Client Name Only • Combine Tape Reports with ATO Audit Report Provides • tape media involved • relative folder location where to find TO client in staging client backup • staging and tape backup dates • staging server name used • tape backup server used

27. ATO - Audit Log • Used to Track Critical Path Component Name Changes Impacting Recovery Procedure From Tape • Critical Path Components Tracked • Tape Backup Server name • Tape Policy Name • Staging Server Name • Staging Server Destination Folder • Audit Process Is Automatic Requires No User Intervention • every critical path component change generates an environment audit record • audit records readily available interactively by client name from Configuration Manager option T=Audit • Combined with tape solution media reports identifies where a TO client data resides on tape system

28. ATO - Audit Record Updates • Environment Change Detected in Critical Path • will occur relatively infrequently over time • displayed in audit summary or detail report • Non Critical Path records • audit record established every time a client is staged • displayed in audit detail report • Audit Log File • any filters or script can be run against audit log contents to format your own custom or alternative report formats • available through ATO interface for display purposes

29. Sample Audit Summary Report • Summary Report Displays Critical Path Change Records Only initiate audit report initiate audit report initiate audit report initiate audit report select client for report select client group selected report type defaults to summary or enter D = detail

30. Sample Audit Detail Report • Detail Report Displays Critical and Non Critical Audit Records selected report d=detail

31. ATOTape Backup ScriptStaging Server SpecsDestination Path

32. ATO - Tape Scripts • Automated Tape Script Generation • criteria taken from environment and client configuration files to generate a suitable script autotapeout.bat or autotapeout.sh located in the environments temp directory • transferred to staging server <AvamarHomePath>/etc/scriptsandexecuted as a pre script • supports incremental and non-incremental • non-incremental script removes staged data after a successful tape backup • Currently supports tape applications Arcserv – Brightstore, HPDP,Networker, Backupexec NetBackup, & TSM • Networker support for client alias name, I.E. tape out data owned by original client name (disabled) • any tape solution with a suitable CLI could be accommodated • User Defined Scripts • user defined scripts are permitted • must contain necessary logic to allow ATO to capture its completion status • refer to ATO doc for example or use an auto generated one as a reference • must be kept on Utility node, suggested location /usr/local/avamar/etc • Tape Backup Status • tape status saved on staging server in <AvamarHomePath>/etc/scripts/autotapeout.stat • tape backup RC and detailed failure information retrieved and logged to ATO event log • a non-zero tape RC is considered a failure

33. ATO - Staging Server & Destination Path Specs • Staging Server Types • Unix, Windows & Linux • must be a valid supported client of both Avamar & Tape solution • Staging Server Specs • can be physical or virtual • no specific ATO requirements other than those required by Tape App & Avamar agents • Non-Incremental Destination Path: • <user-defined-path>/BYDATE/<client-name>/<date-time-buid#>/<backup-data> • Incremental Destination Path: • <user-defined-path>/INCREMENTAL/<client-name>/<backup-data> 1. Path definition shown in RED is not user modifiable 2. Optional path suffix inserted prior to </backup-data> using the -path option or from Fld-16 in client configuration file

34. Sample Networker Save Set Definition • Sample Networker Client Resource • BYDATE • INCREMENTAL • Define Multiple Client Resources as Required • All Staged Clients to a Given Location are Included in Tape Backup • User Staging Path Defined in Client Cfg. File • Each Client Resource May Have its own Media Pool, Group Assignment and Schedule etc.

35. ATO – Networker Recover View Of Tape-Out Data • Recovery View on Staging Server of Incremental TO • Recovery View on Staging Server of non-Incremental TO referred to as BYDATE

36. ATO Incremental Advantage

37. ATO - Incremental Tape-Out • Incremental Concept • incremental staging is relative to previously staged data, 1st stage is always a full • requires staged data remain on the staging server between ATO sessions • refer to –incdel option for a method to resynchronize staged data with backup client • beneficial for FS & NDMP not for SQL, Exchange, Oracle or Lotus Notes • total disk space required equivalent to size of source data involved • staging disk can be any disk type, Arrays, JBOD’s, stand alone etc. • disk I/O performance much less critical when using incremental • makes it feasible to leverage an incremental tape-backup • Performance Gains • typical effective throughput gains versus non-incremental ~7 to 10 times • expect ~600-900 GB’s/Hr forstagingdependent on % incremental change • expect equivalent throughput gains with tape backup phase • expect reductions in tape media usage of ~70-80% • use of concurrent staging servers can improve aggregate throughput • How to Enable Incremental • during the -tapeout phase add –inc option

38. ATO - Incremental AdvantageExample-1 Environment: single node Initial Stage 4.7 GB’s 2.45 min’s Initial Tape Backup 4.7GB’s 4.5 min’s

39. ATO - Incremental Advantage Example-1 Environment: single node Post Initial Stage 4.7 GB’s 15 sec’s Post Initial Tape Backup 4.7 GB’s 1 min

40. ATO - Incremental Advantage Example-2 Environment: single node Initial Stage 10.5 GB’s 11.45 min’s Session continued... Initial Tape Backup 10.5 GB’s 8 min’s

41. ATO - Incremental AdvantageExample-2 Environment: single node Post Initial Stage 10.5 GB’s 30 sec’s Post Initial Tape Backup 10.5 GB’s 1.5 min’s

42. ATO – Incremental Advantage Single Node Incremental Comparison Results • Factors Influencing Test Results • Networker Server is using Windows • Networker configured to use a DL3D VTL • no changed files were involved

43. ATO - Incremental Advantage Example-3 Environment: multi node(4) Initial Stage 44.5 GB’s 12 min’s Initial Tape Backup 44.5 GB’s 23 min’s

44. ATO - Incremental AdvantageExample-3 Environment: multi node(4) Post Initial Stage 44.5 GB’s 45 sec’s Post Initial Tape Backup 44.5 GB’s 30 sec’s

45. ATO – Incremental Advantage Multi Node Incremental Comparison Results • Factors Impacting Test Results • Networker Server & Staging server were the same server, an Avamar spare node • Networker configured using adv_file type device B2D • no changed files were involved • Staging configured to use all-nodes, provided ~10-15% increase in throughput

46. ATO Event Log

47. ATO - Event Log • All–select and –tapeoutActions Logged • Consolidates All Informational & Error Messages • Readily accessible to user for success / failure confirmation or problem diagnosis • To Access Event Log:ato –l or ato -log • most recent event displayed initially • specify default browse direction by entering P=Previous or N=Next then press enter to browse all events in the direction specified • to jump to a specific event#, enter desired event number & press enter • to search event log enter “s” then “h” for search editor syntax, similar to vi commands • Event Messages are Color Coded • informational messages displayed in your default terminal color • errors messages displayed in red • info returned from tape application displayed in white • Each ATO Environment# Maintains Separate Event Logs • Email Alerts Contain Corresponding Event Log Details

48. optional debug flags used to provide additional command trace information Avamar recovery log name located on SS and available within Activity Monitor staging process completed okay tape backup command syntax issued by auto generated tape script details returned from tape backup command issued by tape script ATO - Event Log Sample – Successful Stage & Tape Status

49. ATO - Event Log Sample – Staging Failure Avamar log file name available in Activity Monitor or on the staging server staging failure information, client, buid, error code staging error count > 0 establish retry checkpoint containing only failed buid’s if there’s no successfully staged items, tape backup back-up initiation will be skipped

50. ATO - Event Log Sample - Tape Backup Failure staging completed okay tape backup failed due to a mis-configured tape policy. The path specified to be backed up is not configured for tape backup. suggested rerun syntax to perform only the tape portion of this TO session no staging is required.