Backup methods for a hot site
Download
1 / 82

Backup Methods For a Hot Site - PowerPoint PPT Presentation


  • 129 Views
  • Uploaded on

Backup Methods For a Hot Site. Dieter W. Storr Los Angeles Times 23 August 2005. B/R Methods. Existing Backup Method Experiences Mirroring or Replicating Fast Copy of Data Proposals and Costs Future Technology Lessons learned. Existing Backup Method. From disk (databases) Copy to

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Backup Methods For a Hot Site' - huy


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Backup methods for a hot site

Backup Methods For a Hot Site

Dieter W. Storr

Los Angeles Times

23 August 2005


B r methods
B/R Methods

  • Existing Backup Method

  • Experiences

  • Mirroring or Replicating

  • Fast Copy of Data

  • Proposals and Costs

  • Future Technology

  • Lessons learned

Dieter W. Storr -- www.storrconsulting.com


Existing backup method
Existing Backup Method

  • From disk (databases)

  • Copy to

  • 3490 / 3590-1 / VTS

  • Then, copy to

  • 3590-1 (cartridge)

Dieter W. Storr -- www.storrconsulting.com


ADABAS 6.2.2 Back-up at LA Times

Dieter W. Storr -- www.storrconsulting.com


B r methods1
B/R Methods

“Companies that relied on

tape or on third-party provider

found in many cases they had difficulty meeting their recovery time objectives.”

Source: http://www.drj.com/articles/spr02/1502-07.html

Dieter W. Storr -- www.storrconsulting.com


B r methods2
B/R Methods

“Flaws in tape-based data backup may beleaving enterprises without key information and could lead to legal exposure under emerging laws such as Sarbanes-Oxley, say data backup and recovery experts. “

Source: 15 Apr 2004 | SearchSecurity.com

Dieter W. Storr -- www.storrconsulting.com


B r methods3
B/R Methods

  • In a survey of 500 IT departments completed … found that as many as 20% of routine, nightly backups fail to capture all data.

  • 40% of IT managers had been unable to recover data from a tape when they needed it

  • More than 23% sought to use data stored on tape backups more than 20 times in a year

Source: 15 Apr 2004 | SearchSecurity.com

Dieter W. Storr -- www.storrconsulting.com


B r methods4
B/R Methods

Are tapes really so bad?

LA Times experiences?

Dieter W. Storr -- www.storrconsulting.com


Tape problems
Tape Problems

1 November 2002:

  • Six tape drive errors

  • Delay

Dieter W. Storr -- www.storrconsulting.com


Tape problems1
Tape Problems

24 March 2003:

  • Only two channel paths per tape controller were provided

  • Slow restore time

Dieter W. Storr -- www.storrconsulting.com


Tape problems2
Tape Problems

5 October 2003:

  • 3590 tape drives were not defined to DFSMS (SMS)

  • ADABAS restore and application test cancelled

Dieter W. Storr -- www.storrconsulting.com


Tape problems3
Tape Problems

6 December 2003:

  • VTS problems with GDG datasets

  • End-user functions couldn’t be tested

Dieter W. Storr -- www.storrconsulting.com


Tape problems4
Tape Problems

5 August 2004:

  • Restore jobs had to wait for an input tape that was being used by another restore job

  • Delay

Dieter W. Storr -- www.storrconsulting.com


Tape problems5
Tape Problems

30 October 2004:

  • Packages didn’t arrive in time, due to a thunderstorm that affected FedEx delivery

  • Major delay

Dieter W. Storr -- www.storrconsulting.com


Tape problems6
Tape Problems

30 October 2004:

  • Automated tape library experienced unit address problems during the restore process

  • Delay

Dieter W. Storr -- www.storrconsulting.com


Tape problems7
Tape Problems

30 October 2004:

  • VTS logical tapes were not shipped to Wood Dale (HSM level 2, SAR level 2)

  • Delay

Dieter W. Storr -- www.storrconsulting.com


Tape problems8
Tape Problems

30 October 2004:

  • Confusion about when to load DRP1 and DRP2 tapes, before or after IPL

  • Delay

Dieter W. Storr -- www.storrconsulting.com


Tape problems9
Tape Problems

30 October 2004:

  • ICIS libraries were not backed up to tape

  • Application tests were not possible

Dieter W. Storr -- www.storrconsulting.com


Tape problems10
Tape Problems

8 December 2004:

  • Load problems

  • Tapes were loaded before IPL and not after IPL

  • Major delay

Dieter W. Storr -- www.storrconsulting.com


Tape problems11
Tape Problems

8 December 2004:

  • Experienced problems when trying to restore MIG1 data, e.G. DRADABC0 job

  • Major delay

Dieter W. Storr -- www.storrconsulting.com


Tape problems12
Tape Problems

8 December 2004:

  • Recall sent by FedEx tapes to SunGard

  • One damaged package arrived without tapes

  • Restored DATA one generation back (-1)System was generation (0)

Dieter W. Storr -- www.storrconsulting.com


Tape problems13
Tape Problems

21 March 2005:

  • Level 2 tapes for VTS not being sent off-site (but have been on the list)

  • Application team couldn’t test all data

Dieter W. Storr -- www.storrconsulting.com


Tape problems14
Tape Problems

5 August 2005:

  • 3590-1 cartridges ejected, not found

    • DSS8370W - TMS SHOWS TAPE N00318 OUT OF AREA “DRP1”,SLOT 00031

  • Delay

Dieter W. Storr -- www.storrconsulting.com


Time Warner employee data missing

May 2, 2005: 5:51 PM EDT

NEW YORK (CNN) - Time Warner Inc. said Monday that data on 600,000 current and former employees stored on computer backup tapes was lost by an outside storage company and that the Secret Service is now investigating.

Dieter W. Storr -- www.storrconsulting.com


Lost Backup Tape Held Ameritrade Client Data

Wednesday, April 20, 2005 - LA Times

… package was damaged during shipping between vendors ….. fourth tape is still missing…… The tapes may have included customers’ Social Security numbers …..

Dieter W. Storr -- www.storrconsulting.com


Info On 3.9M Citigroup Customers Lost

Monday, June 6, 2005 – CNN.COM

Citigroup, the nation's biggest financial services company, said that UPS lost the tapes while shipping them to a credit bureau in Texas.

Dieter W. Storr -- www.storrconsulting.com


Costs for tape backups
Costs for Tape Backups

  • SunGard recovery services

  • Offsite tape storage

  • Tape handling

  • Shipping per test

  • Special extra pick-ups

Yearly $150,000

Dieter W. Storr -- www.storrconsulting.com


Costs
Costs

  • Not capable to restore one day

    • $$ ???

  • Last December: 2 weeks to rebuild manually (?) customer tables

  • Does it make sense to restore more than 2 days back ??

Dieter W. Storr -- www.storrconsulting.com


Costs1
Costs

Example:

20 employees x $140 per day x 10 days

= $28,000

And they couldn’t work on other projects

$140 is based on $51,100 yearly income

Dieter W. Storr -- www.storrconsulting.com


Quantitative risk analysis single loss expectancy
Quantitative Risk Analysis Single Loss Expectancy

  • SLE = Single Loss Expectancy

  • EF = Exposure Factor, for example 50% or .50

  • AV = Asset Value, for example $1,000,000

SLE = EF * AV

SLE = .5 x $1,000,000 = $500,000

Dieter W. Storr -- www.storrconsulting.com


B r methods5
B/R Methods

Reducing tapes

Dieter W. Storr -- www.storrconsulting.com


B r methods6
B/R Methods

Reducing tapes

  • Stacking datasets to 3590-1 cartridges

  • Using Delta Save Facility from ADABAS

Dieter W. Storr -- www.storrconsulting.com


B r methods7
B/R Methods

Reducing tapes

  • Using Forward Index Compression (FIC) from ADABAS

  • Using larger block size for 3590 tapes = 256K, supported by ADABAS

Dieter W. Storr -- www.storrconsulting.com


Delta Save Facility (DSF)

Dieter W. Storr -- www.storrconsulting.com


Delta Save Facility

Dieter W. Storr -- www.storrconsulting.com


B r methods8

Within an index block the part of the index value that is identical to the forward part of the previous index value is suppressed.

B/R Methods

Forward Index Compression

Rochester Gas & Electric

Space savings:

  • Normal Index: 37% - 55%

  • Upper Index: 21% - 69%

Dieter W. Storr -- www.storrconsulting.com


B r methods9
B/R Methods identical to the forward part of the previous index value is suppressed.

IBM Magstar 3494 / Virtual Tape Server (VTS)

SunGard

LA Times

Dieter W. Storr -- www.storrconsulting.com


B r methods10
B/R Methods identical to the forward part of the previous index value is suppressed.

VTS problems

LA Times:

  • Completion code A78 RC 18

  • We switched from VTS to 3590-1 cartridges

Dieter W. Storr -- www.storrconsulting.com


B r methods11
B/R Methods identical to the forward part of the previous index value is suppressed.

VTS problems

Virginia Information Technologies Agency:

  • Ran 2003/2004 into the same problem system completion code A78 RC 18

  • We … converted … to 3490/3590 physical tapes

  • Problem solved

Dieter W. Storr -- www.storrconsulting.com


B r methods12
B/R Methods identical to the forward part of the previous index value is suppressed.

Disk to Disk

  • Mirroring

    • Hardware

    • Software

  • Replicating

    • Software

Dieter W. Storr -- www.storrconsulting.com


B r methods enterprise server
B/R Methods – Enterprise Server identical to the forward part of the previous index value is suppressed.

Enterprise Server

NT / 2000 / XP

Hot Site

UNIX

Dieter W. Storr -- www.storrconsulting.com


B r methods open system
B/R Methods – Open System identical to the forward part of the previous index value is suppressed.

Hot Site

Dieter W. Storr -- www.storrconsulting.com


B r methods13
B/R Methods identical to the forward part of the previous index value is suppressed.

Marty Stewart

Disaster Recovery Manager

AnMed Health:

“…we’d rather have a server that’s running slower than having no server at all.”

Dieter W. Storr -- www.storrconsulting.com


Disk mirroring
Disk Mirroring identical to the forward part of the previous index value is suppressed.

ASSO

Benefits

  • Asynchronous disk mirroring can provide better physical protection by supporting extended physical distances.

  • No loss of committed transactions in synchronous storage (mirroring/RAID) on a CPU failure

DATA

ASSO

DATA

Dieter W. Storr -- www.storrconsulting.com


Disk mirroring1
Disk Mirroring identical to the forward part of the previous index value is suppressed.

ASSO

Limitations

  • No protection from data corruption

  • Secondary site is not guaranteed to betransitionally consistent, in the case of asynchronous mirroring.

  • Client application must be re-started after failure and need to be aware of failure

DATA

ASSO

DATA

Dieter W. Storr -- www.storrconsulting.com


Disk mirroring2
Disk Mirroring identical to the forward part of the previous index value is suppressed.

ASSO

Limitations

  • Synchronous mirroring and RAID devices can add overhead to application performance.

  • Redundant/specialized high availability hardware/software can be expensive and restricted to use for backup purposes only.

DATA

ASSO

DATA

Dieter W. Storr -- www.storrconsulting.com


Disk mirroring3
Disk Mirroring identical to the forward part of the previous index value is suppressed.

ASSO

Limitations

  • Secondary copy of data is not available for use – low hardware utilization.

  • Need to replicate everything on disk, no selectivity of data replication

DATA

ASSO

DATA

Dieter W. Storr -- www.storrconsulting.com


Example for disk mirroring
Example For Disk Mirroring identical to the forward part of the previous index value is suppressed.

Back Up / Hot Site

S/390

UNIX

EMC 5700

SRDF remote mirroredsynchronized

OC-3 link

SRDF remote mirroredsynchronized

12-15 miles

EMC 5700

S/390

UNIX

Main Platform

Dieter W. Storr -- www.storrconsulting.com


B r methods14
B/R Methods identical to the forward part of the previous index value is suppressed.

  • Can we buy used Enterprise Servers?

  • Yes…..and inexpensive

  • OP system is free for D/R

Search for “selling used mainframes,” for example:

http://www.used-line.com/fdc3236-find-dealer.htm

http://www.azure.co.uk/

etc.

Dieter W. Storr -- www.storrconsulting.com


Dedicated line broadband speeds and prices
Dedicated line broadband speeds and prices identical to the forward part of the previous index value is suppressed.

  • T-1 - 1.544 megabits per second (24 DS0 lines) Ave. cost $400.-$650./mo.

  • T-3 - 43.232 megabits per second (28 T1s) Ave. cost $6,000.-$16,000./mo.

  • OC-3 - 155 megabits per second (100 T1s)Ave. cost $20,000.-$45,000./mo.

  • OC-12 - 622 megabits per second (4 OC3s) no price

  • OC-48 - 2.5 gigabits per seconds (4 OC12s) no price

  • OC-192 - 9.6 gigabits per second (4 OC48s) no price

    Source: http://www.infobahn.com/research-information.htm

    prices updated: 12 May 2005

Dieter W. Storr -- www.storrconsulting.com


Peer-to-Peer Remote Copy Extended Distance (PPRC-XD) identical to the forward part of the previous index value is suppressed.

PPRC = 60 miles - PPRC-XD = continent

FlashCopy

ESS Shark

- IBM ESS DASD - HDSalso support PPRC

ESS Shark

Also see TimeFinder from EMC

Dieter W. Storr -- www.storrconsulting.com


External back up systems
External Back-up Systems identical to the forward part of the previous index value is suppressed.

Fast Copy of Data

  • Snapshot

    • No data movement

    • A virtual copy by copying pointers

  • Copy Process

    • Physical copy async. from the log. copy

    • No impact on applic. on the original data

Dieter W. Storr -- www.storrconsulting.com


External back up systems1
External Back-up Systems identical to the forward part of the previous index value is suppressed.

Fast Copy of Data

  • Specific Hardware Required

    • Software works only with the hardware

  • Work on Volume Level

    • Some snapshot only tools work also on dataset level

Dieter W. Storr -- www.storrconsulting.com


Snapshot physical copy
Snapshot & Physical Copy identical to the forward part of the previous index value is suppressed.

IBM

  • Hardware: Enterprise Storage Server

  • Software: FlashCopy

    http://www.share.org/proceedings/sh98/data/S3087.PDF

    EMC2

  • Hardware: Symmetrix Remote Data Facility

  • Software: EMC TimeFinder

    http://www.emc.com/interactive_center/media/timefinder/tf_noRC.html

Dieter W. Storr -- www.storrconsulting.com


Flash copy
Flash Copy identical to the forward part of the previous index value is suppressed.

Dieter W. Storr -- www.storrconsulting.com


How it works
How It Works identical to the forward part of the previous index value is suppressed.

Pre-defined time window

Read only: update requests are queued

Suspend

Resume

Read

only

Read / update

Read / update

snap

Source

Data

Snapshot

Physical

Backup

Source: SAG

ADADBS TRANSACTIONS SUSPEND,TTSYN=60,TRESUME=120

Dieter W. Storr -- www.storrconsulting.com


Replication
Replication identical to the forward part of the previous index value is suppressed.

ASSO

DATA

Benefits

  • Warm standby systems can be configured over a Wide Area Network, providing protection from site failures.

  • Ability to more quickly swap to the standby system in the event of failure, as backup database is already on-line.

  • Data corruptionis typically not replicated as transactions are logically reproduced rather than I/O blocks mirrored.

WORK

WORK

DATA

ASSO

Dieter W. Storr -- www.storrconsulting.com


Replication1
Replication identical to the forward part of the previous index value is suppressed.

ASSO

DATA

Benefits

  • Automatic switch over for clients using a switching mechanism, no client restart needed.

  • Originating applications are minimally impacted as replication takes place asynchronously after commit of the originating transaction.

  • The warm standby database is available for read-only operations, allowing better utilization of backup systems.

WORK

WORK

DATA

ASSO

Dieter W. Storr -- www.storrconsulting.com


Replication2
Replication identical to the forward part of the previous index value is suppressed.

ASSO

DATA

Benefits

  • Ability to resynchronize and easily switch back to primary system when it becomes available without loss of data.

WORK

WORK

DATA

ASSO

Dieter W. Storr -- www.storrconsulting.com


Replication3
Replication identical to the forward part of the previous index value is suppressed.

ASSO

DATA

Limitations

  • Warm standby system will be out-of-date by transactions committed at the active database that have not been applied to the standby.

  • Protection is limited to components supporting Warm Standby (e.g. DBMS data sources may be protected but file systems may not be supported).

WORK

WORK

DATA

ASSO

Dieter W. Storr -- www.storrconsulting.com


Entire transaction propagator
Entire Transaction Propagator identical to the forward part of the previous index value is suppressed.

  • The Entire Transaction Propagator allows for asynchronous data replication.

  • Replicated data can be updated and synchronized with master data at user specified intervals.

Dieter W. Storr -- www.storrconsulting.com


Adabas data replication
ADABAS Data Replication identical to the forward part of the previous index value is suppressed.

  • Logical dissemination of ADABAS Data to homogeneous or heterogeneous targets

  • Near real time propagation

  • Event driven at the Transaction level

  • Implemented at the Database/file level for Store, Delete and Update commands

  • Define Replication rules through subscriptions

  • Minimal Impact on normal nucleus activity

  • Strategic for Enterprise Data Sharing

  • Replace Entire Transaction Propagator

Dieter W. Storr -- www.storrconsulting.com


Adabas data replication1

Target identical to the forward part of the previous index value is suppressed.

Field

Origin

DBMS

Target

Target

Field

File

ADABAS Data Replication

z/OS

Image C

z/OS Image A

File

Target

Target

z/OS

Image B

Unix

Server D

Table

DBMS

Dieter W. Storr -- www.storrconsulting.com


Possible hot site solutions
Possible Hot Site Solutions identical to the forward part of the previous index value is suppressed.

Enterprise Server Los Angeles

Shark

Shark

EMC

Converter

ESCON

FICON

Fiber Optic

OC3

OC3

OC3

Shark

EMC

EMC

Own Enterprise Server Hot Site

Dieter W. Storr -- www.storrconsulting.com


Costs for tape backups1
Costs for Tape Backups identical to the forward part of the previous index value is suppressed.

  • SunGard recovery services

  • Offsite tape storage

  • Tape handling

  • Shipping per test

  • Special extra pick-ups

Yearly $150,000

Dieter W. Storr -- www.storrconsulting.com


Costs for real disaster
Costs for Real Disaster identical to the forward part of the previous index value is suppressed.

  • SunGard Declaration Fee

  • D/R Site Daily Usage Fee

  • Office Space Daily Usage Fee

  • Work Group Declaration Fee

  • Work Group Daily Usage Fee

  • LAN Bridge Declaration Fee

  • LAN Bridge Daily Usage Fee

30 Days $475,000

Dieter W. Storr -- www.storrconsulting.com


Costs for own hot site
Costs for Own Hot Site identical to the forward part of the previous index value is suppressed.

  • Used IBM Z800-0X2 Mainframe

  • Used IBM 2105-F20 Shark Storage

  • Used IBM 3494 Library, VTS and Tape Drives

  • 3rd Party Next Day HW Maintenance

  • Printer and Terminal Controller Re-location Costs

  • 3490 Tape Drive and Controller Re-location Costs

  • Other Costs

1st Year $520,000After 5 Years Total $735,000

Dieter W. Storr -- www.storrconsulting.com


Costs for own hot site1
Costs for Own Hot Site identical to the forward part of the previous index value is suppressed.

5 Years SunGard = $750,000

30 Days Real Disaster = $475,000

5 Years Own Facility = $735,000

Dieter W. Storr -- www.storrconsulting.com


Restore times min
Restore Times (Min) identical to the forward part of the previous index value is suppressed.

Dieter W. Storr -- www.storrconsulting.com


Benefits of own hot site
Benefits of Own Hot Site identical to the forward part of the previous index value is suppressed.

  • Financial savings > $150,000 annually providing an almost 5% ROI

  • Reduced recovery time

  • Reduced impact due to road and airport closures

  • Elimination of reliance on external vendors

  • Mainframe and open system can use the same facility

Dieter W. Storr -- www.storrconsulting.com


Grid computing
Grid Computing identical to the forward part of the previous index value is suppressed.

virtual machine

virtual memory

virtual storage

virtual I/O

Dieter W. Storr -- www.storrconsulting.com


Dieter W. Storr -- www.storrconsulting.com identical to the forward part of the previous index value is suppressed.


Grid computing1
Grid Computing identical to the forward part of the previous index value is suppressed.

Interface

User

ComputerElement(s)

Passkey

ResourceBroker

HardwareSoftware

Locations

StorageElement(s)

Information Service

ReplicaCatalog

Dieter W. Storr -- www.storrconsulting.com


Grid computing2
Grid Computing identical to the forward part of the previous index value is suppressed.

Dieter W. Storr -- www.storrconsulting.com


Grid computing3
Grid Computing identical to the forward part of the previous index value is suppressed.

  • BBC builds distributed grid for content sharing (Gridcast).

62.73.167.57/publicdocs/ppt/prompeg307.ppt

file:///C:/My%20Documents/Dieter/my%20presentations/prompeg307.ppt

Dieter W. Storr -- www.storrconsulting.com


Grid computing for backup
Grid Computing For Backup? identical to the forward part of the previous index value is suppressed.

  • Intra or Extra Grid?

  • Pull or Push?

  • Grid Software

http://www.gridcomputing.com/

http://www.gridforum.org/ggf_grid_understand.htm

http://gridcafe.web.cern.ch/gridcafe/animations.html

Dieter W. Storr -- www.storrconsulting.com


Backup methods
Backup Methods identical to the forward part of the previous index value is suppressed.

Mostly used by other companies

Source: DRJ Magazine

  • VTS

  • Disk to disk

    • Is more and more common for enterprise storage servers and AIX server technology, for example.

      Source: @server Magazine

Dieter W. Storr -- www.storrconsulting.com


B r methods15
B/R Methods identical to the forward part of the previous index value is suppressed.

Problems for other companies

  • High third-party hot site costs, approx. $10,000 - $70,000 per month

  • Restore time24-30 hours

Dieter W. Storr -- www.storrconsulting.com


How far is far enough http www drj com articles spr03 1602 02 html
How Far is ‘Far Enough?’ identical to the forward part of the previous index value is suppressed.(http://www.drj.com/articles/spr03/1602-02.html)

  • Alternate Facility

  • Offsite Storage Facility

    Answer = 105 miles

    …so the survey

Dieter W. Storr -- www.storrconsulting.com


Lessons learned http www drj com articles spr02 1502 07 html
Lessons Learned identical to the forward part of the previous index value is suppressed.(http://www.drj.com/articles/spr02/1502-07.html)

  • Distance is keyStreets, bridges, tunnels, airports are closed

  • Tape recovery is not effective

  • All applications are critical

  • Inconsistent back-up is no back-up at all

Dieter W. Storr -- www.storrconsulting.com


Lessons learned http www drj com articles spr02 1502 07 html1
Lessons Learned identical to the forward part of the previous index value is suppressed.(http://www.drj.com/articles/spr02/1502-07.html)

  • People-dependent processes do not suffice

  • Two sites are not enough

  • People are hard to replace but information is irreplaceable

Dieter W. Storr -- www.storrconsulting.com


We should have an excellent hot site
…..we should have an excellent HOT SITE! identical to the forward part of the previous index value is suppressed.

Dieter W. Storr -- www.storrconsulting.com


ad