In2p3 hpss migration report v 5 1 6 2 2 2
This presentation is the property of its rightful owner.
Sponsored Links
1 / 21

IN2P3 HPSS Migration report V 5.1  6.2.2.2 PowerPoint PPT Presentation


  • 81 Views
  • Uploaded on
  • Presentation posted in: General

IN2P3 HPSS Migration report V 5.1  6.2.2.2. Pierre-Emmanuel Brinette IN2P3 Computing Center (Lyon – France) [email protected] Overview. Context of the migration What’s new in HPSS 6.2 IN2P3 HPSS Configuration Migration Planning Metadata Migration Conclusion. Context of migration.

Download Presentation

IN2P3 HPSS Migration report V 5.1  6.2.2.2

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


In2p3 hpss migration report v 5 1 6 2 2 2

IN2P3 HPSS Migration reportV 5.1  6.2.2.2

Pierre-Emmanuel Brinette

IN2P3 Computing Center (Lyon – France)

[email protected]


Overview

Overview

Context of the migration

What’s new in HPSS 6.2

IN2P3 HPSS Configuration

Migration Planning

Metadata Migration

Conclusion

Pierre-Emmanuel Brinette


Context of migration

Context of migration

  • IN2P3 use HPSS as main MSS since 1999.

  • The last major upgrade (to v 5.1) has been made in 2005.

  • Migration to HPSS 6.2 planned since 2007:

    • HPSS 6.2.1 test platform setup in mid 2007.

  • IN2P3 HPSS staff has changed in 2008.

    • The engineers that manage HPSS leave the center.

    • The new team has no experience in HPSS.

  • In 2009 upgrade to HPSS 6.2 is necessary

    • Support for T10k-B drives.

    • Remove End-Of-Life equipments.

Pierre-Emmanuel Brinette

Wednesday, August 27, 2014

3


Overview1

Overview

  • Context of the migration

  • What’s new in HPSS 6.2

  • IN2P3 HPSS Configuration

  • Migration Planning

  • Metadata Migration

  • Conclusion

Pierre-Emmanuel Brinette

Wednesday, August 27, 2014

4


What s new in hpss 6 2 2 2

What’s new in HPSS 6.2.2.2

  • Major changes:

    • DCE is no longer used as security mechanism.

    • Some changes in the HPSS API.

  • New Authentication/Authorization mechanisms:

    • Unix/Unix : /etc/passwd or NIS

    • Kerberos/LDAP.

  • Officially supported platform:

    • AIX 5.3 and RHEL 4.7.

    • DB2 v 9.1.

Pierre-Emmanuel Brinette

Wednesday, August 27, 2014

5


Overview2

Overview

  • Context of the migration

  • What’s new in HPSS 6.2

  • IN2P3 HPSS Configuration

  • Migration Planning

  • Metadata Migration

  • Conclusion

Pierre-Emmanuel Brinette

Wednesday, August 27, 2014

6


In2p3 hpss configuration early 2009

HPSS

HPSS-API

SRB

RFIO

xrootd

dcache

IN2P3 HPSS Configuration (early 2009)

  • 25 M files / 6 Pb

  • + 2000 users

  • Main access protocol is RFIO

  • HPSS is used as tape backend for others storage systems:

    • xrootd: 1 M files /1 Pb

    • SRB used for files transfers

    • dcache 2 M files / 2 Pb for grid

Local users only

Local and remote users

Grid users (SRM)

Pierre-Emmanuel Brinette

Wednesday, August 27, 2014

7


In2p3 hpss configuration early 20091

IN2P3 HPSS Configuration (early 2009)

  • HPSS 5.1

    • 42 disk movers

    • 2600 disk volumes

    • 28 tape movers

    • 66 drives

    • Gbits LAN

Pierre-Emmanuel Brinette

Wednesday, August 27, 2014

8


Overview3

Overview

  • Context of the migration

  • What’s new in HPSS 6.2

  • IN2P3 HPSS Configuration

  • Migration Planning

  • Metadata Migration

  • Conclusion

Pierre-Emmanuel Brinette

Wednesday, August 27, 2014

9


Migration planning

Migration Planning

  • Migration goal

    • Clean-up configuration

    • Remove EOL hardware

    • Decrease the number of volumes

Migrate to HPSS 6.2

Migrate to HPSS 6.2

Migration to HPSS 6

Migrate to HPSS 6.2

Migrate to HPSS 6.2

Pierre-Emmanuel Brinette

Wednesday, August 27, 2014

10


Migration planning1

Migration Planning

New

10 Gb

  • Migration goal

    • Introduce new hardware

    • Use 10 Gbits for mover

New

New

New

Pierre-Emmanuel Brinette

Wednesday, August 27, 2014

11


Migration planning2

Migration Planning

  • IBM Migration requirements

    • Migration procedure applied on the production server.

    • Metadata migration procedure assume a 1 core server and a 1 subsystem.

    • Empty Disk volumes.

    • UID/GID in DCE consistent with new auth/authz system (ie NIS).

Pierre-Emmanuel Brinette

Wednesday, August 27, 2014

12


Migration planning3

Migration Planning

  • IN2P3 CC constraints

    • Change the HPSS Core server and remove secondary server (EOL)

    • Use NIS for Auth/Authz

    • Modify metadata of 4 subsystems

    • Drastically reduce the number of Disk Volumes/hpss mover/servers

    • Introduce the new tape movers

Pierre-Emmanuel Brinette

Wednesday, August 27, 2014

13


Migration planning4

Migration Planning

(Non linear time line)

Dec 08

Feb 09

Apr 09

May 1, 09

May 15, 09

Jun 1, 09

Jun 5, 09

Migration case study with IBM Support

HPSS 6.2 Test platform setup

Writing migration procedure

HPSS 6.2 metadata migration platform setup

Clean-up production config

Setup new production server

Migration

Pierre-Emmanuel Brinette

Wednesday, August 27, 2014

14


Migration planning5

Migration Planning

(Non linear time line)

May 25

May 31

Jun 1

Jun 2

Jun 3

Jun 4

Jun 5

Reduce disk volume

Stop HPSS access

Purge disk

Shutdown HPSS

Export & Convert DB2 metadata

Import in new server

Configure new server

HPSS mover installation

RFIO Installation

Tests

Pierre-Emmanuel Brinette

Wednesday, August 27, 2014

15


Overview4

Overview

  • Context of the migration

  • What’s new in HPSS 6.2

  • IN2P3 HPSS Configuration

  • Migration Planning

  • Metadata Migration

  • Conclusion

Pierre-Emmanuel Brinette

Wednesday, August 27, 2014

16


Metadata migration

Metadata migration

  • HPSS 6.2 introduce new DB2 Tablespace containers scheme

  • Use a production metadata backup after HPSS shutdown

  • On a temporary HPSS systems:

    • Create exactly the same raw devices than the ones used on the production server

    • Restore the backup, run metadata conversion script, and export DB content

  • On the new production system:

    • Create the databases with HPSS install procedure (mkhpss)

    • Populate the databases with the DB content just exported

Pierre-Emmanuel Brinette

Wednesday, August 27, 2014

17


Overview5

Overview

  • Context of the migration

  • What’s new in HPSS 6.2

  • IN2P3 HPSS Configuration

  • Migration Planning

  • Metadata Migration

  • Conclusion

Pierre-Emmanuel Brinette

Wednesday, August 27, 2014

18


Conclusion

Conclusion

  • Migration issues

    • Even with a good preparation we have encountered (small) issues during the migration.

    • Details available on demand.

  • Benefits

    • Supported HPSS version.

    • Stability improved.

    • Configuration simplified.

    • Metadata queries now runs faster.

    • Gain lot of experience for the new staff !

Pierre-Emmanuel Brinette

Wednesday, August 27, 2014

19


Conclusion1

Conclusion

  • After migration

    • Repack of remaining 9940 tapes

    • Remove of last PowderHorn library

    • Repack of T10k-A  T10kB (Big files SC)

    • Repack 9840  T10k-A Sport

  • Future

    • HPSS 7.1 migration in 2010

    • 4th SL8500 with T10K-B

    • Add more disk (probably DDN)

Pierre-Emmanuel Brinette

Wednesday, August 27, 2014

20


In2p3 hpss migration report v 5 1 6 2 2 2

Thanks for your attention

Pierre-Emmanuel Brinette

Wednesday, August 27, 2014

21


  • Login