TeraGrid Archive Replication and Federation Phil Andrews et al

1. TeraGrid Archive Replication and Federation Phil Andrews et al There is nothing more powerful than an idea whose time has come -Victor Hugo …system is in crisis, but nothing will be done before it enters chaos – C. Everett Koop

2. For those who came in late… • In response to encouragement from Steve Meacham, we submitted a proposal, oriented towards Stimulus Funding, to provide a TG replication service. 6 Hardware partners, ~$12M total, primarily in first year. Providing for non-XD funded archives a major concern • Between submission and review, the Stimulus Funding possibility was removed • 16 (17?) member panel was divided on proposal, not recommended for funding. Will describe comments in this talk • Hope to resubmit, spread over 3-4 years, 2-3$M /year The best-laid schemes o' mice an 'men gang aft agley – Robert Burns TeraGrid Quarterly, Sep’09

3. Panel Comments on Proposal: • 5 “Excellent”, 8 “Very Good”, 2 “Good”, one thought a solicitation was essential • Lack of solicitation →significant confusion • Some thought it unnecessarily ambitious • Some thought it insufficiently ambitious A compromise is the art of dividing a cake in such a way that everyone believes he has the biggest piece. -Ludwig Erhard TeraGrid Quarterly, Sept’09

4. Proposal: Confusion and response • Two reviewers though it dependent on XD • Two didn’t understand SDSC’s position • Some didn’t understand why not solicited • Need to “set the scene” better • Need help from NSF The people may be made to follow a path of action, but they may not be made to understand it. -Confucius TeraGrid Quarterly, Sept’09

5. Proposal: too ambitious? • Reviewers didn’t like spending HDW $ early • Some didn’t think all data needed saving • Some didn’t think any data needed saving • Need to establish requirements (survey?) • Need to better justify data archiving New knowledge is the most valuable commodity on earth. The more truth we have to work with, the richer we become. -Kurt Vonnegut TeraGrid Quarterly, Sept’09

6. Proposal: not ambitious enough? • Reviewers didn’t think 4 years was enough • Concerns on software plan/value added • Need help from NSF (either more than 4 years, or why not) • Need more concrete software plan • Need more value added I have no spur to prick the sides of my intent, but only vaulting ambition… -Macbeth TeraGrid Quarterly, Sept’09

7. Proposal realignments: • No Stimulus $: no need for “front loading” • Less concern for non-XD funded archives at SDSC and NCSA, eliminates one motivation • Need a more strategic approach based on long-term HPC sustainability • What will be the future TG archival requirements and policies? • What do we want to be when we grow up? The unexamined life is not worth living - Socrates TeraGrid Quarterly, Sept’09

8. Is Archival Storage part of the TG mission? • Traditionally TG has been cycles oriented • Archival will take bigger slice of the future budget • Sites (prediction) will tend to “unload” archival • User Requirement for archive will not disappear • DataNet doesn’t seem to the answer • Archival must be separated from individual compute competitions to make full shrift • A concerted approach is in all our best interests If not us, who? If not now, when?”- J.F.K. TeraGrid Quarterly, Sept’09

9. Early NICS survey results TeraGrid Quarterly, Sep’09