Report of the BERAC ARM Facility Review Panel Joyce E. Penner, Eugene W. Bierly, Robert Dickinson, Charles Miller1, Nelson A. Seaman1, Anne-Marie Schmoltner2, Paul Try2 1Non-ARM funded 2 Participated in 2005 review
Charge: Assess the effectiveness of the ARM facilities as a national user facility • How well are they serving the needs of the open (non ARM-funded) user community? • What factors are driving the costs of maintaining and operating the facilities? • Were critical recommendations from the 2005 review implemented?
ARM Background • ARM program funds ARM research and ARM infrastructure (sites and services) • 2004: ARM Climate Research Facility designated as a national user facility • ACRF Science Board reviews requests for field programs at the fixed sites and use of the Mobile Facility • ACRF does not fund research, but provides instrumentation, data, and expertise
ARM as a User Facility: External Users • The climate modeling community gains from knowledge generated by the data taken at ARM: Cloud/aerosol interactions; Convective precipitation • Non-ARM funded direct use of the facility: • Mobile facility: 3 out of last 4 users were not ARM-funded • 60% of accepted proposals are non ARM-funded P.I.’s • 1500 users; 931 use archive, 69% are non-funded • Several satellite validation exercises benefit NASA, NOAA
ARM as a User Facility: Possible extensions • Initiate programs to train new scientists (advanced graduate students and beginning post-docs) in the use of the ACRF • Initiate a process of outside review of the documentation of data provided in the archive
Cost Effectiveness of the ARM facility: Metrics • Metrics for cost effectiveness useful for DOE and OMB, but don’t tell the whole story • Scientific excellence metrics useful to capture the impact of the ACRF on the national and international research community.
Cost Metrics • Cost/Product: goal is $100 per product file (counted as the highest level product) (has trended down, but leveled off) • Leveraged Science Costs: Since external users not coming out of ARM, a measure of the costs of these campaigns would show broader impact
Science Metrics • Uptime (98%) or Number of Instrument Operation Hours • Number of Publications Citing ARM or ARM Data: 883 during 2002-2006 • Number of Proposals Received/Number of Proposals Approved (68%); would be nice to have stratified by cost • Only 17% of Mobile Facility requests approved: Strong unaccomodated desire • User and Data Use Statistics: 931 data users: need better feedback about available documentation and need error bars on products
Cost effectiveness and trends • Largest cost factor is labor: • Have worked to keep this cost category flat and/or declining: Savings of $6M compared to projections using 3% inflation • Have also trained staff to maintain instruments (cost savings relative to contracting this out) • Cost saving in internet services through use of higher technology
ACRF is approaching a period of optimal productivity • Risk that ARM Infrastructure will not be able to sustain continued growth in service to the user community under the current funding profile • Scope of ACRF responsibilities has increased with their support of climate data records: Must emphasize data quality
Implementation of recommendations from prior review (2005) • Management team established a formal process to address each recommendation: • Emphasis was given to: • improvement of data quality, • documentation and dissemination, • enhancement of communications to the user community
Web Technology, Data Products and Information • Massive improvement to the ARM web site and organization of data • A more pro-active approach to feedback from users would help • Data quality especially error characteristics could be made more accessible • Linking and cross-cutting of data sets in time could be improved • Grouping of data sets for particular problems has started and should continue
Other recommendations • Revival of the Instrument Development Program is probably not necessary • The criteria for deployment is certainly better enunciated especially with the Mobile Facility • Constructive outreach is being pursued: • A limited number (one to three) of signature data sets should be advertised • More focused attention could be given to applications of the world-wide, regionally diverse, collective data from the ACRF sites to global climate research • Communication of ACRF Accomplishments needs to continue
Other recommendations • Data quality efforts have been expanded, though no easy way to track the improvements • Management/Leadership approaches appear to be evolving to remain effective though we did not spend effort on this in the review
Conclusions • ACRF is effectively used by the broader scientific community • The ACRF Management has worked aggressively to decrease costs of running the ARM facility: can widen metrics to better quantify the value of the facility • ACRF management also aggressively pursued implementation of the 2005 review recommendations, though we had a few suggestions for further improvement