NARR User Issues

1. Wesley Ebisuzaki CPC/NCEP Wesley.Ebisuzaki@noaa.gov http:wwwt.emc.ncep.noaa.gov/mmb/rreanl/ NARR User Issues

2. Topics • Files: contents, availability, formats • Tools: BUFR, GRIB • Obtaining the data: NCEP, NCAR, NCDC • Scripting downloads • What is reality? Is reanalysis truth? An insider's viewpoint.

3. GRIB Files • For grids (horizontal fields) such as 500 mb Z • Lambert conformal grid: 32 km eta grid (#221) • Have grids of lat/lon, display programs • Model grid: not available except at NCEP • No plans to make available, display programs? • Analyses are on-line and forecasts ? • 99% of people will use the GRIB files

4. BUFR • For all input observations except SST, snow, ice, observed precipitation (GRIB) • Input obs BUFR files have QC marks • Analyses/forecasts at station locations (MOLTS) • Much smaller than GRIB files, limited locations • Distribution (informal) • Includes restricted data • Few users (experience from global reanalysis) • Often other sources of data (in easier formats)

5. Misc Files • Restart files: huge, full precision, NCEP-only • Statistics: • Fit of obs to NARR, fit of obs to R2 • Small, can be made available on-line • Observation counts: • Small, should be put on-line • PNG (monitoring plots) • Small, should be put on-line

6. NARR has zero funding. Want more files?Don't expect too much.

7. Despair? • Global Reanalysis has been running this way for years! • CPC has been running global reanalysis for its own operations. CPC and the outside community have shared interest in maintaining a stable assimilation system. NCEP has an interest in others using its products. • However, limited resources, need to be selective, work with partners

8. Status • Real-time is running (4 days late) • Started building 1.5-2 year on-line rotating archive of latest analyses at NCEP • 1979-2003 merged files at NCAR, NCDC • NCDC has 5 TB merged files on-line, trying to support the on-line community by NOMADS • EMC people are working on other projects

9. Tools: BUFR • Situation improving • GrADS can read bufr directly

10. Tools: GRIB • GrADS: display and computations • grib2ctl.pl: make GrADS control files • wgrib: inventory, decode GRIB files, remove records from a file (save space) • lcgrib: regional subset (save space) • copygb: change grid (save space or neurons) • grbpoint.pl: grib -> point value printout (slow) • Gempak (?), NCL (?)

11. Example: GrADS • Make control file (analyses) • grib2ctl.pl -rr cpc_rr_mon.197901 >my.ctl • grib2ctl.pl -rr cpc_rr_mon.%y4%m2 >my.ctl • Make index file (analyses) • gribmap -0 -i my.ctl • Run GrADS • grads • open my.ctl

12. Example: wgrib • Use: inventories, make files smaller, decode data for fortran and C programs • See contents of a grib file • wgrib (options) cpc_mon_197901 • To extract records, decode records, see wgrib home page.

13. Example: lcgrib • Regional subset of Lambert conformal grid • *** MAKE SMALLER FILES *** • lcgrib in-grib out-grib west south east north • West, south, east and north are in degrees • N and E are +ve, S and W are -ve

14. Obtaining Data: NCEP • Rotating archive: latest 1.5 – 2 years of analyses • http://nomad7.ncep.noaa.gov/ncep_data • Other files: TBD • NCEP not set up to deal with the public requests • Go through the web • Go through NCDC, NCAR

15. Obtaining Data: NCAR • Have 1979-2003 merged data • Sure they will get more of the data • Plans, availability?

16. Obtaining Data: NCDC • NCDC has the merged data 1979-2003 and want 2004 ASAP • NCDC will use NOMADS to distribute the data to the on-line community. • NOMADS is project of many institutions including NCDC and NCEP for distributing data over the internet. • NOMADS-NCDC is taking software from NCEP and COLA to distribute GRIB data.

17. Data Access at NCDC • Custom plots • ftp4u: access by selected time, field, region • http access to raw data (like FTP) • GDS (GrADS Data Server): OpeNDAP server (formerly DODS), on-line access to data

18. Difficulties in on-line distribution • Downloading a 5 TB data set is difficult • Need ways to minimize data transfer • Plots (maps) work but limited utility • GDS needs to be optimized for large datasets • ftp4u is a solution

19. FTP4U • Only download data that you need • Select levels • Select variables • Select geographic regions • Select times

20. Making FTP4U Work • Don't request more data than needed • Make one big request, not many small requests of the same file; i.e., make one request to get the winds, temp and heights rather than 3 requests. Do not request long time series. • Shared resource (1 job at a time) • Be prepared to script/automate the downloads • May be impossible to download in a few hours, may require days, weeks and multiple jobs/requests

21. Scripting FTP4U Downloads • FTP4U is normally accessed by a browser • Anything a browser can do, a script can also do • Experience at NCEP is that most plot/ftp4u access is scripted (automated) • NCDC's modifications to the NCEP provided programs make it more difficult to script the download but nothing that a cURL user couldn't handle. NCDC wants to change!

22. Scripting Downloads cont'd • Use cURL, a unix/windows program, to “emulate” a browser • Each browser page becomes a call to cURL. • With a little RTFM was able get script working • sample program will be put on NARR home page

23. Scripting Downloads cont'd • Downside: steep learning curve, may have no local experts for help • Upside: can make large data transfers possible, can be used in other projects (ex. getting fcst data automatically, skewing on-line opinion polls) • Given that we get more data off the web, it's a skill that someone in a large group should eventually have.

24. Obtaining Data: NCDC • It's a learning situation for NCDC • Give feedback (complain), it's the only way to improve the situation. Lessons learned here will help the next time you try getting a different dataset from NCDC's nomad server. • Current issues: full disk (new RAID is will be installed soon), obtaining time series, scripting support

25. What is Reality? • Some people consider reanalysis to be truth • NARR is new, bound to be warts, already found some.

26. What is Reality? • Accuracy of reanalysis depends on many factors • Data going into the analyses • Types, counts, quality of observations • Accuracy of previous analysis • Accuracy of atmospheric/land model • Spatial and temporal filtering • NO EASY ANSWER

27. What is Reality? cont'd • Accuracy • check to the appropriate data counts and biases • Need to add our counts to web • Need to add fit to data to web • check other reanalyses • Fit to observations • Consider sensitivity to modeling assumptions • Sensitivity experiments (data, resolution, model), only available to limited # of people

28. “Trust but verify” Ronald Reagan What is Reality? cont'd

29. URLs • copygb: ftp://ftpprd.ncep.noaa.gov/pub/cpc/wd51we/copygb • change grid type of grib files • copygb: ftp://ftpprd.ncep.noaa.gov/pub/cpc/wd51we/copygb • change grid type of grib files • cURL: http://curl.haxx.se • download URLs, handles post data • GEMPAK: http://my.unidata.ucar.edu/content/software/gempak/index.html • graphics, can handle NARR files with some effort • GrADS: http://grads.iges.org/grads/ • graphics, computations

30. URLs • grbpoint.pl: ftp://ftpprd.ncep.noaa.gov/pub/cpc/wd51we/wgrib.scripts • simple program to extract data from lat/lon • grib2ctl.pl: http://www.cpc.ncep.noaa.gov/products/wesley/grib2ctl.html • makes control files for GrADS • g95: http://www.g95.org/ • f95 compiler for copygb • HIPHOP: http://iacweb.etzh.ch/stall/dominik/hihop/ • grib support for idl • lcgrib: http://www.cpc.ncep.noaa.gov/products/wesley/gribw.html • regional subset of Lambert conformal grib files

31. URLs • NARR: http://wwwt.emc.ncep.noaa.gov/mmb/rreanl/index.html • home page for NARR • NARR rotating archive: http://nomad7.ncep.noaa.gov/ncep_data • NCAR Command Language (NCL): http://ngwww.ucar.edu/ • graphics, computations package that supports grib • NCDC data: http://nomads.ncdc.noaa.gov/#narr_datasets • on-line supplier of NARR data • wgrib: http://www.cpc.ncep.noaa.gov/products/wesley/wgrib.html • inventory, slice & dice grib files