slide1 n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Output of UM Diagnostics Directly in CF NetCD; an end to costly data conversions PowerPoint Presentation
Download Presentation
Output of UM Diagnostics Directly in CF NetCD; an end to costly data conversions

Loading in 2 Seconds...

play fullscreen
1 / 11

Output of UM Diagnostics Directly in CF NetCD; an end to costly data conversions - PowerPoint PPT Presentation


  • 77 Views
  • Uploaded on

NCAS C omputational M odelling S ervice (CMS) Group providing services to the UK academic modelling community. Output of UM Diagnostics Directly in CF NetCD; an end to costly data conversions. Jeff Cole, Grenville Lister, NCAS-CMS, Dept. of Meteorology, University of Reading.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Output of UM Diagnostics Directly in CF NetCD; an end to costly data conversions' - gabe


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

NCAS Computational Modelling Service (CMS) Groupproviding services to the UK academic modelling community

Output of UM Diagnostics Directly in CF NetCD; an end to costly data conversions

Jeff Cole, Grenville Lister, NCAS-CMS, Dept. of Meteorology, University of Reading

slide2

NCAS Computational Modelling Service (CMS) Groupproviding services to the UK academic modelling community

  • Typical workflow
  • Run the model to create 64-bit fields-file output
  • Convert fields files to 32-bit pp (delete fields files)
  • Convert pp data to CF NetCDF (all of it, ad hoc, delete pp files?)
  • Analyze, visualize, process NetCDF data
  • Problems?
  • Low volumes of data - probably not
  • Large volumes of data – greater potential for inefficiency
slide3

NCAS Computational Modelling Service (CMS) Groupproviding services to the UK academic modelling community

  • Cascade
  • 80TB of pp data to NetCDF
  • conversion rate to NetCDF 10hr/TB on the HECToR NERC post processor
  • 800hrs converting pp-NetCDF (doesn’t include fields-file – pp conversion)
  • Model data production 10 wall clock hrs/TB
  • Leeds experience
  • Faster conversions - 4hr/TB
  • “As far as I'm concerned the main issue is the amount of extra disk space the conversion takes up. It essentially doubles what is needed because I am never 100% sure the conversion has worked properly so I never delete the original pp files.”
slide4

NCAS Computational Modelling Service (CMS) Groupproviding services to the UK academic modelling community

Objectives

  • Directly write UM STASH output in NetCDF format
  • Implement the whole UMUI STASH panel
  • Convert PP file metadata into CF metadata
  • Support standard and mean PP files
  • Support NetCDF versions 3 and 4, including data compression
  • Support multiple UM versions
slide5

NCAS Computational Modelling Service (CMS) Groupproviding services to the UK academic modelling community

Coding decisions

  • All NetCDF library calls contained in one f90 module
  • Minimize changes to existing UM code
  • All model metadata is written when NetCDF file is first opened
  • Use STASH domain and time profile names as vertical and time coordinate names
  • Standard names and units are taken from file STASH_to_CF.txt, produced by David Hassell
slide6

NCAS Computational Modelling Service (CMS) Groupproviding services to the UK academic modelling community

Writing metadata

Subroutine ncfile_init(called from initial)

Loop1: loop over Fortran units

if PP file is to be output on this unit

Open NetCDF file

Write global attributes

Loop2: loop over all possible STASH codes

if this STASH code is to be written to a PP file

Write horizontal dimension information (ncfile_write_horiz_dim)

Write vertical dimension information (ncfile_write_vert_dim)

Write time dimension information (ncfile_write_time_dim)

Create NetCDF variable

Save NetCDF variable id, to be used when writing data

Write NetCDF variable attributes

slide7

NCAS Computational Modelling Service (CMS) Groupproviding services to the UK academic modelling community

Writing metadata

Loop3: loop over Fortran units

if PP file is to be output on this unit

Put NetCDF file into data mode

Write horizontal dimension data (ncfile_write_horiz_var)

Write vertical dimension data (ncfile_write_vert_var)

Write time dimension data (ncfile_write_time_var)

slide8

NCAS Computational Modelling Service (CMS) Groupproviding services to the UK academic modelling community

netcdf_mod

ncfile_init

ncfile_write_data

stwork

initial

Atmosphere

slide9

NCAS Computational Modelling Service (CMS) Groupproviding services to the UK academic modelling community

NetCDF timings

HadGEM3A, vn8.2, N96L85, 3 day run, 118 diagnostics, 9 PP files

slide10

NCAS Computational Modelling Service (CMS) Groupproviding services to the UK academic modelling community

slide11

NCAS Computational Modelling Service (CMS) Groupproviding services to the UK academic modelling community

To Do

  • Finish implementing CF attributes (bounds, cell_methods)
  • More work needed on time dimension
  • Complete STASH panel options (meaning, grid subsection, timeseries)
  • Add support for mean PP files
  • Put in code so PP output works when files are re-initialised and for CRUNs
  • Add support for writing NetCDF data in I/O Server code
  • Add UMUI panel to select NetCDF options and code to read NetCDF namelist variables
  • Add support for NetCDF4 data compression
  • Port code to other UM versions, including adding support for the old ocean model
  • Comprehensive testing