NCL Introduction - PowerPoint PPT Presentation

Gabriel
slide1 l.
Skip this Video
Loading SlideShow in 5 Seconds..
NCL Introduction PowerPoint Presentation
Download Presentation
NCL Introduction

play fullscreen
1 / 47
Download Presentation
NCL Introduction
1139 Views
Download Presentation

NCL Introduction

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. NCL Introduction Dennis Shea National Center for Atmospheric Research

  2. Workshop Overview • Objective • comfortable with NCL; minimize learning curve • workshopwill notmake you an expert • access, process and visualize data Labs important: you must invest the time • “osmosis” method of learning does not work • Schedule • day 1: Introduction, File I/O, [lunch] , Demo, Lab • day 2: Graphics [,Python], [lunch] , Web Tour, Lab • day 3: Functions and Data Analysis: Lab • day 4: Lab

  3. Introduction to netCDF NCL variable model is based on the netCDF variable model

  4. netCDF files self describing • (ideally)all info contained within the file • no external information needed to determine file contents portable[machine independent] • Supported by many software tools / languages • NCL, IDL, Matlab, ferret, GrADS, F, C, C++,Java • command line file operators: NCO, CDO • perform various tasks very efficiently on files • http://nco.sourceforge.net • http://www.mpimet.mpg.de/fileadmin/software/cdo • ncview • quick visualization: COARDS • http://meteora.ucsd.edu/~pierce/ncview_home_page.html Many modelling groups use netCDF [IPCC]

  5. Examining a netCDF file • ncdump file_name | less • dumps the entire contents of a file; prints every value • ncdump -h file_name | less • Dumps header information [most commonly used] • NCL equivalent: ncl_filedump file_name | less • ncdump -v U file_name | less • NCL equivalent: ncl_filedump –v U file_name| less • Note: ncdump is a Unidata utility • not a netCDF Operator (NCO / CDO) • not associated with NCL • ncview: visualize file contents [COARDS conven] • ncl_filedump file_name [ more general ] • netCDF3/4, GRIB-1,GRIB-2, HDF, HDF-EOS [HDF5]

  6. Parts of netCDF file ncdump –h foo.nc (or ncl_filedump foo.nc) DIMENSION SIZES dimensions: lat = 64 lon = 128 time = 12 VARIABLES: Names , Types, Attributes, Coordinate Variables variables: floatlat(lat) lat:long_name = "latitude" lat:units = "degrees_north" floatlon(lon) lon:long_name = "longitude" lon:units = "degrees_east" inttime(time) time:long_name = "time" time:units = "Month of Year" doubleT(time, lat, lon) T:long_name = “Temperature” T:units = “C" T:missing_value = 1.e+20f T:_FillValue = 1.e+20f time=UNLIMITED (12 currently) FILE ATTRIBUTES global attributes: title = “Temp: 1999” source = “NCAR” conventions = "None” exercise: ncl_filedump FOO.nc | less

  7. 4.35 4.39 0.27 -3.35 -6.90 4.36 4.66 3.77 -1.66 4.06 9.73 -5.84 0.89 8.46 10.39 17 3.68 5.08 0.14 -5.63 -0.63 -4.12 -2.51 1.76 -1.43 -4.29 0.07 5.85 0.87 8.65 x netCDF/NCL variable • array[could be of length 1 (scalar)] • (may) have additional information name: x type: float [real] shape: 2-dimensions size: 7 (rows) x 5 (columns) values:x(2,3) = 8.46 [row major] long_name: “Temperature” units: “C” named dimensions: x(time,lat) lat: (/ -60, -30 ,0, 30, 60 /) time: (/2000,2001,2002,2003,2004,2005,2006 /) Meta data

  8. visual: simple 2D netCDF Variable coordinate variables (rectilinear grid) • attributes @: • long_name • units Latitude coordinate variable (1D, &) Longitude coordinate variable (1D, &) • NCL is NOT LIMITEDto netCDF conforming variables • eg: 2D coordinate arrays (curvilinear coords)

  9. X Scalar or Array attributes attributes long_name _FillValue units add_offset scale_factor etc. long_name _FillValue units add_offset scale_factor etc. accessed via @ accessed via & X coord var values time lev lat lon etc. coordinates Scalar or Array time lev lat lon etc. netCDF [NCL] Variable model f = addfile(“foo.nc”,”r”) ; grb/hdf x = f->X NCL reads the scalar/array, attributes, and coordinate variables as an object

  10. Detailed Look netCDF Variable (NCL) ncl<return> ; interactive mode ncl0 >f =addfile("UV300.nc", "r") ; open file ncl1 >u = f->U; import STRUCTURE ncl2 >printVarSummary(u) ; overview of variable Classic netCDF Variable Model Variable: u Type: float Total Size: 65536 bytes 16384 values Number of Dimensions: 3 Dimensions and Sizes: [time | 2] x [lat | 64] x [lon | 128] Coordinates: time: [ 1 .. 7 ] lat: [ -87.8638 .. 87.8638 ] lon: [ 0 .. 357.185] Number of Attributes: 5 _FillValue : 1e36 units : m/s long_name : Zonal Wind Component short_name : U missing_value : 1e36 NCL syntax/funcs query use modify add any aspect of data object

  11. Why Know NCL/netCDF Variable Model • variables imported (read) from nc, grib, hdf, hdf-eos • will have the same [consistent] structure • powerful programming aid • facilitates writing netCDF and HDF files • facilitates writing robust functions • functions can query, add, use meta data • some functions access/use the meta data • eg: gsn_csm graphic functions • aware of common netCDF conventions • will query and use for labels and mapping

  12. pdf Vis5D NCL Overview Integrated processing environment • freeware: supported, public domain • portable: linux/unix, windows (cygwin), MacOS • general purpose: unique capabilities • excellent graphics (2D, limited 3D)

  13. NCL Support • Documentation and Examples • http://www.ncl.ucar.edu/ • numerous downloadable examples to get you going • downloadable reference manuals [pdf], FAQ ncl-talk@ucar.edu users must subscribe http://www.ncl.ucar.edu/Support/ncl_talk.shtml [haley@ucar.edu , shea@ucar.edu]

  14. NCAR Command Language • Complete Programming Language • data types • variables • operators • expressions • conditional statements • loops • functions/procedures/graphics • Features • query / manipulate meta data • import data in a variety of formats • array syntax / operations • can use user fortran/C codes and commercial libraries • most functions/procedures ignore missing data

  15. Running NCL • Interactive Mode (Command line) • ncl[options][command-line-arguments]<return> ncl> enter commands ncl> quit <return> • can save interactive commands ncl> record “file_name” ncl> stop record • Batch Mode[ .ncl suffix is optional] • ncl[options][arguments]script.ncl • ncl< script.ncl[also acceptable] • ncl[options][arguments]script.ncl >&!out • ncl[options][arguments]script.ncl >&!out& • appending "&" means put in background • note: the >&!& are csh andtcshsyntax

  16. NCL predefined options • ncl –hnxV [predfined options are preceded by dash] • may be used for interactive or batch mode • informational • ncl –h[display predefined options and usage and exit] • ncl –V[print the NCL version and exit] • action • ncl –x[echo statements as encountered (debug)] • ncl –n[don't enumerate dimensions of values in print()] • multiple predefined options • ncl –nx[ not ncl –n –x ]

  17. Sample Batch Script: sample.ncl load “$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl“ load“$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl“ begin ; optional f1 = addfile("TMP_58-97.nc", "r") ; open netCDF file T = f1->Tmp ; T(time,lev,lat,lon) =>(480,17,73,144) f2=addfile(“P_1958-1997.grb", "r") ; open GRIB file P = f2->Pres ; P(time,lev,lat,lon) f3=addfile(“Q_1958-1997.hdfeos", "r") ; open hdf –eos file Q = f3->Specific_Humidity ; Q(time,lev,lat,lon) pot = T* (1.+0.622*Q) *(1000/P)^0.286; potential temperature (array) print("pot: min="+ min(pot) + " max="+ max(pot) ) wks =gsn_open_wks("ps", “sample") ; open a graphic workstation gsn_define_colormap (wks,"gui_default") ; change from default plot =gsn_csm_contour_map_polar (wks,T(0,5,:,:),False) res = True;change default plot res@cnFillOn = True; use colors res@gsnSpreadColors = True; use entire color map plot = gsn_csm_pres_hgt (wks,T(0,:,{50},:),res) end ; only if begin is present

  18. Language Basics

  19. Outline: NCL Language Basics • special syntax characters • data types • Variables  netCDF/NCL variable model • attributes • _FillValue • named dimensions • coordinate variables • print and printVarSummary • shaping • subscripting

  20. NCL Syntax Characters • ;- comment [can appear anywhere] • @- reference/create attributes • ! - reference/create named dimension • & - reference/create coordinate variable • {…}- coordinate subscripting • $ - enclose strings when (im/ex)port variables via addfile • (/../)- array construct characters • : - array syntax • |- separator for named dimensions • \- continue character [statement to span multiple lines] • ::- syntax for external shared objects (eg, fortran/C) • ->- use to (im/ex)port variables via addfilefunction

  21. Numeric double (64 bit) float (32 bit) long (32 or 64 bit) integer (32 bit) short (16 bit) byte ( 8 bit, 0-255) complex NOT supported non-Numeric string character graphic file logical list Data Types

  22. Simple Variable Assignment a_int = 1 , a_float = 2.0 [2e-5] a_double = 3.2d a_string = "a” , a_logical = True [False] • array constructor characters (/…/) • a_integer = (/1,2,3/) • a_float = (/1.0, 2.0, 3.0/) • a_double = (/1., 2 , 3.2 /) *1d5 • a_string = (/"abcd","e", "Hello, World”/) • a_logical = (/True, False,True/) • a_2darray = (/(/1,2,3/), (/4,5,6/), (/7,8,9/) /)

  23. Conversion between data types • NCL is a‘strongly typed’ language • coercion • implicit conversion of one type to another • automatic coercion when no info is lost • let i be integer and x be float or double • fortran: x=i and i=x • NCL: x=i and i=floattointeger(x) • many functions to perform conversions

  24. Attributes • info about a variable or file [meta data] • attributes can be any data type except file or list • scalar, multi dimensional array (string, numeric) • assign/access with@character • T@units = “deg C” • T@wgts = (/ 0.25, 0.5, 0.25 /) • T@x2d = (/(/1,2,3/), (/4,5,6/), (/7,8,9/) /) • T@_FillValue = -999 • title = x@long_name • attribute functions [isatt, getfilevaratts] • if (isatt(T,"units")) then .... end if • atts = getfilevaratts (fin, "T") • deletecan eliminate an attribute • delete(T@title)

  25. _FillValue • Unidata and NCLreservedattribute • most NCL functions ignore_FillValue • Note: “missing_value” attribute: no special status to NCL • if “T” has “missing_value” attribute but no “_FillValue ” • NCL creates: T@_FillValue = T@missing_value • use built-in function “ismissing” to check for _FillValue • if (any(ismissing(T))) then … end if • NOTE: if (any(T.eq.T@_FillValue)) will not work • netCDF Operators[NCO]: _FillValue attribute • ncview: missing_value attribute • best to create netCDF files with both • NCL: best to not use zero as a _FillValue • OK except when contouring [random bug]

  26. Interactive: Variable_1 • http://www.ncl.ucar.edu/Training/Workshops/interactive.shtml - demonstrate interactive mode • illustrate simple calculations and function use • first view of meta data [attributes]

  27. Arrays • row major ….. like C/C++ • left dimension varies slowest; right varies fastest • dimension numbering left to right [0,1,..] • indicies [subscripts] are zero based [0,N-1] Consider T(:,:,:) left dimension is 0 [ T!0 ] middle dimension is 1 [ T!1 ] right dimension is 2 [ T!2 ]

  28. NCL (netCDF) Dimensions • may be “named” • provides alternative way to reference subscripts • recommendation: always name dimensions • use NCL syntax • assigned with ! character{let T(:,:,:)} • T!0 = "time" ; leftmost [slowest varying] dim • T!1 = "lat“ • T!2 = "lon" ; rightmost [fastest varying] dim • dim names may be renamed, retrieved • T!1 = "LAT" … dName = T!2 • delete can eliminate: delete (T!2) • Named dimensions used to reshape

  29. NCL (netCDF) Coordinate Variables • data coordinates [ eg: time, level, lat, lon ] • strict netCDF definition • 1D numeric array • monotonically in[de]creasing numeric values • can only be assigned to a named dimension • 1D array must be the same size as dimension • should not have _FillValue attribute • coordinate values assigned via &character • T&lat = latitude • used in graphics, coordinate subscripting • coordinate functions • iscoord , isfilevarcoord • deletecan eliminate coordinate array • delete(T&time)

  30. Create and Assign Coordinate Variables • create 1D array • time = (/1980, 1981, 1982 /) ; integer • lon = ispan(0, 355, 5)*1.0 ; integer->float • assign dimension name [same as variable name] • time!0 = “time” • lon!0 = “lon” • assign values to named dimension • time&time = time • lon&lon = lon • let x be 2D: name dimensions • x!0 = “time” … x!1 = “lon” • assign coordinate variables to x • x&time = time … x&lon = lon

  31. Access/Change/Create/Delete Meta Data • @ attributes • u@long_name = "U" • lonName = u@long_name • !named dimensions • u!0 = "TIME" • tName = u!0 • &coordinate variable • u&lat = (/ -90., -85, .... , 85., 90. /) • latitude = u&lat • $ substitute string • x = fin->$variable(n)$ …x = fin->$"T: p"$

  32. X Scalar or Array attributes attributes long_name _FillValue units add_offset scale_factor etc. long_name _FillValue units add_offset scale_factor etc. accessed via @ accessed via & X coord var values time lev lat lon etc. coordinates Scalar or Array time lev lat lon etc. netCDF [NCL] Variable model f = addfile(“foo.nc”, “r”) ; grb/hdf x = f->X NCL reads the scalar/array, attributes, and coordinate variables as an object

  33. Variable Assignment • Variable-to-Variable assignment • consider y = xwhere x is previously defined • if y not defined: • y has same type/shape/values/meta data as x • if y predefined: • y must have same shape and type • or, x must be coerceible to the type of y • y attributes, dimension names and coordinate variables, if they exist, will be over written • Value-only assignment(no meta copy) • U multi-dimensional array with meta data • Uval = (/ U /) or Uval = (/f->U/) • the (/ ... /)operator pair strips meta data

  34. Interactive: Variable_2 • http://www.ncl.ucar.edu/Training/Workshops/interactive.shtml • named dimensions • create a variable data-object [structure] from scratch • standard and coordinate subscripting • reordering via named dimensions

  35. Variable Reshaping/Reordering • functions may require data in specific order • map plot functions want array order T(...,lat,lon) • can and shouldbe done without loops • use NCL syntax or functions • very fast for variables in memory • how? … two approaches: let T(time,lat,lon) • named dimensions: t = T(lat|:, lon|:, time|:) • NCL functions: • ndtooned:t1D =ndtooned(T) • onedtond:t2D =onedtond(t1D, (/N,M/) )

  36. Variable Subscripting (1 of 3) Standard Array Subscripting • ranges: start/end and [optional] stride • indices separated by : • omitting start/end index implies default begin/end Consider T(time,lat,lon) T  entire array [ don't use T(:,:,:) ] T(0,:,::5)  1st time index, all lat, every 5th lon T(0,::-1,:50)  1st time index, reverse lat order, 1st 51 lon T(:1,45,10:20)  1st 2 time indices, 46th index of lat, 10-20 indicies of lon

  37. Variable Subscripting (2 of 3) Coordinate Variable Subscripting • only applies to coordinate variables • same rules apply for ranges, strides, defaults • use curly brackets {…} • standard and coordinate subs can be mixed [if no reorder] T(:,{-30:30},:)  all times/lon, lat -30° to +30° (inclusive) T(0,{-20},{-180:35:3})  1st time, lat nearest - 20°, every 3rd lon between -180° and 35°

  38. Variable Subscripting (3 of 3) Named Dimensions • only used for dimension reordering • indicated by | • dim names must be used for each subscript • named/coordinate subscripting can be mixed Consider T(time,lat,lon) t = T(lat|:, lon|:, time|:)  makes t(lat,lon,time) t = T(time|:,{lon|90:120},{lat|-20:20})  all times, 90-120° lon, -20-20° lat

  39. Standard and Coordinate Subscripting Standard: T(9:13,1:8) Coordinate: T({-10:20},{-170:-110}) Latitude coordinate variable (1D) Combined: T({-10:20}, 1:8) Longitude coordinate variable (1D)

  40. Interactive: Variable_3 • http://www.ncl.ucar.edu/Training/Workshops/interactive.shtml • standard and coordinate subscripting • reorder via named dimensions • shape remains the same [same number of dimensions] • reshape via ndtooned and onedtond • number of dimensions changes

  41. “printing” • printVarSummary • gives gross overview of a variable • print • same info as printVarSummary • prints values • write_matrix • print to standard out or a file • format control of numerical output • can write to file also

  42. printVarSummary • Print out variable (data object) information • type • dimension information • coordinate information (if present) • attributes (if present) • printVarSummary(u) Variable: u Type: double Total Size: 1179648 bytes 147456 values Number of Dimensions: 4 Dimensions / Sizes: [time | 1] x [lev | 18] x [lat | 64] x [lon | 128] Coordinates: time: [4046..4046] lev: [4.809 .. 992.5282] lat: [-87.86379 .. 87.86379] lon: [ 0. 0 .. 357.1875] Number of Attributes: 2 long_name: zonal wind component units: m/s

  43. print (1 of 3) • Prints out all variable information including • meta data, values • T(lat,lon): print (T) Variable: T Type: float Total Size: 32768 bytes 8192 values Number of Dimensions: 2 Dimensions / Sizes: [lat | 64] x [lon | 128] Coordinates: lat: [-87.86379 .. 87.86379] lon: [ 0. 0 .. 357.1875] Number of Attributes: 2 long_name: Temperature units: C (0,0) -31.7 (0,1) -31.4 (0,2) -32.3 (0,3) -33.4 (0,4) -31.3 etc. [entire T array will be printed]

  44. print (2 of 3) • can be used to print asubsetof array • meta data, values • T(lat,lon): print( T(:,103) ) orprint( T(:,{110}) ) • Variable: T (subsection) • Type: float • Total Size: 256 bytes • 64 values • Number of Dimensions: 1 • Dimensions / Sizes: [lat | 64] • Coordinates: • lat: [-87.86379 .. 87.86379] • Number of Attributes: 3 • long_name: Temperature • units: C • lon: 109.6875[ added ] • (0) -40.7 • (1) -33.0 • -25.1 • -20.0 • (4) -15.3 etc.

  45. print (3 of 3) • printwith embedded strings • no meta data • print ( "min(T)="+min(T)+" max(T)="+max(T) ) (0) min(T)=-53.8125 max(T)=25.9736 • sprintf andsprinti provide formatting • often used in graphics • print ( "min(T) = "+ sprintf("%5.2f", min(T)) ) (0) min(T) = -53.81 • sprintican left fill with zeros (ex: let n=3) • fnam = "h" + sprinti ("%0.5i", n) + ".nc" • print("file name = "+fnam) (0) file name = h00003.nc

  46. write_matrix(x[*][*], fmt, opt) • pretty-print 2D array (table) to standard out • integer, float, double • user format control (fmt) • T(N,M), N=7, M=5: write_matrix (T, “5f7.2”, False) 4.35 4.39 0.27 -3.35 -6.90 4.36 4.66 3.77 -1.66 4.06 9.73 -5.84 0.89 8.46 10.39 4.91 4.59 -3.09 7.55 4.56 17 3.68 5.08 0.14 -5.63 -0.63 -4.12 -2.51 1.76 -1.43 -4.29 0.07 5.85 0.87 8.65 • can also create an ASCII file opt = True opt@fout = “foo.ascii” ; file name write_matrix (T, “5f7.2”, opt)