hdf5 tools update
Download
Skip this Video
Download Presentation
HDF5 Tools Update

Loading in 2 Seconds...

play fullscreen
1 / 37

HDF5 Tools Update - PowerPoint PPT Presentation


  • 253 Views
  • Uploaded on

HDF5 Tools Update. Peter Cao - The HDF Group [email protected] November 6, 2007.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' HDF5 Tools Update' - uta


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
hdf5 tools update

HDF5ToolsUpdate

Peter Cao - The HDF Group

[email protected]

November 6, 2007

This report is based upon work supported in part by a Cooperative Agreement with NASA under NASA NNX06AC83A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Aeronautics and Space Administration.

outline
Outline

Overview of current tools

New tools in HDF5

HDF-Java 2.4 release

hdf5 command line tools
HDF5 Command Line Tools

Readers

  • h5dump, h5diff, h5ls
  • new tools: h5check, h5stat
  • Writers
    • h5repack, h5repart, h5import, h5jam/h5unjam
    • new tool: h5copy, h5mkgrp
  • Converters
    • h4toh5, h5toh4, gif2h5, h52gif
where to start
Where to start

Tools located at bin/ directory

  • Use “-h” option for help
  • Online help http://www.hdfgroup.org/hdf5tools.html
  • Email to THG help desk [email protected]
h5dump
h5dump

Dumps file content to stdout in

  • ASCII text
  • XML
  • Binary

4

h5dump h sds h5
h5dump -H SDS.h5

HDF5 "SDS.h5" {

GROUP "/" {

GROUP "Floats" {

DATASET "FloatArray" {

DATATYPE H5T_IEEE_F32LE

DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) }

}

}

DATASET "IntArray" {

DATATYPE H5T_STD_I32LE

DATASPACE SIMPLE { ( 5, 6 ) / ( 5, 6 ) }

}

}

}

h5dump d floats floatarray sds h5
h5dump -d /Floats/FloatArray SDS.h5

HDF5 "SDS.h5" {

DATASET "/Floats/FloatArray" {

DATATYPE H5T_IEEE_F32LE

DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) }

DATA {

(0,0): 0.01, 0.02, 0.03,

(1,0): 0.1, 0.2, 0.3,

(2,0): 1, 2, 3,

(3,0): 10, 20, 30

}

}

}

h5dump binary output
h5dump Binary Output

-b F, --binary=[MEMORY|FILE|LE|BE]

h5dump -d /IntArray -o out_le.bin -b LE SDS.h5

000000000000000 00000001 00000002 00000003 00000004 00000005

0000030 0000000a 0000000b 0000000c 0000000d 0000000e 0000000f

0000060 00000014 00000015 00000016 00000017 00000018 00000019

0000110 0000001e 0000001f 00000020 00000021 00000022 00000023

0000140 00000028 00000029 0000002a 0000002b 0000002c 0000002d

h5diff
h5diff

Show Differences

Between

two objects or two files

h5diff sds h5 sds2 h5
h5diff SDS.h5 SDS2.h5

Dataset: </IntArray> and </IntArray>

5 differences found

report differences
Report differences

Dataset: </IntArray> and </IntArray>

position IntArray IntArray difference

------------------------------------------------------

[ 0 0 ] 0 10 10

[ 1 0 ] 10 100 90

[ 2 0 ] 20 200 180

[ 3 0 ] 30 300 270

[ 4 0 ] 40 400 360

5 differences found

h5diff SDS.h5 SDS2.h5 -r /IntArray

h5repack
h5repack

Copies a file to a new file with

  • Compression filter
  • Data layout
  • Other user options
h5repack new filters
h5repack: new filters

-f{GZIP|SZIP|SHUF|FLET|NBIT|SOFF|NONE}

For example

h5repack -i SDS.h5 -o out.h5 -f GZIP=6

h5repack data layout
h5repack: data layout

-l {CHUNK| COMPA| CONTI}

For example

h5repack -i SDS.h5 -o out.h5 -l CHUNK=2x3

h5repack latest version of file format
h5repack: latest version of file format

The testing file was created with HDF5 1.6.5. The root group of the file contains 100 groups. Each group at the root contains 9 sub groups. Each sub group contains one compound dataset. The compound dataset has 40 members of 32 bit integer

h5repart
h5repart

Repartitions a file into a family of files

For example

h5repart -m 200m int16kx16k.h5 part200m%d.h5

part200m0.h5

200 MB

977 MB

200 MB

part200m1.h5

200 MB

part200m2.h5

200 MB

part200m3.h5

177 MB

part200m4.h5

h5import
h5import

Imports binary/ASCII data into an HDF5 file

  • h5import infile -c config_file [infile -c config_file2 ...] -outfile outfile

For example

h5import float5x4x2.txt -c First_set.conf -o First_set.h5

GROUP "/" {

GROUP "work" {

DATASET "First-set" {

DATATYPE H5T_IEEE_F64LE

DATASPACE SIMPLE { ( 5, 2, 4 ) / ( 8, 8, H5S_UNLIMITED ) }

DATA {

(0,0,0): 1.01, 1.02, 1.03, 1.04,

(0,1,0): 1.11, 1.12, 1.13, 1.14,

(1,0,0): 1.21, 1.22, 1.23, 1.24,

(1,1,0): 1.31, 1.32, 1.33, 1.34,

(2,0,0): 1.41, 1.42, 1.43, 1.44,

}

}

}

}}

PATH work/First-set

INPUT-CLASS TEXTFP

RANK 3

DIMENSION-SIZES 5 2 4

OUTPUT-CLASS FP

OUTPUT-SIZE 64

OUTPUT-ARCHITECTURE IEEE

OUTPUT-BYTE-ORDER LE

CHUNKED-DIMENSION-SIZES 2 2 2

MAXIMUM-DIMENSIONS 8 8 -1

h5jam h5unjam
h5jam/h5unjam
  • h5jam

h5jam -u test_ub.txt -i test_ub.h5

  • h5unjam

h5unjam -i test_ub.h5 -o out_ub.txt -o out_ub.h5

User Block

HDF5

Jam

unjam

User Block

HDF5

slide20
h5ls
  • Lists selected information about file objects in the specified format

For Example,

h5ls -r SDS2.h5

  • /Floats Group
  • /Floats/DoubleArray Dataset {10, 5}
  • /Floats/FloatArray Dataset {4, 3}
  • /Floats/subs Group
  • /IntArray Dataset {5, 6}
gif2h5 h52gif
gif2h5 / h52gif

gif2h5 – converts a GIF file into HDF5

gif2h5apollo17_earth.gif apollo17_earth.h5

h52gif – converts an HDF5 file into GIF

h52gifapollo17_earth.h5 apollo17_earth2.gif -i /apollo17_earth.gif/Image0 -p "/apollo17_earth.gif/Global Palette"

h5toh4 h4toh5
h5toh4 / h4toh5

h5toh4 -- Converts an HDF5 file to an HDF4 file

h4toh5 -- Converts an HDF4 file to an HDF5 file

http://www.hdfgroup.org/h4toh5/

new tools
New tools

h5copy

h5mkgrp

h5stat

h5check

h5copy
h5copy

Copies an object within a file/cross files

/

/

Floats

IntArray

FloatArray

FloatArray

h5copy f shallow
h5copy-f shallow

/

floats

-f shallow

/

f32

64-bit

floats

integers

/

i1

i2

f32

64-bit

floats

f1

f2

f32

64-bit

f1

f2

h5copy f soft
h5copy-f soft

/

/

f1

dset_SL

/f1

-f soft

f1

dset_SL

/f1

/

dset_SL

/f1

h5copy f ref
h5copy-f ref

/

d1

d2

/

dset_ref

-f ref

d2

d1

/

dset_ref

dset_ref

h5mkgrp
h5mkgrp

Creates groups

h5mkgrp test.h5 /info /data /data/test1 /data/test2

/

info

data

test1

test2

h5stat
h5stat

Prints statistics about HDF5 file

  • Header size
  • Raw data size
  • Attributes, datatypes, and etc.

Helps

  • To troubleshoot overhead in HDF5 files
  • To choose specific object’s properties and storage strategies
h5check
h5check

A validation tool that verifies if an HDF5 file is encoded according to the HDF5 File Format Specification

why h5check
Why h5check?
  • Verify if the file is compliant with the File Format
  • As a verification tool required by the application of HDF5 File Format to be an ANSI standard
  • Serves as a watch dog that the HDF5 library implementation is compliant with the File Format
example a compliant file
Example: a compliant file
  • % h5check example1.h5
    • VALIDATING example1.h5
    • FOUND super block signature
    • VALIDATING the super block at 0...
    • VALIDATING the object header at 928...
    • VALIDATING the btree at 384...
    • FOUND btree signature.
    • VALIDATING the local heap at 96...
    • FOUND local heap signature.
    • Result: File is in compliance.
example a non compliant file
Example: a non-compliant file

h5check invalid2.h5

  • FOUND super block signature
  • VALIDATING the super block at 0...
  • VALIDATING the object header at 928...
  • VALIDATING the btree at 384...
  • FOUND btree signature.
  • VALIDATING the SNOD at 1248...
  • FOUND SNOD signature.
  • VALIDATING the object header at 976...
  • check_sym(at 1248): Errors from check_obj_header()
  • decode_validate_messages(): Failure in type->decode().
  • H5O_sdspace_decode(): Bad version number in simple dataspace message.
  • VALIDATING the local heap at 96...
  • FOUND local heap signature.
  • Main(): Errors from check_obj_header().
  • decode_validate_messages(): Failure in type->decode().
  • H5O_attr_decode(): Can\'t decode attribute dataspace.
  • H5O_sdspace_decode(): Bad version number in simple dataspace message.
  • Result: File is not in compliance.
implementation status
Implementation Status

Basic file format components

1.6 - DONE

1.8 -IN PROGRESS

hdf java 2 4
HDF-Java 2.4

http://www.hdfgroup.org/hdf-java-html

ftp://ftp.hdfgroup.org/HDF5/hdf-java

HDF-Java 2.4 Released !!

hdf java 2 4 release
HDF-Java 2.4 Release
  • Build with HDF4 2r2 and HDF5 1.6.6
  • New platforms

Mac intel

Linux 64-bit AMD

Solaris 64-bit

  • 36 new enhancements and 44 bugs fixed
  • Fixed memory leak
  • Test suite
  • Enhanced documentation
memory leak in old version
Memory leak in old version

while (true)

{

H5File f = new H5File

(fname, H5File.READ);

f.open();

f.close();

}

ad