Hdf5 tools update
This presentation is the property of its rightful owner.
Sponsored Links
1 / 37

HDF5 Tools Update PowerPoint PPT Presentation


  • 83 Views
  • Uploaded on
  • Presentation posted in: General

HDF5 Tools Update. Peter Cao - The HDF Group [email protected] November 6, 2007.

Download Presentation

HDF5 Tools Update

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Hdf5 tools update

HDF5ToolsUpdate

Peter Cao - The HDF Group

[email protected]

November 6, 2007

This report is based upon work supported in part by a Cooperative Agreement with NASA under NASA NNX06AC83A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Aeronautics and Space Administration.


Outline

Outline

Overview of current tools

New tools in HDF5

HDF-Java 2.4 release


Hdf5 command line tools

HDF5 Command Line Tools

Readers

  • h5dump, h5diff, h5ls

  • new tools: h5check, h5stat

  • Writers

    • h5repack, h5repart, h5import, h5jam/h5unjam

    • new tool: h5copy, h5mkgrp

  • Converters

    • h4toh5, h5toh4, gif2h5, h52gif


Where to start

Where to start

Tools located at bin/ directory

  • Use “-h” option for help

  • Online help http://www.hdfgroup.org/hdf5tools.html

  • Email to THG help desk [email protected]


H5dump

h5dump

Dumps file content to stdout in

  • ASCII text

  • XML

  • Binary

4


H5dump h sds h5

h5dump -H SDS.h5

HDF5 "SDS.h5" {

GROUP "/" {

GROUP "Floats" {

DATASET "FloatArray" {

DATATYPE H5T_IEEE_F32LE

DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) }

}

}

DATASET "IntArray" {

DATATYPE H5T_STD_I32LE

DATASPACE SIMPLE { ( 5, 6 ) / ( 5, 6 ) }

}

}

}


H5dump d floats floatarray sds h5

h5dump -d /Floats/FloatArray SDS.h5

HDF5 "SDS.h5" {

DATASET "/Floats/FloatArray" {

DATATYPE H5T_IEEE_F32LE

DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) }

DATA {

(0,0): 0.01, 0.02, 0.03,

(1,0): 0.1, 0.2, 0.3,

(2,0): 1, 2, 3,

(3,0): 10, 20, 30

}

}

}


H5dump x sds h5

h5dump -x SDS.h5


H5dump binary output

h5dump Binary Output

-b F, --binary=[MEMORY|FILE|LE|BE]

h5dump -d /IntArray -o out_le.bin -b LE SDS.h5

000000000000000 00000001 00000002 00000003 00000004 00000005

0000030 0000000a 0000000b 0000000c 0000000d 0000000e 0000000f

0000060 00000014 00000015 00000016 00000017 00000018 00000019

0000110 0000001e 0000001f 00000020 00000021 00000022 00000023

0000140 00000028 00000029 0000002a 0000002b 0000002c 0000002d


H5diff

h5diff

Show Differences

Between

two objects or two files


H5diff sds h5 sds2 h5

h5diff SDS.h5 SDS2.h5

Dataset: </IntArray> and </IntArray>

5 differences found


Report differences

Report differences

Dataset:</IntArray> and </IntArray>

positionIntArrayIntArraydifference

------------------------------------------------------

[ 0 0 ]01010

[ 1 0 ]1010090

[ 2 0 ]20200180

[ 3 0 ]30300270

[ 4 0 ]40400360

5 differences found

h5diff SDS.h5 SDS2.h5 -r /IntArray


H5repack

h5repack

Copies a file to a new file with

  • Compression filter

  • Data layout

  • Other user options


H5repack new filters

h5repack: new filters

-f{GZIP|SZIP|SHUF|FLET|NBIT|SOFF|NONE}

For example

h5repack -i SDS.h5 -o out.h5 -f GZIP=6


H5repack data layout

h5repack: data layout

-l {CHUNK| COMPA| CONTI}

For example

h5repack -i SDS.h5 -o out.h5 -l CHUNK=2x3


H5repack latest version of file format

h5repack: latest version of file format

The testing file was created with HDF5 1.6.5. The root group of the file contains 100 groups. Each group at the root contains 9 sub groups. Each sub group contains one compound dataset. The compound dataset has 40 members of 32 bit integer


H5repart

h5repart

Repartitions a file into a family of files

For example

h5repart -m 200m int16kx16k.h5 part200m%d.h5

part200m0.h5

200 MB

977 MB

200 MB

part200m1.h5

200 MB

part200m2.h5

200 MB

part200m3.h5

177 MB

part200m4.h5


H5import

h5import

Imports binary/ASCII data into an HDF5 file

  • h5import infile -c config_file [infile -c config_file2 ...] -outfile outfile

    For example

    h5import float5x4x2.txt -c First_set.conf -o First_set.h5

GROUP "/" {

GROUP "work" {

DATASET "First-set" {

DATATYPE H5T_IEEE_F64LE

DATASPACE SIMPLE { ( 5, 2, 4 ) / ( 8, 8, H5S_UNLIMITED ) }

DATA {

(0,0,0): 1.01, 1.02, 1.03, 1.04,

(0,1,0): 1.11, 1.12, 1.13, 1.14,

(1,0,0): 1.21, 1.22, 1.23, 1.24,

(1,1,0): 1.31, 1.32, 1.33, 1.34,

(2,0,0): 1.41, 1.42, 1.43, 1.44,

}

}

}

}}

PATH work/First-set

INPUT-CLASS TEXTFP

RANK 3

DIMENSION-SIZES 5 2 4

OUTPUT-CLASS FP

OUTPUT-SIZE 64

OUTPUT-ARCHITECTURE IEEE

OUTPUT-BYTE-ORDER LE

CHUNKED-DIMENSION-SIZES 2 2 2

MAXIMUM-DIMENSIONS 8 8 -1


H5jam h5unjam

h5jam/h5unjam

  • h5jam

    h5jam -u test_ub.txt -i test_ub.h5

  • h5unjam

    h5unjam -i test_ub.h5 -o out_ub.txt -o out_ub.h5

User Block

HDF5

Jam

unjam

User Block

HDF5


Hdf5 tools update

h5ls

  • Lists selected information about file objects in the specified format

    For Example,

    h5ls -r SDS2.h5

  • /Floats Group

  • /Floats/DoubleArray Dataset {10, 5}

  • /Floats/FloatArray Dataset {4, 3}

  • /Floats/subs Group

  • /IntArray Dataset {5, 6}


Gif2h5 h52gif

gif2h5 / h52gif

gif2h5 – converts a GIF file into HDF5

gif2h5apollo17_earth.gif apollo17_earth.h5

h52gif – converts an HDF5 file into GIF

h52gifapollo17_earth.h5 apollo17_earth2.gif -i /apollo17_earth.gif/Image0 -p "/apollo17_earth.gif/Global Palette"


H5toh4 h4toh5

h5toh4 / h4toh5

h5toh4 -- Converts an HDF5 file to an HDF4 file

h4toh5 -- Converts an HDF4 file to an HDF5 file

http://www.hdfgroup.org/h4toh5/


New tools

New tools

h5copy

h5mkgrp

h5stat

h5check


H5copy

h5copy

Copies an object within a file/cross files

/

/

Floats

IntArray

FloatArray

FloatArray


H5copy f shallow

h5copy-f shallow

/

floats

-f shallow

/

f32

64-bit

floats

integers

/

i1

i2

f32

64-bit

floats

f1

f2

f32

64-bit

f1

f2


H5copy f soft

h5copy-f soft

/

/

f1

dset_SL

/f1

-f soft

f1

dset_SL

/f1

/

dset_SL

/f1


H5copy f ref

h5copy-f ref

/

d1

d2

/

dset_ref

-f ref

d2

d1

/

dset_ref

dset_ref


H5mkgrp

h5mkgrp

Creates groups

h5mkgrp test.h5 /info /data /data/test1 /data/test2

/

info

data

test1

test2


H5stat

h5stat

Prints statistics about HDF5 file

  • Header size

  • Raw data size

  • Attributes, datatypes, and etc.

    Helps

  • To troubleshoot overhead in HDF5 files

  • To choose specific object’s properties and storage strategies


H5check

h5check

A validation tool that verifies if an HDF5 file is encoded according to the HDF5 File Format Specification


Why h5check

Why h5check?

  • Verify if the file is compliant with the File Format

  • As a verification tool required by the application of HDF5 File Format to be an ANSI standard

  • Serves as a watch dog that the HDF5 library implementation is compliant with the File Format


Example a compliant file

Example: a compliant file

  • % h5check example1.h5

    • VALIDATING example1.h5

    • FOUND super block signature

    • VALIDATING the super block at 0...

    • VALIDATING the object header at 928...

    • VALIDATING the btree at 384...

    • FOUND btree signature.

    • VALIDATING the local heap at 96...

    • FOUND local heap signature.

    • Result: File is in compliance.


Example a non compliant file

Example: a non-compliant file

h5check invalid2.h5

  • FOUND super block signature

  • VALIDATING the super block at 0...

  • VALIDATING the object header at 928...

  • VALIDATING the btree at 384...

  • FOUND btree signature.

  • VALIDATING the SNOD at 1248...

  • FOUND SNOD signature.

  • VALIDATING the object header at 976...

  • check_sym(at 1248): Errors from check_obj_header()

  • decode_validate_messages(): Failure in type->decode().

  • H5O_sdspace_decode(): Bad version number in simple dataspace message.

  • VALIDATING the local heap at 96...

  • FOUND local heap signature.

  • Main(): Errors from check_obj_header().

  • decode_validate_messages(): Failure in type->decode().

  • H5O_attr_decode(): Can't decode attribute dataspace.

  • H5O_sdspace_decode(): Bad version number in simple dataspace message.

  • Result: File is not in compliance.


Implementation status

Implementation Status

Basic file format components

1.6 - DONE

1.8 -IN PROGRESS


Hdf java 2 4

HDF-Java 2.4

http://www.hdfgroup.org/hdf-java-html

ftp://ftp.hdfgroup.org/HDF5/hdf-java

HDF-Java 2.4 Released !!


Hdf java 2 4 release

HDF-Java 2.4 Release

  • Build with HDF4 2r2 and HDF5 1.6.6

  • New platforms

    Mac intel

    Linux 64-bit AMD

    Solaris 64-bit

  • 36 new enhancements and 44 bugs fixed

  • Fixed memory leak

  • Test suite

  • Enhanced documentation


Memory leak in old version

Memory leak in old version

while (true)

{

H5File f = new H5File

(fname, H5File.READ);

f.open();

f.close();

}


  • Login