Author: Quincey Koziol

# HDF5 File Format Discussion

### Quincey Koziol
koziol@ncsa.uiuc.edu
May 15, 2003

1. ### Document's Audience:

   - Current H5 library designers and knowledgeable external developers.

2. ### Background Reading:

   HDF5 File Format Specification
     This describes the current HDF5 file format.

3. ### Introduction:

   What is this document about?
     This document attempts to explain the HDF5 file format specification with a few examples and describes some potential improvements to the format specification.

4. ### File Format Examples:

   This section has several small programs and describes the format of a file created with each of them.

   Example program one - Create an empty file:

       #include "hdf5.h"
       #include

       int main()
       {
           hid_t fid;      /* File ID */
           herr_t ret;     /* Generic return value */

           /* Create the file */
           fid=H5Fcreate("example1.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
           assert(fid>=0);

           /* Close the file */
           ret=H5Fclose(fid);
           assert(ret>=0);

           return(0);
       }

   Super Block
   byte | byte | byte | byte
   --- | --- | --- | ---
   \211 | 'H' | 'D' | 'F'
   \r | \n | \032 | \n
   0 | 0 | 0 | 0
   0 | 8 | 8 | 0
   4 | 16
   0x00000003
   0
   0xffffffffffffffff
   ?
   0xffffffffffffffff
   0
   928
   H5G_CACHED_STAB (1)
   0
   384
   96

       %h5debug example1.h5

       Reading signature at address 0 (rel)
       File Super Block...
       File name:                                         example1.h5
       File access flags                                  0x00000000
       File open reference count:                         1
       Address of super block:                            0 (abs)
       Size of user block:                                0 bytes
       Super block version number:                        0
       Free list version number:                          0
       Root group symbol table entry version number:      0
       Shared header version number:                      0
       Size of file offsets (haddr_t type):               8 bytes
       Size of file lengths (hsize_t type):               8 bytes
       Symbol table leaf node 1/2 rank:                   4
       Symbol table internal node 1/2 rank:               16
       File consistency flags:                            0x00000003
       Base address:                                      0 (abs)
       Free list address:                                 UNDEF (rel)
       Address of driver information block:               UNDEF (rel)
       Root group symbol table entry:
          Name offset into private heap:                  0
          Object header address:                          928
          Dirty:                                          Yes
          Cache info type:                                Symbol Table
          Cached information:
             B-tree address:                              384
             Heap address:                                96

   Root Group Object Header
   byte | byte | byte | byte
   --- | --- | --- | ---
   1 | 0 | 2
   1
   32
   0x0011 | 16
   0x01 | 0
   384
   96
   0 | 0
   0x00 | 0

       %h5debug example1.h5 928

       New address: 928
       Reading signature at address 928 (rel)
       Object Header...
       Dirty:                                             0
       Version:                                           1
       Header size (in bytes):                            16
       Number of links:                                   1
       Number of messages (allocated):                    2 (32)
       Number of chunks (allocated):                      1 (8)
       Chunk 0...
          Dirty:                                          0
          Address:                                        944
          Size in bytes:                                  32
       Message 0...
          Message ID (sequence number):                   0x0011 stab(0)
          Shared message:                                 No
          Constant:                                       Yes
          Raw size in obj header:                         16 bytes
          Chunk number:                                   0
          Message Information:
             B-tree address:                              384
             Name heap address:                           96
       Message 1...
          Message ID (sequence number):                   0x0000 null(0)
          Shared message:                                 No
          Constant:                                       No
          Raw size in obj header:                         0 bytes
          Chunk number:                                   0
          Message Information:

   Root Group Local Heap
   byte | byte | byte | byte
   --- | --- | --- | ---
   'H' | 'E' | 'A' | 'P'
   0
   256
   8
   128

       %h5debug example1.h5 96

       New address: 96
       Reading signature at address 96 (rel)
       Local Heap...
       Dirty:                                             0
       Header size (in bytes):                            32
       Address of heap data:                              128
       Data bytes allocated on disk:                      256
       Data bytes allocated in core:                      256
       Free Blocks (offset, size):
          Block #0:                                        8,      248
       Percent of heap used:                              3.12%
       Data follows (`__' indicates free region)...
            0: 00 00 00 00 00 00 00 00  __ __ __ __ __ __ __ __ ........
           16: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
           32: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
           48: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
           64: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
           80: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
           96: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
          112: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
          128: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
          144: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
          160: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
          176: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
          192: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
          208: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
          224: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
          240: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __

   Root Group B-tree
   byte | byte | byte | byte
   --- | --- | --- | ---
   'T' | 'R' | 'E' | 'E'
   0 | 0 | 0
   0xffffffffffffffff
   0xffffffffffffffff

       %h5debug example1.h5 384 96

       New address: 384
       Reading signature at address 384 (rel)
       Tree type ID:                                      H5B_SNODE_ID
       Size of node:                                      544
       Size of raw (disk) key:                            8
       Dirty flag:                                        False
       Number of initial dirty children:                  0
       Level:                                             0
       Address of left sibling:                           UNDEF
       Address of right sibling:                          UNDEF
       Number of children (max):                          0 (32)

   Example program two - Create a file with a single dataset in it:

       #include "hdf5.h"
       #include

       int main()
       {
           hid_t fid;      /* File ID */
           hid_t sid;      /* Dataspace ID */
           hid_t did;      /* Dataset ID */
           herr_t ret;     /* Generic return value */

           /* Create the file */
           fid=H5Fcreate("example2.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
           assert(fid>=0);

           /* Create a scalar dataspace for the dataset */
           sid=H5Screate(H5S_SCALAR);
           assert(sid>=0);

           /* Create a trivial dataset */
           did=H5Dcreate(fid, "Dataset", H5T_NATIVE_INT, sid, H5P_DEFAULT);
           assert(did>=0);

           /* Close the dataset */
           ret=H5Dclose(did);
           assert(ret>=0);

           /* Close the dataspace */
           ret=H5Sclose(sid);
           assert(ret>=0);

           /* Close the file */
           ret=H5Fclose(fid);
           assert(ret>=0);

           return(0);
       }

   Super Block
   byte | byte | byte | byte
   --- | --- | --- | ---
   \211 | 'H' | 'D' | 'F'
   \r | \n | \032 | \n
   0 | 0 | 0 | 0
   0 | 8 | 8 | 0
   4 | 16
   0x00000003
   0
   0xffffffffffffffff
   ?
   0xffffffffffffffff
   0
   928
   H5G_CACHED_STAB (1)
   0
   384
   96

       %h5debug example2.h5

       Reading signature at address 0 (rel)
       File Super Block...
       File name:                                         example2.h5
       File access flags                                  0x00000000
       File open reference count:                         1
       Address of super block:                            0 (abs)
       Size of user block:                                0 bytes
       Super block version number:                        0
       Free list version number:                          0
       Root group symbol table entry version number:      0
       Shared header version number:                      0
       Size of file offsets (haddr_t type):               8 bytes
       Size of file lengths (hsize_t type):               8 bytes
       Symbol table leaf node 1/2 rank:                   4
       Symbol table internal node 1/2 rank:               16
       File consistency flags:                            0x00000003
       Base address:                                      0 (abs)
       Free list address:                                 UNDEF (rel)
       Address of driver information block:               UNDEF (rel)
       Root group symbol table entry:
          Name offset into private heap:                  0
          Object header address:                          928
          Dirty:                                          Yes
          Cache info type:                                Symbol Table
          Cached entry information:
             B-tree address:                              384
             Heap address:                                96

   Root Group Object Header
   byte | byte | byte | byte
   --- | --- | --- | ---
   1 | 0 | 2
   1
   32
   0x0011 | 16
   0x01 | 0
   384
   96
   0 | 0
   0x00 | 0

       %h5debug example2.h5 928

       New address: 928
       Reading signature at address 928 (rel)
       Object Header...
       Dirty:                                             0
       Version:                                           1
       Header size (in bytes):                            16
       Number of links:                                   1
       Number of messages (allocated):                    2 (32)
       Number of chunks (allocated):                      1 (8)
       Chunk 0...
          Dirty:                                          0
          Address:                                        944
          Size in bytes:                                  32
       Message 0...
          Message ID:                                     0x0011 stab(0)
          Shared message:                                 No
          Constant:                                       Yes
          Raw size in obj header:                         16 bytes
          Chunk number:                                   0
          Message Information:
             B-tree address:                              384
             Name heap address:                           96
       Message 1...
          Message ID:                                     0x0000 null(0)
          Shared message:                                 No
          Constant:                                       No
          Raw size in obj header:                         0 bytes
          Chunk number:                                   0
          Message Information:

   Root Group Local Heap
   byte | byte | byte | byte
   --- | --- | --- | ---
   'H' | 'E' | 'A' | 'P'
   0
   256
   16
   128

       %h5debug example2.h5 96

       New address: 96
       Reading signature at address 96 (rel)
       Local Heap...
       Dirty:                                             0
       Header size (in bytes):                            32
       Address of heap data:                              128
       Data bytes allocated on disk:                      256
       Data bytes allocated in core:                      256
       Free Blocks (offset, size):
          Block #0:                                       16,      240
       Percent of heap used:                              6.25%
       Data follows (`__' indicates free region)...
             0: 00 00 00 00 00 00 00 00  44 61 74 61 73 65 74 00 ........Dataset.
            16: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
            32: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
            48: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
            64: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
            80: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
            96: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
           112: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
           128: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
           144: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
           160: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
           176: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
           192: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
           208: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
           224: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
           240: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __

   Root Group B-tree
   byte | byte | byte | byte
   --- | --- | --- | ---
   'T' | 'R' | 'E' | 'E'
   0 | 0 | 1
   0xffffffffffffffff
   0xffffffffffffffff
   0
   1248
   8

       %h5debug example2.h5 384 96

       New address: 384
       Reading signature at address 384 (rel)
       Tree type ID:                                      H5B_SNODE_ID
       Size of node:                                      544
       Size of raw (disk) key:                            8
       Dirty flag:                                        False
       Number of initial dirty children:                  0
       Level:                                             0
       Address of left sibling:                           UNDEF
       Address of right sibling:                          UNDEF
       Number of children (max):                          1 (32)
       Child 0...
          Address:                                        1248
          Left Key:
             Heap offset:                                 0
             Name :
          Right Key:
             Heap offset:                                 8
             Name :                                       Dataset

   Root Group B-tree Symbol Table Node
   byte | byte | byte | byte
   --- | --- | --- | ---
   'S' | 'N' | 'O' | 'D'
   1 | 0 | 1
   8
   976
   0
   0
   0

       %h5debug example2.h5 1248 96

       New address: 1248
       Reading signature at address 1248 (rel)
       Symbol Table Node...
       Dirty:                                             No
       Size of Node (in bytes):                           328
       Number of Symbols:                                 1 of 8
       Symbol 0:
          Name:                                           `Dataset'
          Name offset into private heap:                  8
          Object header address:                          976
          Dirty:                                          No
          Cache info type:                                Nothing Cached

   '/Dataset' Object Header
   byte | byte | byte | byte
   --- | --- | --- | ---
   Version: 1 | Reserved: 0 | Number of Header Messages: 6
   Object Reference Count: 1
   Total Object Header Size: 256
   Fill Value Header Message
   Message Type: 0x0005 | Message Data Size: 8
   Flags: 0x01 | Reserved: 0
   Version: 1 | Space Allocation Time: 2 (Late) | Fill Value Writing Time: 0 (At allocation) | Fill Value Defined: 0 (Undefined)
   Fill Value Datatype Size: 0 (Use dataset's datatype for fill-value datatype)
   Datatype Header Message
   Message Type: 0x0003 | Message Data Size: 16
   Flags: 0x01 | Reserved: 0
   Version: 0x1 | Class: 0x0 (Fixed-Point)
   | Fixed-Point Bit-Field: 0x08 (Little-endian, No padding, Signed)
   Size: 4
   Bit Offset: 0 | Bit Precision: 32
   Message Alignment Filler: -
   Dataspace Header Message
   Message Type: 0x0001 | Message Data Size: 8
   Flags: 0x00 | Reserved: 0
   Version: 1 | Rank: 0 (Scalar) | Flags: 0x00 (No maximum dimensions, no permutation information) | Reserved: 0
   Reserved: 0
   Layout Header Message
   Message Type: 0x0008 | Message Data Size: 24
   Flags: 0x00 | Reserved: 0
   Version: 1 | Rank: 1 (Dataspace rank+1) | Class: 1 (Contiguous) | Reserved: 0
   Reserved: 0
   Address: 0xffffffffffffffff (Undefined)
   Dimension 0 Size: 4 (Datatype size)
   Message Alignment Filler: -
   Modification Date & Time Header Message
   Message Type: 0x0012 | Message Data Size: 8
   Flags: 0x00 | Reserved: 0
   Version: 1 | Reserved: 0
   Seconds Since Epoch: 1052401700 (2003-05-08 08:48:20 CDT)
   Null Header Message
   Message Type: 0x0000 | Message Data Size: 144
   Flags: 0x00 | Reserved: 0

       %h5debug example2.h5 976

       New address: 976
       Reading signature at address 976 (rel)
       Object Header...
       Dirty:                                             0
       Version:                                           1
       Header size (in bytes):                            16
       Number of links:                                   1
       Number of messages (allocated):                    6 (32)
       Number of chunks (allocated):                      1 (8)
       Chunk 0...
          Dirty:                                          0
          Address:                                        992
          Size in bytes:                                  256
       Message 0...
          Message ID (sequence number):                   0x0005 `fill_new' (0)
          Shared:                                         No
          Constant:                                       Yes
          Raw size in obj header:                         8 bytes
          Chunk number:                                   0
          Message Information:
             Version:                                     1
             Space Allocation Time:                       Late
             Fill Time:                                   On Allocation
             Fill Value Defined:                          Undefined
             Size:                                        0
             Data type:
       Message 1...
          Message ID (sequence number):                   0x0003 data_type(0)
          Shared message:                                 No
          Constant:                                       Yes
          Raw size in obj header:                         16 bytes
          Chunk number:                                   0
          Message Information:
             Type class:                                  integer
             Size:                                        4 bytes
             Byte order:                                  little endian
             Precision:                                   32 bits
             Offset:                                      0 bits
             Low pad type:                                zero
             High pad type:                               zero
             Sign scheme:                                 2's comp
       Message 2...
          Message ID (sequence number):                   0x0001 simple_dspace(0)
          Shared message:                                 No
          Constant:                                       No
          Raw size in obj header:                         8 bytes
          Chunk number:                                   0
          Message Information:
             Rank:                                        0
       Message 3...
          Message ID (sequence number):                   0x0008 layout(0)
          Shared message:                                 No
          Constant:                                       No
          Raw size in obj header:                         24 bytes
          Chunk number:                                   0
          Message Information:
             Data address:                                UNDEF
             Number of dimensions:                        1
             Size:                                        {4}
       Message 4...
          Message ID (sequence number):                   0x0012 mtime_new(0)
          Shared message:                                 No
          Constant:                                       No
          Raw size in obj header:                         8 bytes
          Chunk number:                                   0
          Message Information:
             Time:                                        2003-03-05 14:52:00 CST
       Message 5...
          Message ID (sequence number):                   0x0000 null(0)
          Shared message:                                 No
          Constant:                                       No
          Raw size in obj header:                         144 bytes
          Chunk number:                                   0
          Message Information: