/** \page FMTDISC HDF5 File Format Discussion

Document's Audience:
  Current H5 library designers and knowledgeable external developers.
Background Reading:
  \ref FMT2
  This describes the current HDF5 file format.

\section sec_fmtdisc_intro Introduction What is this document about?
This document attempts to explain the HDF5 file format specification with a few examples and describes some potential improvements to the format specification. \section sec_fmtdisc_exam File Format Examples This section has several small programs and describes the format of a file created with each of them. Example program one - Create an empty file: \code #include "hdf5.h" #include
    int main()
    {
        hid_t fid;      /* File ID */
        herr_t ret;     /* Generic return value */

        /* Create the file */
        fid=H5Fcreate("example1.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
        assert(fid>=0);

        /* Close the file */
        ret=H5Fclose(fid);
        assert(ret>=0);

        return(0);
    }
    \endcode

Super Block
byte | byte | byte | byte
--- | --- | --- | ---
\211 | 'H' | 'D' | 'F'
\\r | \\n | \032 | \\n
0 | 0 | 0 | 0
0 | 8 | 8 | 0
4 | 16
0x00000003
0
0xffffffffffffffff
?
0xffffffffffffffff
0
928
H5G_CACHED_STAB (1)
0
384
96

\code %h5debug example1.h5 Reading signature at address 0 (rel) File Super Block... File name: example1.h5 File access flags 0x00000000 File open reference count: 1 Address of super block: 0 (abs) Size of user block: 0 bytes Super block version number: 0 Free list version number: 0 Root group symbol table entry version number: 0 Shared header version number: 0 Size of file offsets (haddr_t type): 8 bytes Size of file lengths (hsize_t type): 8 bytes Symbol table leaf node 1/2 rank: 4 Symbol table internal node 1/2 rank: 16 File consistency flags: 0x00000003 Base address: 0 (abs) Free list address: UNDEF (rel) Address of driver information block: UNDEF (rel) Root group symbol table entry: Name offset into private heap: 0 Object header address: 928 Dirty: Yes Cache info type: Symbol Table Cached information: B-tree address: 384 Heap address: 96 \endcode

Root Group Object Header
byte | byte | byte | byte
--- | --- | --- | ---
1 | 0 | 2
1
32
0x0011 | 16
0x01 | 0
384
96
0 | 0
0x00 | 0

\code %h5debug example1.h5 928 New address: 928 Reading signature at address 928 (rel) Object Header... Dirty: 0 Version: 1 Header size (in bytes): 16 Number of links: 1 Number of messages (allocated): 2 (32) Number of chunks (allocated): 1 (8) Chunk 0... Dirty: 0 Address: 944 Size in bytes: 32 Message 0... Message ID (sequence number): 0x0011 stab(0) Shared message: No Constant: Yes Raw size in obj header: 16 bytes Chunk number: 0 Message Information: B-tree address: 384 Name heap address: 96 Message 1... Message ID (sequence number): 0x0000 null(0) Shared message: No Constant: No Raw size in obj header: 0 bytes Chunk number: 0 Message Information: \endcode

Root Group Local Heap
byte | byte | byte | byte
--- | --- | --- | ---
'H' | 'E' | 'A' | 'P'
0
256
8
128

    \code
    %h5debug example1.h5 96

    New address: 96
    Reading signature at address 96 (rel)
    Local Heap...
    Dirty:                                             0
    Header size (in bytes):                            32
    Address of heap data:                              128
    Data bytes allocated on disk:                      256
    Data bytes allocated in core:                      256
    Free Blocks (offset, size):
       Block #0:                                        8,      248
    Percent of heap used:                              3.12%
    Data follows (`__' indicates free region)...
         0: 00 00 00 00 00 00 00 00  __ __ __ __ __ __ __ __ ........
        16: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        32: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        48: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        64: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        80: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        96: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
       112: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
       128: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
       144: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
       160: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
       176: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
       192: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
       208: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
       224: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
       240: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __

    \endcode

Root Group B-tree
byte | byte | byte | byte
--- | --- | --- | ---
'T' | 'R' | 'E' | 'E'
0 | 0 | 0
0xffffffffffffffff
0xffffffffffffffff

\code %h5debug example1.h5 384 96 New address: 384 Reading signature at address 384 (rel) Tree type ID: H5B_SNODE_ID Size of node: 544 Size of raw (disk) key: 8 Dirty flag: False Number of initial dirty children: 0 Level: 0 Address of left sibling: UNDEF Address of right sibling: UNDEF Number of children (max): 0 (32) \endcode Example program two - Create a file with a single dataset in it: \code #include "hdf5.h" #include
    int main()
    {
        hid_t fid;      /* File ID */
        hid_t sid;      /* Dataspace ID */
        hid_t did;      /* Dataset ID */
        herr_t ret;     /* Generic return value */

        /* Create the file */
        fid=H5Fcreate("example2.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
        assert(fid>=0);

        /* Create a scalar dataspace for the dataset */
        sid=H5Screate(H5S_SCALAR);
        assert(sid>=0);

        /* Create a trivial dataset */
        did=H5Dcreate(fid, "Dataset", H5T_NATIVE_INT, sid, H5P_DEFAULT);
        assert(did>=0);

        /* Close the dataset */
        ret=H5Dclose(did);
        assert(ret>=0);

        /* Close the dataspace */
        ret=H5Sclose(sid);
        assert(ret>=0);

        /* Close the file */
        ret=H5Fclose(fid);
        assert(ret>=0);

        return(0);
    }
    \endcode

Super Block
byte | byte | byte | byte
--- | --- | --- | ---
\211 | 'H' | 'D' | 'F'
\\r | \\n | \032 | \\n
0 | 0 | 0 | 0
0 | 8 | 8 | 0
4 | 16
0x00000003
0
0xffffffffffffffff
?
0xffffffffffffffff
0
928
H5G_CACHED_STAB (1)
0
384
96

\code %h5debug example2.h5 Reading signature at address 0 (rel) File Super Block... File name: example2.h5 File access flags 0x00000000 File open reference count: 1 Address of super block: 0 (abs) Size of user block: 0 bytes Super block version number: 0 Free list version number: 0 Root group symbol table entry version number: 0 Shared header version number: 0 Size of file offsets (haddr_t type): 8 bytes Size of file lengths (hsize_t type): 8 bytes Symbol table leaf node 1/2 rank: 4 Symbol table internal node 1/2 rank: 16 File consistency flags: 0x00000003 Base address: 0 (abs) Free list address: UNDEF (rel) Address of driver information block: UNDEF (rel) Root group symbol table entry: Name offset into private heap: 0 Object header address: 928 Dirty: Yes Cache info type: Symbol Table Cached entry information: B-tree address: 384 Heap address: 96 \endcode

Root Group Object Header
byte | byte | byte | byte
--- | --- | --- | ---
1 | 0 | 2
1
32
0x0011 | 16
0x01 | 0
384
96
0 | 0
0x00 | 0

\code %h5debug example2.h5 928 New address: 928 Reading signature at address 928 (rel) Object Header... Dirty: 0 Version: 1 Header size (in bytes): 16 Number of links: 1 Number of messages (allocated): 2 (32) Number of chunks (allocated): 1 (8) Chunk 0... Dirty: 0 Address: 944 Size in bytes: 32 Message 0... Message ID: 0x0011 stab(0) Shared message: No Constant: Yes Raw size in obj header: 16 bytes Chunk number: 0 Message Information: B-tree address: 384 Name heap address: 96 Message 1... Message ID: 0x0000 null(0) Shared message: No Constant: No Raw size in obj header: 0 bytes Chunk number: 0 Message Information: \endcode

Root Group Local Heap
byte | byte | byte | byte
--- | --- | --- | ---
'H' | 'E' | 'A' | 'P'
0
256
16
128

    \code
    %h5debug example2.h5 96

    New address: 96
    Reading signature at address 96 (rel)
    Local Heap...
    Dirty:                                             0
    Header size (in bytes):                            32
    Address of heap data:                              128
    Data bytes allocated on disk:                      256
    Data bytes allocated in core:                      256
    Free Blocks (offset, size):
       Block #0:                                       16,      240
    Percent of heap used:                              6.25%
    Data follows (`__' indicates free region)...
          0: 00 00 00 00 00 00 00 00  44 61 74 61 73 65 74 00 ........Dataset.
         16: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
         32: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
         48: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
         64: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
         80: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
         96: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        112: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        128: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        144: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        160: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        176: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        192: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        208: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        224: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        240: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
    \endcode

Root Group B-tree
byte | byte | byte | byte
--- | --- | --- | ---
'T' | 'R' | 'E' | 'E'
0 | 0 | 1
0xffffffffffffffff
0xffffffffffffffff
0
1248
8

\code %h5debug example2.h5 384 96 New address: 384 Reading signature at address 384 (rel) Tree type ID: H5B_SNODE_ID Size of node: 544 Size of raw (disk) key: 8 Dirty flag: False Number of initial dirty children: 0 Level: 0 Address of left sibling: UNDEF Address of right sibling: UNDEF Number of children (max): 1 (32) Child 0... Address: 1248 Left Key: Heap offset: 0 Name : Right Key: Heap offset: 8 Name : Dataset \endcode

Root Group B-tree Symbol Table Node
byte | byte | byte | byte
--- | --- | --- | ---
'S' | 'N' | 'O' | 'D'
1 | 0 | 1
8
976
0
0
0

\code %h5debug example2.h5 1248 96 New address: 1248 Reading signature at address 1248 (rel) Symbol Table Node... Dirty: No Size of Node (in bytes): 328 Number of Symbols: 1 of 8 Symbol 0: Name: `Dataset' Name offset into private heap: 8 Object header address: 976 Dirty: No Cache info type: Nothing Cached \endcode

'/Dataset' Object Header
byte | byte | byte | byte
--- | --- | --- | ---
Version: 1 | Reserved: 0 | Number of Header Messages: 6
Object Reference Count: 1
Total Object Header Size: 256
Fill Value Header Message
Message Type: 0x0005 | Message Data Size: 8
Flags: 0x01 | Reserved: 0
Version: 1 | Space Allocation Time: 2 (Late) | Fill Value Writing Time: 0 (At allocation) | Fill Value Defined: 0 (Undefined)
Fill Value Datatype Size: 0 (Use dataset's datatype for fill-value datatype)
Datatype Header Message
Message Type: 0x0003 | Message Data Size: 16
Flags: 0x01 Reserved: 0
Version: 0x1 | Class: 0x0 (Fixed-Point)
| Fixed-Point Bit-Field: 0x08 (Little-endian, No padding, Signed)
Size: 4
Bit Offset: 0 | Bit Precision: 32
Message Alignment Filler: -

Dataspace Header Message
Message Type: 0x0001 | Message Data Size: 8
Flags: 0x00 | Reserved: 0
Version: 1 | Rank: 0 (Scalar) | Flags: 0x00 (No maximum dimensions, no permutation information) | Reserved: 0
Reserved: 0

Layout Header Message
Message Type: 0x0008 | Message Data Size: 24
Flags: 0x00 | Reserved: 0
Version: 1 | Rank: 1 (Dataspace rank+1) | Class: 1 (Contiguous) | Reserved: 0
Reserved: 0
Address: 0xffffffffffffffff (Undefined)
Dimension 0 Size: 4 (Datatype size)
Message Alignment Filler: -

Modification Date & Time Header Message
Message Type: 0x0012 | Message Data Size: 8
Flags: 0x00 | Reserved: 0
Version: 1 | Reserved: 0
Seconds Since Epoch: 1052401700 (2003-05-08 08:48:20 CDT)

Null Header Message
Message Type: 0x0000 | Message Data Size: 144
Flags: 0x00 | Reserved: 0

\code %h5debug example2.h5 976 New address: 976 Reading signature at address 976 (rel) Object Header... Dirty: 0 Version: 1 Header size (in bytes): 16 Number of links: 1 Number of messages (allocated): 6 (32) Number of chunks (allocated): 1 (8) Chunk 0... Dirty: 0 Address: 992 Size in bytes: 256 Message 0... Message ID (sequence number): 0x0005 `fill_new' (0) Shared: No Constant: Yes Raw size in obj header: 8 bytes Chunk number: 0 Message Information: Version: 1 Space Allocation Time: Late Fill Time: On Allocation Fill Value Defined: Undefined Size: 0 Data type: Message 1... Message ID (sequence number): 0x0003 data_type(0) Shared message: No Constant: Yes Raw size in obj header: 16 bytes Chunk number: 0 Message Information: Type class: integer Size: 4 bytes Byte order: little endian Precision: 32 bits Offset: 0 bits Low pad type: zero High pad type: zero Sign scheme: 2's comp Message 2... Message ID (sequence number): 0x0001 simple_dspace(0) Shared message: No Constant: No Raw size in obj header: 8 bytes Chunk number: 0 Message Information: Rank: 0 Message 3... Message ID (sequence number): 0x0008 layout(0) Shared message: No Constant: No Raw size in obj header: 24 bytes Chunk number: 0 Message Information: Data address: UNDEF Number of dimensions: 1 Size: {4} Message 4... Message ID (sequence number): 0x0012 mtime_new(0) Shared message: No Constant: No Raw size in obj header: 8 bytes Chunk number: 0 Message Information: Time: 2003-03-05 14:52:00 CST Message 5... Message ID (sequence number): 0x0000 null(0) Shared message: No Constant: No Raw size in obj header: 144 bytes Chunk number: 0 Message Information: \endcode */