HDF5 Software Changes

HDF5 documents and links Introduction to HDF5 | HDF5 User’s Guide HDF5 Reference Manual HDF5 Application Developer's Guide

# HDF5 Software Changes from Release to Release

This document is designed to help developers maintain HDF5-based applications as they are used with successive releases of HDF5. To that end, this document lists the changes at each release that a maintenance developer needs to be aware of:

- New or changed features or tools
- Syntax and behavioral changes in the existing application programming interface (the API)
- Certain types of changes in configuration or build processes

Bugfixes and performance enhancements in the C library are automatically picked up by the C++, Fortran, and Java libraries.

The release notes also list changes made to the library, but these notes tend to be more at a more detail-oriented level. The release notes may include new features, bugs fixed, supported configuration features, platforms on which the library has been tested, and known problems. The release note files are listed below and can be found at the top level of the HDF5 source code tree in the release_docs directory of the distributed HDF5 source code.

`RELEASE.txt` | Technical notes regarding the current release of the HDF5 Library
`HISTORY-1_8.txt` | Technical notes for 1.8.x releases of the HDF5 Library by release
`HISTORY-1_0-1_8_0_rc3.txt` `` | Technical notes for releases prior to the 1.8.0 release of the HDF5 Library by release

## Release 1.10.0 of March 2016 (current release) versus Release 1.8.16

This section lists interface-level changes and other user-visible changes in behavior in the transition from HDF5 Release 1.8.16 to Release 1.10.0.

### Changed Type

`hid_t`
  Changed from a 32-bit to a 64-bit value.

  `hid_t` is the type is used for all HDF5 identifiers. This change, which is necessary to accomodate the capacities of modern computing systems, therefore affects all HDF5 applications. If an application has been using HDF5’s `hid_t` the type, recompilation will normally be sufficient to take advantage of HDF5 Release 1.10.0. If an application uses an integer type instead of HDF5’s `hid_t` type, those identifiers must be changed to a 64-bit type when the application is ported to the 1.10.x series.

### New Features and Feature Sets

Several new features are introduced in HDF5 Relesase 1.10.0.
  Single-Writer / Multiple-Reader or SWMR
  Collective Metadata I/O
  Fine-tuning the metadata cache
  File Space Management
  Virtual Datasets or VDS
  Partial Edge Chunk Options
  Relative Pathnames for External Links
  Property List Encoding and Decoding
More substantial lists follow, including new and modified C functions and Fortran subroutines.

### New Features, including associated C Functions and Fortran Wrappers

The following features are described and documented in New Features in HDF5 Release 1.10.0. On this page, we list each feature and its associated C functions and Fortran wrappers.

Single-writer / Multiple-reader, commonly called SWMR:

  `H5Fstart_swmr_write` | Enables SWMR writing mode for a file.
  `H5DOappend` | Appends data to a dataset along a specified dimension. (This is a high-level API.)
  `H5Pget_append_flush` | Retrieves the values of the append property that is set up in the dataset access property list.
  `H5Pset_append_flush` | Sets two actions to perform when the size of a dataset’s dimension being appended reaches a specified boundary.
  `H5Pget_object_flush_cb``` | Retrieves the object flush property values from the file access property list.
  `H5Pset_object_flush_cb` | Sets a callback function to invoke when an object flush occurs in the file.
  `H5Odisable_mdc_flushes` | Prevents metadata entries for an HDF5 object from being flushed from the metadata cache to storage.
  `H5Oenable_mdc_flushes` | Returns the cache entries associated with an HDF5 object to the default metadata flush and eviction algorithm.
  `H5Oare_mdc_flushes_disabled`
  | Determines if an HDF5 object (dataset, group, committed datatype) has had flushes of metadata entries disabled.
  `H5Fdisable_mdc_flushes` | Globally prevents dirty metadata entries from being flushed from the metadata cache to storage.
  `H5Fenable_mdc_flushes` | Returns a file’s metadata cache to the standard eviction and flushing algorithm.
  `H5Fare_mdc_flushes_disabled`
  | Determines if flushes have been globally disabled for a file’s metadata cache.
  `H5Fget_mdc_flush_disabled_obj_ids`
  | Returns a list of all object identifiers for which flushes have been disabled in a file’s metadata cache.
  Command-line Tools:
  `h5watch` | Allows users to output new records appended to a dataset under SWMR access as it grows. The functionality is similar to the Unix user command `tail` with the follow option, which outputs appended data as the file grows.
  `h5format_convert` | This tool allows users to convert the indexing type of a chunked dataset made with a 1.10.x version of the HDF5 Library when the latest file format is used to the 1.8.x version 1 B-tree indexing type. For example, datasets created using SWMR access, can be converted to be accessed by the HDF5 1.18 library and tools. The tool does not rewrite raw data, but it does rewrite HDF5 metadata.

Collective Metadata I/O:

  `H5Pset_coll_metadata_write` `h5pset_coll_metadata_write_f` | Establishes I/O mode property setting, collective or independent, for metadata writes.
  `H5Pget_coll_metadata_write` `h5pget_coll_metadata_write_f` | Retrieves I/O mode property setting for metadata writes.
  | `H5Pset_all_coll_metadata_ops``` `h5pset_all_coll_metadata_ops_f``` | Establishes I/O mode, collective or independent, for metadata read operations.
  `H5Pget_all_coll_metadata_ops` `h5pget_all_coll_metadata_ops_f` | Retrieves I/O mode for metadata read operations.

Fine-tuning the Metadata Cache:

  `H5Fget_metadata_read_retries_info`
  | Retrieves the collection of read retries for metadata items with checksum.
  `H5Pget_metadata_read_attempts`
  | Retrieves the number of read attempts from a file access property list.
  `H5Pset_metadata_read_attempts`
  | Sets the number of read attempts in a file access property list.
  `H5Dflush` | Causes all buffers associated with a dataset to be immediately written to disk without removing the data from the cache.
  `H5Drefresh` | Causes all buffers associated with a dataset to be cleared and immediately re-loaded with updated contents from disk storage.
  `H5Gflush` | Causes all buffers associated with a group to be immediately flushed to disk without removing the data from the cache.
  `H5Grefresh` | Causes all buffers associated with a group to be cleared and immediately re-loaded with updated contents from disk storage.
  `H5Oflush` | Causes all buffers associated with an object to be immediately flushed to disk without removing the data from the cache.
  `H5Orefresh` | Causes all buffers associated with an object to be cleared and immediately re-loaded with updated contents from disk storage.
  `H5Tflush` | Causes all buffers associated with a committed datatype to be immediately flushed to disk without removing the data from the cache.
  `H5Trefresh` | Causes all buffers associated with a committed datatype to be cleared and immediately re-loaded with updated contents from disk storage.
  `H5Fget_intent` | Determines the read/write or read-only status of a file.
  Logging APIs:
  `H5Pset_mdc_log_options``` | Sets metadata cache logging options.
  `H5Pget_mdc_log_options` | Gets metadata cache logging options.
  `H5Fstart_mdc_logging` | Starts logging metadata cache events if logging was previously enabled.
  `H5Fstop_mdc_logging` | Stops logging metadata cache events if logging was previously enabled and is currently ongoing.
  `H5Pget_mdc_logging_status`
  | Gets the current metadata cache logging status.

File Space Management:

  `H5Fget_free_sections` |  | Retrieves free-space section information for a file.
  `H5Fget_freespace` |  | Returns the amount of free space in a file.
  `H5Fget_info2` |  | Returns global information for a file.
  `H5Pset_file_space` |  | Sets the file space management strategy and/or the free-space section threshold for an HDF5 file.
  `H5Pget_file_space` |  | Retrieves the file space management strategy and/or the free-space section threshold for an HDF5 file.
  The following tool has been modified to preserve or modify file freepace information appropriately when processing files employing the VDS feature:
  `h5repack` | Repacks HDF5 files with various options, including the ability to change the applied filters. This version of `h5repack` understands the file free space feature and handles the file and metadata appropriately.

Virtual Dataset or VDS:

  `H5Pset_virtual` `h5pset_virtual_f` | Sets the mapping between virtual and source datasets.
  `H5Pget_virtual_count` `h5pget_virtual_count_f` | Retrieves the number of mappings for the virtual dataset.
  `H5Pget_virtual_vspace` `h5pget_virtual_vspace_f` | Retrieves a dataspace identifier for the selection within the virtual dataset used in the mapping.
  `H5Pget_virtual_srcspace` `h5pget_virtual_srcspace_f` | Retrieves a dataspace identifier for the selection within the source dataset used in the mapping.
  `H5Pget_virtual_dsetname` `h5pget_virtual_dsetname_f` | Retrieves the name of a source dataset used in the mapping.
  `H5Pget_virtual_filename` `h5pget_virtual_filename_f` | Retrieves the filename of a source dataset used in the mapping.
  `H5Pset_virtual_printf_gap` `h5pset_virtual_printf_gap_f` | Sets maximum number of missing source files and/or datasets with printf-style names when getting the extent of an unlimited virtual dataset.
  `H5Pget_virtual_printf_gap` `h5pget_virtual_printf_gap_f` | Returns maximum number of missing source files and/or datasets with printf-style names when getting the extent for an unlimited virtual dataset.
  `H5Pset_virtual_view` `h5pset_virtual_view_f` | Sets the view of the virtual dataset to include or exclude missing mapped elements.
  `H5Pget_virtual_view` `h5pget_virtual_view_f` | Retrieves the view of a virtual dataset.
  Supporting Functions:
  `H5Sis_regular_hyperslab` `h5sis_regular_hyperslab_f` | Determines whether a hyperslab selection is regular.
  `H5Sget_regular_hyperslab``` `h5sget_regular_hyperslab_f``` | Retrieves a regular hyperslab selection.
  Modified Functions: The following pre-exising functions have been modified to understand virtual datasets.
  `H5Pset_layout` `h5pset_layout_f` | Specifies the layout to be used for a dataset. Virtual dataset, `H5D_VIRTUAL`, has been added to the list of layouts available through this function.
  `H5Pget_layout` `h5pget_layout_f` | Retrieves the layout in use for a dataset. Virtual dataset, `H5D_VIRTUAL`, has been added to the list of layouts.

Partial Edge Chunks:

  `H5Pset_chunk_opts` | Sets a partial edge chunk option in a dataset access property list.
  `H5Pget_chunk_opts` | Retrieves partial edge chunk option setting from a dataset access property list.

Relative Pathnames for External Links:

  `H5Pset_elink_prefix` | These functions enable the user to specify the locations of external files. (These functions are not yet documented.)
  `H5Pget_elink_prefix`

Property List Encoding and Decoding:

  `H5Pencode` |
  `H5Pdecode` |

Additional New Functions:
  The following new functions appear in HDF5 Release 1.10.0 but are not yet documented:

          H5Dformat_convert
          H5Dget_chunk_index_type

          H5FDlock
          H5FDunlock

          H5Fformat_convert

          H5LDget_dset_dims
          H5LDget_dset_elmts
          H5LDget_dset_type_size

### New and Changed Elements of the Packet Table (H5PT) High-level API

In the C Interface
  Replacement functions:
    `H5PTcreate`
      Takes a property list identifier to provide flexibility on creation properties.
      `H5PTcreate_fl` has been removed.
    `H5PTfree_vlen_buff`
      Replaces `H5PTfree_vlen_readbuff`.

  New functions:
    Two accessor functions have been added.
    `H5PTget_dataset`
      Returns the identifier of the dataset associated a packet table.
    `H5PTget_type`
      Returns the identifier of the datatype used by a packet table.
    `H5PTis_varlen`
      Determines whether a type is variable-length.

In the C++ Interface
  Overloaded constructor
    An overloaded constructor has been added.
    `FL_PacketTable`
      Takes a property list identifier to provide flexibility on creation properties.>/dd>
    `H5PTfree_vlen_buff`
      Replaces `H5PTfree_vlen_readbuff`.

  Accessor wrappers
    Two accessor wrappers are added to class `PacketTable`.
    `PacketTable::GetDataset()`
      Returns the identifier of the dataset associated a packet table.
    `PacketTable::GetDatatype()`
      Returns the identifier of the datatype used by a packet table.

  Other wrappers
    `PacketTable::FreeBuff()`
      Replaces `VL_PacketTable::FreeReadBuff()`.
    `PacketTable::IsVariableLength()`
      Replaces `VL_PacketTable::IsVariableLength()`.

  Overloaded functions:
    Where a member functions has a `char*` as an argument, an overloaded functions has been added to provide the `const char*` argument.
    The existing version will be deprecated in a future release.

### Java Interface Changes

Integration into Main HDF5 Library
  The Java HDF5 JNI library has been integrated into the HDF5 repository.
  Configure option:
    `--enable-java`
  CMake option:
    `HDF5_BUILD_JAVA:BOOL=ON`

  Prior to the 1.10.x series, the HDF5 Java tools were built from an independent repository and were not as fully integrated with HDF5. were built from an independent repository and were not as fully integrated with HDF5.

Package Hierarchy Change
  The package hierarchy has changed to `hdf.hdflib.hdf5`.

  Prior to the 1.10.x series, the hierarchy was `ncsa.hdf.hdflib.hdf5`.

New Java APIs
  A number of new APIs have been added in the Java interface, including APIs for the VDS and SWMR features.

### Functions with Changed Behavior

`H5Lexists`
  The behavior of this function has changed in this release. When testing the pathname `/` (a slash representing the root of an HDF5 file) `H5Lexists` now returns successfully with the value `1` (one). See the entry in the HDF5 Reference Manual for `H5Lexists` for more information.

### API Compatibility

See API Compatibility Macros in HDF5 for details on the following.

New API Compatibility Flag
  A new `v18` flag was added enabling the building of HDF5 such that the default API is compatible with the HDF5 Release 1.8.x API:
  `--with-default-api-version=v18`

New versioned functions and associated compatibility macros
  Two functions and a struct have been converted to a versioned form in this release. Compatibility macros have been created for each.

  `H5Fget_info`
    The original function is renamed to `H5Fget_info1` and deprecated.
    A new version of the function, `H5Fget_info2`, is introduced.
    The compatiblity macro `H5Fget_info` is introduced.

  `H5F_info_t`
    This is the struct used by the `H5Fget_info` functions and macro.
    The original struct is renamed to `H5F_info1_t` and deprecated.
    A new version of the struct, `H5F_info2_t`, is introduced.
    The compatiblity macro `H5F_info_t` is introduced.

  `H5Rdereference`
    The original function is renamed to `H5Rdereference1` and deprecated.
    A new version of the function, `H5Rdereference2`, is introduced.
    The compatiblity macro `H5Rdereference` is introduced.

### Autotools Configuration and Large File Support

Autotools configuration has been extensively reworked and autotool’s handling of large file support has been overhauled in this release.

See the following sections in `RELEASE.txt`:

- “Autotools Configuration Has Been Extensively Reworked”
- “LFS Changes”

`RELEASE.txt` is found in the `release_docs/` subdirectory at the root level of the HDF5 code distribution.

### Compatibility Report and Comments

Compatibility report for Release 1.10.0 versus Release 1.8.16
  API compatibility report for the HDF5 Library between 1.10.0 and 1.8.16 versions on x86

Comments regarding the report
  In the C interface, the `hid_t` change from 32-bit to 64-bit was made in order to address a performance problem that arose when the library “ran out” of valid object identifiers to issue and thus needed to employ an expensive algorithm to find previously issued identifiers that could be re-issued. This problem is avoided by switching the size of the `hid_t` type to 64-bit integers instead of 32-bit integers in order to make the pool of available integers significantly larger. (`H5E_major_t` and `H5E_minor_t` are aliased to `hid_t` which is why they changed size as well). (An alternate solution to this problem was applied in release HDF5 1.8.5 but this is the cleaner/preferred solution and had to wait until 1.10.0 to be included).

  `hbool_t` will now be defined as a `_Bool` type when configure determines that it’s available.

  Public structs that have members of type `hid_t` or `hbool_t` are affected by the above changes accordingly.

  The `H5Fget_info` function was renamed due to the introduction of a newer version of the function which returns additional information. The `H5Rdereference` function was renamed due to the introduction of a newer version of the function which allows a data access property list to be passed in. Both changes are accompanied with compatibility macros, so while existing code will need to be recompiled in order to use the newer library version, these functions do not need to be changed in application code using them provided that the HDF5 API compatibility macros are configured appropriately.

## Releases in the 1.8.x Series

Change notes for HDF5 Release 1.8.0 to the release immediately preceding the current release are published in “HDF5 Software Changes from Release to Release for HDF5 Releases 1.8.x”.

## Releases in the 1.6.x and 1.4.x Series

Change notes for HDF5 Release 1.6.10 and earlier releases are published in “HDF5 Software Changes from Release to Release for HDF5 Releases 1.4.0 through 1.6.10”.

HDF5 documents and links Introduction to HDF5 | HDF5 User’s Guide HDF5 Reference Manual HDF5 Application Developer's Guide

Last modified: 8 April 2016