HDF5 documents and links Introduction to HDF5 HDF5 User’s Guide Other High-level API documents | In the HDF5 Reference Manual H5DS H5IM H5LT H5PT H5TB Optimized H5 H5A H5D H5E H5F H5G H5I H5L H5O H5P H5PL H5R H5S H5T H5Z Tools Datatypes Fortran

# HDF5 Optimized Functions

HDF5 functions described is this section are implemented in the HDF5 High-level Library as optimized functions. These functions generally require careful setup and testing as they enable an application to bypass portions of the HDF5 Library’s I/O pipeline for performance purposes.

These functions are distributed in the standard HDF5 distribution and are available any time the HDF5 High-level Library is available.

The C Interfaces:

Dataset I/O Optimization H5DOwrite_chunk |

/ast modified: 28 February 2013
Name: H5DOwrite_chunk
Signature:
  herr_t `H5DOwrite_chunk`( hid_t`dset_id`, hid_t`dxpl_id`, uint32_t`filter_mask`, hsize_t *`offset`, size_t`data_size`, const void *`buf` )

Purpose:
  Writes a raw data chunk from a buffer directly to a dataset.

Description:
  `H5DOwrite_chunk` writes a raw data chunk from the application memory buffer `buf` directly to its logical destination `offset` in a dataset `dset_id`. Typically, the data in `buf` is preprocessed in memory by a custom transformation, such as compression. The data chunk will bypass the library’s internal data transfer pipeline, including filters, and will be written directly to the file.

  `dxpl_id` is a data transfer property list identifier. (Currently unused.)

  `filter_mask` is a mask providing a record of which filters are used with the chunk. The default value of the mask is zero (`0`), indicating that all enabled filters are applied. A filter is skipped if the bit corresponding to the filter’s position in the pipeline (`0 ≤ position < 32`) is turned on. This mask is saved with the data chunk in the file.

  `offset` is an array specifying the logical position of the first element of the data chunk in the dataset’s dataspace. The length of the offset array must equal the number of dimensions, or rank, of the dataspace. The values in `offset` must not exceed the dimension limits and must specify a point that falls on a data chunk boundary.

  `data_size` is the size in bytes of the raw data chunk, representing the number of bytes to be read from the buffer `buf`. If the data chunk has been precompressed, `data_size` should be the size of the compressed data.

  `buf` is the memory buffer for the data chunk.

Parameters:

  hid_t`dset_id` | IN: Identifier for the dataset to write to
  hid_t`dxpl_id` | IN: Transfer property list identifier for this I/O operation (Currently unused)
  uint32_t`filter_mask` | IN: Mask for identifying the filters in use
  hsize_t *`offset` | IN: Logical position of the chunk’s first element in the dataspace
  size_t`data_size` | IN: Size of the actual data to be written in bytes
  const void *`buf` | IN: Buffer containing data to be written to the file

Returns:
  Returns a non-negative value if successful; otherwise returns a negative value.

See Also:
  “Using the Direct Chunk Write Function” in “Advanced Topics in HDF5” (PDF only)
  Design document for this feature: “RFC: Direct Chunk Write” (PDF only)

Example Usage:
  The following code illustrates the use of `H5DOwrite_chunk` to write an entire dataset, chunk by chunk:

      #include <zlib.h>
      #include <math.h>
      #define DEFLATE_SIZE_ADJUST(s) (ceil(((double)(s))*1.001)+12)
                      :
                      :
      size_t       buf_size = CHUNK_NX*CHUNK_NY*sizeof(int);
      const Bytef *z_src = (const Bytef*)(direct_buf);
      Bytef       *z_dst;             /* Destination buffer            */
      uLongf       z_dst_nbytes = (uLongf)DEFLATE_SIZE_ADJUST(buf_size);
      uLong        z_src_nbytes = (uLong)buf_size;
      int          aggression = 9;    /* Compression aggression setting */
      uint32_t     filter_mask = 0;
      size_t       buf_size = CHUNK_NX*CHUNK_NY*sizeof(int);

      /* Create the data space */
      if((dataspace = H5Screate_simple(RANK, dims, maxdims)) < 0)
                  goto error;

      /* Create a new file */
      if((file = H5Fcreate(FILE_NAME5, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT)) < 0)
                  goto error;

      /* Modify dataset creation properties, i.e. enable chunking and compression */
      if((cparms = H5Pcreate(H5P_DATASET_CREATE)) < 0)
                  goto error;

      if((status = H5Pset_chunk( cparms, RANK, chunk_dims)) < 0)
                  goto error;

      if((status = H5Pset_deflate( cparms, aggression)) < 0)
                  goto error;

      /* Create a new dataset within the file using cparms creation properties */
      if((dset_id = H5Dcreate2(file, DATASETNAME, H5T_NATIVE_INT, dataspace, H5P_DEFAULT,
                              cparms, H5P_DEFAULT)) < 0)
              goto error;

      /* Initialize data for one chunk */
      for(i = n = 0; i < CHUNK_NX; i++)
          for(j = 0; j < CHUNK_NY; j++)
              direct_buf[i][j] = n++;

      /* Allocate output (compressed) buffer */
      outbuf = malloc(z_dst_nbytes);
      z_dst = (Bytef *)outbuf;

      /* Perform compression from the source to the destination buffer */
      ret = compress2(z_dst, &z_dst_nbytes, z_src, z_src_nbytes, aggression);

      /* Check for various zlib errors */
      if(Z_BUF_ERROR == ret) {
              fprintf(stderr, "overflow");
              goto error;
      } else if(Z_MEM_ERROR == ret) {
              fprintf(stderr, "deflate memory error");
              goto error;
      } else if(Z_OK != ret) {
              fprintf(stderr, "other deflate error");
              goto error;
      }

      /* Write the compressed chunk data repeatedly to cover all the
       * chunks in the dataset, using the direct write function.     */
      for(i=0; i<NX/CHUNK_NX; i++) {
          for(j=0; j<NY/CHUNK_NY; j++) {
              status = H5DOwrite_chunk(dset_id, H5P_DEFAULT, filter_mask,
                                      offset, z_dst_nbytes, outbuf);
              offset[1] += CHUNK_NY;
          }
          offset[0] += CHUNK_NX;
          offset[1] = 0;
      }

      /* Overwrite the first chunk with uncompressed data.  Set the filter mask to
       * indicate the compression filter is skipped */
      filter_mask = 0x00000001;
      offset[0] = offset[1] = 0;
      if(H5DOwrite_chunk(dset_id, H5P_DEFAULT, filter_mask, offset, buf_size,
                   direct_buf) < 0)
              goto error;

      /* Read the entire dataset back for data verification converting ints to longs */
      if(H5Dread(dataset, H5T_NATIVE_LONG, H5S_ALL, H5S_ALL, H5P_DEFAULT,
                   outbuf_long) < 0)
              goto error;

      /* Data verification here */
                  :
                  :

History:

  Release | Change
  1.8.11 | C function introduced in this release.