
                         NCSA HDF Newsletter #1
                           December 12, 1989	

Contents:
	The HDF Newsletter
	The Current State of HDF
	HDF Release 3.0
	The HDF Vgroup: A More General HDF Structure
	HDF File Conversion 
	The NCSA HDF Staff


      --------------------------- ** ---------------------------

                          The HDF Newsletter

As HDF has begun to spread beyond NCSA many of you have requested 
some form of regular communication among all HDF users.  The HDF 
Newsletter is an occasional newsletter designed to provide that 
communication.  It is meant to serve as a clearinghouse of 
information about HDF.

The contents of this issue reflect the kinds of things I expect to 
include in the Newsletter, but I am quite open to suggestions.  I 
will print anything about HDF that seems as though it might be of 
interest to the HDF community.

We will distribute the Newsletter via both surface mail and email.  
If we have your email address, we will use email.  It is cheaper 
and (we hope) easier to manage than surface mail.

Mike Folk
NCSA
University of Illinois
605 E. Springfield Ave.
Champaign, IL 61820
217-244-0647
mfolk@ncsa.uiuc.edu (internet)
12409@ncsavmsa

    --------------------------- ** ---------------------------

                    The current state of HDF

FUNCTIONALITY
HDF Release 2.0, released last February and since then revised 
only for bug fixes, is still the most up-to-date official version 
of HDF.  It includes interfaces and file formats for handling 8-
bit raster images (with or withoout palettes), and regular gridded 
multidimensional floating point data sets.  It includes utilities 
for listing the contents of an HDF file (hdfls), for converting 
raw raster data to HDF (r8tohdf) and vice versa (hdftor8), and for 
displaying images or sequences of images from HDF files (hdfseq 
and hdfrseq).

NCSA WORKSTATION TOOLS
Most of the workstation tools now support HDF files.  This 
includes CompositeTool on the Sun, which previously did not 
support HDF.  (By the end of the year ImageTool (also Sun) will 
also support HDF files.)  The Mac tools are NCSA Image, NCSA 
PalEdit, NCSA DataScope, and NCSA Layout.  NCSA CompositeTool is a 
Sun version of Layout.  Our two X-Windows tools are NCSA X-Image, 
and NCSA X-DataSlice.

OFFICIAL PLATFORMS
Architectures that we have HDF running on: Cray-2 and Cray X-MP; 
Alliant (Concentrix), SGI 4, Sun-3, and Sun Sparc machines, 
Macintosh and IBM-PC.  And VMS, sort of (see below).

UNOFFICIAL PLATFORMS
People outside of NCSA have ported HDF to: Convex, Stellar, 
Connection Machine, Symbolics Machine, and Amiga.  (Do you know of 
any others?)  I think most of these people would be happy to give 
you advice on any special thing you need to do if you want to port 
HDF to one of these machines, but I probably shouldn't publish 
their names without their permission.  Let me know if you are 
interested in any particular port, and I will give your name to 
the relevant person.

SEMI-OFFICIAL PLATFORMS -- VMS & CTSS
VMS. A lot of you have asked for HDF support on VMS systems, so we 
made a quick attempt to port HDF to VMS last spring.  It was 
problematic from the beginning because we don't have any real VMS 
expertise here.  It actually went reasonably well in most 
respects, except for one major problem, which we're still 
struggling with: VMS-C writes files "stream-LF" format, which does 
not transfer well via ftp and some other transfer programs.  There 
is a VMS utility called FIXATR that we supply with VMS HDF, which 
converts stream-LF files to "fixed-512" files, which seems to be 
more amenable to transfer outside of the VMS environment.  This is 
not a great solution, since it involves an extra step whenever 
files go to or from VMS, but it may at least make it possible.  
Any suggestions for better solutions are certainly welcome.

CTSS. CTSS is a different story.  Not only don't we have CTSS/HDF 
expertise at NCSA; we also don't have access to CTSS any more.  If 
you are having problems getting HDF to work on CTSS, we will be 
happy to try to help you, but frankly we haven't been very 
successful diagnosing such problems recently.  If we can't help 
you, we may be able to put you in touch with someone who has 
successfully implemented HDF on CTSS.


     --------------------------- ** ---------------------------


                          HDF Release 3.0

After many delays, HDF 3.0 seems now to be ready for release. All 
of the features of HDF 2.0 are present in HDF 3.0, plus several 
new features.  

HDF 3.0 has been in available as a beta release for about 5 
months, and it seems pretty bug free.  But certain parts of it 
have hardly been used by anyone, so please let us know if 
something doesn't seem to work the way you think it should.

New features of HDF 3.0 include the following:

An interface for basic i/o of 24-bit raster images, which includes 
the following routines:

  DF24setil:   sets the interlace to be used when writing out the 
               RIS24 for the image.

  DF24addimage:appends a 24-bit raster image set to the file.

  DF24getdims: retrieves the dimensions and interlace of the
               image.

  DF24getimage: retrieves the image and stores it in an array.

  DF24reqil:   specifies an interlace to be used in place of the
               interlace indicated in the file when the next 
               raster image is read.


An interface for annotating HDF data objects and files, which 
includes the following routines:

  DFANgetlablen: gets length of label of a tag/ref
  
  DFANgetlabel:  gets label of tag/ref

  DFANgetdesclen: gets length of description of tag/ref

  DFANgetdesc:   gets description of tag/ref

  DFANputlabel:  puts label of tag/ref

  DFANputdesc:   puts description of tag/ref

  DFANlastref:   returns ref of last annotation read or written

  DFANlablist:   gets list of labels for a particular tag


An interface for input and output of 8-bit palettes, including the 
following routines:

  DFPaddpal:    appends a palette to a file.

  DFPgetpal:    reads in the next palette in the file.

  DFPputpal:    writes a palette to a file.

  DFPnpals:     indicates number of palettes in a file.

  DFPwriteref:  sets the reference number of the next palette to
                be written.

  DFPreadref:   gets the reference number of the next palette to
                be retrieved.

  DFPrestart:   specifies that the next call to DFPgetpal reads
                first palette in the file, rather than the next.

  DFPlastref:   returns value of the reference number most
                recently read or written.


Scientific data set routines for storing and retreiving subsets 
(slices) of scientific data, and for choosing optional storage 
formats and data types:

  DFSDstartslice: prepares to write part of dataset to file.

  DFSDputslice:   writes part of a dataset to a file.

  DFSDendslice:   indicates write completion for part of dataset.

  DFSDgetslice:   reads part of a dataset.

  DFSDsettype:    specifies data attributes: data type and 
                  representation, system type, and array order.


* new utilities, including the following:

  hdfed:    lets you browse in an HDF file and manipulate some of
            the data

  fptohdf:  converts floating point data to HDF floating point 
            data and/or 8-bit raster images

  r24tohdf: converts a raw RGB 24-bit image to an 8-bit RIS8 with 
            a palette

  paltohdf: converts a raw palette to hdf format

  hdftopal: converts palette in an hdf file to raw format

    --------------------------- ** ---------------------------


              HDF Vgroup--A More General Structure

HDF currently supports only two major types of scientific data: 
raster data and regular gridded multidimensional arrays.  Recently 
we have added an HDF structure that promises to expand 
significantly the types of data that we can support.  This 
structure, currently called Vgroup (the name may change), provides 
two important new structures:

    1. a general grouping structure that lets the user form
       groups out of any set of HDF objects, including other
       Vgroups

    2. a general structure made up of a set of record-like
       structures, each record being made up of a set of
       fields.  Fields can be use-defined or predefined.

Vgroups look very promising for a number of important scientific 
application areas not currently supported by HDF, including finite 
element and non-rectilinear mesh data.  We have talked with a 
number of scientists who work with this kind of data, and our 
general impression is that there is a need for a standard in this 
area and that Vgroups could well provide the standard.

The idea for Vgroup springs from a need to store 3-D polygonal 
data, with vertices, polygons (connectivity lists), and various 
associated values with each vertex or polygon. 

When Jason Ng took over the Vgroup project, he began talking to a 
lot of potential users from many different disciplines about how 
they might be able to use Vroups.  Their responses were so varied, 
that Jason immediately began looking for ways to generalize the 
concept so that it could handle many different kinds of data. The 
result is a very general HDF structure that "groups" one or more 
other HDF structures.  The structures in a Vgroup can be anything 
you want them to be including other Vgroups.  

For example, a Raster Image Set could probably be stored as a 
Vgroup.  The members of the Vgroup would be a palette, a dimension 
record, and an image.  But with the Vgoup concept we could now go 
a step further and group several Raster Image Sets, in an 
animation, for example.

While the Vgroup idea provides a general structure for linking HDF 
items together, we still need a structure for representing things 
like sets of vertices and connectivity lists.  The structure that 
we use for this is a very familiar one--a field and record 
structure.  Store 3-D vertices, we define three fields per 
element, corresponding to the x, y and z coordinates that define 
each vertex.  A vertex set is a fixed number of vertex records.  A 
polygon set is similarly defined.  If there are four vertices per 
polygon, each record consists of four vertex numbers; these 
numbers appear an order that describes the connectivity of the 
polygon.

In keeping with our desire to standardize those items that are 
likely to be accessed by different programs in different 
environments, certain types of sets will be predefined.  A 3-D 
vertex set will have exactly three fields per vertex, for 
instance.  But those who have the need are free to define their 
own dataset types.  For example, you might for some reason want to 
store scalar values in the same dataset that you store your 
vertices.  You are free to do this, but must recognize that you 
are building a non-standard dataset.  (Unless, of course, enough 
users ask us to make THAT one of the standard types.)

There are still some issues yet to be settled with respect to 
Vgroups, but we think that we are pretty close to having the major 
design of it pinned down.  The interface is now undergoing a major 
overhaul.  We expect to release a Beta version of it in mid-
January for any of you who would like to look at it and play with 
it.

Of course we welcome all comments and questions you have about 
Vgroups.  We don't want to freeze this structure too soon, because 
we see it as an important building block to HDF in the future.  On 
the other hand, we want to get it into use as soon as is 
reasonably possible.  If don't want to wait for the Beta release, 
contact us and we will send you the draft of the documentation.

     --------------------------- ** ---------------------------


                        HDF File Conversions

A frequent question that arises is "How can I translate between 
file format xxx and HDF?"  We want very much to support 
translators between HDF and other formats, but have so far had 
trouble finding the resources to write them.  Here is a list of 
some of the translators that we would like to have.  If you have a 
translator, know of one, are interested in working on one, etc., 
please let us know.  


FITS--Flexible Image Transport System
FITS is the standard format used for astronomical images and other 
digital arrays.  We have small collaborative project with the 
Space TelescopeScience Institute to translate basic FITS to HDF.  
We hope this will lead to a more elaborate project later.

CGM--Computer Graphics Metafile 
CGM is a very widespread file format that is used primarily for 
describing pictures.  Though CGM and HDF have different roles to 
play in scientific visualization, it would be nice to be able to 
look at CGM pictures using HDF-based tools, and vice versa.  Some 
programs that might help us do this: a CGM cell array-to-HDF 
converter, a rasterizer that converts CGM 2-D pictures to HDF 
raster, and a converter  that converts CGM text to HDF.  (HDF 
currently does not support text.)

(We have heard about a tool called CGM-Maker developed at Los 
Alamos that converts between CGM and Pict files, among others.  
Since NCSA Layout reads and writes both Pict and HDF files, CGM-
Maker and Layout together provide a kind of primitive filter 
between the two formats.)

netCDF
The netCDF interface allows users to share scientific data in a 
form that is self-describing and network transparent, and is very 
much in the spirit of HDF.  It is a well-designed, flexible 
interface, and one that would benefit HDF users enormously if it 
could be incorporated into the HDF library.  We are very 
interested in adapting HDF to support the netCDF interface, and 
also in writing translators that convert between the HDF and 
netCDF file formats.

TIFF--Tag Image File Format
Several HDF users have requested translators to and from this 
common image format and HDF.

     --------------------------- ** ---------------------------

                            The HDF Staff

In the last year and a half, the HDF project has expanded from one 
programmer to six people.  We lost Swami Natarajan, who finished 
his Ph.D last summer and took a job a Texas A & M.  We really miss 
him and continue to appreciate the work he did for us. Fortunately 
he is still in touch via email.

Mike Folk is the project manager for HDF.  Mike joined NCSA in 
August, 1988, having last worked at Oklahoma State University as a 
professor in the computer science department.

ChinChau Low has taken over Swami's responsibilities for 
maintaining HDF.  ChinChau is a graduate student in Computer 
Science at the University of Illinois.   ChinChau joined us in 
Fall, 1988, and has worked on virtually all aspects of HDF. He is 
crucial to maintaining HDF and also to all additions.

Jason Likkai Ng is a full-time staff member from Malaysia (via 
Cornell, Milwaukee, and La Jolla) who joined us in May.  Jason's 
main responsibility is the Vgroup project described earlier.

Peter Webb, Brian Calvert and Drew Hess joined the HDF group this 
fall semester.  Peter last worked at Schlumberger; Brian joins us 
from Motorola.  Peter and Brian are graduate students in Computer 
Science; Drew is an undergraduate in Computer Science.  

Peter's current projects include (1) installation of a revision 
control system to help manage HDF code development, and (2) 
finding ways to speed up HDF's performance.  Brian's project is 
Polyview, an SGI-based interactive tool for displaying polygonal 
data stored in the Vgroup format described above.  Drew is 
currently working on an HDF utility for taking slices out of 3-D 
scientific datasets.

