
                   NCSA HDF Newsletter #2
                       August 20, 1990

CONTENTS 
	HDF Version 3.1
	VMS
	DFSDgetslice and DFSDputslice documentation errors 
	Other releases
		The HDF V-set
		PolyView 1.0 (2-D and 3-D polygon viewer)
		NCSA Image 3.0 (Macintosh): late September
		Import2HDF (Mac): beta release
		Reformat, a Unix-based file converter: beta release
		Enhanced fptohdf coming soon
	HDF 4.0 in the works
	HDF and netCDF
			

Mike Folk (mfolk@ncsa.uiuc.edu)
--------------------------- ** ---------------------------

                      HDF Version 3.1

During the first half of July, we released Version 3.1 of HDF.  

The differences between HDF 3.0 and HDF 3.1 are not major.  (See 
the list below.)  If you have been using HDF 3.0, you should not 
need to change anything to your code or to the way you compile and 
link programs that invoke HDF.  If you have a program that runs 
correctly with HDF 3.0 but does not run correctly under HDF 3.1,  
let us know immediately.  (mfolk@ncsa.uiuc.edu (217-244-0647) or 
clow@ncsa.uiuc.edu (217-2440014))

--- Changes in 3.10 ---

These are changes made in Release 3.10:

    * fixed a bug concerning checking the status of opening a file
      with unbuffered i/o

    * Added function DF24readref and DFGRreadref for random access
      of 24-bit rasters

    * Added function DF24restart

    * Added function DF24setil

    * Speeded up the DFSDgetdata, DFSDputdata, DFSDadddata,
      DFSDgetslice and DFSDputslice functions, especially for 
      UNICOS machines.  (I/o performance on the Crays should be
      at least 10 times faster, and in some cases much more than
      that.) 

    * Repaired the DFSDgetslice and DFSDputslice routines, which
      did not work correctly.  Also, the treatment of these 
      routines in the  HDF 3.0 manual is incorrect.  The correct
      versions are described briefly later in this newsletter.

    * Added several functions to the annotations interface.  
      Most of the new functions are to allow you to annotate HDF
      files.  The already-existing annotation functions are for 
      annotating HDF objects within HDF files.  The new functions
      include:  DFANaddfid, DFABaddfds, DFANgetfidlen, DFANgetfid,
      DFANgetdslen, DFANgetfds, DFANaddfann, DFANgetfannlen, 
      DFANgetfann and DFANlastref.

    * Revised DFANlablist so that it returns all reference numbers
      for a given tag

    * Fixed bug with DFSDgetdata where it did not move to the next
      SDG

    * Added some macros to make passing character arrays from
      fortran to C easier

    * Other minor modifications. 

 --- Updated manual soon ---

An updated version of the HDF manual should be available soon.  We 
will send out another announcement when it is.

--- Supported platforms ---

With this release, we now support the HDF library on the following 
platforms:

	Cray 2 and X-MP (UNICOS)
	Suns (Unix)
	SGI (Unix)
	Alliant (Concentrix)
	Macintosh (Mac OS)
	IBM PC (MS-DOS)
	Vax (VMS)

The Macintosh, PC, and VMS versions are somewhat special, 
unfortunately:

* On the Mac, you need MPW C and Languages Systems Fortran (we're 
trying to make it work with Absoft Fortran as well).  You need MPW 
C even if you only want to write Fortran calls, because HDF makes 
calls to MPW C routines that are proprietary and cannot be 
included with the library.  We are also playing with AUX 2.0 on 
the Mac, and may support an AUX-based HDF soon.

* The PC version currently requires Lattice C, but we are 
currently switching over to Microsoft C, which seems to be more 
popular.

* See the writeup on VMS HDF below.


--- Other platforms ---

There are several other machines that HDF is running on 
successfully, but that we can't officially support because we 
don't have them here.  We have started a HINTS file (on anonymous 
ftp in the HDF/src directory) where we put hints on how you might 
go about porting HDF to your particular platform.  When people 
send us information about new machines that they have ported HDF 
to, we put that information in this file.  We can provide 
information (or put you in touch with someone who can) on how you 
can probably make HDF work on the following machines:

	Cray (CTSS)
	Apollo
	MIPS machines (e.g. DEC 3100 (Ultrix))
	Convex 
	Stellar

In general, if you have a Unix platform, there is a good chance 
that it is similar enough to one of the machines that we do 
support that you can get HDF working on it.

Also, a group at UCLA has been pouring a lot of effort into porting
HDF to their IBM 3090 MVS and VM systems.  They have had a lot
of success, and we expect them to be able to share their results
with us soon.  Stay tuned.


--------------------------- ** ---------------------------

                           VMS

The VMS port finally seems to be working pretty well, at least if 
the lack of bug reports on it is any indication.  We owe a real 
debt of thanks to several people for substantial amounts of help 
getting VMS to work, including Joe St. Sauver at Oregon, Ron 
Schmucker at Lawrence Livermore National Laboratory, Brian Wallace 
at Oak Ridge National Laboratory, and Bob Mark of the U.S. 
Geological Survey.  Future suggestions for ways to improve our VMS 
support are extremely welcome.

I am pleased to report that we managed to scrounge a VaxStation II 
a couple of months ago, and just recently have gotten it to be 
functional with VMS.  Now, with luck we can keep a really stable 
HDF working.

 --- fixatr ---

One frequent question we get asked is how to use the fixatr 
routine.  The fixatr routine needs to be used to convert between 
VMS's Stream-LF format, which VMS C reads and writes, and fixed-
512 format, which ftp and other transfer programs work best with.  
The README.VMS file explains how to use fixatr.

One problem that some people have encountered that has not been 
properly covered in the README.VMS file is the need to identify 
correctly the file recformat.exe that is invoked on the second 
line of fixatr.cld.  After executing makefix.com (enter: @makefix) 
you need to find out the full path name to recformat.exe.  This 
path name has to be substituted in the proper place on line 2 of 
the file fixatr.cld, which originally reads:

	image disk$system:[fixatr]recformat

For instance, if the directory that contains recformat.exe is 
sys$login:[hdf], you change the line to read 

	image sys$login:[hdf]recformat.exe

Also, before executing fixatr, you need to execute the command:

	set command fixatr

--- Utilities on VMS ---

Another problem several people have encountered has been in using 
the utilities, such as hdfls and fptohdf.  If a program name (e.g. 
hdfls) is to be treated as a command procedure, VMS requires that 
you assign (using :==) the name to the full path of the 
corresponding executable procedure.  For example, if the full path 
name for the executable hdfls is sys$login:[hdf]hdfls.exe, you 
would enter

	hdfls :== sys$login:[hdf]hdfls.exe

We have included with the new release on anonymous ftp a file 
called setuputils.com that does this for you for all of the hdf 
utilities.  (You have to change a path name within setuputils.com 
to correspond to your system before you execute it.)

 --- hdfrseq ---

The utility hdfrseq requires special treatment when use from a VMS 
machine.  The problem is as follows: hdfrseq sends a stream of 
bytes via telnet to your terminal.  If this stream of bytes 
doesn't have a line-feed every 512 bytes or less, VMS in its 
infinite wisdom adds one for you.  This of corrupts the image, and 
you get wierd streaks in your output.

Brian Wallace has given us a simple solution to this problem:  You 
can eliminate the extra line-feed by entering the following line 
before you execute hdfrseq: 

set terminal/nowrap

(After executing hdfrseq, you may want to set the terminal back to 
its original modes with something like "set terminal/wrap.")

In release 3.1 this is done for you, as hdfrseq is included in a 
".com" file that sets "nowrap" before executing hdfrseq.

--------------------------- ** ---------------------------

                DFSDgetslice and DFSDputslice 
                 documentation errors fixed


The documentation for DFSDputslice and DFSDgetslice is wrong in 
the most recently published version of the HDF manual.  Most 
important is that the parameter lists shown in the documentation 
are incorrect.  Here are some excerpts from the new documentation 
that explain how the routines should be called.


 --- Writing Parts of a Scientific Dataset ---

To store an array in slices, make calls to DFSDstartslice, 
DFSDputslice, and DFSDendslice in the following order:

DFSDstartslice(filename)
DFSDputslice(windims, data, dims)
DFSDputslice(windims, data, dims)
...
DFSDputslice(windims, data, dims)
DFSDendslice()


 --- DFSDstartslice --- 

FORTRAN:
INTEGER FUNCTION dfsdstartslice(filename)
CHARACTER*64 filename

C:
int DFSDstartslice(filename)
char *filename;    /* name of HDF file */

Purpose:  To prepare the system to write a slice to a file. 
Returns:  0 on success; -1 on failure.

Before DFSDstartslice is called, DFSDsetdims must be called to 
specify the dimensions of the dataset to be written to the file. 
DFSDstartslice always appends a new dataset to an existing file. 


 --- DFSDputslice --- 

FORTRAN:
INTEGER FUNCTION DFSDputslice(windims, source, dims)
INTEGER windims(*) 
REAL	source()
INTEGER	dims(*)

C:
int DFSDputslice(windims, source, dims)
int32 windims[];    /* dimensions of slice*/
float32 *source;    /* array for storing slice*/
int32 dims[];        /* dimensions of array source*/

Purpose:  To write a slice to an SDS 
Returns:  0 on success; -1 on failure.

DFSDputslice stores part of an array to the dataset last declared 
by DFSDsetdims.  Slices must be stored contiguously.  

Array windims ("window dimensions") specifies the size of the 
slice to be written. windims has as many elements as there are 
dimensions in the entire SDS array. source is an array containing 
the slice. 


 --- DFSDendslice --- 

FORTRAN:
INTEGER FUNCTION DFSDendslice()

C:
int DFSDendslice()
Purpose:  To specify that the entire dataset has been written.
Returns:  0 on success; -1 on failure.

DFSDendslice must be called after all the slices are written. It 
checks to ensure that the entire dataset has been written, and if 
it has not, returns an error code.


 --- Example: Writing slices to a 10x12 SDS. --- 
/****************************************************
*
* Example C code:    Write out slices of different sizes 
*        from a 10 x 12 array.
*
****************************************************/

...

    int rank;
    int dimsizes[2], windims[2];
    float data[10][12];

    /* code that builds the array goes here */
    ...

    dimsizes[0]=10;
    dimsizes[1]=12;

    DFSDsetdims(2,dimsizes);

    /* write out scientific data set in slices */
    DFSDstartslice(filename);

    windims[0]=2; windims[1]=12;    /* {(1,1) to (2,12)} */
    DFSDputslice(windims, &data[0][0], dimsizes);

    windims[0]=4; windims[1]=12;    /* {(3,1) to (6,12)} */
    DFSDputslice(windims, &data[2][0], dimsizes);

    windims[0]=1; windims[1]=4;    /* {(7,1) to (7,4)} */
    DFSDputslice(windims, &data[6][0], dimsizes);

    windims[0]=1; windims[1]=8;    /* {(7,5) to (7,12)} */
    DFSDputslice(windims, &data[6][4], dimsizes);

    windims[0]=3; windims[1]=12;    /* {(8,1) to (10,12)} */
    DFSDputslice(windims, &data[7][0], dimsizes);

    DFSDendslice();

...


 --- Reading Part of a Scientific Dataset --- 

The routine DFSDgetslice lets you read in a slice from an SDS.  A 
slice is an array of elements that is a subarray, or "hypercube", 
of  the SDS from which it is read.  (Note that, for the purposes 
of reading slices, the definition of a slice is more general that 
it is for writing slices.)

A slice can be described with two one-dimensional arrays, one 
containing the coordinates of the corner that is nearest to the 
origin and the other containing the sizes of the slices 
dimensions.   



 --- DFSDgetslice --- 

FORTRAN:
INTEGER FUNCTION DFSDgetslice(filename,winst,windims dest, dims)
CHARACTER*(*) filename
INTEGER    winst
INTEGER    windims
REAL    dest
INTEGER    dims

Purpose:  To read part of an SDS from a file.
Returns:  0 on success; -1 on failure.

DFSDgetslice accesses the dataset last accessed by DFSDgetdims. If 
DFSDgetdims has not been called for the named file, DFSDgetslice 
gets a slice from the next dataset in the file.

Array winst specifies the coordinates of the start of the slice.  
Array winend gives the size of the slice.  The number of elements 
in winst and winend must be equal to the rank of the dataset. For 
example, if the file contains a three dimensional dataset, winst 
may contain the values {2, 4, 3}, while windims contains the 
values {3,1,4}. This will extract a 3 x 4, two-dimensional slice, 
containing the elements between (2,4,3) and (4,4,6) from the 
original dataset. 

dest is the array into which the slice is read. It must be at 
least as big as the desired slice.

dims is an array containing the actual dimensions of the array 
dest. The user assigns values to dims before calling DFSDgetslice. 

NOTE: All the parameters on the call assume FORTRAN-style 1-based 
arrays.


 --- Example --- 

/****************************************************
*
* Example C code:    Read in slices from a 10 x 12 array. 
*
****************************************************/
#include "df.h"
...

	int  i, rank;
	int32 dimsizes[2];

	DFSDgetdims(filename, &rank, dimsizes, 2);

	/* starting at (3,4) read 4 x 6 window */
	getit("myfile", 3,4,4,6); 

	/* starting at (1,10) read 10 x 2 window */
	getit("myfile", 1,10,10,2);

	printf("\n");

}

getit(filename, st0, st1, rows, cols)
int st0, st1, rows, cols;
char *filename;
{
	int i, j;
	int32 winst[2], windims[2], dims[2];
	float32 data[500];

	winst[0]=st0; winst[1]=st1;
	dims[0] = windims[0] = rows;
	dims[1] = windims[1] = cols;
	DFSDgetslice(filename, winst, windims, data,dims);

	for (i=0; i<rows; i++)  {
		printf("\n");
		for (j=0; j<cols; j++)
		 printf("%5.0f%c",data[i*cols+j], ' ');
	}
	printf("\n");
}

--------------------------- ** ---------------------------

                       Other Releases

Here are some short takes on other items of interest.  All of 
these items can be gotten from our anonymous ftp server 
("ftp.ncsa.uiuc.edu" (128.174.20.50)).  For those that are 
officially released, you can also order them on tape (and 
sometimes on disk) through our technical resource catalog.  To 
obtain a catalog, contact:

	NCSA Documentation Orders
	152 Computing Applications Building
	605 East Springfield Avenue
	Champaign, IL 61820
	(217) 244-0072


 --- HDF Vset --- 

The HDF Vset (formerly "vgroup") structures and interface are now 
available in a separate directory on anonymous ftp.  We described 
this structure in the previous newsletter.  Here is the gist of 
what we said:

Vset provides two important new structures:

    1. a general grouping structure that lets the user form groups 
out of any set of HDF objects, including other Vgroups

    2. a general structure made up of a set of record-like 
structures, each record being made up of a set of fields.  Fields 
can be use-defined or predefined.

Vgroups are useful for a number of important scientific  
application areas, including finite element and non-rectilinear 
mesh data, and  3-D polygonal  data.

The primary use that have made so far of Vsets is in storing data 
for use with our SGI-based polyview program.  For Polyview, we 
store 3-D vertices, connectivity lists (polygons), and associated 
scalar data.  See the description of Polyview below.

Vset is currently in a separate library, but we plan to integrate 
it with the regular HDF library with the next full release of HDF, 
planned for late Fall of this year.  You can find Vset in a 
separate directory on anonymous ftp, or contact Jason Ng ((217)-
244-8524; likkai@ncsa.uiuc.edu).



 --- PolyView 1.0 (2-D and 3-D polygon viewer) ---

PolyView is an interactive visualization tool for HDF Vset data. 
PolyView  displays an HDF Vset of polygons or points that describe 
a two or three dimensional, interactive image with optional 
annotation.  PolyView-produced images may be written to a RIS8 HDF 
format file.  The program also allows you to:

	-- change display projection
	-- render image as points, lines, or polygons
	-- choose constant or gouraud shaded polygons
	-- load and manipulate the colormap
	-- animate a series of vdata sets
	-- view a fly-by of the data using a script file

PolyView is only supported on the Silicon Graphics Personal IRIS 
4D/20G (24-bit color, 24-bit Z-buffer).  It makes extensive use of 
Z-buffering.  Although it has not been ported to or tested on 
higher-end IRISes, it should run on most recent models.

Sample code which demonstrates the creation of an HDF
Vset file is included.

 --- NCSA Image 3.0 (Macintosh): late September --

This is the next upgrade for the Image program.  It will support 
several new features:

     * Distributed computing capabilities
     * HDF list windows (discussed below)
     * Enhanced 3D support
     * Bug fixes and minor upgrades
     * Animation from a single file

The HDF list window allows one to view the tag/refs in an HDF 
file.  In Image three kinds of tag/refs are shown (RIG, SDG and 
IP8).  The HDF list window allows one to select any of the 
tag/refs and display its "contents."  The list window also allows 
one to annotate and label any of the tag/refs in the window.

The code for the list window was written in a portable fashion for 
MPW C 3.0. It should be easy to add the list window to most Mac 
programs.  We have already added it to Layout and plan on putting 
it into the other Mac tools (DataScope and PalEdit).


 ---. Import2HDF (Macintosh): Beta release ---

Import2HDF is a program that allows HDF users to convert files in 
other formats to HDF format. This program provides rudimentary 
display capability for HDF file contents. When you open an HDF 
file it simply lists the contents of the file in a window. From 
here, individual elements (RISs and SDSs) can be displayed. 
Annotations for data groups can be added or changed. 

This program is currently in BETA release form, and should be 
released in early fall.


 --- Reformat, a Unix-based file converter: Beta release ---

This utility provides a mechanism by which images can be 
transformed from one storage format to another.  It was initially 
developed to facilitate the conversion of TIFF, FITS and GIF files 
to HDF files, so the conversion routines are much the same as 
those used in the NCSA MacIntosh tool, Import2HDF.

The graphical user interface requires X version 11, release 4 and 
the Athena Widget set.  It has been tested with release 2 and 
higher servers.  

There is also a command line interface that does not require X.


 ---  fptohdf enhancement coming --

Bob Weaver at INEL recently sent me a greatly improved version of 
fptohdf.  We haven't installed this new version on anonymous ftp, 
but we hope to do so soon.  The current version accepts only 2-D 
data sets that are text or hdf SDS files.  The new version can 
accept 3-D scientific data sets, and also 32-bit and 64-bit raw 
binary files.

We will put the new version on anonymous ftp soon.  (We'll call it 
fp2hdf to distinguish it from the old version.)



--------------------------- ** ---------------------------

                      HDF 4.0 in the works

HDF is currently undergoing a major overhaul.  The primary goals 
of the rewrite of HDF are 

    * to improve the underlying code structure, based on what we
      have learned over the past two years
    * to allow multiple file access
    * to integrate the Vset interface into HDF
    * to improve HDF's error handling facilities.

All of the current interfaces will be supported in HDF 4.0, but 
there will also be new, corresponding interfaces that permit 
greater user control over file access.

We will provide more details on HDF 4.0 in a future Newsletter.


--------------------------- ** ---------------------------

                         HDF and netCDF


NetCDF is an interface for data access produced by the Unidata 
Program Center at the University Center for Atmospheric Research.  
It is a excellent interface, providing a very effective data 
abstraction model for describing scientific data.  We discussed 
with Unidata the possibility of incorporating the netCDF interface 
in HDF, and have concluded that it is something that we should do.  
We are currently looking for funding to undertake this project, 
which would be fairly substantial.

We will give more details about this effort in a future 
newsletter.
--------------------------- ** ---------------------------



