NCO Homepage
Author: Charles S. Zender
Keywords: NCO Homepage, netCDF, netCDF operator, GCM, HDF, scientific data

## NCO News and Announcements from 20030624 (version 2.8.3) and earlier

2003/06/24: A potpurri of old news which is no longer needed on the homepage

- As of version 2.8.3, `ncap` consumes significantly less memory than before.
- New “binary” operator `ncbo` does file addition, subtraction, multiplication, and division
- `ncap` adds user-invokable type conversion functions `char()`, `short()`, `int()`, `float()`, `double()`.
- `libnco_c++` reads into and writes from type `long double`
- Developer code has fully integrated `autoconf`/`libtool` build procedure
- NCO allows `ncap` to unpack variables stored with `scale_factor` and `add_offset` convention
- NCO includes `ncap`, a scriptable operator capable of performing arbitrary arithmetic operations
- NCO code includes libnco_c++, a new C++ interface to netCDF
- NCO now supports multithreading with OpenMP-compliant compilers
- NCO code contains working packing and unpacking primitives

2002/02/03: NCO version 2.2 is released. Differences with version 2.0

- `ncap`, the netCDF Arithmetic Processor
- `libnco_c++`, a C++ API for libnetcdf
- Various minor bugfixes introduced in 2.X releases

2001/05/07: NCO version 2.0 is released. Differences with version 1.2.2 are completely internal and oriented toward enhancing I/O performance and modernizing the NCO API:

- NCO 2.x is now completely based on the netCDF 3.x API. NCO 1.2.x will continue to be maintained for its ability to work with HDF4 files, but no new features will be added.

2001/02/11: NCO version 1.2.2 is released. Differences with version 1.2.1 include three new features and one bugfix:

- Weights are allowed with `ncwa` in min/max/ttl operations. New "var:att" syntax allows `ncrename` to rename attributes for specific variables rather than whole file, e.g.,

      ncrename -a os_name:eunuchs,unix in.nc out.nc

  Previously, `ncrename` would rename the same attribute for all variables in a file.
- `ncks` has new `-q` option to allow for printing arrays without printing associated dimensions
- The build system has been modified so that `make HDF4=Y` will build NCO with HDF4 compatibility (previously this had to be done by hand).
- The internal representations of `NC_CHAR` and `NC_BYTE` data have been swapped in order to agree with the netCDF standard. The effects of this last change should not be visible to the user.

2000/08/28: NCO version 1.2.1 is released. Differences with version 1.2 include one new feature and one bugfix.

1. Operators now perform automatic type conversion of integers to doubles before arithmetic. This feature allows `ncwa`, `ncra`, and `ncea` to handle averaging of longs, shorts, bytes, and chars in a more satisfactory manner. One immediate benefit is that `ncwa` no longer complains when user's try to apply weights (e.g., gaussian weights) all less than one to the averaging of an integer (e.g., `nlons`). Formerly `ncwa` would issue a warning, a workaround, and then dump core (because weights converted to integers were all zeros). Now that arithmetic is performed in double precision (rather than in the precision of the variable), `ncwa` handles this situation correctly.

2. A bug introduced in NCO 1.2 which broke the `-x` switch has been fixed

3. Thanks to Henry Butowsky for implementing the type conversion feature
Thanks to Ethan Davis for improving the build procedure

2000/07/30: nco-1.2 is released. Summary of new features and user-visible changes:

- The names of three of the operators have been made somewhat obsolete because they are no longer just arithmetic averagers. ncra, ncea, and ncwa have all been given brain transplants so that the type of operation they perform may be specified on the command line. The following operations are now supported by the new `-y` switch: average (default), total, minimum, maximum, root-mean-square, square of the mean, mean of the squares, square root. For example,

      ncra -O -y rms -n 12,4,1 h0001.nc 2000_rms.nc # Root-mean-square
      ncea -O -y min -n 12,4,1 h0001.nc 2000_min.nc # Minimum
      ncwa -O -y max -a lon,lat h0001.nc 200001_max_x.nc # Maxima

  Averaging is still the default so no existing usage needs to be modified. It is also possible to compute spatial and temporal variances and standard deviations using a combination of three or four NCO commands.
- All operators may now be DODS clients. DODS enables network transparent data access to any DODS server. Thus DODS-enabled NCO can operate on remote files accessible through any DODS server without transferring the files. Only the required data (e.g., the variable or hyperslab specified) are transferred. See the Users Guide for complete details.
- Fortran compatibility has been dropped. If a volunteer wants to continue to maintain the option to perform arithmetic in Fortran for reasons of speed, contact the project.

Summary of administrative changes:

- NCO is now an Open Source project licensed under the GPL. These new capabilities are the result of contributions from Henry Butowski, Bill Kocik, and Roland Schweitzer. For more information on how to contribute to NCO, visit the NCO administration homepage.
- New NCO releases and important announcements will be made on the netCDF list and on the new `nco-announce` mailing list.
- Questions, suggestions, ideas, and pleas for help should be posted to the appropriate discussion forum at the NCO administration site. NCO developers monitor these open forums, and your message will get a quicker response if it is posted on one of these forums:
  - Open Discussion Forum: General discussion, discussion of tips, tricks, future directions for NCO.
  - Help Forum: Installation problems, user's questions (e.g., "How do I average x and y...?")
  - Developers forum: Discussion of patches, technical questions related to modifying NCO, and discussion of specific items on the TODO list and how best to address them

Note: You must first register in order to post to these forums. Once registered you may choose to "monitor" the forums and to receive email when there are any postings.

2000/05/15: nco-1.1.48 fixes a bug introduced in nco-1.1.15 (November 1998) with `ncdiff` version 1.5. That bug could cause incorrect answers when the broadcast feature was used and dimensions were defined in a certain order (that appears to be favored by oceanographers). All NCO users are urged to upgrade NCO in order to avoid the potential `ncdiff` bug as well as to take advantage of the new features in `ncra` and `ncrcat`.

2000/05/10: nco-1.1.47 adds capability of `ncra`, and `ncrcat` to handle superfluous initial files. These multifile operators used to require that hyperslabs along the record dimension contain valid points in the first input file, although superfluous trailing files were allowed. Now superfluous files are also allowed anywhere in the file argument list for coordinate value hyperslabs (e.g., `-d time,2.0,3.0`), and at the beginning and end of the file argument list for dimension index hyperslabs (e.g., `-d time,2,3`). This can dramatically simplify analysis of file collections when the location of certain record coordinates or indices is not known a priori, or is tedious to compute. Users may now operate on specific segments of a timeseries (e.g., February values) without knowing which files contain February. Simply input the entire timeseries of files to the operators and they will automagically find and operate on only the February values. For example,

`ncra -O -d time,2.0,3.0 -n 1000,4,1 h0001.nc feb.nc`

`ncrcat -O -F -d time,2,,12 -n 1000,4,1 h0037.nc feb.nc`

Sylvia Murphy of the NCAR Community Climate System Model Support Network organized a workshop in February, 2000 which devoted some time to NCO. Here are links to the very useful pamphlet and slide presentation she developed to introduce and to summarize the NCO operators. The pamphlet is an excellent primer on NCO's capabilities, though not as definitive as the Users Guide.

2000/03/05: Source and prebuilt binary executable RPMs for NCO are now available for download.

2000/01/26: Fixed longstanding NCO bug which caused infrequent but unpredictable core dumps on machines with PIDs longer than 6 digits, e.g., all recent SGI machines. A core dump could occur on these machines when the PID was seven digits long and the filename length was one less than a multiple of eight. Kudos to Juliana Rew and Brian Eaton for identifying this bug. It is recommended that all users upgrade to the the new version. NCO is now distributed under the terms of the GNU General Public License (GPL).

1999/12/26: Added support for printing character arrays as strings in `ncks`. Added documentation and examples of stride capability to `ncks`, `ncra`, and `ncrcat` portions of Users Guide.

1999/12/14: Added feature to `ncatted` so that C-language escape sequences may be used in editing character attributes. Text formatting of attributes now supports `\n`, `\t`, and most other `printf`-style sequences. See the Users Guide for details.

1999/12/06: Fixed bug in `ncatted` so that character-valued attributes (strings) may contain commas, and be of zero length. Formerly `ncatted` would drop all text after the first comma.

1999/10/04: The build configuration has been simplified and made more robust. Now typing `make test` will invoke a self-test where NCO attempts to download a small (~65 kb) file on which to perform a battery of NCO tests. If the equations output by the `make test` command are numerically accurate, then you can have high confidence that NCO is installed and working correctly.

1999/08/31: A bug involving `ncwa`, weights, and missing values has been identified and fixed. When `ncwa` was used to average variables with user-specified weights (i.e., -w wgt was used) and every value in a target hyperslab was the missing value (i.e., missing_value), then `ncwa` would erroneously set the average value of the target hyperslab to 0.0 rather than to missing_value. Thus some values which should have been missing_value in the output file were, instead, set to 0.0. No operators besides `ncwa` were affected by this bug. This bug is fixed in nco-1.1.38. A new feature has been implemented in `ncwa`. `ncwa` used to exit with an error whenever it was called without any user-specified dimensions to average, i.e., when the `-a` was not invoked. Now when `ncwa` is invoked without any user-specified dimensions, it averages over all dimensions. Thus the behavior of `-a` is now analogous to that of `-d` and of `-v`. See the Users Guide for details.

1999/08/29: NCO now builds completely with pure C source files by default. In the past, NCO used Fortran routines to perform floating point arithmetic and pure C builds could be manually enabled by specifying -DC_ONLY in the build environment. The interface between Fortran and C, however, was very difficult to maintain on the wide variety of platforms supported by NCO. Now pure C builds are the default and Fortran arithmetic must be manually enabled by specifying -DUSE_FORTRAN_ARITHMETIC in the build environment. See the Users Guide for details. The Fortran code is no longer supported and will probably be deprecated in future releases.

1999/08/05: NCAR users will be pleased to know that NCO now takes advantage of the local `msrcp` command, if it exists in a standard location on the local network. This allows all operators to retrieve files from the NCAR mass storage unit when `msrcp` is available on the local system. This change is transparent to the user, so that commands should still specify the paths to the files on the mass store with, e.g., `ncks -O -R -l ./ /ZENDER/nc/in.nc foo.nc`. See the Users Guide for more details.

1999/07/03: `ncatted` now replaces missing values of variables whose missing_value attribute is changed. When the internal floating point representation of a missing value, e.g., 1.0e36, differs between machines then netCDF files produced on those machines will have incompatible missing values. `ncatted` now changes the missing_value attribute and the missing data self-consistently. This allows the user to change the missing values in files from different machines to a single value so that the files may then be concatenated together, e.g., by `ncrcat`, without losing any information.

1999/05/09: `ncks` now alphabetizes output by default. To turn off alphabetization and return to the previous behavior (variables reported in the order they were written to disk) simply specify ``-a'` on the command line. Yes, that means the behavior of ``-a'` has been toggled.

`ncrcat` and `ncra` now support a stride in the record dimension (and only the record dimension!), e.g., `ncra -d time,1,100,12 in1.nc in2.nc ... out.nc` should now work correctly. Strides in any dimension remain available with `ncks`, of course.

`ncrcat` and `ncra` now support index-based hyperslabbing in the record dimension across files, e.g., `ncra -d time,1,100 in1.nc in2.nc ... out.nc` The Users Guide claimed this feature had already been implemented, but, due to an oversight, that was not true. Attempting to use this new feature in older versions of NCO resulted in an "index out of range" error. Thanks to John Truesdale for pointing this out.

1999/04/20: Fixed bug where `ncrcat` and `ncra` omitted the last slice of the record dimension when ``-F'` (Fortran indexing) was user-specified AND user-specified hyperslab information was provided for some dimensions BUT not for the record dimension. This bug only affected `ncrcat` and `ncra` and only under these conditions. Thanks to John Sheldon for pointing this out.

1999/04/04: By default, extracted variables appear in the same order in which they were written to the source netCDF file. However, this order is often essentially random, making it hard to locate a particular variable in a dump of many variables. To help address this, `ncks` will now alphabetize variables. Simply specify ``-a'` on the command line.

1999/01/21: An optional token has been added to NCO which eliminates all netCDF 3.x calls and thus restores compatibility of NCO with HDF files.

1998/12/02: `ncwa` has been updated again since a user discovered a bug which occurred when a weight (``-w'`) and a mask (``-m'`) were both given to `ncwa`. Unfortunately, `ncwa` sometimes produced incorrect answers in this scenario so if you ever used weights and masks at the same time you are strongly urged to upgrade your NCO and rerun the commands. This bug only affected variables averaged with both the weight and mask options enabled at the same time, no other variables would have been affected.

The ``-n'` and ``-W'` normalization options to `ncwa` have been removed, possibly permanently, but at least until their implementation is simplified and redesigned. The ``-N'` option to `ncwa` remains the same. All the new behavior is fully documented in the rewritten and expanded `ncwa` section in the NCO Users Guide.

1998/11/24: Choose whether to weight your coordinates in `ncwa` A ``-I'` switch has been added to `ncwa`. Henforth, by default, `ncwa` applies weights or masks to coordinate variables just like any other variable. Previously, `ncwa` did not weight coordinate variables at all, and always produced simple arithmetic averages of coordinate variables. Use ``ncwa -I'` to recover the old behavior (i.e., arithmetic rather than masked, weighted averages of coordinate variables). A problem where `ncwa` would refuse to average variables which did not contain any of the averaged dimensions has also been fixed.

1998/07/07: Reduce bloating! The ``-h'` switch has been enabled on all operators. Using this switch suppresses the automatic history feature in NCO. Now you can decide whether each change to a file warrants an additional line in the history attribute. The functionality of the old ``ncks -h'` switch is now invoked with ``ncks -m'` (the mnemonic is `metadata').