$Header$ -*-text-*-

The netCDF Operators NCO version 4.6.8 are ready. 

http://nco.sf.net (Homepage, Mailing lists)
http://github.com/nco (Source Code, Releases, Developers)

What's new?
Many new features in 4.6.8 are, as usual, related to climatologies and
regridding. ncclimo can now be given an explicit list of seasons to
compute instead of, or in addition to, MAM,JJA,SON,DJF. ncremap has
improved mask and format handling. There are also a collection of
minor bugfixes and improvements, including JSON tweaks, ISO8601
printing, and GCC7 and netCDF 4.5.x compatibility. 

This release includes an important ncwa fix that prevents incorrect
answers when hyperslabs are used with masks and/or weights.
Upgrade if you use ncwa with -d hyperslabs.

PLEASE READ THIS IF YOU USE NCKS TO PRINT DATA:
The ncks default printing will change to CDL in NCO 4.7.0.
For 20+ years ncks has, by default, printed the text representation
of a file in what we now call "traditional" mode. This mode best
facilitates meticulous data examination in line-by-line format.
ncdump produces CDL format that is more useful for most NCO users.
ncdump has always printed clean CDL and for many years there was
little point in defaulting ncks printed output to CDL.
However, ncks CDL mode now rivals ncdump in many ways.
In particular, ncks --cln now prints times as human-readable calendar
dates, the last ncdump feature that I used which ncks lacked. 
Hence in NCO 4.7.0 ncks default printed output will change to CDL. 
Then one will type simply "ncks" instead of "ncks --cdl". 
We just added a new "--trd" option to print traditional output.
Add --trd to your scripts and their behavior will not change.
Otherwise your printing scripts will start to print CDL.
You have been warned :)

Work on NCO 4.7.0 has commenced. Planned changes include printing
CDL by default, and planned improvements include support for conda
installs on MS Windows, and more ncclimo and ncremap features.

Enjoy,
Charlie

NEW FEATURES (full details always in ChangeLog):

A. ncap2 now fully implements NCO chunking maps and policies.
   Previously ncap2 preserved existing chunking in variables,
   but could not be told to do anythin different. Now all command-line
   chunking behavior supported by NCO works in ncap2.
   ncap2 --cnk_plc=unchunk -S cnk.nco in.nc4 out.nc4
   http://nco.sf.net/nco.html#cnk

B. ncks CDL printing supports finer-grained control of date formats,
   including an ISO 8601 "T" option. Previously ncks printed
   UDUnits-compliant times as dates when invoked with the --cdl and
   --cal options. A third option, --dt_fmt, now exposes finer
   control of the format with short, regular, and ISO8601 options:
   ncks -H -m -v time_bnds -C --cdl --dt_fmt=1 ~/nco/data/in.nc
   ncks -H -m -v time_bnds -C --cdl --dt_fmt=2 ~/nco/data/in.nc
   ncks -H -m -v time_bnds -C --cdl --dt_fmt=3 ~/nco/data/in.nc
   dt_fmt:   Output:
   0,1       1964-03-13 09:08:16 (Default, short format)
   2	     1964-03-13 09:08:16.000000 (regular format)
   3         1964-03-13T09:08:16.000000 (ISO8601 "T" format)
   Note that --dt_fmt automatically implies --cal.
   http://nco.sf.net/nco.html#cln

C. Some data are best evaluated with custom-defined seasons, e.g., JFM
   instead of DJF, or two-month seasons such as FM or ON. ncclimo now
   supports up to eleven (and counting) seasons, although by default it
   only computes MAM, JJA, SON, and DJF. As of NCO 4.6.8, use the
   seasons option to specify additional or alternate seasons: 
   ncclimo --seasons=jfm,jas,ann -s 1980 -e 1983 -i drc_in -o drc_out
   Use "–seasons=none" to completely turn-off seasonal and annual-mean
   climatologies:
   ncclimo --seasons=none -s 1980 -e 1983 -i drc_in -o drc_out
   http://nco.sf.net/nco.html#ncclimo

D. ncremap --msk_src and --msk_dst options now accept the value 'none'
   to prevent the regridder from interpreting any variable as a mask
   in files from which source or destination grids are inferred.
   ncremap -i ~/ed.nc --msk_src=none -d ~/ed.nc -o ~/foo.nc
   ncremap -i ~/ed.nc -d ~/ed.nc --msk_dst=none -o ~/foo.nc
   http://nco.sf.net/nco.html#ncremap

E. ncremap now accepts standard NCO arguments for output file type.
   The -3, -4, -5, -6, and -7 flags, or their long-option equivalents
   --fl_fmt=fmt or --file_format=fmt where fmt is classic, netcdf4,
   64bit_data, 64bit_offset, or netcdf4_classic (or some synonyms)
   causes the output files created by ncremap, e.g., the regridded
   files and the grid files, to have the requested format.
   The ERWG and TempestRemap weight-generators support only a subset
   of these formats.
   ncremap -7 -i ~/deepthroat.nc -d ~/wapo.nc -o ~/foo.nc
   http://nco.sf.net/nco.html#fl_fmt_ncremap

BUG FIXES:

A. The ncap2 implementation of mibs()/mabs()/mebs() is fixed.
   Thanks to Dominique Briand for reporting this.

B. JSON now prints "null" instead of "NaN" for non-normal
   floating-point values like NaN and Infinity.
   This makes output fully JSON-compliant, although it simultaneously 
   makes it impossible for JSON parsers to determine whether a "null"
   floating-point value is NaN, or +/-Infinity.
   Thanks to Bob Simons for noticing this.

C. Fix longstanding (since ~4.2.3) ncwa issue where hyperslabs
   were not correctly handled for masks and weights. Hyperslabs
   that started after the first array element could lead to incorrect
   answers. Thanks to Tony Leboissetier for reporting the problem,
   and to Pedro Vicente for fixing it.   	    
   http://nco.sf.net#bug_ncwa_hyp_msk_wgt

D. Ensure ncclimo and ncremap load netCDF libraries 4.4.x+ on
   certain clusters so that CDF5 capabilities always available.
   Thanks to Milena Veneziani for reporting this.

KNOWN PROBLEMS DUE TO NCO:

   This section of ANNOUNCE reports and reminds users of the
   existence and severity of known, not yet fixed, problems. 
   These problems occur with NCO 4.6.8 built/tested under
   MacOS 10.12.6 with netCDF 4.4.1.1 on HDF5 1.10.1 and with
   Linux with netCDF 4.5.1-development (20170811) on HDF5 1.8.19.

A. NOT YET FIXED (NCO problem)
   Correctly read arrays of NC_STRING with embedded delimiters in ncatted arguments

   Demonstration:
   ncatted -D 5 -O -a new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc ~/foo.nc
   ncks -m -C -v att_var ~/foo.nc

   20130724: Verified problem still exists
   TODO nco1102
   Cause: NCO parsing of ncatted arguments is not sophisticated
   enough to handle arrays of NC_STRINGS with embedded delimiters.

B. NOT YET FIXED (NCO problem?)
   ncra/ncrcat (not ncks) hyperslabbing can fail on variables with multiple record dimensions

   Demonstration:
   ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc

   20140826: Verified problem still exists
   20140619: Problem reported by rmla
   Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
   Workaround: Convert to fixed dimensions then hyperslab

KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:

A. NOT YET FIXED (netCDF4 or HDF5 problem?)
   Specifying strided hyperslab on large netCDF4 datasets leads
   to slowdown or failure with recent netCDF versions.

   Demonstration with NCO <= 4.4.5:
   time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
   Demonstration with NCL:
   time ncl < ~/nco/data/ncl.ncl   
   20140718: Problem reported by Parker Norton
   20140826: Verified problem still exists
   20140930: Finish NCO workaround for problem
   Cause: Slow algorithm in nc_var_gets()?
   Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
   Workaround #2: Convert file to netCDF3 first, then use stride

B. NOT YET FIXED (netCDF4 library bug)
   Simultaneously renaming multiple dimensions in netCDF4 file can corrupt output

   Demonstration:
   ncrename -O -d lev,z -d lat,y -d lon,x ~/nco/data/in_grp.nc ~/foo.nc # Completes but file is unreadable
   ncks -v one ~/foo.nc

   20150922: Confirmed problem reported by Isabelle Dast, reported to Unidata
   20150924: Unidata confirmed problem
   20160212: Verified problem still exists in netCDF library
   20160512: Ditto
   20161028: Verified problem still exists with netCDF 4.4.1
   20170323: Verified problem still exists with netCDF 4.4.2-development
   20170323: https://github.com/Unidata/netcdf-c/issues/381

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/fxm
   More details: http://nco.sf.net/nco.html#ncrename_crd

C. NOT YET FIXED (netCDF4 library bug)
   Renaming a non-coordinate variable to a coordinate variable fails in netCDF4

   Demonstration:
   ncrename -O -v non_coord,coord ~/nco/data/in_grp.nc ~/foo.nc # Fails (HDF error)

   20170323: Confirmed problem reported by Paolo Oliveri, reported to Unidata
   20170323: https://github.com/Unidata/netcdf-c/issues/381

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/fxm
   More details: http://nco.sf.net/nco.html#ncrename_crd

D. FIXED in netCDF Development branch as of 20161116 and in maintenance release 4.4.1.1
   nc-config/nf-config produce erroneous switches that cause NCO builds to fail
   This problem affects netCDF 4.4.1 on all operating systems.
   Some pre-compiled netCDF packages may have patched the problem.
   Hence it does not affect my MacPorts install of netCDF 4.4.1.

   Demonstration:
   % nc-config --cflags # Produces extraneous text that confuses make
   Using nf-config: /usr/local/bin/nf-config
   -I/usr/local/include -I/usr/local/include -I/usr/include/hdf

   If your nc-config output contains the "Using ..." line, you are
   affected by this issue. 

   20161029: Reported problem to Unidata
   20161101: Unidata confirmed reproducibility, attributed to netCDF 4.4.1 changes
   20161116: Unidata patch is in tree for netCDF 4.4.2 release
   20161123: Fixed in maintenance release netCDF 4.4.1.1

E. NOT YET FIXED (would require DAP protocol change?)
   Unable to retrieve contents of variables including period '.' in name
   Periods are legal characters in netCDF variable names.
   Metadata are returned successfully, data are not.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:
   ncks -O -C -D 3 -v var_nm.dot -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to find variable

   20130724: Verified problem still exists. 
   Stopped testing because inclusion of var_nm.dot broke all test scripts.
   NB: Hard to fix since DAP interprets '.' as structure delimiter in HTTP query string.

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47

F. NOT YET FIXED (would require DAP protocol change)
   Correctly read scalar characters over DAP.
   DAP non-transparency: Works locally, fails through DAP server.
   Problem, IMHO, is with DAP definition/protocol

   Demonstration:
   ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc

   20120801: Verified problem still exists
   Bug report not filed
   Cause: DAP translates scalar characters into 64-element (this
   dimension is user-configurable, but still...), NUL-terminated
   strings so MD5 agreement fails 

"Sticky" reminders:

A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g., 
   HDF4: AMSR MERRA MODIS ...
   HDF5: GLAS ICESat Mabel SBUV ...
   HDF-EOS5: AURA HIRDLS OMI ...

B. Pre-built executables for many OS's at:
   http://nco.sf.net#bnr

