GNU Astronomy Utilities NEWS                          -*- outline -*-

* Noteworthy changes in release 0.3 (library 1.0.0) (2017-06-01) [stable]

  This is a full re-write of Gnuastro. Most importantly, Gnuastro now has a
  new generic data contaihtner (`gal_data_t'). This new container can now
  deal natively with all standard numeric data types, work in RAM or
  HDD/SSD, keep data in any dimensions and has enabled many other very
  useful features. Some of the most prominent of the new features are
  discussed below. It is strongly recommended to review the respective
  section of the Gnuastro manual/book for a better feeling of all the new
  features.

  As discussed below, some program names have changed, if you have a
  previous version of Gnuastro installed from source, it is recommended to
  uninstall it first (with `make uninstall' using the corresponding
  tarball), then install this new version. Building Gnuastro can be slow,
  so please build in parallel with Make's `-j8' option (to build on 8
  threads).

** New programs or library features

  Library functions that deal with datasets now use this generic data
  container for inputs and outputs, significantly simplifying their
  API. Nearly all library functions have been re-written with much more
  clear names, argument lists and individual purpose. Some example library
  functions are shown below, also see the "Library demos" section of the
  book for some complete working example:

     -- `gal_table_read' and `gal_table_write' will read and write data to
        plain text, FITS ASCII and FITS Binary formats.

     -- `gal_fits_img_read' and `gal_fits_img_write' can read a FITS image
        to memory or write a FITS image from memory.

  Gnuastro now defines a simple comment line format to keep basic
  information in a plain text table, see the "Gnuastro text table format"
  section of the book. In short for every column, a comment line like below
  can be used to give a name, units, comments, or a type to a column. This
  allows a FITS binary table for eample to be written to plain text and
  converted back to binary without loosing any information (except for very
  small floating point errors if not enough decimals are printed).

      # Column N: NAME [UNIT, TYPE, BLANK] COMMENT

  The new Fits program replaces the old Header program. But besides
  reading/checking FITS header keywords, it can now also work on FITS
  extensions/HDUs. For example with no options, it will list all the HDUs
  in a FITS file along with basic information. It can copy a whole HDU to
  another file, or delete a HDU from a FITS file. To get the previous
  behavior of listing all the keywords in a FITS HDU, you can run it with
  the `-p' option.

  All programs now write data into the second HDU of a FITS file to allow a
  clean first HDU. Note that following CFITSIO, HDU counting still starts
  from zero, so FITS images and tables written by Gnuastro in a new file
  can always be accessed with the `--hdu=1' option (which is now the
  default).

  If any program is run within a Git version controlled directory, a
  `COMMIT' header keyword will be added to the created FITS files, see the
  "Output headers" section of the book for a discussion on the usefulness
  of this new feature.

  BuildProgram: a new program to easily compile, link and run a C program
  you have written with Gnuastro's libraries without having to worry about
  which libraries (Gnuastro dependencies) your program needs. Debugging
  (`-g'), optimizations (`-O'), warnings (`-W'), include search path
  (`-I'), link search path (`-L'), and linked libraries (`-l') compiler
  options are also supported. BuildProgram will greatly facilitate the easy
  usage of Gnuastro's libraries.

** Removed programs or library features

  The following program names have been renamed:

    ImageCrop       --> Crop
    ImageWarp       --> Warp
    ImageStatistics --> Statistics
    Header          --> Fits

  SubtractSky has been removed. The Statistics program now has tools to
  estimate the Sky value and it can be subtracted with Arithmetic.

** New features

  All Gnuastro programs that read and write tables can now do so in plain
  text table format or in FITS ASCII or FITS Binary tables. Depending on
  the filename or with the new `--tableformat' common option to all
  programs.

  The option management system in all Gnuastro programs has been completely
  re-written with many new features, some of the most important ones are
  listed below. For developers, you will notice that there is no more usage
  of macros and adding new options has become much more easier.

     -- All programs will now also look for a `gnuastro.conf' configuration
        file to keep common options for all programs in every directory.

     -- The `--lastconfig' opion can be used on the command-line or in any
        configuration file to stop parsing any further configuration files.

     -- The `--config' option can now be used to identify any arbitrary
        file to be parsed as a configuration file. Any file that is given
        to this option is parsed immediately.

     -- The `--printparams' option now also prints the short documentation
        of each option (same description in `--help') after its value.

  It is now possible to choose columns in tables based on column names as
  well as column numbers. It is also possible to search for columns based
  on searching in their units or comments. When using column numbers,
  counting now starts from 1 (one), not 0 (as before). See the new
  "Selecting table columns" section for more on these new features.

  Where relevant, all programs now accept a `--type' option that you can
  use to specify the numerical datatype of the output.

  With the new common option `--minmapsize', you can specify a minimum size
  of an array (in bytes) to store data in SSD/HDD and not in RAM. This can
  be instrumental when you are dealing with large datasets, or even smaller
  ones, but when your RAM is getting full.

  Making a log file is now optional and users have to explicitly ask for it
  with the `--log' option.

  Slower building of Gnuastro: Binary operators (e.g., plus or multiply)
  are now done in the native type of the input dataset. Doing so for all
  the different combinations of types, greatly slows down the initial
  compilation of Gnuastro (after running `make'). So for every type there
  is now a `--enable-bin-op-*' configure time option. When the dataset's
  type isn't compiled (only for the binary operators), it will be converted
  to a compiled type and then converted back in the end.

  Arithmetic: all operations are done in the native data type of the
  dataset. Until now, it would convert the data internally to double
  precision floating point, do the requested operation and write the data
  back in the proper type. So this new implementation is much more
  efficient.

  Arithmetic: two new classes of operators: type conversion operators to
  all standard types and integer-only operators (all C bit-wise operators
  and the modulo operator).

  ConvertType: can also print the input dataset to the command-line
  (`stdout'). To use this feature set the output filename to `stdout'.

  Convolve now has the `--minsharpspec' option to specify the minimum
  spectrum value to use in deconvolution (matching PSFs).

  Crop: when in WCS mode it can still only work on aligned images. However,
  very small floating point errors in writing the WCS (for example 10e-13
  degrees) are now acceptable. Until now these would cause Crop to complain
  and abort.

  Crop: the name of the crop can be pre-determined based on the values in a
  given in a table column. This allows your object's IDs to be directly
  used as the crop's file name for example.

  MakeCatalog now also reports the surface brightness in
  mag/arcsec^2. Until now, MakeCatalog would only print the dataset's
  surface brightness in units of magnitudes/pixel. But that is not nicely
  comparable to other datasets. Hence, using a simple calculation (from the
  pixel projected size, fully derived in the book), it now also reports the
  surface brightness in magnitudes/arcsec^2 also.

  MakeProfiles: Profile codes now start from `1' (until now they started
  from `0').

  MakeProfiles: now accepts the radial function of profiles as
  human-readable strings instead of a code for each profile (which was very
  cryptic, although codes are also still acceptable). For example in the
  profile column you can now write `sersic' instead of the code `1'.

  NoiseChisel: the new `--cleandilated' option will remove dilated objects
  that have a low S/N (it is mainly useful on very clean or mock images).
  For non-clean noise, it will result in a decrease of completeness. With
  this new option, NoiseChisel will also print detection S/N values when
  run with the `--checkdetsn' option.

  Statistics: now reads table columns as well as images and does basic
  operations on them. It can also only work on a certain range of the data
  (instead of the whole set). Alternatively, you can define the range on
  another reference column, but use values of the main column.

  Statistics: all its single-valued measurements can now be done on a
  tessellation (tile grid) over the input dataset.

  Statistics: can now estimate the Sky value on the input dataset using the
  mode's quantile similar to what NoiseChisel does to find its initial
  threshold.

  Statistics: has several new single valued calculations: `--quantile',
  `--quantfunc' (quantile function), `--mode', `--modequant', `--modesym',
  and `--modesymvalue'.

  Warp: align the image with the celestial coordiates using the `--align'
  option.

  Warp: standard modular warpings can now be requested without an input
  matrix, using the following options: `--shear', `--flip', `--project',
  `--rotate', `--scale', `--translate'. Any number of these transformations
  (along with the `--align' option) can be called on the command-line and
  they will be applied in the same order to create one warping matrix. By
  default the WCS will also be corrected.

** Changes in behavior

  Mask image options have been removed from all programs. Instead, all
  programs can work directly on data with blank values. So when some pixels
  must be masked, the Arithmetic program's `where' operator can be used to
  select special pixels and set them to blank. In particular bit-wise
  operations are now available in Arithmetic to use bit-mask
  images. Managing all these different choices in every program would only
  confuse the user (with too many options).

  Arithmetic: the `x' letter is now used to represent the multiplication
  operator. Previously it was `*' which needed quotation and was thus very
  inconvenient.

  Convolve: the old `--frequency' and `--spatial' options have been removed
  and are replaced by `--domain' which accepts values of `frequency' and
  `spatial'.

  Convolve: the old `--viewfreqsteps' was changed to `--checkfreqsteps' to
  fit with the general style of such check images in all Gnuastro's
  programs.

  Crop: `--section' syntax is now inclusive in both bounds.

  Crop: only checks if the center of a crop is filled when the crop was
  defined by its center (for example with `--ra' and `--dec'). The verbose
  outputs of Crop are also not cryptic 0s or 1s. The are human readable
  text.

  Crop: doesn't have separate `--imgmode' and `--wcsmode' options any
  more. There is now a single `--mode' option which accepts values of `img'
  or `wcs'.

  MakeProfiles: the old `--inputascanvas' is now called `--clearcanvas'.

  MakeProfiles: until now, it would abort with an error when the input
  columns had blank values. But for masking, it might happen that you set a
  blank magnitude. So this check has now been removed when reading the
  magnitude column.

  NoiseChisel: default value of the `--minskyfrac' option (new name for the
  old `--minbfrac') is now 0.7 as opposed to 0.5. This will allow much
  better estimation of noise properties (by default). It may be slightly
  too high for a crowded field, but the users can change it on the
  command-line (or in a configuration file) for such datasets.

  NoiseChisel: when it is run with any of the `--check' options, it will
  abort after all the check images have been created. This is very useful
  for checking your parameters until each step and not be distracted (or
  have to wait) for later steps to finish.

  Statistics: will not make a histogram and cumulative frequency files, or
  calculate sigma-clipped results by default (with no options). It will
  just print some basic information.

  Table: Previously, if a column was requested, the `-i' option would be
  ignored. But it often happens that the users forget a column name after
  already typing several of their desired columns. So the opposite behavior
  is preferred. Because when more than a couple of columns are needed, you
  will probably forget the column identifiers of the last few and having to
  retype everything is very frustrating. Something like how `--help' takes
  precedence over all other options.

  Table: to select column(s) by regular expression searching, the name now
  has to be put in `/ /' (similar to AWK). If a value isn't in `/ /', the
  programs will only select a column with the exact match.

  Warp: when a 2 by 2 matrix is given, the FITS pixel positions (which
  define the center of a pixel as an integer) are automatically implemented
  internally, see "Invoking Warp" in the manual for more.

  Warp: the old `--nofitscorrect' option has been changed to
  `--centeroncorner' to be more clear. The new option is now more general
  than before and also works on warping with a matrix, not just on modular
  warpings.

  Warp: the old `--nowcscorrection' option has been given a more clear name
  of `--keepwcs'. With this option, Warp will not apply the warp the
  input's WCS structure.

  Warp: the old `--maxblankfrac' option has been changed to
  `--coveredfrac'. Until now, Warp would only look for the fraction of
  input blank/NaN pixel area over the output pixel. But this would be
  useless on the edges of the image. So the new `--coveredfrac' option
  takes the acceptable fraction of output pixel area that must be covered
  by input pixels in order to give that output pixel a value. You can use
  this to set edge pixels that are not fully covered in the new grid to
  blank and have a flat warped image.

** Bug fixes

  Using `%zu' to print `size_t' variables for clean build on 32-bit
  systems.

  Crash in Table for some operating systems due to memory is now fixed (bug
  #49347).

  Table's man-page is now created and installed (bug #49418).

  Fixes in the documentation (sr #109170, bug #49419).

  Check for malloc returning valid pointer (bug #49459).

  Segfault in mesh interpolation corrected (bug #49588).

  Corrected bad status usage in calls to wcsp2s and wcss2p (bug #49752).

  Strickter checking in Crop's polygon point list (bug #48978).

  Correction in alignment and getting pixel scale (bug #50072).

  Decomposing PCi_j and CDELTi matices in output WCS (bug #50073).

  Using image naxes[n] when checkcenter is larger (bug #50099).

  Memory leak in MakeCatalog corrected (bug #51118).

  Fix copy-paste error in MakeCatalog flag arrays (bug #51130).





* Noteworthy changes in release 0.2 (library 0.0.0) (2016-10-03) [stable]

** Bug fixes

  Linker errors on some operating systems have been fixed (bug #48076).

  Several memory allocation, checks or redundancies have been fixed: bugs
  #48453, #48516, #48603, #48611, #48571, #48650, #48657, #48692, #48770,
  #47866, #48899, #49049, #49007.

  ImageCrop no longer crashes with very long output file names (bugs #46241
  and #45380).

** New programs or headers

  Table: a new utility to read and write FITS binary and ASCII tables. It
  can also print the column information or select columns using regular
  expressions (task #13579).

  Shared libraries and headers are now installed. The libraries can be used
  in C and C++ programs. This release includes the following headers:
  `gnuastro.h', `array.h', `box.h', `fits.h', `linkedlist.h', `mesh.h',
  `polygon.h', `qsort.h', `spatialconvolve.h', `statistics.h', `threads.h',
  `wcs.h', `txtarray.h' (task #13765).

  Gnuastro now comes with a script in its top source directory
  (`tmpfs-config-make') to configure and build it in the tmpfs (on the
  RAM), for those systems that have it. See the new "Configure and build in
  RAM" section in the book for more (task #14100).

** New features

  MakeProfiles also accepts WCS positions (task #13566).

  Flat profiles in MakeProfiles can be given a profile specific value. The
  new `--mforflatpix' option MakeProfile will use the value in the
  magnitude column as a fixed value for each pixel. This can be very useful
  in defining a mask, or creating segmentation maps or labeled images (task
  #14115).

  MakeProfiles can now use input image as canvas. Instead of specifying the
  WSC and image size paramteres manually. With the new `--inputascanvas'
  option, MakeProfiles will get this information (along with blank pixels)
  from an already existing image (task #14116).

  Type of output in MakeProfiles and Arithmetic can be specified with the
  `--type' option.

  Magnitude error column in MakeCatalog with the `--magnitudeerr' option.

  Arithmetic now has new conditional (task #13870) and logical operators
  (task #14153) along with an operator for actions only when conditions are
  true: `where'. The new `isblank' operator will also enable you to select
  blank, or masked, pixels (task #14146).

  The `--noerodequant' in NoiseChisel enables it to detect small and
  sharper profiles by disabling erosion on pixels above a certain quantile
  (task #14139).

  MakeCatalog can also calculate the upper limit magnitude for each given
  object in the image by randomly positioning the object's footprint over
  undetected regions of the image (task #14167).

  The source tarball is now also distributed with Lzip for a much better
  compression ratio and more robust archival file format.

** Changes in behavior

  The two MakeProfiles options `--setconsttonan', `--setconsttomin' have
  been removed (see `--mforflatpix' above for their alternative).

  MakeCatalog makes clump catalog only when asked (when the `WCLUMPS'
  header exists in the objects HDU). This can be very useful in cases like
  aperture photometry, when the object labels are not generated by
  NoiseChisel and so a clump image doesn't exist (task #14122).

  Default cosmological parameters in CosmiCalculator set to Plank 2015
  results: A&A (2016), 594, A13 (arXiv 1502.01589).

  The `--envseed' option (to read random number generator type and seed
  from the environment) to MakeProfiles and MakeNoise can also be given in
  the configuration files.





* Noteworthy changes in release 0.1 (2016-05-30) [stable]

** Bug fixes

  MakeCatalog's problem in checking the sizes of all input images is now
  fixed.

  NoiseChisel's problem with reading the `--kernel' option is now
  corrected (bug #46750).

  lib/mesh.c's problem in correctly calculating the mesh sizes was
  corrected (bug #47611).

  `make check' will not look into system utility configuration files. In
  the previous release, if Gnuastro was already installed, the
  configuration files already present on the system would also be read. Now
  only configuration files in the tested package are used (bug #47833).

  Ghostscript's version is now checked at configure time after its
  existence. ConvertType uses the `-dPDFFitPage' option to Ghostscript
  which was introduced in version 9.10, so older versions would pass
  configure but at `make check' time, the PDF test would fail. Now this
  test is skipped (bug #47868).

  Most tests would fail when `make check -jN' was run (to do the checks on
  N threads). A dependency structure has now been defined to fix this
  problem and greatly speed up the testing process (bug #47957).

** New utilities

  Arithmetic: For arithmetic operations on the pixels of input images. With
  this utility, it is now possible to add multiple images with each other,
  or easily calculate a median image. It as a large set of other arithmetic
  operations and some functions which can be done on the input image(s). An
  unlimited number of input images can be given, the images will only be
  loaded when necessary and will be freed as soon as they are no longer
  necessary.

  CosmicCalculator: For doing cosmological calculations at a given
  redshift.

** New features

  All the utilities that would produce a log file now have a `--nolog'
  option to avoid printing a log file.

  The tiled image compression convention (.fits.fz, created with `fpack')
  files can now be used as input in the utilities.

  ImageCrop can now also crop a polygon from the input image. The polygon
  vertices can be given in the world or image coordinates. The simple
  `--polygon' option will keep the insides of the polygon while the
  `--outpolygon' will keep the outside of the polygon.

  ImageCrop and ImageWarp can now read the WCS information of a FITS header
  from a specific region with the `--hstartwcs' and `--hendwcs'. In some
  older FITS images, when the WCS distortions were not as standardized as
  now, there were cases which would confuse WCSLIB.

  NoiseChisel can now save the grown clumps image instead of the original
  clumps image in the output with the `--grownclumps' option.

  Convolve can now do deconvolution with the `--makekernel' option.

  MakeProfiles now has a `--setconsttonan' option which will fill the
  contant profiles with a NaN (blank) value, not a number, allowing the
  creations of elliptical masked regions for example.

  Header can now import a keyword directly from a string with the `--asis'
  option.

  MakeCatalog can now output the geometric (average position independent of
  pixel flux value) positions of the objects too.

  MakeCatalog can now produce the object's elliptical parameters (for
  example semi-major axis, semi-minor axis, and position angle). This can
  also be done both in standard flux weighted and geometric methods too.

  MakeCatalog now has a `--threshold' function to only use pixels above a
  given threshold in each object or clump. This is useful to avoid diffuse
  regions in calculations.

  MakeCatalog now has a `--noriverbrightness' option. With this option it
  is possible to calculate the clump flux without subtracting the river
  pixels on its circumference.

  The number of CPU threads is no longer a configuration option, it is now
  determined at runtime for each program. Therefore it is now easily
  possible to built Gnuastro on one system to use on another (commonly done
  in the GNU/Linux package managers). Therefore ./configure no longer has a
  `--with-numthreads' option.

  Every commit in Gnuastro's history (after implementing this feature) can
  now be given a unique version number. Since the version number is printed
  in possible outputs, this feature can help reproducibility, even when the
  an official/stable release isn't used.

  The AUTHORS file is now automatically generated from the version
  controlled history. Also all the authors that have contributed to
  Gnuastro are included in the second (copyright) page of the PDF book.

  All the bootstrapped directories are now moved with a new `bootstrapped'
  directory in the top source directory. This significantly cleans up this
  directory, allowing users to more easily find the hand-written Gnuastro
  source files they like.

  A `bug-gnuastro' Info page was created so users can easily go to that
  page for information on how to submit bug reports. It is accessible on
  the command line with the command `info bug-gnuastro'.

** Changes in behavior

  The separate utilities no longer have a separate version number. With the
  introduction of unofficial version numbers generated from each commit in
  Gnuastro's history, the utility version numbers would cause confusion.

  MakeProfiles will now add a suffix to the individual images and put them
  in the output directory if specified.

  NoiseChisel and MakeCatalog now use the median mesh standard deviation to
  define the over-all depth of the image. Previously they used the maximum
  value (or the least depth).

  NoiseChisel no longer outputs a sky subtracted image. This job can now be
  done with the new Arithmetic utility.

  NoiseChisel's `--segsnhistnbins' option was renamed to
  `--clumpsnhistnbins'.

** Improvements

  NoiseChisel's default quantiles were changed to 0.95 from 0.99. Since the
  old value was too severe.

  NoiseChisel's S/N thresholds are now found from the distribution of
  pseudo-detections and clumps from the full image, not within each large
  mesh. This was done to increase the accuracy of the S/N threshold. There
  were commonly not enough points in large mesh sizes and this would add
  scatter. With ImageCrop's polygon capabilities, it is now easily possible
  to cut out the region that has uniform noise properties (depth and
  correlated noise). Therefore the old `--checkdetectionsn' and
  `checkclumpsn' options are no longer present.

  When building from the version controlled source, the whole bootstrapping
  process is done with one script. In the previous version, all the
  separate operations should have been done by hand (as instructed in the
  old manual).

  All the build steps now report what was done and suggest the next
  step. This feature can be disabled with the `--disable-guide-message' at
  configure time.
