.. _whatsnew-7.0:

**************************
What's New in Astropy 7.0?
**************************

Overview
========

Astropy 7.0 is a major release that adds significant new functionality since
the 6.1 release.

In particular, this release includes:

* :ref:`whatsnew-7.0-table-masked-quantity`
* :ref:`whatsnew-7.0-table-frames`
* :ref:`whatsnew-7.0-table-notebook-backends`
* :ref:`whatsnew-7.0-table-init-columns-order`
* :ref:`whatsnew_7_0_table_pformat_default_changes`
* :ref:`whatsnew_7_0_quantity_to_string_formatter`
* :ref:`whatsnew_7_0_numpy_constructors_like_quantity`
* :ref:`whatsnew_7_0_ecsv_meta_default_dict`
* :ref:`whatsnew_7_0_contributor_doc_improvement`
* :ref:`whatsnew_7_0_typing_stats`
* :ref:`whatsnew_7_0_unit_conversion_array_like`
* :ref:`whatsnew_7_0_modeling_speedup`
* :ref:`whatsnew_7_0_parallel_fitting`
* :ref:`whatsnew_7_0_rgb_image_visualization_enhancement`
* :ref:`whatsnew_7_0_lorentz2d_model`
* :ref:`whatsnew_7_0_faster_io_ascii_guessing`
* :ref:`whatsnew_7_0_votable_1_5`
* :ref:`whatsnew_7_0_simplenorm`
* :ref:`whatsnew_7_0_sigmaclippedstats`
* :ref:`whatsnew_7_0_wcsaxes_default_axes`
* :ref:`whatsnew-7.0-masked-coordinates`

In addition to these major changes, Astropy v7.0 includes a large number of
smaller improvements and bug fixes, which are described in the :ref:`changelog`.
By the numbers:

* 1440 commits have been added since 6.1
* 213 issues have been closed since 6.1
* 466 pull requests have been merged since 6.1
* 52 people have contributed since 6.1
* 22 of which are new contributors

.. _whatsnew-7.0-table-masked-quantity:

Full ``MaskedQuantity`` Support in ``QTable``
=============================================

Masked quantities were already used in many table functions, like reading from
files, and are now fully supported throughout, i.e., ``MaskedQuantity`` are
now always used in ``QTable`` to represent masked quantities (or when the
``QTable`` is created with ``masked=True``). This removes the last vestiges of
a work-around where a normal ``Quantity`` was used with a stub of a mask, and
fixes functions like reading of table data from a list of dict that includes
quantities with missing entries, and aggregation of ``MaskedQuantity`` in
table groups.

.. _whatsnew-7.0-table-frames:

Coordinate frames can now be stored in tables
=============================================

Coordinate frames like ``ICRS`` and ``AltAz`` can now be stored in tables, as
was already the case for ``SkyCoord`` and the underlying representations such
as ``SphericalRepresentation``.

This includes all frames, also those that do not have associated data, such as
a ``SkyOffsetFrame`` in which the RA, Dec of the origin might represent a
pointing directions for a tiled observation, and the position angle the roll of
a spacecraft.

.. _whatsnew-7.0-table-notebook-backends:

Table ``show_in_notebook`` is back with ``ipydatagrid``
=======================================================

Due to popular demand, the :meth:`~astropy.table.Table.show_in_notebook`
method is revived (instead of deprecated) with a new backend that would
require an optional dependency, ``ipydatagrid``. As a result, a new module
called `astropy.table.notebook_backends` is added to support different
backends for rendering Astropy tables in Jupyter notebooks.

.. image:: https://raw.githubusercontent.com/jupyter-widgets/ipydatagrid/main/static/ipydatagrid_1.gif
   :width: 450px
   :alt: Animated DataGrid usage example from ipydatagrid

.. _whatsnew-7.0-table-init-columns-order:

Ordering of table columns constructed from rows
===============================================

The column order in a `~astropy.table.Table` constructed from a `list` or rows (`dict`
or `~astropy.table.Row`) may change using ``astropy`` version ``7.0`` *if the
first row has missing values*.

Before ``7.0``, the column ordering was determined from the first row if it contained
values for all the columns, or by sorting the final column names alphabetically if it
did not. Starting with ``7.0``, columns are always added in the order they appear
when iterating over  the `list` of rows.

For example, create a table as shown below:

    >>> from astropy.table import Table
    >>> data = [{'b': 10, 'c': 7, },
    ...         {'a': 15, 'c': 35, 'b': 20}]
    >>> t = Table(data)  # or Table(rows=data), which is equivalent


Before ``7.0`` the table would look like this::

     a   b   c
    --- --- ---
     --  10   7
     15  20  35


Starting with ``7.0`` the table would instead look like this::

     b   c   a
    --- --- ---
     10   7  --
     20  35  15

.. _whatsnew_7_0_table_pformat_default_changes:

``Table.pformat`` is now independent of terminal dimensions
===========================================================

``Table.pformat`` and ``Column.pformat`` do not truncate their outputs according
to terminal height and width by default any more. The new default behavior is
intended to be less surprising.

Truncating representations to fit the current terminal is still supported but
now requires explicitly passing ``max_lines=None`` and/or ``max_width=None``.

``Table.pformat_all`` is deprecated as it is now fully redundant.


.. _whatsnew_7_0_quantity_to_string_formatter:

``Quantity.to_string`` supports ``formatter`` for formatting
==============================================================

The :meth:`~astropy.units.Quantity.to_string` method now supports a ``formatter`` parameter.
This feature supports both Python's format specification mini-language via format strings and
custom formatting through callables. This enables users to have explicit and consistent control
over the numerical representation of quantities, accommodating a wide range of formatting needs.

Previously, the method primarily relied on the ``precision`` parameter for format control, which dictated
the number of significant digits and did not provide much freedom in the latex format.

Example:

.. code-block:: python

    >>> from astropy import units as u
    >>> q = u.Quantity(123.456, u.m)
    >>> custom_format = lambda x: f"\\approx {float(x):.1f}"
    >>> q.to_string(formatter=custom_format, format='latex')
    '$\\approx 123.5 \\; \\mathrm{m}$'
    >>> q.to_string(formatter='.3e', format='latex')
    '$1.235 \\times 10^{2} \\; \\mathrm{m}$'
    >>> q.to_string(precision=3, format='latex')
    '$123 \\; \\mathrm{m}$'

.. _whatsnew_7_0_numpy_constructors_like_quantity:

NumPy constructor functions with a ``like`` argument are now supported with ``Quantity``
========================================================================================

We added support for constructing ``Quantity`` arrays from 21 NumPy functions
via the ``like`` keyword argument.

Example:

.. code-block:: python

    >>> import numpy as np
    >>> from astropy import units as u
    >>> q = u.Quantity(1.0, u.m)
    >>> np.arange(0, 10, 1, like=q)
    <Quantity [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] m>
    >>> np.eye(3, like=q)
    <Quantity [[1., 0., 0.],
               [0., 1., 0.],
               [0., 0., 1.]] m>
    >>> np.full((3, 3), 1*u.s, like=q)
    <Quantity [[1., 1., 1.],
               [1., 1., 1.],
               [1., 1., 1.]] s>


The unit of the output ``Quantity`` is defined from the first quantity argument
where it is meaningful. Otherwise, and by default, the output unit will be that
of the ``like`` argument itself.

Here's the entire list of functions affected

* `~numpy.arange`
* `~numpy.empty`
* `~numpy.ones`
* `~numpy.zeros`
* `~numpy.full`
* `~numpy.array`
* `~numpy.asarray`
* `~numpy.asanyarray`
* `~numpy.ascontiguousarray`
* `~numpy.asfortranarray`
* `~numpy.require`
* `~numpy.fromfunction`
* `~numpy.fromstring`
* `~numpy.fromiter`
* `~numpy.fromfile`
* `~numpy.frombuffer`
* `~numpy.identity`
* `~numpy.loadtxt`
* `~numpy.genfromtxt`
* `~numpy.eye`
* `~numpy.tri`

.. _whatsnew_7_0_ecsv_meta_default_dict:

Change default type for ``meta`` attribute to ``dict`` and update ECSV writer
=============================================================================

We have changed the default type for the ``meta`` attribute in ``Table`` and ``Column``
(and subclasses) from ``OrderedDict`` to ``dict``. Since Python 3.8 the ``dict`` class
is ordered by default, so there is no need to use ``OrderedDict``.

In addition, the ECSV table writer in :ref:`io-ascii` was updated to consistently
write the ``meta`` attribute as an ordered map using the  ``!!omap`` tag. This
convention conforms to the `ECSV specification
<https://github.com/astropy/astropy-APEs/blob/main/APE6.rst>`_ and is supported by
existing ECSV readers. Previously the ``meta`` attribute could be written as an ordinary
YAML map, which is not guaranteed to preserve the order of the keys.

.. _whatsnew_7_0_contributor_doc_improvement:

Improve the Contributor Documentation
=====================================

The `Contributor documentation <https://docs.astropy.org/en/latest/index_dev.html>`_ has
been significantly improved. It now includes a `Quickstart Guide
<https://docs.astropy.org/en/latest/development/quickstart.html>`_ with concise
instructions on setting up a development environment and making a pull request. In
addition, the developer documentation was reorganized and simplified where possible to
improve readability and accessibility. We welcome continued feedback on how to make
contributing to Astropy even easier and more enjoyable.

.. _whatsnew_7_0_typing_stats:

Typing in astropy.stats
=======================

The ``astropy.stats`` module is now fully typed. This is the first subpackage for
which this the case.

.. _whatsnew_7_0_unit_conversion_array_like:

Converting units on dask and other array-like objects
=====================================================

It is now possible to pass in array-like objects such as dask arrays as the
value in ``Unit.to`` and have those arrays not be converted to Numpy arrays:

.. doctest-requires:: dask

    >>> from dask import array as da
    >>> from astropy import units as u
    >>> arr = da.arange(10)
    >>> u.m.to(u.km, value=arr)
    dask.array<mul, shape=(10,), dtype=float64, chunksize=(10,), chunktype=numpy.ndarray>

Note that it is not yet possible to use ``Quantity`` with dask arrays directly.

.. _whatsnew_7_0_modeling_speedup:

Performance improvements in astropy.modeling
============================================

There have been significant improvements to the performance of non-linear fitters in
:mod:`astropy.modeling`, with typical speedups of 2x for simple models and 4x or
more for compound models.

.. _whatsnew_7_0_parallel_fitting:

Fitting models in parallel with N-dimensional data
==================================================

A new function, :func:`~astropy.modeling.fitting.parallel_fit_dask`, has been
added to the :mod:`astropy.modeling` module. This function makes it easy to fit
many parts of an N-dimensional array in parallel, such as fitting all the
spectra in a spectral cube. This makes use of the `dask
<https://www.dask.org/>`_ package to efficiently parallelize the problem,
running it either on multiple processes of a single machine or in a distributed
environment. A simple example might be:

.. doctest-skip::

    >>> from astropy.modeling.models import Gaussian1D
    >>> from astropy.modeling.fitting import parallel_fit_dask, TRFLSQFitter
    >>> model_fit = parallel_fit_dask(model=Gaussian1D(),
    ...                               fitter=TRFLSQFitter(),
    ...                               data=data,
    ...                               world=wcs,
    ...                               fitting_axes=0)

where ``data`` is a 3-D array, and ``wcs`` is the :class:`~astropy.wcs.WCS`
object associated with the data. A full example can be found at
:ref:`parallel-fitting`.

.. _whatsnew_7_0_rgb_image_visualization_enhancement:

RGB image visualization enhancements
====================================


The `RGB image visualization functionality <https://docs.astropy.org/en/latest/visualization/rgb.html>`_
in ``astropy.visualization`` has been expanded to support more flexible methods
for creating composite RGB images.

A new function :func:`~astropy.visualization.make_rgb` allows for creating RGB
images with independent scaling on each filter, using arbitrary stretch and
interval functions (instances of subclasses of
:class:`~astropy.visualization.BaseStretch` and
:class:`~astropy.visualization.BaseInterval`, respectively).

Additionally, the :func:`~astropy.visualization.make_lupton_rgb` function
(which performs interconnected R, G, B image scaling) now also supports
arbitrary stretch and interval functions, in addition to the default Lupton
asihn stretch.


.. plot::
   :context: reset
   :nofigs:

    import numpy as np
    import matplotlib.pyplot as plt
    from astropy.visualization import make_rgb, make_lupton_rgb, ManualInterval, LogStretch
    from astropy.io import fits
    from astropy.utils.data import get_pkg_data_filename

    # Read in the three images downloaded from here:
    g_name = get_pkg_data_filename('visualization/reprojected_sdss_g.fits.bz2')
    r_name = get_pkg_data_filename('visualization/reprojected_sdss_r.fits.bz2')
    i_name = get_pkg_data_filename('visualization/reprojected_sdss_i.fits.bz2')
    g = fits.getdata(g_name)
    r = fits.getdata(r_name)
    i = fits.getdata(i_name)

    fig, axes = plt.subplots(1, 2)
    fig.set_size_inches(10,4)


.. plot::
   :context:
   :include-source:

    intervals = [ManualInterval(vmin=0, vmax=np.percentile(img,99.95)) for img in [i,r,g]]

    rgb_log = make_rgb(i, r, g, interval=intervals, stretch=LogStretch(a=1000))
    rgb_log_lupton = make_lupton_rgb(i, r, g, interval=intervals, stretch_object=LogStretch(a=5))

    axes[0].imshow(rgb_log, origin='lower')
    axes[1].imshow(rgb_log_lupton, origin='lower')

.. _whatsnew_7_0_lorentz2d_model:

New ``Lorentz2D`` model
=======================

A new 2D Lorentzian model has been added to the ``astropy.modeling``
package.

.. _whatsnew_7_0_faster_io_ascii_guessing:

Faster guessing of formats in ``astropy.io.ascii``
==================================================

The performance of guessing the table format when reading large files with
``astropy.io.ascii`` has been improved. Now the process uses at most
10000 lines of the file to check if it matches the format. This behavior can
be configured, e.g.:

.. code-block:: python

    >>> from astropy.io.ascii import conf
    >>> conf.guess_limit_lines = 100000

and the limit can be disabled by setting this value to `None`. When reading large tables
you should specify the format of the table explicitly if possible.

.. _whatsnew_7_0_votable_1_5:

Support VOTable version 1.5
===========================

The `Astropy VOTable parser <https://docs.astropy.org/en/stable/io/votable/index.html>`_
now supports version 1.5 of the VOTable standard.  The main new feature is that the
``COOSYS`` specification now has a ``refposition`` attribute analogous to that for ``TIMESYS``.

At this writing, version 1.5 is a proposed standard, but it is expected to be approved as an
official recommendation soon.

.. _whatsnew_7_0_simplenorm:

New ``SimpleNorm`` class
========================

A new convenience class, :class:`~astropy.visualization.SimpleNorm`,
has been added to the ``astropy.visualization`` module.
This class provides a simple interface to create
a :class:`~astropy.visualization.ImageNormalize`
normalization object that can be used with Matplotlib's
:meth:`~matplotlib.axes.Axes.imshow` method. It also provides
a :meth:`~astropy.visualization.SimpleNorm.imshow` method that
wraps Matplotlib's :meth:`matplotlib.axes.Axes.imshow` method and
automatically sets the normalization.

Here's an example using the :func:`~astropy.visualization.SimpleNorm`
function with its :meth:`~astropy.visualization.SimpleNorm.imshow`
method:

.. plot::
    :include-source:

    import numpy as np
    import matplotlib.pyplot as plt
    from astropy.visualization import SimpleNorm

    # Generate a test image
    image = np.arange(65536).reshape((256, 256))

    # Create an ImageNormalize object
    snorm = SimpleNorm('sqrt', percent=98)

    # Display the image
    fig, ax = plt.subplots()
    axim = snorm.imshow(image, ax=ax, origin='lower')
    fig.colorbar(axim)

.. _whatsnew_7_0_sigmaclippedstats:

New ``SigmaClippedStats`` class
===============================

A new convenience class, :class:`~astropy.stats.SigmaClippedStats`, has
been added to the :mod:`~astropy.stats` module. This class provides a
convenient way to compute statistics of an array with sigma-clipping. A
simple example might be:

.. doctest-skip::

    >>> import numpy as np
    >>> from astropy.stats import SigmaClippedStats
    >>> rng = np.random.default_rng(seed=42)
    >>> data = rng.exponential(scale=100, size=500)
    >>> stats = SigmaClippedStats(data, sigma=3, maxiters=10)
    >>> stats.min(), stats.max(), stats.sum()  # doctest: +FLOAT_CMP
    (np.float64(0.8129422833034009), np.float64(255.34193997940474), np.float64(36783.895498717866))
    >>> stats.mean(), stats.median(), stats.std()  # doctest: +FLOAT_CMP
    (np.float64(79.61882142579624), np.float64(60.01103363578014), np.float64(65.4457063794851))
    >>> stats.mode(), stats.var(), stats.mad_std()  # doctest: +FLOAT_CMP
    (np.float64(20.79545805574793), np.float64(4283.1404835097765), np.float64(62.608350894722484))
    >>> stats.biweight_location(), stats.biweight_scale()  # doctest: +FLOAT_CMP
    (np.float64(67.98055399699436), np.float64(64.82889460022386))

.. _whatsnew_7_0_wcsaxes_default_axes:

Automatic placement of axis and tick labels for ``WCSAxes``
===========================================================

``WCSAxes`` now automatically selects which coordinates are displayed on which axes
of a plot, a change from the previous behavior of using the ordering of the
world axes in the WCS object.

The selection is done by first placing the coordinates on the different axes in
the order in which they appear in the WCS and then trying different permutations
of coordinates on different axes to see if it is possible to increase the number
of tick labels. This addresses the situation where ticks are shown along an axis
where the world coordinate hardly varies when it varies substantially along the
other axis.

.. plot::

    import matplotlib.pyplot as plt

    from astropy.wcs import WCS

    wcs = WCS(naxis=2)
    wcs.wcs.ctype = 'RA---TAN', 'DEC--TAN'
    wcs.wcs.crval = 10, 20
    wcs.wcs.crpix = 30, 30
    wcs.wcs.cdelt = 0.1, 0.1
    wcs.wcs.cunit = 'deg', 'deg'
    wcs.wcs.crota = 0, 89
    wcs.wcs.set()

    fig = plt.figure(figsize=(10, 5))
    ax0 = fig.add_subplot(1, 2, 1, projection=wcs)
    ax0.set_title("Previous default")
    ax0.coords[0].set_axislabel_position("b")
    ax0.coords[0].set_ticklabel_position("b")
    ax0.coords[1].set_axislabel_position("l")
    ax0.coords[1].set_ticklabel_position("l")
    ax0.coords[0].tick_params(color="red", labelcolor="red")
    ax0.coords[0].set_axislabel(ax0.coords[0].get_axislabel(), color="red")
    ax0.coords[1].tick_params(color="blue", labelcolor="blue")
    ax0.coords[1].set_axislabel(ax0.coords[1].get_axislabel(), color="blue")
    ax0.coords.grid()

    ax1 = fig.add_subplot(1, 2, 2, projection=wcs)
    ax1.set_title("New default")
    ax1.coords.grid()
    ax1.coords[0].tick_params(color="red", labelcolor="red")
    ax1.coords[0].set_axislabel(ax1.coords[0].get_axislabel(), color="red")
    ax1.coords[1].tick_params(color="blue", labelcolor="blue")
    ax1.coords[1].set_axislabel(ax1.coords[1].get_axislabel(), color="blue")

    fig.tight_layout()

This default can be overridden by manually setting the axes to use with
`~.CoordinateHelper.set_ticks_position`,
`~.CoordinateHelper.set_ticklabel_position`, and
`~.CoordinateHelper.set_axislabel_position`. In practice, you should be able
to just call `~.CoordinateHelper.set_ticklabel_position` because axis labels
and ticks will default to being shown on the same axes as tick labels.

.. _whatsnew-7.0-masked-coordinates:

Support for masks in ``coordinates``
====================================

Initial support for masks has been implemented for |SkyCoord|, as well as for
coordinate frames and coordinate representations.

Like more generally for |Masked| instances, one can access the mask using
`~astropy.coordinates.SkyCoord.mask` and check whether the underlying data are
masked using `~astropy.coordinates.SkyCoord.masked`. Like for all masked
classes, one can access underlying unmasked data using
`~astropy.utils.masked.MaskableShapedLikeNDArray.unmasked` and one can get a
new instance with masked data replaced with a fill value using
:meth:`~astropy.utils.masked.MaskableShapedLikeNDArray.filled`.

By default, the mask combines all masks of the underlying representation
components, including those on possible differentials like radial velocity and
proper motion. It is possible to make other choices using the new
`~astropy.coordinates.SkyCoord.get_mask()` method. For instance,
``sc.get_mask("ra", "dec")`` would get the combined masks of just the angular
components, which may be useful if dealing with, e.g., a catalogue of stars
where distances are available only for some of the objects.

For concrete examples, see :ref:`astropy-coordinates-masks`.

Full change log
===============

To see a detailed list of all changes in version v7.0, including changes in
API, please see the :ref:`changelog`.

Contributors to the 7.0 release
===============================

The people who have contributed to the code for this release are:

.. hlist::
  :columns: 4

  -  Albert Y. Shih
  -  Albert Zhang  *
  -  Alfio Puglisi  *
  -  Andreas Faisst
  -  Brandie-M  *
  -  Brett Morris
  -  Brigitta Sipőcz
  -  Clément Robert
  -  Deen-Dot  *
  -  Derek Homeier
  -  Eero Vaher
  -  Hans Moritz Günther
  -  Henrike F  *
  -  Hugo Buddelmeijer
  -  Jeff Jennings  *
  -  Jost Migenda  *
  -  Larry Bradley
  -  Leo Singer
  -  Marten van Kerkwijk
  -  Maximilian Linhoff
  -  Maximillian Weber  *
  -  Melissa Weber Mendonça  *
  -  Michael Kelley  *
  -  Mihai Cara
  -  Mridul Seth
  -  Nabil Freij
  -  Nathaniel Starkman
  -  P. L. Lim
  -  Parkerwise  *
  -  Peter Teuben
  -  Sam Holt
  -  Sam Van Kooten
  -  Sedona Price  *
  -  Shane Maloney  *
  -  Simon Alinder  *
  -  Simon Conseil
  -  SteinMatthiasPTB  *
  -  Stuart Mumford
  -  Tanvi Pooranmal Meena
  -  Thomas Robitaille
  -  Thomas Vandal  *
  -  Tom Aldcroft
  -  Tom Donaldson
  -  Zach Burnett  *
  -  Zhen-Kai Gao  *
  -  rachel guo  *
  -  sharath  *

Where a * indicates that this release contains their first contribution to astropy.
