# Translation synchronisation

## The need for a synchronization process

The files which are really used for building &d-i; packages are hosted in the git repository used for the development of &d-i; packages. Each &d-i; package sources is hosted in a subdirectory of the `packages/` directory, with a very classical Debian package source tree organization, including the `debian/po` directory where the package's translations are hosted.

These files need to be synchronised with the so-called master files from the packages/po directory, since translators only work on these master files. This synchronization must work both ways with changes on the English original strings going from individual packages to the master files and translation changes/updates from the master files going to individual packages:

- When &d-i; packages templates are changed, the individual `debian/po/templates.pot` file will change. These changes have to be moved to files in `packages/po`;

- When translations are updated in the `packages/po` directory, these changes have to be moved back into individual `debian/po/*.po` files for each package. This does not need to be done at each translation update, but must be done before building packages.

The task of keeping track of the correct synchronisation is one of the most important tasks of the &i18n-coords;. It is handled by the `scripts/l10n/l10n-sync` script.

## Localization files synchronisation process

### Synchronization process

The `l10n-sync` script is run periodically by a cron job on dillon.debian.org. Its runs are monitored by the &i18n-coords;. See the crontab file. It will update the general `template.pot` and all PO files from individual `templates.pot` files from all &d-i; packages.

In case of changings (commits) to the master templates files or master PO files (under packages/po) done by the l10n-sync script, the &i18n-coords; will need to perform a `git push` on dillon in the respective directory, to get the changings into GIT.

While running, this script will also run `debconf-updatepo` for each &d-i; package and will commit back the regenerated templates.pot files (package maintainers often forget running `debconf-udpatepo` when commiting changes to English templates).

During normal development, this synchronization process is run once a day. During release preparations, the frequency might be increased to speed up the full cycle.

The general process is:

1. Initial step: synchronize the whole repository;

2. Individual packages update step. For each &d-i; package:

   1. synchronize the local copy with the &d-i; GIT;

   2. run `debconf-updatepo`;

   3. commit back the files in `debian/po`.

3. Master templates file update step:

   1. merge all `templates.pot` files to `packages/po/template.pot`.

   2. commit this file.

4. Master PO files update step. For each PO file in `packages/po`:

   1. synchronize with &d-i; GIT;

   2. if the script is run with the `--merge` option, merge translations from the reference PO file (the reference file is given priority for identical strings);

   3. update with `packages/po/template.pot`. This step uses the `msgmerge`. The resulting PO file formatting details will depend on the version of this utility. For that reasons, the synchronization script should always be run from machines using the same Debian release version (these days it permanently runs on dillon anyway);

   4. commit back the changed file to &d-i; GIT.

5. Individual packages PO files update step. For each &d-i; package:

   1. synchronize the local copy with the &d-i; GIT (in case some update occurred in the meantime);

   2. update `debian/po/*.po` files with master files;

   3. commit back the changes to &d-i; GIT;

This system minimizes race conditions which could trigger conflicts.

The conflict windows have been minimised as much as possible in the `l10n-sync` script. However, in order to limit the number of commits made by the script, files are not commited as soon as they are modified (except by using the `--atomic-commits` switch which slows down the whole process a lot and triggers a lot of commit actions). So, the script is sometimes likely to trigger conflicts. For that reason, it will stop working on a GIT copy where GIT conflict files are present. This explains why this script must always be monitored even when it is scheduled to be executed periodically.

### Synchronization script use

The l10n synchronisation script is kept in &d-i; repository in the `scripts/l10n-utilities` directory and is named `l10n-sync`.

This is a shell script (which may contain some bashisms until a skilled shell programmer cleans it out).

The script uses some command-line switches which may affect its behaviour. Some of these switches are mostly present for historical reasons and are kept because they may have some new use in the future:

- `--debug` will trigger more output by the script. Otherwise, the script reports about its actions but commands output (such as `git`, `debconf-updatepo`, `msgmerge`) will be redirected to `/dev/null`;

- `--online` triggers `git pull` commands before working on files. Except for testing, there is no reason for not using this switch.

- `--commit` allows the script to commit files back to the &d-i; repository. Otherwise, the modified files are kept in the local GIT copy. Of course, this switch should be used in production. Omitting it is useful only for testing.

- the `--atomic` switch instructs `l10n-sync` to commit files as soon as they are modified. This makes the script quite slow and may trigger dozens of commits. As &d-i; commits are sent to the development IRC channel and are followed by several &d-i; maintainers, this switch should not be used except in very rare occasions;

- the `--atomic-updates` switch instructs l10n-sync to issue a `git pull` command before working on each package. This may be very CPU, network- and time-consuming and should only be used when there is some need to limit possible conflicts;

- the `--debconf-updatepo` defines the command for calling the `debconf-updatepo` utility. It allows using a specially crafted `debconf-updatepo` switch, often the `--skip-merge` switch which only updates `templates.pot` files and not PO files (using this speeds up the general synchronization script);

- the `--sort-order` switch allows giving at the command line the order the packages should be dealt with. This allows to put a crafted order in the generated `templates.pot` file so that the translators begin to work on the most important packages;

- the `--merge` switch allows merging master files from another branch. It is used, for instance, on the sarge branch for merging translations coming from trunk.

The script needs the location of the local copy of the &d-i; repository as an argument. It makes some simple checks about the copy.

When using this script with commits, the &d-i; copy must be as clean as possible. It should not be used for development tasks. The script checks for possible GIT conflict files and aborts if it finds some.

Prospective languages (see prospective) are handled in a special way: for all such languages, the translations are not copied in the individual packages directories.

### The PROSPECTIVE list

When languages are on early stages of translation after they have been added through the new language process (new-language), they are temporarily listed in a file named `packages/po/PROSPECTIVE`.

Languages listed in that file are excluded from synchronization, and, therefore, PO files are not created in individual packages. As a consequence, activating a language means that the language code is removed from the `PROSPECTUVE` file.

### List of handled packages

A special file, named `packages/po/packages_list`, lists the &d-i; packages which are included in the master files. This file also sorts these packages by order or priority for translations. The strings at the beginning of the files are to be translated first.

The &d-i; packages maintainers must request for their packages to be included in this file, when they estimate that their package is ready for translation. Only &i18n-coords; can integrate packages there, after checking that the strings have been reviewed.