@c -*- coding: utf-8; mode: texinfo; -*-

@node Working with source code
@chapter Working with source code

The LilyPond project uses @uref{https://git-scm.com/, Git} as a version
control system.  This section is intended at getting new contributors
started with Git, and helping senior developers with less frequently
used procedures.

@menu
* Setting up::
* Git cheat sheet::
* Lifecycle of a merge request::
* Writing good commit messages::
* Commit access::
* Further Git documentation resources::
* Repository directory structure::
* Other repositories::
@end menu



@node Setting up
@section Setting up

@menu
* Installing Git::
* Cloning and forking the repository::
* Configuring Git::
@end menu

@node Installing Git
@subsection Installing Git

On UNIX systems (such as GNU/Linux, Mac OS X, FreeBSD), the easiest way
to download and install Git is through a package manager.  Alternatively,
you can visit the @uref{http://git-scm.com/, Git website} for downloadable
installers.

For convenience, you may also install a graphical front-end to Git.
Packaged in the installers come @code{gitk} (for browsing the history)
and @code{git-gui} (for committing).  Git's official website provides a
@uref{https://git-scm.com/downloads/guis/, list of GUI clients},
including free software for various platforms.


@node Cloning and forking the repository
@subsection Cloning and forking the repository

Clone the @uref{https://gitlab.com/lilypond/lilypond, LilyPond repository}
to get the source code and its history:

@example
git clone git@@gitlab.com:lilypond/lilypond.git
@end example

New contributors need to fork it in order to push branches.  Using a fork
is no longer necessary (but may be convenient) when you are given
developer access.  You need to create an account on
@uref{https://gitlab.com/, GitLab}.  Visit
@uref{https://gitlab.com/lilypond/lilypond} and press @qq{Fork} on
the top right.  After the fork is created, set up a new remote:

@example
cd lilypond
git remote add fork git@@gitlab.com:@var{your-username}/lilypond.git
@end example

To list remote repositories that are configured, along with their URLs:

@example
git remote -v
@end example

You should have @code{origin} pointing to the official LilyPond repository,
and @code{fork} pointing to your your private fork.


@node Configuring Git
@subsection Configuring Git

Settings apply to any repository on your computer, unless you leave out
the @code{--global} option.

You first need configure some basic settings required for keeping track of
commit authors:

@example
git config --global user.name "@var{John Smith}"
git config --global user.email @var{john@@example.com}
@end example

It is also recommended to enable colored output:

@example
git config --global color.ui auto
@end example

If none of your editor-related environment variables are set, the default
text editor used for writing commit messages, etc., is usually @command{vim}.
If your are not familiar with it, change it to an editor that you are
comfortable with; for example, Gedit:

@example
git config --global core.editor gedit
@end example

Finally, and in some ways most importantly, let's make sure that
we can easily see the state of our working copy.  If you are not using
LilyDev or Windows, add the following lines to your @file{~/.bashrc}:

@example
export PS1="\u@@\h \w\$(__git_ps1)$ "
export GIT_PS1_SHOWDIRTYSTATE=true
export GIT_PS1_SHOWUNTRACKEDFILES=true
export GIT_PS1_SHOWUPSTREAM=auto
@end example

After starting a new terminal, your prompt will show the current
branch (this term and others are explained below).  Additionally, some
symbols next to the branch name indicate certain states.  A star @qq{*}
means that there are unstaged changes.  With a plus @qq{+}, the changes
are staged.  If there are untracked files, a percent @qq{%} will appear.
Finally, we can also see the state of the local repository compared
to upstream: @qq{=} means up to date, @qq{<} is behind, @qq{>} is ahead,
@qq{<>} means they have diverged.

You may need to install the additional @code{bash-completion}
package.




@node Git cheat sheet
@section Git cheat sheet

The intent of this section is to get you working on LilyPond
quickly.  If you want to learn about Git, go read
@ref{Further Git documentation resources}.

Also, these instructions are designed to eliminate the most common
problems we have found in using Git.  If you already know Git and
have a different way of working, great!  Feel free to ignore these
advice.


@unnumberedsubsec Pulling recent changes

As LilyPond's source code is continously improved, it is wise to
integrate recent changes into your local copy whenever you start a
working session.  On the @code{master} branch (this term is explained
below), run:

@example
git pull
@end example


@unnumberedsubsec Viewing the history

Each change is contained in a @emph{commit} with an explanatory message.
To list commits starting from the latest:

@example
git log
@end example

Press @key{Enter} to see more or @key{Q} to exit.


@subheading Start work: make a new branch

The Git workflow is based on branches, which can be viewed as different
copies of the source code with concurrent changes that are eventually
merged.  You start a contribution by creating a branch, freezing the
initial state of the source code you will base your work onto.  Ultimately,
your branch will be @emph{merged} in @emph{master}.  This latter special
branch centralizes all features developed simultaneously and is the
source for unstable releases.

@warning{Remember, @strong{never} directly commit to @code{master}.}

Let's pretend you want to add a section to the Contributor's Guide
about using branches.  To create a new branch for this:

@example
git branch cg-add-branches
@end example


@unnumberedsubsec Switching branches

Switching branches is somehow like @qq{loading a file}, although in
this case it is really @qq{loading a directory and subdirectories
full of files}.  The command to use is @code{git checkout}.

@example
git checkout master
git checkout cg-add-branches
git checkout origin/release/unstable
@end example

Branches that begin with @code{origin/} are part of the remote
repository, rather than your local repository, so when you check them
out you get a  temporary local branch.  Therefore, do not commit to
these either.  Always work in a local branch.


@unnumberedsubsec Listing branches

To list local branches:

@example
git branch
@end example

If you want remote branches too:

@example
git branch -a
@end example

In the output, the current branch is prefixed with a star.


@unnumberedsubsec Staging and committing files

Now edit files.  To show a summary of your edits:

@example
git status
@end example

For every file that you modified or added, first preview your changes:

@example
git diff @var{file}
@end example

If everything looks right:

@example
git add @var{file}
@end example

Then commit your changes:

@example
git commit
@end example

A text editor window appears for you to write a commit message. See
@ref{Writing good commit messages}.


@unnumberedsubsec Amending and reverting changes

To add some more changes to the latest commit, stage them using
@code{git add}, then run:

@example
git commit --amend
@end example

This also works for rephrasing the commit message.

To revert changes to a file that has not been committed yet:

@example
git checkout @var{filename}
@end example

To get back to the last commit, @emph{discarding all changes}:

@example
git reset --hard HEAD
@end example


@unnumberedsubsec Uploading your branch for review

To upload the current branch on the remote repository:

@example
git push -u fork cg-add-branches
@end example

This sets the remote branch so subsequent pushes are simpler:

@example
git push
@end example

The next section covers how to create a merge request from your branch.


@unnumberedsubsec Deleting branches

After the merge request has passed testing and was merged to
@code{master}, or after a failed experiment, you can delete your
local branch.

@example
git checkout master
git branch -d cg-add-branches
@end example

As a safety measure, this will fail if the commits of
@code{cg-add-branches} are not present in @code{master}.  This can be
because you used GitLab to rebase your branch, which modifies the commit
data and changes the hash.  If you are sure that the branch is not needed
anymore, replace the @code{-d} on the final line with a @code{-D} instead.




@node Lifecycle of a merge request
@section Lifecycle of a merge request

@menu
* Uploading a patch for review::
* Automated and manual testing::
* Patch countdown::
* Merging to master::
* Abandoned patches::
@end menu


@node Uploading a patch for review
@subsection Uploading a patch for review

Any non-trivial change should be reviewed as a merge request:

@example
@uref{https://gitlab.com/lilypond/lilypond/-/merge_requests}
@end example

Ensure your branch differs from latest @code{master} by just the changes
to be uploaded.

Make sure that @code{make}, @code{make test} and @code{make doc} succeed.
Even if the individual commits contain incomplete features, they must
@strong{all} pass these tests.

Branches pushed on the main repository should start with @code{dev/}.

After pushing, create a merge request to start the review cycle.  There
are multiple options for this as
@uref{https://docs.gitlab.com/ee/user/project/merge_requests/,
outlined in GitLab's documentation}. This will also ask you for a message
that will accompany your patch.

If you are not a member of the team and create the merge request
from a fork, consider enabling the box to @qq{Allow commits from
members who can merge to the target branch}.  This makes it possible
for somebody with permissions to rebase your changes and merge them
for you.  Please refer to @ref{Merging to master} for more details.


@node Automated and manual testing
@subsection Automated and manual testing

When a merge request is opened, a bot automatically adds the
@code{Patch::new} label to it, and it enters the countdown cycle.  GitLab
triggers automated testing which ensures that the patch completes
@code{make}, @code{make test}, and @code{make doc}.  Afterwards, the Patch
Meister manually runs @code{make check} to verify regression tests.  If
both steps pass, the merge request is labelled @code{Patch::review}.  On
failure, the status is updated to @code{Patch::needs_work}.


@node Patch countdown
@subsection Patch countdown

The Patch Meister reviews the tracker periodically, to list patches
which have been on review for at least 24 hours.  For each patch, the
Handler reviews any discussion on the merge request, to determine
whether the patch can go forward.  If there is any indication that a
developer thinks the patch is not ready, the Handler marks it
@code{Patch::needs_work} and makes a comment regarding the reason,
referring to the comment if needed.

Patches with explicit approval, or at least no negative comment, are
updated to @code{Patch::countdown}.  The countdown is a 48-hour waiting
period in which any final reviews or complaints should be made.

The Patch Meister sends an email to the developer list.  The subject
line has a fixed formatting, to enable filtering by email clients, like
so:

@example
PATCHES: Countdown for February 30th
@end example

The text of the email sets the deadline for this countdown batch.  At
present, batches are done on Tuesday, Thursday and Sunday evenings.

At the next countdown, if no problems were found, the patch will be
set to @code{Patch::push}.  New contributors should ask for it to be
merged.  Developers merge their patches themselves, see @ref{Merging
to master} and @ref{Commit access}.

Alternately, your patch may be set to @code{Patch::needs_work},
indicating that you should fix something (or at least discuss why the
patch needs no modification).  It also happens that patches waiting
for minor fixes are put on countdown a second time.

Successive revisions are made in response to comments are uploaded by
pushing to the same branch.  GitLab automatically keeps track of all
pushed commits and allows to compare revisions with each other.

As in most organisations of unpaid volunteers, fixed procedures are
useful in as much as they get the job done.  In our community, there
is room for senior developers to bypass normal patch handling flows,
particularly now that the testing of patches is largely automated.
Similarly, the minimum age of 24 hours can reasonably be waived if
the patch is minor and from an experienced developer.


@node Merging to master
@subsection Merging to master

Before allowing a merge request to be merged, GitLab ensures the
following:

@enumerate
@item
The merge must be fast-forward.  In most cases, this can be
achieved by @q{rebasing} the branch with the most recent commits
from @code{master}.  This can be handled via GitLab, if no conflicts
arise.  Otherwise, or if preferred, the operation can
be performed locally.

@item
The (possibly rebased) changes must have passed automatic testing.
This ensures that the @code{master} branch is always clean and ready
for development and translation.

@end enumerate

After rebasing, GitLab will immediately start the automatic testing
pipeline.  At the moment, all steps may take up to one hour to
complete.  If you are confident about the rebased result of your
changes, you may click @qq{Merge when pipeline succeeds} to avoid
waiting for the tests.  On failure, the merge will be aborted and
no harm is done to the @code{master} branch.

Because GitLab enforces fast-forward merges, this means only one
set of changes can be rebased and merged at once.  A second merge
request would be rejected later on because it does not contain the
commit(s) merged first.  To avoid wasting testing resources, please
avoid this situation and check first if a pipeline with a scheduled
merge is already running.  View the
@uref{https://gitlab.com/lilypond/lilypond/-/merge_requests, list of
merge requests} and verify that no merge request with @code{Patch::push}
status has a blue @qq{timer} icon.


@subsubheading How to merge a branch without rebasing

It is generally recommended to rebase commits before merging to get
a linear history.  However, this is not always possible or wanted.
This particularly holds for the @code{translation} branch and
@code{release/unstable} which cannot be force-pushed.  For these
cases, use the following procedure:

@enumerate
@item
Merge the branch manually using the command line. The example
assumes no pending changes in the local @code{master} branch and
merges the @code{translation} branch:

@example
git checkout master
git pull
git merge translation
git push origin HEAD:translation
@end example

@item
Open a merge request at GitLab.  This will immediately trigger
automatic testing as described above.

@item
Accept the merge request once the testing finishes, or use the
button to @qq{Merge when pipeline succeeds}.

@end enumerate


@node Abandoned patches
@subsection Abandoned patches

Roughly at six month intervals, the Patch Meister can list the
patches which have been set to @code{Patch::needs_work} and send the
results to the developer list for review.  In most cases, these
patches should be marked @code{Patch::abandoned} but this should come
from the developer if possible.




@node Writing good commit messages
@section Writing good commit messages

Your commit message should begin with a one-line summary describing the
change (no more than 50 characters long), and if necessary a blank line
followed by more explanatory text (wrapped at 72 characters).  Here is
how a good commit message looks like:

@example
Doc: add Baerenreiter and Henle solo cello suites

Added comparison of solo cello suite engravings to new essay with
high-resolution images.  Fixed cropping on Finale example.

Closes #1234.
@end example

The @qq{Closes} part is specially recognized by GitLab.  See the
documentation for
@uref{https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically,
closing issues automatically}.

Commit messages often start with a short prefix describing the general
location of the changes.  Commit affecting the documentation in English
(or in several languages simultaneously) should be prefixed with
@qq{Doc:}.  When the commit affects only one of the translations, use
@qq{Doc-**:}, where @var{**} is the two-letter language code.  For the
website, this is @qq{Web:} or @qq{Web-**}.  Commits that change CSS files
should use @qq{Web:@tie{}CSS} or @qq{Doc:@tie{}CSS:}.  Finally, changes
to a single file are often prefixed with the name of the file involved.

The imperative form, e.g. @qq{Include this in that}, is strongly
preferred over the descriptive form @qq{That is now included in this}.

See also @uref{https://chris.beams.io/posts/git-commit/, this blog post}
for details on how to write good commit messages.




@node Commit access
@section Commit access

New contributors are not able to push branches directly to the main
repository -- only members of the LilyPond development team have
@emph{commit access}.  If you are a contributor and are interested in
joining the development team, contact the Project Manager through the
mailing list (@email{lilypond-devel@@gnu.org}).  Generally, only
contributors who have already provided a number of patches which have
been merged to the main repository will be considered for membership.

If you have been approved by the Project Manager, use the following
procedure to obtain commit access.

@itemize
@item
Navigate to @uref{https://gitlab.com/lilypond} and @q{Request access}
to the group.  Make sure that your account can be related to your
activity on the mailing list.  If in doubt, please post the user name
after requesting access.

Note that you will not have commit access until the Project Manager
activates your membership.  Once your membership is activated, LilyPond
should appear under the heading @qq{Groups} on your profile page.

@item
Generate and register an SSH key pair.  Excellent instructions are provided in
@uref{https://docs.gitlab.com/ee/ssh/, GitLab's documentation}.

@item
Configure Git to use the SSH protocol (instead of the GIT or HTTP
protocols).  From your local Git repository, enter:

@example
git remote set-url origin git@@gitlab.com:lilypond/lilypond.git
@end example

@item
After your membership has been activated and you've configured Git
to use SSH, test the connection with:

@example
git pull --verbose
@end example

SSH should issue the following warning:

@example
The authenticity of host 'gitlab.com' can't be established.
ECDSA key fingerprint is SHA256:HbW3g8zUjNSksFbqTiUWPWg2Bq1x8xdGUrliXFzSnUw.
Are you sure you want to continue connecting (yes/no/[fingerprint])?
@end example

Make sure the key fingerprint displayed matches the one above or one of the
@uref{https://docs.gitlab.com/ee/user/gitlab_com/index.html#ssh-host-keys-fingerprints,
others published by GitLab}.  If it doesn't, respond @qq{no} and check that
you configured Git properly in the previous step.  If it does match, respond
@qq{yes}.  SSH should then issue another warning:

@example
Warning: Permanently added 'gitlab.com' (ECDSA) to the list of known hosts.
@end example

The list of known hosts is stored in the file
@file{~/.ssh/known_hosts}.

At this point, you are prompted for your passphrase if you have
one, then Git will attempt a pull.

If @command{git pull --verbose} fails, you should see
error messages like these:

@example
Permission denied (publickey).
fatal: The remote end hung up unexpectedly
@end example

If you get the above error, you may have made a mistake when
registering your SSH key.  If the key is properly registered
and it still doesn't work after an hour, ask for help on the
mailing list.

If @command{git pull --verbose} succeeds, the output
will include a @q{From} line that shows @q{ssh} as the protocol:

@example
From git@@gitlab.com:lilypond/lilypond.git
@end example


@item
Test your commit access with a dry run:

@example
git push --dry-run --verbose
@end example

@item
Repeat the steps from generating an SSH key through to testing
your commit access, for each machine from which you will be
making commits, or you may simply copy the files from your
local @file{~/.ssh} folder to the same folder on the other
machine.

@end itemize




@node Further Git documentation resources
@section Further Git documentation resources

@itemize
@item
In-depth tutorials: @uref{http://git-scm.com/documentation}

@item
Book about Git: @uref{http://progit.org/,Pro Git}

@item
GitLab user documentation: @uref{https://docs.gitlab.com/ee/user/index.html}

@item
Github help: @uref{http://help.github.com/}
(very highly recommended by Graham)

@end itemize

Finally, please remember to read the man page of every Git command you
will find in this manual in case you want to discover alternate methods
or just understand how it works.  The man pages can also
@uref{http://www.kernel.org/pub/software/scm/git/docs/, be read online}.




@node Repository directory structure
@section Repository directory structure

@c TODO: integrate the roadmap better
@verbatiminclude ROADMAP




@node Other repositories
@section Other repositories

TODO: to be updated (infrastructure, static-files, lilypond-extra is
no more).

We have a few other code repositories.

@menu
* lilypond-extra::
* Grand Unified Builder (GUB)::
* LilyPad::
@end menu

@node lilypond-extra
@subheading lilypond-extra

There is a separate repository for general administrative scripts,
as well as pictures and media files for the website.  People
interested in working on the website should download this
repository, and set their @code{$LILYPOND_WEB_MEDIA_GIT}
environment variable to point to that repository.

@example
@uref{https://github.com/gperciva/lilypond-extra}
@end example

To configure an environment variable in Bash (the default for most
GNU/Linux distributions),

@example
export LILYPOND_WEB_MEDIA_GIT=$HOME/dir/of/lilypond-extra/
@end example

Be aware that @code{lilypond-extra} is the definitive source for some binary
files -- in particular PDF versions of papers concerning LilyPond.  To add
further PDFs of this sort, all that is necessary is to add the PDF to
@code{lilypond-extra} and then add a reference to it in the documentation.  The
file will then be copied to the website when @code{make website} is run.

However, pictures that are also used in the documentation build are mastered in
the main Git repository.  If any of these is changed, it should be updated in
Git, and then the updates copied to @code{lilypond-extra}.


@node Grand Unified Builder (GUB)
@subheading Grand Unified Builder (GUB)

Another item of interest might be the Grand Unified Builder, our
cross-platform building tool.  Since it is used by other projects as
well, it is not stored in our main repository.  For more info, see
@uref{https://lilypond.org/gub, GUB's home page}.

The location for the repository is:

@example
@uref{http://github.com/gperciva/gub}
@end example


@node LilyPad
@subheading LilyPad

Our binary releases on MacOS X and Windows contain a lightweight
text editor.

To make any modifications the Windows editor, you will need to do the
following:

@enumerate
@item
Clone the git repository from @code{https://github.com/gperciva/lilypad}

@item
Make changes to the source, and check it compiles.  In a Windows environment
@code{MinGW} provides both a @code{Git} installation and a @code{gcc}
compiler.  This can be obtained from @code{http://www.mingw.org/}

@item
Update the version which is contained in the @file{rsrc.rc}.  Check
this compiles, too.

@item
Commit the changes with an informative commit message.

@item
Push the changes to github.  You will need to use syntax similiar to this:

@example
git push https://UserName@@github.com/gperciva/lilypad.git
@end example

You will need to have push access to the git repository for this to be
successful.

@item
Make a tarball of the source code to be used by GUB by pulling the updated
repository from GitHub.  Ensure that the tarball has the correct Version
number.

@item
Copy the tarball to @code{https://lilypond.org/downloads/gub-sources/lilypad/}.
You will need to have SSH access to @code{lilypond.org}.  If you do not,
contact the Release Manager via the lilypond-devel mailing list.

@item
Update GUB to make it use the new tarball by editing
@file{gub/specs/lilypad.py} and changing the @code{source =} line to
point to the new source.

@item
Push this updated @file{lilypad.py} version to the GUB repository on
GitHub.

@item
Test the changes with a new GUB compile.

@end enumerate
