Integration with GitHub

EasyBuild provides several features that integrate with GitHub, where the different EasyBuild repositories are located.

From the EasyBuild command line eb several options are available to reach out to GitHub, which are documented below.

Requirements

Depending on which GitHub integration features you want to use, there are a couple of requirements:

See also Checking status of GitHub integration (–check-github).

Configuration

The following sections discuss the EasyBuild configuration options relevant to the GitHub integration features.

Providing a GitHub username (--github-user)

To specify your GitHub username, do one of the following:

  • use the --github-user configuration option on the eb command line
  • define the $EASYBUILD_GITHUB_USER environment variable
  • specify github-user in your EasyBuild configuraton file

(see also Configuring EasyBuild)

Installing a GitHub token (--install-github-token)

Note

requires: GitHub username + keyring Python package

A GitHub token is a string of 40 hexidecimal (lowercase) characters that is tied to your GitHub account, allowing you to access the GitHub API authenticated.

Using a GitHub token is beneficial with respect to rate limitations, and enables write permissions on GitHub (e.g., posting comments, creating gists, opening pull requests, etc.).

To obtain a GitHub token:

  • visit https://github.com/settings/tokens/new and log in with your GitHub account
  • enter a token description, for example: “EasyBuild“
  • make sure (only) the gist and repo scopes are fully enabled
  • click Generate token
  • copy-paste the generated token

Note

You will only be able to copy-paste the generated token right after you have created it. The value corresponding to an existing token can not be retrieved later through the GitHub interface.

Please keep your token secret at all times; it allows fully authenticated access to your GitHub account!

You can install the GitHub token in your keyring using EasyBuild, so it can pick it up when it needs to, using eb --install-github-token:

$ eb --github-user example --install-github-token
Token: <copy-paste-your-40-character-token-here>
Validating token...
Token seems to be valid, installing it.
Token 'e3a..0c2' installed!

EasyBuild will validate the provided token, to check that authenticated access to your GitHub account works as expected.

Note

EasyBuild will never print the full token value, to avoid leaking it. For debugging purposes, only the first and last 3 characters will be shown.

Specify location of working directories (--git-working-dirs-path)

You can specify the location of your Git working directories using --git-working-dirs-path.

The provided path should be the parent directory of the location of the working directories (i.e. clones) of the EasyBuild repositories (easybuild-easyconfigs, etc.); the assumption is that you keep them all in a single parent directory.

Although not strictly required, this is useful for speeding up --new-pr and --update-pr, since it allows that the repository can be copied & updated, rather than being cloned from scratch.

Checking status of GitHub integration (--check-github)

To check the status of your setup w.r.t. GitHub integration, the --check-github command line option can be used.

Using this will trigger EasyBuild to perform a number of checks, and report back on what the test results mean for the different GitHub integration features.

If all requirements are taken care of in your setup, you should see output like this:

$ eb --check-github

== temporary log file in case of crash /tmp/eb-xWCpWl/easybuild-hGnKS5.log

Checking status of GitHub integration...

Making sure we're online... OK

* GitHub user... example => OK
* GitHub token... e3f..0c8 (len: 40) => OK (validated)
* git command... OK ("git version 2.7.4 (Apple Git-66); ")
* GitPython module... OK
* push access to example/easybuild-easyconfigs repo @ GitHub... OK
* creating gists... OK
* location to Git working dirs...  OK (/home/example/git-working-dirs)

All checks PASSed!

Status of GitHub integration:
* --from-pr: OK
* --new-pr: OK
* --review-pr: OK
* --update-pr: OK
* --upload-test-report: OK

Note

Checking whether push access to GitHub works may take some time, since a recent clone of the easybuild-easyconfigs GitHub repository will be created in the process (at a temporary location).

See also Requirements.

Using easyconfigs from pull requests (--from-pr)

(supported since EasyBuild v1.13.0)

Via the --from-pr command line option (available since EasyBuild v1.13.0), easyconfig files that are added or modified by a particular pull request to the easybuild-easyconfigs GitHub repository can be used (regardless of whether the pull request is merged or not).

This can be useful to employ easyconfig files that are not available yet in the active EasyBuild installation, or to test new contributions by combining --from-pr with -upload-test-report (see Uploading test reports (–upload-test-report)).

When --from-pr is used, EasyBuild will download all modified files (easyconfig files and patches) to a temporary directory before processing them.

For example, to use the GCC v4.9.2 easyconfigs contributed via easyconfigs pull request #1177:

$ eb --from-pr 1177 --dry-run
== temporary log file in case of crash /tmp/eb-88quZc/easybuild-62fFdo.log
Dry run: printing build status of easyconfigs and dependencies
 * [ ] /tmp/eb-88quZc/files_pr1177/GCC-4.9.2-CLooG-multilib.eb (module: GCC/4.9.2-CLooG-multilib)
 * [ ] /tmp/eb-88quZc/files_pr1177/GCC-4.9.2-CLooG.eb (module: GCC/4.9.2-CLooG)
 * [ ] /tmp/eb-88quZc/files_pr1177/GCC-4.9.2.eb (module: GCC/4.9.2)
== temporary log file /tmp/eb-88quZc/easybuild-62fFdo.log has been removed.
== temporary directory /tmp/eb-88quZc has been removed.

Note

To avoid GitHub rate limiting, let EasyBuild know which GitHub account should be used to query the GitHub API, and provide a matching GitHub token; see also Installing a GitHub token (–install-github-token).

Synergy with --robot

Since EasyBuild v1.15.0, the temporary directory containing the easyconfigs (and patch files) from the specified pull request is prepended to the robot search path, to ensure that easyconfigs that were modified in the respective pull request are picked up via --robot when they are required. Thus, for easyconfig files that are both available in the pull request and are available locally, the ones from the specified pull request will be preferred.

For example, to build and install HPL with the intel/2015a toolchain, both of which are contributed via easyconfigs pull request #1238:

$ eb --from-pr 1238 --dry-run --robot $HOME/easyconfigs
== temporary log file in case of crash /tmp/eb-A1fRvw/easybuild-Eqc8Oi.log
Dry run: printing build status of easyconfigs and dependencies
 * [x] /home/example/easyconfigs/g/GCC/GCC-4.9.2.eb (module: GCC/4.9.2)
 * [x] /home/example/easyconfigs/i/icc/icc-2015.1.133-GCC-4.9.2.eb (module: icc/2015.1.133-GCC-4.9.2)
 * [x] /home/example/easyconfigs/i/ifort/ifort-2015.1.133-GCC-4.9.2.eb (module: ifort/2015.1.133-GCC-4.9.2)
 * [x] /home/example/easyconfigs/i/iccifort/iccifort-2015.1.133-GCC-4.9.2.eb (module: iccifort/2015.1.133-GCC-4.9.2)
 * [x] /home/example/easyconfigs/i/impi/impi-5.0.2.044-iccifort-2015.1.133-GCC-4.9.2.eb (module: impi/5.0.2.044-iccifort-2015.1.133-GCC-4.9.2)
 * [x] /home/example/easyconfigs/i/iimpi/iimpi-7.2.3-GCC-4.9.2.eb (module: iimpi/7.2.3-GCC-4.9.2)
 * [x] /home/example/easyconfigs/i/imkl/imkl-11.2.1.133-iimpi-7.2.3-GCC-4.9.2.eb (module: imkl/11.2.1.133-iimpi-7.2.3-GCC-4.9.2)
 * [ ] /tmp/eb-A1fRvw/files_pr1238/intel-2015a.eb (module: intel/2015a)
 * [ ] /tmp/eb-A1fRvw/files_pr1238/HPL-2.1-intel-2015a.eb (module: HPL/2.1-intel-2015a)
== temporary log file /tmp/eb-A1fRvw/easybuild-Eqc8Oi.log has been removed.
== temporary directory /tmp/eb-A1fRvw has been removed.

Note that the easyconfigs that are required to resolve dependencies and are available locally in $HOME/easyconfigs are being picked up as needed.

Specifying particular easyconfig files

Since EasyBuid v2.0.0 the particular easyconfigs to be used can be specified, rather than using all easyconfigs that are touched by the pull request (which is the default if no easyconfigs are specified alongside --from-pr).

For example, to only use CMake-3.0.0-intel-2015a.eb from easyconfigs pull request #1330, and ignore the other easyconfigs being contributed in that same pull request for netCDF, WRF, ...:

$ eb --from-pr 1330 CMake-3.0.0-intel-2015a.eb --dry-run --robot $HOME/easyconfigs
== temporary log file in case of crash /tmp/eb-QhM_qc/easybuild-TPvMkJ.log
Dry run: printing build status of easyconfigs and dependencies
 * [x] /home/example/easyconfigs/g/GCC/GCC-4.9.2.eb (module: GCC/4.9.2)
 * [x] /home/example/easyconfigs/i/icc/icc-2015.1.133-GCC-4.9.2.eb (module: icc/2015.1.133-GCC-4.9.2)
 * [x] /home/example/easyconfigs/i/ifort/ifort-2015.1.133-GCC-4.9.2.eb (module: ifort/2015.1.133-GCC-4.9.2)
 * [x] /home/example/easyconfigs/i/iccifort/iccifort-2015.1.133-GCC-4.9.2.eb (module: iccifort/2015.1.133-GCC-4.9.2)
 * [x] /home/example/easyconfigs/i/impi/impi-5.0.2.044-iccifort-2015.1.133-GCC-4.9.2.eb (module: impi/5.0.2.044-iccifort-2015.1.133-GCC-4.9.2)
 * [x] /home/example/easyconfigs/i/iimpi/iimpi-7.2.3-GCC-4.9.2.eb (module: iimpi/7.2.3-GCC-4.9.2)
 * [x] /home/example/easyconfigs/i/imkl/imkl-11.2.1.133-iimpi-7.2.3-GCC-4.9.2.eb (module: imkl/11.2.1.133-iimpi-7.2.3-GCC-4.9.2)
 * [x] /home/example/easyconfigs/i/intel/intel-2015a.eb (module: intel/2015a)
 * [x] /home/example/easyconfigs/n/ncurses/ncurses-5.9-intel-2015a.eb (module: ncurses/5.9-intel-2015a)
 * [ ] /tmp/eb-QhM_qc/files_pr1330/CMake-3.0.0-intel-2015a.eb (module: CMake/3.0.0-intel-2015a)
== temporary log file /tmp/eb-QhM_qc/easybuild-TPvMkJ.log has been removed.
== temporary directory /tmp/eb-QhM_qc has been removed.

Again, note that locally available easyconfigs that are required to resolve dependencies are being picked up as needed.

Uploading test reports (--upload-test-report)

(supported since EasyBuild v1.13.0)

Note

requires that a GitHub token was required gist permissions is available, cfr. Installing a GitHub token (–install-github-token)

For every installation performed with EasyBuild, a test report is generated. By default, the test report is copied in the installation directory, right next to the log file (see also Understanding EasyBuild logs).

Using --upload-test-report, the test report can also be pushed to GitHub (as a gist, cfr. https://gist.github.com) to share it with others.

Each test report includes:

  • an overview of the easyconfigs being processed
  • time & date
  • the exact eb command line that was used
  • the full EasyBuild configuration that was in place
  • information about the system on which EasyBuild was used (hostname, OS, architecture, etc.)
  • the list of modules that was loaded
  • the full environment of the session in which eb was run (note: can be filtered, see Filtering the environment details (–test-report-env-filter))

For each easyconfig that failed to install a partial log will be uploaded as a separate gist, and a link to this gist will be included in the test report.

If --upload-test-report is combined with --from-pr, a comment referring to the test report (incl. a brief summary) will be placed in the respective pull request. This makes it a very powerful tool when testing contributions.

Note

If you want to easily access a test report without uploading it to GitHub, use --dump-test-report.

Example:

$ eb --from-pr 3153 --force --upload-test-report
== temporary log file in case of crash /tmp/eb-aqk20q/easybuild-wuyZBV.log
== processing EasyBuild easyconfig /tmp/eb-aqk20q/files_pr3153/EasyBuild/EasyBuild-2.8.1.eb
== building and installing EasyBuild/2.8.1...
...
== COMPLETED: Installation ended successfully
== Results of the build can be found in the log file /home/example/software/EasyBuild/2.8.1/easybuild/easybuild-EasyBuild-2.8.1-20160603.090702.log
== Test report uploaded to https://gist.github.com/1cb2db8a2913a1b8ddbf1c6fee3ff83c and mentioned in a comment in easyconfigs PR#3153
== Build succeeded for 1 out of 1
== Temporary log file(s) /tmp/eb-aqk20q/easybuild-wuyZBV.log* have been removed.
== Temporary directory /tmp/eb-aqk20q has been removed.

The resulting test report can be viewed at https://gist.github.com/1cb2db8a2913a1b8ddbf1c6fee3ff83c .

Note

It is common to use --force in combination with --upload-test-report, to ensure that all easyconfigs in the pull request are rebuilt, resulting in a complete test report.

Filtering the environment details (--test-report-env-filter)

Since the environment of the session in which eb was used may contain sensitive information, it can be filtered through --test-report-env-filter.

This configuration option takes a regular expression that is used to determine which environment variables can be included in the test report (based on their name). Environment variables for which the name matches the specified regular expression will not be included in the test report.

An example of a typical setting:

export EASYBUILD_TEST_REPORT_ENV_FILTER='^SSH|USER|HOSTNAME|UID|.*COOKIE.*'

Reviewing easyconfig pull requests (--review-pr)

A useful tool when reviewing pull requests for the easybuild-easyconfigs repository that add new or update existing easyconfig files is --review-pr.

The ‘files’ tab in the GitHub interface shows the changes being made to existing files; using --review-pr the differences with one or more other similar easyconfig files, for example the one(s) with the same toolchain (version) and/or software version, can also be evaluated.

This is very useful to quickly see how easyconfig files in pull requests differ from existing easyconfig files, and to maintain consistency across easyconfig files where desired.

The --review-pr output consists of a ‘multidiff’ view per easyconfig file that is being touched by the specified pull request. The exact format of the output depends on whether EasyBuild is configured to allow colored output (enabled by default, see --color).

Search criteria for similar easyconfigs

The set of existing similar easyconfig files is determined by specific search criteria; the first one that results in a non-empty set of easyconfigs is retained.

The search criteria consists of a combination of the software version criterion with additional restrictions.

The software version criterion is one of the criterions below (considered in order), with x.y.z the software version of the easyconfig file from the pull request:

  • exact same software version
  • same major/minor software version (same x and y)
  • same major software version (same x)
  • no (partial) version match (so consider any version)

The addition restrictions are the following (also considered in order):

  • matching versionsuffix and toolchain name/version
  • matching versionsuffix and toolchain name (any toolchain version)
  • matching versionsuffix (any toolchain name/version)
  • matching toolchain name/version (any versionsuffix)
  • matching toolchain name (any versionsuffix, toolchain version)
  • no extra requirements (any versionsuffix, toolchain name/version)

Submitting new and updating pull requests (--new-pr, --update-pr)

(supported since EasyBuild v2.6.0)

EasyBuild provides two simple yet powerful features that make contributing back to the central EasyBuild repositories significantly easier and less error-prone, especially for people who are not very familiar with git and/or GitHub yet:

  • --new-pr to create new pull requests
  • --update-pr to update existing pull requests

Note

Both --new-pr and --update-pr are still experimental, meaning that their behaviour may change in future releases and that they require having --experimental enabled; see also Experimental features.

Submitting pull requests (--new-pr)

To create a new pull request, the --new-pr command line option can be used, provided that the necessary requirements are fullfilled (see Requirements).

In its simplest form, you just provide the location of the file(s) that you want to include in the pull request:

$ eb --new-pr test.eb --experimental

This takes care of all the steps required to make a contribution, i.e.:

  • set up a working copy of the relevant EasyBuild repository (e.g., easybuild-easyconfigs)
  • create a new ‘feature’ branch, starting from the up-to-date develop branch
  • renaming easyconfig files according to their name, version, versionsuffix and toolchain
  • moving easyconfig files to the right location in the repository (e.g. easybuild/easyconfigs/e/EasyBuild/)
  • staging and committing the files in the feature branch
  • pushing the feature branch to your fork of the relevant EasyBuild repository on GitHub
  • creating the pull request, targetting the develop branch of the central EasyBuild repository (e.g. hpcugent/easybuild-easyconfigs)

It should be clear that automating this whole procedure with a single simple eb command greatly lowers the bar for contributing back, especially since it even alleviates the need for interacting directly with git entirely!

The working copy of the EasyBuild repository is created in a temporary location, and cleaned up once the pull request has been created. EasyBuild does not make changes to an existing working copy you may have in place already (cfr. Specify location of working directories (–git-working-dirs-path)).

Example

For example, to create a pull request for a new version of, let’s say, EasyBuild:

$ eb --new-pr ~/WIP/newEB.eb --experimental
== temporary log file in case of crash /tmp/eb-mWKR9u/easybuild-cTpf2W.log
== copying /home/example/git-working-dirs/easybuild-easyconfigs...
== fetching branch 'develop' from https://github.com/hpcugent/easybuild-easyconfigs.git...

Opening pull request
* target: hpcugent/easybuild-easyconfigs:develop
* from: boegel/easybuild-easyconfigs:20160530131447_new_pr_EasyBuild281
* title: "{tools}[dummy/dummy] EasyBuild v2.8.1"
* description:
"""
(created using `eb --new-pr`)

"""
* overview of changes:
 .../easyconfigs/e/EasyBuild/EasyBuild-2.8.1.eb     | 35 ++++++++++++++++++++++
 1 file changed, 35 insertions(+)

Opened pull request: https://github.com/hpcugent/easybuild-easyconfigs/pull/3153

Yes, it’s that easy!

Updating existing pull requests (--update-pr)

Similarly to creating new pull requests, existing pull requests can be easily updated using eb --update-pr (regardless of whether or not they were created with --new-pr).

The usage is equally simple, for example to update pull request #1234 just list the changed/new file(s):

$ eb --update-pr 1234 ~/WIP/changed.eb --experimental

Again, this take care of the whole procedure required to update an existing pull request:

  • set up a working copy of the relevant EasyBuild repository (e.g., easybuild-easyconfigs)
  • determining the branch corresponding to the pull request, which should be updated by pushing a new commit to it
  • checking out that branch
  • renaming easyconfig files according to their name, version, versionsuffix and toolchain
  • moving easyconfig files to the right location in the repository (e.g. easybuild/easyconfigs/e/EasyBuild/)
  • staging and committing the (changed/new) files
  • pushing the updated branch to GitHub

Again, not a single git command to be executed; the only thing that is required is the ID of the pull request that should be updated.

Just like with --new-pr, this is done in a temporary working copy of the repository, no changes are made to a possible existing working copy.

Example

For example, to update pull request #3153 with a changed easyconfig file:

eb --update-pr 3153 ~/WIP/newEB.eb --experimental
== temporary log file in case of crash /tmp/eb-gO2wJu/easybuild-37Oo2z.log
== Determined branch name corresponding to hpcugent/easybuild-easyconfigs PR #3153: 20160530131447_new_pr_EasyBuild281
== copying /home/example/git-working-dirs/easybuild-easyconfigs...
== fetching branch '20160530131447_new_pr_EasyBuild281' from https://github.com/boegel/easybuild-easyconfigs.git...
Overview of changes:
 easybuild/easyconfigs/e/EasyBuild/EasyBuild-2.8.1.eb | 3 +++
 1 file changed, 3 insertions(+)

Updated hpcugent/easybuild-easyconfigs PR #3159 by pushing to branch boegel/20160530131447_new_pr_EasyBuild281

Controlling pull request metadata

You can control the metadata for pull requests using the following configuration options:

  • --pr-branch-name: branch name for new pull requests
  • --pr-commit-msg: commit message to use when creating new or updating existing pull requests
  • --pr-descr: pull request description
  • --pr-title: pull request title

EasyBuild will use sensible defaults for each of these, see below.

Default branch name for new pull requests

The branch name for new pull requests will be composed from:

  • a timestamp, down to the second in an attempt to make it unique
    • example: 20160513141133 for a pull request created on May 13th 2016, 2:11:33 PM
  • a label new_pr
  • the software name and version of the first easyconfig file, with some filtering (e.g. remove .‘s)
    • example: GCC530 for GCC v5.3.0

Full example: 20160513141133_new_pr_GCC530

Although there is usually no reason to change this default, it can be done if desired using --pr-branch-name when opening a new pull request with --new-pr.

Default commit message

The default commit message is very simple: it specifies for each easyconfig file whether it was a new file being added, or an existing file being modifed, for example “add easyconfig GCC-5.3.0.eb, modify easyconfig GCC-4.9.3.eb”.

It is highly recommended to provide a more descriptive commit message via --pr-commit-msg whenever existing easyconfig files are being modified, both with --new-pr and --update-pr.

Default pull request description

By default, the pull description will only contain the following text:

(created using eb --new-pr)

It is generally advised to provide more descriptive information, although the changes made by the pull request may be self-explanatory (e.g. when only adding new easyconfig files).

To change this default text, you can either use --pr-descr or edit the description via the GitHub interface after the pull request has been opened.

Particularly useful information to specify here is dependencies on other pull requests, by copy-pasting the respective URLs with a short descriptive message like ‘depends on PR <URL>‘.

Default pull request title

The pull request title is derived from the easyconfig files being changed/added, taking into account the recommendation for easyconfig pull requests to clearly specify module class, toolchain, software name/version, as follows: {<module_class>}[<toolchain>] <software_name> v<software_version>.

For example, when opening a pull request for an easyconfig for Python 2.7.11 with the intel/2016a toolchain, the default pull request title will be something like: {lang}[intel/2016a] Python v2.7.11 .

If multiple easyconfig files are provided, the respective software names/versions will be included separated by a ,, up until the first 3 easyconfig files (to avoid excessively lenghty pull request titles).

In case (only) existing easyconfig files are being changed, it’s advisable to provide a more descriptive title using --pr-title.

Configuring --new-pr and --update-pr

By default, --new-pr and --update-pr affect pull requests to the central hpcugent/easybuild-easyconfigs repository.

However, this can be changed with the following configurations options:

  • --pr-target-account (default: hpcugent): target GitHub account for new pull requests
  • --pr-target-branch (default: develop): target branch for new pull requests
  • --pr-target-repo (default: easybuild-easyconfigs): target repository for new pull requests

Synergy with --dry-run/-D and --extended-dry-run/-x

Both --new-pr and --update-pr are ‘dry run-aware’, in the sense that you can combine them with either --dry-run/-D- or --extended-dry-run/-x to preview the pull request they would create/update without actually doing so.

For example:

$ eb --new-pr EasyBuild-2.9.0.eb --experimental -D
== temporary log file in case of crash /tmp/eb-1ny69k/easybuild-UR1Wr4.log
== copying /home/example/git-working-dirs/easybuild-easyconfigs...
== fetching branch 'develop' from https://github.com/hpcugent/easybuild-easyconfigs.git...

Opening pull request [DRY RUN]
* target: hpcugent/easybuild-easyconfigs:develop
* from: boegel/easybuild-easyconfigs:20160603105641_new_pr_EasyBuild290
* title: "{tools}[dummy/dummy] EasyBuild v2.9.0"
* description:
"""
(created using `eb --new-pr`)

"""
* overview of changes:
 .../easyconfigs/e/EasyBuild/EasyBuild-2.9.0.eb     | 35 ++++++++++++++++++++++
 1 file changed, 35 insertions(+)

The only difference between using --dry-run and --extended-dry-run is that the latter will show the full diff of the changes (equivalent to git diff), while the former will only show a summary of the changes (equivalent to git diff --stat, see example above).