diff --git a/source/developer/bulk.rst b/source/developer/bulk.rst index c4feeb5..42d3fd5 100644 --- a/source/developer/bulk.rst +++ b/source/developer/bulk.rst @@ -1,199 +1,296 @@ The ``bulk`` branch =================== -Sometimes we need to do maintenance or make changes to lots of recipes at once. -This happens most often when there is a new Bioconductor release: all -`bioconductor-*` recipes need to be updated, and the corresponding packages -need to be built. +.. datechanged:: 2025-01-04 + Updated based on recent BioC 3.20 builds -This ends up taking substantial compute time on CI infrastructure. If this were -run on the same CI infrastructure that processes pull requests, this might -consume CI time needed for the typical functioning of the daily activity in the -bioconda repository. The ``bulk`` branch is a mechanism for the Bioconda core -team to perform large-scale changes on a different CI system, without tying up -the CI infrastructure used by contributors on individual pull requests. - -The bulk branch reads its configuration (which version of bioconda-utils and -miniconda) from the ``bulk`` branch of `bioconda-common`. +The ``bulk`` branch is used for large-scale maintenance tasks, like supporting +a new version of Python or building all `bioconductor-*` recipes for a new +release of Bioconductor. These tasks can take up substantial compute time, so +they are run on separate CI workflows to avoid disrupting normal daily +activity in the bioconda-recipes repository. **The bulk branch immediately uploads successfully built packages to Anaconda.** As such, only the bioconda core team has the ability to push to this branch. -Interacting with the bulk branch --------------------------------- +.. _bulk-jobs: + +Managing a bulk run +------------------- -When changes are made on the bulk branch (committed and pushed), the CI system -decides whether it will run its jobs or not. +#. **Merge master into bulk.** Start with a clean slate by merging master into + bulk, preferring what's on master: -* If any of the pushed commit messages contains the substring ``[ci run]`` the CI jobs are executed. -* If not, no CI jobs are executed. + .. code-block:: bash -The reason for this behavior is that we want to avoid race conditions caused by -multiple CI jobs spawned from different commits, possibly from different -people, to be exectuted at the same time. + git checkout bulk + git merge master -s recursive -X theirs -In order to simplify interactions with the bulk CI, bioconda-utils offers therefore -some dedicated subcommands: + There may be a few remaining conflicts to fix; in all cases you should + prefer what's on the master branch. -* **bulk-commit**: ``bioconda-utils bulk-commit `` commits the changes on your - local clone of https://github.com/bioconda/bioconda-recipes to the bulk branch while marking the commit - as being eligible for a CI run (by automatically prefixing the message with ``[ci run]``). - The **bulk-commit** subcommand does not push the commit. This enables you to do multiple fine-grained commits - and push them in one pass via a subsequent ``git push`` that triggers a single CI run. -* **bulk-trigger-ci**: ``bioconda-utils bulk-trigger-ci`` creates an empty commit that is - immediately pushed automatically to the bulk branch, thereby triggering a CI run. This can be used - to restart the CI run in case all of the previous runs are finished without build failures but there - are still packages that need to be built (and haven't been before because the job runtime limits were - reached and the CI has terminated them (usually this happens after somewhat more than 5 hours)). +#. **Make initial changes.** Typically one person is responsible for this part. + + * For pinning updates, see :ref:`update-pinnings` + * For Bioconductor releases, see :ref:`update-bioconductor` + * If a Bioconductor release *also* uses a new R version, complete a pinning + migration first. You will need to wait for (or better yet, help out with) + conda-forge building packages for the new R version. See + :ref:`update-bioconductor` for more details. + * For other changes, you will likely need to write your own tooling. + +#. **Commit and push** these changes to bulk. + +#. **Start a run.** Use ``bioconda-utils bulk-trigger-ci`` to start a run. Or + include the string ``[ci run]`` in a commit message. + +#. **Address build failures.** Inspect all build + failures (see :ref:`handling-build-failures`). This is the time consuming + part. For each failure, decide whether the recipe shall be skiplisted or + whether you would like to fix it. In general, spend time to fix highly + depended-upon packages and anything else that is obvious and easy. For the + rest, mark the recipes as skiplisted in the build failure file. It will be + ignored by subsequent CI runs and put into a table in the bioconda-recipes + wiki. This strategy is good because the bulk branch update should be + performed as fast as possible to avoid redundant work between master and + bulk. Also, skiplisting democratizes the update effort. + + * Push commits as soon as they are done, so other people know the build + failure has been addressed. It may be helpful to prefix your commit + message with the recipe name, for easy viewing on the `bulk branch Actions + dashboard + `__ + or with ``git log --oneline --pretty=format:"%ad %an %s" --date=iso + | grep -v "ci skip" | grep -v "ci run" | less`` + * Your local branch may be out of date with the remote, since the bulk runs + commit any build failure yamls. So a typical series of commands is ``git + commit ...``, ``git pull origin bulk --rebase``, ``git push origin bulk``. + +#. **Iterate.** If no untreated failure remains, run ``bioconda-utils + bulk-trigger-ci`` to start another run, address build failures, and start + another run. *Warning, this process can take weeks.* See :ref:`bulk-notes` for + some pointers. + +#. **Merge master into bulk again.** Once all the packages have either been + successfully built or skiplisted, locally pull the master branch and merge + it into bulk. Usually, conflicts can occur here due to build-numbers having + been increased in the master branch while you did your changes in bulk. For + such cases, increase the build number to ``max(build_number_master, + build_number_bulk)`` and commit all of those. Repeat this until master is + merged without any conflicts. + +#. **Update common.sh for master** Ensure that `bioconda-common/common.sh + `_ points + to the same version of bioconda-utils that the ``bulk`` branch has been + using. + +#. **Merge bulk into master.** Open a PR to merge bulk into master and merge it. + +#. **Work on fixing skiplisted recipes.** Shortly afterwards, you will find all + remaining build failures in the `bioconda-recipes wiki + `_. You + can let your colleagues and the community know about the updated build + failure table and ask for help. In addition, any automatic or manual updates + to recipes on this list that succeed will automatically remove them from + this list over time. + + +.. _update-pinnings: Updating pinnings ----------------- -Pinnings are updated for example when we are supporting a new version of -Python. These are versions of base packages that are supported, and form the -basis of the build string hashes at the end of conda package names. A recent -example is updating pinnings to support Python 3.10. +Bioconda uses `conda-forge-pinning +`__ +for consistency with the conda-forge ecosystem. You can read more about global +pinnings in the corresponding `conda-forge docs +`__. + +Bioconda also uses `bioconda-specific pinning +`__ +which has some overrides and bioinformatics-specific pinnings. -1. Update `bioconda pinnings +Pinnings are typically updated in an *ad hoc* fashion, but generally coincide +with version bumps in underlying packages, or when we are ready to support +a new Python or R version. Here is what you need to do: + +#. Follow the **Merge master into bulk** step in :ref:`bulk-jobs`. + +#. Update `conda-forge-pinning` in `bioconda-utils requirements + `__. + +#. Update `bioconda pinnings `_. This may take a few tries; you may need to make changes to match conda-forge's pinnings. Merge these changes into the master branch of - bioconda-utils (which will create or update a Release Please PR) and merge + bioconda-utils (which will create or update a Release Please PR). Merge in the Release Please PR to create a new version of bioconda-utils. -2. Allow autobump to pick up the new version and create a PR in - bioconda-recipes for the new version of bioconda-utils. This usually takes - an hour. Then merge the corresponding PR in bioconda-recipes. You now have - a new bioconda-utils package to use which contains those pinnings. - -3. Update ``common.sh`` (see `here - `_) **only on the bulk +#. Update ``common.sh`` (see `here + `_) **only on the bulk branch in bioconda-common**, to match the newly-updated bioconda-utils version. Changing the pinnings will likely trigger many recipes to require rebuilding. Since the bioconda-recipes/bulk branch reads from the bioconda-common/bulk branch, this allows bulk to run a different version of bioconda-utils. Once a bulk migration is complete, you can update the master - branch of bioconda-common. + branch of bioconda-common to point to the bioconda-utils version used for bulk. -4. In bioconda-recipes, merge master into bulk to start with a clean slate. - Since bulk is infrequently updated, there may be substantial conflicts - caused by running the default ``git checkout bulk && git merge master``. - This tends to happen most with build numbers. But since we want to prefer - using whatever is in the master branch, we can merge master into bulk, while - preferring master version in any conflicts, with: +#. Run ``bioconda-utils update-pinnings`` in the bulk branch. This will go + through all the pinnings, figure out what recipes they're used with, and + bump the recipes' build numbers appropriately. Note, this may take a few GB + of RAM and a bit of time. - .. code-block:: bash +#. **IMPORTANT:** if you are also doing a Bioconductor release with a new + version of R, then **revert changes to all Bioconductor packages** from the + above command with ``git checkout -- recipes/bioconductor-*``. This way, we + avoid new builds of BioC packages for a new version of R that they may not + be compatible with. Once the pinning updates have been completed on bulk by + following the rest of the steps in :ref:`bulk-jobs`, then start over again + but follow the :ref:`updating-bioconductor` steps to make the initial + changes on bulk. - git checkout bulk - git merge master -s recursive -X theirs +Then continue following the steps in :ref:`bulk-jobs`. - There may be a few remaining conflicts to fix; in all cases you should - prefer what's on the master branch. +.. _updating-bioconductor: + +Updating Bioconductor +--------------------- + +Bioconductor gets updated twice a year (spring and fall). All Bioconductor +packages are designed to work together within a Bioconductor release, so we +need to update all packages simultaneously, building packages in order of the +dependency tree. + +**Bioconductor releases are tied to an R version.** We need to wait until +conda-forge finishes, or at least gets to an advanced stage of building +packages for the new version of R. + +To view the current status of a conda-forge r-base migration, visit the `conda-forge status dashboard `_. In the bioconda-recipes repository, use the ``missingCranPackages.py`` script to generate a list of all incomplete packages that are referenced in a bioconda recipe. Example: ``python ./scripts/bioconductor/missingCranPackages.py --format markdown --migration_id r-base44_and_m2w64-ucrt`` The markdown output can be added to an Issue for tracking purposes. + +Then, we need to first go through the +:ref:`update-pinnings` workflow (while ensuring Bioconductor packages DO NOT +have their build numbers updated). This ensures the non-BioConductor packages +are built for the new version of R. + +Then we can proceed with updating Bioconductor packages: + +#. Follow the **Merge master into bulk** step in :ref:`bulk-jobs`. + +#. Identify the latest BioConductor version, and update all BioConductor + recipes in the bulk branch with the following. This will take time because + there are thousands of Bioconductor recipes, and the tarballs are downloaded + for all of them: + + .. code-block:: bash + + bioconda-utils bioconductor-skeleton update-all-packages --bioc-version $BIOC_VERSION + +#. The `bioconductor-data-packages` will have changed with the URLs to data + packages. Manually bump the version to match the current date reflect this. -5. Run ``bioconda-utils update-pinnings`` in - the bulk branch. This will go through all the pinnings, figure out what - recipes they're used with, and bump the recipes' build numbers - appropriately. - -6. Then, **bulk-commit** and push the changes. - -7. Once the CI run has finished, inspect all build failures (see :ref:`handling-build-failures`). - For each failure, decide whether the recipe shall be skiplisted or whether you would like to fix it. - In general it is advisable to fix all libraries on which many recipes depend and anything else - that is obvious and easy. For the rest, mark the recipes as skiplisted in the build failure file. - It will be ignored by subsequent CI runs and put into a table in the bioconda-recipes wiki. - This strategy is good because the bulk branch update should be performed as fast as possible to avoid - redundant work between master and bulk. Also, skiplisting democratizes the update effort. - -8. If no untreated failure remains, **bulk-commit** (see above) and push the changes and visit - step 6-7 again. If the run has finished without any build failure and did not time out before checking all - recipes, you can go on with step 7. - -9. Once all the packages have either been successfully built or skiplisted, pull the master branch and merge it into bulk. - Usually, conflicts can occur here due to build-numbers having been increased in the master branch while you - did your changes in bulk. For such cases (which should be not so many) you can just increase the build number to - ``max(build_number_master, build_number_bulk)`` and **bulk-commit** all of those in a row. - Repeat this until master is merged without any conflicts. - Ensure that `bioconda-common/common.sh `_ points to the same version of - bioconda-utils that the ``bulk`` branch has been using. Then, merge bulk into master and push the changes. - -10. Shortly afterwards, you will find all remaining build failures in the - `bioconda-recipes wiki `_. - You can let your colleagues and the community know about the updated build failure table and ask for help. - In addition, any automatic or manual updates to recipes on this list that succeed will automatically - remove them from this list over time. - -.. _handling-build-failues: +#. Commit and push the changes. + +Then continue following the steps in :ref:`bulk-jobs`. + +.. note:: + + Sometimes BioConductor packages get updated shortly after release, and the + originally-released version is removed. This may happen before the bulk + branch builds the version originally specified by the + ``bioconductor-skeleton`` run, which results in errors trying to download the source. + + In such cases, run ``bioconda-utils bioconductor-skeleton + --force`` to update it. If it was a data package, then you will need to + manually bump the version of ``bioconductor-data-packages`` as well. + +.. _handling-build-failures: Handling build failures ~~~~~~~~~~~~~~~~~~~~~~~ Build failures are stored in a file ``build_failure..yaml`` next to each -failing recipe. You can list all build failures stored in the current branch of -bioconda-recipes via the command ``bioconda-utils list-build-failures recipes -config.yml``. This reads the yaml files from failing recipes, and prints -a table on stdout that will be sorted by the number of dependencies and package -downloads, which should help for prioritizing the fixing work. +failing recipe. These are committed back to the bulk branch after every failed +recipe with the ``--record-build-failures`` argument. You can list all build +failures stored in the current branch of bioconda-recipes via the command +``bioconda-utils list-build-failures recipes config.yml``. This reads the yaml +files from failing recipes, and prints a table on stdout that will be sorted by +the number of dependencies and package downloads, which should help for +prioritizing the fixing work. + +Since the ``list-build-failures`` command can take time to run, it is often +more convenient to search the build logs for the latest bulk run. Useful search +strings are: + * ``BUILD FAILED`` for generic failures + * ``failed linting`` for linting errors + * ``TEST FAILED`` for mulled-build failures + +Or, inspect the git log to see what build failures were added in the last day: + +.. code-block:: bash + git log --since="1.days ago" --pretty=format:"%ad %h %s" --date=iso | grep "\[ci skip\] add build failure record" + + +The build failure files look like this by default: -This file can look e.g. like this: .. code-block:: yaml recipe_sha: 37fa4d78a2ee8b18065a0bd0f594ad1e9587bb4ac7edf1b4629a9f10fa45d0a5 # The shas256 hash of the recipe at which it failed to build. - skiplist: true # Set to true to skiplist this recipe so that it will be ignored as long as its latest commit is the one given above. + skiplist: false # Set to true to skiplist this recipe so that it will be ignored as long as its latest commit is the one given above. log: |2- -Based on the log, you can decide whether and how the recipe can be fixed or whether it shall be skiplisted for -fixing it later in the future. -Notably, any update to the recipe automatically de-skiplists it, because the skiplist -entry is only valid together with the hash listed in the first line. +If a failed recipe is a leaf (i.e., it is not a dependency for any other +recipe), then it **WILL** be automatically skiplisted (``skiplist: true``) due +to the ``--skiplist-leafs`` argument, and need to be handled later. -It is possible to further annotate and even manually create build failure records via the `bioconda-utils` CLI. -Check out all possibilities in the corresponding help message: +Based on this log, you can decide whether and how the recipe can be fixed or +skiplisted for fixing it later. To help others in the future, add information +in the "reason" field if you have any ideas of where to start fixing the +package. -.. code-block:: bash +You can manually edit the build failure yamls, or use the command line tool: - bioconda-utils annotate-build-failures --help +.. code-block:: bash -Skiplisted recipes from the master branch are automatically displayed in a `wiki page `_, -so that others can pick them up for providing a fix. + bioconda-utils annotate-build-failures \ + -c 'dependency issue' \ + -r 'package xyz needs to be added to conda-forge' \ + -s \ + recipes/packagename +Which will make the build failure look like this: -Updating Bioconductor ---------------------- - -Bioconductor gets updated twice a year (spring and fall), where all BioC -packages get released with updated versions at the same time. This in turn -requires updating the packages on Bioconda. This is a perfect use-case for the -bulk branch. The process is generally the same as above but without the -pinnings updates and with some Bioconductor-specific helper scripts. - -1. Execute step 4 from above. +.. code-block:: yaml -2. Identify the latest BioConductor version, and update all BioConductor - recipes with: + recipe_sha: 37fa4d78a2ee8b18065a0bd0f594ad1e9587bb4ac7edf1b4629a9f10fa45d0a5 # The shas256 hash of the recipe at which it failed to build. + skiplist: true # Set to true to skiplist this recipe so that it will be ignored as long as its latest commit is the one given above. + log: |2- + + category: |- + dependency issue + reason: |- + package xyz needs to be added to conda-forge - .. code-block:: bash +Any update to the meta.yaml automatically de-skiplists it, because the skiplist +entry is only valid together with the hash listed in the first line. - bioconda-utils bioconductor-skeleton update-all-packages --bioc-version $BIOC_VERSION +It is possible to further annotate and even manually create build failure +records via the `bioconda-utils` CLI. Check out all possibilities in the +corresponding help message: -3. Execute step 6 from above. +.. code-block:: bash -4. Execute step 7 from above. - Alternatively, use the - [rootNodes.py](https://github.com/bioconda/bioconda-recipes/blob/master/scripts/bioconductor/rootNodes.py) - from the bioconda-recipes repo to help figure out what the primary root - nodes are for the currently-remaining packages to be built. This looks at - recently-built packages, removes them from the DAG of recipes to be built, - and then reports to stdout the remaining root nodes. This information can be - used to strategically edit the ``build-fail-blacklist`` file to prioritize - the building of those root nodes. Once builds seem to be stabilizing, remove the temporary edits to the - ``build-fail-blacklist``. + bioconda-utils annotate-build-failures --help -5. Execute step 8-10 from above. +Skiplisted recipes from the master branch are automatically displayed in +a `wiki page +`_, so that +others can pick them up for providing a fix. Failures that only exist on the Bulk branch, but not on the master branch, are displayed on `this wiki page +`_. Note that this page is only updated when a build failure yaml file is updated and the commit message does not contain "[ci skip]". +.. _bulk-notes: Notes on working with bulk branch --------------------------------- @@ -202,13 +299,20 @@ Some unordered notes on working with the bulk branch: - Remember that successfully-built packages are immediately pushed to Anaconda. -- You may want to coordinate the timing of fixes and pushes (say, via gitter). - This is because the bulk branch has ``fail-fast: false`` set to allow - parallel jobs to progress as much as possible. Multiple people pushing to - bulk means that there is a risk of trying to build the same recipes multiple - times. In such a case, only the first package will be actually uploaded and - subsequent packages will a failure on the upload step. So there is no danger - to the channel, it's just poor use of CI resources. +- Use ``--subdag-depth`` (see visualization in `#950 + `__) to restrict what is + built, especially in early stages. This will hopefully reduce the frequency + of recipes skiplisted only because their dependencies happened to not get + built yet due to being on a different worker. + +- Bulk migrations can take weeks. Plan accordingly. + +- The bulk branch has ``fail-fast: false`` set to allow parallel jobs to + progress as much as possible. If multiple people trigger a bulk run, jobs + will run simultaneously and likely will do duplicate work. Whichever worker + successfully pushes a package first wins and the other will fail when trying + to push. So there is no danger to the channel, it's just poor use of CI + resources. - The logs are awkward to read and hard to find exactly where failures occur. One way to do this is to go to the bottom where there is a report of which @@ -218,21 +322,39 @@ Some unordered notes on working with the bulk branch: nothing to do will be reported in a short stanza, so you can use those as structural markers to indicate where there's no useful log info. +- Here are some search strings to help narrow down issues: + * ``BUILD FAILED`` for generic failures + * ``failed linting`` for linting errors + * ``TEST FAILED`` for mulled-build failures + - Instead of using the search functionality in the CI logs, download the raw log (from gear menu at top right) to use your browser search functionality, which is often much easier to use (for example, Chrome shows occurrences of search term throughout the document in the scrollbar, which makes digging for the actual error a lot easier). -- You may see a lot of output for Python packages in particular. This is because for - bioconda-utils to figure out whether it needs to build the package, it needs - to know what the hash is for the package. This in turn requires figuring out - all the dependencies to see which of them are pinned and then using those to - calculate a hash. So it may appear that it's doing a lot of work for packages - that don't need to be rebuilt, but that work needs to be done simply to - figure out if a rebuild is needed, and so this is expected. - -- For ``linux-64``, ``osx-64``, and ``osx-arm64`` the bulk runs take place on GitHub Actions, - and the configuration is in :file:`.github/workflows/Bulk.yml`. For - ``linux-aarch64``, the builds take place on CircleCI and the configuration is - in :file:`.circleci/config.yml`. +- You may see a lot of output for Python packages in particular. To determine + whether the recipe needs to be built, we need to compute the hash for the + build string. This in turn requires figuring out all the dependencies to see + which of them are pinned and then using those to calculate a hash. This needs + to be done for each version of Python that we support. So it may appear that + it's doing a lot of work for packages that don't need to be rebuilt, but that + work needs to be done simply to figure out if a rebuild is needed, and so + this is expected. + +- For ``linux-64``, ``osx-64``, and ``osx-arm64`` the bulk runs take place on + GitHub Actions, and the configuration is in + :file:`.github/workflows/Bulk.yml`. For ``linux-aarch64``, the builds take + place on CircleCI and the configuration is in :file:`.circleci/config.yml`. + +- Jobs time out at 6 hours on GitHub Actions and 1 hour on Circle CI. These limits are likely to be hit early in the process. If the timeout is reached, wait for all jobs to complete (pass, fail, or timeout), and trigger a new run. + +- You may end up with a lot of skiplisted leaf packages -- especially from + packages whose dependencies were not built yet because they were on + a different worker. ``--subdag-depth`` (described above) can help with this. + On one hand, merging these into master will let others in the community + contribute (remember, only core team can push to branch). But on the other + hand, removing the build failure yamls later on during a bulk migration can + take advantage of the bulk branch's resources. Currently, you'll need to + manually find the build failures to try to remove which can be + time-consuming, so work out the best balance for yourself.