Merge pull request #305 from nf-core/dev

Prepare nf-core tools release 1.6
nf-core · Apr 9, 2019 · 4ab80b8 · 4ab80b8
2 parents 1855d34 + 95624ed
commit 4ab80b8
Show file tree

Hide file tree

Showing 31 changed files with 455 additions and 545 deletions.
diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md
@@ -6,7 +6,7 @@ We try to manage the required tasks for nf-core/tools using GitHub issues, you p
 
 However, don't be put off by this template - other more general issues and suggestions are welcome! Contributions to the code are even more welcome ;)
 
-> If you need help using or developing nf-core/tools then the best place to go is the Gitter chatroom where you can ask us questions directly: https://gitter.im/nf-core/Lobby
+> If you need help using or developing nf-core/tools then the best place to ask is the nf-core `tools` channel on [Slack](https://nf-core-invite.herokuapp.com/).
 
 ## Contribution workflow
 If you'd like to write some code for nf-core/tools, the standard workflow
@@ -24,7 +24,7 @@ If you're not used to this workflow with git, you can start with some [basic doc
 
 ## Style guide
 Google provides an excellent [style guide](https://github.com/google/styleguide/blob/gh-pages/pyguide.md), which
-is an best practise extension of [PEP](https://www.python.org/dev/peps/), the Python Enhancement Proposals. Have a look at the 
+is a best practise extension of [PEP](https://www.python.org/dev/peps/), the Python Enhancement Proposals. Have a look at the
 [docstring](https://github.com/google/styleguide/blob/gh-pages/pyguide.md#38-comments-and-docstrings) section, which is in particular
 important, as nf-core tool's code documentation is generated out of these automatically.
 
@@ -68,4 +68,4 @@ nf-core lint nf-core-testpipeline
 ```
 
 ## Getting help
-For further information/help, please consult the [nf-core/tools documentation](https://github.com/nf-core/tools#documentation) and don't hesitate to get in touch on [Gitter](https://gitter.im/nf-core/Lobby)
+For further information/help, please consult the [nf-core/tools documentation](https://github.com/nf-core/tools#documentation) and don't hesitate to get in touch on the nf-core `tools` channel on [Slack](https://nf-core-invite.herokuapp.com/).
diff --git a/.travis.yml b/.travis.yml
@@ -26,10 +26,13 @@ install:
   - sudo apt-get install npm && npm install -g markdownlint-cli # For documentation linting
 
 script:
+  # Test the main code in this repo
   - pytest --cov=nf_core .
+  - markdownlint . -c ${TRAVIS_BUILD_DIR}/.github/markdownlint.yml
+  # Test the pipeline template code
   - nf-core create -n testpipeline -d "This pipeline is for testing" -a "Testing McTestface"
   - nf-core lint nf-core-testpipeline
-  - markdownlint . -c ${TRAVIS_BUILD_DIR}/.github/markdownlint.yml
+  - markdownlint nf-core-testpipeline -c nf-core-testpipeline/.github/markdownlint.yml
 
 after_success:
   - codecov --rcfile=${TRAVIS_BUILD_DIR}/.github/coveragerc

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,37 @@
 # nf-core/tools: Changelog
 
+## v1.6
+
+#### Syncing
+* Code refactoring to make the script more readable
+* No travis build failure anymore on sync errors
+* More verbose logging
+
+#### Template pipeline
+* awsbatch `work-dir` checking moved to nextflow itself. Removed unsatisfiable check in main.nf template.
+* Fixed markdown linting
+* Tools CI testing now runs markdown lint on compiled template pipeline
+* Migrated large portions of documentation to the [nf-core website](https://github.com/nf-core/nf-co.re/pull/93)
+* Removed Gitter references in `.github/` directories for `tools/` and pipeline template.
+* Changed `scrape_software_versions.py` to output `.csv` file
+* Added `export_plots` parameter to multiqc config
+
+#### Tools helper code
+* Drop [nf-core/rnaseq](https://github.com/nf-core/rnaseq]) from `blacklist.json` to make template sync available
+* Updated main help command to sort the subcommands in a more logical order
+* Updated readme to describe the new `nf-core launch` command
+* Fix bugs in `nf-core download`
+  * The _latest_ release is now fetched by default if not specified
+  * Downloaded pipeline files are now properly executable.
+* Fixed bugs in `nf-core list`
+  * Sorting now works again
+  * Output is partially coloured (better highlighting out of date pipelines)
+  * Improved documentation
+* Fixed bugs in `nf-core lint`
+  * The order of conda channels is now correct, avoiding occasional erroneous errors that packages weren't found ([#207](https://github.com/nf-core/tools/issues/207))
+* Add reporting of ignored errored process
+  * As a solution for [#103](https://github.com/nf-core/tools/issues/103))
+
 ## [v1.5](https://github.com/nf-core/tools/releases/tag/1.5) - 2019-03-13 Iron Shark
 
 #### Template pipeline
@@ -14,6 +46,7 @@
 * Completion email now includes MultiQC report if not too big
 * `params.genome` is now checked if set, to ensure that it's a valid iGenomes key
 * Together with nf-core/configs, helper function now checks hostname and suggests a valid config profile
+* `awsbatch` executor requires the `tracedir` not to be set to an `s3` bucket.
 
 #### Tools helper code
 * New `nf-core launch` command to interactively launch nf-core pipelines from command-line

diff --git a/Dockerfile b/Dockerfile
@@ -3,5 +3,5 @@ LABEL authors="[email protected],[email protected]"
       description="Docker image containing base requirements for the nfcore pipelines"
 
 # Install procps so that Nextflow can poll CPU usage
-RUN apt-get update && apt-get install -y procps && apt-get clean -y 
+RUN apt-get update && apt-get install -y procps && apt-get clean -y
 RUN conda install conda=4.6.7
diff --git a/README.md b/README.md
@@ -9,13 +9,18 @@ A python package with helper tools for the nf-core community.
 
 ## Table of contents
 
-* [Installation](#installation)
-* [Listing pipelines](#listing-pipelines) (`nf-core list`)
-* [Downloading pipelines for offline use](#downloading-pipelines-for-offline-use) (`nf-core download`)
-* [Listing software licences](#pipeline-software-licences): List software licences for a given workflow (`nf-core licences`)
-* [Creating a new workflow](#creating-a-new-workflow) (`nf-core create`)
-* [Checking a pipeline against nf-core guidelines](#linting-a-workflow) (`nf-core lint`)
-* [Bumping a pipeline version number](#bumping-a-pipeline-version-number) (`nf-core bump-version`)
+* [`nf-core` tools installation](#installation)
+* [`nf-core list` - List available pipelines](#listing-pipelines)
+* [`nf-core launch` - Run a pipeline with interactive parameter prompts](#launch-a-pipeline)
+* [`nf-core download` - Download pipeline for offline use](#downloading-pipelines-for-offline-use)
+* [`nf-core licences` - List software licences in a pipeline](#pipeline-software-licences)
+* [`nf-core create` - Create a new workflow from the nf-core template](#creating-a-new-workflow)
+* [`nf-core lint` - Check pipeline code against nf-core guidelines](#linting-a-workflow)
+* [`nf-core bump-version` - Update nf-core pipeline version number](#bumping-a-pipeline-version-number)
+
+
+The nf-core tools package is written in Python and can be imported and used within other packages.
+For documentation of the internal Python functions, please refer to the [Tools Python API docs](https://nf-co.re/tools-docs/).
 
 ## Installation
 
@@ -25,6 +30,12 @@ You can install `nf-core/tools` from [PyPI](https://pypi.python.org/pypi/nf-core
 pip install nf-core
 ```
 
+Alternatively, it can be installed using [bioconda](https://bioconda.github.io/recipes/nf-core/README.html):
+
+```bash
+conda install -c bioconda nf-core
+```
+
 If you would like the development version instead, the command is:
 
 ```bash
@@ -50,7 +61,7 @@ The command `nf-core list` shows all available nf-core pipelines along with thei
 
 An example of the output from the command is as follows:
 
-```bash
+```console
 $ nf-core list
 
                                           ,--./,-.
@@ -60,23 +71,27 @@ $ nf-core list
                                           `._,._,'
 
 
-Name               Version    Published    Last Pulled    Default local is latest release?
------------------  ---------  -----------  -------------  ----------------------------------
-nf-core/hlatyping  1.1.0      5 days ago   9 minutes ago  Yes
-nf-core/methylseq  1.1        1 week ago   2 months ago   No
-nf-core/chipseq    dev        -            -              No
-nf-core/eager      dev        -            -              No
-nf-core/exoseq     dev        -            -              No
-nf-core/mag        dev        -            -              No
-nf-core/rnaseq     dev        -            -              No
-nf-core/smrnaseq   dev        -            -              No
-nf-core/vipr       dev        -            -              No
+Name                       Version    Released      Last Pulled     Have latest release?
+-------------------------  ---------  ------------  --------------  ----------------------
+nf-core/rnaseq             1.3        4 days ago    27 minutes ago  Yes
+nf-core/hlatyping          1.1.4      3 weeks ago   1 months ago    No
+nf-core/eager              2.0.6      3 weeks ago   -               -
+nf-core/mhcquant           1.2.6      3 weeks ago   -               -
+nf-core/rnafusion          1.0        1 months ago  -               -
+nf-core/methylseq          1.3        1 months ago  3 months ago    No
+nf-core/ampliseq           1.0.0      3 months ago  -               -
+nf-core/deepvariant        1.0        4 months ago  -               -
+nf-core/atacseq            dev        -             1 months ago    No
+nf-core/bacass             dev        -             -               -
+nf-core/bcellmagic         dev        -             -               -
+nf-core/chipseq            dev        -             1 months ago    No
+nf-core/clinvap            dev        -             -               -
 ```
 
 To narrow down the list, supply one or more additional keywords to filter the pipelines based on matches in titles, descriptions and topics:
 
-```bash
-nf-core list rna rna-seq
+```console
+$ nf-core list rna rna-seq
 
                                           ,--./,-.
           ___     __   __   __   ___     /,-._.--~\
@@ -85,16 +100,109 @@ nf-core list rna rna-seq
                                           `._,._,'
 
 
-Name              Version    Published     Last Pulled    Default local is latest release?
-----------------  ---------  ------------  -------------  ----------------------------------
-nf-core/rnaseq    1.0        20 hours ago  -              No
-nf-core/smrnaseq  dev        -             -              No
+Name               Version    Released      Last Pulled     Have latest release?
+-----------------  ---------  ------------  --------------  ----------------------
+nf-core/rnaseq     1.3        4 days ago    28 minutes ago  Yes
+nf-core/rnafusion  1.0        1 months ago  -               -
+nf-core/lncpipe    dev        -             -               -
+nf-core/smrnaseq   dev        -             -               -
 ```
 
-You can sort the results by latest release (default), name (alphabetical) or number of GitHub stars using the `-s`/`--stars` option.
+You can sort the results by latest release (`-s release`, default),
+when you last pulled a local copy (`-s pulled`),
+alphabetically (`-s name`),
+or number of GitHub stars (`-s stars`).
+
+```console
+$ nf-core list -s stars
+
+                                          ,--./,-.
+          ___     __   __   __   ___     /,-._.--~\
+    |\ | |__  __ /  ` /  \ |__) |__         }  {
+    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
+                                          `._,._,'
+
+
+Name                         Stargazers  Version    Released      Last Pulled     Have latest release?
+-------------------------  ------------  ---------  ------------  --------------  ----------------------
+nf-core/rnaseq                       81  1.3        4 days ago    30 minutes ago  Yes
+nf-core/methylseq                    22  1.3        1 months ago  3 months ago    No
+nf-core/ampliseq                     21  1.0.0      3 months ago  -               -
+nf-core/chipseq                      20  dev        -             1 months ago    No
+nf-core/deepvariant                  15  1.0        4 months ago  -               -
+nf-core/eager                        14  2.0.6      3 weeks ago   -               -
+nf-core/rnafusion                    14  1.0        1 months ago  -               -
+nf-core/lncpipe                       9  dev        -             -               -
+nf-core/exoseq                        8  dev        -             -               -
+nf-core/mag                           8  dev        -             -               -
+```
 
 Finally, to return machine-readable JSON output, use the `--json` flag.
 
+## Launch a pipeline
+Some nextflow pipelines have a considerable number of command line flags that can be used.
+To help with this, the `nf-core launch` command uses an interactive command-line wizard tool to prompt you for
+values for running nextflow and the pipeline parameters.
+
+If the pipeline in question has a `parameters.settings.json` file following the [nf-core parameter JSON schema](https://nf-co.re/parameters.schema.json), parameters will be grouped and have associated description text and variable typing.
+
+Nextflow `params` variables are saved in to a JSON file called `nfx-params.json` and used by nextflow with the `-params-file` flag.
+This makes it easier to reuse these in the future.
+
+It is not essential to run the pipeline - the wizard will ask you if you want to launch the command at the end.
+If not, you finish with the `params` JSON file and a nextflow command that you can copy and paste.
+
+```console
+$ nf-core launch rnaseq
+
+                                          ,--./,-.
+          ___     __   __   __   ___     /,-._.--~\
+    |\ | |__  __ /  ` /  \ |__) |__         }  {
+    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
+                                          `._,._,'
+
+
+INFO: Launching nf-core/rnaseq
+Main nextflow options
+
+Config profile to use
+ -profile [standard]: docker
+
+Unique name for this nextflow run
+ -name [None]: test_run
+
+Work directory for intermediate files
+ -w [./work]:
+
+Resume a previous workflow run
+ -resume [y/N]:
+
+Release / revision to use
+ -r [None]: 1.3
+
+
+Parameter group: Main options
+Do you want to change the group's defaults? [y/N]: y
+
+Input files
+Specify the location of your input FastQ files.
+ --reads ['data/*{1,2}.fastq.gz']: '/path/to/reads_*{R1,R2}.fq.gz'
+
+[..truncated..]
+
+Nextflow command:
+  nextflow run nf-core/rnaseq -profile "docker" -name "test_run" -r "1.3" --params-file "/Users/ewels/testing/nfx-params.json"
+
+
+Do you want to run this command now? [y/N]: y
+
+INFO: Launching workflow!
+N E X T F L O W  ~  version 19.01.0
+Launching `nf-core/rnaseq` [evil_engelbart] - revision: 37f260d360 [master]
+
+[..truncated..]
+```
+
 
 ## Downloading pipelines for offline use
 Sometimes you may need to run an nf-core pipeline on a server or HPC system that has no internet connection. In this case you will need to fetch the pipeline files first, then manually transfer them to your system.
@@ -103,7 +211,7 @@ To make this process easier and ensure accurate retrieval of correctly versioned
 
 By default, the pipeline will just download the pipeline code. If you specify the flag `--singularity`, it will also download any singularity image files that are required.
 
-```bash
+```console
 $ nf-core download methylseq --singularity
 
                                           ,--./,-.
@@ -124,7 +232,7 @@ INFO: Downloading 1 singularity container
 nf-core-methylseq-1.0.simg [762.28MB]  [####################################]  780573/780572
 ```
 
-```bash
+```console
 $ tree -L 2 nf-core-methylseq-1.0/
 
 nf-core-methylseq-1.0/
@@ -150,7 +258,7 @@ nf-core-methylseq-1.0/
 ## Pipeline software licences
 Sometimes it's useful to see the software licences of the tools used in a pipeline. You can use the `licences` subcommand to fetch and print the software licence from each conda / PyPI package used in an nf-core pipeline.
 
-```bash
+```console
 $ nf-core licences rnaseq
 
                                           ,--./,-.
@@ -191,7 +299,7 @@ With a given pipeline name, description and author, it makes a starter pipeline
 After creating the files, the command initialises the folder as a git repository and makes an initial commit. This first "vanilla" commit which is identical to the output from the templating tool is important, as it allows us to keep your pipeline in sync with the base template in the future.
 See the [nf-core syncing docs](http://nf-co.re/sync) for more information.
 
-```bash
+```console
 $ nf-core create
 
                                           ,--./,-.
@@ -217,7 +325,7 @@ INFO: Done. Remember to add a remote and push to GitHub:
 Once you have run the command, create a new empty repository on GitHub under your username (not the `nf-core` organisation, yet).
 On your computer, add this repository as a git remote and push to it:
 
-```bash
+```console
 git remote add origin https://github.com/ewels/nf-core-nextbigthing.git
 git push --set-upstream origin master
 ```
@@ -235,7 +343,7 @@ This is the same test that is used on the automated continuous integration tests
 
 For example, the current version looks something like this:
 
-```bash
+```console
 $ cd path/to/my_pipeline
 $ nf-core lint .
 
@@ -268,7 +376,7 @@ The command uses results from the linting process, so will only work with workfl
 
 Usage is `nf-core bump-version <pipeline_dir> <new_version>`, eg:
 
-```bash
+```console
 $ cd path/to/my_pipeline
 $ nf-core bump-version . 1.0
 

diff --git a/bin/blacklist.json b/bin/blacklist.json
@@ -7,7 +7,6 @@
     "exoseq",
     "nascent",
     "neutronstar", 
-    "rnaseq",
     "smrnaseq",
     "vipr"
   ]