Merge pull request #274 from nf-core/dev

Dev > Master for release 1.5

.github/CONTRIBUTING.md

@@ -22,6 +22,21 @@ is as follows:
 
 If you're not used to this workflow with git, you can start with some [basic docs from GitHub](https://help.github.com/articles/fork-a-repo/) or even their [excellent interactive tutorial](https://try.github.io/).
 
+## Style guide
+Google provides an excellent [style guide](https://github.com/google/styleguide/blob/gh-pages/pyguide.md), which
+is an best practise extension of [PEP](https://www.python.org/dev/peps/), the Python Enhancement Proposals. Have a look at the 
+[docstring](https://github.com/google/styleguide/blob/gh-pages/pyguide.md#38-comments-and-docstrings) section, which is in particular
+important, as nf-core tool's code documentation is generated out of these automatically.
+
+In order to test the documentation, you have to install Sphinx on the machine, where the documentation should be generated.
+
+Please follow Sphinx's [installation instruction](http://www.sphinx-doc.org/en/master/usage/installation.html).
+
+Once done, you can run `make clean` and then `make html` in the root directory of `nf-core tools`, where the `Makefile` is located.
+
+The HTML will then be generated in `docs/api/_build/html`.
+
+
 ## Tests
 When you create a pull request with changes, [Travis CI](https://travis-ci.org/) will run automatic tests.
 Typically, pull-requests are only fully reviewed when these tests are passing, though of course we can help out before then.

.github/markdownlint.yml

@@ -0,0 +1,10 @@
+# Markdownlint configuration file
+default: true,
+line-length: false
+no-multiple-blanks: 0
+blanks-around-headers: false
+blanks-around-lists: false
+header-increment: false
+no-duplicate-header:
+    siblings_only: true
+no-bare-urls: false # tools only - the {{ jinja variables }} break URLs and cause this to error

.travis.yml

@@ -3,15 +3,14 @@ language: python
 jdk: openjdk8
 python:
   - '2.7'
-  - '3.4'
   - '3.5'
   - '3.6'
 before_install:
   # PRs to master are only ok if coming from dev branch
   - '[ $TRAVIS_PULL_REQUEST = "false" ] || [ $TRAVIS_BRANCH != "master" ] || ([ $TRAVIS_PULL_REQUEST_SLUG = $TRAVIS_REPO_SLUG ] && [ $TRAVIS_PULL_REQUEST_BRANCH = "dev" ])'
   # Check that the changelog has been updated if this is a PR
   - git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*" && git fetch
-  - '[ $TRAVIS_PULL_REQUEST = "false" ] || [ $(git diff --name-only HEAD origin/${TRAVIS_BRANCH} | grep CHANGELOG.md | wc -l) -eq 1 ]'
+  - '[ $TRAVIS_PULL_REQUEST = "false" ] || [ $(git diff --name-only HEAD origin/${TRAVIS_BRANCH} | grep CHANGELOG.md | wc -l) -ge 1 ]'
 install:
   # Install Nextflow
   - mkdir /tmp/nextflow
@@ -22,15 +21,24 @@ install:
   - cd ${TRAVIS_BUILD_DIR}
   - pip install --upgrade pip # Get rid of dependency resolve issues from older pip versions
   - pip install .
-  - pip install codecov 'pytest==3.6.4' pytest-datafiles pytest-cov mock
+  - pip install codecov 'pytest==3.6.4' pytest-datafiles pytest-cov mock jsonschema
+  - pip install Sphinx sphinxcontrib-napoleon # For autodoc generation
+  - sudo apt-get install npm && npm install -g markdownlint-cli # For documentation linting
 
 script:
-  - python -m pytest --cov=nf_core .
+  - pytest --cov=nf_core .
   - nf-core create -n testpipeline -d "This pipeline is for testing" -a "Testing McTestface"
   - nf-core lint nf-core-testpipeline
+  - markdownlint . -c ${TRAVIS_BUILD_DIR}/.github/markdownlint.yml
 
 after_success:
-  - codecov
+  - codecov --rcfile=${TRAVIS_BUILD_DIR}/.github/coveragerc
+
+jobs:
+  include:
+    - stage: docs generation
+      if: (branch = "master" OR branch = "dev") AND type = push
+      script: bash ./bin/push.sh
 
 deploy:
   provider: pypi

CHANGELOG.md

@@ -1,15 +1,53 @@
 # nf-core/tools: Changelog
 
+## v1.5dev
+
+#### Template pipeline
+* Dropped Singularity file
+* Summary now logs details of the cluster profile used if from [nf-core/configs](https://github.com/nf-core/configs)
+* Dockerhub is used in favor of Singularity Hub for pulling when using the Singularity profile
+* Changed default container tag from latest to dev
+* Brought the logo to life
+* Change the default filenames for the pipeline trace files
+* Remote fetch of nf-core/configs profiles fails gracefully if offline
+* Remove `process.container` and just directly define `process.container` now.
+
+#### Tools helper code
+* New `nf-core launch` command to interactively launch nf-core pipelines from command-line
+  * Works with a `parameters.settings.json` file shipped with each pipeline
+  * Discovers additional `params` from the pipeline dynamically
+* Drop Python 3.4 support
+* `nf-core list` now only shows a value for _"is local latest version"_ column if there is a local copy.
+* Lint markdown formatting in automated tests
+  * Added `markdownlint-cli` for checking Markdown syntax in pipelines and tools repo
+* Syncing now reads from a `blacklist.json` in order to exclude pipelines from being synced if necessary.
+* Added nf-core tools API description to assist developers with the classes and functions available.
+  * Docs are automatically built by Travis CI and updated on the nf-co.re website.
+* Introduced test for filtering remote workflows by keyword.
+* Build tools python API docs
+  * Use Travis job for api doc generation and publish
+
+* `nf-core bump-version` now stops before making changes if the linting fails
+* Code test coverage
+  * Introduced test for filtering remote workflows by keyword
+* Linting updates
+  * Now properly searches for conda packages in default channels
+  * Now correctly validates version pinning for packages from PyPI
+  * Updates for changes to `process.container` definition
+
+#### Other
+* Bump `conda` to 4.6.7 in base nf-core Dockerfile
+
 ## [v1.4](https://github.com/nf-core/tools/releases/tag/1.4) - 2018-12-12 Tantalum Butterfly
 
 #### Template pipeline
 * Institutional custom config profiles moved to github `nf-core/configs`
-    * These will now be maintained centrally as opposed to being shipped with the pipelines in `conf/`
-    * Load `base.config` by default for all profiles
-    * Removed profiles named `standard` and `none`
-    * Added parameter `--igenomesIgnore` so `igenomes.config` is not loaded if parameter clashes are observed
-    * Added parameter `--custom_config_version` for custom config version control. Can use this parameter to provide commit id for reproducibility. Defaults to `master`
-    * Deleted custom configs from template in `conf/` directory i.e. `uzh.config`, `binac.config` and `cfc.config`
+  * These will now be maintained centrally as opposed to being shipped with the pipelines in `conf/`
+  * Load `base.config` by default for all profiles
+  * Removed profiles named `standard` and `none`
+  * Added parameter `--igenomesIgnore` so `igenomes.config` is not loaded if parameter clashes are observed
+  * Added parameter `--custom_config_version` for custom config version control. Can use this parameter to provide commit id for reproducibility. Defaults to `master`
+  * Deleted custom configs from template in `conf/` directory i.e. `uzh.config`, `binac.config` and `cfc.config`
 * `multiqc_config` and `output_md` are now put into channels instead of using the files directly (see issue [#222](https://github.com/nf-core/tools/issues/222))
 * Added `local.md` to cookiecutter template in `docs/configuration/`. This was referenced in `README.md` but not present.
 * Major overhaul of docs to add/remove parameters, unify linking of files and added description for providing custom configs where necessary
@@ -25,8 +63,8 @@
 
 ## [v1.3](https://github.com/nf-core/tools/releases/tag/1.3) - 2018-11-21
 * `nf-core create` command line interface updated
-    * Interactive prompts for required arguments if not given
-    * New flag for workflow author
+  * Interactive prompts for required arguments if not given
+  * New flag for workflow author
 * Updated channel order for bioconda/conda-forge channels in environment.yaml
 * Increased code coverage for sub command `create` and `licenses`
 * Fixed nasty dependency hell issue between `pytest` and `py` package in Python 3.4.x
@@ -38,18 +76,18 @@
 
 ## [v1.2](https://github.com/nf-core/tools/releases/tag/1.2) - 2018-10-01
 * Updated the `nf-core release` command
-    * Now called `nf-core bump-versions` instead
-    * New flag `--nextflow` to change the required nextflow version instead
+  * Now called `nf-core bump-versions` instead
+  * New flag `--nextflow` to change the required nextflow version instead
 * Template updates
-    * Simpler installation of the `nf-core` helper tool, now directly from PyPI
-    * Bump minimum nextflow version to `0.32.0` - required for built in `manifest.nextflowVersion` check and access to `workflow.manifest` variables from within nextflow scripts
-    * New `withName` syntax for configs
-    * Travis tests fail if PRs come against the `master` branch, slightly refactored
-    * Improved GitHub contributing instructions and pull request / issue templates
+  * Simpler installation of the `nf-core` helper tool, now directly from PyPI
+  * Bump minimum nextflow version to `0.32.0` - required for built in `manifest.nextflowVersion` check and access to `workflow.manifest` variables from within nextflow scripts
+  * New `withName` syntax for configs
+  * Travis tests fail if PRs come against the `master` branch, slightly refactored
+  * Improved GitHub contributing instructions and pull request / issue templates
 * New lint tests
-    * `.travis.yml` test for PRs made against the `master` branch
-    * Automatic `--release` option not used if the travis repo is `nf-core/tools`
-    * Warnings if depreciated variables `params.version` and `params.nf_required_version` are found
+  * `.travis.yml` test for PRs made against the `master` branch
+  * Automatic `--release` option not used if the travis repo is `nf-core/tools`
+  * Warnings if depreciated variables `params.version` and `params.nf_required_version` are found
 * New `nf-core licences` subcommand to show licence for each conda package in a workflow
 * `nf-core list` now has options for sorting pipeline nicely
 * Latest version of conda used in nf-core base docker image
@@ -60,20 +98,20 @@
 Very large release containing lots of work from the first nf-core hackathon, held in SciLifeLab Stockholm.
 
 * The [Cookiecutter template](https://github.com/nf-core/cookiecutter) has been merged into tools
-    * The old repo above has been archived
-    * New pipelines are now created using the command `nf-core create`
-    * The nf-core template and associated linting are now controlled under the same version system
+  * The old repo above has been archived
+  * New pipelines are now created using the command `nf-core create`
+  * The nf-core template and associated linting are now controlled under the same version system
 * Large number of template updates and associated linting changes
-    * New simplified cookicutter variable usage
-    * Refactored documentation - simplified and reduced duplication
-    * Better `manifest` variables instead of `params` for pipeline name and version
-    * New integrated nextflow version checking
-    * Updated travis docker pull command to use tagging to allow release tests to pass
-    * Reverted Docker and Singularity syntax to use `ENV` hack again
+  * New simplified cookicutter variable usage
+  * Refactored documentation - simplified and reduced duplication
+  * Better `manifest` variables instead of `params` for pipeline name and version
+  * New integrated nextflow version checking
+  * Updated travis docker pull command to use tagging to allow release tests to pass
+  * Reverted Docker and Singularity syntax to use `ENV` hack again
 * Improved Python readme parsing for PyPI
 * Updated Travis tests to check that the correct `dev` branch is being targeted
 * New sync tool to automate pipeline updates
-    * Once initial merges are complete, a nf-core bot account will create PRs for future template updates
+  * Once initial merges are complete, a nf-core bot account will create PRs for future template updates
 
 ## [v1.0.1](https://github.com/nf-core/tools/releases/tag/1.0.1) - 2018-07-18
 

Dockerfile

@@ -4,4 +4,4 @@ LABEL authors="[email protected],[email protected]"
 
 # Install procps so that Nextflow can poll CPU usage
 RUN apt-get update && apt-get install -y procps && apt-get clean -y 
-RUN conda install conda=4.5.11
+RUN conda install conda=4.6.7

README.md

@@ -1,6 +1,6 @@
-<img src="docs/images/nf-core-logo.png" width="400">
+# ![nf-core/tools](docs/images/nf-core-logo.png)
 
-# [nf-core/tools](https://github.com/nf-core/tools)
+## [nf-core/tools](https://github.com/nf-core/tools)
 [![Build Status](https://travis-ci.org/nf-core/tools.svg?branch=master)](https://travis-ci.org/nf-core/tools)
 [![codecov](https://codecov.io/gh/nf-core/tools/branch/master/graph/badge.svg)](https://codecov.io/gh/nf-core/tools)
 [![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg?style=flat-square)](http://bioconda.github.io/recipes/nf-core/README.html)
@@ -21,13 +21,13 @@ A python package with helper tools for the nf-core community.
 
 You can install `nf-core/tools` from [PyPI](https://pypi.python.org/pypi/nf-core/) using pip as follows:
 
-```
+```bash
 pip install nf-core
 ```
 
 If you would like the development version instead, the command is:
 
-```
+```bash
 pip install --upgrade --force-reinstall git+https://github.com/nf-core/tools.git
 ```
 
@@ -50,7 +50,7 @@ The command `nf-core list` shows all available nf-core pipelines along with thei
 
 An example of the output from the command is as follows:
 
-```
+```bash
 $ nf-core list
 
                                           ,--./,-.
@@ -75,7 +75,7 @@ nf-core/vipr       dev        -            -              No
 
 To narrow down the list, supply one or more additional keywords to filter the pipelines based on matches in titles, descriptions and topics:
 
-```
+```bash
 nf-core list rna rna-seq
 
                                           ,--./,-.
@@ -103,7 +103,7 @@ To make this process easier and ensure accurate retrieval of correctly versioned
 
 By default, the pipeline will just download the pipeline code. If you specify the flag `--singularity`, it will also download any singularity image files that are required.
 
-```
+```bash
 $ nf-core download methylseq --singularity
 
                                           ,--./,-.
@@ -123,7 +123,8 @@ INFO: Downloading workflow files from GitHub
 INFO: Downloading 1 singularity container
 nf-core-methylseq-1.0.simg [762.28MB]  [####################################]  780573/780572
 ```
-```
+
+```bash
 $ tree -L 2 nf-core-methylseq-1.0/
 
 nf-core-methylseq-1.0/
@@ -149,7 +150,7 @@ nf-core-methylseq-1.0/
 ## Pipeline software licences
 Sometimes it's useful to see the software licences of the tools used in a pipeline. You can use the `licences` subcommand to fetch and print the software licence from each conda / PyPI package used in an nf-core pipeline.
 
-```
+```bash
 $ nf-core licences rnaseq
 
                                           ,--./,-.
@@ -190,7 +191,7 @@ With a given pipeline name, description and author, it makes a starter pipeline
 After creating the files, the command initialises the folder as a git repository and makes an initial commit. This first "vanilla" commit which is identical to the output from the templating tool is important, as it allows us to keep your pipeline in sync with the base template in the future.
 See the [nf-core syncing docs](http://nf-co.re/sync) for more information.
 
-```
+```bash
 $ nf-core create
 
                                           ,--./,-.
@@ -234,7 +235,7 @@ This is the same test that is used on the automated continuous integration tests
 
 For example, the current version looks something like this:
 
-```
+```bash
 $ cd path/to/my_pipeline
 $ nf-core lint .
 
@@ -256,7 +257,7 @@ WARNING: Test Warnings:
   http://nf-co.re/errors#8: Conda package is not latest available: bwameth=0.2.0, 0.2.1 available
 ```
 
-You can find extensive documentation about each of the lint tests in the [lint errors documentation](docs/lint_errors.md).
+You can find extensive documentation about each of the lint tests in the [lint errors documentation](https://nf-co.re/errors).
 
 
 ## Bumping a pipeline version number
@@ -267,7 +268,7 @@ The command uses results from the linting process, so will only work with workfl
 
 Usage is `nf-core bump-version <pipeline_dir> <new_version>`, eg:
 
-```
+```bash
 $ cd path/to/my_pipeline
 $ nf-core bump-version . 1.0
 
@@ -295,7 +296,7 @@ INFO: Updating version in nextflow.config
  + version = '1.0'
 
 INFO: Updating version in nextflow.config
- - container = 'nfcore/mypipeline:latest'
+ - container = 'nfcore/mypipeline:dev'
  + container = 'nfcore/mypipeline:1.0'
 
 INFO: Updating version in .travis.yml

bin/blacklist.json

@@ -0,0 +1,14 @@
+{
+  "pipelines": [
+    "atacseq",
+    "bcellmagic",
+    "chipseq",
+    "epitopeprediction",
+    "exoseq",
+    "nascent",
+    "neutronstar", 
+    "rnaseq",
+    "smrnaseq",
+    "vipr"
+  ]
+}

bin/push.sh

@@ -0,0 +1,25 @@
+#!/bin/sh
+
+create_api_doc() {
+    make --directory ./docs/api html
+}
+
+remove_source_files() {
+    git rm -r --cache .
+}
+
+commit_website_files() {
+  git checkout --orphan api-doc
+  create_api_doc
+  remove_source_files
+  git add docs
+  git commit --message "Travis build: $TRAVIS_BUILD_NUMBER"
+}
+
+upload_files() {
+  git remote add nf-core "https://$NF_CORE_BOT@github.com/nf-core/tools.git" > /dev/null 2>&1
+  git push --force --quiet --set-upstream nf-core api-doc
+}
+
+commit_website_files
+upload_files