Add individual tasks input files to TaskArray

[chris-mbiomics]

Update TaskArrayCollector.groovy, fixes #5701

Making this a draft PR since I don't know if this is the best way

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	b43785b
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/67a0aba7477ce10008f728f6

[chris-mbiomics]

@pditommaso Since you were able to fix the other task array issue so quickly, may I ask you for your feedback on this?

[jorgee]

I have reproduced the error and checked that the proposed fix is working. The solution looks fine but it could also be implemented in the taskbean as done for the working dir. You would not need to create the input parameters, just add the paths. The pipeline suggested to test could be added as validation test. @pditommaso do you think it is needed?

[pditommaso]

Yeah, the TaskBean looks a better solution

[chris-mbiomics]

I agree with that assessment. Just got to know the nextflow codebase, thank you for your feedback!

I'll change my PR.

Regarding a test: I would feel better if the test bucket was not owned by my company, but also owned by seqera/nextflow. Do you have a second bucket to test with? Then I can also adapt one of the tests.

[pditommaso]

You can use this bucket

nextflow/validation/google.config

Line 11 in 0a29236

workDir = 'gs://rnaseq-nf/scratch'

[bentsherman]

Even simpler, just override getInputFilesMap() in TaskArrayRun:

    @Override
    Map<String,Path> getInputFilesMap() {
        Map<String,Path> result = [:]
        for( final handler : children )
            result.putAll(handler.task.getInputFilesMap())
        return result
    }

Everything else should work out-of-the-box. My only concern is that if two tasks stage an input file with the same name, they will clobber each other here.

We might need to refactor this code here:

nextflow/plugins/nf-google/src/main/nextflow/cloud/google/batch/GoogleBatchScriptLauncher.groovy

Lines 73 to 76 in 039c8f3

	// remap input files to container mounted paths
	for( Map.Entry<String,Path> entry : new HashMap<>(bean.inputFiles).entrySet() ) {
	bean.inputFiles.put( entry.key, toContainerMount(entry.value, true) )
	}

When the task is an array, we don't need to set the bean input files, we just need to call toContainerMount() on all of the input file paths. But the problem is that at this point we no longer have the TaskRun.

Should we just add another field to the TaskBean?

List<Path> arrayInputFiles

[chris-mbiomics]

just switched out the branches to use the taskbean based implementation suggested above. I think that's the easiest – since it's a list it doesn't have issues with multiple files with the same names from different tasks colliding and replacing each other.
It also follows the same pattern as with the working directory.

@pditommaso For the test case, I would need a second, different bucket, which would not be host to the workDir

[chris-mbiomics]

Is this an issue with aws batch aswell? I didn't touch that code because the TaskBeans arrayWorkDirs wasn't used by aws batch

[pditommaso]

I'd suggest @jorgee to take over this and review all cases

[bentsherman]

AWS Batch should not need to be changed because it does not need the bucket mounts. Google Batch needs them only because it uses gcsfuse by default to access the inputs.

Fusion is technically doing a similar thing to Google Batch:

nextflow/modules/nextflow/src/main/groovy/nextflow/fusion/FusionScriptLauncher.groovy

Lines 56 to 59 in 0a29236

	// remap input files to container mounted paths
	for( Map.Entry<String,Path> entry : new HashMap<>(bean.inputFiles).entrySet() ) {
	bean.inputFiles.put( entry.key, toContainerMount(entry.value, scheme) )
	}

But I don't think the array parent needs the input files for Fusion. They are only used to populate the staging commands in the wrapper script, which the array job doesn't need.

So in summary, this new arrayInputFiles should only be needed for Google Batch on gcsfuse.

[pditommaso]

Closing in favour of #5739