.command.run permission denied in 3.16.1, 3.17.0, 3.18.0

[givtrah]

Description of the bug

New versions of rnaseq pipeline fails when running the test pipeline under docker (and singularity, conda was not tested). 3.16.1, 3.17.0 and 3.18.0 is affected. 3.16.0 and 3.14.0 tested fine.

Command used and terminal output

run nf-core/rnaseq -r 3.16.0 -profile test,docker --outdir testout

Works

run nf-core/rnaseq -r 3.16.1 -profile test,docker --outdir testout

not working (3.17.0 and 3.18.0 gives the same error):

-[nf-core/rnaseq] Pipeline completed with errors-
WARN: Directive `process.shell` cannot contain new-line characters - offending value: [bash

set -e # Exit if a tool returns a non-zero status/exit code
set -u # Treat unset variables and parameters as an error
set -o pipefail # Returns the status of the last command to exit with a non-zero status or zero if all successfully execute
]
ERROR ~ Error executing process > 'NFCORE_RNASEQ:PREPARE_GENOME:GUNZIP_ADDITIONAL_FASTA (gfp.fa.gz)'

Caused by:
  Process `NFCORE_RNASEQ:PREPARE_GENOME:GUNZIP_ADDITIONAL_FASTA (gfp.fa.gz)` terminated with an error exit status (126)


Command executed:

  # Not calling gunzip itself because it creates files
  # with the original group ownership rather than the
  # default one for that user / the work directory
  gzip \
      -cd \
       \
      gfp.fa.gz \
      > gfp.fa
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_RNASEQ:PREPARE_GENOME:GUNZIP_ADDITIONAL_FASTA":
      gunzip: $(echo $(gunzip --version 2>&1) | sed 's/^.*(gzip) //; s/ Copyright.*$//')
  END_VERSIONS

Command exit status:
  126

Command output:
  (empty)

Command error:
  /bin/bash: line 10: .command.run: Permission denied

Work dir:
  /home/givtrah/test/work/f5/a89cb7f2a3d6a658b5cdc768a35afe

Container:
  quay.io/nf-core/ubuntu:22.04

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

 -- Check '.nextflow.log' file for details
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting

 -- Check '.nextflow.log' file for details

Relevant files

No response

System information

No response

[lisch7]

Same Issue. I thought it was an internet problem, but it looks like someone else is having the same problem I am!

[givtrah]

I spent two days trying to figure out where my permission problem was, all to no avail until I recalled last time I ran it just fine using a previous version and... voila (also, great fun first troubleshooting that sample names for some reason can no longer be numeric when the error said something about a space in the filename.. sigh).

Of course it could still be specific to my setup, so good to hear I'm not alone in having this problem.

[lisch7]

I spent two days trying to figure out where my permission problem was, all to no avail until I recalled last time I ran it just fine using a previous version and... voila (also, great fun first troubleshooting that sample names for some reason can no longer be numeric when the error said something about a space in the filename.. sigh).

Of course it could still be specific to my setup, so good to hear I'm not alone in having this problem.

Our experience is very similar, I have no problem at all if I specify version 3.16.0. Also, I have installed other piplines such as fetchings, sarek, differntialabundance,all without problems. But it's worth mentioning that I've installed 3.18.0 successfully before, but because of a slight problem with docker, I deleted all the containers, but it didn't work when I pulled it again

[fbartusch]

A user on our HPC-cluster has the same problem. I also thought our setup was the problem until I found this issue.

[sebastiantonn]

Same issue.
3.17.0 gives error like above, 3.16.0 seems to work.
Did not test other versions.

[MatthiasZepper]

It is the first time that I hear about all those issues, but the

Directive process.shell cannot contain new-line characters - offending value: [bash

part of your error message piqued my interest. Searching for issues in the nf-core organization quickly brought up this template fix by @ewels and an associated Nextflow issue.

Admittedly, I do not know what I can recommend to you to act on the problem. Since a change in the pipeline template was required, once probably has to consider these pipeline versions incompatible with certain Nextflow versions?

But maybe just providing a custom config with

// Set bash options
process.shell = [
    "bash",
    "-C",         // No clobber - prevent output redirection from overwriting files.
    "-e",         // Exit if a tool returns a non-zero status/exit code
    "-u",         // Treat unset variables and parameters as an error
    "-o",         // Returns the status of the last command to exit..
    "pipefail"    //   ..with a non-zero status or zero if all successfully execute
]

to nextflow run suffices. Not tested, though...

I think, this warrants a blog post from the nf-core maintainers once there is a good solution in place.

[ewels]

It looks like the dev branch of this pipeline already has the fix:

rnaseq/nextflow.config

Lines 307 to 315 in 15c160f

	// Set bash options
	process.shell = [
	"bash",
	"-C", // No clobber - prevent output redirection from overwriting files.
	"-e", // Exit if a tool returns a non-zero status/exit code
	"-u", // Treat unset variables and parameters as an error
	"-o", // Returns the status of the last command to exit..
	"pipefail" // ..with a non-zero status or zero if all successfully execute
	]

So if you run the pipeline with -r dev it'll probably work. But we should prioritise a release ASAP.

This particular error only shows up when several specific criteria are met:

• Running with Nextflow 24.12.0-edge or later (so likely coming up now because of the 25.04 stable release)
• Pipeline has nf-core template update >= 3.0.0 and < 3.2.0

Fixes are:

• Use an older version of Nextflow
• Use an older version of the pipeline
• Release a new version of the pipeline with the included fix

An explanation with more detail of what's going on can be found here: nf-core/tools#3416 (comment)

[ewels]

oh, and can also overwrite the above syntax with a custom config too - probably the easiest fix!

Untested, but:

nf_config_fix.config

 // Set bash options 
 process.shell = [ 
     "bash", 
     "-C",         // No clobber - prevent output redirection from overwriting files. 
     "-e",         // Exit if a tool returns a non-zero status/exit code 
     "-u",         // Treat unset variables and parameters as an error 
     "-o",         // Returns the status of the last command to exit.. 
     "pipefail"    //   ..with a non-zero status or zero if all successfully execute 
 ] 

nextflow run nf-core/rnaseq -c nf_config_fix.config ...

[givtrah]

Thanks for all the suggestions and for getting back to me about this bug.

I can confirm that running the dev branch using -r dev works for both docker and singularity containers.

For now I just used 3.16.0.

[arinwongprommoon]

@ewels Just commenting to say thank you for suggesting a custom config (in #1554 (comment)).

I came across a very similar issue with a different Nextflow-based pipeline (avantonder/assembleBAC#28), and appending my current config with your text seems to have gotten me past the error. I was using Nextflow 24.10.5.

Hopefully this is helpful, at least in letting people know how widespread the issue is.

[ewels]

Yup, looks like that pipeline was originally based on the nf-core template. So even though it's not an nf-core pipeline, it's the same situation.

[nttg8100]

I think the unhappy test cases for this feature is missing when the process.shell is provided via config. Hope to see new test cases to cover this on new release.
https://github.com/nextflow-io/nextflow/blob/9e675c6aba28f5b2a76217b1d97e4ef5da4e6f81/modules/nextflow/src/main/groovy/nextflow/processor/TaskConfig.groovy#L392C4-L406C6
https://github.com/nextflow-io/nextflow/blob/9e675c6aba28f5b2a76217b1d97e4ef5da4e6f81/modules/nextflow/src/test/groovy/nextflow/processor/TaskConfigTest.groovy#L35C5-L53C6