diff --git a/CHANGELOG.md b/CHANGELOG.md index 1bb0fbf..d08c71b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +### `Added` + +- `--mzml_convert` parameter to control Thermo `.raw` conversion. Default (unset) auto-selects based on `--diann_version`: converts via ThermoRawFileParser for DIA-NN < 2.1.0, passes `.raw` natively to DIA-NN for >= 2.1.0. Explicit `true` forces conversion (useful for `--mzml_statistics` or to work around DIA-NN Thermo reader issues like [DiaNN#1468](https://github.com/vdemichev/DiaNN/issues/1468)); explicit `false` requires DIA-NN >= 2.1.0 and skips TRFP entirely (closes [#66](https://github.com/bigbio/quantmsdiann/issues/66)). + ## [2.0.0] bigbio/quantmsdiann — Rome - 2026-04-18 ### `Added` diff --git a/docs/parameters.md b/docs/parameters.md index 0844985..53ab84b 100644 --- a/docs/parameters.md +++ b/docs/parameters.md @@ -22,11 +22,12 @@ This document lists every pipeline parameter organised by category. Default valu ## 3. File Preparation (Spectrum Preprocessing) -| Parameter | Type | Default | Description | -| ------------------- | ------- | ------- | ----------------------------------------------------------------------------------------------------------------------------- | -| `--reindex_mzml` | boolean | `true` | Force re-indexing of input mzML files at the start of the pipeline for safety. | -| `--mzml_statistics` | boolean | `false` | Compute MS1/MS2 statistics from mzML files. Generates `*_ms_info.parquet` files for QC. Bruker `.d` files are always skipped. | -| `--mzml_features` | boolean | `false` | Compute MS1-level features during the mzML statistics step. Only available for mzML files. | +| Parameter | Type | Default | Description | +| ------------------- | ------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `--reindex_mzml` | boolean | `true` | Force re-indexing of input mzML files at the start of the pipeline for safety. | +| `--mzml_statistics` | boolean | `false` | Compute MS1/MS2 statistics from mzML files. Generates `*_ms_info.parquet` files for QC. Bruker `.d` files are always skipped. | +| `--mzml_features` | boolean | `false` | Compute MS1-level features during the mzML statistics step. Only available for mzML files. | +| `--mzml_convert` | boolean | _auto_ | Convert Thermo `.raw` to `.mzML` via ThermoRawFileParser before DIA-NN. Unset = auto: convert when DIA-NN < 2.1.0, pass `.raw` through when DIA-NN >= 2.1.0. Set `false` to force native `.raw` (requires DIA-NN >= 2.1.0); set `true` to force conversion (e.g. to enable mzML statistics). | ## 4. Search Parameters diff --git a/docs/usage.md b/docs/usage.md index 421524c..e526e0a 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -22,7 +22,7 @@ The input file must be in [Sample-to-data-relationship format (SDRF)](https://pu The pipeline supports the following mass spectrometry data file formats: -- **`.raw`** - Thermo RAW files (automatically converted to mzML) +- **`.raw`** - Thermo RAW files. Converted to `.mzML` via ThermoRawFileParser for DIA-NN < 2.1.0, passed through natively for DIA-NN >= 2.1.0. Control via `--mzml_convert`. - **`.mzML`** - Open standard mzML files - **`.d`** - Bruker timsTOF files (processed natively by DIA-NN) - **`.dia`** - DIA-NN native binary format (passed through without conversion) @@ -39,6 +39,23 @@ The pipeline includes several preprocessing steps that can be controlled via par - **`--mzml_features`** (default: `false`) -- Compute MS1-level features during the mzML statistics step. Only available for mzML files. +- **`--mzml_convert`** (default: _auto_) -- Controls whether Thermo `.raw` files are converted to `.mzML` via ThermoRawFileParser before being fed to DIA-NN. + + DIA-NN 2.1.0 (2025-03-25) added native Thermo `.raw` support on Linux, so the conversion step is no longer strictly required. Skipping it saves one container invocation and an I/O pass per `.raw` file — non-trivial on Astral-scale datasets. + + | Setting | Behaviour | + | ---------------------- | ---------------------------------------------------------------------------------------------------------------------- | + | unset (default) | Auto: convert via TRFP for DIA-NN < 2.1.0, pass `.raw` through natively for DIA-NN >= 2.1.0. | + | `--mzml_convert true` | Always convert `.raw` to `.mzML` via TRFP. Use this to enable `--mzml_statistics`, or as a workaround for DIA-NN bugs. | + | `--mzml_convert false` | Never convert. Pass `.raw` files straight to DIA-NN. Requires DIA-NN >= 2.1.0 (fails fast otherwise). | + + The parameter has no effect when no `.raw` files are present in the input (e.g. all `.mzML`, `.d`, or `.dia`), or when `--local_input_type mzML` is combined with `--root_folder` so no `.raw` extensions reach the file-preparation branching step — the pipeline will emit a warning in that case. + + > [!WARNING] + > DIA-NN's Linux Thermo reader has known issues on some acquisition schemes / instruments — see [DiaNN#1468](https://github.com/vdemichev/DiaNN/issues/1468) (`Instrument index not available for requested device`) and similar reports. If you hit such an issue, fall back to TRFP conversion with `--mzml_convert true`. + > + > Native `.raw` inputs do not produce `*_ms_info.parquet` QC files; combine `--mzml_convert true` with `--mzml_statistics true` if you need those statistics. + ### Bruker/timsTOF Data For Bruker timsTOF datasets, DIA-NN recommends manually fixing MS1 and MS2 mass accuracy (typically 10-15 ppm) rather than using automatic calibration. There are two ways to set this: diff --git a/lib/VersionUtils.groovy b/lib/VersionUtils.groovy index f340a61..093cab3 100644 --- a/lib/VersionUtils.groovy +++ b/lib/VersionUtils.groovy @@ -31,4 +31,22 @@ class VersionUtils { static boolean versionAtLeast(String version, String required) { return compare(version, required) >= 0 } + + /** + * Minimum DIA-NN version that supports native Linux Thermo .raw reading. + * Used by stageInMode closures in DIA-NN per-file process modules. + */ + static final String NATIVE_RAW_MIN_VERSION = '2.1.0' + + /** + * Returns true when DIA-NN processes should receive .raw files directly + * (without prior ThermoRawFileParser conversion), based on pipeline params. + * + * @param params Nextflow params map (must have diann_version and mzml_convert) + */ + static boolean isNativeRawMode(params) { + if (params.mzml_convert == false) return true + if (params.mzml_convert != null) return false + return versionAtLeast(params.diann_version?.toString() ?: '1.8.1', NATIVE_RAW_MIN_VERSION) + } } diff --git a/modules/local/diann/assemble_empirical_library/main.nf b/modules/local/diann/assemble_empirical_library/main.nf index a1cb2ca..c0c069e 100644 --- a/modules/local/diann/assemble_empirical_library/main.nf +++ b/modules/local/diann/assemble_empirical_library/main.nf @@ -4,6 +4,10 @@ process ASSEMBLE_EMPIRICAL_LIBRARY { label 'diann' label 'error_retry' + // DIA-NN's native Thermo .raw reader fails on symlinked files (Thermo SDK limitation). + // Use 'copy' when .raw files are passed directly to DIA-NN (DIA-NN >= 2.1.0 without TRFP conversion). + stageInMode { VersionUtils.isNativeRawMode(params) ? 'copy' : 'symlink' } + container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ? 'https://containers.biocontainers.pro/s3/SingImgsRepo/diann/v1.8.1_cv1/diann_v1.8.1_cv1.img' : 'docker.io/biocontainers/diann:v1.8.1_cv1' }" diff --git a/modules/local/diann/individual_analysis/main.nf b/modules/local/diann/individual_analysis/main.nf index 814895f..db23607 100644 --- a/modules/local/diann/individual_analysis/main.nf +++ b/modules/local/diann/individual_analysis/main.nf @@ -4,6 +4,10 @@ process INDIVIDUAL_ANALYSIS { label 'diann' label 'error_retry' + // DIA-NN's native Thermo .raw reader fails on symlinked files (Thermo SDK limitation). + // Use 'copy' when .raw files are passed directly to DIA-NN (DIA-NN >= 2.1.0 without TRFP conversion). + stageInMode { VersionUtils.isNativeRawMode(params) ? 'copy' : 'symlink' } + container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ? 'https://containers.biocontainers.pro/s3/SingImgsRepo/diann/v1.8.1_cv1/diann_v1.8.1_cv1.img' : 'docker.io/biocontainers/diann:v1.8.1_cv1' }" diff --git a/modules/local/diann/preliminary_analysis/main.nf b/modules/local/diann/preliminary_analysis/main.nf index c09d41a..6c6b91d 100644 --- a/modules/local/diann/preliminary_analysis/main.nf +++ b/modules/local/diann/preliminary_analysis/main.nf @@ -4,6 +4,10 @@ process PRELIMINARY_ANALYSIS { label 'diann' label 'error_retry' + // DIA-NN's native Thermo .raw reader fails on symlinked files (Thermo SDK limitation). + // Use 'copy' when .raw files are passed directly to DIA-NN (DIA-NN >= 2.1.0 without TRFP conversion). + stageInMode { VersionUtils.isNativeRawMode(params) ? 'copy' : 'symlink' } + container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ? 'https://containers.biocontainers.pro/s3/SingImgsRepo/diann/v1.8.1_cv1/diann_v1.8.1_cv1.img' : 'docker.io/biocontainers/diann:v1.8.1_cv1' }" diff --git a/nextflow.config b/nextflow.config index 0d48e94..f3abd20 100644 --- a/nextflow.config +++ b/nextflow.config @@ -30,6 +30,11 @@ params { reindex_mzml = true mzml_statistics = false mzml_features = false + // Convert Thermo .raw to .mzML via ThermoRawFileParser before feeding DIA-NN. + // null = auto: convert when DIA-NN < 2.1.0 (no native Linux .raw reader), + // pass .raw straight through when DIA-NN >= 2.1.0. + // Set true to force conversion, false to force native .raw (requires DIA-NN >= 2.1.0). + mzml_convert = null // shared search engine parameters met_excision = true // Met-excision is enabled by default diff --git a/nextflow_schema.json b/nextflow_schema.json index 31823d7..307d35d 100644 --- a/nextflow_schema.json +++ b/nextflow_schema.json @@ -115,6 +115,12 @@ "description": "Compute with mzmlstatistics step the features at MS1 level and output to a RAW file, only available for mzML files", "help_text": "Compute with mzmlstatistics step the features at MS1 level and output to a RAW file, only available for mzML files", "fa_icon": "far fa-check-square" + }, + "mzml_convert": { + "type": "boolean", + "description": "Convert Thermo .raw to .mzML via ThermoRawFileParser before feeding DIA-NN. If unset, the pipeline converts when DIA-NN < 2.1.0 and passes .raw files natively when DIA-NN >= 2.1.0.", + "help_text": "DIA-NN 2.1.0 added native Thermo .raw support on Linux, so conversion via ThermoRawFileParser is no longer strictly required. Set `--mzml_convert false` to feed .raw files directly to DIA-NN (saves one container invocation and an I/O pass per file). Set `--mzml_convert true` to force conversion (e.g. for QC / mzML statistics, or to work around DIA-NN Thermo reader issues such as [DiaNN#1468](https://github.com/vdemichev/DiaNN/issues/1468)). The parameter has no effect when inputs are already mzML / .d / .dia or when `--local_input_type mzML` is used with `--root_folder`.", + "fa_icon": "far fa-check-square" } }, "fa_icon": "far fa-chart-bar" diff --git a/subworkflows/local/file_preparation/main.nf b/subworkflows/local/file_preparation/main.nf index 6327cbb..84fff22 100644 --- a/subworkflows/local/file_preparation/main.nf +++ b/subworkflows/local/file_preparation/main.nf @@ -33,6 +33,33 @@ workflow FILE_PREPARATION { ch_versions = ch_versions.mix(DECOMPRESS.out.versions) ch_rawfiles = ch_branched_input.uncompressed.mix(DECOMPRESS.out.decompressed_files) + // + // Resolve whether Thermo .raw should be converted to mzML via TRFP + // or passed natively to DIA-NN. DIA-NN added native Linux .raw support + // in 2.1.0; older versions must go through TRFP. + // + boolean native_raw_supported = VersionUtils.versionAtLeast(params.diann_version, '2.1.0') + boolean convert_raw + if (params.mzml_convert == null) { + convert_raw = !native_raw_supported + } else { + convert_raw = params.mzml_convert as boolean + if (!convert_raw && !native_raw_supported) { + error("--mzml_convert false (native .raw) requires DIA-NN >= 2.1.0. " + + "Current version: ${params.diann_version}. " + + "Use -profile diann_v2_1_0 (or later) or set --mzml_convert true.") + } + } + + // Warn if the user set --mzml_convert but every input is already mzML / .d / .dia + // (i.e. no Thermo .raw will reach TRFP) — including when --local_input_type mzML + // overrides SDRF extensions under --root_folder. + if (params.mzml_convert != null && params.root_folder && params.local_input_type && + params.local_input_type.toString().toLowerCase() != 'raw') { + log.warn "--mzml_convert=${params.mzml_convert} has no effect: " + + "--local_input_type '${params.local_input_type}' under --root_folder means no Thermo .raw files will be processed." + } + // // Divide mzml files ch_rawfiles @@ -76,9 +103,16 @@ workflow FILE_PREPARATION { ch_results = ch_results.mix(ch_branched_input.mzML) } - THERMORAWFILEPARSER( ch_branched_input.raw ) - // Output: spectra (tuple val(meta), path(mzML/mgf/parquet)), log, versions via topic channel - ch_results = ch_results.mix(THERMORAWFILEPARSER.out.spectra) + if (convert_raw) { + // Convert Thermo .raw to .mzML via ThermoRawFileParser (default for DIA-NN < 2.1.0). + THERMORAWFILEPARSER( ch_branched_input.raw ) + // Output: spectra (tuple val(meta), path(mzML/mgf/parquet)), log, versions via topic channel + ch_results = ch_results.mix(THERMORAWFILEPARSER.out.spectra) + } else { + // Pass Thermo .raw straight through to DIA-NN (native reader, DIA-NN >= 2.1.0). + // See https://github.com/vdemichev/DiaNN/issues/1468 for known caveats. + ch_results = ch_results.mix(ch_branched_input.raw) + } ch_results.map{ it -> [it[0], it[1]] }.set{ indexed_mzml_bundle }