Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,12 @@
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### `Added`

- `--mzml_convert` parameter to control Thermo `.raw` conversion. Default (unset) auto-selects based on `--diann_version`: converts via ThermoRawFileParser for DIA-NN < 2.1.0, passes `.raw` natively to DIA-NN for >= 2.1.0. Explicit `true` forces conversion (useful for `--mzml_statistics` or to work around DIA-NN Thermo reader issues like [DiaNN#1468](https://github.com/vdemichev/DiaNN/issues/1468)); explicit `false` requires DIA-NN >= 2.1.0 and skips TRFP entirely (closes [#66](https://github.com/bigbio/quantmsdiann/issues/66)).

## [2.0.0] bigbio/quantmsdiann — Rome - 2026-04-18

### `Added`
Expand Down
11 changes: 6 additions & 5 deletions docs/parameters.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,12 @@ This document lists every pipeline parameter organised by category. Default valu

## 3. File Preparation (Spectrum Preprocessing)

| Parameter | Type | Default | Description |
| ------------------- | ------- | ------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `--reindex_mzml` | boolean | `true` | Force re-indexing of input mzML files at the start of the pipeline for safety. |
| `--mzml_statistics` | boolean | `false` | Compute MS1/MS2 statistics from mzML files. Generates `*_ms_info.parquet` files for QC. Bruker `.d` files are always skipped. |
| `--mzml_features` | boolean | `false` | Compute MS1-level features during the mzML statistics step. Only available for mzML files. |
| Parameter | Type | Default | Description |
| ------------------- | ------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--reindex_mzml` | boolean | `true` | Force re-indexing of input mzML files at the start of the pipeline for safety. |
| `--mzml_statistics` | boolean | `false` | Compute MS1/MS2 statistics from mzML files. Generates `*_ms_info.parquet` files for QC. Bruker `.d` files are always skipped. |
| `--mzml_features` | boolean | `false` | Compute MS1-level features during the mzML statistics step. Only available for mzML files. |
| `--mzml_convert` | boolean | _auto_ | Convert Thermo `.raw` to `.mzML` via ThermoRawFileParser before DIA-NN. Unset = auto: convert when DIA-NN < 2.1.0, pass `.raw` through when DIA-NN >= 2.1.0. Set `false` to force native `.raw` (requires DIA-NN >= 2.1.0); set `true` to force conversion (e.g. to enable mzML statistics). |

## 4. Search Parameters

Expand Down
19 changes: 18 additions & 1 deletion docs/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ The input file must be in [Sample-to-data-relationship format (SDRF)](https://pu

The pipeline supports the following mass spectrometry data file formats:

- **`.raw`** - Thermo RAW files (automatically converted to mzML)
- **`.raw`** - Thermo RAW files. Converted to `.mzML` via ThermoRawFileParser for DIA-NN < 2.1.0, passed through natively for DIA-NN >= 2.1.0. Control via `--mzml_convert`.
- **`.mzML`** - Open standard mzML files
- **`.d`** - Bruker timsTOF files (processed natively by DIA-NN)
- **`.dia`** - DIA-NN native binary format (passed through without conversion)
Expand All @@ -39,6 +39,23 @@ The pipeline includes several preprocessing steps that can be controlled via par

- **`--mzml_features`** (default: `false`) -- Compute MS1-level features during the mzML statistics step. Only available for mzML files.

- **`--mzml_convert`** (default: _auto_) -- Controls whether Thermo `.raw` files are converted to `.mzML` via ThermoRawFileParser before being fed to DIA-NN.

DIA-NN 2.1.0 (2025-03-25) added native Thermo `.raw` support on Linux, so the conversion step is no longer strictly required. Skipping it saves one container invocation and an I/O pass per `.raw` file — non-trivial on Astral-scale datasets.

| Setting | Behaviour |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| unset (default) | Auto: convert via TRFP for DIA-NN < 2.1.0, pass `.raw` through natively for DIA-NN >= 2.1.0. |
| `--mzml_convert true` | Always convert `.raw` to `.mzML` via TRFP. Use this to enable `--mzml_statistics`, or as a workaround for DIA-NN bugs. |
| `--mzml_convert false` | Never convert. Pass `.raw` files straight to DIA-NN. Requires DIA-NN >= 2.1.0 (fails fast otherwise). |

The parameter has no effect when no `.raw` files are present in the input (e.g. all `.mzML`, `.d`, or `.dia`), or when `--local_input_type mzML` is combined with `--root_folder` so no `.raw` extensions reach the file-preparation branching step — the pipeline will emit a warning in that case.

> [!WARNING]
> DIA-NN's Linux Thermo reader has known issues on some acquisition schemes / instruments — see [DiaNN#1468](https://github.com/vdemichev/DiaNN/issues/1468) (`Instrument index not available for requested device`) and similar reports. If you hit such an issue, fall back to TRFP conversion with `--mzml_convert true`.
>
> Native `.raw` inputs do not produce `*_ms_info.parquet` QC files; combine `--mzml_convert true` with `--mzml_statistics true` if you need those statistics.

### Bruker/timsTOF Data

For Bruker timsTOF datasets, DIA-NN recommends manually fixing MS1 and MS2 mass accuracy (typically 10-15 ppm) rather than using automatic calibration. There are two ways to set this:
Expand Down
18 changes: 18 additions & 0 deletions lib/VersionUtils.groovy
Original file line number Diff line number Diff line change
Expand Up @@ -31,4 +31,22 @@ class VersionUtils {
static boolean versionAtLeast(String version, String required) {
return compare(version, required) >= 0
}

/**
* Minimum DIA-NN version that supports native Linux Thermo .raw reading.
* Used by stageInMode closures in DIA-NN per-file process modules.
*/
static final String NATIVE_RAW_MIN_VERSION = '2.1.0'

/**
* Returns true when DIA-NN processes should receive .raw files directly
* (without prior ThermoRawFileParser conversion), based on pipeline params.
*
* @param params Nextflow params map (must have diann_version and mzml_convert)
*/
static boolean isNativeRawMode(params) {
if (params.mzml_convert == false) return true
if (params.mzml_convert != null) return false
return versionAtLeast(params.diann_version?.toString() ?: '1.8.1', NATIVE_RAW_MIN_VERSION)
}
}
4 changes: 4 additions & 0 deletions modules/local/diann/assemble_empirical_library/main.nf
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ process ASSEMBLE_EMPIRICAL_LIBRARY {
label 'diann'
label 'error_retry'

// DIA-NN's native Thermo .raw reader fails on symlinked files (Thermo SDK limitation).
// Use 'copy' when .raw files are passed directly to DIA-NN (DIA-NN >= 2.1.0 without TRFP conversion).
stageInMode { VersionUtils.isNativeRawMode(params) ? 'copy' : 'symlink' }

container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
'https://containers.biocontainers.pro/s3/SingImgsRepo/diann/v1.8.1_cv1/diann_v1.8.1_cv1.img' :
'docker.io/biocontainers/diann:v1.8.1_cv1' }"
Expand Down
4 changes: 4 additions & 0 deletions modules/local/diann/individual_analysis/main.nf
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ process INDIVIDUAL_ANALYSIS {
label 'diann'
label 'error_retry'

// DIA-NN's native Thermo .raw reader fails on symlinked files (Thermo SDK limitation).
// Use 'copy' when .raw files are passed directly to DIA-NN (DIA-NN >= 2.1.0 without TRFP conversion).
stageInMode { VersionUtils.isNativeRawMode(params) ? 'copy' : 'symlink' }

container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
'https://containers.biocontainers.pro/s3/SingImgsRepo/diann/v1.8.1_cv1/diann_v1.8.1_cv1.img' :
'docker.io/biocontainers/diann:v1.8.1_cv1' }"
Expand Down
4 changes: 4 additions & 0 deletions modules/local/diann/preliminary_analysis/main.nf
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ process PRELIMINARY_ANALYSIS {
label 'diann'
label 'error_retry'

// DIA-NN's native Thermo .raw reader fails on symlinked files (Thermo SDK limitation).
// Use 'copy' when .raw files are passed directly to DIA-NN (DIA-NN >= 2.1.0 without TRFP conversion).
stageInMode { VersionUtils.isNativeRawMode(params) ? 'copy' : 'symlink' }

container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
'https://containers.biocontainers.pro/s3/SingImgsRepo/diann/v1.8.1_cv1/diann_v1.8.1_cv1.img' :
'docker.io/biocontainers/diann:v1.8.1_cv1' }"
Expand Down
5 changes: 5 additions & 0 deletions nextflow.config
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,11 @@ params {
reindex_mzml = true
mzml_statistics = false
mzml_features = false
// Convert Thermo .raw to .mzML via ThermoRawFileParser before feeding DIA-NN.
// null = auto: convert when DIA-NN < 2.1.0 (no native Linux .raw reader),
// pass .raw straight through when DIA-NN >= 2.1.0.
// Set true to force conversion, false to force native .raw (requires DIA-NN >= 2.1.0).
mzml_convert = null

// shared search engine parameters
met_excision = true // Met-excision is enabled by default
Expand Down
6 changes: 6 additions & 0 deletions nextflow_schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,12 @@
"description": "Compute with mzmlstatistics step the features at MS1 level and output to a RAW file, only available for mzML files",
"help_text": "Compute with mzmlstatistics step the features at MS1 level and output to a RAW file, only available for mzML files",
"fa_icon": "far fa-check-square"
},
"mzml_convert": {
"type": "boolean",
"description": "Convert Thermo .raw to .mzML via ThermoRawFileParser before feeding DIA-NN. If unset, the pipeline converts when DIA-NN < 2.1.0 and passes .raw files natively when DIA-NN >= 2.1.0.",
"help_text": "DIA-NN 2.1.0 added native Thermo .raw support on Linux, so conversion via ThermoRawFileParser is no longer strictly required. Set `--mzml_convert false` to feed .raw files directly to DIA-NN (saves one container invocation and an I/O pass per file). Set `--mzml_convert true` to force conversion (e.g. for QC / mzML statistics, or to work around DIA-NN Thermo reader issues such as [DiaNN#1468](https://github.com/vdemichev/DiaNN/issues/1468)). The parameter has no effect when inputs are already mzML / .d / .dia or when `--local_input_type mzML` is used with `--root_folder`.",
"fa_icon": "far fa-check-square"
}
},
"fa_icon": "far fa-chart-bar"
Expand Down
40 changes: 37 additions & 3 deletions subworkflows/local/file_preparation/main.nf
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,33 @@ workflow FILE_PREPARATION {
ch_versions = ch_versions.mix(DECOMPRESS.out.versions)
ch_rawfiles = ch_branched_input.uncompressed.mix(DECOMPRESS.out.decompressed_files)

//
// Resolve whether Thermo .raw should be converted to mzML via TRFP
// or passed natively to DIA-NN. DIA-NN added native Linux .raw support
// in 2.1.0; older versions must go through TRFP.
//
boolean native_raw_supported = VersionUtils.versionAtLeast(params.diann_version, '2.1.0')
boolean convert_raw
if (params.mzml_convert == null) {
convert_raw = !native_raw_supported
} else {
convert_raw = params.mzml_convert as boolean
if (!convert_raw && !native_raw_supported) {
error("--mzml_convert false (native .raw) requires DIA-NN >= 2.1.0. " +
"Current version: ${params.diann_version}. " +
"Use -profile diann_v2_1_0 (or later) or set --mzml_convert true.")
}
}

// Warn if the user set --mzml_convert but every input is already mzML / .d / .dia
// (i.e. no Thermo .raw will reach TRFP) — including when --local_input_type mzML
// overrides SDRF extensions under --root_folder.
if (params.mzml_convert != null && params.root_folder && params.local_input_type &&
params.local_input_type.toString().toLowerCase() != 'raw') {
log.warn "--mzml_convert=${params.mzml_convert} has no effect: " +
"--local_input_type '${params.local_input_type}' under --root_folder means no Thermo .raw files will be processed."
}

//
// Divide mzml files
ch_rawfiles
Expand Down Expand Up @@ -76,9 +103,16 @@ workflow FILE_PREPARATION {
ch_results = ch_results.mix(ch_branched_input.mzML)
}

THERMORAWFILEPARSER( ch_branched_input.raw )
// Output: spectra (tuple val(meta), path(mzML/mgf/parquet)), log, versions via topic channel
ch_results = ch_results.mix(THERMORAWFILEPARSER.out.spectra)
if (convert_raw) {
// Convert Thermo .raw to .mzML via ThermoRawFileParser (default for DIA-NN < 2.1.0).
THERMORAWFILEPARSER( ch_branched_input.raw )
// Output: spectra (tuple val(meta), path(mzML/mgf/parquet)), log, versions via topic channel
ch_results = ch_results.mix(THERMORAWFILEPARSER.out.spectra)
} else {
// Pass Thermo .raw straight through to DIA-NN (native reader, DIA-NN >= 2.1.0).
// See https://github.com/vdemichev/DiaNN/issues/1468 for known caveats.
ch_results = ch_results.mix(ch_branched_input.raw)
}

ch_results.map{ it -> [it[0], it[1]] }.set{ indexed_mzml_bundle }

Expand Down
Loading