Skip to content

Add --mzml_convert parameter for native Thermo .raw support - #68

Merged
ypriverol merged 3 commits into
devfrom
claude/add-root-file-type-Xjb9L
Apr 24, 2026
Merged

ypriverol merged 3 commits into
devfrom
claude/add-root-file-type-Xjb9L

Conversation

@ypriverol

Copy link
Copy Markdown
Contributor

Description

This PR adds a new --mzml_convert parameter to control whether Thermo .raw files are converted to .mzML via ThermoRawFileParser before being fed to DIA-NN.

Motivation

DIA-NN 2.1.0 (released 2025-03-25) added native Thermo .raw support on Linux, eliminating the need for conversion via ThermoRawFileParser in many cases. Skipping this conversion step saves one container invocation and an I/O pass per .raw file — a non-trivial optimization on large datasets.

However, DIA-NN's native Thermo reader has known issues on some acquisition schemes (e.g., DiaNN#1468), and conversion is still required to generate mzML-based QC statistics. This parameter provides explicit control over the behavior.

Changes

Workflow Logic (subworkflows/local/file_preparation/main.nf):

  • Added version-aware logic to determine whether to convert .raw files based on --diann_version
  • Default behavior (unset): auto-selects conversion for DIA-NN < 2.1.0, native pass-through for >= 2.1.0
  • Explicit true: forces conversion via ThermoRawFileParser
  • Explicit false: requires DIA-NN >= 2.1.0, passes .raw files directly to DIA-NN
  • Added validation to fail fast if native .raw is requested with DIA-NN < 2.1.0
  • Added warning when --mzml_convert is set but no .raw files will be processed

Documentation:

  • Updated docs/usage.md with detailed explanation of the parameter, behavior table, and known caveats
  • Updated docs/parameters.md with parameter definition
  • Updated nextflow_schema.json with parameter schema and help text
  • Updated nextflow.config with default value and inline comments
  • Updated CHANGELOG.md with feature entry

Closes

Addresses #66

PR checklist

  • This comment contains a description of changes (with reason).
  • Usage Documentation in docs/usage.md is updated.
  • CHANGELOG.md is updated.
  • nextflow_schema.json is updated.
  • nextflow.config is updated with parameter defaults.

https://claude.ai/code/session_01FwKnS4YJFjKMTaTg5WHy8b

)

DIA-NN 2.1.0 added native Thermo .raw support on Linux, making the
ThermoRawFileParser step optional. Introduce --mzml_convert with an
auto default that picks TRFP vs. native based on --diann_version, plus
a version guard that rejects explicit --mzml_convert false for < 2.1.0.
Emit a warning when --local_input_type under --root_folder means no
.raw will reach the conversion branch (parameter is a no-op).
@coderabbitai

coderabbitai Bot commented Apr 22, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 328a2dd8-332c-4d3e-8c4b-dfae565096cf

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch claude/add-root-file-type-Xjb9L

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

github-actions Bot commented Apr 22, 2026 •

Copy link
Copy Markdown

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 912c4c0

+| ✅ 106 tests passed       |+
#| ❔  19 tests were ignored |#
#| ❔   1 tests had warnings |#
!| ❗   4 tests had warnings |!
Details

❗ Test warnings:

  • files_exist - File not found: conf/igenomes.config
  • files_exist - File not found: conf/igenomes_ignored.config
  • files_exist - File not found: .github/workflows/awstest.yml
  • files_exist - File not found: .github/workflows/awsfulltest.yml

❔ Tests ignored:

❔ Tests fixed:

✅ Tests passed:

Run details

  • nf-core/tools version 3.5.2
  • Run at 2026-04-22 09:39:02

@yueqixuan

Copy link
Copy Markdown
Collaborator

I tested this PR locally on Linux using the test dataset (PXD026600) with both versions 2.5.0 and 2.3.2. Unfortunately, both versions failed with an error from the underlying Thermo library:

  [0:01] File #1/1
  [0:01] Loading run RD139_Narrow_UPS1_0_1fmol_inj1.raw
  ThermoRaw exception: Instrument index not available for requested device (Parameter 'instrumentIndex')
  Cannot read the metadata from the .raw file
  ERROR: cannot load the file, skipping
  ERROR: DIA-NN tried but failed to load the following files: RD139_Narrow_UPS1_0_1fmol_inj1.raw

@yueqixuan

yueqixuan commented Apr 24, 2026 •

Copy link
Copy Markdown
Collaborator

Hi @ypriverol ,
Just an update on this issue. After some testing, it turns out this bug has nothing to do with DIA-NN missing the dotnet-sdk-8.0 (actually, it turns out DIA-NN doesn't seem to lack SDK 8.0 when I run it directly on its own.).

DIA-NN 2.5.0 Academia  (Data-Independent Acquisition by Neural Networks)
Compiled on Apr 12 2026 10:45:33
Current date and time: Fri Apr 24 02:48:44 2026
Logical CPU cores: 20
Thread number set to 6
DIA-NN will carry out FASTA digest for in silico lib generation
In silico digest will involve cuts at K*,R*
But excluding cuts at *P
Modification Carbamidomethyl with mass delta 57.0215 at C will be considered as fixed
Modification Oxidation with mass delta 15.9949 at M will be considered as variable
Min precursor m/z set to 400
Max precursor m/z set to 800
Min fragment m/z set to 200
Max fragment m/z set to 1800
Maximum number of missed cleavages set to 2
Min peptide length set to 7
Max peptide length set to 30
Min precursor charge set to 2
Max precursor charge set to 4
Maximum number of variable modifications set to 2
Deep learning will be used to generate a new in silico spectral library from peptides provided
N-terminal methionine excision enabled
A fast algorithm will be used to select the MS2 mass accuracy setting
Only peak groups with correlation-based score exceeding 2 will be considered; will be deactivated for the second MBR pass
Peak groups with correlation-based score at 1 below the maximum will be excluded; will be deactivated for the second MBR pass
DIA-NN will disable machine learning during the calibration stage to save memory, this may lead to somewhat lower identification numbers; will be deactivated for the second MBR pass
Protein inference will not be performed
The spectral library (if generated) will retain the original spectra but will include empirically-aligned RTs
Existing .quant files will be used
Mass accuracy will be determined separately for different runs
Scan windows will be inferred separately for different runs
Scan window radius set to 6
Implicit protein grouping: genes; this determines which peptides are considered 'proteotypic' and thus affects protein FDR calculation
Normalisation disabled
Precursor/protein x samples expression level matrices will be saved along with the main report
Output will be filtered at 0.01 FDR
Precursor/protein x sample matrices will be filtered at 0.01 precursor & protein-level FDR
Protein x sample matrices will be filtered at 0.05 protein q-value
Legacy (direct) quantification mode
.NET SDK 8.0.407 or later found, Thermo .raw support enabled
WARNING: note the mass accuracy settings used by DIA-NN, automatic optimisation will not be performed as at least one of MS1/MS2 mass accuracies is user-provided
Mass accuracy will be fixed to 2e-05 (MS2) and 1e-05 (MS1)
WARNING: incorrect settings, the in silico-predicted library must be generated in a separate pipeline step and then used to process the raw data, now without activating FASTA digest
WARNING: make sure to enable Protein inference in the GUI or set the proteotypicity definition using --pg-level
WARNING: peptidoform scoring enabled because variable modifications have been declared; to disable, use --no-peptidoforms
The following variable modifications will be localised: UniMod:35 

I ran the test dataset directly using the DIA-NN 2.5.0 Docker image (without the SDK installed), and it executed successfully.
The real issue lies in Nextflow's symlink mechanism. The Thermo native reader in DIA-NN fails to parse the metadata when .raw files are mounted as symlinks in the container. Once I configured process { stageInMode = 'copy' } for the process, the quantmsdiann pipeline successfully finished the test:

N E X T F L O W  ~  version 25.10.4
Launching `../../../github_test_0422/quantmsdiann/main.nf` [lonely_maxwell] DSL2 - revision: 616f896425
[d3/308061] Submitted process > BIGBIO_QUANTMSDIANN:QUANTMSDIANN:INPUT_CHECK:SAMPLESHEET_CHECK (PXD026600.sdrf.tsv)
[54/248737] Submitted process > BIGBIO_QUANTMSDIANN:QUANTMSDIANN:CREATE_INPUT_CHANNEL:SDRF_PARSING (PXD026600.sdrf.tsv)
[ea/c803df] Submitted process > BIGBIO_QUANTMSDIANN:QUANTMSDIANN:DIA:INSILICO_LIBRARY_GENERATION (REF_EColi_K12_UPS1_combined.fasta)
[76/e221e3] Submitted process > BIGBIO_QUANTMSDIANN:QUANTMSDIANN:DIA:TUNE_PRELIMINARY_ANALYSIS (RD139_Narrow_UPS1_0_25fmol_inj1)
[10/e9009d] Submitted process > BIGBIO_QUANTMSDIANN:QUANTMSDIANN:DIA:TUNE_PRELIMINARY_ANALYSIS (RD139_Narrow_UPS1_0_1fmol_inj2)
[1f/740724] Submitted process > BIGBIO_QUANTMSDIANN:QUANTMSDIANN:DIA:TUNE_PRELIMINARY_ANALYSIS (RD139_Narrow_UPS1_0_1fmol_inj1)
[ed/a3688c] Submitted process > BIGBIO_QUANTMSDIANN:QUANTMSDIANN:DIA:TUNE_ASSEMBLE_LIBRARY (PXD026600.sdrf)
[9a/3fd1ba] Submitted process > BIGBIO_QUANTMSDIANN:QUANTMSDIANN:DIA:FINE_TUNE_MODELS (fine_tune)
[24/b34cdd] Submitted process > BIGBIO_QUANTMSDIANN:QUANTMSDIANN:DIA:TUNED_LIBRARY_GENERATION (REF_EColi_K12_UPS1_combined.fasta)
[f9/67d34a] Submitted process > BIGBIO_QUANTMSDIANN:QUANTMSDIANN:DIA:PRELIMINARY_ANALYSIS (RD139_Narrow_UPS1_0_1fmol_inj2)
[16/335b4b] Submitted process > BIGBIO_QUANTMSDIANN:QUANTMSDIANN:DIA:PRELIMINARY_ANALYSIS (RD139_Narrow_UPS1_0_1fmol_inj1)
[a3/7090f4] Submitted process > BIGBIO_QUANTMSDIANN:QUANTMSDIANN:DIA:PRELIMINARY_ANALYSIS (RD139_Narrow_UPS1_0_25fmol_inj2)
[3e/aa0793] Submitted process > BIGBIO_QUANTMSDIANN:QUANTMSDIANN:DIA:PRELIMINARY_ANALYSIS (RD139_Narrow_UPS1_0_25fmol_inj1)
[31/3ffa04] Submitted process > BIGBIO_QUANTMSDIANN:QUANTMSDIANN:DIA:ASSEMBLE_EMPIRICAL_LIBRARY (PXD026600.sdrf)
[b4/7f8388] Submitted process > BIGBIO_QUANTMSDIANN:QUANTMSDIANN:DIA:INDIVIDUAL_ANALYSIS (RD139_Narrow_UPS1_0_25fmol_inj1)
[be/5e8b4a] Submitted process > BIGBIO_QUANTMSDIANN:QUANTMSDIANN:DIA:INDIVIDUAL_ANALYSIS (RD139_Narrow_UPS1_0_1fmol_inj1)
[b4/016b40] Submitted process > BIGBIO_QUANTMSDIANN:QUANTMSDIANN:DIA:INDIVIDUAL_ANALYSIS (RD139_Narrow_UPS1_0_25fmol_inj2)
[7d/bed9bb] Submitted process > BIGBIO_QUANTMSDIANN:QUANTMSDIANN:DIA:INDIVIDUAL_ANALYSIS (RD139_Narrow_UPS1_0_1fmol_inj2)
[0c/422df1] Submitted process > BIGBIO_QUANTMSDIANN:QUANTMSDIANN:DIA:FINAL_QUANTIFICATION (PXD026600.sdrf)
[75/1a966a] Submitted process > BIGBIO_QUANTMSDIANN:QUANTMSDIANN:DIA:DIANN_MSSTATS (diann_msstats)
[47/62faf4] Submitted process > BIGBIO_QUANTMSDIANN:QUANTMSDIANN:SUMMARY_PIPELINE
-�[0;35m[bigbio/quantmsdiann]�[0;32m Pipeline completed successfully�[0m-

@ypriverol

Copy link
Copy Markdown
Contributor Author

@yueqixuan this means this PR bigbio/quantms-containers#16 is not needed?

@yueqixuan

Copy link
Copy Markdown
Collaborator

I previously built the local image using the Dockerfile from the DIA-NN archive. I just double-checked, and dotnet-sdk-8.0 is indeed required, so this PR (bigbio/quantms-containers#16) is necessary. Once we fix the symlink issue on top of this, quantmsdiann should be able to read .raw files directly.

@ypriverol

Copy link
Copy Markdown
Contributor Author

Question here @yueqixuan @jpfeuffer @timosachsenberg @claude @copilot @enryH:

Should we make raw -> .dia the default behaviour? do mzML in the middle?

…ailure

DIA-NN's native Thermo reader fails when .raw files are staged as
symlinks in the Nextflow work directory (Thermo SDK limitation).

Add a dynamic stageInMode closure to PRELIMINARY_ANALYSIS,
INDIVIDUAL_ANALYSIS, and ASSEMBLE_EMPIRICAL_LIBRARY that switches to
'copy' staging when native .raw mode is active (DIA-NN >= 2.1.0 with
mzml_convert != true), and uses 'symlink' otherwise.

Extract the shared mode-detection logic into VersionUtils.isNativeRawMode()
to avoid duplication across modules.

Agent-Logs-Url: https://github.com/bigbio/quantmsdiann/sessions/657483e0-5833-4c95-a84d-76a61f5f5908

Co-authored-by: ypriverol <52113+ypriverol@users.noreply.github.com>

Copilot AI commented Apr 24, 2026

Copy link
Copy Markdown
Contributor

Hi @ypriverol ,
Just an update on this issue. After some testing, it turns out this bug has nothing to do with DIA-NN missing the dotnet-sdk-8.0 (actually, it turns out DIA-NN doesn't seem to lack SD...

Fixed in commit 5aabdda. DIA-NN per-file processes (PRELIMINARY_ANALYSIS, INDIVIDUAL_ANALYSIS, ASSEMBLE_EMPIRICAL_LIBRARY) now use a dynamic stageInMode closure that switches to 'copy' when native .raw mode is active (DIA-NN >= 2.1.0 with mzml_convert != true), and falls back to 'symlink' otherwise. The shared detection logic is centralized in VersionUtils.isNativeRawMode(params) to avoid duplication.

@ypriverol
ypriverol merged commit a25b015 into dev Apr 24, 2026
20 checks passed
ypriverol added a commit that referenced this pull request Apr 24, 2026
Populate the [Unreleased] section (bumped to 2.1.0dev by the version
manifest change) with entries for the --local_input_type default switch,
the schema enum / runtime validation, the Bruker .d archive variants,
the pride_codon_slurm resource cleanup, and the native-raw stageInMode
fix merged from #68.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@ypriverol
ypriverol deleted the claude/add-root-file-type-Xjb9L branch May 5, 2026 05:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support native Thermo .raw input for DIA-NN ≥ 2.1.0 (skip TRFP conversion)

4 participants