Skip to content

New release 2.0.0 including dda workflow for quantmsdiann - #36

Merged
ypriverol merged 84 commits into
mainfrom
dev
Apr 18, 2026
Merged

ypriverol merged 84 commits into
mainfrom
dev

Conversation

@ypriverol

@ypriverol ypriverol commented Apr 6, 2026 •

Copy link
Copy Markdown
Contributor

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the pipeline conventions in the contribution docs
  • If necessary, also make a PR on the bigbio/quantmsdiann branch on the nf-core/test-datasets repository.
  • Make sure your code lints (nf-core pipelines lint).
  • Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
  • Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

Summary by CodeRabbit

  • Documentation

    • Updated DOI/badges and many parameter names; added DDA mode, InfinDIA notes, new DIA‑NN options, convert_dotd and revised usage/parameters docs.
  • New Features

    • DDA analysis mode support, new DIA‑NN flags (scoring, channel normalization, InfinDIA, export options) and version-aware behavior; new test profiles.
  • Bug Fixes

    • Improved error propagation and safer logging to surface failures earlier.
  • Chores

    • Removed legacy .d→mzML converter and bumped pipeline manifest version.

ypriverol and others added 18 commits April 3, 2026 11:26
13-task plan covering robustness fixes, DDA support, new DIA-NN params,
InfinDIA groundwork, comprehensive documentation, and issue cleanup.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Without pipefail, if the command before tee fails, tee returns 0 and
the Nextflow task appears to succeed. This masked failures in
generate_cfg, diann_msstats, samplesheet_check, and sdrf_parsing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
These are the longest-running tasks and most susceptible to transient
failures (OOM, I/O timeouts). The error_retry label enables automatic
retry on signal exits (130-145, 104, 175).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Guard ch_searchdb and ch_experiment_meta with ifEmpty to fail fast
with clear error messages instead of hanging indefinitely.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds conf/diann_versions/v2_3_2.config with ghcr.io/bigbio/diann:2.3.2
container. Use -profile diann_v2_3_2 to opt in. Default stays 1.8.1.
Enables DDA support and InfinDIA features.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- New param diann_dda (boolean, default: false)
- Version guard: requires DIA-NN >= 2.3.2
- Passes --dda to all 5 DIA-NN modules when enabled
- Accepts DDA acquisition method in SDRF when diann_dda=true
- Added --dda to blocked lists in all modules

Closes #5

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- test_dda: BSA dataset with diann_dda=true on DIA-NN 2.3.2
- test_dia_skip_preanalysis: tests previously untested skip path
Both added to extended_ci.yml stage 2a.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- diann_light_models: 10x faster in-silico library generation
- diann_export_quant: fragment-level parquet export
- diann_site_ms1_quant: MS1 apex intensities for PTM quantification
All require DIA-NN >= 2.0.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Experimental support for InfinDIA (DIA-NN 2.3.0+). Passes --infin-dia
to library generation when enabled. Version guard enforces >= 2.3.0.
No test config — InfinDIA requires large databases.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Complete reference for all ~70 pipeline parameters grouped by category
with types, defaults, descriptions, and version requirements.

Closes #1

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- DDA mode documentation with limitations
- Missing param sections (preprocessing, extra_args scope, verbose output)
- DIA-NN version selection guide
- Parquet vs TSV output explanation
- MSstats format section
- pmultiqc citation added
- README updated with version table and parameter reference link

Closes #3, #9, #15

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add version guard for DIA-NN 2.0+ params (--light-models,
  --export-quant, --site-ms1-quant) to prevent crashes with 1.8.1
- Add *.site_report.parquet as optional output in FINAL_QUANTIFICATION
  for site-level PTM quantification

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. test_dda.config: Add diann_version = '2.3.2' so the version guard
   doesn't reject DDA mode (default is 1.8.1, guard requires >= 2.3.2)

2. quantmsdiann.nf: Update branch condition to also match "dda"
   acquisition method. Previously "dda".contains("dia") was false,
   causing all DDA files to be silently dropped from processing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
These flags exist in DIA-NN 1.8.x but were removed in 2.3.x, causing
'unrecognised option' warnings. Only pass them for versions < 2.3.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Apr 6, 2026 •

Copy link
Copy Markdown

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough
📝 Walkthrough
🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main objective: releasing version 2.0.0 with DDA workflow support. It aligns with the substantial changes across configuration, documentation, and workflow modules.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch dev

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

github-actions Bot commented Apr 6, 2026 •

Copy link
Copy Markdown

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 1a61d17

+| ✅ 106 tests passed       |+
#| ❔  19 tests were ignored |#
#| ❔   1 tests had warnings |#
!| ❗   4 tests had warnings |!
Details

❗ Test warnings:

  • files_exist - File not found: conf/igenomes.config
  • files_exist - File not found: conf/igenomes_ignored.config
  • files_exist - File not found: .github/workflows/awstest.yml
  • files_exist - File not found: .github/workflows/awsfulltest.yml

❔ Tests ignored:

❔ Tests fixed:

✅ Tests passed:

Run details

  • nf-core/tools version 3.5.2
  • Run at 2026-04-18 13:50:22

ypriverol and others added 2 commits April 6, 2026 21:10
Bruker .d to mzML conversion via tdf2mzml is no longer needed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@ypriverol ypriverol changed the title update zenodo link New release 2.0.0 including dda workflow for quantmsdiann Apr 7, 2026
@ypriverol
ypriverol marked this pull request as draft April 7, 2026 05:47
ypriverol and others added 6 commits April 7, 2026 07:20
- Merge dev branch (version bump, tdf2mzml removal, lint fixes, DOI update)
- Update test_dda.config to use PXD022287 HeLa DDA dataset with subset FASTA
- Add test_dda profile to CI matrix in ci.yml

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The test_dda profile uses ghcr.io/bigbio/diann:2.3.2 which is a
private container requiring authentication. Add Docker login step
(matching merge_ci.yml) conditioned on test_dda profile.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove implementation plan from repo, add docs/plans/ to .gitignore
- Add lib/VersionUtils.groovy for semantic version comparison
  (prevents string comparison bugs like '2.10.0' < '2.3')
- Update all version guards in dia.nf and module scripts to use
  VersionUtils.versionLessThan/versionAtLeast

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
DDA analysis support is a major feature warranting a major version bump.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Rename output version→versions in sdrf_parsing/meta.yml
- Add ch_ prefix to input_file→ch_input_file in input_check/meta.yml
- Fix grammar in pmultiqc and diann_msstats meta.yml descriptions
- Fix glob pattern in decompress_dotd/meta.yml (double-dot expansion)
- Update CITATIONS.md to link published Nature Methods article
- Fix schema_input.json error messages (source name, whitespace)
- Standardize quantmsdiann keyword in utils meta.yml

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ypriverol and others added 14 commits April 13, 2026 11:01
- New v2_5_0.config version profile (ghcr.io/bigbio/diann:2.5.0)
- Add diann_v2_5_0 profile to nextflow.config
- Block --parent flag in BlockedFlags.groovy COMMON list
  (container-managed, overriding breaks model discovery)
- Document fine-tuning workflow in docs/usage.md:
  - How to generate tuning libraries
  - How to fine-tune RT/IM/fragment models
  - How to use fine-tuned models via --extra_args
- Update version tables in docs/usage.md and AGENTS.md

New DIA-NN 2.5.0 CLI flags (passable via --extra_args):
  --tokens, --rt-model, --fr-model, --im-model (model selection)
  --aa-eq (amino acid equivalence for reannotation)
  --tune-lib, --tune-rt, --tune-im, --tune-fr (fine-tuning)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Document all new DIA-NN 2.5.0 flags available via --extra_args:
model selection (--tokens, --rt-model, --fr-model, --im-model),
fine-tuning (--tune-lib, --tune-rt, --tune-im, --tune-fr, etc.),
and --aa-eq for amino acid equivalence.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add aa_eq param (default: false) that maps to DIA-NN's --aa-eq flag.
When enabled, I&L, Q&E, N&D are treated as equivalent amino acids
during reannotation — essential for entrapment FDR benchmarks.

- Added to nextflow.config, nextflow_schema.json, docs/parameters.md
- Passed to all 5 DIA-NN modules
- Added to BlockedFlags.groovy COMMON list (pipeline-managed)
- Moved from "via extra_args" to proper pipeline parameter in docs

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… for Vadim

Rewrite the fine-tuning documentation to:
- Explain what tokens/dict.txt are (neural network encoding of modifications)
- Clarify that --tune-lib cannot be combined with --gen-spec-lib
  (separate DIA-NN invocations, confirmed in DIA-NN #1499)
- Document the full two-run workflow with concrete commands
- Propose the future integrated FINE_TUNE_MODELS step with a
  question for @vdemichev about the right integration point

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When --enable_fine_tuning is enabled, the pipeline:
1. Runs the standard flow through ASSEMBLE_EMPIRICAL_LIBRARY
2. Fine-tunes RT/IM (optionally fragment) models on the empirical library
3. Re-generates the in-silico library using tuned models (--tokens, --rt-model, --im-model)
4. Uses the tuned library for INDIVIDUAL_ANALYSIS and FINAL_QUANTIFICATION

New parameters:
  --enable_fine_tuning (default: false) — enable the fine-tuning step
  --tune_fr (default: false) — also fine-tune the fragmentation model
  --tune_lr (default: DIA-NN's 0.0005) — fine-tuning learning rate

New module: modules/local/diann/fine_tune_models/
  - Takes empirical library + FASTA + diann_config
  - Runs diann --tune-lib --tune-rt --tune-im
  - Outputs: dict.txt, rt.d0.pt, im.d0.pt (optionally fr.d0.pt)

INSILICO_LIBRARY_GENERATION updated with optional tuned model inputs
(pass [] when not fine-tuning, actual files when fine-tuning).

Version guard: requires DIA-NN >= 2.0.
Cannot be combined with --skip_preliminary_analysis.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Per Vadim Demichev's recommendation, fine-tuning should run as a
separate phase BEFORE the main analysis, not mid-pipeline:

Phase 0 (when --enable_fine_tuning):
  1. INSILICO_LIBRARY_GENERATION (default models)
  2. TUNE_PRELIMINARY_ANALYSIS (on --tune_n_files subset)
  3. TUNE_ASSEMBLE_LIBRARY (empirical library from subset)
  4. FINE_TUNE_MODELS (train RT/IM/fragment models)
  5. TUNED_LIBRARY_GENERATION (re-generate with tuned models)

Phase 1 (always, the standard pipeline):
  Uses tuned in-silico library (or default if no fine-tuning)
  PRELIMINARY_ANALYSIS → ASSEMBLE → INDIVIDUAL → FINAL

New parameter: --tune_n_files (default: 3) — number of files
for the tuning search subset.

Uses Nextflow process aliases (TUNE_PRELIMINARY_ANALYSIS,
TUNE_ASSEMBLE_LIBRARY, TUNED_LIBRARY_GENERATION) to avoid
duplicate process invocation conflicts.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
feat: add DIA-NN 2.5.0 support with model fine-tuning documentation
feat: add optional FINE_TUNE_MODELS step for DL model fine-tuning

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR prepares the quantmsdiann pipeline for a 2.0.0 release by adding beta DDA support and an optional DIA-NN model fine-tuning phase, while also refactoring/renaming DIA-NN parameters and hardening version/flag handling across modules.

Changes:

  • Add DDA mode support (auto-detected from SDRF or forced via --dda) with DIA-NN version guards and a new CI test profile.
  • Introduce an optional “Phase 0” model fine-tuning workflow and new DIA-NN feature flags (scoring mode, InfinDIA, export options, channel normalization).
  • Centralize “pipeline-managed flag” stripping into lib/BlockedFlags.groovy, add semantic version comparison via lib/VersionUtils.groovy, and update docs/config/schema accordingly.

Reviewed changes

Copilot reviewed 59 out of 61 changed files in this pull request and generated 13 comments.

Show a summary per file
File Description
workflows/quantmsdiann.nf Routes DDA acquisitions into the DIA/DIA-NN workflow path.
workflows/dia.nf Adds version guards, DDA detection, optional fine-tuning phase, and report output handling changes.
subworkflows/local/utils_nfcore_quantms_pipeline/meta.yml Updates keyword metadata to quantmsdiann.
subworkflows/local/input_check/meta.yml Renames documented output key.
subworkflows/local/file_preparation/meta.yml Removes tdf2mzml component reference.
subworkflows/local/create_input_channel/main.nf Groups SDRF rows per file and supports DIA vs DDA acquisition detection.
nextflow_schema.json Renames/adds DIA-NN parameters (DDA, scoring, InfinDIA, fine-tuning, q-values, etc.).
nextflow.config Bumps manifest version to 2.0.0 and updates/renames params and profiles.
modules/local/utils/tdf2mzml/meta.yml Removes deprecated tdf2mzml module metadata.
modules/local/utils/tdf2mzml/main.nf Removes deprecated tdf2mzml conversion process.
modules/local/utils/mzml_statistics/main.nf Updates quantms-utils container version.
modules/local/utils/decompress_dotd/meta.yml Updates input pattern documentation.
modules/local/sdrf_parsing/meta.yml Renames version output to versions in metadata.
modules/local/sdrf_parsing/main.nf Adds pipefail for safer SDRF parsing error propagation.
modules/local/samplesheet_check/main.nf Updates quantms-utils container and adds pipefail.
modules/local/pmultiqc/meta.yml Improves output description wording.
modules/local/pmultiqc/main.nf Updates pmultiqc container version.
modules/local/diann/preliminary_analysis/meta.yml Clarifies log capture behavior.
modules/local/diann/preliminary_analysis/main.nf Uses centralized blocked-flag stripping; adds scoring/DDA flags and robust log capture.
modules/local/diann/insilico_library_generation/meta.yml Adds is_dda input and clarifies logging.
modules/local/diann/insilico_library_generation/main.nf Adds DDA/InfinDIA/tuned-model support; uses centralized blocked-flag stripping; robust log capture.
modules/local/diann/individual_analysis/meta.yml Updates log output naming/description.
modules/local/diann/individual_analysis/main.nf Uses centralized blocked-flag stripping; adds scoring/DDA flags; version-conditional flags; robust log capture.
modules/local/diann/generate_cfg/main.nf Updates quantms-utils container version and adds pipefail.
modules/local/diann/fine_tune_models/meta.yml Adds new fine-tuning module metadata.
modules/local/diann/fine_tune_models/main.nf Adds new fine-tuning process to generate tuned DIA-NN model artifacts.
modules/local/diann/final_quantification/meta.yml Splits main report outputs (parquet vs TSV) and clarifies logging.
modules/local/diann/final_quantification/main.nf Uses centralized blocked-flag stripping; adds new q-value params and feature flags; robust log capture.
modules/local/diann/diann_msstats/meta.yml Minor doc wording adjustments.
modules/local/diann/diann_msstats/main.nf Updates quantms-utils container and uses new q-value param; adds pipefail.
modules/local/diann/assemble_empirical_library/meta.yml Clarifies log capture behavior.
modules/local/diann/assemble_empirical_library/main.nf Uses centralized blocked-flag stripping; adds scoring/DDA flags; robust log capture.
lib/VersionUtils.groovy Adds semantic version comparison helpers for DIA-NN feature guards.
lib/BlockedFlags.groovy Centralizes stripping of pipeline-managed DIA-NN flags from extra_args.
docs/usage.md Updates usage docs for new params/features, adds DDA section and fine-tuning docs.
docs/parameters.md Updates parameter reference with renamed/new DIA-NN params and new q-value controls.
conf/tests/test_latest_dia.config Renames diann_normalize → normalize.
conf/tests/test_full_dia.config Renames diann_normalize → normalize.
conf/tests/test_dia_skip_preanalysis.config Adds new test profile for skip_preliminary_analysis.
conf/tests/test_dia_quantums.config Renames diann_normalize → normalize.
conf/tests/test_dia_parquet.config Renames params to new names (normalize, report_decoys).
conf/tests/test_dia_dotd.config Renames diann_normalize → normalize.
conf/tests/test_dia_2_2_0.config Renames diann_normalize → normalize.
conf/tests/test_dia.config Renames diann_normalize → normalize.
conf/tests/test_dda.config Adds new test profile for DDA mode with DIA-NN 2.3.2 container.
conf/pride_codon_slurm.config Removes hardcoded DIA-NN container and documents version selection.
conf/modules/dia.config Switches module arg plumbing to params.extra_args.
conf/diann_versions/v2_5_0.config Adds DIA-NN 2.5.0 version profile config.
conf/diann_versions/v2_3_2.config Adds DIA-NN 2.3.2 version profile config.
conf/diann_versions/v2_2_0.config Removes engine forcing; documents engine selection via profiles.
conf/diann_versions/v2_1_0.config Removes engine forcing; documents engine selection via profiles.
assets/schema_input.json Improves validation error messages.
README.md Updates DOI and reflects parameter renames / pipeline summary changes.
CITATIONS.md Updates pipeline publication link.
CITATION.cff Adds a CFF citation file for software citation.
CHANGELOG.md Adds a 2.0.0 release entry describing the new features and breaking changes.
AGENTS.md Updates supported DIA-NN versions and pipeline overview.
.gitignore Ignores docs/plans/.
.github/workflows/extended_ci.yml Expands extended CI matrix to include DDA and skip-preanalysis tests.
.github/workflows/ci.yml Adds test_dda to CI and handles GHCR auth-based skipping for forks.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread workflows/dia.nf Resolved
Comment thread docs/usage.md Resolved
Comment thread docs/parameters.md Resolved
Comment thread workflows/dia.nf
Comment on lines +128 to +130
// Run in-silico library generation first (with default models) for the tuning search
INSILICO_LIBRARY_GENERATION(ch_searchdb, ch_diann_cfg_val, ch_is_dda, [], [], [])
tune_speclib = INSILICO_LIBRARY_GENERATION.out.predict_speclib

Copilot AI Apr 18, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fine-tuning path calls INSILICO_LIBRARY_GENERATION(..., [], [], []) to represent “no tuned models”. With the current module signature using path(...) inputs, passing [] is likely to result in an empty input and the process never executing (or a staging error). Once the module inputs are made optional, consider passing Channel.empty()/null consistently to make the intent explicit and avoid type ambiguity.

Copilot uses AI. Check for mistakes.
Comment thread workflows/dia.nf Resolved
Comment thread docs/usage.md Resolved
Comment thread nextflow_schema.json Resolved
ypriverol and others added 8 commits April 18, 2026 11:18
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Add missing blank line before heading.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Update reproducibility example from -r 1.0.0 to -r 2.0.0
- Update "Latest DIA-NN version" comment from 2.1.0 to 2.5.0

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@ypriverol
ypriverol merged commit 8d85759 into main Apr 18, 2026
47 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants