Repository navigation
Conversation
13-task plan covering robustness fixes, DDA support, new DIA-NN params, InfinDIA groundwork, comprehensive documentation, and issue cleanup. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Without pipefail, if the command before tee fails, tee returns 0 and the Nextflow task appears to succeed. This masked failures in generate_cfg, diann_msstats, samplesheet_check, and sdrf_parsing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
These are the longest-running tasks and most susceptible to transient failures (OOM, I/O timeouts). The error_retry label enables automatic retry on signal exits (130-145, 104, 175). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Guard ch_searchdb and ch_experiment_meta with ifEmpty to fail fast with clear error messages instead of hanging indefinitely. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds conf/diann_versions/v2_3_2.config with ghcr.io/bigbio/diann:2.3.2 container. Use -profile diann_v2_3_2 to opt in. Default stays 1.8.1. Enables DDA support and InfinDIA features. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- New param diann_dda (boolean, default: false) - Version guard: requires DIA-NN >= 2.3.2 - Passes --dda to all 5 DIA-NN modules when enabled - Accepts DDA acquisition method in SDRF when diann_dda=true - Added --dda to blocked lists in all modules Closes #5 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- test_dda: BSA dataset with diann_dda=true on DIA-NN 2.3.2 - test_dia_skip_preanalysis: tests previously untested skip path Both added to extended_ci.yml stage 2a. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- diann_light_models: 10x faster in-silico library generation - diann_export_quant: fragment-level parquet export - diann_site_ms1_quant: MS1 apex intensities for PTM quantification All require DIA-NN >= 2.0. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Experimental support for InfinDIA (DIA-NN 2.3.0+). Passes --infin-dia to library generation when enabled. Version guard enforces >= 2.3.0. No test config — InfinDIA requires large databases. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Complete reference for all ~70 pipeline parameters grouped by category with types, defaults, descriptions, and version requirements. Closes #1 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- DDA mode documentation with limitations - Missing param sections (preprocessing, extra_args scope, verbose output) - DIA-NN version selection guide - Parquet vs TSV output explanation - MSstats format section - pmultiqc citation added - README updated with version table and parameter reference link Closes #3, #9, #15 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add version guard for DIA-NN 2.0+ params (--light-models, --export-quant, --site-ms1-quant) to prevent crashes with 1.8.1 - Add *.site_report.parquet as optional output in FINAL_QUANTIFICATION for site-level PTM quantification Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. test_dda.config: Add diann_version = '2.3.2' so the version guard
doesn't reject DDA mode (default is 1.8.1, guard requires >= 2.3.2)
2. quantmsdiann.nf: Update branch condition to also match "dda"
acquisition method. Previously "dda".contains("dia") was false,
causing all DDA files to be silently dropped from processing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
These flags exist in DIA-NN 1.8.x but were removed in 2.3.x, causing 'unrecognised option' warnings. Only pass them for versions < 2.3. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 Walkthrough📝 Walkthrough🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Bruker .d to mzML conversion via tdf2mzml is no longer needed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Merge dev branch (version bump, tdf2mzml removal, lint fixes, DOI update) - Update test_dda.config to use PXD022287 HeLa DDA dataset with subset FASTA - Add test_dda profile to CI matrix in ci.yml Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The test_dda profile uses ghcr.io/bigbio/diann:2.3.2 which is a private container requiring authentication. Add Docker login step (matching merge_ci.yml) conditioned on test_dda profile. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove implementation plan from repo, add docs/plans/ to .gitignore - Add lib/VersionUtils.groovy for semantic version comparison (prevents string comparison bugs like '2.10.0' < '2.3') - Update all version guards in dia.nf and module scripts to use VersionUtils.versionLessThan/versionAtLeast Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
DDA analysis support is a major feature warranting a major version bump. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Rename output version→versions in sdrf_parsing/meta.yml - Add ch_ prefix to input_file→ch_input_file in input_check/meta.yml - Fix grammar in pmultiqc and diann_msstats meta.yml descriptions - Fix glob pattern in decompress_dotd/meta.yml (double-dot expansion) - Update CITATIONS.md to link published Nature Methods article - Fix schema_input.json error messages (source name, whitespace) - Standardize quantmsdiann keyword in utils meta.yml Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Minor update
- New v2_5_0.config version profile (ghcr.io/bigbio/diann:2.5.0) - Add diann_v2_5_0 profile to nextflow.config - Block --parent flag in BlockedFlags.groovy COMMON list (container-managed, overriding breaks model discovery) - Document fine-tuning workflow in docs/usage.md: - How to generate tuning libraries - How to fine-tune RT/IM/fragment models - How to use fine-tuned models via --extra_args - Update version tables in docs/usage.md and AGENTS.md New DIA-NN 2.5.0 CLI flags (passable via --extra_args): --tokens, --rt-model, --fr-model, --im-model (model selection) --aa-eq (amino acid equivalence for reannotation) --tune-lib, --tune-rt, --tune-im, --tune-fr (fine-tuning) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Document all new DIA-NN 2.5.0 flags available via --extra_args: model selection (--tokens, --rt-model, --fr-model, --im-model), fine-tuning (--tune-lib, --tune-rt, --tune-im, --tune-fr, etc.), and --aa-eq for amino acid equivalence. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add aa_eq param (default: false) that maps to DIA-NN's --aa-eq flag. When enabled, I&L, Q&E, N&D are treated as equivalent amino acids during reannotation — essential for entrapment FDR benchmarks. - Added to nextflow.config, nextflow_schema.json, docs/parameters.md - Passed to all 5 DIA-NN modules - Added to BlockedFlags.groovy COMMON list (pipeline-managed) - Moved from "via extra_args" to proper pipeline parameter in docs Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… for Vadim Rewrite the fine-tuning documentation to: - Explain what tokens/dict.txt are (neural network encoding of modifications) - Clarify that --tune-lib cannot be combined with --gen-spec-lib (separate DIA-NN invocations, confirmed in DIA-NN #1499) - Document the full two-run workflow with concrete commands - Propose the future integrated FINE_TUNE_MODELS step with a question for @vdemichev about the right integration point Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When --enable_fine_tuning is enabled, the pipeline: 1. Runs the standard flow through ASSEMBLE_EMPIRICAL_LIBRARY 2. Fine-tunes RT/IM (optionally fragment) models on the empirical library 3. Re-generates the in-silico library using tuned models (--tokens, --rt-model, --im-model) 4. Uses the tuned library for INDIVIDUAL_ANALYSIS and FINAL_QUANTIFICATION New parameters: --enable_fine_tuning (default: false) — enable the fine-tuning step --tune_fr (default: false) — also fine-tune the fragmentation model --tune_lr (default: DIA-NN's 0.0005) — fine-tuning learning rate New module: modules/local/diann/fine_tune_models/ - Takes empirical library + FASTA + diann_config - Runs diann --tune-lib --tune-rt --tune-im - Outputs: dict.txt, rt.d0.pt, im.d0.pt (optionally fr.d0.pt) INSILICO_LIBRARY_GENERATION updated with optional tuned model inputs (pass [] when not fine-tuning, actual files when fine-tuning). Version guard: requires DIA-NN >= 2.0. Cannot be combined with --skip_preliminary_analysis. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Per Vadim Demichev's recommendation, fine-tuning should run as a separate phase BEFORE the main analysis, not mid-pipeline: Phase 0 (when --enable_fine_tuning): 1. INSILICO_LIBRARY_GENERATION (default models) 2. TUNE_PRELIMINARY_ANALYSIS (on --tune_n_files subset) 3. TUNE_ASSEMBLE_LIBRARY (empirical library from subset) 4. FINE_TUNE_MODELS (train RT/IM/fragment models) 5. TUNED_LIBRARY_GENERATION (re-generate with tuned models) Phase 1 (always, the standard pipeline): Uses tuned in-silico library (or default if no fine-tuning) PRELIMINARY_ANALYSIS → ASSEMBLE → INDIVIDUAL → FINAL New parameter: --tune_n_files (default: 3) — number of files for the tuning search subset. Uses Nextflow process aliases (TUNE_PRELIMINARY_ANALYSIS, TUNE_ASSEMBLE_LIBRARY, TUNED_LIBRARY_GENERATION) to avoid duplicate process invocation conflicts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
feat: add DIA-NN 2.5.0 support with model fine-tuning documentation
feat: add optional FINE_TUNE_MODELS step for DL model fine-tuning
There was a problem hiding this comment.
Pull request overview
This PR prepares the quantmsdiann pipeline for a 2.0.0 release by adding beta DDA support and an optional DIA-NN model fine-tuning phase, while also refactoring/renaming DIA-NN parameters and hardening version/flag handling across modules.
Changes:
- Add DDA mode support (auto-detected from SDRF or forced via
--dda) with DIA-NN version guards and a new CI test profile. - Introduce an optional “Phase 0” model fine-tuning workflow and new DIA-NN feature flags (scoring mode, InfinDIA, export options, channel normalization).
- Centralize “pipeline-managed flag” stripping into
lib/BlockedFlags.groovy, add semantic version comparison vialib/VersionUtils.groovy, and update docs/config/schema accordingly.
Reviewed changes
Copilot reviewed 59 out of 61 changed files in this pull request and generated 13 comments.
Show a summary per file
| File | Description |
|---|---|
| workflows/quantmsdiann.nf | Routes DDA acquisitions into the DIA/DIA-NN workflow path. |
| workflows/dia.nf | Adds version guards, DDA detection, optional fine-tuning phase, and report output handling changes. |
| subworkflows/local/utils_nfcore_quantms_pipeline/meta.yml | Updates keyword metadata to quantmsdiann. |
| subworkflows/local/input_check/meta.yml | Renames documented output key. |
| subworkflows/local/file_preparation/meta.yml | Removes tdf2mzml component reference. |
| subworkflows/local/create_input_channel/main.nf | Groups SDRF rows per file and supports DIA vs DDA acquisition detection. |
| nextflow_schema.json | Renames/adds DIA-NN parameters (DDA, scoring, InfinDIA, fine-tuning, q-values, etc.). |
| nextflow.config | Bumps manifest version to 2.0.0 and updates/renames params and profiles. |
| modules/local/utils/tdf2mzml/meta.yml | Removes deprecated tdf2mzml module metadata. |
| modules/local/utils/tdf2mzml/main.nf | Removes deprecated tdf2mzml conversion process. |
| modules/local/utils/mzml_statistics/main.nf | Updates quantms-utils container version. |
| modules/local/utils/decompress_dotd/meta.yml | Updates input pattern documentation. |
| modules/local/sdrf_parsing/meta.yml | Renames version output to versions in metadata. |
| modules/local/sdrf_parsing/main.nf | Adds pipefail for safer SDRF parsing error propagation. |
| modules/local/samplesheet_check/main.nf | Updates quantms-utils container and adds pipefail. |
| modules/local/pmultiqc/meta.yml | Improves output description wording. |
| modules/local/pmultiqc/main.nf | Updates pmultiqc container version. |
| modules/local/diann/preliminary_analysis/meta.yml | Clarifies log capture behavior. |
| modules/local/diann/preliminary_analysis/main.nf | Uses centralized blocked-flag stripping; adds scoring/DDA flags and robust log capture. |
| modules/local/diann/insilico_library_generation/meta.yml | Adds is_dda input and clarifies logging. |
| modules/local/diann/insilico_library_generation/main.nf | Adds DDA/InfinDIA/tuned-model support; uses centralized blocked-flag stripping; robust log capture. |
| modules/local/diann/individual_analysis/meta.yml | Updates log output naming/description. |
| modules/local/diann/individual_analysis/main.nf | Uses centralized blocked-flag stripping; adds scoring/DDA flags; version-conditional flags; robust log capture. |
| modules/local/diann/generate_cfg/main.nf | Updates quantms-utils container version and adds pipefail. |
| modules/local/diann/fine_tune_models/meta.yml | Adds new fine-tuning module metadata. |
| modules/local/diann/fine_tune_models/main.nf | Adds new fine-tuning process to generate tuned DIA-NN model artifacts. |
| modules/local/diann/final_quantification/meta.yml | Splits main report outputs (parquet vs TSV) and clarifies logging. |
| modules/local/diann/final_quantification/main.nf | Uses centralized blocked-flag stripping; adds new q-value params and feature flags; robust log capture. |
| modules/local/diann/diann_msstats/meta.yml | Minor doc wording adjustments. |
| modules/local/diann/diann_msstats/main.nf | Updates quantms-utils container and uses new q-value param; adds pipefail. |
| modules/local/diann/assemble_empirical_library/meta.yml | Clarifies log capture behavior. |
| modules/local/diann/assemble_empirical_library/main.nf | Uses centralized blocked-flag stripping; adds scoring/DDA flags; robust log capture. |
| lib/VersionUtils.groovy | Adds semantic version comparison helpers for DIA-NN feature guards. |
| lib/BlockedFlags.groovy | Centralizes stripping of pipeline-managed DIA-NN flags from extra_args. |
| docs/usage.md | Updates usage docs for new params/features, adds DDA section and fine-tuning docs. |
| docs/parameters.md | Updates parameter reference with renamed/new DIA-NN params and new q-value controls. |
| conf/tests/test_latest_dia.config | Renames diann_normalize → normalize. |
| conf/tests/test_full_dia.config | Renames diann_normalize → normalize. |
| conf/tests/test_dia_skip_preanalysis.config | Adds new test profile for skip_preliminary_analysis. |
| conf/tests/test_dia_quantums.config | Renames diann_normalize → normalize. |
| conf/tests/test_dia_parquet.config | Renames params to new names (normalize, report_decoys). |
| conf/tests/test_dia_dotd.config | Renames diann_normalize → normalize. |
| conf/tests/test_dia_2_2_0.config | Renames diann_normalize → normalize. |
| conf/tests/test_dia.config | Renames diann_normalize → normalize. |
| conf/tests/test_dda.config | Adds new test profile for DDA mode with DIA-NN 2.3.2 container. |
| conf/pride_codon_slurm.config | Removes hardcoded DIA-NN container and documents version selection. |
| conf/modules/dia.config | Switches module arg plumbing to params.extra_args. |
| conf/diann_versions/v2_5_0.config | Adds DIA-NN 2.5.0 version profile config. |
| conf/diann_versions/v2_3_2.config | Adds DIA-NN 2.3.2 version profile config. |
| conf/diann_versions/v2_2_0.config | Removes engine forcing; documents engine selection via profiles. |
| conf/diann_versions/v2_1_0.config | Removes engine forcing; documents engine selection via profiles. |
| assets/schema_input.json | Improves validation error messages. |
| README.md | Updates DOI and reflects parameter renames / pipeline summary changes. |
| CITATIONS.md | Updates pipeline publication link. |
| CITATION.cff | Adds a CFF citation file for software citation. |
| CHANGELOG.md | Adds a 2.0.0 release entry describing the new features and breaking changes. |
| AGENTS.md | Updates supported DIA-NN versions and pipeline overview. |
| .gitignore | Ignores docs/plans/. |
| .github/workflows/extended_ci.yml | Expands extended CI matrix to include DDA and skip-preanalysis tests. |
| .github/workflows/ci.yml | Adds test_dda to CI and handles GHCR auth-based skipping for forks. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // Run in-silico library generation first (with default models) for the tuning search | ||
| INSILICO_LIBRARY_GENERATION(ch_searchdb, ch_diann_cfg_val, ch_is_dda, [], [], []) | ||
| tune_speclib = INSILICO_LIBRARY_GENERATION.out.predict_speclib |
There was a problem hiding this comment.
This fine-tuning path calls INSILICO_LIBRARY_GENERATION(..., [], [], []) to represent “no tuned models”. With the current module signature using path(...) inputs, passing [] is likely to result in an empty input and the process never executing (or a staging error). Once the module inputs are made optional, consider passing Channel.empty()/null consistently to make the intent explicit and avoid type ambiguity.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Add missing blank line before heading. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Update reproducibility example from -r 1.0.0 to -r 2.0.0 - Update "Latest DIA-NN version" comment from 2.1.0 to 2.5.0 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PR checklist
nf-core pipelines lint).nextflow run . -profile test,docker --outdir <OUTDIR>).nextflow run . -profile debug,test,docker --outdir <OUTDIR>).docs/usage.mdis updated.docs/output.mdis updated.CHANGELOG.mdis updated.README.mdis updated (including new tool citations and authors/contributors).Summary by CodeRabbit
Documentation
New Features
Bug Fixes
Chores