You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Workstation DIANN vs. quantmsdiann, a small benchmark #85
I have been stewarding the use of quantmsdiann in my group.
We have seen poorer than usual performance as measured by protein and precursor hits.
I like to believe these discordant results come from inconsistent parameters, I tried to keep them the same, but sometimes it was difficult to translate DIANN GUI to quantms params. I have now benchmarked a demo dataset and can show some results.
TLDR; workstation DIANN has better coverage
Point plot, showing the number of proteins identified (coming from `diann_report.stats.tsv`), for the same 25 samples, across computation environments (local vs. hpc) and different DIANN versions in the hpc. Could not test local environment for different DIANN versions because our workstation has a long queue of jobs :). Points are connected symbolizing the same LCMS sample across runs. I tried to keep parameters consistent across DIANN runs.
Samples
25 samples coming from collected tissue, processed FFPE samples on frame slides.
Same samples as shown in openDVP manuscript for the TNBC experiments.
These were all cellular neighborhood guided samples, meaning that I collected and area of 25000um^2 x 5um thick tissue (about 100 cells).
Some samples are tumour cells and some are immune cells.
I purposefully added some empty samples.
I am happy to test things and parameters out.
Perhaps there can be a more straightforward way to translate DIANN GUI params to params.yml (this might be version dependent :/ ).
Thank you for your efforts building this amazing pipeline, excited to push it to its full potential.
Command used and terminal output
Relevant files
No response
System information
Nextflow: 25.10.4 build 11173
Hardware: Windows Workstation, and SLURM HPC
Container engine: Singularity
OS: Windows for local, Linux on HPC
quantmsdiann hash: 4605642
@josenimo I was doing a bit of clauding using the param files; this is what I got:
Mass accuracy strategy - mass_acc_automatic: true + quick_mass_acc: true vs the GUI's pinned --mass-acc 15 --mass-acc-ms1 15. On timsTOF this is the one with documented impact (per the recent Bruker warning commits). @vdemichev could be this the problem; that quick_mass_acc can lock in too-tight or too-loose windows file-by-file
--smart-profiling — set by the GUI, never emitted by the pipeline.
--peak-center — same.
Suggested fix for the user shrinks to: turn off auto mass accuracy (mass_acc_automatic: false, keep 15/15) and pass diann_extra_args: "--smart-profiling --peak-center". That should put the pipeline run on equal footing with the GUI run. @josenimo Can you try that?
The 1.8.1 GUI log seems fine. So I assume numbers are real.
@vdemichev could be this the problem; that quick_mass_acc can lock in too-tight or too-loose windows file-by-file
Extremely unlikely. I cannot imagine this having such an effect. That said, on timsTOFs always OK to fix to 15ppm anyway. I think when we developed the pipeline there was no timsTOF support in quantms at all, right? That's why we did not discuss fixing mass accs.
--smart-profiling — set by the GUI, never emitted by the pipeline.
--peak-center — same.
Neither will have a non-negligible effect.
I looked at the uploaded quantms output, I cannot figure out where the DIA-NN logs are. I will likely be able to explain this if I see the logs.
@vdemichev No conversion is happening; here the log from the repo that @josenimo provided:
DIA-NN 1.8.1 (Data-Independent Acquisition by Neural Networks)
Compiled on Apr 15 2022 08:45:18
Current date and time: Mon May 4 17:05:55 2026
Logical CPU cores: 32
diann --lib empirical_library.speclib --fasta UP000005640_9606_wConts.fasta --f Bluto_20251020_JoN_P37E24_991_Cancer_cnDVP_1_S3-A7_1_14821.d --f Bluto_20251020_JoN_P37E35_991_Cancer_cnDVP_5_S3-B6_1_14823.d --f Bluto_20251020_JoN_P37E45_991_Cancer_cnDVP_3_S3-C6_1_14825.d --f Bluto_20251020_JoN_P37E56_991_Cancer_cnDVP_4_S3-D9_1_14827.d --f Bluto_20251024_JoN_P37E01_992_Cancer_cnDVP_01_S6-H7_1_14847.d --f Bluto_20251024_JoN_P37E01_992_Cancer_cnDVP_02_S6-H10_1_14849.d --f Bluto_20251024_JoN_P37E01_992_Cancer_cnDVP_03_S6-H11_1_14851.d --f Bluto_20251024_JoN_P37E01_992_Cancer_cnDVP_04_S6-C3_1_14845.d --f Bluto_20251024_JoN_P37E01_992_Cancer_cnDVP_05_S6-C2_1_14843.d --f Bluto_20251027_JoN_P37E01_991_Immune_cnDVP_01_S4-D12_1_14955.d --f Bluto_20251027_JoN_P37E01_991_Immune_cnDVP_02_S4-D7_1_14957.d --f Bluto_20251027_JoN_P37E01_991_Immune_cnDVP_03_S4-C2_1_14961.d --f Bluto_20251027_JoN_P37E01_991_Immune_cnDVP_04_S4-E9_1_14965.d --f Bluto_20251027_JoN_P37E01_991_Immune_cnDVP_05_S4-G6_1_14969.d --f Bluto_20251027_JoN_P37E01_991_Immune_cnDVP_06_S4-B4_1_14971.d --f Bluto_20251027_JoN_P37E01_992_Cancer_cnDVP_06_S4-B6_1_14949.d --f Bluto_20251027_JoN_P37E01_992_Cancer_cnDVP_07_S4-B7_1_14950.d --f Bluto_20251027_JoN_P37E01_992_Cancer_cnDVP_08_S4-B8_1_14951.d --f Bluto_20251027_JoN_P37E01_992_Cancer_cnDVP_09_S4-B9_1_14952.d --f Bluto_20251027_JoN_P37E01_992_Immune_cnDVP_01_S4-D5_1_14953.d --f Bluto_20251027_JoN_P37E01_992_Immune_cnDVP_02_S4-E11_1_14959.d --f Bluto_20251027_JoN_P37E01_992_Immune_cnDVP_03_S4-D2_1_14963.d --f Bluto_20251027_JoN_P37E01_992_Immune_cnDVP_04_S4-G4_1_14966.d --f Bluto_20251027_JoN_P37E01_992_Immune_cnDVP_05_S4-G2_1_14967.d --f Bluto_20251027_JoN_P37E01_992_Immune_cnDVP_06_S4-B5_1_14972.d --threads 12 --verbose 3 --temp ./quant/ --relaxed-prot-inf --pg-level 2 --matrices --out diann_report.tsv --qvalue 0.01 --matrix-qvalue 0.01 --matrix-spec-q 0.05 --use-quant --fixed-mod Carbamidomethyl,57.021464,C --var-mod Oxidation,15.994915,M
Thread number set to 12
Highly heuristic protein grouping will be used, to reduce the number of protein groups obtained; this mode is recommended for benchmarking protein ID numbers; use with caution for anything else
Implicit protein grouping: genes; this determines which peptides are considered 'proteotypic' and thus affects protein FDR calculation
Precursor/protein x samples expression level matrices will be saved along with the main report
Output will be filtered at 0.01 FDR
Precursor/protein x sample matrices will be filtered at 0.01 precursor & protein-level FDR
Protein x sample matrices will be also filtered using run-specific protein q-value
Existing .quant files will be used
Modification Carbamidomethyl with mass delta 57.0215 at C will be considered as fixed
Modification Oxidation with mass delta 15.9949 at M will be considered as variable
DIA-NN will optimise the mass accuracy automatically using the first run in the experiment. This is useful primarily for quick initial analyses, when it is not yet known which mass accuracy setting works best for a particular acquisition scheme.
25 files will be processed
[0:00] Loading spectral library empirical_library.speclib
[0:00] Library annotated with sequence database(s): UP000005640_9606_wConts.fasta
[0:00] Spectral library loaded: 2985 protein isoforms, 2832 protein groups and 6848 precursors in 6547 elution groups.
[0:00] Loading protein annotations from FASTA UP000005640_9606_wConts.fasta
[0:00] Annotating library proteins with information from the FASTA database
[0:00] Protein names missing for some isoforms
[0:00] Gene names missing for some isoforms
[0:00] Library contains 2984 proteins, and 2935 genes
[0:00] Initialising library
[0:00] Cross-run analysis
[0:00] Reading quantification information: 25 files
[0:00] Quantifying peptides
[0:01] Assembling protein groups
[0:01] Quantifying proteins
[0:01] Calculating q-values for protein and gene groups
[0:01] Calculating global q-values for protein and gene groups
[0:01] Writing report
[0:04] Report saved to diann_report.tsv.
[0:04] Saving precursor levels matrix
[0:04] Precursor levels matrix (1% precursor and protein group FDR) saved to diann_report.pr_matrix.tsv.
[0:04] Saving protein group levels matrix
[0:04] Protein group levels matrix (1% precursor FDR and protein group FDR) saved to diann_report.pg_matrix.tsv.
[0:04] Saving gene group levels matrix
[0:04] Gene groups levels matrix (1% precursor FDR and protein group FDR) saved to diann_report.gg_matrix.tsv.
[0:04] Saving unique genes levels matrix
[0:04] Unique genes levels matrix (1% precursor FDR and protein group FDR) saved to diann_report.unique_genes_matrix.tsv.
[0:04] Stats report saved to diann_report.stats.tsv
Finished
I agree, this is less than some single samples had on the GUI run [0:00] Spectral library loaded: 2985 protein isoforms, 2832 protein groups and 6848 precursors in 6547 elution groups.
command.log from the ASSEMBLE_EMPIRICAL_LIBRARY process, for DIANN 2.3.2
DIA-NN 2.3.2 Academia (Data-Independent Acquisition by Neural Networks)
Compiled on Jan 22 2026 05:01:57
Current date and time: Mon May 4 15:57:03 2026
Logical CPU cores: 32
Thread number set to 4
The spectral library (if generated) will retain the original spectra but will include empirically-aligned RTs
Existing .quant files will be used
Mass accuracy will be determined separately for different runs
Scan windows will be inferred separately for different runs
A spectral library will be generated
Modification Carbamidomethyl with mass delta 57.0215 at C will be considered as fixed
Modification Oxidation with mass delta 15.9949 at M will be considered as variable
DIA-NN will automatically optimise the mass accuracy for each acquisition, use this mode only to determine the approximate optimal settings
WARNING: combining reuse of .quant files with automatic optimisation of mass accuracies or scan window will lead to results that are different from those of the original analysis that produced the .quant files and is strongly not recommended
WARNING: protein inference is enabled but no FASTA provided - is this intended?
WARNING: peptidoform scoring enabled because variable modifications have been declared; to disable, use --no-peptidoforms
The following variable modifications will be localised: UniMod:35
25 files will be processed
[0:00] Loading spectral library silicolibrarygeneration-lib.predicted.speclib
[0:01] Library annotated with sequence database(s): UP000005640_9606_wConts.fasta
[0:01] Gene names missing for some isoforms
[0:01] Library contains 20196 proteins, and 19841 genes
[0:02] Spectral library loaded: 20197 protein isoforms, 24707 protein groups and 1967413 precursors in 1076460 elution groups.
[0:04] Initialising library
[0:10] Library RAM: 210MB (core), 269MB (fragments), 51MB (names), 8MB (protein_ids), 0MB (proteins)
WARNING: it is strongly recommended to enable MBR when analysing with a large library, if this is a quantitative analysis
[0:10] Cross-run analysis
[0:10] Reading quantification information: 25 files
[0:16] Averaged recommended settings for this experiment: Mass accuracy = 10ppm, MS1 accuracy = 20ppm, Scan window = 8
[0:18] Quantifying peptides
[0:32] Quantification parameters: 0.377837, 0.0045897, 0.00364257, 0.162628, 0.309544, 0.247538, 0.220062, 0.14895, 0.153145, 0.206395, 0.143001, 0.166704, 0.58776, 0.153642, 0.223596, 0.117568
[0:34] Assembling protein groups
[0:35] Quantifying proteins
[0:35] Calculating q-values for protein and gene groups
[0:38] Calculating global q-values for protein and gene groups
[0:38] Protein groups with global q-value <= 0.01: 2364
[0:39] Compressed report saved to report.parquet. Use R 'arrow' or Python 'PyArrow' package to process
[0:39] Stats report saved to report.stats.tsv
WARNING: FASTA database not provided, protein PTM sites will not be quantified.
[0:39] Generating spectral library:
[0:39] 7550 target and 79 decoy precursors saved
[0:39] Spectral library saved to empirical_library.parquet
The following warnings or errors (in alphabetic order) were detected at least the indicated number of times:
WARNING: FASTA database not provided, protein PTM sites will not be quantified. : 1
WARNING: combining reuse of .quant files with automatic optimisation of mass accuracies or scan window will lead to results that are different from those of the original analysis that produced the .quant files and is strongly not recommended : 1
WARNING: it is strongly recommended to enable MBR when analysing with a large library, if this is a quantitative analysis : 1
WARNING: peptidoform scoring enabled because variable modifications have been declared; to disable, use --no-peptidoforms : 1
WARNING: protein inference is enabled but no FASTA provided - is this intended? : 1
Finished
Thanks @josenimo; Can you provide this: assemble_empirical_library.log
I think; what @vdemichev also ask for was the first iteration in any of the files; the preliminary_analysis If you can provide some of the logs files for the preliminary analysis it would be also great.
No, is that quantmsdiann and diann only takes one enzyme, as many of the tools, in fact, not really an enzyme but a cut pattern. In your SDRF, you provided two enzymes, Lys-C and Trypsin; first Lys-C and then Trypsin; the parser takes the first one (which is wrong); but we don't have a mechanism to combine both into a pattern; I will try to implement it. This is mainly common in this case.
For you, if you switch the enzymes in SDRF, it should be enough; first, Trypsin and then Lys-C. Thanks a lot @vdemichev for your feedback and help. Good we catch the bug.
Lys-C + trypsin is effectively just a 'better trypsin' from the software perspective, so should be specified as trypsin.
Yasset, maybe change strategy to 'combine all specificities'? And if incompatible, issue a warning (which will warrant increasing missed cleavages).
We actually, when reanalyzing data, we annotate using Trypsin or Trypsin/P, but it is actually good for the lab to annotate the Lys-C in the SDRF even if they don't have an effect.
I will improve the logic of sdrf conversion. PR coming.
So nice to see this solve PR ongoing in here; Thanks @vdemichev for helping us here; I actually fixed already and it will be included in 2.1.0 version. Thanks @josenimo and hope you can convince the team now; to go for quantmsdiann. 😆!!!!
Activity
ypriverol commentedon May 6, 2026
@josenimo I was doing a bit of clauding using the param files; this is what I got:
Mass accuracy strategy -
mass_acc_automatic: true + quick_mass_acc: true vs the GUI's pinned--mass-acc 15 --mass-acc-ms1 15. On timsTOF this is the one with documented impact (per the recent Bruker warning commits). @vdemichev could be this the problem; that quick_mass_acc can lock in too-tight or too-loose windows file-by-file--smart-profiling — set by the GUI, never emitted by the pipeline.
--peak-center — same.
Suggested fix for the user shrinks to: turn off auto mass accuracy (mass_acc_automatic: false, keep 15/15) and pass diann_extra_args: "--smart-profiling --peak-center". That should put the pipeline run on equal footing with the GUI run. @josenimo Can you try that?
vdemichev commentedon May 6, 2026
Hi @josenimo ,
The 1.8.1 GUI log seems fine. So I assume numbers are real.
Extremely unlikely. I cannot imagine this having such an effect. That said, on timsTOFs always OK to fix to 15ppm anyway. I think when we developed the pipeline there was no timsTOF support in quantms at all, right? That's why we did not discuss fixing mass accs.
Neither will have a non-negligible effect.
I looked at the uploaded quantms output, I cannot figure out where the DIA-NN logs are. I will likely be able to explain this if I see the logs.
Best,
Vadim
vdemichev commentedon May 6, 2026
For 2.3.2:
params_2026-05-04_14-16-15.json:
If it's .d converted to mzML, output will be nonsense.
Will also lead to nonsense.
P.S. I cannot seem to download any files from MDC storage
ypriverol commentedon May 6, 2026
@vdemichev No conversion is happening; here the log from the repo that @josenimo provided:
ypriverol commentedon May 6, 2026
This is the param some of them do not have effect; because they are overrided by SDRF:
docs: incorporate DIA-NN author feedback and rename callout
ypriverol commentedon May 6, 2026
@josenimo for the log files of the PRELIMINARY/INDIVIDUAL steps you can compress all the logs files in the pipeline and provided here.
vdemichev commentedon May 6, 2026
Empirical lib has very few precursors, so issue is with the first pass
josenimo commentedon May 6, 2026
I agree, this is less than some single samples had on the GUI run
[0:00] Spectral library loaded: 2985 protein isoforms, 2832 protein groups and 6848 precursors in 6547 elution groups.Will post log files tomorrow first thing
josenimo commentedon May 6, 2026
will check the nextflow
workfolder to see what the ASSEMBLE_EMPIRICAL_LIBRARY command didjosenimo commentedon May 7, 2026
command.logfrom theASSEMBLE_EMPIRICAL_LIBRARYprocess, for DIANN 2.3.2josenimo commentedon May 7, 2026
here is the
.command.shshowing the command line DIANN command used with its parameters:command.sh
ypriverol commentedon May 7, 2026
Thanks @josenimo; Can you provide this: assemble_empirical_library.log
I think; what @vdemichev also ask for was the first iteration in any of the files; the preliminary_analysis If you can provide some of the logs files for the preliminary analysis it would be also great.
24 remaining items
josenimo commentedon May 7, 2026
ah I see the difference,
in local DIANN 1.8.1 run log:
line 18:
In silico digest will involve cuts at K*,R*in HPC DIANNs:
In silico digest will involve cuts at K*right?
ypriverol commentedon May 7, 2026
Yes; checking the SDRF parsing, you defined two enzymes in the SDRF:
NT=Lys-C;AC=MS:1001309NT=Trypsin;AC=MS:1001251The parsing is picking the first one Lys-C; Maybe in this case, we need to define Trypsin, but I will try to see if we can avoid the error.
josenimo commentedon May 7, 2026
to be sure I understand,
then it is a
sdrf-pipelinesissue not parsing the SDRF table correctlyypriverol commentedon May 7, 2026
No, is that quantmsdiann and diann only takes one enzyme, as many of the tools, in fact, not really an enzyme but a cut pattern. In your SDRF, you provided two enzymes, Lys-C and Trypsin; first Lys-C and then Trypsin; the parser takes the first one (which is wrong); but we don't have a mechanism to combine both into a pattern; I will try to implement it. This is mainly common in this case.
For you, if you switch the enzymes in SDRF, it should be enough; first, Trypsin and then Lys-C. Thanks a lot @vdemichev for your feedback and help. Good we catch the bug.
vdemichev commentedon May 7, 2026
Lys-C + trypsin is effectively just a 'better trypsin' from the software perspective, so should be specified as trypsin.
Yasset, maybe change strategy to 'combine all specificities'? And if incompatible, issue a warning (which will warrant increasing missed cleavages).
ypriverol commentedon May 7, 2026
We actually, when reanalyzing data, we annotate using Trypsin or Trypsin/P, but it is actually good for the lab to annotate the Lys-C in the SDRF even if they don't have an effect.
I will improve the logic of sdrf conversion. PR coming.
josenimo commentedon May 7, 2026
meanwhile, will rerun with SRDF table with only trypsin
docs: add 'Reproducing a DIA-NN GUI run' section to usage.md
josenimo commentedon May 7, 2026
It works as expected, great stuff everyone :D
vdemichev commentedon May 7, 2026
Super!
ypriverol commentedon May 7, 2026
So nice to see this solve PR ongoing in here; Thanks @vdemichev for helping us here; I actually fixed already and it will be included in 2.1.0 version. Thanks @josenimo and hope you can convince the team now; to go for quantmsdiann. 😆!!!!
josenimo commentedon May 7, 2026
they are now fully back on board ;)
also great marketing to actually use the latest DIANN version
ypriverol commentedon May 7, 2026
Issue tackle in #89