Skip to content
abiome-orgPublic

About

CV3 synthetic-lethality benchmarking.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

908 Commits

Folders and files

Repository files navigation

SLP

SLP is a benchmark and modeling repository for predicting synthetic lethality (SL) and selective gene dependencies across held-out gene families and cancer cell lines.

The repository has two main components:

  • SLB (src/slbench): Synthetic Lethality Benchmark, a unified dataset and evaluation pipeline covering human cell-line screens and combinatorial genetic interaction screens across multiple species.
  • SLp (src/slp and models/): Baseline predictors, literature reimplementations, and candidate models evaluated on SLB.

Synthetic Lethality Benchmark (SLB)

Task Formulation

Each example in the benchmark is a (state_id, gene_b) pair:

  • state_id defines a cellular context, consisting of a baseline profile (X0) and a set of pre-existing genetic perturbations (P0).
  • gene_b is a candidate target gene being perturbed in that context.
  • The model predicts dy, the continuous fitness change when gene_b is knocked out in state_id (more negative values indicate stronger fitness defects or synthetic lethality).

This representation combines single-gene cancer dependency screens and combinatorial double-knockout screens in a single schema:

Assay Type State (S) Candidate (B) Ground Truth Label
DepMap (1,178 human cancer cell lines) X0: cell-line gene expression and copy number (x0_human.npz). P0: damaging mutations and deep deletions (lesions.parquet). Any gene targeted by CRISPR knockout. Selective dependency: fitness effect relative to the cell line's overall shift, the gene's mean effect in the same lineage, and the screening library offset (computed on training lines). Positive if <= -0.8 below expectation and <= -6 robust SDs; negative if > -0.2.
Combinatorial pair screens (human in 60 contexts, S. cerevisiae, S. pombe, and 4 minor species) X0: parental cell line profile (human) or wild-type background. P0: parental lesions plus a query gene A knocked out (after in states.parquet). The second gene in the pair. Consensus genetic interaction call from published screens where the highest-reliability source has reliability >= 0.78 (see src/slbench/screens.py).

Benchmark Files (data/bench/)

  • states.parquet: Metadata for each state (state_id, species, assay, x0_id, lineage, and after listing genes knocked out after baseline measurement).
  • lesions.parquet: Damaging mutations and deep deletions per state.
  • x0_human.npz: Baseline gene expression and copy-number matrices for human cell lines.
  • aux_deleteome.npz: Transcriptome profiles across 1,481 single-gene deletion strains in S. cerevisiae.
  • orthologs.parquet and families.parquet: Cross-species orthology mappings and within-species gene family assignments.
  • train/<species>.parquet: Training data per species (state_id, gene_b, dy, sel, label, label_scored).
  • {dev,heldout,test}_inputs.parquet: Evaluation split inputs (row_id, state_id, species, assay, gene_b, col).
  • dev_labels.parquet: Public labels for the dev split.

Dataset Splits

Splits are constructed to evaluate generalization to unseen gene families (in pair screens) and unseen cell lines (in DepMap):

  • Pair screens: Genes are grouped into within-species families based on connected components of paralogs and overlapping ORFs. Each gene pair is assigned to the highest-priority split of its two genes (test > heldout > dev > train). Eligible gene families are partitioned 40% to test, 30% to heldout, and 30% to dev. Families with 100 or more genes (and genes listed in reference/pinned_train) are assigned to train. Because gene families are defined within each species rather than across orthologs, orthologs of held-out genes in other species remain in train to allow cross-species transfer.
  • DepMap: Cell lines are partitioned into test (17%), heldout (12%), dev (11%), and train (remaining lines).
  • Label access: dev labels are included in the repository workspace for tuning and optional final refitting. heldout and test labels are stored separately in SLB_SEALED and accessed only through slbench eval, which logs every evaluation and enforces query budgets (1,200 heldout and 60 test evaluations per build, with at most one test evaluation per model).

Evaluation Protocol

Models may use the training partition, external biological datasets (such as perturbation screens, expression compendia, protein interaction networks, literature, and pretrained foundation models), and dev labels for a final refit, subject to three constraints:

  1. No held-out outcome leakage: Models must not access SLB_SEALED, withheld dev/heldout/test interaction values, the raw pair-screen source tables listed in src/slbench/screens.py for evaluated pairs, or DepMap gene-effect scores for held-out cell lines. Single-gene perturbation profiles of evaluated genes (such as deletion expression profiles or fitness fingerprints on training cell lines) are permitted, whereas pairwise synthetic lethality or genetic interaction measurements involving held-out genes are excluded.
  2. No assay-design features: Models must not use features derived from the sparsity pattern of which gene pairs were chosen for screening (e.g., node degree in the assayed pair graph, partner-set overlap, or the composition of evaluation rows), as these reflect experimental selection rather than biology.
  3. Contamination reporting: Pretrained models or text corpora whose training data may overlap published screens are permitted, provided their performance on post-cutoff screens is measured and documented in models/<name>/modelcard.md. External pairwise datasets can be checked for overlap with held-out families via uv run slbench check-leakage.

Scoring Metric (src/slbench/score.py)

Evaluation is restricted to rows with a defined binary ground-truth label (label_scored). In DepMap, candidate genes with high copy number in a given cell line (log2 ratio + 1 > 1.7) are excluded from scoring to avoid confounding from copy-number cutting toxicity.

For each split, the scorer computes within-state AUROC after removing main effects and adjusting for propensity strata:

  1. Interaction residualization: Within each connected component of scored (state, gene) entries, predictions are standardized by median and median absolute deviation (MAD), clipped to +-1e6, and residualized by jointly fitting and subtracting a state main effect, a gene main effect, and a rank-1 state x gene bilinear term. This ensures that predictions depending only on state-level features, gene-level features, or a single multiplicative factor evaluate to zero residual regardless of matrix sparsity.
  2. Within-gene percentile normalization: Residuals are converted to percentiles across states for the same candidate gene B within each assay, removing per-gene scale differences.
  3. Propensity stratification: Within each state, positive and negative candidates are compared only within matched propensity strata (src/slbench/propensity.py). Strata are formed by crossing out-of-fold risk bands (estimated from training positive rate, baseline expression and copy number in DepMap, and experimental design covariates in pair screens) with training positive-rate terciles, context measurement counts, and (for DepMap) expression and copy-number quintiles. Each assay uses the finest band width that retains at least half of its positive labels in scorable states.
  4. Within-state AUROC: Mann-Whitney U statistics are summed across strata to produce a single AUROC per state. States require at least 20 valid positive-negative comparisons to be scored, and an assay requires at least 5 scored states and 20 positives.
  5. Hierarchical aggregation: State AUROCs are averaged within each assay weighted by sqrt(positives), and assay scores are averaged within each species weighted by sqrt(positives). The headline SLB score is the unweighted mean across human, S. cerevisiae, and S. pombe.

Confidence intervals (95%) are estimated via Poisson bootstrap over gene families (for pair screens) and cell lines (for DepMap). Output on heldout and test is rounded to three decimal places.

Replicate Concordance and Noise Ceiling

Because experimental screens have measurement noise, a noise-free model cannot achieve an AUROC of 1.0. Empirical replicate correlations (r) and the corresponding simulated noise-free AUROC ceilings are shown below:

Assay Replicate Comparison Replicate r Estimated Noise-Free AUROC
DepMap Independent CRISPR libraries (Avana vs. KY) across 214 shared cell lines 0.85 ~0.94
HAP1 (Billmann 2026) Independent replicate screens of identical query knockouts 0.87 ~0.96
Pair screens (all scored calls) Minimum source reliability threshold (reference/label_reliability.tsv) >= 0.78 >= 0.91

On dev, evaluating the ground-truth binary or continuous labels directly (the oracle) yields a headline score of 0.974 / 0.966.

Quickstart

# Build the benchmark tables from raw sources in data/raw
uv run slbench build

# Generate predictions on dev using the label-free prior baseline
uv run slp prior --split dev --out runs/ref/prior_dev.parquet

# Score predictions on dev with 200 bootstrap resamples
uv run slbench eval runs/ref/prior_dev.parquet --split dev --boot 200

# Compare two prediction files on heldout
uv run slbench compare A.parquet B.parquet --split heldout

Prediction files must be Parquet tables containing row_id (matching the split's input file) and a finite numeric dy (|dy| < 1e12) for every input row. Evaluation takes roughly 30 seconds per split.

Results

Reference Baselines

Full evaluation logs for all models are maintained in leaderboard.md. The built-in reference baselines are:

  • prior: A label-free heuristic that scores each candidate gene B by averaging its percentile rank across four relationships to the perturbed genes in state S: paralog identity, high-confidence protein-protein interaction, Gene Ontology similarity, and ESM-2 embedding cosine similarity.
  • random: Uniform random predictions.
  • oracle: Ground-truth labels (label_scored / continuous sel) evaluated on dev.
Model Dev Dev 95% CI Heldout Heldout 95% CI Human (Dev / Heldout) S. cerevisiae (Dev / Heldout) S. pombe (Dev / Heldout)
prior 0.520 0.503–0.545 0.543 0.514–0.570 0.479 / 0.506 0.566 / 0.568 0.516 / 0.556
random (30 seeds, dev) 0.502 SD 0.009 - - SD 0.013 SD 0.007 SD 0.019
oracle (binary / continuous, dev) 0.974 / 0.966 - - - 0.985 / 0.982 - -

Verification tests in tests/test_bench.py and tests/test_score.py confirm that non-interaction shortcuts (including constant predictions, training positive rates, mean selective effects, measurement frequency, baseline expression/copy number, and partner-set degree or overlap) score near chance (~0.500).

Test Set Benchmark and Literature Comparison

All models in the table below were trained on the SLB training split and evaluated once on the sealed test split (scripts/rank.py).

  • ours entries are models developed in this repository.
  • lit. entries are published synthetic lethality methods, cancer-genomics pipelines, and pretrained biological or text foundation models (using released weights with a trained pair head, authors' code, or faithful reimplementations as documented in each models/<name>/modelcard.md).
  • Several published cancer-genomics methods only support human data; their predictions on yeast splits default to a constant (0.500 AUROC), so their performance on human assays should be read from the test human column.
# model kind paper / source implementation test (95% CI) heldout (95% CI) test human / scer / spom
1 SLATE ours Kronecker GP + two-tower over gene kernels and single-gene profiles 0.690 (0.667-0.713) 0.706 (0.681-0.728) 0.715 / 0.668 / 0.687
2 sl_virtualcell_r3b1 ours Kronecker GP + two-tower over gene kernels and single-gene profiles (r3b1) 0.689 (0.668-0.711) 0.704 (0.677-0.730) 0.718 / 0.674 / 0.675
3 sl_virtualcell_r2b1 ours Kronecker GP + two-tower over gene kernels and single-gene profiles (r2b1) 0.688 (0.664-0.710) 0.701 (0.675-0.728) 0.715 / 0.674 / 0.675
4 sl_reasoner_r2b1 ours Kernel regression over gene views and perturbation fingerprints 0.678 (0.657-0.701) 0.684 (0.660-0.707) 0.690 / 0.668 / 0.674
5 ensemble_chorus_v5 ours 6-model per-block ensemble 0.675 (0.654-0.699) 0.684 (0.657-0.709) 0.694 / 0.646 / 0.685
6 kron_gp_lattice ours Masked-Kronecker GP + two-tower over label-free gene kernels 0.664 (0.643-0.685) 0.676 (0.650-0.704) 0.699 / 0.643 / 0.649
7 sl_worldmodel_r2b1 ours Gene-set transformer + kernel ridge over perturbation profiles 0.664 (0.648-0.684) 0.662 (0.641-0.686) 0.653 / 0.671 / 0.670
8 jepa_cell_phenome ours Nearest-gene vote LightGBM + cold-start pair factorization 0.658 (0.632-0.682) 0.659 (0.633-0.681) 0.663 / 0.638 / 0.674
9 fba_synth_attn ours Nearest-gene vote LightGBM with learned attention 0.658 (0.637-0.687) 0.659 (0.628-0.687) 0.676 / 0.636 / 0.662
10 lit_text_qwen3_4b lit. foundation Qwen3-4B text embedding (Qwen Team, 2025 (Qwen3)) released weights + our pair head 0.591 (0.567-0.616) 0.598 (0.569-0.625) 0.569 / 0.596 / 0.607
11 lit_text_harrier_0p6b lit. foundation Harrier-OSS 0.6B (Microsoft, 2026 (harrier-oss-v1)) released weights + our pair head 0.587 (0.563-0.610) 0.603 (0.572-0.632) 0.563 / 0.595 / 0.602
12 lit_text_embeddinggemma2 lit. foundation EmbeddingGemma 2 (Google, 2026 (EmbeddingGemma)) released weights + our pair head 0.584 (0.557-0.611) 0.595 (0.567-0.625) 0.554 / 0.581 / 0.615
13 lit_sl2mf lit. SL-ML Liu et al., IEEE/ACM TCBB 2020 ("SL2MF: Predicting synthetic… 0.584 (0.563-0.607) 0.588 (0.559-0.614) 0.527 / 0.596 / 0.630
14 lit_text_medcpt lit. foundation MedCPT article encoder (Jin et al., Bioinformatics 2023 (Med… released weights + our pair head 0.583 (0.560-0.603) 0.589 (0.556-0.617) 0.556 / 0.587 / 0.606
15 lit_text_biogpt lit. foundation BioGPT (Luo et al., Briefings in Bioinformatics 2022 (BioGPT)) released weights + our pair head 0.582 (0.559-0.606) 0.602 (0.569-0.631) 0.546 / 0.597 / 0.604
16 lit_label_vote lit. genomics heuristic (nearest-neighbour transfer of labels; the baselin… own heuristic 0.575 (0.549-0.596) 0.584 (0.563-0.612) 0.545 / 0.581 / 0.600
17 lit_gcatsl lit. SL-ML Long et al., Bioinformatics 2021 ("Graph contextualized atte… port of authors' code 0.571 (0.550-0.594) 0.556 (0.532-0.584) 0.528 / 0.589 / 0.595
18 lit_protrek_650m_seq lit. foundation ProTrek 650M (sequence encoder) (Su et al., 2024 (ProTrek)) released weights + our pair head 0.570 (0.547-0.596) 0.567 (0.539-0.594) 0.538 / 0.573 / 0.600
19 lit_esm2_3b lit. foundation ESM-2 3B (Lin et al., Science 2023) released weights + our pair head 0.567 (0.548-0.591) 0.534 (0.513-0.561) 0.551 / 0.568 / 0.583
20 lit_grsmf lit. SL-ML Huang et al., BMC Bioinformatics 2019 ("Predicting synthetic… port of authors' code 0.565 (0.539-0.589) 0.555 (0.526-0.580) 0.559 / 0.574 / 0.562
21 lit_esmc_6b lit. foundation ESM C 6B (EvolutionaryScale/Biohub ESM Cambrian, 2026) released weights + our pair head 0.563 (0.541-0.588) 0.551 (0.527-0.575) 0.531 / 0.557 / 0.602
22 lit_cmfw lit. SL-ML Liany, Jeyasekharan, Rajan, Bioinformatics 2020 ("Predicting… reimplementation 0.557 (0.532-0.579) 0.542 (0.518-0.562) 0.520 / 0.563 / 0.588
23 strict_track_donor ours Cross-species LightGBM + kernel regression 0.551 (0.530-0.573) 0.551 (0.521-0.577) 0.534 / 0.566 / 0.553
24 prior reference label-free prior 0.545 (0.528-0.568) 0.543 (0.514-0.570) 0.529 / 0.562 / 0.545
25 lit_li2019 lit. SL-ML Li, Lu, Zhang, Liu, Chen, Huang, Cai, J Cell Biochem 2019 ("… reimplementation 0.540 (0.517-0.562) 0.538 (0.513-0.564) 0.495 / 0.570 / 0.554
26 lit_nt2_500m_tx lit. foundation Nucleotide Transformer v2 500M, transcript (Dalla-Torre et a… released weights + our pair head 0.533 (0.510-0.558) 0.523 (0.502-0.551) 0.515 / 0.506 / 0.580
27 lit_evo2_1b_tx lit. foundation Evo 2 1B base, transcript (Brixi et al., 2025 (Evo 2)) released weights + our pair head 0.529 (0.510-0.546) 0.513 (0.493-0.535) 0.500 / 0.509 / 0.576
28 lit_slmgae lit. SL-ML Hao et al., IEEE JBHI 2021 ("Prediction of synthetic lethal… port of authors' code 0.528 (0.509-0.551) 0.542 (0.515-0.568) 0.482 / 0.562 / 0.541
29 lit_elisl lit. SL-ML Tepeli, Seale, Gonçalves, Bioinformatics 2024 ("ELISL: early… reimplementation 0.525 (0.511-0.541) 0.505 (0.492-0.520) 0.574 / 0.500 / 0.500
30 lit_tcga_depmap lit. genomics Shi et al. TCGA_DEPMAP: mapping translational dependencies a… reimplementation 0.524 (0.510-0.538) 0.516 (0.501-0.530) 0.571 / 0.500 / 0.500
31 lit_sinatra lit. SL-ML Jacunski, Dixon, Tatonetti, PLoS Comput Biol 2015 ("Connecti… reimplementation 0.523 (0.500-0.541) 0.511 (0.486-0.539) 0.505 / 0.535 / 0.528
32 lit_dekegel2019 lit. genomics De Kegel B, Ryan CJ. Paralog buffering contributes to the va… reimplementation 0.520 (0.504-0.535) 0.496 (0.481-0.510) 0.561 / 0.500 / 0.500
33 lit_select lit. genomics Lee JS et al. Synthetic lethality-mediated precision oncolog… authors' code 0.519 (0.506-0.534) 0.520 (0.506-0.533) 0.558 / 0.500 / 0.500
34 lit_isle lit. genomics Lee JS et al. Harnessing synthetic lethality to predict the… port of authors' code 0.518 (0.504-0.531) 0.526 (0.511-0.544) 0.554 / 0.500 / 0.500
35 lit_llm_qwen3_4b_yn lit. foundation Qwen3-4B-Instruct-2507 Yes/No prompt: the open-weights LLM Q… reimplementation 0.518 (0.500-0.537) 0.516 (0.494-0.544) 0.542 / 0.500 / 0.512
36 lit_daisy lit. genomics Jerby-Arnon L et al. Predicting cancer-specific vulnerabilit… reimplementation 0.518 (0.505-0.533) 0.509 (0.497-0.524) 0.555 / 0.500 / 0.500
37 lit_slayer2025 lit. genomics Cohen Z et al. SLAYER: A computational framework for identif… port of authors' code 0.515 (0.501-0.529) 0.523 (0.509-0.536) 0.544 / 0.500 / 0.500
38 lit_tahoe_x1_70m lit. foundation Tahoe-x1 70M (Gandhi et al., Tahoe Therapeutics 2025) released weights + our pair head 0.515 (0.496-0.531) 0.518 (0.505-0.533) 0.544 / 0.500 / 0.500
39 lit_state_se600m lit. foundation State SE-600M (Adduri et al., Arc Institute 2025 (State)) released weights + our pair head 0.515 (0.499-0.528) 0.501 (0.487-0.517) 0.545 / 0.500 / 0.500
40 lit_paris lit. genomics Benfatto S et al. Uncovering cancer vulnerabilities by machi… reimplementation 0.512 (0.500-0.530) 0.522 (0.503-0.535) 0.536 / 0.500 / 0.500
41 lit_sbsl lit. SL-ML Seale, Tepeli, Gonçalves, Bioinformatics 2022 ("Overcoming s… reimplementation 0.511 (0.497-0.523) 0.509 (0.494-0.525) 0.532 / 0.500 / 0.500
42 lit_ddgcn lit. SL-ML Cai et al., Bioinformatics 2020 ("Dual-dropout graph convolu… port of authors' code 0.511 (0.489-0.536) 0.498 (0.474-0.525) 0.513 / 0.491 / 0.530
43 lit_slant lit. genomics Benstead-Hume G et al. Predicting synthetic lethal interacti… reimplementation 0.510 (0.488-0.528) 0.514 (0.494-0.535) 0.508 / 0.521 / 0.501
44 lit_flister2025 lit. genomics Flister MJ et al. Genetic mapping of all human paralog pairs… reimplementation 0.510 (0.497-0.524) 0.499 (0.482-0.512) 0.530 / 0.500 / 0.500
45 lit_nsf4sl lit. SL-ML Wang et al., Bioinformatics 2022 ("NSF4SL: negative-sample-f… port of authors' code 0.509 (0.495-0.523) 0.508 (0.492-0.523) 0.527 / 0.500 / 0.500
46 lit_mvgcnisl lit. SL-ML Fan et al., Front Genet 2022 (published 2023; "Multi-view gr… port of authors' code 0.506 (0.492-0.524) 0.495 (0.479-0.510) 0.518 / 0.500 / 0.500
47 lit_kebabci2026 lit. genomics Kebabci N, Ajmal H, Adams DJ, Ryan CJ. A predicted cancer de… reimplementation 0.504 (0.492-0.517) 0.499 (0.485-0.513) 0.511 / 0.500 / 0.500
48 lit_dekegel2021 lit. genomics De Kegel B, Quinn N, Thompson NA, Adams DJ, Ryan CJ. Compreh… reimplementation 0.503 (0.483-0.526) 0.511 (0.484-0.533) 0.531 / 0.506 / 0.473
49 random reference random scores (seed 0) 0.503 (0.481-0.524) 0.509 (0.484-0.531) 0.495 / 0.507 / 0.507
50 lit_bisep lit. genomics Wappett M et al. Multi-omic measurement of mutually exclusiv… authors' code 0.503 (0.494-0.511) 0.504 (0.499-0.508) 0.509 / 0.500 / 0.500
51 lit_kr4sl lit. SL-ML Zhang et al., Bioinformatics 2023 (ISMB; "KR4SL: knowledge g… reimplementation 0.502 (0.487-0.518) 0.508 (0.498-0.523) 0.507 / 0.500 / 0.500
52 lit_haider2025 lit. genomics Haider S et al. The transcriptomic architecture of common ca… reimplementation 0.501 (0.481-0.514) 0.503 (0.488-0.516) 0.502 / 0.500 / 0.500
53 lit_kg4sl lit. SL-ML Wang et al., Bioinformatics 2021 ("KG4SL: knowledge graph ne… port of authors' code 0.500 (0.485-0.518) 0.499 (0.484-0.512) 0.500 / 0.500 / 0.500
54 lit_cilantro_sl lit. foundation CILANTRO-SL: Geneformer encodes bulk cell-line expression as… port of authors' code 0.500 (0.486-0.516) 0.498 (0.482-0.514) 0.500 / 0.500 / 0.500
55 lit_geneformer_v2_104m lit. foundation Geneformer v2 104M (Theodoris et al., Nature 2023; Chen et al released weights + our pair head 0.495 (0.481-0.508) 0.511 (0.497-0.529) 0.485 / 0.500 / 0.500
56 lit_mge4sl lit. SL-ML Lai et al., EMBC 2021 ("Predicting synthetic lethality in hu… port of authors' code 0.494 (0.475-0.518) 0.495 (0.469-0.516) 0.490 / 0.500 / 0.492

Summary of Findings

  • Top-performing models: SLATE (models/slate) achieves 0.690 [0.667, 0.713] on test and 0.706 [0.681, 0.728] on heldout. It combines a Kronecker Gaussian process and a two-tower architecture over label-free gene similarity kernels (sequence, structure, PPI, pathway, and text embeddings) with single-gene functional profiles of the perturbed gene (DepMap training-line dependency fingerprints, yeast deletion phenomes, and matched K562/RPE1 single-gene screens).
  • Model ablations: Without single-gene perturbation profiles of the held-out query gene, the best single model (kron_gp_lattice) reaches 0.664 on test (0.676 on heldout), and a multi-model ensemble (ensemble_chorus_v5) reaches 0.675 on test (0.684 on heldout). Adding LLM chain-of-thought reranking, scaling perturbation world-model width past ~1M parameters, or fine-tuning current single-cell foundation models did not yield statistically significant gains over the kernel and profile predictors.
  • Literature methods and foundation models: Among external baselines, pair heads trained on frozen gene-text embeddings (lit_text_qwen3_4b at 0.591, lit_text_harrier_0p6b at 0.587) and protein language models (lit_protrek_650m_seq at 0.570, lit_esm2_3b at 0.567) transfer best to unseen gene families. Matrix-factorization and graph-attention SL methods that incorporate biological side information (SL2MF at 0.584, GCATSL at 0.571, GRSMF at 0.565, CMF-W at 0.557) outperform the prior baseline (0.545), whereas graph methods that rely strictly on existing SL edges (DDGCN, NSF4SL, KG4SL, MGE4SL) perform near chance on held-out gene families. Human-specific cancer-genomics pipelines (ELISL, TCGA_DEPMAP, DAISY, SELECT, ISLE) achieve 0.55–0.57 AUROC on the human block, compared to 0.715 for SLATE.

Limitations and Technical Notes

  • Rank-1 residualization: The scorer removes additive effects and a single rank-1 multiplicative (state x gene) term per connected component. Higher-rank non-biological product terms are only partially removed if combined with a larger decoy factor.
  • Query budgets: heldout and test evaluations are rate-limited (1,200 and 60 evaluations per build, respectively), but repeated queries within the budget could in principle leak information if used for iterative fitting.
  • Candidate coverage: In pair screens, scored states correspond to held-out query genes while candidate genes B belong to training families (since propensity stratification requires training-set statistics for B). In HAP1 screens, individual query states have fewer candidate genes, resulting in wider propensity strata than in DepMap.
  • Cell-line donor overlap: DepMap cell lines are split by cell-line ID rather than patient donor. Across the dataset, 49 cell lines from 22 patients span multiple splits, and 21 training lines share kinship with evaluation lines (slp.depmap_clean.drop_lines()). Excluding these training lines shifts DepMap dev AUROC by -0.002 to +0.005, which is within bootstrap noise (SD 0.004–0.007).
  • Baseline omics availability: Measured baseline expression and copy-number profiles (X0) are provided only for human cell lines; yeast screens share a uniform wild-type baseline, and the four minor species have no baseline omics table.

Repository Layout

  • src/slbench/: Benchmark construction and evaluation package.
    • sources/ and screens.py: Parsers and consensus interaction calling for published pair screens.
    • depmap.py: DepMap selective dependency calculation and cell-line data processing.
    • families.py, homology.py, ids*.py: Paralog/ORF family clustering, orthology tables, and identifier normalization.
    • build.py and propensity.py: Dataset partitioning, table assembly, and out-of-fold propensity stratification.
    • score.py, evaluate.py, and leakage.py: Frozen evaluation metric, CLI evaluator, query logging, and external data leakage checker.
  • src/slp/: Reference baselines (prior, random), feature and outcome store loaders, and DepMap cleaning utilities.
  • models/<name>/: Self-contained uv projects for each model, including source code, pyproject.toml, and modelcard.md.
  • scripts/: Data-building pipelines (features/, outcomes/, evo/, lit_sl/), project scaffolding (newmodel), and evaluation table generation (rank.py).
  • leaderboard.md: Complete table of evaluated models and scores across splits.
  • reference/: Data download manifests and source reliability estimates.
  • data/ (gitignored): Raw downloads (data/raw), intermediate tables (data/interim), and built benchmark files (data/bench).

License

Code is released under the MIT License. Benchmark data and external datasets remain subject to the licenses of their original publications.

About

CV3 synthetic-lethality benchmarking.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages