SLP is a benchmark and modeling repository for predicting synthetic lethality (SL) and selective gene dependencies across held-out gene families and cancer cell lines.
The repository has two main components:
- SLB (
src/slbench): Synthetic Lethality Benchmark, a unified dataset and evaluation pipeline covering human cell-line screens and combinatorial genetic interaction screens across multiple species. - SLp (
src/slpandmodels/): Baseline predictors, literature reimplementations, and candidate models evaluated on SLB.
Each example in the benchmark is a (state_id, gene_b) pair:
state_iddefines a cellular context, consisting of a baseline profile (X0) and a set of pre-existing genetic perturbations (P0).gene_bis a candidate target gene being perturbed in that context.- The model predicts
dy, the continuous fitness change whengene_bis knocked out instate_id(more negative values indicate stronger fitness defects or synthetic lethality).
This representation combines single-gene cancer dependency screens and combinatorial double-knockout screens in a single schema:
| Assay Type | State (S) |
Candidate (B) |
Ground Truth Label |
|---|---|---|---|
| DepMap (1,178 human cancer cell lines) | X0: cell-line gene expression and copy number (x0_human.npz). P0: damaging mutations and deep deletions (lesions.parquet). |
Any gene targeted by CRISPR knockout. | Selective dependency: fitness effect relative to the cell line's overall shift, the gene's mean effect in the same lineage, and the screening library offset (computed on training lines). Positive if <= -0.8 below expectation and <= -6 robust SDs; negative if > -0.2. |
| Combinatorial pair screens (human in 60 contexts, S. cerevisiae, S. pombe, and 4 minor species) | X0: parental cell line profile (human) or wild-type background. P0: parental lesions plus a query gene A knocked out (after in states.parquet). |
The second gene in the pair. | Consensus genetic interaction call from published screens where the highest-reliability source has reliability >= 0.78 (see src/slbench/screens.py). |
states.parquet: Metadata for each state (state_id,species,assay,x0_id,lineage, andafterlisting genes knocked out after baseline measurement).lesions.parquet: Damaging mutations and deep deletions per state.x0_human.npz: Baseline gene expression and copy-number matrices for human cell lines.aux_deleteome.npz: Transcriptome profiles across 1,481 single-gene deletion strains in S. cerevisiae.orthologs.parquetandfamilies.parquet: Cross-species orthology mappings and within-species gene family assignments.train/<species>.parquet: Training data per species (state_id,gene_b,dy,sel,label,label_scored).{dev,heldout,test}_inputs.parquet: Evaluation split inputs (row_id,state_id,species,assay,gene_b,col).dev_labels.parquet: Public labels for thedevsplit.
Splits are constructed to evaluate generalization to unseen gene families (in pair screens) and unseen cell lines (in DepMap):
- Pair screens: Genes are grouped into within-species families based on connected components of paralogs and overlapping ORFs. Each gene pair is assigned to the highest-priority split of its two genes (
test > heldout > dev > train). Eligible gene families are partitioned 40% totest, 30% toheldout, and 30% todev. Families with 100 or more genes (and genes listed inreference/pinned_train) are assigned totrain. Because gene families are defined within each species rather than across orthologs, orthologs of held-out genes in other species remain intrainto allow cross-species transfer. - DepMap: Cell lines are partitioned into
test(17%),heldout(12%),dev(11%), andtrain(remaining lines). - Label access:
devlabels are included in the repository workspace for tuning and optional final refitting.heldoutandtestlabels are stored separately inSLB_SEALEDand accessed only throughslbench eval, which logs every evaluation and enforces query budgets (1,200heldoutand 60testevaluations per build, with at most onetestevaluation per model).
Models may use the training partition, external biological datasets (such as perturbation screens, expression compendia, protein interaction networks, literature, and pretrained foundation models), and dev labels for a final refit, subject to three constraints:
- No held-out outcome leakage: Models must not access
SLB_SEALED, withhelddev/heldout/testinteraction values, the raw pair-screen source tables listed insrc/slbench/screens.pyfor evaluated pairs, or DepMap gene-effect scores for held-out cell lines. Single-gene perturbation profiles of evaluated genes (such as deletion expression profiles or fitness fingerprints on training cell lines) are permitted, whereas pairwise synthetic lethality or genetic interaction measurements involving held-out genes are excluded. - No assay-design features: Models must not use features derived from the sparsity pattern of which gene pairs were chosen for screening (e.g., node degree in the assayed pair graph, partner-set overlap, or the composition of evaluation rows), as these reflect experimental selection rather than biology.
- Contamination reporting: Pretrained models or text corpora whose training data may overlap published screens are permitted, provided their performance on post-cutoff screens is measured and documented in
models/<name>/modelcard.md. External pairwise datasets can be checked for overlap with held-out families viauv run slbench check-leakage.
Evaluation is restricted to rows with a defined binary ground-truth label (label_scored). In DepMap, candidate genes with high copy number in a given cell line (log2 ratio + 1 > 1.7) are excluded from scoring to avoid confounding from copy-number cutting toxicity.
For each split, the scorer computes within-state AUROC after removing main effects and adjusting for propensity strata:
- Interaction residualization: Within each connected component of scored
(state, gene)entries, predictions are standardized by median and median absolute deviation (MAD), clipped to+-1e6, and residualized by jointly fitting and subtracting a state main effect, a gene main effect, and a rank-1state x genebilinear term. This ensures that predictions depending only on state-level features, gene-level features, or a single multiplicative factor evaluate to zero residual regardless of matrix sparsity. - Within-gene percentile normalization: Residuals are converted to percentiles across states for the same candidate gene
Bwithin each assay, removing per-gene scale differences. - Propensity stratification: Within each state, positive and negative candidates are compared only within matched propensity strata (
src/slbench/propensity.py). Strata are formed by crossing out-of-fold risk bands (estimated from training positive rate, baseline expression and copy number in DepMap, and experimental design covariates in pair screens) with training positive-rate terciles, context measurement counts, and (for DepMap) expression and copy-number quintiles. Each assay uses the finest band width that retains at least half of its positive labels in scorable states. - Within-state AUROC: Mann-Whitney U statistics are summed across strata to produce a single AUROC per state. States require at least 20 valid positive-negative comparisons to be scored, and an assay requires at least 5 scored states and 20 positives.
- Hierarchical aggregation: State AUROCs are averaged within each assay weighted by
sqrt(positives), and assay scores are averaged within each species weighted bysqrt(positives). The headline SLB score is the unweighted mean across human, S. cerevisiae, and S. pombe.
Confidence intervals (95%) are estimated via Poisson bootstrap over gene families (for pair screens) and cell lines (for DepMap). Output on heldout and test is rounded to three decimal places.
Because experimental screens have measurement noise, a noise-free model cannot achieve an AUROC of 1.0. Empirical replicate correlations (r) and the corresponding simulated noise-free AUROC ceilings are shown below:
| Assay | Replicate Comparison | Replicate r |
Estimated Noise-Free AUROC |
|---|---|---|---|
| DepMap | Independent CRISPR libraries (Avana vs. KY) across 214 shared cell lines | 0.85 | ~0.94 |
| HAP1 (Billmann 2026) | Independent replicate screens of identical query knockouts | 0.87 | ~0.96 |
| Pair screens (all scored calls) | Minimum source reliability threshold (reference/label_reliability.tsv) |
>= 0.78 | >= 0.91 |
On dev, evaluating the ground-truth binary or continuous labels directly (the oracle) yields a headline score of 0.974 / 0.966.
# Build the benchmark tables from raw sources in data/raw
uv run slbench build
# Generate predictions on dev using the label-free prior baseline
uv run slp prior --split dev --out runs/ref/prior_dev.parquet
# Score predictions on dev with 200 bootstrap resamples
uv run slbench eval runs/ref/prior_dev.parquet --split dev --boot 200
# Compare two prediction files on heldout
uv run slbench compare A.parquet B.parquet --split heldoutPrediction files must be Parquet tables containing row_id (matching the split's input file) and a finite numeric dy (|dy| < 1e12) for every input row. Evaluation takes roughly 30 seconds per split.
Full evaluation logs for all models are maintained in leaderboard.md. The built-in reference baselines are:
prior: A label-free heuristic that scores each candidate geneBby averaging its percentile rank across four relationships to the perturbed genes in stateS: paralog identity, high-confidence protein-protein interaction, Gene Ontology similarity, and ESM-2 embedding cosine similarity.random: Uniform random predictions.oracle: Ground-truth labels (label_scored/ continuoussel) evaluated ondev.
| Model | Dev | Dev 95% CI | Heldout | Heldout 95% CI | Human (Dev / Heldout) | S. cerevisiae (Dev / Heldout) | S. pombe (Dev / Heldout) |
|---|---|---|---|---|---|---|---|
prior |
0.520 | 0.503–0.545 | 0.543 | 0.514–0.570 | 0.479 / 0.506 | 0.566 / 0.568 | 0.516 / 0.556 |
random (30 seeds, dev) |
0.502 | SD 0.009 | - | - | SD 0.013 | SD 0.007 | SD 0.019 |
oracle (binary / continuous, dev) |
0.974 / 0.966 | - | - | - | 0.985 / 0.982 | - | - |
Verification tests in tests/test_bench.py and tests/test_score.py confirm that non-interaction shortcuts (including constant predictions, training positive rates, mean selective effects, measurement frequency, baseline expression/copy number, and partner-set degree or overlap) score near chance (~0.500).
All models in the table below were trained on the SLB training split and evaluated once on the sealed test split (scripts/rank.py).
oursentries are models developed in this repository.lit.entries are published synthetic lethality methods, cancer-genomics pipelines, and pretrained biological or text foundation models (using released weights with a trained pair head, authors' code, or faithful reimplementations as documented in eachmodels/<name>/modelcard.md).- Several published cancer-genomics methods only support human data; their predictions on yeast splits default to a constant (0.500 AUROC), so their performance on human assays should be read from the
test humancolumn.
| # | model | kind | paper / source | implementation | test (95% CI) | heldout (95% CI) | test human / scer / spom |
|---|---|---|---|---|---|---|---|
| 1 | SLATE | ours | Kronecker GP + two-tower over gene kernels and single-gene profiles | 0.690 (0.667-0.713) | 0.706 (0.681-0.728) | 0.715 / 0.668 / 0.687 | |
| 2 | sl_virtualcell_r3b1 | ours | Kronecker GP + two-tower over gene kernels and single-gene profiles (r3b1) | 0.689 (0.668-0.711) | 0.704 (0.677-0.730) | 0.718 / 0.674 / 0.675 | |
| 3 | sl_virtualcell_r2b1 | ours | Kronecker GP + two-tower over gene kernels and single-gene profiles (r2b1) | 0.688 (0.664-0.710) | 0.701 (0.675-0.728) | 0.715 / 0.674 / 0.675 | |
| 4 | sl_reasoner_r2b1 | ours | Kernel regression over gene views and perturbation fingerprints | 0.678 (0.657-0.701) | 0.684 (0.660-0.707) | 0.690 / 0.668 / 0.674 | |
| 5 | ensemble_chorus_v5 | ours | 6-model per-block ensemble | 0.675 (0.654-0.699) | 0.684 (0.657-0.709) | 0.694 / 0.646 / 0.685 | |
| 6 | kron_gp_lattice | ours | Masked-Kronecker GP + two-tower over label-free gene kernels | 0.664 (0.643-0.685) | 0.676 (0.650-0.704) | 0.699 / 0.643 / 0.649 | |
| 7 | sl_worldmodel_r2b1 | ours | Gene-set transformer + kernel ridge over perturbation profiles | 0.664 (0.648-0.684) | 0.662 (0.641-0.686) | 0.653 / 0.671 / 0.670 | |
| 8 | jepa_cell_phenome | ours | Nearest-gene vote LightGBM + cold-start pair factorization | 0.658 (0.632-0.682) | 0.659 (0.633-0.681) | 0.663 / 0.638 / 0.674 | |
| 9 | fba_synth_attn | ours | Nearest-gene vote LightGBM with learned attention | 0.658 (0.637-0.687) | 0.659 (0.628-0.687) | 0.676 / 0.636 / 0.662 | |
| 10 | lit_text_qwen3_4b | lit. foundation | Qwen3-4B text embedding (Qwen Team, 2025 (Qwen3)) | released weights + our pair head | 0.591 (0.567-0.616) | 0.598 (0.569-0.625) | 0.569 / 0.596 / 0.607 |
| 11 | lit_text_harrier_0p6b | lit. foundation | Harrier-OSS 0.6B (Microsoft, 2026 (harrier-oss-v1)) | released weights + our pair head | 0.587 (0.563-0.610) | 0.603 (0.572-0.632) | 0.563 / 0.595 / 0.602 |
| 12 | lit_text_embeddinggemma2 | lit. foundation | EmbeddingGemma 2 (Google, 2026 (EmbeddingGemma)) | released weights + our pair head | 0.584 (0.557-0.611) | 0.595 (0.567-0.625) | 0.554 / 0.581 / 0.615 |
| 13 | lit_sl2mf | lit. SL-ML | Liu et al., IEEE/ACM TCBB 2020 ("SL2MF: Predicting synthetic… | 0.584 (0.563-0.607) | 0.588 (0.559-0.614) | 0.527 / 0.596 / 0.630 | |
| 14 | lit_text_medcpt | lit. foundation | MedCPT article encoder (Jin et al., Bioinformatics 2023 (Med… | released weights + our pair head | 0.583 (0.560-0.603) | 0.589 (0.556-0.617) | 0.556 / 0.587 / 0.606 |
| 15 | lit_text_biogpt | lit. foundation | BioGPT (Luo et al., Briefings in Bioinformatics 2022 (BioGPT)) | released weights + our pair head | 0.582 (0.559-0.606) | 0.602 (0.569-0.631) | 0.546 / 0.597 / 0.604 |
| 16 | lit_label_vote | lit. genomics | heuristic (nearest-neighbour transfer of labels; the baselin… | own heuristic | 0.575 (0.549-0.596) | 0.584 (0.563-0.612) | 0.545 / 0.581 / 0.600 |
| 17 | lit_gcatsl | lit. SL-ML | Long et al., Bioinformatics 2021 ("Graph contextualized atte… | port of authors' code | 0.571 (0.550-0.594) | 0.556 (0.532-0.584) | 0.528 / 0.589 / 0.595 |
| 18 | lit_protrek_650m_seq | lit. foundation | ProTrek 650M (sequence encoder) (Su et al., 2024 (ProTrek)) | released weights + our pair head | 0.570 (0.547-0.596) | 0.567 (0.539-0.594) | 0.538 / 0.573 / 0.600 |
| 19 | lit_esm2_3b | lit. foundation | ESM-2 3B (Lin et al., Science 2023) | released weights + our pair head | 0.567 (0.548-0.591) | 0.534 (0.513-0.561) | 0.551 / 0.568 / 0.583 |
| 20 | lit_grsmf | lit. SL-ML | Huang et al., BMC Bioinformatics 2019 ("Predicting synthetic… | port of authors' code | 0.565 (0.539-0.589) | 0.555 (0.526-0.580) | 0.559 / 0.574 / 0.562 |
| 21 | lit_esmc_6b | lit. foundation | ESM C 6B (EvolutionaryScale/Biohub ESM Cambrian, 2026) | released weights + our pair head | 0.563 (0.541-0.588) | 0.551 (0.527-0.575) | 0.531 / 0.557 / 0.602 |
| 22 | lit_cmfw | lit. SL-ML | Liany, Jeyasekharan, Rajan, Bioinformatics 2020 ("Predicting… | reimplementation | 0.557 (0.532-0.579) | 0.542 (0.518-0.562) | 0.520 / 0.563 / 0.588 |
| 23 | strict_track_donor | ours | Cross-species LightGBM + kernel regression | 0.551 (0.530-0.573) | 0.551 (0.521-0.577) | 0.534 / 0.566 / 0.553 | |
| 24 | prior | reference | label-free prior | 0.545 (0.528-0.568) | 0.543 (0.514-0.570) | 0.529 / 0.562 / 0.545 | |
| 25 | lit_li2019 | lit. SL-ML | Li, Lu, Zhang, Liu, Chen, Huang, Cai, J Cell Biochem 2019 ("… | reimplementation | 0.540 (0.517-0.562) | 0.538 (0.513-0.564) | 0.495 / 0.570 / 0.554 |
| 26 | lit_nt2_500m_tx | lit. foundation | Nucleotide Transformer v2 500M, transcript (Dalla-Torre et a… | released weights + our pair head | 0.533 (0.510-0.558) | 0.523 (0.502-0.551) | 0.515 / 0.506 / 0.580 |
| 27 | lit_evo2_1b_tx | lit. foundation | Evo 2 1B base, transcript (Brixi et al., 2025 (Evo 2)) | released weights + our pair head | 0.529 (0.510-0.546) | 0.513 (0.493-0.535) | 0.500 / 0.509 / 0.576 |
| 28 | lit_slmgae | lit. SL-ML | Hao et al., IEEE JBHI 2021 ("Prediction of synthetic lethal… | port of authors' code | 0.528 (0.509-0.551) | 0.542 (0.515-0.568) | 0.482 / 0.562 / 0.541 |
| 29 | lit_elisl | lit. SL-ML | Tepeli, Seale, Gonçalves, Bioinformatics 2024 ("ELISL: early… | reimplementation | 0.525 (0.511-0.541) | 0.505 (0.492-0.520) | 0.574 / 0.500 / 0.500 |
| 30 | lit_tcga_depmap | lit. genomics | Shi et al. TCGA_DEPMAP: mapping translational dependencies a… | reimplementation | 0.524 (0.510-0.538) | 0.516 (0.501-0.530) | 0.571 / 0.500 / 0.500 |
| 31 | lit_sinatra | lit. SL-ML | Jacunski, Dixon, Tatonetti, PLoS Comput Biol 2015 ("Connecti… | reimplementation | 0.523 (0.500-0.541) | 0.511 (0.486-0.539) | 0.505 / 0.535 / 0.528 |
| 32 | lit_dekegel2019 | lit. genomics | De Kegel B, Ryan CJ. Paralog buffering contributes to the va… | reimplementation | 0.520 (0.504-0.535) | 0.496 (0.481-0.510) | 0.561 / 0.500 / 0.500 |
| 33 | lit_select | lit. genomics | Lee JS et al. Synthetic lethality-mediated precision oncolog… | authors' code | 0.519 (0.506-0.534) | 0.520 (0.506-0.533) | 0.558 / 0.500 / 0.500 |
| 34 | lit_isle | lit. genomics | Lee JS et al. Harnessing synthetic lethality to predict the… | port of authors' code | 0.518 (0.504-0.531) | 0.526 (0.511-0.544) | 0.554 / 0.500 / 0.500 |
| 35 | lit_llm_qwen3_4b_yn | lit. foundation | Qwen3-4B-Instruct-2507 Yes/No prompt: the open-weights LLM Q… | reimplementation | 0.518 (0.500-0.537) | 0.516 (0.494-0.544) | 0.542 / 0.500 / 0.512 |
| 36 | lit_daisy | lit. genomics | Jerby-Arnon L et al. Predicting cancer-specific vulnerabilit… | reimplementation | 0.518 (0.505-0.533) | 0.509 (0.497-0.524) | 0.555 / 0.500 / 0.500 |
| 37 | lit_slayer2025 | lit. genomics | Cohen Z et al. SLAYER: A computational framework for identif… | port of authors' code | 0.515 (0.501-0.529) | 0.523 (0.509-0.536) | 0.544 / 0.500 / 0.500 |
| 38 | lit_tahoe_x1_70m | lit. foundation | Tahoe-x1 70M (Gandhi et al., Tahoe Therapeutics 2025) | released weights + our pair head | 0.515 (0.496-0.531) | 0.518 (0.505-0.533) | 0.544 / 0.500 / 0.500 |
| 39 | lit_state_se600m | lit. foundation | State SE-600M (Adduri et al., Arc Institute 2025 (State)) | released weights + our pair head | 0.515 (0.499-0.528) | 0.501 (0.487-0.517) | 0.545 / 0.500 / 0.500 |
| 40 | lit_paris | lit. genomics | Benfatto S et al. Uncovering cancer vulnerabilities by machi… | reimplementation | 0.512 (0.500-0.530) | 0.522 (0.503-0.535) | 0.536 / 0.500 / 0.500 |
| 41 | lit_sbsl | lit. SL-ML | Seale, Tepeli, Gonçalves, Bioinformatics 2022 ("Overcoming s… | reimplementation | 0.511 (0.497-0.523) | 0.509 (0.494-0.525) | 0.532 / 0.500 / 0.500 |
| 42 | lit_ddgcn | lit. SL-ML | Cai et al., Bioinformatics 2020 ("Dual-dropout graph convolu… | port of authors' code | 0.511 (0.489-0.536) | 0.498 (0.474-0.525) | 0.513 / 0.491 / 0.530 |
| 43 | lit_slant | lit. genomics | Benstead-Hume G et al. Predicting synthetic lethal interacti… | reimplementation | 0.510 (0.488-0.528) | 0.514 (0.494-0.535) | 0.508 / 0.521 / 0.501 |
| 44 | lit_flister2025 | lit. genomics | Flister MJ et al. Genetic mapping of all human paralog pairs… | reimplementation | 0.510 (0.497-0.524) | 0.499 (0.482-0.512) | 0.530 / 0.500 / 0.500 |
| 45 | lit_nsf4sl | lit. SL-ML | Wang et al., Bioinformatics 2022 ("NSF4SL: negative-sample-f… | port of authors' code | 0.509 (0.495-0.523) | 0.508 (0.492-0.523) | 0.527 / 0.500 / 0.500 |
| 46 | lit_mvgcnisl | lit. SL-ML | Fan et al., Front Genet 2022 (published 2023; "Multi-view gr… | port of authors' code | 0.506 (0.492-0.524) | 0.495 (0.479-0.510) | 0.518 / 0.500 / 0.500 |
| 47 | lit_kebabci2026 | lit. genomics | Kebabci N, Ajmal H, Adams DJ, Ryan CJ. A predicted cancer de… | reimplementation | 0.504 (0.492-0.517) | 0.499 (0.485-0.513) | 0.511 / 0.500 / 0.500 |
| 48 | lit_dekegel2021 | lit. genomics | De Kegel B, Quinn N, Thompson NA, Adams DJ, Ryan CJ. Compreh… | reimplementation | 0.503 (0.483-0.526) | 0.511 (0.484-0.533) | 0.531 / 0.506 / 0.473 |
| 49 | random | reference | random scores (seed 0) | 0.503 (0.481-0.524) | 0.509 (0.484-0.531) | 0.495 / 0.507 / 0.507 | |
| 50 | lit_bisep | lit. genomics | Wappett M et al. Multi-omic measurement of mutually exclusiv… | authors' code | 0.503 (0.494-0.511) | 0.504 (0.499-0.508) | 0.509 / 0.500 / 0.500 |
| 51 | lit_kr4sl | lit. SL-ML | Zhang et al., Bioinformatics 2023 (ISMB; "KR4SL: knowledge g… | reimplementation | 0.502 (0.487-0.518) | 0.508 (0.498-0.523) | 0.507 / 0.500 / 0.500 |
| 52 | lit_haider2025 | lit. genomics | Haider S et al. The transcriptomic architecture of common ca… | reimplementation | 0.501 (0.481-0.514) | 0.503 (0.488-0.516) | 0.502 / 0.500 / 0.500 |
| 53 | lit_kg4sl | lit. SL-ML | Wang et al., Bioinformatics 2021 ("KG4SL: knowledge graph ne… | port of authors' code | 0.500 (0.485-0.518) | 0.499 (0.484-0.512) | 0.500 / 0.500 / 0.500 |
| 54 | lit_cilantro_sl | lit. foundation | CILANTRO-SL: Geneformer encodes bulk cell-line expression as… | port of authors' code | 0.500 (0.486-0.516) | 0.498 (0.482-0.514) | 0.500 / 0.500 / 0.500 |
| 55 | lit_geneformer_v2_104m | lit. foundation | Geneformer v2 104M (Theodoris et al., Nature 2023; Chen et al | released weights + our pair head | 0.495 (0.481-0.508) | 0.511 (0.497-0.529) | 0.485 / 0.500 / 0.500 |
| 56 | lit_mge4sl | lit. SL-ML | Lai et al., EMBC 2021 ("Predicting synthetic lethality in hu… | port of authors' code | 0.494 (0.475-0.518) | 0.495 (0.469-0.516) | 0.490 / 0.500 / 0.492 |
- Top-performing models: SLATE (
models/slate) achieves 0.690 [0.667, 0.713] ontestand 0.706 [0.681, 0.728] onheldout. It combines a Kronecker Gaussian process and a two-tower architecture over label-free gene similarity kernels (sequence, structure, PPI, pathway, and text embeddings) with single-gene functional profiles of the perturbed gene (DepMap training-line dependency fingerprints, yeast deletion phenomes, and matched K562/RPE1 single-gene screens). - Model ablations: Without single-gene perturbation profiles of the held-out query gene, the best single model (
kron_gp_lattice) reaches 0.664 ontest(0.676 onheldout), and a multi-model ensemble (ensemble_chorus_v5) reaches 0.675 ontest(0.684 onheldout). Adding LLM chain-of-thought reranking, scaling perturbation world-model width past ~1M parameters, or fine-tuning current single-cell foundation models did not yield statistically significant gains over the kernel and profile predictors. - Literature methods and foundation models: Among external baselines, pair heads trained on frozen gene-text embeddings (
lit_text_qwen3_4bat 0.591,lit_text_harrier_0p6bat 0.587) and protein language models (lit_protrek_650m_seqat 0.570,lit_esm2_3bat 0.567) transfer best to unseen gene families. Matrix-factorization and graph-attention SL methods that incorporate biological side information (SL2MFat 0.584,GCATSLat 0.571,GRSMFat 0.565,CMF-Wat 0.557) outperform thepriorbaseline (0.545), whereas graph methods that rely strictly on existing SL edges (DDGCN,NSF4SL,KG4SL,MGE4SL) perform near chance on held-out gene families. Human-specific cancer-genomics pipelines (ELISL,TCGA_DEPMAP,DAISY,SELECT,ISLE) achieve 0.55–0.57 AUROC on the human block, compared to 0.715 for SLATE.
- Rank-1 residualization: The scorer removes additive effects and a single rank-1 multiplicative (
state x gene) term per connected component. Higher-rank non-biological product terms are only partially removed if combined with a larger decoy factor. - Query budgets:
heldoutandtestevaluations are rate-limited (1,200 and 60 evaluations per build, respectively), but repeated queries within the budget could in principle leak information if used for iterative fitting. - Candidate coverage: In pair screens, scored states correspond to held-out query genes while candidate genes
Bbelong to training families (since propensity stratification requires training-set statistics forB). In HAP1 screens, individual query states have fewer candidate genes, resulting in wider propensity strata than in DepMap. - Cell-line donor overlap: DepMap cell lines are split by cell-line ID rather than patient donor. Across the dataset, 49 cell lines from 22 patients span multiple splits, and 21 training lines share kinship with evaluation lines (
slp.depmap_clean.drop_lines()). Excluding these training lines shifts DepMapdevAUROC by -0.002 to +0.005, which is within bootstrap noise (SD 0.004–0.007). - Baseline omics availability: Measured baseline expression and copy-number profiles (
X0) are provided only for human cell lines; yeast screens share a uniform wild-type baseline, and the four minor species have no baseline omics table.
src/slbench/: Benchmark construction and evaluation package.sources/andscreens.py: Parsers and consensus interaction calling for published pair screens.depmap.py: DepMap selective dependency calculation and cell-line data processing.families.py,homology.py,ids*.py: Paralog/ORF family clustering, orthology tables, and identifier normalization.build.pyandpropensity.py: Dataset partitioning, table assembly, and out-of-fold propensity stratification.score.py,evaluate.py, andleakage.py: Frozen evaluation metric, CLI evaluator, query logging, and external data leakage checker.
src/slp/: Reference baselines (prior,random), feature and outcome store loaders, and DepMap cleaning utilities.models/<name>/: Self-containeduvprojects for each model, including source code,pyproject.toml, andmodelcard.md.scripts/: Data-building pipelines (features/,outcomes/,evo/,lit_sl/), project scaffolding (newmodel), and evaluation table generation (rank.py).leaderboard.md: Complete table of evaluated models and scores across splits.reference/: Data download manifests and source reliability estimates.data/(gitignored): Raw downloads (data/raw), intermediate tables (data/interim), and built benchmark files (data/bench).
Code is released under the MIT License. Benchmark data and external datasets remain subject to the licenses of their original publications.