Input integrity
Genome build, chromosome mapping, coordinates, REF/ALT alleles, duplicates, missingness, and sequence context.
I started PopGenLM Bench with 100 variants and a question I could test end to end: could I reproduce a genomic model’s scores and trace them back to the right alleles? Each release added a harder test. In v0.4, I ask whether adding a model score changes which linked variants I would follow up at an Arabidopsis expression locus.
The benchmark brings my population-genetics experience into the software itself, through reference checks, explicit score definitions, matched comparisons, and reproducible reports. Follow the learning story in Genomes to AI.
v0.4.0 · Latest release
v0.4 adds frozen discovery/holdout inputs, restartable model scoring, candidate-matched comparisons, and a shared CPU reproduction path. I can now inspect how a model changes the shortlist at an expression locus, including the variants it displaces. The endpoint is held-out association-signal recovery in the same study—not independent causal-nucleotide validation. Causal-SNP recovery remains unestimable: zero eligible loci.
| Magnitude rule | Held-out signals: model vs matched baseline | Top-five gains / losses |
|---|---|---|
| PlantCAD2-S · 512 bp | 18/34 versus 17/34 | 2 / 1 |
| PlantCAD2-S · 4 kb | 18/34 versus 17/34 | 3 / 2 |
| Botanic1-S · 512 bp | 17/34 versus 17/34 | 1 / 1 |
| Botanic1-S · 4 kb | 19/34 versus 17/34 | 3 / 1 |
| GPN Brassicales · 512 bp | 17/34 versus 17/34 | 1 / 1 |
| Original PlantCAD · l32 · 512 bp | 17/34 versus 17/34 | 1 / 1 |
| nemo · B. napus transfer | 8/14 versus 8/14 | 0 / 0 |
| AlphaGenome · mammalian-to-plant transfer | 2/4 versus 1/4 | 1 / 0 |
Signed PlantCAD2-S (22/34 versus 17/34) and signed Botanic1-S (21/34 versus 17/34) were explored on the original loci and remain exploratory. No fitted weights or confirmatory causal claim is made. The chromosome-block uncertainty is descriptive, with only five blocks on the full panel and two on the AlphaGenome subset. Exact metrics and protocol.
1:249256:T:C from rank 6 to 4; 1:247721:C:T leaves the shortlist, rank 3 to 6. A gene model, association evidence, LD, site conservation and model outputs share a TAIR10.1 axis. PopGenLM adaptation from frozen 1001 Genomes, TAIR10.1, GSE80744 and model evidence; the association target is a proxy, not causal truth. Exact membership · Open SVG · Download PNG.The package also shows the loss at AT4G02405: its target moves from rank 3 to 11. Showing both outcomes changed how I report the benchmark: a net gain of one signal does not mean every locus improves.
Each figure reflects the method’s output. Exact checkpoint revisions, score units, strand handling, coverage and evaluation endpoints are in the release evidence table. Original raw outputs remain distinct from oriented/harmonized scores. “Original PlantCAD-S” is a historical label: the verified checkpoint actually used is kuleshov-group/PlantCaduceus_l32, not a verified Small checkpoint.
| Model / evidence | Role and candidate coverage | Figure · focused Colab |
|---|---|---|
| PhyloP | Site-level conservation reference; 1,195/1,205 covered, ten missing; never allele-oriented | Figure · Colab |
| GPN Brassicales | 512-bp sequence likelihood; 1,205/1,205; v0.3 population evidence retained | Figure · Colab |
| Original PlantCAD-S / l32 | 512-bp sequence likelihood; 1,205/1,205; v0.3 evidence retained | Figure · Colab |
| PlantCAD2-S | Context/likelihood comparison at 512 bp and 4 kb; 1,205/1,205 each | Figure · Colab |
| Botanic1-S | Completed 512-bp/4-kb zero-shot comparison; 1,205/1,205 each; separate ATAC reproduction limited below | Figure · Colab |
| nemo | Predicted expression delta; B. napus transfer, 507/1,205; native A. thaliana resources missing | Figure · Colab |
| AlphaGenome | Mammalian-to-plant RNA-seq transfer; 194/1,205, five predefined loci, four evaluable | Figure · Colab |
Figure provenance and limits. All are PopGenLM adaptations from cached evidence. AlphaGenome adapts Avsec et al. Fig. 4b and 6b: paired REF/ALT predictions and differences, with the focal SNP and gene model. Its display is human plus-strand neuronal stem-cell polyA RNA-seq track 0, API v1 ALL_FOLDS, SDK 0.11.0; ranking retains all 667 tracks. Observed Arabidopsis expression by genotype is a separate GSE80744 panel, not measured RNA-seq coverage or mammalian-track calibration. The four eligible AlphaGenome target-rank changes are AT3G23110:6→4, AT3G53880:47→41, AT4G14840:15→13, AT4G33940:1→2. AT2G22240 is illustrative and ineligible.
The nemo figure adapts the expression-contrast idea of Rockenbach et al. Fig. 3a, showing predicted ALT−REF expression changes by promoter/terminator position and the coverage-matched 8/14 versus 8/14 result. It does not reproduce native Arabidopsis accuracy: family-safe native weights/folds, reconciled expression inputs and scalers remain missing. B. napus transfer and native A. thaliana nemo are not interchangeable.
In Barozet et al., bioRxiv v3, Fig. 4b compares predicted and observed ATAC-seq signal on held-out peaks, and Fig. 4d shows the representative Arabidopsis HY5 locus. PopGenLM’s separate Arabidopsis ATAC-seq reproduction remains blocked: verified accessibility measurements, held-out chromosome 4 peaks/labels/split, the fine-tuned ATAC head or official predictions, and matched ChromBPNet outputs are unavailable. The completed zero-shot context comparison is not a substitute for accessibility prediction.
Overview Colab · Public CPU input bundle · Executed notebooks · Download checksums.
Eight notebooks passed in a clean local CPU environment, each with a fresh kernel and archive extraction. They verify pinned checksums, reconstruct the available score-based ranks, and check paired outcomes. Local execution is distinct from hosted testing: anonymous Colab previews are checked; no notebook is claimed to have run in a signed-in Google-hosted runtime.
Code is Apache-2.0; model outputs retain their own terms. Raw Botanic scores are excluded: its public notebook verifies published rank/count arithmetic and displays the frozen LLR/context figure, rather than reconstructing omitted raw scores. AlphaGenome reproduces cached-score ranks and the illustrative track, not every full 667-track API aggregation. Exact reproduction scope, separately obtained inputs and provider notices.
This work uses the Botanic™ model developed by Living Models, available under the Botanic™ Model License; research publication, as-is, with the modifications stated in the release. AlphaGenome outputs and derivatives are subject to its Output Terms of Use. Neither provider’s outputs are relicensed as Apache data.
I froze the accession split and candidate set before scoring, retained raw allele/strand definitions, and required each model and baseline to rank the same SNPs. That decision prevents coverage differences from becoming apparent model gains. Restartable shards, reference checks, checksum-pinned inputs and one shared evaluation implementation make the result inspectable beyond RAMSES. The gain and loss loci make the biological consequence visible. The result justifies transparent follow-up shortlists and independent tests; it does not identify causal nucleotides.
Read the v0.4 Genomes to AI release note alongside my journal.
Ath_PhyloP.bedGraph.gz; Tian et al. (2020), 10.1093/nar/gkz1020. Track URL and exact hash.Producing a score is an inference step. Establishing what that score means is an evaluation problem.
DNA language models learn statistical regularities from sequence context and can assign scores to large numbers of candidate variants. But a technically valid score is not automatically biological evidence. Before a score is interpreted, its coordinate system, alleles, reference sequence, model revision, orientation, provenance, statistical behaviour, and relationship to independent biological observations need to be checked.
Population and evolutionary genomics provide evidence against which model scores can be evaluated. PopGenLM Bench is designed as the reproducible layer between model output and biological claim: validate the inputs, preserve inference provenance, orient scores explicitly, quantify associations and uncertainty, and state what the result does — and does not — support.
Genome build, chromosome mapping, coordinates, REF/ALT alleles, duplicates, missingness, and sequence context.
Model revision, upstream code, packages, parameters, device, inputs, outputs, runtime, and hashes.
Separate raw REF-to-ALT model scores from population minor-vs-major scores so the biological comparison is explicit.
Evaluate model concordance with allele frequency, conservation, and genomic context, while testing LD sensitivity and summarizing population structure.
v0.3.0 · Released
v0.3.0 added the ability to integrate frozen model outputs with genomic annotations and LD-aware descriptive checks. It integrates population-oriented GPN and PlantCAD scores with evolutionary and genomic evidence for 10,000 real Arabidopsis thaliana population variants selected using a deterministic procedure. The release is organized around a frozen 10,000 × 39 annotated master table, with exclusive genomic-context annotations, signed site-oriented PhyloP conservation, fixed physical-bin and LD-sensitivity analyses, genotype-quality summaries, and UMAP population-structure summaries.
The v0.3 release assets are frozen and hash-verified. Its primary population score is log P(minor allele) − log P(major allele). Raw model scores retain log P(ALT) − log P(REF). PhyloP is site-oriented, not allele-oriented, so its sign is not treated as an allele preference.
The sample-quality summaries contain 1,135 samples in the sample-quality table, 1,093 samples in the primary call-rate set, 996 samples in the strict set, and geographic metadata for 1,131 samples.
These are descriptive associations. They do not show that either model directly measures fitness, selection, causality, or pathogenicity. v0.3.0 claims no hypothesis-test p-values.
The public notebook loads, verifies, summarizes, and displays the frozen v0.3.0 assets. It does not rerun PlantCAD/GPN inference, VCF processing, UMAP fitting, or bootstrap production analysis.
The notebook is pinned to v0.3.0 and checks the released files before loading the descriptive summaries and displaying the frozen figures.
v0.2.0 · Released
v0.2.0 added deterministic population sampling, exact reference validation and explicit minor-versus-major score orientation. It uses deterministically selected biallelic SNPs from the 1001 Genomes Project, validates every REF allele against TAIR10.1, combines population summaries with reproducible GPN Brassicales scores, and evaluates the relationship between minor-allele frequency and model score.
The primary population-oriented score is log P(minor) − log P(major). Negative values mean that GPN assigns lower sequence likelihood to the population minor allele than to the major allele.
For variants in which ALT is the minor allele, the raw score log P(ALT) − log P(REF) already has the required orientation. For the 246 ALT-major variants, the sign is reversed. This transformation makes the population comparison explicit; it does not create the observed genome-wide association.
Across the 10,000-variant benchmark:
The result is therefore weak but reproducible concordance, not strong predictive separation. The distributions overlap extensively.
The positive association remains under stricter data filters:
| Analysis | n | Spearman ρ |
|---|---|---|
| All variants | 10,000 | 0.0611 |
| Call rate ≥ 0.95 | 6,761 | 0.0581 |
| MAC ≥ 5 | 5,057 | 0.0876 |
| Call rate ≥ 0.95 and MAC ≥ 5 | 3,279 | 0.1002 |
The association is also positive separately on chromosomes 1–5 and remains positive after excluding each chromosome in turn. This supports the conclusion that the genome-wide direction is not being driven by a single chromosome.
The release notebook is intentionally lightweight. It starts from the frozen population + GPN benchmark rather than rerunning the expensive 10,000-variant GPN inference.
It:
The notebook is pinned to the v0.2.0 release and can be run directly in Google Colab. It verifies the released artifacts before reproducing the main analysis.
v0.1.0 · Released
v0.1.0 established the engineering foundation: verify a score table, preserve allele/metadata contracts, and reproduce a report end to end. It remains an active, frozen PopGenLM Bench analysis fixture. It contains 100 deterministic SNVs with verified GPN scores, together with the score table, metadata, schema checks, descriptive statistics, bootstrap uncertainty, figures, and a CPU-compatible Colab notebook.
The important distinction is that v0.1.0 is an engineering and analysis fixture, not a sampled population benchmark. It established a reproducible analysis path before population-aware evaluation was added in v0.2.0.
v0.1.0 validates score-table structure, metadata, descriptive analysis, bootstrap uncertainty, and figure generation. It does not by itself support population-genetic or evolutionary claims.
The central design principle is separation of concerns. Model scoring, reference validation, population orientation, statistical evaluation, and biological interpretation remain distinct steps. This makes failure modes inspectable and helps prevent a model score from being promoted directly into a biological claim.
The learning path is chronological: v0.1 verified a 100-variant fixture end to end; v0.2 made the reference and population allele comparison explicit; v0.3 added evolutionary evidence and LD checks; v0.4 tests which linked SNPs a researcher would follow up. Each step adds something a researcher can verify or run; the planned stable release will make the exact version citable.
Regulatory Variant Prioritization Pilot. Candidate-matched per-locus rankings, seven evidence sources, and CPU reproduction from the public release bundle.
Evolutionary evidence. Multi-model comparison with PhyloP, genomic context, and LD sensitivity.
Population-aware benchmark. 10,000 Arabidopsis thaliana SNPs from 1001 Genomes, TAIR10.1 reference checks, and explicit minor-versus-major score orientation.
Reproducible analysis foundation. A verified 100-variant GPN fixture with validation and reproducible reporting.
Freeze the scoring rules before testing them on new, non-overlapping Arabidopsis loci. Evaluate association-signal recovery and report causal-SNP recovery if enough independently supported regulatory SNPs qualify. If the evidence is sparse, document the eligible cases and evidence gaps. Stabilize the model adapters and precomputed-score inputs, add a containerized workflow, and test whether an external researcher can reproduce the results.
Incorporate beta feedback, freeze the public data and score schemas and the evaluation protocol, and add end-to-end release checks and documented case studies. Archive the tagged software and benchmark version with a DOI so researchers can reproduce and cite the exact version they used.
git checkout v0.3.0
pip install -e ".[dev,v03]"
pytest -q
ruff check src tests scripts
sha256sum -c data/benchmarks/v0.3/SHA256SUMS.txtThe public notebook loads, verifies, summarizes, and displays frozen assets; it does not rerun PlantCAD/GPN inference, VCF processing, UMAP fitting, or bootstrap production analysis.
The released population analysis can be reproduced from the frozen artifacts without rerunning GPN inference:
git clone https://github.com/tahirali-biomics/popgenlm-bench.git
cd popgenlm-bench
git checkout v0.2.0
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -e ".[dev,viz]"
pytest -q
ruff check src tests scripts
python scripts/analyze_v02_population_gpn.py
python scripts/plot_v02_population_gpn.pyThe full v0.2.0 release is validated by 58 automated tests and GitHub Actions on Python 3.11 and 3.12.
PopGenLM Bench uses public demonstration and benchmark data. It does not contain unpublished Arabidopsis lyrata results, restricted teaching sources, student work, private credentials, or institution-owned material without permission.