CellFateBench is a scientific software and benchmark-engineering project for evaluating reasoning over single-cell genomics workflows.

The project was designed around a practical question:

How can single-cell analysis outputs be turned into reproducible benchmark tasks with public prompts, hidden answer keys, oracle outputs, scoring, calibration, Docker validation, and CI?

What CellFateBench does

Single-cell genomics workflows often produce outputs such as:

• clusters;
• embeddings;
• marker tables;
• pseudotime summaries;
• spatial patterns;
• topology summaries;
• RNA velocity layers.

Those outputs still require interpretation.

A solver may need to decide which state is likely upstream, whether a branch is terminal, whether a spatial pattern is meaningful, whether a ring-like pattern supports a cyclic claim, or whether RNA velocity evidence is strong enough to support a directionality statement.

CellFateBench focuses on that reasoning layer.

It converts single-cell analysis contexts into structured benchmark assets:

• public benchmark tasks;
• hidden answer keys;
• oracle outputs;
• deterministic validators;
• scoring outputs;
• calibration logs;
• difficulty rebalancing outputs;
• reproducible pipelines;
• Docker validation;
• GitHub Actions CI.

The project is not a notebook-only analysis. It is structured as a reproducible scientific software repository.

Repository architecture

The repository is organised around a clear separation of source code, workflow scripts, tests, benchmark assets, documentation, and generated outputs.

cellfatebench-single-cell-analysis/
├── benchmark_tasks/
│   ├── public/
│   ├── hidden/
│   ├── oracle_outputs/
│   └── calibration_logs/
├── configs/
├── data/
│   ├── raw/
│   ├── processed/
│   ├── reference/
│   └── synthetic/
├── docs/
├── results/
│   ├── figures/
│   ├── reports/
│   └── tables/
├── sample_solver_answers/
├── scripts/
├── src/cellfatebench/
├── tests/
├── Dockerfile
├── Makefile
├── environment.yml
└── README.md

The key design decision is that benchmark assets are explicit and inspectable.

Public tasks are not mixed with hidden answers. Oracle outputs are separate. Scoring code is separate from task generation. Pipelines are exposed through Makefile commands.

That structure makes the project easier to review, test, and extend.

Two benchmark layers: v1 and v2

CellFateBench currently has two layers.

This design allows the project to balance two needs:

1. controlled hidden truth for deterministic scoring;
2. public dataset context for biological realism.

v1: controlled synthetic benchmark

The v1 layer uses controlled synthetic single-cell data.

This is important because benchmark scoring requires known answers. In many real datasets, the true biological state, lineage structure, or spatial domain assignment may be uncertain. Synthetic data allows the benchmark to define hidden truth and use that hidden truth for deterministic evaluation.

The v1 dataset includes:

• 900 synthetic cells;
• 60 genes;
• designed root or progenitor state;
• transition state;
• terminal states;
• branch labels;
• pseudotime values;
• spatial coordinates;
• spatial domains;
• topology design.

Generated synthetic outputs include:

data/synthetic/synthetic_cell_metadata.csv
data/synthetic/synthetic_expression_matrix.csv
data/synthetic/synthetic_gene_metadata.csv
data/synthetic/synthetic_hidden_truth.json

The synthetic hidden-truth file is central to the v1 benchmark. It allows tasks to be scored against known answers.

v1 task families

The v1 benchmark contains three task families.

1. Trajectory and pseudotime reasoning

Files:

benchmark_tasks/public/trajectory_pseudotime_tasks.json
benchmark_tasks/hidden/trajectory_pseudotime_answers.json
benchmark_tasks/oracle_outputs/trajectory_pseudotime_oracle_outputs.json

These tasks test reasoning about:

• root-state inference;
• terminal-state inference;
• transition-state placement;
• early-to-late pseudotime ordering;
• masked terminal-state recovery.

2. Spatial pattern reasoning

Files:

benchmark_tasks/public/spatial_pattern_tasks.json
benchmark_tasks/hidden/spatial_pattern_answers.json
benchmark_tasks/oracle_outputs/spatial_pattern_oracle_outputs.json

These tasks test reasoning about:

• spatially variable genes;
• domain-specific marker enrichment;
• masked spatial-domain recovery;
• unsupported spatial claims.

3. Topological persistence reasoning

Files:

benchmark_tasks/public/topological_persistence_tasks.json
benchmark_tasks/hidden/topological_persistence_answers.json
benchmark_tasks/oracle_outputs/topological_persistence_oracle_outputs.json

These tasks test reasoning about:

• bifurcating structure;
• branch count;
• ring-like spatial signals;
• false-positive loop claims;
• the difference between spatial topology and cell-fate topology.

The topology layer uses GUDHI-based summaries to support topology-aware benchmark tasks.

Public tasks, hidden answers, and oracle outputs

A key benchmark-design pattern in CellFateBench is the separation between public prompts and hidden answers.

benchmark_tasks/public/
benchmark_tasks/hidden/
benchmark_tasks/oracle_outputs/

Public tasks are solver-facing.

Hidden answers contain expected outputs and scoring-relevant evidence.

Oracle outputs show reference-style answers with rationale, confidence, and supporting evidence.

This structure helps prevent answer leakage and makes the benchmark easier to review.

A simplified benchmark structure looks like this:

public task
    |
    | visible to solver
    v
solver answer
    |
    | compared privately
    v
hidden answer key
    |
    | scored by validators
    v
score report

Oracle outputs provide a human-readable reference, but they are not used as public prompts.

v2: public RNA velocity extension

The v2 layer adds a public RNA velocity benchmark extension using the scVelo pancreas dataset.

The dataset preparation validates:

• 3,696 cells;
• 27,998 genes;
• spliced RNA velocity layer;
• unspliced RNA velocity layer;
• cluster annotation column;
• 8 annotation groups.

The project deliberately avoids committing the large raw H5AD file. Instead, it loads the public dataset through code and writes lightweight derived outputs.

Generated v2 dataset outputs include:

results/tables/velocity_dataset_summary.csv
results/tables/velocity_layer_summary.csv

The local public H5AD file is ignored through .gitignore, which protects the repository from accidental large-file commits.

That is important for a scientific software repo: data can be reproducible without being committed directly.

RNA velocity tasks

The v2 task generator creates RNA velocity reasoning tasks.

Files:

benchmark_tasks/public/velocity_reasoning_tasks.json
benchmark_tasks/hidden/velocity_reasoning_answers.json
benchmark_tasks/oracle_outputs/velocity_reasoning_oracle_outputs.json

The current v2 benchmark includes six velocity reasoning tasks:

The v2 layer is careful not to claim more than it computes.

It validates spliced and unspliced layers and creates benchmark reasoning tasks around RNA velocity context. It does not yet claim to compute a full scVelo velocity graph inside the benchmark pipeline.

That limitation is documented in the README and limitations file.

Scoring design

CellFateBench uses transparent scoring.

The v1 scoring layer supports:

• expected answer matching;
• Boolean claim correctness;
• required evidence-term coverage;
• confidence field presence;
• partial credit.

The v2 velocity scoring layer adds behaviour-focused scoring:

• correctness;
• evidence support;
• uncertainty discipline;
• no-overclaim discipline;
• penalty for unsupported precision claims.

This is important because a scientific answer is not only judged by the final label. A solver can provide the correct answer with weak evidence. Another solver can make a plausible claim but overstate confidence.

The v2 scoring layer makes those behaviours visible.

Velocity solver evaluation is implemented in:

src/cellfatebench/velocity_solver_evaluation.py
scripts/13_evaluate_velocity_solvers.py

Generated outputs include:

results/tables/velocity_solver_performance_summary.csv
results/tables/velocity_task_performance_summary.csv
results/figures/velocity_solver_score_by_profile.png
results/figures/velocity_task_pass_rate.png

Calibration and difficulty rebalancing

CellFateBench includes calibration assets because a benchmark should not only generate questions. It should also review task difficulty and likely failure modes.

The v1 layer includes design-stage calibration:

src/cellfatebench/calibration.py
benchmark_tasks/calibration_logs/design_stage_calibration_log.json

The v2 layer adds empirical sample-solver calibration:

src/cellfatebench/velocity_calibration.py
scripts/14_generate_velocity_calibration.py
benchmark_tasks/calibration_logs/empirical_velocity_calibration_log.json
results/tables/velocity_task_difficulty_rebalanced.csv
results/figures/velocity_task_difficulty_rebalance.png

This calibration is intentionally scoped.

It is based on local sample solver profiles. It does not claim frontier-model calibration. That would require running the benchmark against actual frontier models or expert human solvers.

Makefile workflow

The project exposes core workflows through a Makefile.

make test
make pipeline
make pipeline-v2

Other useful commands include:

make dataset
make trajectory
make spatial
make topology-summary
make topology-tasks
make calibration
make score
make velocity-data
make velocity-tasks
make velocity-evaluate
make velocity-calibration

This gives reviewers and future contributors stable commands rather than asking them to run internal Python modules manually.

Running the project locally

Clone the repository:

git clone https://github.com/gbadedata/cellfatebench-single-cell-analysis.git
cd cellfatebench-single-cell-analysis

Create the environment:

conda env create -f environment.yml
conda activate cellfatebench

Run the tests:

make test

Run the v1 pipeline:

make pipeline

Run the v2 pipeline:

make pipeline-v2

Expected result:

57 passed
CellFateBench full pipeline completed.
CellFateBench v2 public RNA velocity pipeline completed.

The public scVelo pancreas dataset may emit AnnData old-format warnings during tests. These warnings come from the upstream dataset format and do not indicate failure of CellFateBench logic.

Docker validation

The repository includes Docker support.

Build the image:

docker build -t cellfatebench:latest .

Run the test suite inside Docker:

docker run --rm cellfatebench:latest make test

Run the v1 pipeline inside Docker:

docker run --rm cellfatebench:latest make pipeline

Run the v2 pipeline inside Docker:

docker run --rm cellfatebench:latest make pipeline-v2

This validates that the benchmark can run in a clean container environment.

GitHub Actions CI

The project is validated through GitHub Actions.

CI checks:

• test suite;
• v1 pipeline;
• v2 pipeline;
• expected output files;
• Docker build;
• Docker test execution;
• Docker v1 pipeline;
• Docker v2 pipeline.

Workflow evidence is available here:

GitHub Actions CI

The CI badge is visible at the top of the README.

Documentation structure

The documentation layer includes:

README.md
docs/methods.md
docs/evidence_map.md
docs/reviewer_guide.md
docs/limitations.md
docs/project_design.md
docs/v2_velocity_extension_plan.md

Each document has a specific role.

This documentation structure is part of the engineering work. It makes the repository easier to evaluate.

What was validated

At the time of this write-up, the project had:

• 57 passing tests;
• v1 pipeline passing;
• v2 pipeline passing;
• Docker validation in CI;
• GitHub Actions passing on main;
• README upgraded with badges;
• methods, evidence map, reviewer guide, and limitations updated for v2;
• large public H5AD file ignored and protected from accidental commit.

This is the difference between a script and a reviewable scientific software project.

Key engineering decisions

Several engineering decisions helped make the project more robust.

1. Public and hidden assets are separated

Public benchmark tasks do not contain hidden answer fields.

2. Large public datasets are not committed

The public H5AD file is loaded through code and ignored locally.

3. Pipelines are exposed through Makefile targets

This makes execution easier for reviewers.

4. CI validates more than tests

The workflow validates tests, pipelines, expected outputs, and Docker execution.

5. Limitations are documented

The project does not overclaim clinical validity, biological discovery, frontier-model calibration, or full RNA velocity graph computation.

6. The benchmark is designed for extension

The current structure can support future task families such as full velocity graphs, latent time, spatial neighbourhood reasoning, multi-omic tasks, and expert calibration.

Future improvements

Planned improvements include:

• full scVelo velocity graph computation;
• latent-time summaries;
• gene-level velocity confidence summaries;
• real solver submissions;
• frontier-model calibration;
• human expert calibration;
• semantic answer matching;
• richer task-specific rubrics;
• expanded public datasets;
• spatial-neighbourhood reasoning;
• multi-omic benchmark tasks;
• release versioning;
• optional dashboard or web review interface.

Final thoughts

CellFateBench was built to show that single-cell genomics benchmarking can go beyond outputs.

The project asks whether a solver can reason from evidence, separate supported claims from unsupported ones, report uncertainty, and avoid overclaiming.

That is why the project is positioned as:

A benchmark-engineering project for the reasoning layer of single-cell genomics.

Explore the repository:

GitHub repository

Read the README:

Project README

Review CI validation:

GitHub Actions CI

Read the flagship Medium article:

Medium article