Skip to content

Datasheet for the CHAMBER-Bench v1.0 data artifacts

This is a datasheet in the style of Gebru et al., Datasheets for Datasets (2021): a structured description of released data — motivation, composition, collection, uses, distribution, and maintenance — so a consumer can decide whether and how to use it. It covers the three dataset artifacts released with CHAMBER-Bench v1.0. Each artifact is prepared deterministically from the committed repository state by scripts/release/prepare_hosting.py, ships with its own manifest, SHA-256 checksum file, dataset card, and Croissant metadata (a machine-readable JSON-LD dataset description), and is hosted on Hugging Face with a Zenodo archival copy.

The three artifacts:

  1. chamber-bench-partner-sets — the versioned partner zoo: public members' parameter files and checkpoints, per-member cards, and behavioural fingerprints. Private members appear as identity hashes only.
  2. chamber-bench-leaderboard-bundles — the result bundles behind every leaderboard row, the checkpoints the learned rows load, and the admission-report archives behind each task's verdict.
  3. chamber-bench-reference-trajectories — the fingerprint probe episodes and the power-pilot bundles: the evidence behind partner identities and the campaigns' sample-size choices.

Motivation

Why were these created? CHAMBER-Bench scores cooperation with black-box partners, which creates three evidence obligations: anyone must be able to (a) instantiate the exact partners a score was measured against, (b) recompute the score from raw episode records, and (c) audit the admission verdicts that decide which tasks count. One artifact serves each obligation.

Who created and funded them? Farhad Safaei, the project maintainer (see MAINTENANCE.md at the repository root). No external funding; no third-party data providers.

Composition

What do the instances represent? Simulation records only. There are no people, no personal data, no customer or workplace data, and no offensive content anywhere in these artifacts; every byte was generated by the simulator and tooling in this repository.

chamber-bench-partner-sets. One directory per partner-set version (cocarry_partners@v1, cocarry_partners@v2, handover_place_partners@v1, stage1_pickplace_as_partners@v1). For each public member: its construction parameters (JSON), its identity hash (SHA-256 over the serialized artifact), its rendered card, and — for learned members — its checkpoint file. For each private member: the identity hash and card only; the parameters are deliberately withheld (see Uses). Learned checkpoints are PyTorch .pt files with JSON sidecars, about 3 MB each.

chamber-bench-leaderboard-bundles. The verbatim result-bundle directories listed in the per-task LEADERBOARD_BUNDLES.txt manifests (13 bundles: 10 for cocarry@v1, 3 for handover_place@v1), each containing bundle.json (schema v3, ADR-028), per-seed episode records (JSONL), partners.json, REPRO.txt, prereg.yaml, and SHA256SUMS.txt; plus the campaign report, the committed checkpoint-selection artifacts, the 15 ego/pair checkpoints the learned rows load (content-addressed .pt files), and the admission-report archives (spikes/results/admission/, one per admission decision: co-carry ADMITTED, handover-place ADMITTED, pick-and-place CONTROL — the evidence behind each task's verdict). An episode record carries per-episode success, stress telemetry, seed and substream labels, and the partner identity — not raw simulator frames.

chamber-bench-reference-trajectories. The fingerprint probe archives (spikes/results/partner-fingerprints/, one verifiable bundle per partner-set member plus fingerprints.json per set, including the dropped-candidates record) and the power-pilot bundles (the run_purpose: power exploratory runs that sized the campaigns' preregistered episode counts — committed evidence, excluded from every leaderboard manifest), each an ADR-028 bundle tree.

Is any information missing? Yes, by design: private partner members' parameters (withheld to keep an unseen evaluation split meaningful) and the raw training runs behind the learned checkpoints (only the selected checkpoints and the selection evidence are released; training is reproducible from the preregistered configuration).

Collection process

All data was generated in simulation (ManiSkill v3 / SAPIEN, plus one CPU diagnostic task) by the exact tooling in this repository, under preregistered specifications locked with signed git tags before the runs. Every bundle records the producing command, git commit, dependency-lock hash, platform fingerprint, and seed schedule, and passes chamber-eval verify — which re-checks file hashes and recomputes the summary statistics from the raw records. Collection dates are July 2026 (stamped in each bundle and directory name). No human subjects, no annotation, no crowdworkers, no scraping.

Preprocessing and labeling

None beyond what the tooling does: episode records are written directly by the evaluation harness; fingerprints are deterministic functions of probe episodes; no filtering, cleaning, or human labeling occurred. The "raw" data (episode JSONL) ships inside the bundles.

Uses

Intended. Verifying and reproducing CHAMBER-Bench v1.0 leaderboard rows; evaluating new methods against the public partner set; auditing admission verdicts; studying partner behaviour via fingerprints and probe episodes.

Discouraged / out of scope. Training on the probe or evaluation episodes and then reporting leaderboard numbers (this defeats the unseen-partner condition — the protocol's private split and maintainer spot-check exist to catch it); treating results as transferring to real robots (simulation only); using the safety telemetry as certification evidence (the stack is research-grade, not a certified safety product).

Impact considerations. The artifacts contain no personal data, so the main responsible-use concern is misrepresentation: quoting rows without their honest labels (oracle reference, non-AHT upper anchor) or without the per-partner breakdown. The dataset cards restate the labels; consumers should carry them along.

Distribution

Hosted as three datasets on Hugging Face under the maintainer's namespace, with a Zenodo archival deposit per release (DOI 10.5281/zenodo.20128468 is the software concept DOI; the data deposit carries its own versioned DOI). Licence: Apache-2.0, same as the repository. Each artifact ships manifest.json (file list with sizes and SHA-256), SHA256SUMS.txt, a README.md dataset card, and croissant.jsonld (Croissant JSON-LD with the Responsible-AI fields). Consumers verify a download with sha256sum -c SHA256SUMS.txt and, for bundle directories, chamber-eval verify.

Hosted artifacts

Dataset Location
chamber-bench-partner-sets https://huggingface.co/datasets/fsafaei/chamber-bench-partner-sets
chamber-bench-leaderboard-bundles https://huggingface.co/datasets/fsafaei/chamber-bench-leaderboard-bundles
chamber-bench-reference-trajectories https://huggingface.co/datasets/fsafaei/chamber-bench-reference-trajectories

Maintenance

Single maintainer; contact and cadence in MAINTENANCE.md. Artifacts are versioned, never mutated: an updated partner set or a new campaign produces a new versioned upload, and superseded versions remain available for at least 24 months. Errata are announced in the repository CHANGELOG and on the dataset cards.