Ledger
Public experiment ledger
One place to read a hypothesis, the variant, the traffic split, the Bayesian probability of a winner, the stopping rule, the decision, and the evidence.
In plain English
An experiment shows two versions of a next step and watches which one is clicked. This site will not run that test until it is approved in the repo. Until then every visitor sees the ordinary page.
Consent decides who can be counted. The measurement choice lives in this browser. After you accept, the browser also sets a consent cookie the edge can see. Without that cookie, the edge assigns control, sets no assignment cookie, and does not count the visit. A cookie is written only for a test that is actually running.
The probability is a Bayesian read of two coin-flip rates. The credible interval is the band those rates probably sit in. The stopping rule refuses to name a winner before the sample and the calendar say the read is old enough. When the sample is small, the decision is the sentence “Not enough data yet.”
Each change of state is hashed, and each hash includes the one before it. The root is a short fingerprint of the whole list. You can check an entry on this page. A changed entry will not match.
Demo mode. KV or Upstash is not configured, so live tallies are not saved. The real experiment shows zeros. The example section is labeled example data.
This request from the edge: control, not counted, no_consent.
Real experiment. It is configured and it is not running.
Next step after a lab
After a lab, a next-step link that says Packages is clicked more often than a next-step link that says Another lab.
- Status
- Awaiting approval
- Variants
- Another lab (control) → /lab/recommendationsPackages (packages) → /packages
- Traffic split
- 20.0% of consented visitors enter the test, split by weight. 10.0% are holdout and always see control. Everyone else sees control and is not counted. Without consent, everyone sees control and is not counted.
- Stopping rule
- Beta-binomial with a uniform prior. A winner needs 200 exposures on every arm, 14 days, and at least a 95% probability that one rate is higher. A smaller sample never wins.
- Decision
- Not enough data yet.
Evidence
Holdout. 0 exposures, 0 conversions.
Another lab. 0 exposures, 0 conversions.
Packages. 0 exposures, 0 conversions.
Another lab: mean 50.0%, 95% interval 2.5%–97.5%
Packages: mean 50.0%, 95% interval 2.5%–97.5%
Packages would open the contact page. Another lab would open the recommendations lab. Those links are not added to lab pages while this experiment is awaiting approval.
Merkle log
Root of the real experiment. Verify an entry here, or paste one.
2bcf83edc26f86de6e3262deec5b830134d2aae67fb9062bfd8d58dd372a629f
configured
c174282d6c44abdebe8bdfc74c804433af560812125308310a8c8f97fa7fda31
awaiting_approval
b1b34cf4e5fdc66d7ce996c3ba7c75c0bc429d990270e3622a81e57a8e818b6d
Causal check
Not enough data yet.
The holdout comparison needs the minimum sample on both the holdout and the exposed group. Markov removal for this experiment has no path counts yet.
Example data. These counts are not this site’s traffic.
Example: next step after a lab
Example only. A Packages link is clicked more often than Another lab.
- Status
- Awaiting approval
- Variants
- Another lab (control) → /lab/recommendationsPackages (packages) → /packages
- Traffic split
- 20.0% of consented visitors enter the test, split by weight. 10.0% are holdout and always see control. Everyone else sees control and is not counted. Without consent, everyone sees control and is not counted.
- Stopping rule
- Beta-binomial with a uniform prior. A winner needs 200 exposures on every arm, 14 days, and at least a 95% probability that one rate is higher. A smaller sample never wins.
- Decision
- Winner: Packages
Evidence
Holdout. 200 exposures, 18 conversions.
Another lab. 400 exposures, 40 conversions.
Packages. 400 exposures, 72 conversions.
Another lab: mean 10.2%, 95% interval 7.4%–13.3%
Packages: mean 18.2%, 95% interval 14.5%–22.1%. Probability this rate is higher: 99.9%.
Example data. Not this site’s traffic.
Example causal check
Holdout conversion 9.0%. Exposed conversion 14.0%. Difference 5.0%.
Markov removal on this example
Chance of reaching contact from the lab, then the same chance with one step removed. A positive effect means that step was helping.
- another-lab: 24.0% → 33.3% (-9.3%)
- packages: 24.0% → 8.0% (+16.0%)
Shared removal example
Example data. The same removal the causal attribution lab draws. Baseline chance of converting, then each page’s share of the drop when that page is removed.
- home: 39.3%
- lab: 26.2%
- consent: 9.8%
- packages: 24.6%
Edge A/B testing, causal attribution, verifiable analytics are the worked examples. This ledger is the live gate.
Example log
5308acd15fc9c8abfa6d5903e71af688d846283aaa2bd4cbeebd80aa403a0c6b
example_configured
8fbb0aaf872266f1272db40904f5b7554d86fabee56b798315f517d46ad85f76
example_counts
d8aea25391184390c25d8aff9c75d32a4d0e49eb18d48f4b8b3261eb5b39591b
Suggest a hypothesis
In the site chat, start the message with “Hypothesis:” or “Hipótesis:”. The same scorer runs there. Only an edit to experiments/approved.json, by Neil, can turn a test on.
The experiments lab is a separate A/B and bandit demo.
WalkthroughHow we built itShowHide
Example vendor setup is marked below. eVar and event numbers are examples. This lab does not call Adobe.
The ledger is a public page. The edge assigns a variant only after consent and only when experiments/approved.json says the experiment is approved and running. Counts stay in KV or Upstash when those variables are set. Otherwise the page says it is in demo mode and the sample numbers are labeled example data.
Read the approval file
isLive is true only when the catalog status is running, the row is not example data, and approved.json has that id with approved set to true. The seeded next-step test fails those checks, so it does not run.
Assign before render
The proxy reads the consent cookie. Without it, or with Global Privacy Control, the visitor is control and is not counted, and no assignment cookie is set. A live test hashes a visitor id into a holdout, a 20 percent test slice, or the uncounted remainder.
Refuse an early winner
The posterior is a beta-binomial with a uniform prior. The decision stays “Not enough data yet.” until every arm has the minimum sample and the test has run for the minimum number of days, even if one arm converted every time.
Hash each state change
configured and awaiting_approval are entries in an append-only log. Each hash covers the previous hash. The page shows the Merkle root and checks an entry in the browser.
Score ideas without shipping them
The form and a chat message that starts with Hypothesis: keep the idea text only. The score is a rubric. The nightly job stays off unless EXPERIMENT_AGENT=1, and a winner opens a draft pull request only when EXPERIMENT_REPO_TOKEN is set. Nothing is merged.
From the code
Simplified gate. The real function also rejects example rows.
export function isLive(experiment, file) {
if (experiment.example) return false
if (experiment.status !== "running") return false
return file.experiments.some((entry) => entry.id === experiment.id && entry.approved === true)
}Simplified stopping rule. A short or small sample returns before any winner is named.
if (!sampleReady || !timeReady) {
return { label: "Not enough data yet.", winnerId: null }
}Google and Adobe
After Accept, a live experiment queues experiment_conversion with experiment_id and variant. Adobe is an example of the same fields. This site does not call Adobe.
When a test is actually running, a click on its call to action queues experiment_conversion after Accept. experiment_id and variant are the parameters. This site does not call gtag('set'). The seeded test is not running, so that push does not fire.
Shape of the push. It is sent only after Accept and only for a live assignment.
dataLayer.push({
event: "experiment_conversion",
experiment_id: "lab-next-step",
variant: "packages",
conversion_name: "packages_click",
})Example implementation. This site does not call Adobe. eVar41, prop41, and event41 are example variable numbers.
Example alloy payload. _example is not a real tenant.
alloy("sendEvent", {
xdm: {
eventType: "web.webinteraction.linkClicks",
_example: { experimentId: "lab-next-step", variant: "packages" },
},
})Example hit. These variable numbers are not configured on this site.
s.eVar41 = "lab-next-step"
s.prop41 = "packages"
s.events = "event41"
s.tl(true, "o", "experiment_conversion")Translation
Each lab field, the GA4 name, and the example Adobe variable. Scope is the part that usually gets lost in the translation.
Experiment
GA4. experiment_id
Adobe. eVar41, example
Scope. Event. One test id, not a visitor id.
Variant
GA4. variant
Adobe. prop41, example
Scope. Event. control, packages, or the holdout’s control.
Conversion
GA4. conversion_name on experiment_conversion
Adobe. event41, example
Scope. Event. packages_click or lab_click.
Consent
GA4. The push is skipped when consent is off.
Adobe. No Adobe call either way.
Scope. The edge does not count a visit without the consent cookie.