How to read your results
An experiment runs two bidding strategies, A and B, through the same simulated market many times and compares what each one earned and spent.
Reading a result
- Result sums up conversion value in one sentence: what B earned against A, at what spend, and in how many of the repeats B came out ahead.
- Better / worse marks each change in the direction that counts for that metric: more conversion value is better, a lower cost per acquisition is better.
- No clear difference means the change is inside its margin: it could be chance, so the experiment cannot tell the two strategies apart on that metric.
- Margin (±) is two standard errors of the difference between B and A across repeats. A change larger than its margin is unlikely to be luck within this simulated market.
- B ahead in is the share of repeats in which B did better than A. "—" marks metrics like spend, where neither direction is better on its own.
- Repeats and seed: each repeat is a fresh simulated day (or days) that both strategies face together. The seed fixes the randomness, so the same seed reproduces the same numbers.
Metrics
- Conversion Value
- The total value of the sales or sign-ups the strategy produced. Higher is better.
- Conversions
- How many sales or sign-ups it produced. Higher is better.
- Spend
- What it paid in auctions. Neither direction is better on its own: compare it with what it bought.
- ROAS
- Return on ad spend: conversion value divided by spend. A ROAS of 3 earns $3 per $1. Higher is better.
- CPA
- Cost per acquisition: spend divided by conversions, what one sale or sign-up costs. Lower is better.
- Impressions
- How many auctions it won, so how often its ad was shown. Neither direction is better on its own.
- Impression Share
- Impressions divided by the opportunities it targeted: how much of its audience it reached. Higher is better.
- Lost Budget Share
- The share of targeted opportunities it missed because its daily budget had already run out. Lower is better.
- Budget Utilization
- Spend divided by the budget it had: 100% means it spent everything. Neither direction is better on its own.
What the model leaves out
The market is synthetic and not calibrated against real campaign data, so results show how strategies behave under these assumptions, not a forecast of a real campaign.
- Second-price auction per impression with a reserve price; bids carry random per-user noise.
- Click and conversion rates depend on segment and hour, not on which advertiser wins.
- Conversions are attributed immediately to the impression; no delays, frequency or position effects.
- Daily budgets reset each day; pacing scales bids hourly using only past spend.
- Both strategies face the same simulated market in each repeat (common random numbers).
Known model behaviors
- A competitor that adapts its bids can make a higher reserve price cheaper: it lowers its own bids in response, so prices can fall. With competitors that do not adapt, a higher reserve buys fewer impressions at a higher price, as expected.
- Pacing only scales bids; it never skips auctions. Once competitors run out of budget, every eligible impression is won at the reserve price, so spend cannot fall below about the reserve times the volume targeted, however strongly pacing holds back.
New to bidding? Take the five-week crash course or see ad auction terms, explained.