How Agencies Run CTV Incrementality Tests for Clients

Clients asking whether CTV is working isn’t a new problem. What’s new is the expectation that agencies can answer it with data. Incrementality testing is the method: a controlled holdout that separates the lift a CTV campaign drove from the revenue that would have happened anyway. This is how performance agencies structure, run, and report that test inside a client campaign.

What incrementality testing actually measures

An incrementality test has one job: isolate the causal lift from a CTV campaign.

The mechanism is a holdout group. A portion of the eligible audience, typically 10 to 20 percent, is withheld from seeing the ad. Everything else stays identical — the audience pool, the creative, the media spend. At the end of the test window, you compare conversion rates between the exposed group and the holdout.

The difference is the lift. That number is what survives the budget meeting.

Post-view attribution can’t produce this. It credits a CTV ad for any conversion that followed an impression, whether or not the impression caused it. Incrementality removes the coincidence. For clients tracking blended ROAS across channels, it’s also the number that makes CTV comparable to Meta and Google — the same causal logic, not impression-adjacent correlation next to last-click attribution.

Read what an incremental customer actually is if the client needs a primer before you set up the test.

How to structure a holdout test for a client campaign

Three decisions define the test design: holdout size, test window, and the conversion event.

Holdout size. Ten to twenty percent of the eligible audience is standard. Too small and the results won’t reach statistical significance. Too large and you’re burning potential conversions to run a test. For most mid-market client campaigns, 15 percent is the right starting point — enough statistical power without sacrificing too much of the addressable audience.

Test window. The window needs to cover at least one full buying cycle for the client’s category. A weekly-repeat subscription product needs a shorter window than a gift-heavy brand where purchases concentrate around a season. Getting this wrong in either direction produces a result that understates or overstates the channel’s contribution, and there’s no clean way to rerun it mid-campaign.

Conversion event. Define it before the test runs. Website purchases, subscription starts, form completions — whichever is closest to revenue. If the client tracks multiple conversion types, start with the primary one and resist the temptation to layer in secondary events during the test window.

One structural requirement holds regardless of category: the holdout needs to be in place before the campaign goes live. Running a test on an audience that’s already been exposed contaminates the control group and produces an unusable read.

Learn how to track and measure CTV ad performance alongside incrementality if you’re new to CTV measurement setup.

How to size the cooldown window to your client’s category

The cooldown window is the period between when a user sees an ad and when you start counting conversions. It exists because most people don’t convert the day they see the ad.

Setting it too short misses real lift. Setting it too long pulls in organic conversions that had nothing to do with the campaign. The right number maps to the client’s actual purchase behavior, not a platform default.

For most categories:

For a gifting or seasonal apparel brand, where purchase volume concentrates around a few key moments in the year, the cooldown needs to cover enough of a standard buying cycle to produce a meaningful read. Running the same window you’d use for a DTC supplement brand won’t work.

When in doubt, run it longer. A result that understates lift because the window closed too early is the harder outcome to recover from in a client conversation — you can’t go back and reopen a window that’s already closed.

See why setting the right attribution window matters for CTV for more on how window setup affects reported results.

Why the test needs to be running before peak spend

A clean incrementality test needs time. For most client campaigns, four to six weeks is the minimum to accumulate statistically significant results. For seasonal businesses, that means the test has to be set up and running before the peak period begins.

Running a test during peak spend — when purchase intent is elevated across the board — produces results that are impossible to interpret cleanly. The lift you read includes behavior changes that have nothing to do with your client’s CTV campaign, and there’s no way to separate them after the fact.

The practical implication: if a client’s budget is increasing — a product launch, a seasonal peak, a new channel push — the holdout needs to be in place before that spend goes up. A test that runs at the start of a scaled campaign, when the question is whether CTV works at all, is what produces the number you can actually act on.

Plan for at least four to six weeks between holdout launch and the moment you need actionable results. That buffer is what gives you time to complete the test, read the data, and adjust before conditions change.

How to report lift results to clients in budget-meeting terms

Lift data means different things to different clients. The agency’s job is to translate the test result into the metric the client already tracks and reports internally.

Three ways to frame it:

Incremental revenue. If the exposed group converted at 4.2% and the holdout at 2.8%, the lift is 1.4 percentage points. Apply that delta to total audience size and media spend to show what the campaign caused in dollar terms — not what happened near the campaign, what the campaign caused.

Incremental ROAS. Divide incremental revenue by media spend. This maps CTV performance directly to the language clients use to evaluate Meta and Google. That’s what earns CTV a permanent line in the media plan rather than a test budget to be cut when pressure comes.

CAC comparison. If the client tracks customer acquisition cost across channels, run the incremental CPA from the CTV test against their blended CAC from other channels. TYR, a performance athletic apparel brand, saw a 24.2% blended CAC reduction tracked through Northbeam’s attribution model in its first 60 days on Vibe.co — alongside 234.6% revenue growth in the same period. Read the TYR case study for the full attribution methodology breakdown.

The goal is to put the CTV number next to the client’s Meta and Google numbers. When CTV speaks the same language as the rest of the performance stack, it stays in the budget. See how transparent CTV management drives higher ROAS for more on reporting setup.

Running incrementality natively in Northbeam with Vibe

Northbeam’s native CTV incrementality testing launched with Vibe as the only CTV platform supported at launch. That distinction matters for agencies: it means the holdout setup, cooldown window, and lift read all run inside Northbeam — no separate measurement tool, no manual reconciliation between platforms.

The incrementality result lives in the same reporting environment as the rest of the client’s attribution data. For clients using Northbeam as their source of truth across Meta, Google, and CTV, the lift number comes from the same system they already trust. That’s what makes it credible in a budget review.

For agencies running Northbeam accounts, this removes the overhead of building a separate measurement layer for every client running CTV. Vibe holds a 97 satisfaction score on G2, and the average campaign launch time on Vibe is six days per platform data — which means there’s enough runway to get a test running, let it breathe, and have results before most seasonal peaks. Read about the Northbeam and Vibe incrementality partnership and how agencies are using the native workflow.

For agencies using Haus for measurement, the Vibe integration with Haus Analytics supports incrementality testing through the same holdout methodology. See how to test incrementality with Vibe and Haus for the step-by-step setup guide.

Vibe is also built for agency workflows across media planning, reporting, and client campaign management — the incrementality workflow is one piece of a broader client reporting stack.

FAQ

What’s the difference between incrementality testing and post-view attribution?

Post-view attribution credits a CTV ad for any conversion that followed an impression — whether or not the impression caused the purchase. Incrementality testing uses a holdout group to isolate the causal effect: you compare conversion rates between the exposed group and the holdout, and the delta is the lift the campaign actually drove. The two measurements can produce very different numbers, and incrementality is the one that holds up in a budget review.

How long does a CTV incrementality test need to run?

Most tests need four to six weeks to reach statistical significance. For seasonal or gifting categories, the window should cover at least one full purchase cycle. The test also needs to be in place before peak spend begins — running a holdout during a spike in purchase intent produces results that can’t be cleanly attributed to the campaign itself.

Can an agency run an incrementality test without a native tool like Northbeam?

Yes, by manually suppressing a portion of the eligible audience from targeting and tracking conversions separately by cohort. It’s operationally heavier, requires coordinating data from multiple sources, and is harder to present cleanly to clients. Native incrementality inside Northbeam runs the test inside the client’s existing attribution environment, so the result doesn’t require a separate system the client has to be walked through or a data reconciliation step the agency has to manage.

How do you choose the right holdout size for a client campaign?

Ten to twenty percent of the eligible audience is standard. Fifteen percent is a solid starting point for most mid-market client campaigns. A holdout that’s too small won’t reach statistical significance, especially for clients with lower monthly conversion volume. Too large burns conversion opportunities for the sake of the test. Start at 15 percent and adjust based on the client’s conversion volume — lower volume typically means the holdout needs to be larger to produce a clean read.