Ad experiments: A/B test banner ads on your own site

Ad experiments are for the ads you show on your own site: a house ad for your own product, a sponsor's banner on a directory, a call to action in the sidebar of your docs. Privatus Analytics shows your creatives in a slot, counts how often each one is shown, seen and clicked, and tells you which one gets the most clicks.

It works without cookies, without anything stored in the visitor's browser and without a visitor id, like the rest of Privatus Analytics.

How it works, in one minute#

1. You create a placement: a slot of one size, such as a 300 by 250 sidebar or a 728 by 90 leaderboard.
2. You upload its creatives: images of that shape, each with the address a click goes to.
3. You paste one tag where the slot should appear on your site.
4. On every view the loader picks a creative by its weight and shows it. It counts the impression, a viewable impression once the creative has been seen, and the click.
5. To find the best creative, you start an ad experiment on the placement. When the result is clear enough it ends by itself, and the winner can be shown to everyone.

Plans#

Every impression is an event and counts toward your monthly events. Viewable impressions and clicks do not.

Add a placement#

Open Experiments, then Ad placements, and press New placement. Give it:

• A name, for you, such as "Profile sidebar".
• A key, which the tag on your site carries, such as profile-sidebar. Lowercase letters, numbers and dashes. It is made from the name when you leave it empty.
• A size. Pick a common one or type your own:

• A label, if you like, shown above the ad, such as "Sponsored". Saying that something is an ad is good practice, and the law in some places.

One size a placement. For a leaderboard on desktops and a banner on phones, make two placements and show each with your own CSS. A slot your page hides is never counted.

Creatives#

On the placement's page, press Add creative.

• Image. A JPEG, PNG, WebP or GIF up to 2 MB, of the placement's shape and at least its size. Twice the size looks sharpest on screens with dense pixels. An animated GIF stays animated. The image is turned into WebP at the placement's size and at twice that, with its metadata removed, and served from our media host.
• Link. An https address, or a page of your own site such as /pricing. An outside address opens in a new tab.
• Campaign parameters. On by default: utm_source is your site's domain, utm_medium is house_ad, utm_campaign is the placement's key and utm_content is the creative's name. For a page of a site you measure, the clicks then show up in Campaigns and in your goals. A parameter the link already has is never replaced.
• Alternative text, for people who use a screen reader.
• Weight. Outside an experiment, creatives rotate by weight. A creative with 70 is shown about twice as often as one with 35.
• In the rotation. Switch a creative off to keep it and its numbers without showing it.
• When and where (Pro and up): a start and an end, devices and pages (one a line, with one * for many, such as /tools/*). These are checked in the visitor's browser on every view, so a start or an end is never late.

The image is checked when you upload it, so a file of the wrong shape is refused on the form. It is ready a few seconds later.

A creative that an experiment compares can't be deleted, because the results need it. Switch it off instead.

Put the tag on your site#

The placement's page shows the tag. It has two parts:

<script async src="https://privatusads.com/embed/ads.js" data-site="pa_XXXXXXXXXX"></script>
<div data-pa-ad="profile-sidebar" style="width:100%;max-width:300px;aspect-ratio:300/250"></div>

• The script goes once on a page, however many placements it has. It is about 2 KB compressed.
• The slot goes where the ad should appear. Its size is set before the script runs, so the page does not move when the creative arrives. When there is nothing to show, the slot is hidden.

It works with pages that change without a reload (Turbo). The slot config is kept at the edge for up to an hour and refreshed within seconds of any change you make.

What is counted#

• Impression: the creative was shown in a slot the page displays.
• Viewable impression: at least half of the creative was on screen for one unbroken second, in a tab that was in view. For a creative of 242,500 pixels or more, 30% is enough. This is the IAB and MRC rule.
• Click: the creative was clicked. A second click on the same creative from the same visitor address within ten minutes is not counted again.
• Click rate: clicks per viewable impression. It is the number experiments compare.

For each of them Privatus Analytics keeps the placement, the creative, the experiment if there is one, the time, the device, the country and the page (its host and path, with your path masks). Nothing about the visitor is kept.

Bots#

Every plan leaves out what the basic bot filter catches: known bots, headless browsers and visits from data centers. They are never counted and never billed.

On Pro and up, every hit also gets a bot score, and its band (likely human, unsure or likely bot) is kept with it. Tick Likely humans only on a placement or an experiment to count only what the score calls likely human.

Test creatives against each other#

On the placement's page press Test these creatives, or open Experiments and press Test ads.

1. Pick two or more creatives of the placement whose images are ready. The first is the original.
2. Give each a share, or leave the shares empty for an even split.
3. Choose when it ends: the shortest and longest duration, the viewable impressions each creative needs (1,000 by default) and the confidence.
4. On Pro and up, limit the test to some devices or pages. Elsewhere the placement keeps its usual rotation.
5. Press Start.

The results show each creative's impressions, viewable impressions, clicks and click rate, its uplift against the original and the chance that it really is better. A winner is called the same way as in a page experiment: once the shortest duration has passed, every creative has its viewable impressions and one beats the original with the confidence you chose.

When a creative wins and Send everyone to the winner is on, the placement shows the winner on every view until you turn it off. Then it rotates its creatives again.

A placement runs one experiment at a time. Ad experiments count toward the same limit as page experiments.

Steer to the leader#

On Pro and up, choose Steer to the leader. This makes the experiment a multi-armed bandit: once a day, more views go to the creative most likely to have the best click rate, and every creative keeps at least 10%, so one that started badly can still recover. Nothing moves until every creative has 300 viewable impressions. The method is Thompson sampling, explained under The method.

The method#

This is what happens behind the numbers, in brief.

Picking a creative. Every placement has a small list of its creatives and their shares, which the script fetches once per page. On every view the script draws a random number and picks a creative in proportion to its share: with shares of 70 and 30, the first is shown about seven times in ten. Nothing remembers which creative a visitor saw, so a visitor can see a different one on the next page. That is why the results count views, not people.

Comparing the creatives. Each creative is compared with the original on its click rate: clicks divided by viewable impressions. Privatus Analytics uses the same statistics as for page experiments:

• A two-proportion z-test gives the p-value: how likely a difference this big would be if the two creatives were really the same.
• With more than two creatives, Holm's method raises the p-values, so more comparisons don't mean more false winners.
• The likely range is the difference in click rate plus or minus the margin for the confidence you chose.
• The chance to beat the original treats each creative's real click rate as unknown, describes what it could be from the clicks and views so far (a Beta distribution), and works out how often the creative would come out ahead.

A winner is only called once the shortest duration has passed, every creative has its viewable impressions and one beats the original at your confidence. The check runs once an hour.

Steering. With "Steer to the leader", once a day the creatives' chance of being the best is worked out by drawing 4,000 possible click rates for each from the same Beta distributions and counting how often each one is the highest. This is known as Thompson sampling, a kind of multi-armed bandit. Each creative keeps 10%, and the rest is shared in proportion to those chances. The new shares reach your site within seconds.

Sponsor reports#

On Pro and up, a placement's page can make a link to a live report for someone with no account, such as a sponsor. It shows the impressions, viewable impressions, clicks and click rate of the placement's creatives, or the ones you choose, in total and day by day for the last 7, 30 or 90 days. What the bot score calls likely bots is left out.

The link can expire on a date, and deleting it stops it working at once. If the workspace moves to a plan without sponsor reports, or the site is locked by its plan, the link stops working but is kept: an upgrade brings it back. The report says nothing about your visitors.

Ad blockers#

Some ad blockers hide slots and block the script. Privatus Analytics does not try to get around them. A view that is blocked is never counted and never billed, so the numbers you see are those of the people who saw the ads.

Privacy#

The script sets no cookie, uses no local storage and reads no fingerprinting API. A creative is picked at random by its weight on every view, so nothing has to remember who saw what. The visitor's address and browser are read in memory to leave out bots and to tell the device and country, then dropped. See what we store.

API and MCP#

Placements, creatives, ad experiments and sponsor links have the same operations in the API and over MCP as in the app: ad_placements_list, ad_placements_get, ad_placements_create, ad_placements_update, ad_placements_embed, ad_creatives_create (with image_url to have the image downloaded), ad_creatives_update, ad_report_links_create and experiments_create with kind: "ad". Over MCP nothing is deleted: deleting a placement, a creative or a link is done by a person in the app.

Limits#

• Images up to 2 MB, from 10 to 2,000 pixels a side.
• Up to 10 creatives an experiment, and 20 page patterns a creative.
• One click per creative and visitor address in ten minutes.