Ad Testing Pitfalls


Gain an understanding of potential pitfalls and issues surrounding creative testing such as is used for ads, in order to support a client presentation regarding how testing results and actual market impact are not always aligned.

Early Findings

Data Integrity

History & Selection Effects

  • Specific challenges to A/B creative testing include history effects and selection effects.
  • History effects occur when testing is run between two different times without accounting for the impacts of timing, such as 20 hours on a weekday one month and then 20 hours on a weekend another month, or during a holiday season peak vs. low season.
  • Selection effects involve testing only one element of ads, without accounting for the impact of other marketing strategies being run simultaneously.
  • For example, testing one ad, while simultaneously running a pay-per-click ad on another platform that is directing traffic to a website, and not differentiating the results could cause data issues that make it hard to determine the impact of a campaign.
  • Sampling distortion effects can also be an issue, which are primarily caused when the sample size is not large enough to overcome random chance.

Selecting Testing Approach

  • Because there are so many potential testing strategies, it's often challenging to determine which type of tests will yield the most accurate or useful results.
  • For example, Facebook ads offer a split-test feature, but for localized or custom ads, a more complex approach may be more effective.
  • Knowing what to test within an ad is also often a challenge, because involves marketers being able to determine which changes will both be easy to implement and create a measurable impact.
  • Ad testing methods need to be selected in a way that generates the most information on why the customer behaves the way they do that the marketer can apply the knowledge and have a foundational understanding of what motivated the behavior.

Proposed next steps:

You need to be the project owner to select a next step.
Provide 3-5 best practices for market tests to avoid history effects, selection effects, and sampling distortion in order to maintain data integrity in creative testing methods.
Provide an overview of what types of testing key platforms are providing as part of their services, such as Facebook, Instagram, and Google offering split-testing or other related ad testing options built into the platforms.