Ad Testing Pitfalls
Gain an understanding of potential pitfalls and issues surrounding creative testing such as is used for ads, in order to support a client presentation regarding how testing results and actual market impact are not always aligned.
History & Selection Effects
- Specific challenges to A/B creative testing include history effects and selection effects.
- History effects occur when testing is run between two different times without accounting for the impacts of timing, such as 20 hours on a weekday one month and then 20 hours on a weekend another month, or during a holiday season peak vs. low season.
- Selection effects involve testing only one element of ads, without accounting for the impact of other marketing strategies being run simultaneously.
- For example, testing one ad, while simultaneously running a pay-per-click ad on another platform that is directing traffic to a website, and not differentiating the results could cause data issues that make it hard to determine the impact of a campaign.
- Sampling distortion effects can also be an issue, which are primarily caused when the sample size is not large enough to overcome random chance.
Selecting Testing Approach
- Because there are so many potential testing strategies, it's often challenging to determine which type of tests will yield the most accurate or useful results.
- For example, Facebook ads offer a split-test feature, but for localized or custom ads, a more complex approach may be more effective.
- Knowing what to test within an ad is also often a challenge, because involves marketers being able to determine which changes will both be easy to implement and create a measurable impact.
- Ad testing methods need to be selected in a way that generates the most information on why the customer behaves the way they do that the marketer can apply the knowledge and have a foundational understanding of what motivated the behavior.
Proposed next steps:
You need to be the project owner to select a next step.