Gain an understanding of potential pitfalls and issues surrounding creative testing such as is used for ads, in order to support a client presentation regarding how testing results and actual market impact are not always aligned.
History effects occur when testing is run between two different times without accounting for the impacts of timing, such as 20 hours on a weekday one month and then 20 hours on a weekend another month, or during a holiday season peak vs. low season.
Selection effects involve testing only one element of ads, without accounting for the impact of other marketing strategies being run simultaneously.
For example, testing one ad, while simultaneously running a pay-per-click ad on another platform that is directing traffic to a website, and not differentiating the results could cause data issues that make it hard to determine the impact of a campaign.
Sampling distortion effects can also be an issue, which are primarily caused when the sample size is not large enough to overcome random chance.
Selecting Testing Approach
Because there are so many potential testing strategies, it's often challenging to determine which type of tests will yield the most accurate or useful results.
For example, Facebook ads offer a split-test feature, but for localized or custom ads, a more complex approach may be more effective.
Knowing what to test within an ad is also often a challenge, because involves marketers being able to determine which changes will both be easy to implement and create a measurable impact.
Ad testing methods need to be selected in a way that generates the most information on why the customer behaves the way they do that the marketer can apply the knowledge and have a foundational understanding of what motivated the behavior.
Only the project owner can select the next research path.