Data Integrity

• One key challenge with testing creative ads is maintaining data integrity. 
•  One source notes that it's not typical to have test data come out evenly, with some ads getting more impressions than others, which often creates noise in the data.
• The lack of even data makes it challenging to use data sets from testing to accurately determine the winning ad. 
• It's also easy for marketers to overlook data inconsistencies, and therefore overlook factors that are leading to conversions because of these variances.

History & Selection Effects

• Specific challenges to A/B creative testing include history effects and selection effects.  
•  History effects occur when testing is run between two different times without accounting for the impacts of timing, such as 20 hours on a weekday one month and then 20 hours on a weekend another month, or during a holiday season peak vs. low season.
•  Selection effects involve testing only one element of ads, without accounting for the impact of other marketing strategies being run simultaneously.
• For example, testing one ad, while simultaneously running a pay-per-click ad on another platform that is directing traffic to a website, and not differentiating the results could cause data issues that make it hard to determine the impact of a campaign.
•  Sampling distortion effects can also be an issue, which are primarily caused when the sample size is not large enough to overcome random chance.

Selecting Testing Approach

• Because there are so many potential testing strategies, it's often challenging to determine which type of tests will yield the most accurate or useful results.
• For example, Facebook ads offer a split-test feature, but for localized or custom ads, a more complex approach may be more effective.
•  Knowing what to test within an ad is also often a challenge, because involves marketers being able to determine which changes will both be easy to implement and create a measurable impact.
• Ad testing methods need to be selected in a way that generates the most information on why the customer behaves the way they do that the marketer can apply the knowledge and have a foundational understanding of what motivated the behavior.