Why do virtually all marketers insist on doing their own tests rather than believing tests done by third parties?
Because there is a general distrust in the numbers for almost all research studies in the industry from the big to the small. People even distrust their own numbers, coming out of their own research studies.
This is actually a healthier sign than it seems; a right balance must be struck on the epistemology of the advertising measurement business. We must begin from a skeptical stance. As in all science. In this case the science is important because it affects billions of dollars, and people who can’t afford to lose their careers.
Our recommendations as to how to achieve this right balance:
- Give more credence to third party research studies done by well-known and trusted research companies.
- Give more credence to random control trials, they actually measure causal differences; less credence to results based on correlations.
- When measuring advertising effects, ROI and full funnel branding, if the perceived need for broadcast in the test eliminates the idea of a random control trial (which is most purely applicable to addressable media types only), consider big data singlesource such as provided by NCS, Neustar, 605, VideoAmp, Comscore, TiVo Research and others.
- As benchmarks to track at high level use marketing mix modeling, but it is too slow and inexact for tactical decision making. It is still valuable as a compass on your business. It forces you to think longer range ahead and to consider what you need to learn.
The six studies reported at right (below on mobile) are all done by trusted objective third party organizations including the nonprofit ARF’s Cognition Council (study 5) and Google Analytics (study 4). The third-party research companies are NCS (study 1), 605 (study 2), Neustar (study 3), and Simmons (study 6).
The Neustar study is a random control trial, therefore conclusively proves a near-doubling (+95%) of incremental campaign-produced sales. The method tested was the selection of target audiences based on content consumption aligned to the content in the ad. The brand was a major national retail chain. Sales were measured by the chain in concert with IRI. This is proof that the RMT targeting method, causally nearly doubles sales effects as compared to the lookalike targets the client had been using before.
Study 4 was also a random control trial. The outcome measure was site visit. The benchmark was quintupled.
The NCS and 605 studies align with one another very closely. For example, in the RMT analysis of the NCS data, the average increase in sales lift was +36%, in the 605 study, purchase intent increased +37%.
In the statistical analysis performed by the ARF Cognition Council, 11 RMT Motivations accounted for 48% of the IRI sales effect. As a comparison, the average sales effect attributed to paid media advertising by market mix modeling is only 7%. This suggests that the content put into the ads has much more impact on sales than the media placements you make. Future MMM should include RMT encoding as demonstrated by the ARF, that way more of the sales effects will be explainable and therefore controllable.