374. IJRM. A|B Testing.
August 7, 2025
Listen to full episode :
In this episode Johannes Boegershausen, an assistant professor of marketing at the Rotterdam School of Management, Erasmus University discusses the limitations and challenges of A|B testing on digital platforms like Google and Meta. Boegershausen concludes by emphasizing the importance of being mindful of the limitations of data obtained from these platforms and recommends not over-extrapolating results from narrow contexts to broader marketing strategies.
Key points discussed:
Definition and Importance of A|B Testing: A|B tests are essential tools for evidence-based marketing, where study participants are randomly assigned to different treatments to compare results.
Divergent Delivery: On platforms like Meta, A|B tests aren't truly randomized because the platform’s algorithms optimize ad delivery non-randomly, affecting the results.
Implications for Marketers and Researchers: This non-random optimization skews the data, making it challenging to draw valid inferences. This particularly affects academic research that seeks to understand broader patterns.
Optimization Effects: The study also found that optimizing ads for clicks showed more pronounced divergent delivery. This suggests selecting different optimization metrics can affect the reliability of A|B tests.
Practical Advice for Brands: If marketers want more reliable data, they should conduct A|B testing on their own websites where they can control randomization.
Combining Lab and Online Studies: For academic research, combining lab studies with online platform studies could offer more comprehensive insights.
Caution for Extrapolation: Findings from these platform-based tests should not be over-extrapolated. Results are often context-specific and temporal.
Click here for the Research Article
The Up Next podcast’s access to this content is courtesy of the International Journal of Research in Marketing, an international, double-blind peer-reviewed journal for marketing academics and practitioners. IJRM aims to contribute to the marketing discipline by providing high-quality, original research which advances marketing knowledge and techniques. As marketers increasingly draw on diverse and sophisticated methods, IJRM‘s target audience is comprised of marketing scholars, practitioners (e.g., marketing research and consulting professionals) and policymakers.
IJRM aims to be at the forefront of the marketing field with a particular emphasis on bringing timely ideas to market. The journal embraces innovative research with the potential to spur future research and influence practice. Hence, it welcomes contributions in various aspects of marketing. The editors, while accepting a wide array of scholarly contributions from different disciplinary approaches, especially encourage research that is novel, visionary or path breaking.
Johannes Boegershausen is an Assistant Professor of Marketing at the Rotterdam School of Management at Erasmus University Rotterdam. Johannes' research examines marketplace morality, stigma, and well-being. He also develops frameworks for improving online data collection methods, such as web scraping or online platform studies. His research has been published in journals such as the Journal of Marketing, Journal of Consumer Research, Journal of Consumer Psychology, and International Journal of Research in Marketing. He has received grants from organizations like the Marketing Science Institute and Accessibility Standards Canada. Johannes has received the 2019 C.W. Park Young Contributor Award from the Society for Consumer Psychology and was a finalist for the 2022 AMA/Marketing Science Institute/H. Paul Root Award. In 2023, he was one of Poets & Quants’ Top 50 Undergraduate Professors. Johannes holds a Ph.D. in Marketing from the University of British Columbia. Prior to joining Erasmus, he worked as an Assistant Professor of Marketing at the University of Amsterdam.
Follow & Learn More
International Journal of Research in Marketing
Interviewee
Johannes Boegershausen
Co Authors
Yann Cornil
Shangwen Yi
Dave Hardisty