Summary
Andrea Bronzini, a seasoned expert in experimentation and conversion optimization, shares insights on the limitations of traditional A/B testing frameworks, the impact of noise on experiment results, and introduces innovative tools to improve decision-making in product growth.
Tools Mentioned
Noise Explorer
Explore the effect of stochastic noise on your experiments and fine-tune your decision policy. https://confidentstory.com/noise/
Noise Check
Evaluate the results of split tests or compare KPIs to rule out noise wobbling. https://confidentstory.com/noisecheck/
GTMsplit
Run split tests in Google Tag Manager (100% free). https://confidentstory.com/gtmsplit/
Key topics
Limitations of traditional A/B testing frameworks
Impact of stochastic noise on experiment outcomes
The importance of uncertainty in decision-making
Tools for simulating experiment outcomes and understanding noise
Reevaluating the use of statistical significance in experiments
Decision policies and experiment duration considerations
The role of noise in KPIs and long-term metrics
Chapters
00:00 Introduction to Andrea Bronzini and the episode’s focus
01:13 The industry’s stagnation in experimentation practices
03:15 Fundamentals of healthy experimentation and decision policies
08:44 Understanding stochastic noise and its visual representation
12:53 The importance of uncertainty over point estimates
19:34 Simulating experiment outcomes to inform decision-making
23:43 The impact of noise on KPIs and long-term metrics
27:21 Reevaluating decision thresholds and significance levels
32:33 Practical steps for product teams to improve decision quality
36:27 Tools for noise detection and better experiment analysis
38:12 Closing remarks and where to learn more about Andrea’s tools
