ConversionXL (Georgi Georgiev) – Statistics for AB Testing

BECOME A PROPER OPTIMIZER AND KNOW YOUR STATS

You won’t be able to tell whether your tests suck if you aren’t proficient in A/B testing statistics.

A lot of your tests are not winners. If you want to be the person who advocates proper scientific approach in your team, learn to call bullshit.

IN 8 SESSIONS, YOU’LL LEARN

• How to run A/B tests with a sound statistical design.
• How to ask the right questions, avoid common mistakes, and get insights through statistics.
• An in-depth understanding of statistical hypothesis testing and concepts like statistical significance, confidence intervals, statistical power, and others.

UNDERSTAND THE COMPLEXITIES INVOLVED IN PLANNING AND EVALUATING A/B TESTS

Avoid costly testing mistakes stemming from misuse and misunderstanding of statistics, and improve the ROI of all your A/B testing efforts, with the guidance of Georgi Georgiev.

THIS COURSE IS RIGHT FOR YOU IF…

• You can’t define statistical significance correctly without looking it up on Google.
• Your A/B tests produce a lot of “winners,” but your clients aren’t seeing improvements.
• You’re planning and analyzing A/B tests, but you don’t understand the statistical underpinnings of the testing process.
• You’re not confident in the outcomes of your tests and are unsure how much trust to put in them
• You have an in-house statistical tool you want to improve, or you use a third-party A/B testing software you want to understand better

THIS COURSE IS PROBABLY NOT FOR YOU IF…

• You are just starting with CRO and have little to no practical experience with A/B testing.
• You don’t employ A/B tests as a primary method to evaluate CRO work.
• You are a professional statistician or experimental design specialist.

SKILLS YOU SHOULD HAVE BEFORE TAKING THIS COURSE:

• Some experience in conversion rate optimization.
• Basic understanding of how A/B testing works.
• Some experience with an A/B testing software.

ABOUT YOUR INSTRUCTOR,
GEORGI GEORGIEV

He is the mastermind behind the Analytics-ToolKit.com, a software as a service used by web analysts and CRO practitioners.

For the past 10 years, the owner of WebFocus has been delivering world-class marketing and analytic services.

There are multiple marketing events that Georgi is a lecturer in. He is the author of three papers and multiple articles on A/B testing.

You will be able to in just 8 sessions.

• Plan maximally efficient A/B tests.
• Correctly interpret A/B testing statistics.
• Navigate the complexities of MVT, segmentation, multiple KPIs, and concurrent tests.
• Plan and analyze sequential tests.

YOUR FULL COURSE CURRICULUM

STATISTICS FOR A/B TESTING

Lesson 1

What is the reason for the A/B test? The basics of causality inference.

The groundwork will be laid in the first class in order to understand more advanced concepts in subsequent classes. Basic concepts are important for developing a probabilistic mindset.

Topics are covered.

• Correlation and causation. Observational analysis versus controlled experiments.
• Sampling and natural variance and their implications for drawing insights from data.
• Null-hypothesis statistical tests – history and basics of causal inference.
• Control and randomization in A/B tests – why we need them and how they work?
• One-sided and two-sided tests, composite vs. point hypothesis.

Lesson 2

There are statistical significance and confidence intervals.

One of the most abused concepts is statistical significance. You will learn what it is. Common misuses and misunderstandings, their consequences, and how to avoid them will be discussed.

Topics are covered.

• What is statistical significance.
• Common misuses of statistical significance and how to avoid them.
• Common misinterpretations and how to avoid them.
• A/A, A/B/A, A/A/B/B testing – when are they appropriate, and what can they be used for?
• Confidence intervals.

Lesson 3

A/B tests have sample size and statistical power.

Why is statistical power neglected? There are trade-offs involved in planning A/B tests and how to avoid under- and over-powered tests.

Topics are covered.

• What is statistical power and why does it matter?
• The relationship between power and other statistical parameters: significance, sample size, & minimum detectable effect.
• Under-powered and over-powered tests.

Lesson 4

There are multivariate and concurrent tests.

Learn how to plan and analyze a test. The practice of running multiple concurrent tests is examined.

Topics are covered.

• Complexities introduced by testing more than one variant versus control
• When is an A/B/n test preferred to a simple A/B test?
• Do’s and don’ts of running concurrent tests

Lesson 5

There are multiple KPIs and non-binomial tests.

Gain deeper insights by segmenting. We will look at the details of using multiple outcome metrics for a test and non-binomial metrics such as revenue per user.

Topics are covered.

• Segmenting A/B testing data for maximum insights.
• Complexities in running tests with more than one outcome metric.
• Analyzing non-binomial data such as revenue and time on site.

Lesson 6

The testing is sequential.

The future of A/B testing is sequential testing. Learn how to plan and analyze a sequential test.

Topics are covered.

• Shortcomings of classical fixed-sample tests
• The issue of optional stopping
• The alpha-spending approach to sequential testing
• Planning and analyzing a sequential test
• Adaptive tests – benefits and drawbacks

Lesson 7

Asking the right questions can speed up testing.

How can we maximize efficiency in A/B tests? To reflect the questions we answered.

Topics are covered.

• One vs two-tailed significance tests
• Non-inferiority testing
• Nontraditional hypothesis and analysis

Lesson 8

A/B tests should be planned.

How to combine everything from the past seven courses to run highly efficient A/B tests that result in great returns is the cherry on top.

Topics are covered.

• Costs and benefits in A/B testing
• Planning ROI-positive A/B tests

Sale Page: https://conversionxl.com/institute/online-course/statistics-for-ab-testing/
Archive: http://archive.is/T3Opr