A/B Testing Intuition with Python | by Nam Vu | Dec, 2020

[ad_1]


1. Creating some data

bcr = 0.1 # Baseline conversion rate
lift = 0.02 # Difference between the 2 groups
size = 4000 # Total size of data, each group will have 2000
df, summary = generate_data_equal_size(size, bcr, lift)
Generated data summary

2. Binomial distribution

Binomial distribution
fig, ax = plt.subplots(figsize=(8,5))
data_a = binom(n_A, p_A).pmf(xx)
data_b = binom(n_B, p_B).pmf(xx)
ax.bar(xx, data_a, alpha=0.5)
ax.bar(xx, data_b, alpha=0.5)
plt.xlabel('converted')
plt.ylabel('probability')
Binomial distribution
Normal distribution of sample mean
outcomes = []
for i in range(100000):
p = 0.1
outcome = np.random.binomial(1, p)
outcomes.append(outcome)
sns.countplot(outcomes)
Count plot Bernoulli trial
sample_means = []# Simulating 1000 samples
for i in range(1000):
# For each sample, we simulate 2000 users
sample = random.choices(outcomes, k=2000)
mean = np.mean(sample)
sample_means.append(mean)
sns.distplot(sample_means)
Distribution for sample mean

3. Forming the hypothesis

fig, ax = plt.subplots()
xx = np.linspace(0.07, 0.17, 1000)

data_a_norm = norm.pdf(xx, p_A, np.sqrt(p_A*(1-p_A) / n_A))
data_b_norm = norm.pdf(xx, p_B, np.sqrt(p_B*(1-p_B) / n_B))

sns.lineplot(xx, data_a_norm, color='blue', ax=ax)
sns.lineplot(xx, data_b_norm, color='red', ax=ax)
ax.axvline(p_A, color='cyan', linestyle='--')
ax.axvline(p_B, color='orange', linestyle='--')
plt.xlabel('converted')
plt.ylabel('probability')
Normal approximation for distribution of mean of group A vs B
Hypothesis
Mean and variance for d
Deriving standard deviation

4. Calculating the sample size

plot_null(p_A=0.1, p_B=0.12, n_A=2000, n_B=2000)
Null hypothesis plot (code)
# Since we draw the graph for 3, let's just assume pooled SE is calculated with bcr=0.1 and lift=0.02
SE_0 = calculate_SE(0.1, 0.12, 2000, 2000)
SE_1 = calculate_SE(0.1, 0.12, 2000, 2000)
SE_2 = calculate_SE(0.1, 0.12, 2000, 2000)
plot_multiple_alt(0, 0.01, 0.02, SE_0, SE_1, SE_2)
Power of different hypothesis with 2000 sample size
# Since we draw the graph for 3, let's just assume pooled SE is calculated with bcr=0.1 and lift=0.02
SE_0 = calculate_SE(0.1, 0.12, 4000, 4000)
SE_1 = calculate_SE(0.1, 0.12, 4000, 4000)
SE_2 = calculate_SE(0.1, 0.12, 4000, 4000)
plot_multiple_alt(0, 0.01, 0.02, SE_0, SE_1, SE_2)
Power of different hypothesis with 4000 sample size
Example of finding sample size
calc_sample_size(alpha=0.05, beta=0.2, p=0.1, delta=0.02, method='pooled_se') # 3843calc_sample_size(alpha=0.05, beta=0.2, p=0.1, delta=0.02, method='evanmiller') #3623

Read More …

[ad_2]


Write a comment