​

Running A/B Experiments

​

How Experiments Work

1. You create an experiment in the dashboard, defining variants (e.g., “control” and “variant_a”) and traffic allocation (e.g., 50/50).
2. The experiment definition is included in the config bundle that the SDK downloads.
3. When your code calls getVariant(), the SDK computes the assignment locally using a deterministic hash.
4. The same user always gets the same variant — across sessions, across platforms, and even offline.

• Servable surface experiments — test a whole managed surface (paywall, onboarding flow, in-app message, or survey) with no app code. See below.
• Code-level experiments — read the variant with getVariant() and branch yourself. See SDK Usage.

​

Servable Surface Experiments

1. In the Console, create an experiment on the surface. The control variant references the live entity you already present; the treatment variant carries an alternate configuration of that same surface.
2. Start the experiment.
3. Your app keeps presenting the surface exactly as before — e.g. AppDNA.presentPaywall(id:). No experiment code is involved.

• Treatment cohort → renders the treatment configuration in place of the live entity.
• Control cohort, users not in the audience, and users on older SDK versions → render the live entity unchanged.

​

Assignment Algorithm

Experiment: "paywall-test"
  control:   buckets 0-4999   (50%)
  variant_a: buckets 5000-9999 (50%)

User "user-123" → bucket 7342 → variant_a

​

SDK Usage

// Get the assigned variant (synchronous on iOS)
let variant = AppDNA.experiments.getVariant("paywall-test")

switch variant {
case "control":
    showStandardPaywall()
case "variant_a":
    showNewPaywall()
default:
    showStandardPaywall()
}

// Get the assigned variant (synchronous on Android)
val variant = AppDNA.experiments.getVariant("paywall-test")

when (variant) {
    "control" -> showStandardPaywall()
    "variant_a" -> showNewPaywall()
    else -> showStandardPaywall()
}

// getVariant is async on Flutter -- returns Future<String?>
final variant = await AppDNA.experiments.getVariant("paywall-test");

if (variant == "variant_a") {
  showNewPaywall();
} else {
  showStandardPaywall();
}

// getVariant is async on React Native -- returns Promise<string | null>
const variant = await AppDNA.experiments.getVariant("paywall-test");

if (variant === "variant_a") {
  showNewPaywall();
} else {
  showStandardPaywall();
}

​

Exposure Tracking

• Calling getVariant("paywall-test") multiple times in the same session only records one exposure.
• A new exposure is recorded when the user starts a new session.
• There is no public trackExposure() method and no way to defer or disable automatic exposure — exposure fires on the first getVariant() call for each experiment.

​

Experiment Lifecycle

​

What happens when an experiment is stopped?

1. The config bundle is regenerated with the experiment marked as completed.
2. All users now receive the winning variant from getVariant(), regardless of their original assignment.
3. Exposure events are no longer tracked for completed experiments.

​

Statistical Analysis

• Conversion rates per variant with confidence intervals
• Statistical significance (p-value) based on a frequentist approach
• Sample size and exposure counts per variant
• Lift (percentage improvement of treatment over control)

​

Best Practices

1. Always handle the default case. If getVariant() returns nil/null (experiment not found or archived), fall back to the control experience.
2. Use meaningful experiment keys. Choose descriptive keys like "onboarding-v2" or "paywall-annual-price" rather than generic names like "test-1".
3. Track custom conversion events. In addition to exposure tracking, use AppDNA.track() to record conversion events (e.g., "purchase_completed") that you configure as goals in the dashboard.
4. Test in sandbox first. Run experiments in the sandbox environment before deploying to production. Sandbox experiments are completely isolated from production data.
5. Do not change traffic allocation mid-experiment. Changing the allocation while an experiment is running can bias your results. If you need to adjust, consider stopping the experiment and starting a new one.