​

Running A/B Experiments

​

How Experiments Work

1. You create an experiment in the Console, with two to five variants (e.g., “control” and “variant_a”) and a weight for each (e.g., 50/50), and optionally a Traffic Allocation below 100% and targeting rules.
2. When you start it, its definition is written to your app’s config. The SDK reads that config when it fetches remote config (see Config TTL and Refresh).
3. When your code calls getVariant(), the SDK decides on the device whether the user is in the experiment (platform, targeting rules, traffic allocation — see Who Is in an Experiment) and, if so, computes the variant locally using a deterministic hash.
4. The same user ID always gets the same answer for the same experiment — across sessions, on every platform, and offline.

• Servable surface experiments — test a whole managed surface (paywall, onboarding flow, in-app message, or survey) with no app code. See below.
• Code-level experiments — read the variant with getVariant() and branch yourself. See SDK Usage.

​

Servable Surface Experiments

1. In the Console, create an experiment on the surface. The control variant references the live entity you already present; the treatment variant carries an alternate configuration of that same surface.
2. Start the experiment. The control entity must be published, not a draft, and only one running experiment can use a given entity.
3. Your app keeps presenting the surface exactly as before — e.g. AppDNA.presentPaywall(id:). No experiment code is involved.

• Treatment cohort → renders the treatment configuration in place of the live entity.
• Control cohort, devices on a platform the experiment does not target, and users on older SDK versions → render the live entity unchanged. So does a treatment user whose treatment configuration has not loaded.
• Users who are not in the experiment — outside its targeting rules or its traffic allocation — → render the live entity, and no exposure is recorded for them.

​

Assignment Algorithm

bucket = murmur3_32("<experimentId>.<salt>.<userId>") % 10000

Variants: control (weight 0.5), variant_a (weight 0.5)
  control:   buckets 0-4999
  variant_a: buckets 5000-9999

​

Who Is in an Experiment

1. The experiment is running.
2. It targets this platform (iOS, Android or both).
3. Every targeting rule passes. You set them in the Console’s Targeting step, and an admin or owner of the organization can change them later on the experiment page: Edit Configuration while the experiment is a draft or paused (served when you start or resume it), and Edit targeting on the Targeting card while it is running (served at once). Devices apply a change at their next config fetch; a user who still matches keeps the same variant, a user who no longer matches gets no variant from then on, and exposures already recorded stay in the results. The same holds for targeting sent to PUT /api/v1/experiments/<id>. The rules:

   Rule	How the device checks it
   Countries	The device’s country, not the IP address: the first two-letter ISO country code (such as US) among the current locale’s region and then the device’s other preferred languages’ regions, the same on iOS and Android (the SIM’s and the mobile network’s country are not read). A language-only locale or an area code such as 419 (Latin America) is not a country. If you select countries and no country is found, the user is not in the experiment. No countries selected = every country.
   Minimum App Version	Your app’s version (iOS CFBundleShortVersionString, Android versionName) must be at least this version. Versions are compared number by number (2.9.3 is below 2.10); a missing part counts as 0 and a suffix such as -beta is ignored. If the app’s version cannot be read, the user is not in the experiment.
   New users only	The app was installed on this device at or after the moment the experiment started. An app update keeps the install date; reinstalling the app, or restoring it onto a device from a backup or a device transfer, counts as a new install. iOS: the SDK’s own install marker, excluded from backups and written by the first configure() of an install (on the first launch of this SDK version where an older AppDNA SDK already ran, it takes the app container’s creation date once); an app that adds the AppDNA SDK in an update dates its existing users from that update. Android: PackageInfo.firstInstallTime, which predates the SDK.
   User trait conditions	Every condition must match the user’s traits — the ones your app passes to identify(userId, traits) (each call with traits replaces the traits your app set); on iOS, also the country, region, city and timezone the SDK takes from the IP address at start-up. Those location traits stay through identify, a change of user and reset(); a trait your app sets with the same name wins. There is no API to set traits for an anonymous user, so a condition on a trait your app sets only matches after identify. Operators: =, !=, >, >=, <, <=, in, not in (comma-separated values), contains, exists. Numbers written as text in the Console are compared as numbers; >, >=, < and <= need a numeric trait (a missing trait, text that is not a number, or a true/false trait does not match), and a true/false trait equals true / false, not 1 / 0. A condition the SDK cannot read (no trait name) excludes the user.

4. The user is inside the Traffic Allocation: a second, independent hash puts each user in an allocation bucket,
   allocation_bucket = murmur3_32("<experimentId>.<salt>.<userId>.allocation") % 10000
   
   and the user is in the experiment when allocation_bucket < traffic_allocation × 10000 (at 30%, buckets 0-2999). An experiment without a traffic allocation (one saved before the field existed) includes everyone; an allocation that is not a number includes no one. Because the variant uses a different hash, the allocation does not change the variant split, and raising the allocation keeps every user who was already in, with the same variant.

​

SDK Usage

// Get the assigned variant (synchronous on iOS)
let variant = AppDNA.experiments.getVariant("YOUR_EXPERIMENT_ID")

switch variant {
case "control":
    showStandardPaywall()
case "variant_a":
    showNewPaywall()
default:
    showStandardPaywall()
}

// Get the assigned variant (synchronous on Android)
val variant = AppDNA.experiments.getVariant("YOUR_EXPERIMENT_ID")

when (variant) {
    "control" -> showStandardPaywall()
    "variant_a" -> showNewPaywall()
    else -> showStandardPaywall()
}

// getVariant is async on Flutter -- returns Future<String?>
final variant = await AppDNA.experiments.getVariant("YOUR_EXPERIMENT_ID");

if (variant == "variant_a") {
  showNewPaywall();
} else {
  showStandardPaywall();
}

// getVariant is async on React Native -- returns Promise<string | null>
const variant = await AppDNA.experiments.getVariant("YOUR_EXPERIMENT_ID");

if (variant === "variant_a") {
  showNewPaywall();
} else {
  showStandardPaywall();
}

​

Exposure Tracking

• Calling getVariant("YOUR_EXPERIMENT_ID") again does not record another exposure while the app keeps running.
• The SDK remembers exposures in memory only. After the app is restarted, or after AppDNA.reset(), the next assignment records a new exposure. A new session within the same app run does not.
• No exposure is recorded when getVariant() returns nil / null — including for users outside the experiment’s targeting or traffic allocation.
• There is no public trackExposure() method and no way to defer or disable automatic exposure.

​

Experiment Lifecycle

​

Automatic transitions

• If one of its stop-loss rules fires, it is paused.
• When it reaches its maximum runtime (the Console’s default is 90 days), it is completed. If the analysis has found a winner, that winner is recorded.

​

What happens when an experiment is completed?

• No winner — keep the current experience. The control is the live experience, so “the control won” is completed this way too. Nothing is rolled out.
• A treatment variant, shown with its latest result on the primary metric (value, uplift, p-value). A treatment can be picked only if the latest results show it as a statistically significant improvement over control on the primary metric, in the direction the metric should move, with its confidence interval clear of the control value. Treatments that do not qualify are shown disabled with the reason.

1. The winner, if any, is recorded on the experiment.
2. The experiment is removed from your app’s config. After their next config fetch, devices get nil/null from getVariant() and show the live entity of a servable experiment.
3. Until a device fetches config again, it keeps the experiment definition it already has and keeps assigning variants and recording exposures. A config bundle generated while the experiment was running still contains it, so a fresh install that cannot reach the network uses that definition until its first successful fetch.
4. If Auto-Deploy Winner is on, the winner is rolled out (see below).

​

Auto-Deploy Winner

• Deploy trigger: automatically — the rollout happens at completion.
• Deploy trigger: after approval — the completed experiment shows a Deploy winner button; the rollout happens when an admin clicks it.
• Notify team on deploy — an inbox notification for a rollout, an approval that is waiting, and a rollout that failed.

​

Statistical Analysis

• Sample size — the number of exposed users
• Rate (or revenue per user, for a revenue metric) with a confidence interval (95% by default)
• Uplift — percentage change versus control
• P-value, power and whether the result is significant

​

Best Practices

1. Always handle the default case. If getVariant() returns nil/null (experiment not started, paused, completed, archived, not targeting this platform, or the user is outside its targeting or traffic allocation), fall back to the control experience.
2. Use the experiment’s ID, not its name. getVariant() looks experiments up by the generated ID. Choose descriptive variant IDs (letters, digits and underscores) such as "annual_first" — those are the values your code compares against.
3. Track custom conversion events. In addition to exposure tracking, use AppDNA.track() to record conversion events (e.g., "purchase_completed"), and use the same event name as the experiment’s metric.
4. Test with a sandbox key first. An experiment belongs to the app, not to an environment: once started, it is served to builds using a sandbox key and builds using a production key alike. Events sent with a sandbox key are stored apart from production events, and experiment results are computed from production events only.
5. Decide the allocation before you start. Variants, weights, traffic allocation and the primary metric cannot be changed while an experiment is running or paused. To change them, complete the experiment and start a new one.