Saturday, November 11, 2023

Optimizely

https://docs.developers.optimizely.com/feature-experimentation/docs/how-bucketing-works-feature-experimentation

Deterministic – A user sees the same variation across all devices they use every time they see your experiment, thanks to how we hash user IDs. In other words, a returning user is not reassigned to a new variatio

During bucketing, the SDKs rely on the MurmurHash function to hash the user ID and experiment ID to an integer that maps to a bucket range, which represents a variation. MurmurHash is deterministic, so a user ID will always map to the same variation as long as the experiment conditions don’t change. This also means that any SDK will always output the same variation, as long as user IDs and user attributes are consistently shared between systems.

For example, imagine you are running an experiment with two variations (A and B), with an experiment traffic allocation of 40%, and a 50/50 distribution between the two variations. Optimizely will assign each user a number between 0 and 10000 to determine if they qualify for the experiment, and if so, which variation they will see. If they are in buckets 0 to 1999, they see variation A; if they are in buckets 2000 to 3999, they see variation B. If they are in buckets 4000 to 10000, they will not participate in the experiment at all. These bucket ranges are deterministic: if a user falls in bucket 1083, they will always be in bucket 1083.

This operation is highly efficient because it occurs in memory, and there is no need to block on a request to an external service. It also permits bucketing across channels and multiple languages, as well as experimenting without strong network connectivity.

https://www.youtube.com/watch?v=kwNVdSXMGX8&t=20s

What is A/B testing?

https://www.youtube.com/watch?v=hHXEjcJkcbc

https://docs.developers.optimizely.com/feature-experimentation/docs/mutually-exclusive-experiments

How exclusion groups work

For experiments that are not mutually exclusive, Optimizely Feature Experimentation uses a unique value for each experiment to bucket the user. The unique value determines whether a user enters a particular experiment. Because this value is random, unique and not mutually exclusive across experiments, some users may enter multiple experiments. For more information, view the documentation on how bucketing works.

For example, imagine two experiments: A and B. Each receives 20% traffic allocation (the percentage of total traffic that is eligible for the experiment).

Here is the expected traffic allocation:

16% of traffic falls in experiment A only
16% of traffic falls in experiment B only
4% of traffic falls in both experiment A and experiment B
64% of traffic is not in any experiment

In the example above, experiment A's and experiment B's results may be skewed. If users who see both A and B behave differently from those who see just A or B, then the results for A and B are skewed by the overlap. This is called an interaction effect.

If experiments A and B are mutually exclusive, Optimizely Feature Experimentation chooses the same random value (unique to the exclusion group) to bucket users in experiments A and B. This method ensures that experiments cannot overlap for the same users. If experiments A and B are mutually exclusive, the traffic allocation looks something like this: