Experiments

Experiments are a form of A/B testing used by Discord in the client- and server-side of their applications to serve different experiences or behaviours to different users randomly and/or based on location, client version, etc.

At their core, rollouts are simply YAML inputted into the Discord admin panel; however, not all of the data is required by clients. Therefore, the actual experiments clients see are minified and complex to decipher.

Even the marketing website uses A/B tests to hook users into the app. Therefore, the app needs a unique way to identify the person using the website without using authentication (for users initially visiting the landing page), so it resorts to using "fingerprints".

These aren't the usual fingerprints generated by collecting information about the browser; instead, they are snowflakes generated by unauthenticated requests to Get Experiment Assignments. It is expected that fingerprints are sent in the X-Fingerprint header in all subsequent requests to the API until authentication, in order to track A/B tests and allow access to API-locked portions of experiments.

A fingerprint is comprised of a snowflake and a hashed cryptographic value. It looks like this: 1084179945133187083.JQddgNMmwJPghoBtFmaH7jTmdsw.

When registering a new account, the fingerprint is passed, and (if valid) is used as the created user's ID. This is done in order to preserve experiments across to the registered user. Therefore, a user account's creation time theoretically represents the first time they visited Discord's marketing website.

Experiment rollouts are defined in populations based on the user or guild's rollout position and can have filters to narrow each population's availability.

2023-02_stage_boosting (1816004721)

### Treatment 1

Filters

Guild Features: [COMMUNITY]
Member Count Range: 1000 - null
Hash Range: Hash Key: 1816004721, Target: 10000

Position Ranges

5000-9500, 9500-10000

### Control

Filters

Guild Features: [COMMUNITY]
Member Count Range: 1000 - null

Position Ranges

0 - 10000

### None

Position Ranges

0 - 10000

Control and None are represented in the API as the integer values 0 and -1, respectively. Just as a warning, there are some instances of experiments having specific unnecessary treatments labelled as Control indexed as 1.

A rollout position is calculated using the following pseudo code, where exp_name is the human readable name for the experiment and resource_id is the user, fingerprint or guild's ID. This position is used in conjunction with the rollout populations and filters to figure out what the assigned bucket for the experiments are by simply checking which treatment the population is included in.

result = mmh3.hash('exp_name:resource_id', signed=False) % 10000

There are two types of experiments, user experiments and guild experiments. Metadata and human-readable experiment names are available in the user-facing clients. However, API objects do not contain such data, except for guild experiments which may have a human-readable name provided for hash calculations, overriding the one in clients.

Most of the below objects are represented as arrays following the order the fields are documented in.

User experiment data returned by the API is very limited. In contrast to the wide range of data the API provides for guild rollouts, the only values we can programmatically retrieve from the API are the user or fingerprint's assigned bucket for the experiment.

This object is represented as an array of the following fields:

¹ The bucket for A/A tested experiments should always be None (-1) unless an override is present for the resource.

[4130837190, 0, 10, -1, 0, 1932, 0]

The data provided here is more detailed, because the client has to figure out itself the assigned bucket for each guild. It may seem daunting to parse this given the sheer amount of arrays, but it's really quite simple.

This object is represented as an array of the following fields:

¹ Failure to meet the holdout experiment's requirements will result in a bucket of None (-1).

² The bucket for A/A tested experiments should always be None (-1) unless an override is present for the resource.

[
  1405831955,
  "2021-06_guild_role_subscriptions",
  0,
  [
    [
      [
        [
          -1,
          [
            {
              "s": 7200,
              "e": 10000
            }
          ]
        ],
        [
          1,
          [
            {
              "s": 0,
              "e": 7200
            }
          ]
        ]
      ],
      [
        [
          2294888943,
          [
            [2690752156, 1405831955],
            [1982804121, 10000]
          ]
        ]
      ]
    ]
  ],
  [],
  [
    [
      [
        [
          [
            1,
            [
              {
                "s": 0,
                "e": 10000
              }
            ]
          ]
        ],
        [[1604612045, [[1183251248, ["GUILD_ROLE_SUBSCRIPTIONS"]]]]]
      ]
    ]
  ],
  null,
  null,
  0
]

The population object defines a set of filters and position ranges required to meet specific buckets.

This object is represented as an array of the following fields:

[
  [
    [
      -1,
      [
        {
          "s": 7200,
          "e": 10000
        }
      ]
    ]
  ],
  [
    [
      2294888943,
      [
        [2690752156, 1405831955],
        [1982804121, 10000]
      ]
    ]
  ]
]

If the filters in a given population are satisfied and a range includes the resource's rollout position, the resource is then eligible for the given bucket.

This object is represented as an array of the following fields:

[
  1,
  [
    {
      "s": 0,
      "e": 4750
    }
  ]
]

This object defines the filters required to be eligible for the ranges. All provided filters must be satisfied for the resource to be eligible for the given bucket.

The filters are an object represented as an array of arrays. The first item in the nested array is a 32-bit unsigned Murmur3 hashed representation of the key, and the second item is the value, with the value being another array-represented object. All structures below are represented in this way.

¹ The guild age is determined from the guild's ID. See the snowflake documentation for more information. The age can be calculated using the following pseudocode, where resource_id is the guild's ID:

timestamp = ((resource_id >> 22) + 1420070400000) / 1000
guild_age = (time.time() - timestamp) / 86400

[
  [
    1604612045, // guild_has_feature
    [
      [
        1183251248, // guild_features
        ["ROLE_SUBSCRIPTIONS_ENABLED"]
      ]
    ]
  ]
]

An override represents a manual setting by Discord employees to grant a guild early or specific access to an experiment.

{
  "b": 1,
  "k": ["882680660588904448", "882703776794959873", "859533785225494528", "859533828754505741"]
}

Returns the user experiment assignments and optionally guild experiment rollouts for the user or fingerprint.

Generates a new fingerprint.