Experiments

Experiments are a form of A/B testing used by Discord in the client- and server-side of their applications to serve different experiences or behaviours to different users randomly and/or based on location, client version, etc.

At their core, rollouts are simply YAML inputted into the Discord admin panel; however, not all of the data is required by clients. Therefore, the actual experiments clients see are minified and complex to decipher.

Fingerprints

Even the marketing website uses A/B tests to hook users into the app. Therefore, the app needs a unique way to identify the person using the website without using authentication (for users initially visiting the landing page), so it resorts to using "fingerprints".

These aren't the usual fingerprints generated by collecting information about the browser; instead, they are snowflakes generated by unauthenticated requests to Get Experiment Assignments. It is expected that fingerprints are sent in the X-Fingerprint header in all subsequent requests to the API until authentication, in order to track A/B tests and allow access to API-locked portions of experiments.

A fingerprint is comprised of a snowflake and a hashed cryptographic value. It looks like this: 1084179945133187083.JQddgNMmwJPghoBtFmaH7jTmdsw.

When registering a new account, the fingerprint is passed, and (if valid) is used as the created user's ID. This is done in order to preserve experiments across to the registered user. Therefore, a user account's creation time theoretically represents the first time they visited Discord's marketing website.

Rollouts

Experiment rollouts are defined in populations based on the user or guild's rollout position and can have filters to narrow each population's availability.

Example Rollout

2023-02_stage_boosting (1816004721)

### Treatment 1

Filters

Guild Features: [COMMUNITY]
Member Count Range: 1000 - null
Hash Range: Hash Key: 1816004721, Target: 10000

Position Ranges

5000-9500, 9500-10000

### Control

Filters

Guild Features: [COMMUNITY]
Member Count Range: 1000 - null

Position Ranges

0 - 10000

### None

Position Ranges

0 - 10000

Treatments

Control and None are represented in the API as the integer values 0 and -1, respectively. Just as a warning, there are some instances of experiments having specific unnecessary treatments labelled as Control indexed as 1.

Rollout Positions

A rollout position is calculated using the following pseudo code, where exp_name is the human readable name for the experiment and resource_id is the user, fingerprint or guild's ID. This position is used in conjunction with the rollout populations and filters to figure out what the assigned bucket for the experiments are by simply checking which treatment the population is included in.

result = mmh3.hash('exp_name:resource_id', signed=False) % 10000

Data Structures

There are two types of experiments, user experiments and guild experiments. Metadata and human-readable experiment names are available in the user-facing clients. However, API objects do not contain such data, except for guild experiments which may have a human-readable name provided for hash calculations, overriding the one in clients.

Most of the below objects are represented as arrays following the order the fields are documented in.

User Experiments

User experiment data returned by the API is very limited. In contrast to the wide range of data the API provides for guild rollouts, the only values we can programmatically retrieve from the API are the user or fingerprint's assigned bucket for the experiment.

User Experiment Structure

This object is represented as an array of the following fields:

¹ The bucket for A/A tested experiments should always be None (-1) unless an override is present for the resource.

Example User Experiment

[4130837190, 0, 10, -1, 0, 1932, 0, 0]

Guild Experiments

The data provided here is more detailed, because the client has to figure out itself the assigned bucket for each guild. It may seem daunting to parse this given the sheer amount of arrays, but it's really quite simple.

Guild Experiment Structure

This object is represented as an array of the following fields:

¹ Used to categorize multiple experiments together for coordinated rollouts.

² The bucket for A/A tested experiments should always be None (-1) unless an override is present for the resource.

³ Failure to meet the holdout experiment's requirements will result in a bucket of None (-1).

Example Guild Experiment

[
  1405831955,
  "2021-06_guild_role_subscriptions",
  0,
  [
    [
      [
        [
          -1,
          [
            {
              "s": 7200,
              "e": 10000
            }
          ]
        ],
        [
          1,
          [
            {
              "s": 0,
              "e": 7200
            }
          ]
        ]
      ],
      [
        [
          2294888943,
          [
            [2690752156, 1405831955],
            [1982804121, 10000]
          ]
        ]
      ]
    ]
  ],
  [],
  [
    [
      [
        [
          [
            1,
            [
              {
                "s": 0,
                "e": 10000
              }
            ]
          ]
        ],
        [[1604612045, [[1183251248, ["GUILD_ROLE_SUBSCRIPTIONS"]]]]]
      ]
    ]
  ],
  null,
  null,
  0,
  0
]

Experiment Population Object

The population object defines a set of filters and position ranges required to meet specific buckets.

Experiment Population Structure

This object is represented as an array of the following fields:

Example Experiment Population

[
  [
    [
      -1,
      [
        {
          "s": 7200,
          "e": 10000
        }
      ]
    ]
  ],
  [
    [
      2294888943,
      [
        [2690752156, 1405831955],
        [1982804121, 10000]
      ]
    ]
  ]
]

Experiment Population Range Object

If the filters in a given population are satisfied and a range includes the resource's rollout position, the resource is then eligible for the given bucket.

Experiment Population Range Structure

This object is represented as an array of the following fields:

Experiment Population Rollout Structure

Example Experiment Population Range

[
  1,
  [
    {
      "s": 0,
      "e": 4750
    }
  ]
]

Experiment Population Filters Object

This object defines the filters required to be eligible for the ranges. All provided filters must be satisfied for the resource to be eligible for the given bucket.

The filters are an object represented as an array of arrays. The first item in the nested array is a 32-bit unsigned Murmur3 hashed representation of the key, and the second item is the value, with the value being another array-represented object. All structures below are represented in this way.

Experiment Population Filters Structure

¹ The guild age is determined from the guild's ID. See the snowflake documentation for more information. The age can be calculated using the following pseudocode, where resource_id is the guild's ID:

timestamp = ((resource_id >> 22) + 1420070400000) / 1000
guild_age = (time.time() - timestamp) / 86400

Experiment Population Guild Feature Filter Structure

Experiment Population Range Filter Structure

Experiment Population ID Filter Structure

Experiment Population Hub Type Filter Structure

Experiment Population Vanity URL Filter Structure

Experiment Population Range By Hash Filter Structure

Example Experiment Population Filters

[
  [
    1604612045, // guild_has_feature
    [
      [
        1183251248, // guild_features
        ["ROLE_SUBSCRIPTIONS_ENABLED"]
      ]
    ]
  ]
]

Experiment Bucket Override Object

An override represents a manual setting by Discord employees to grant a guild early or specific access to an experiment.

Experiment Bucket Override Structure

Experiment Bucket Override Example

{
  "b": 1,
  "k": ["882680660588904448", "882703776794959873", "859533785225494528", "859533828754505741"]
}

Endpoints

Get Experiment Assignments

Returns the user experiment assignments and optionally guild experiment rollouts for the user or fingerprint.

Query String Parameters

¹ Including this parameter requires valid authentication.

Experiment Platform

Response Body

Create Fingerprint

Generates a new fingerprint.

Response Body