Experiments

Experiments are a form of A/B testing used by Discord in the client- and server-side of their applications to serve different experiences or behaviours to different users randomly and/or based on location, client version, etc.

At their core, rollouts are simply YAML inputted into the Discord admin panel; however, not all of the data is required by clients. Therefore, the actual experiments clients see are minified and complex to decipher.

Fingerprints

Even the marketing website uses A/B tests to hook users into the app. Therefore, the app needs a unique way to identify the person using the website without using authentication (for users initially visiting the landing page), so it resorts to using "fingerprints".

These aren't the usual fingerprints generated by collecting information about the browser; instead, they are snowflakes generated by unauthenticated requests to Get Experiment Assignments. It is expected that fingerprints are sent in the X-Fingerprint header in all subsequent requests to the API until authentication, in order to track A/B tests and allow access to API-locked portions of experiments.

A fingerprint is comprised of a snowflake and a hashed cryptographic value. It looks like this: 1084179945133187083.JQddgNMmwJPghoBtFmaH7jTmdsw.

When registering a new account, the fingerprint is passed, and (if valid) is used as the created user's ID. This is done in order to preserve experiments across to the registered user. Therefore, a user account's creation time theoretically represents the first time they visited Discord's marketing website.

Rollouts

Experiment rollouts are defined in populations based on the user or guild's rollout position and can have filters to narrow each population's availability.

Example Rollout
2023-02_stage_boosting (1816004721)


### Treatment 1


Filters


Guild Features: [COMMUNITY]
Member Count Range: 1000 - null
Hash Range: Hash Key: 1816004721, Target: 10000


Position Ranges


5000-9500, 9500-10000


### Control


Filters


Guild Features: [COMMUNITY]
Member Count Range: 1000 - null


Position Ranges


0 - 10000


### None


Position Ranges


0 - 10000

Treatments

Control and None are represented in the API as the integer values 0 and -1, respectively. Just as a warning, there are some instances of experiments having specific unnecessary treatments labelled as Control indexed as 1.

Rollout Positions

A rollout position is calculated using the following pseudo code, where exp_name is the human readable name for the experiment and resource_id is the user, fingerprint or guild's ID. This position is used in conjunction with the rollout populations and filters to figure out what the assigned bucket for the experiments are by simply checking which treatment the population is included in.

result = mmh3.hash('exp_name:resource_id', signed=False) % 10000

Data Structures

There are two types of experiments, user experiments and guild experiments. Metadata and human-readable experiment names are available in the user-facing clients. However, API objects do not contain such data, except for guild experiments which may have a human-readable name provided for hash calculations, overriding the one in clients.

Most of the below objects are represented as arrays following the order the fields are documented in.

User Experiments

User experiment data returned by the API is very limited. In contrast to the wide range of data the API provides for guild rollouts, the only values we can programmatically retrieve from the API are the user or fingerprint's assigned bucket for the experiment.

User Experiment Structure

This object is represented as an array of the following fields:

FieldTypeDescription
hashinteger32-bit unsigned Murmur3 hash of the experiment's name
revisionintegerCurrent version of the rollout
bucketintegerThe requesting user or fingerprint's assigned experiment bucket
overrideintegerWhether the user or fingerprint has an override for the experiment (-1 for false, 0 for true)
populationintegerThe internal population group the requesting user or fingerprint is in
hash_resultintegerThe calculated rollout position to use, prioritized over local calculations
aa_mode 1integerThe experiment's A/A testing mode, represented as an integer-casted boolean
trigger_debuggingintegerWhether the experiment's analytics trigger debugging is enabled, represented as an integer-casted boolean

1 The bucket for A/A tested experiments should always be None (-1) unless an override is present for the resource.

Example User Experiment
[4130837190, 0, 10, -1, 0, 1932, 0, 0]

Guild Experiments

The data provided here is more detailed, because the client has to figure out itself the assigned bucket for each guild. It may seem daunting to parse this given the sheer amount of arrays, but it's really quite simple.

Guild Experiment Structure

This object is represented as an array of the following fields:

FieldTypeDescription
hashinteger32-bit unsigned Murmur3 hash of the experiment's name
hash_key 1?stringA human-readable experiment name (formatted as year-month_name) to use for hashing calculations, prioritized over the client name
revisionintegerCurrent version of the rollout
populationsarray[experiment population object]The experiment rollout's populations
overrides 2array[experiment bucket override object]Specific bucket overrides for the experiment
overrides_formatted 2array[array[experiment population object]]Populations of overrides for the experiment
holdout_name 3?stringThe human-readable experiment name (formatted as year-month_name) that the experiment is dependent on
holdout_bucket 3?integerThe required bucket for the experiment the experiment is dependent on
aa_mode 2integerThe experiment's A/A testing mode, represented as an integer-casted boolean
trigger_debuggingintegerWhether the experiment's analytics trigger debugging is enabled, represented as an integer-casted boolean

1 Used to categorize multiple experiments together for coordinated rollouts.

2 The bucket for A/A tested experiments should always be None (-1) unless an override is present for the resource.

3 Failure to meet the holdout experiment's requirements will result in a bucket of None (-1).

Example Guild Experiment
[
  1405831955,
  "2021-06_guild_role_subscriptions",
  0,
  [
    [
      [
        [
          -1,
          [
            {
              "s": 7200,
              "e": 10000
            }
          ]
        ],
        [
          1,
          [
            {
              "s": 0,
              "e": 7200
            }
          ]
        ]
      ],
      [
        [
          2294888943,
          [
            [2690752156, 1405831955],
            [1982804121, 10000]
          ]
        ]
      ]
    ]
  ],
  [],
  [
    [
      [
        [
          [
            1,
            [
              {
                "s": 0,
                "e": 10000
              }
            ]
          ]
        ],
        [[1604612045, [[1183251248, ["GUILD_ROLE_SUBSCRIPTIONS"]]]]]
      ]
    ]
  ],
  null,
  null,
  0,
  0
]

Experiment Population Object

The population object defines a set of filters and position ranges required to meet specific buckets.

Experiment Population Structure

This object is represented as an array of the following fields:

FieldTypeDescription
rangesarray[experiment population range object]The ranges for this population
filtersexperiment population filters objectThe filters that the resource must satisfy to be in this population
Example Experiment Population
[
  [
    [
      -1,
      [
        {
          "s": 7200,
          "e": 10000
        }
      ]
    ]
  ],
  [
    [
      2294888943,
      [
        [2690752156, 1405831955],
        [1982804121, 10000]
      ]
    ]
  ]
]

Experiment Population Range Object

If the filters in a given population are satisfied and a range includes the resource's rollout position, the resource is then eligible for the given bucket.

Experiment Population Range Structure

This object is represented as an array of the following fields:

FieldTypeDescription
bucketintegerThe bucket this range grants
rolloutarray[experiment population rollout object]The range rollout
Experiment Population Rollout Structure
FieldTypeDescription
sintegerThe start of this range
eintegerThe end of this range
Example Experiment Population Range
[
  1,
  [
    {
      "s": 0,
      "e": 4750
    }
  ]
]

Experiment Population Filters Object

This object defines the filters required to be eligible for the ranges. All provided filters must be satisfied for the resource to be eligible for the given bucket.

The filters are an object represented as an array of arrays. The first item in the nested array is a 32-bit unsigned Murmur3 hashed representation of the key, and the second item is the value, with the value being another array-represented object. All structures below are represented in this way.

Experiment Population Filters Structure
FieldTypeDescription
guild_has_feature?experiment population guild feature filter objectThe guild features that are eligible
guild_id_range?experiment population range filter objectThe range of snowflake resource IDs that are eligible
guild_age_range_days? 1experiment population range filter objectThe range of guild ages (in days) that are eligible
guild_member_count_range?experiment population range filter objectThe range of guild member counts that are eligible
guild_ids?experiment population ID filter objectA list of resource IDs that are eligible
guild_hub_types?experiment population hub type filter objectA list of hub types that are eligible
guild_has_vanity_url?experiment population vanity URL filter objectWhether the guild must or must not have a vanity to be eligible
guild_in_range_by_hash?experiment population range by hash filter objectThe special rollout position limits on the population

1 The guild age is determined from the guild's ID. See the snowflake documentation for more information. The age can be calculated using the following pseudocode, where resource_id is the guild's ID:

timestamp = ((resource_id >> 22) + 1420070400000) / 1000
guild_age = (time.time() - timestamp) / 86400
Experiment Population Guild Feature Filter Structure
FieldTypeDescription
guild_featuresarray[string]The guild features eligible for this population; only one feature is required for eligibility
Experiment Population Range Filter Structure
FieldTypeDescription
min_id?snowflakeThe exclusive minimum for this range, if any
max_id?snowflakeThe exclusive maximum for this range, if any
Experiment Population ID Filter Structure
FieldTypeDescription
guild_idsarray[snowflake]The list of snowflake resource IDs that are eligible for this population
Experiment Population Hub Type Filter Structure
FieldTypeDescription
guild_hub_typesarray[integer]The type of hubs that are eligible for this population
Experiment Population Vanity URL Filter Structure
FieldTypeDescription
guild_has_vanity_urlbooleanThe required vanity URL holding status for this population
Experiment Population Range By Hash Filter Structure
FieldTypeDescription
hash_keyintegerThe 32-bit unsigned Murmur3 hash of the key used to determine eligibility
targetintegerThe rollout position limit for this population
Example Experiment Population Filters
[
  [
    1604612045, // guild_has_feature
    [
      [
        1183251248, // guild_features
        ["ROLE_SUBSCRIPTIONS_ENABLED"]
      ]
    ]
  ]
]

Experiment Bucket Override Object

An override represents a manual setting by Discord employees to grant a guild early or specific access to an experiment.

Experiment Bucket Override Structure
FieldTypeDescription
bintegerBucket assigned to these resources
karray[snowflake]Resources granted access to this bucket
Experiment Bucket Override Example
{
  "b": 1,
  "k": ["882680660588904448", "882703776794959873", "859533785225494528", "859533828754505741"]
}

Endpoints

Get Experiment Assignments

GET/experiments

Returns the user experiment assignments and optionally guild experiment rollouts for the user or fingerprint.

Query String Parameters
FieldTypeDescription
with_guild_experiments?booleanWhether to include guild experiments in the returned data
Response Body
FieldTypeDescription
fingerprint?stringA generated fingerprint of the current date and time
assignmentsarray[experiment assignment object]The experiment assignments for this user or fingerprint
guild_experiments?array[guild experiment object]Guild experiment rollouts for the client to assign

Create Fingerprint

POST/auth/fingerprint
Deprecated

Generates a new fingerprint.

Response Body
FieldTypeDescription
fingerprintstringThe generated fingerprint of the current date and time