Experiments
Experiments are a form of A/B testing used by Discord in the client- and server-side of their applications to serve different experiences or behaviours to different users randomly and/or based on location, client version, etc.
At their core, rollouts are simply YAML inputted into the Discord admin panel; however, not all of the data is required by clients. Therefore, the actual experiments clients see are minified and complex to decipher.
Fingerprints
Even the marketing website uses A/B tests to hook users into the app. Therefore, the app needs a unique way to identify the person using the website without using authentication (for users initially visiting the landing page), so it resorts to using "fingerprints".
These aren't the usual fingerprints generated by collecting information about the browser; instead, they are snowflakes generated
by unauthenticated requests to Get Experiment Assignments. It is expected that fingerprints are sent
in the X-Fingerprint
header in all subsequent requests to the API until authentication, in order to track A/B tests and allow
access to API-locked portions of experiments.
A fingerprint is comprised of a snowflake and a hashed cryptographic value. It looks like this: 1084179945133187083.JQddgNMmwJPghoBtFmaH7jTmdsw
.
When registering a new account, the fingerprint is passed, and (if valid) is used as the created user's ID. This is done in order to preserve experiments across to the registered user. Therefore, a user account's creation time theoretically represents the first time they visited Discord's marketing website.
Rollouts
Experiment rollouts are defined in populations based on the user or guild's rollout position and can have filters to narrow each population's availability.
Example Rollout
Treatments
Control
and None
are represented in the API as the integer values 0
and -1
, respectively. Just as a warning, there are some instances of experiments having specific unnecessary treatments
labelled as Control
indexed as 1
.
Rollout Positions
A rollout position is calculated using the following pseudo code, where exp_name
is the human readable name for the experiment and resource_id
is the user, fingerprint or guild's ID.
This position is used in conjunction with the rollout populations and filters to figure out what the assigned bucket for the experiments are by simply checking which treatment the population is included in.
Data Structures
There are two types of experiments, user experiments and guild experiments. Metadata and human-readable experiment names are available in the user-facing clients. However, API objects do not contain such data, except for guild experiments which may have a human-readable name provided for hash calculations, overriding the one in clients.
Most of the below objects are represented as arrays following the order the fields are documented in.
User Experiments
User experiment data returned by the API is very limited. In contrast to the wide range of data the API provides for guild rollouts, the only values we can programmatically retrieve from the API are the user or fingerprint's assigned bucket for the experiment.
User Experiment Structure
This object is represented as an array of the following fields:
Field | Type | Description |
---|---|---|
hash | integer | 32-bit unsigned Murmur3 hash of the experiment's name |
revision | integer | Current version of the rollout |
bucket | integer | The requesting user or fingerprint's assigned experiment bucket |
override | integer | Whether the user or fingerprint has an override for the experiment (-1 for false, 0 for true) |
population | integer | The internal population group the requesting user or fingerprint is in |
hash_result | integer | The calculated rollout position to use, prioritized over local calculations |
aa_mode 1 | integer | The experiment's A/A testing mode, represented as an integer-casted boolean |
trigger_debugging | integer | Whether the experiment's analytics trigger debugging is enabled, represented as an integer-casted boolean |
1 The bucket for A/A tested experiments should always be None (-1
) unless an override is present for the resource.
Example User Experiment
[4130837190, 0, 10, -1, 0, 1932, 0, 0]
Guild Experiments
The data provided here is more detailed, because the client has to figure out itself the assigned bucket for each guild. It may seem daunting to parse this given the sheer amount of arrays, but it's really quite simple.
Guild Experiment Structure
This object is represented as an array of the following fields:
Field | Type | Description |
---|---|---|
hash | integer | 32-bit unsigned Murmur3 hash of the experiment's name |
hash_key 1 | ?string | A human-readable experiment name (formatted as year-month_name ) to use for hashing calculations, prioritized over the client name |
revision | integer | Current version of the rollout |
populations | array[experiment population object] | The experiment rollout's populations |
overrides 2 | array[experiment bucket override object] | Specific bucket overrides for the experiment |
overrides_formatted 2 | array[array[experiment population object]] | Populations of overrides for the experiment |
holdout_name 3 | ?string | The human-readable experiment name (formatted as year-month_name ) that the experiment is dependent on |
holdout_bucket 3 | ?integer | The required bucket for the experiment the experiment is dependent on |
aa_mode 2 | integer | The experiment's A/A testing mode, represented as an integer-casted boolean |
trigger_debugging | integer | Whether the experiment's analytics trigger debugging is enabled, represented as an integer-casted boolean |
1 Used to categorize multiple experiments together for coordinated rollouts.
2 The bucket for A/A tested experiments should always be None (-1
) unless an override is present for the resource.
3 Failure to meet the holdout experiment's requirements will result in a bucket of None (-1
).
Example Guild Experiment
[1405831955,"2021-06_guild_role_subscriptions",0,[[[[-1,[{"s": 7200,"e": 10000}]],[1,[{"s": 0,"e": 7200}]]],[[2294888943,[[2690752156, 1405831955],[1982804121, 10000]]]]]],[],[[[[[1,[{"s": 0,"e": 10000}]]],[[1604612045, [[1183251248, ["GUILD_ROLE_SUBSCRIPTIONS"]]]]]]]],null,null,0,0]
Experiment Population Object
The population object defines a set of filters and position ranges required to meet specific buckets.
Experiment Population Structure
This object is represented as an array of the following fields:
Field | Type | Description |
---|---|---|
ranges | array[experiment population range object] | The ranges for this population |
filters | experiment population filters object | The filters that the resource must satisfy to be in this population |
Example Experiment Population
[[[-1,[{"s": 7200,"e": 10000}]]],[[2294888943,[[2690752156, 1405831955],[1982804121, 10000]]]]]
Experiment Population Range Object
If the filters in a given population are satisfied and a range includes the resource's rollout position, the resource is then eligible for the given bucket.
Experiment Population Range Structure
This object is represented as an array of the following fields:
Field | Type | Description |
---|---|---|
bucket | integer | The bucket this range grants |
rollout | array[experiment population rollout object] | The range rollout |
Experiment Population Rollout Structure
Field | Type | Description |
---|---|---|
s | integer | The start of this range |
e | integer | The end of this range |
Example Experiment Population Range
[1,[{"s": 0,"e": 4750}]]
Experiment Population Filters Object
This object defines the filters required to be eligible for the ranges. All provided filters must be satisfied for the resource to be eligible for the given bucket.
The filters are an object represented as an array of arrays. The first item in the nested array is a 32-bit unsigned Murmur3 hashed representation of the key, and the second item is the value, with the value being another array-represented object. All structures below are represented in this way.
Experiment Population Filters Structure
Field | Type | Description |
---|---|---|
guild_has_feature? | experiment population guild feature filter object | The guild features that are eligible |
guild_id_range? | experiment population range filter object | The range of snowflake resource IDs that are eligible |
guild_age_range_days? 1 | experiment population range filter object | The range of guild ages (in days) that are eligible |
guild_member_count_range? | experiment population range filter object | The range of guild member counts that are eligible |
guild_ids? | experiment population ID filter object | A list of resource IDs that are eligible |
guild_hub_types? | experiment population hub type filter object | A list of hub types that are eligible |
guild_has_vanity_url? | experiment population vanity URL filter object | Whether the guild must or must not have a vanity to be eligible |
guild_in_range_by_hash? | experiment population range by hash filter object | The special rollout position limits on the population |
1 The guild age is determined from the guild's ID. See the snowflake documentation for more information.
The age can be calculated using the following pseudocode, where resource_id
is the guild's ID:
Experiment Population Guild Feature Filter Structure
Field | Type | Description |
---|---|---|
guild_features | array[string] | The guild features eligible for this population; only one feature is required for eligibility |
Experiment Population Range Filter Structure
Field | Type | Description |
---|---|---|
min_id | ?snowflake | The exclusive minimum for this range, if any |
max_id | ?snowflake | The exclusive maximum for this range, if any |
Experiment Population ID Filter Structure
Field | Type | Description |
---|---|---|
guild_ids | array[snowflake] | The list of snowflake resource IDs that are eligible for this population |
Experiment Population Hub Type Filter Structure
Field | Type | Description |
---|---|---|
guild_hub_types | array[integer] | The type of hubs that are eligible for this population |
Experiment Population Vanity URL Filter Structure
Field | Type | Description |
---|---|---|
guild_has_vanity_url | boolean | The required vanity URL holding status for this population |
Experiment Population Range By Hash Filter Structure
Field | Type | Description |
---|---|---|
hash_key | integer | The 32-bit unsigned Murmur3 hash of the key used to determine eligibility |
target | integer | The rollout position limit for this population |
Example Experiment Population Filters
[[1604612045, // guild_has_feature[[1183251248, // guild_features["ROLE_SUBSCRIPTIONS_ENABLED"]]]]]
Experiment Bucket Override Object
An override represents a manual setting by Discord employees to grant a guild early or specific access to an experiment.
Experiment Bucket Override Structure
Field | Type | Description |
---|---|---|
b | integer | Bucket assigned to these resources |
k | array[snowflake] | Resources granted access to this bucket |
Experiment Bucket Override Example
{"b": 1,"k": ["882680660588904448", "882703776794959873", "859533785225494528", "859533828754505741"]}
Endpoints
Get Experiment Assignments
GET
/experiments
Returns the user experiment assignments and optionally guild experiment rollouts for the user or fingerprint.
Query String Parameters
Field | Type | Description |
---|---|---|
with_guild_experiments? | boolean | Whether to include guild experiments in the returned data |
platform? 1 | string | Whether to also include experiments for the given platform |
1 Including this parameter requires valid authentication.
Experiment Platform
Value | Description |
---|---|
DEVELOPER_PORTAL | The developer portal |
Response Body
Field | Type | Description |
---|---|---|
fingerprint? | string | A generated fingerprint of the current date and time |
assignments | array[experiment assignment object] | The experiment assignments for this user or fingerprint |
guild_experiments? | array[guild experiment object] | Guild experiment rollouts for the client to assign |
Create Fingerprint
POST
/auth/fingerprint
Generates a new fingerprint.
Response Body
Field | Type | Description |
---|---|---|
fingerprint | string | The generated fingerprint of the current date and time |