A/B Testing

An A/B test is a method of comparing templates against a default template to see how their performance compares. Users specify a default template and up to twenty template variants to compare, the comparison is based on the user selected metric. Currently there are two supported modes of audience selection (which recipients receive the variant templates): a fixed number of recipients per variant can be specified, alternatively a percentage of recipients per variant can be specified. There are two supported modes of behavior selection once the A/B test completes. In Learning Mode once the test has completed subsequent transmissions will revert to using the default template. In Bayesian mode the best performing template (the “winner”) as determined by a Bayesian algorithm will be used in subsequent transmissions.

A/B Test Properties

Note: Bayesian test mode is coming soon!
Property Type Description Notes
id string The identifier for this A/B test
name string A human readable name for this A/B test
status string The current state of the test. Possible values: draft, scheduled, running, completed, cancelled GET only
winning_template_id string The “winner” of the A/B test (only present if the state is completed) GET only
version integer The current version number of the test. The version increments each time the A/B test is modified. GET only
default_template object Details for the default template. See Template Properties
variants array Specifies which variants to test, as well as how messages are distributed to each variant. See Template Properties
metric string One of count_unique_clicked, count_unique_confirmed_opened
audience_selection string Determines how to distribute messages for templates. Each template will receive either a percent of the total of all messages or a set number of messages determined by the template’s sample_size. Options are percent, sample_size
test_mode string Either bayesian or learning
start_time string ISO Date specifying when the test should begin
end_time string (Optional) ISO Date specifying when the test should end Defaults to (30 days from start_time - engagement timeout)
total_sample_size int (Optional) Total number of messages to send as part of the test
confidence_level float (Optional) Specify a confidence level at which point the test should end Defaults to 0.95
engagement_timeout int (Optional) The amount of time, in hours, a test waits to collect results after the end_time to make a decision on winner and/or mark test as completed Defaults to 24 hours
created_at string ISO Date of A/B Test Creation GET only
updated_at string ISO Date of the last time the A/B test was updated GET only

Template Properties

Property Type Description Notes
template_id string The template id
sample_size integer The number of injections to send using this template Required if A/B test has total_sample_size defined
percent integer The percent of injections to send using this template Required if A/B test does not have total_sample_size defined
Note: Tests will run until one of the following criteria are met: The end_time has passed, messages equal to the total_sample_size have been sent, or the confidence_level (Bayesian mode only) has been reached. In Bayesian mode, reaching the specified confidence_level for a template will cause it to become the "winner". If a test ends and the confidence_level has not been reached, the default template will be considered the "winner".

A/B Tests

Create an A/B Test

POST/api/v1/ab-test

Examples

Request: A/B Test using a percentage for distribution

POST /api/v1/ab-test

{
  "id": "payment-confirmation",
  "name": "Payment Confirmation",
  "metric": "count_unique_confirmed_opened",
  "audience_selection": "percent",
  "start_time": "2018-04-03T22:08:33Z",
  "test_mode": "bayesian",
  "confidence_level": 0.99,
  "default_template": {
    "template_id": "default_payment_confirmation_template",
    "percent": 50
  },
  "variants": [
    {
      "template_id": "payment_confirmation_variant1",
      "percent": 25
    },
    {
      "template_id": "payment_confirmation_variant2",
      "percent": 25
    }
  ]
}

Response (HTTP status code: 200)

{
  "results": {
    "id": "payment-confirmation"
  }
}

Response (HTTP status code: 400)

{
  "errors": [
    {
      "message": "Variants must have a template_id"
    }
  ]
}

Request: create a test using sample_size for distribution

POST /api/v1/ab-test

{
  "id": "payment-confirmation",
  "name": "Payment Confirmation",
  "metric": "count_unique_confirmed_opened",
  "audience_selection": "sample_size",
  "start_time": "2018-04-03T22:08:33+00:00",
  "test_mode": "learning",
  "total_sample_size": 60000,
  "default_template": {
    "template_id": "default_payment_confirmation_template",
    "sample_size": 40000
  },
  "variants": [
    {
      "template_id": "payment_confirmation_variant1",
      "sample_size": 10000
    },
    {
      "template_id": "payment_confirmation_variant2",
      "sample_size": 10000
    }
  ]
}

Response (HTTP status code: 200)

{
  "results": {
    "id": "payment-confirmation"
  }
}

Response (HTTP status code: 400)

{
  "errors": [
    {
      "message": "Variants must have a template_id"
    }
  ]
}

List All A/B Tests

GET/api/v1/ab-test

Examples

Request

GET /api/v1/ab-test

Response (HTTP status code: 200)

{
  "results": [
    {
      "id": "payment-confirmation",
      "name": "Payment Confirmation",
      "version": 2,
      "status": "running",
      "metric": "count_unique_confirmed_opened",
      "audience_selection": "percent",
      "start_time": "2018-04-03T22:08:33+00:00",
      "test_mode": "bayesian",
      "confidence_level": 0.99,
      "engagement_timeout": 24,
      "default_template": {
        "template_id": "default_payment_confirmation_template",
        "percent": 60
      },
      "variants": [
        {
          "template_id": "payment_confirmation_variant1",
          "percent": 10
        },
        {
          "template_id": "payment_confirmation_variant2",
          "percent": 30
        }
      ],
      "created_at": "2018-04-02T22:08:33+00:00",
      "updated_at": "2018-04-02T22:08:33+00:00"
    },
    {
      "id": "password-reset",
      "name": "Password Reset",
      "version": 2,
      "status": "completed",
      "winning_template_id": "password_reset_variant2",
      "metric": "count_unique_clicked",
      "audience_selection": "percent",
      "start_time": "2018-04-03T22:08:33+00:00",
      "test_mode": "bayesian",
      "confidence_level": 0.99,
      "engagement_timeout": 24,
      "default_template": {
        "template_id": "default_password_reset_template",
        "percent": 70
      },
      "variants": [
        {
          "template_id": "password_reset_variant1",
          "percent": 15
        },
        {
          "template_id": "password_reset_variant2",
          "percent": 15
        }
      ],
      "created_at": "2018-04-02T22:08:33+00:00",
      "updated_at": "2018-04-02T22:08:33+00:00"
    }
  ]
}

A/B Tests Resource

Get an A/B Test

GET/api/v1/ab-test/{id}{?version}
URI Parameters
id(required)  

A/B Test ID

Example: password-reset
version

If passed return information about the specific version of the A/B test. If not specified, return information about the latest version.

Examples

Request

GET /api/v1/ab-test/password-reset?version=

Response (HTTP status code: 200)

{
  "results": {
    "id": "password-reset",
    "name": "Password Reset",
    "version": 2,
    "status": "scheduled",
    "metric": "count_unique_confirmed_opened",
    "audience_selection": "sample_size",
    "start_time": "2018-04-03T22:08:33+00:00",
    "test_mode": "bayesian",
    "confidence_level": 0.99,
    "total_sample_size": 60000,
    "engagement_timeout": 24,
    "default_template": {
      "template_id": "default_password_reset_template",
      "sample_size": 20000
    },
    "variants": [
      {
        "template_id": "password_reset_variant1",
        "sample_size": 20000
      },
      {
        "template_id": "password_reset_variant2",
        "sample_size": 20000
      }
    ],
    "created_at": "2018-04-02T22:08:33+00:00",
    "updated_at": "2018-04-02T22:08:33+00:00"
  }
}

Response (HTTP status code: 404)

{
  "errors": [
    {
      "message": "A/B test password-reset does not exist"
    }
  ]
}

Update an A/B Test

Update an A/B Test

PUT/api/v1/ab-test/{id}
URI Parameters
id(required)  

A/B Test ID

Example: password-reset

Modify an A/B test properties

Note: Updating an A/B test creates a new version of the test if its latest version is cancelled or completed. This effectively causes the test to restart. Tests in `running` state must be cancelled before updating. The winning_template_id will be reset for the new version. If winning_template_id existed in the previous version, the template_id of the new version will default to that value, unless overridden by the update.

Examples

Request

PUT /api/v1/ab-test/password-reset

{
  "total_sample_size": 100000,
  "default_template": {
    "template_id": "default_password_reset_template",
    "sample_size": 70000
  },
  "variants": [
    {
      "template_id": "password_reset_variant1",
      "sample_size": 10000
    },
    {
      "template_id": "password_reset_variant2",
      "sample_size": 20000
    }
  ]
}

Response (HTTP status code: 200)

{
  "results": {
    "version": 2
  }
}

Response (HTTP status code: 409)

{
  "errors": [
    {
      "message": "A/B test password-reset is running"
    }
  ]
}

Cancel an A/B Test

Cancel an A/B Test

POST/api/v1/ab-test/{id}/cancel
URI Parameters
id(required)  

A/B Test ID

Example: password-reset

Examples

Request: Cancel an A/B test

POST /api/v1/ab-test/password-reset/cancel

Response (HTTP status code: 200)

{
  "results": {
    "status": "cancelled"
  }
}

Response (HTTP status code: 404)

{
  "errors": [
    {
      "message": "A/B test password-reset does not exist"
    }
  ]
}

A/B Test Drafts

A/B Test drafts allow a user to set a name and default template on create, and configure tests over several updates before setting a start time.

Note: Only the id, name and default_template are allowed when creating a draft.

Create an A/B Test draft

POST/api/v1/ab-test/draft

Examples

Request: create a draft with only default_template

POST /api/v1/ab-test/draft

{
  "id": "payment-confirmation",
  "name": "Payment Confirmation",
  "default_template": {
    "template_id": "default_payment_confirmation_template",
    "percent": 50
  }
}

Response (HTTP status code: 200)

{
  "results": {
    "id": "payment-confirmation"
  }
}

Response (HTTP status code: 400)

{
  "errors": [
    {
      "message": "default_template must have a template_id"
    }
  ]
}

A/B Test Draft Resource

Get an A/B Test Draft

GET/api/v1/ab-test/draft/{id}
URI Parameters
id(required)  

A/B Test ID

Example: my-draft-test

Examples

Request

GET /api/v1/ab-test/draft/my-draft-test

Response (HTTP status code: 200)

{
  "results": {
    "id": "my-draft-test",
    "name": "my draft",
    "version": 1,
    "status": "draft",
    "default_template": {
      "template_id": "my-test-temp",
      "percent": 50
    },
    "created_at": "2018-07-10T21:55:34.960Z",
    "updated_at": "2018-07-11T21:55:47.176Z"
  }
}

Update an A/B Test Draft

PUT/api/v1/ab-test/draft/{id}
URI Parameters
id(required)  

A/B Test ID

Example: payment-confirmation

Examples

Request

PUT /api/v1/ab-test/draft/payment-confirmation

{
  "metric": "count_unique_confirmed_opened",
  "audience_selection": "percent",
  "test_mode": "bayesian",
  "confidence_level": 0.99,
  "variants": [
    {
      "template_id": "payment_confirmation_variant1",
      "percent": 25
    },
    {
      "template_id": "payment_confirmation_variant2",
      "percent": 25
    }
  ]
}

Response (HTTP status code: 200)

{
  "results": {
    "id": "payment-confirmation"
  }
}

Response (HTTP status code: 400)

{
  "errors": [
    {
      "message": "Variants must have a template_id"
    }
  ]
}

Schedule an A/B Test Draft

Schedule an A/B Test

POST/api/v1/ab-test/draft/{id}/schedule
URI Parameters
id(required)  

A/B Test ID

Example: payment-confirmation

Examples

Request

POST /api/v1/ab-test/draft/payment-confirmation/schedule

{
  "start_time": "2018-04-03T22:08:33+00:00",
  "end_time": "2018-04-15T22:08:33+00:00",
  "engagement_timeout": 4
}

Response (HTTP status code: 200)

{
  "results": {
    "id": "payment-confirmation"
  }
}

Response (HTTP status code: 400)

{
  "errors": [
    {
      "message": "default_template must have a template_id"
    }
  ]
}