A/B Testing

An A/B test is a method of comparing templates to see which performs best. Each A/B test has a default template and up to 20 template variants to compare. You can configure whether we compare the templates based on the number of unique opens or unique clicks.

You can also choose how many messages use each variant in one of two ways:

A percentage of messages
A fixed number of messages

The test will then run until one of the following criteria is met:

The end_time passes
The number of messages accepted equals or exceeds the total_sample_size
The confidence_level has been reached (Bayesian Mode only)

To use the A/B test, send it as the transmission content. Learn more about sending an A/B test.

Once the test ends, we wait the number of hours set in engagement_timeout before marking the test completed. This gives time for the metrics to come in from the emails that were last sent.

The test_mode defines what happens once the A/B test completes. In learning mode, once the test completes all future transmissions use the default template. In bayesian mode, we use a Bayesian algorithm to determined the best performing template. and use that winning template for future transmissions.

In Bayesian mode, when a template reaches the set confidence_level it becomes the winner. If a test ends by hitting one of the other conditions and the confidence_level has not been reached, the default template will be considered the winner.

A/B Tests only support single recipient transmissions currently

Versions

A/B test versions allow you to run multiple tests within the same A/B test. This lets you iterate without changing your transmissions call. The version increments each time the A/B test is updated after its first completed or cancelled run.

A/B Test Object

Example

{
  "id": "password-reset",
  "name": "Password Reset",
  "version": 2,
  "status": "scheduled",
  "metric": "count_unique_confirmed_opened",
  "audience_selection": "sample_size",
  "start_time": "2018-04-03T22:08:33+00:00",
  "test_mode": "bayesian",
  "confidence_level": 0.99,
  "total_sample_size": 60000,
  "engagement_timeout": 24,
  "default_template": {
    "template_id": "default_password_reset_template",
    "sample_size": 20000
  },
  "variants": [
    {
      "template_id": "password_reset_variant1",
      "sample_size": 20000
    },
    {
      "template_id": "password_reset_variant2",
      "sample_size": 20000
    }
  ],
  "created_at": "2018-04-02T22:08:33+00:00",
  "updated_at": "2018-04-02T22:08:33+00:00"
}

Request Body

id string

Unique alphanumeric ID.

name string

Display name.

status enum

The current state of the test.

Possible Values: draft, scheduled, running, completed, cancelled

winning_template_id string

The winner of the A/B test. Only present when the state is completed.

version number

The current version of the test.

default_template object

Details for the default template. See Template Object.

variants array

Specifies which variants to test, as well as how messages are distributed to each variant. See Template Object.

metric enum

Which metric to use when comparing the templates.

Possible Values: count_unique_clicked, count_unique_confirmed_opened

audience_selection enum

Determines how to distribute messages for templates. Each template will receive either a percent of the total of all messages or a set number of messages.

Possible Values: sample_size, percent

test_mode enum

Possible Values: bayesian, learning

start_time string

ISO Date specifying when the test should begin.

end_time string

ISO Date specifying when the test should end. Defaults to 30 days from start_time - engagement_timeout.

total_sample_size number

Total number of messages to send as part of the test.

confidence_level number

Specify a confidence level at which point the test should end.

engagement_timeout number

The amount of time, in hours, a test waits to collect results after the end_time to determine the winner.

created_at string

ISO Date of A/B Test creation.

updated_at string

ISO Date of the last time the A/B test was updated.

Template Object

Attributes

template_id string required

The template id.

sample_size number

The number of injections to send using this template. Required when audience_selection is sample_size.

percent number

The percent of injections to send using this template. Required when audience_selection is percent.

count_unique_clicked number

Read only. The number of unique clicks. Only present when the A/B test's metric type is count_unique_clicked.

count_unique_confirmed_opened number

Read only. The number of confirmed opens. Only present when the A/B test's metric type is count_unique_confirmed_opened.

count_accepted number

Read only. Messages an ISP or other remote domain accepted (less Out-of-Band Bounces).

engagement_rate number

Read only. A number between 0 and 1 calculated by (count_unique_clicked or count_unique_confirmed_opened) / count_accepted.

The templates must have tracking enabled for the A/B test to properly run.

Request: A/B Test using a percentage for distribution

POST /api/v1/ab-test

{
  "id": "payment-confirmation",
  "name": "Payment Confirmation",
  "metric": "count_unique_confirmed_opened",
  "audience_selection": "percent",
  "start_time": "2018-04-03T22:08:33Z",
  "test_mode": "bayesian",
  "confidence_level": 0.99,
  "default_template": {
    "template_id": "default_payment_confirmation_template",
    "percent": 50
  },
  "variants": [
    {
      "template_id": "payment_confirmation_variant1",
      "percent": 25
    },
    {
      "template_id": "payment_confirmation_variant2",
      "percent": 25
    }
  ]
}

Response

{
  "results": {
    "id": "payment-confirmation"
  }
}

{
  "errors": [
    {
      "message": "Variants must have a template_id"
    }
  ]
}

Request: create a test using sample_size for distribution

POST /api/v1/ab-test

{
  "id": "payment-confirmation",
  "name": "Payment Confirmation",
  "metric": "count_unique_confirmed_opened",
  "audience_selection": "sample_size",
  "start_time": "2018-04-03T22:08:33+00:00",
  "test_mode": "learning",
  "total_sample_size": 60000,
  "default_template": {
    "template_id": "default_payment_confirmation_template",
    "sample_size": 40000
  },
  "variants": [
    {
      "template_id": "payment_confirmation_variant1",
      "sample_size": 10000
    },
    {
      "template_id": "payment_confirmation_variant2",
      "sample_size": 10000
    }
  ]
}

Response

{
  "results": {
    "id": "payment-confirmation"
  }
}

{
  "errors": [
    {
      "message": "Variants must have a template_id"
    }
  ]
}

Create an A/B Test

POST/api/v1/ab-test

Request Body

id string

Unique alphanumeric ID used to reference the A/B test. It is auto-generated from the name, if not provided.

name string required

Display name.

default_template object required

Details for the default template. See Template Object.

variants array of objects required

Specifies which variants to test, as well as how messages are distributed to each variant. See Template Object.

metric enum required

Which metric to use when comparing the templates.

Possible Values: count_unique_clicked, count_unique_confirmed_opened

audience_selection enum required

Determines how to distribute messages for templates. Each template will receive either a percent of the total of all messages or a set number of messages.

Possible Values: percent, sample_size

test_mode enum required

Possible Values: bayesian, learning

start_time string required

ISO Date specifying when the test should begin.

end_time string

ISO Date specifying when the test should end. Defaults to 30 days from start_time - engagement_timeout.

total_sample_size number

Total number of messages to send as part of the test.

confidence_level number, default is 0.95

Specify a confidence level at which point the test should end.

engagement_timeout number, default is 24

The amount of time, in hours, a test waits to collect results after the end_time to determine the winner.

Returns the A/B test ID, when successful.

Request

GET /api/v1/ab-test/password-reset

Response

{
  "results": {
    "id": "password-reset",
    "name": "Password Reset",
    "version": 2,
    "status": "scheduled",
    "metric": "count_unique_confirmed_opened",
    "audience_selection": "sample_size",
    "start_time": "2018-04-03T22:08:33+00:00",
    "test_mode": "bayesian",
    "confidence_level": 0.99,
    "total_sample_size": 60000,
    "engagement_timeout": 24,
    "default_template": {
      "template_id": "default_password_reset_template",
      "sample_size": 20000,
      "count_unique_confirmed_opened": 1398,
      "count_accepted": 20321,
      "engagement_rate": 0.068795
    },
    "variants": [
      {
        "template_id": "password_reset_variant1",
        "sample_size": 20000,
        "count_unique_confirmed_opened": 343,
        "count_accepted": 18908,
        "engagement_rate": 0.01814
      },
      {
        "template_id": "password_reset_variant2",
        "sample_size": 20000,
        "count_unique_confirmed_opened": 890,
        "count_accepted": 22987,
        "engagement_rate": 0.038717
      }
    ],
    "created_at": "2018-04-02T22:08:33+00:00",
    "updated_at": "2018-04-02T22:08:33+00:00"
  }
}

{
  "errors": [
    {
      "message": "A/B test password-reset does not exist"
    }
  ]
}

Retrieve an A/B Test

GET/api/v1/ab-test/{id}{?version}

Parameters

id string required

version number

If passed, returns information about the specific version of the A/B test. If not specified, returns information about the latest version.

Request

PUT /api/v1/ab-test/password-reset

{
  "total_sample_size": 100000,
  "default_template": {
    "template_id": "default_password_reset_template",
    "sample_size": 70000
  },
  "variants": [
    {
      "template_id": "password_reset_variant1",
      "sample_size": 10000
    },
    {
      "template_id": "password_reset_variant2",
      "sample_size": 20000
    }
  ]
}

Response

{
  "results": {
    "version": 2
  }
}

{
  "errors": [
    {
      "message": "A/B test password-reset is running"
    }
  ]
}

Update an A/B Test

PUT/api/v1/ab-test/{id}

Parameters

id string required

You can not update an A/B test while it is running. Tests in running state must be cancelled before updating.

If the A/B test's latest version is cancelled or completed, updating it creates a new version, effectively restarting the test. The winning_template_id will be reset for the new version. If winning_template_id existed in the previous version, the default_tempate.template_id of the new version will default to that value, unless overridden by the update.

Request Body

name string

Display name.

default_template object

Details for the default template. See Template Object.

variants array of objects

Specifies which variants to test, as well as how messages are distributed to each variant. See Template Object.

metric enum

Which metric to use when comparing the templates.

Possible Values: count_unique_clicked, count_unique_confirmed_opened

audience_selection enum

Determines how to distribute messages for templates. Each template will receive either a percent of the total of all messages or a set number of messages.

Possible Values: percent, sample_size

test_mode enum

Possible Values: bayesian, learning

start_time string

ISO Date specifying when the test should begin.

end_time string

ISO Date specifying when the test should end. Defaults to 30 days from start_time - engagement_timeout.

total_sample_size number

Total number of messages to send as part of the test.

confidence_level number, default is 0.95

Specify a confidence level at which point the test should end.

engagement_timeout number, default is 24

The amount of time, in hours, a test waits to collect results after the end_time to determine the winner.

Returns

Returns the new version.

Request

POST /api/v1/ab-test/password-reset/cancel

Response

{
  "results": {
    "status": "cancelled"
  }
}

{
  "errors": [
    {
      "message": "A/B test password-reset does not exist"
    }
  ]
}

Cancel an A/B Test

POST/api/v1/ab-test/{id}/cancel

Parameters

id string required

Any transmissions using this A/B test will send the default_template.

Returns the status.

Request

DELETE /api/ab-test/password-reset

Response

// Empty response body

{
  "errors": [
    {
      "message": "Resource could not be found"
    }
  ]
}

{
  "errors": [
    {
      "message": "A/B test 'password-reset' must have status 'cancelled' or 'completed' to use this endpoint"
    }
  ]
}

Delete an A/B Test

DELETE/api/ab-test/{id}

Parameters

id string required

Request

GET /api/v1/ab-test?status=completed

Response

{
  "results": [
    {
      "id": "payment-confirmation",
      "name": "Payment Confirmation",
      "version": 2,
      "status": "running",
      "metric": "count_unique_confirmed_opened",
      "audience_selection": "percent",
      "start_time": "2018-04-03T22:08:33+00:00",
      "test_mode": "bayesian",
      "confidence_level": 0.99,
      "engagement_timeout": 24,
      "default_template": {
        "template_id": "default_payment_confirmation_template",
        "percent": 60,
        "count_unique_confirmed_opened": 1000,
        "count_accepted": 100000,
        "engagement_rate": 0.01
      },
      "variants": [
        {
          "template_id": "payment_confirmation_variant1",
          "percent": 10,
          "count_unique_confirmed_opened": 489,
          "count_accepted": 9000,
          "engagement_rate": 0.054333
        },
        {
          "template_id": "payment_confirmation_variant2",
          "percent": 30,
          "count_unique_confirmed_opened": 320,
          "count_accepted": 68933,
          "engagement_rate": 0.004642
        }
      ],
      "created_at": "2018-04-02T22:08:33+00:00",
      "updated_at": "2018-04-02T22:08:33+00:00"
    },
    {
      "id": "password-reset",
      "name": "Password Reset",
      "version": 2,
      "status": "completed",
      "winning_template_id": "password_reset_variant2",
      "metric": "count_unique_clicked",
      "audience_selection": "percent",
      "start_time": "2018-04-03T22:08:33+00:00",
      "test_mode": "bayesian",
      "confidence_level": 0.99,
      "engagement_timeout": 24,
      "default_template": {
        "template_id": "default_password_reset_template",
        "percent": 70,
        "count_unique_clicked": 8909,
        "count_accepted": 3423230,
        "engagement_rate": 0.002602
      },
      "variants": [
        {
          "template_id": "password_reset_variant1",
          "percent": 15,
          "count_unique_clicked": 398,
          "count_accepted": 90302,
          "engagement_rate": 0.004407
        },
        {
          "template_id": "password_reset_variant2",
          "percent": 15,
          "count_unique_clicked": 231,
          "count_accepted": 73039,
          "engagement_rate": 0.003162
        }
      ],
      "created_at": "2018-04-02T22:08:33+00:00",
      "updated_at": "2018-04-02T22:08:33+00:00"
    }
  ]
}

List All A/B Tests

GET/api/v1/ab-test{?status}

Parameters

status string

If provided, filters the list of A/B tests returned by the provided status. Available options are: draft, scheduled, running, cancelled, completed.

Returns an array of latest versions of your A/B tests.

A/B Test Drafts

A/B Test drafts allow you to build a A/B test in a series of steps instead of all at once.

Request

POST /api/v1/ab-test/draft

{
  "id": "payment-confirmation",
  "name": "Payment Confirmation",
  "default_template": {
    "template_id": "default_payment_confirmation_template",
    "percent": 50
  }
}

Response

{
  "results": {
    "id": "payment-confirmation"
  }
}

{
  "errors": [
    {
      "message": "default_template must have a template_id"
    }
  ]
}

Create an A/B Test draft

POST/api/v1/ab-test/draft

Request Body

id string

Unique alphanumeric ID used to reference the A/B test. It is auto-generated from the name, if not provided.

name string required

Display name.

default_template object required

Details for the default template. See Template Object.

Request

GET /api/v1/ab-test/draft/my-draft-test

Response

{
  "results": {
    "id": "my-draft-test",
    "name": "my draft",
    "version": 1,
    "status": "draft",
    "default_template": {
      "template_id": "my-test-temp",
      "percent": 50
    },
    "created_at": "2018-07-10T21:55:34.960Z",
    "updated_at": "2018-07-11T21:55:47.176Z"
  }
}

Retrieve an A/B Test draft

GET/api/v1/ab-test/draft/{id}

Parameters

id string required

Request

PUT /api/v1/ab-test/draft/payment-confirmation

{
  "metric": "count_unique_confirmed_opened",
  "audience_selection": "percent",
  "test_mode": "bayesian",
  "confidence_level": 0.99,
  "variants": [
    {
      "template_id": "payment_confirmation_variant1",
      "percent": 25
    },
    {
      "template_id": "payment_confirmation_variant2",
      "percent": 25
    }
  ]
}

Response

{
  "results": {
    "id": "payment-confirmation"
  }
}

{
  "errors": [
    {
      "message": "Variants must have a template_id"
    }
  ]
}

Update an A/B Test draft

PUT/api/v1/ab-test/draft/{id}

Parameters

id string required

Request Body

name string

Display name.

default_template object

Details for the default template. See Template Object.

variants array of objects

Specifies which variants to test, as well as how messages are distributed to each variant. See Template Object.

metric enum

Which metric to use when comparing the templates.

Possible Values: count_unique_clicked, count_unique_confirmed_opened

audience_selection enum

Determines how to distribute messages for templates. Each template will receive either a percent of the total of all messages or a set number of messages.

Possible Values: percent, sample_size

test_mode enum

Possible Values: bayesian, learning

start_time string

ISO Date specifying when the test should begin.

end_time string

ISO Date specifying when the test should end. Defaults to 30 days from start_time - engagement_timeout.

total_sample_size number

Total number of messages to send as part of the test.

confidence_level number, default is 0.95

Specify a confidence level at which point the test should end.

engagement_timeout number, default is 24

The amount of time, in hours, a test waits to collect results after the end_time to determine the winner.

Returns the A/B test ID.

Request

DELETE /api/ab-test/draft/password-reset

Response

// Empty response body

{
  "errors": [
    {
      "message": "Resource could not be found"
    }
  ]
}

{
  "errors": [
    {
      "message": "A/B test 'password-reset' must have status 'draft' to use this endpoint"
    }
  ]
}

Delete an A/B Test Draft

DELETE/api/ab-test/draft/{id}

Parameters

id string required

Request

POST /api/v1/ab-test/draft/payment-confirmation/schedule

{
  "start_time": "2018-04-03T22:08:33+00:00",
  "end_time": "2018-04-15T22:08:33+00:00",
  "engagement_timeout": 4
}

Response

{
  "results": {
    "id": "payment-confirmation"
  }
}

{
  "errors": [
    {
      "message": "default_template must have a template_id"
    }
  ]
}

Schedule an A/B Test draft

POST/api/v1/ab-test/draft/{id}/schedule

Parameters

id string required

Request Body

start_time string required

ISO Date specifying when the test should begin.

end_time string

ISO Date specifying when the test should end. Defaults to 30 days from start_time - engagement_timeout.

engagement_timeout number, default is 24

The amount of time, in hours, a test waits to collect results after the end_time to determine the winner.