Skip to main content

Experiments (A/B Tests)

An A/B test, called an experiment in Perspective, asks one question: does a changed version of your funnel convert better than the one you have now? Perspective keeps your funnel as it is (the original) and makes a copy with one idea changed (the variant). When the test runs, each visitor is sent to one of the two, following a split such as 50/50. After enough visitors have come through, you compare the share of visitors who converted on each side, the conversion rate. The version with the higher conversion rate wins, as long as the gap is unlikely to be luck.

:::note Availability Experiments are available on AI/HTML funnels for accounts with A/B testing enabled. The MCP server follows the same setting as the Perspective app: if your account does not have it, these tools are not listed. Classic (non-HTML) funnels cannot be tested. :::

Key ideas​

  • One test at a time. A funnel has at most one current test, which is either draft (set up, not live yet) or running (live, visitors are being split). Finished tests stay in the history as concluded (ended with a significant result) or aborted (ended without one), and never change again.
  • Two versions, one funnel. The original and the variant are two pages behind the same funnel URL. You edit either one with update_funnel by naming the version.
  • Publishing is the switch. Nothing reaches a visitor until the funnel is published. That is true for starting a test, and also for changes made to a test that is already running.

The lifecycle​

  1. Create the test with create_experiment. You give it a name, a hypothesis ("a shorter form will convert better") and, optionally, a split. The test is saved as a draft, and Perspective's AI writes the variant from the hypothesis. The original is not touched. Poll get_funnel_job_status with the returned jobId until the variant is written.
  2. Edit either version with update_funnel, passing version: "original" or version: "variant". While a test exists, every edit has to say which version it is for.
  3. Change the split or the name with update_experiment, for example from 50/50 to 70/30.
  4. Start the test with publish_funnel. Publishing a funnel with a draft test puts both versions live and starts splitting visitors. The first call asks for confirmation instead of publishing (see Starting a test).
  5. Read the results with get_experiment_results while the test runs.
  6. End the test with end_experiment and keep one version. Every visitor then sees the kept version, and the split stops. A test that never started can be deleted with discard_experiment instead.

get_experiment and list_experiments tell you, at any step, whether a test exists and what state it is in.

:::warning Changes go live on publish While a test is running, editing either version (update_funnel) or changing the split (update_experiment) is saved at once, but visitors do not see it yet. They keep getting the old content and the old split until the funnel is published again with publish_funnel.

Think of it like a printed menu: you can rewrite the menu in the kitchen, but guests only see the change once you print and hand out the new copies. Publishing is the printing step.

On a draft test there is nothing to republish yet, but keep in mind that the next publish of the funnel, whatever it is for, starts the test. :::

Tools​

ToolWhat it doesKey inputsConfirm with the user first
get_experimentRead the funnel's current test, or a past one by idfunnelId, experimentId?No
list_experimentsList the funnel's tests, newest firstfunnelId, status?No
create_experimentSet up a draft test and have the AI write the variantfunnelId, title, hypothesis, weights?Yes
update_experimentRename the current test or change its splitfunnelId, title?, weights?No (republishing a running test does)
get_experiment_resultsHow each version is doing, and whether there is an answer yetfunnelId, experimentId?No
end_experimentEnd a running test and keep one versionfunnelId, winnerYes, including which version to keep
discard_experimentDelete a test that has not started, with its variantfunnelIdYes

Two funnel tools take part in the test flow too. update_funnel edits one version through its version input, and publish_funnel starts or refreshes the test. Both are documented on the Funnels page.

get_experiment​

Reads a funnel's A/B test. Without experimentId it returns the current test (draft or running), or data: null with a message when the funnel has none. Having no test is a normal result, not an error. Call it before editing or publishing a funnel to find out whether a test exists.

NameTypeRequiredDescription
funnelIdstringYesID of the AI/HTML funnel (from list_funnels or create_funnel)
experimentIdstringNoID of a specific test (from list_experiments), in any status. Omit for the current test

Returns { data: { id, title, status, startedAt?, endedAt?, winnerArmId?, arms }, message? }. Each item in arms is one version of the page:

FieldDescription
idID of the arm; winnerArmId refers to it
versionoriginal or variant
weightPercent of the funnel's visitors sent to this version. On a running test this is the saved split, which reaches visitors at the next publish
draftIdID of the draft holding this version's page
rolecontrol or challenger; the same information as version

list_experiments​

Lists a funnel's tests, newest first: the current one and the past ones. Each item is a summary without arms; call get_experiment with its id for the full test. Without status, an empty data array means the funnel has no saved tests (a discarded draft is deleted, so it is not listed); with it, only that no saved test has those statuses.

NameTypeRequiredDescription
funnelIdstringYesID of the AI/HTML funnel
statusstring[]NoOnly return tests in these statuses, e.g. ["concluded", "aborted"] for past tests. Omit for every status

Returns { data: [{ id, title, status, startedAt?, endedAt?, winnerArmId? }] }.

create_experiment​

Sets up an A/B test on an AI/HTML funnel, with the variant written by AI from a hypothesis. The test is created as a draft and the AI starts writing the variant right away. The original is not changed, and nothing is live yet: no visitor is split and the published funnel stays as it is.

NameTypeRequiredDescription
funnelIdstringYesID of the AI/HTML funnel
titlestringYesName of the test, up to 40 characters
hypothesisstringYesWhat the variant should change and why, in plain words, e.g. "A shorter form with only name and email will convert better than the current five fields." The AI writes the variant from this, so be specific
weights{ original, variant }NoTraffic split as whole percents from 5 to 95 that add up to 100, e.g. { "original": 70, "variant": 30 }. Omit for 50/50

Returns { data, jobId?, message }. data is the new test in the same shape as get_experiment. Poll get_funnel_job_status with jobId until it reports completed; its summary describes what the variant changes.

If the AI could not start writing the variant, the test is still created, but jobId is missing and the variant is an exact copy of the original. To retry, call update_funnel with version: "variant" and a prompt that restates the whole hypothesis. To start over instead, discard the test and create it again. When a plan limit stopped the AI, for example because the account has run out of AI credits, message says what lifts the limit; retry once it is lifted.

Errors (nothing is created):

  • FUNNEL_GENERATION_IN_PROGRESS: an AI generation is still running on the funnel. Wait for it with get_funnel_job_status.
  • EXPERIMENT_ALREADY_EXISTS: the funnel already has a draft or running test. Look at it with get_experiment and change it with update_experiment.

update_experiment​

Renames the funnel's current test or changes how traffic is split between the two versions. Works on a draft or running test; a test that has ended cannot be changed. Pass at least one of title and weights.

NameTypeRequiredDescription
funnelIdstringYesID of the AI/HTML funnel
titlestringNoNew name, up to 40 characters
weights{ original, variant }NoNew split, whole percents from 5 to 95 that add up to 100

Returns { data, message? }: data is the test after the change, and message, present when the split changed, says when the new split reaches visitors. On a running test, a new split applies only after the next publish_funnel (see Changes go live on publish); on a draft test it applies when the test starts.

Errors: NOTHING_TO_UPDATE when neither field is passed. NO_CURRENT_EXPERIMENT when the funnel has no draft or running test.

Starting a test​

There is no separate "start" tool. A draft test starts the next time the funnel is published, and publish_funnel guards this with a confirmation step:

  1. The first publish_funnel call on a funnel with a draft test does not publish. It fails with EXPERIMENT_START_NOT_CONFIRMED and returns the test's name and split.
  2. Confirm with the user that they want to start the test: both versions go live and visitors start being split. Agreeing to "publish my funnel" is not the same as agreeing to start a test.
  3. Call publish_funnel again with the same inputs plus confirmStartExperiment: true.

After a publish, the response's message says what happened to the test:

  • Started: the test is live and visitors are split.
  • Refreshed: the test was already running; the latest edits and split now reach visitors.
  • Not started: the user's role does not allow starting tests. The funnel is still published, showing only the original, and the test stays a draft.
  • Failed: the funnel is published, but starting or refreshing the test failed. A draft test stays a draft, and a running test keeps serving each version as of the previous publish. Publishing again retries it.

Publishing a funnel with a draft test always tries to start the test. To publish it without the test, discard the test first.

get_experiment_results​

Shows how each version is doing and whether the test has an answer yet. Without experimentId it reads the current test; with one, any past test.

NameTypeRequiredDescription
funnelIdstringYesID of the AI/HTML funnel
experimentIdstringNoID of a past or current test (from list_experiments). Omit for the current test

Returns { data, message }. message is a one-sentence plain-language reading of the numbers, such as "Not enough data yet: 42 % of the ~3,100 visitors this test needs." data is null when the funnel has no test or the test has not started.

data contains:

  • experiment: id, title, status (running, concluded or aborted).
  • range: from and to, the period the numbers cover (from the test start to its end, or to now).
  • arms: one entry per version, the original first: id, version, sessions (visitors), conversions, conversionRate (0 to 1) and weight.
  • significance: null while a version has had no visitors. Otherwise:
FieldMeaning
progressHow much of the needed data the test has collected, from 0 to 1
isSignificanttrue when the gap between the versions is unlikely to be chance (95 % level)
leadingVersionThe version with the higher conversion rate, or null if they are equal
absoluteLiftVariant rate minus original rate, e.g. 0.02 = 2 percentage points
relativeLiftabsoluteLift divided by the original's rate, e.g. 0.25 = 25 % better. null while the original has no conversions
visitorsNeededEstimated total visitors (both versions) needed for a significant result, if the current gap holds. null while both convert equally

:::info progress is data collected, not the chance of winning Picture flipping two coins to see which one lands heads more often. After 10 flips each, a 6-to-4 gap could easily be luck. After 10,000 flips, the same gap is very unlikely to be luck. progress tells you how far along those flips you are.

So 60 % progress does not mean a 60 % chance that the variant wins. It means the test has 60 % of the data it needs to tell the versions apart. The Perspective app shows this number as "Confidence". :::

Two more rules for reading results:

  • progress reaches 1 exactly when isSignificant turns true.
  • Significant only means the gap is unlikely to be chance, not that the variant is better. The variant has won only when isSignificant is true and leadingVersion is variant. If the original leads, the change made things worse.

end_experiment​

Ends a running test and keeps one of its two versions. Every visitor then sees the kept version, the other one stops being served, and traffic is no longer split. Ending is final: a test cannot be restarted. To test again, create a new one.

Before calling it, show the user the latest results and let them choose which version to keep. If the chosen version is not significantly ahead, the test can still be ended, but the numbers do not prove that version is better.

NameTypeRequiredDescription
funnelIdstringYesID of the AI/HTML funnel
winner"original" | "variant"YesThe version to keep

Returns { data, message }. data is the ended test: status is concluded when the kept version's result was significant, aborted when it was not, and winnerArmId is the kept arm. message says what the funnel serves now:

  • Live funnel: every visitor sees the kept version. The split can take up to a minute to stop everywhere.
  • Offline funnel: nothing is republished. The kept version goes live at the next publish_funnel.
  • Promotion failed: the test has ended, but putting the kept version live failed, so visitors may still be split. Publishing the funnel again fixes it.

Errors (nothing is changed):

  • NO_RUNNING_EXPERIMENT: the funnel has no running test, or it has already ended.
  • EXPERIMENT_NOT_STARTED: the test is still a draft. Use discard_experiment instead.
  • EXPERIMENT_RESULTS_UNAVAILABLE: the results could not be read. Try again shortly.

discard_experiment​

Deletes a test that has not started (draft). The test and its variant, with every edit made to it, are permanently deleted. The original is untouched, so the funnel has no test afterwards and a new one can be created right away.

NameTypeRequiredDescription
funnelIdstringYesID of the AI/HTML funnel

Returns { message }.

Errors: NO_DRAFT_EXPERIMENT when the funnel has no test that has not started. A running test cannot be discarded; end it with end_experiment, which keeps one version.

Errors shared by every tool​

  • FUNNEL_NOT_FOUND: the funnel does not exist, is not accessible to you, or is a classic funnel. Check the id with list_funnels; a funnel with isHtmlFunnel: false is classic and cannot be tested.
  • EXPERIMENT_NOT_FOUND (tools that take experimentId): that test is not on this funnel. list_experiments shows the valid ids.
  • Tools: Funnels — update_funnel with version, publish_funnel with confirmStartExperiment
  • Scopes: Tool Scopes — which scopes include the experiment tools
  • Guide: Metrics & analytics — the abTest chart filter for per-page conversion by version