Building a test in your theme code

Most tests are built in Ordinary’s visual editor. Build the variant in your theme code instead when the change needs Liquid or theme data: a price or subscription plan read from the product, a new section, a menu item (menus usually live outside sections), or one change that spans several templates.

The idea is simple. Your theme ships both versions of the thing you are testing. Ordinary decides which one each visitor sees, keeps that choice for the life of the test, and scores the result on your reconciled revenue like any other test.

The markup

Wrap each version in an element that names the test and the group it belongs to:

<li data-ordinary-only="{{ test_id }}:control">
  <a href="/pages/quiz">Quiz</a>
</li>
<li data-ordinary-only="{{ test_id }}:variant" data-oyx-d="flex" style="display:none">
  <a href="/products/discovery-set">Discovery Set</a>
</li>

Four rules:

• The test ID is the ID shown on the test’s page in Ordinary. It has to match exactly.
• The group key after the colon must match each group’s Key on the test’s Test groups tab — not its Name. Each group has both: the Key (lowercase, up to 16 characters, for example control or variant) is what your markup uses, and the Name is only the label you see in Ordinary’s reports. The default keys are control and variant. Renaming a group doesn’t change its key, so your markup keeps working.
• Every group except the control ships hidden, with style="display:none" and a data-oyx-d attribute holding the display value to restore when it is shown (block, flex, grid, inline-block, list-item, and the other common values). Ordinary only switches versions while the test is active, so a hidden variant stays hidden before Start and after End.
• The control can be the page as it is. A section that only exists in the variant needs no control copy; the control group simply sees nothing there.

A whole section can be one arm: wrap the section’s content in a single <div> carrying the attributes, and ship two copies of the section in the template, one per arm. Put the test ID and the arm in the section’s settings rather than in the code, so the theme editor shows them.

A group brings its own styling

A test on a path runs on every template served at that path, so a section used as a group has to look right wherever it appears. Keep the section’s styling in the section’s own file, scoped to that section. Styling kept in a separate section that only one template includes, or written for the section’s name in one particular template, stays behind when the section is used on another template, and visitors there see the group without its layout.

When Ordinary checks a test’s page, it also compares the page with the other templates the test runs on, such as an alternate template used by a split-URL test on the same path. If a group is styled by a separate section on one of them and not on another, the check names both sections and both pages. Styling in a linked stylesheet file is not part of that comparison, so open each group on every template before you start.

Keep the test ID in one place

Every test in Ordinary has its own ID, and a test you delete and create again gets a new one. A theme that still carries the old ID shows the variant to nobody and counts nobody. Store the ID in a theme setting or a section setting, never hardcoded in a snippet, and update it whenever the test is recreated. When you start a test, Ordinary checks the page for its markup and shows what it found on the test’s page; read that before you move on.

Theme-code tests and the visual editor

Choose Theme code when you create the test. A test built in your theme code does not open in the visual editor: its groups live in the theme, so the editor has nothing to change, and its page pattern and metric stay as you set them on the test’s page. While the test is still a draft, Build in visual editor on its page switches it to the editor; from then on Ordinary puts each group’s changes on your theme when the test starts, and the pre-Start check for your markup no longer applies to it. Once a test has started, how it is delivered is fixed.

Who counts as a visitor

A visitor counts in a same-page test when the test’s content was on the page they loaded and Ordinary was not hiding it from them. Visitors on pages without the markup are not counted, and a test placed inside another test’s variant counts only the visitors who could see it. That is what lets you run one test inside another without diluting the result.

One element, one test

Give each change its own test instead of bundling several into one variant. Separate tests tell you which change moved the number; a bundle only tells you that something did. Tests on the same page run side by side, and each is read as its own result.

Whole-page redesigns

Shopify allows 25 sections per page template, and a same-page test has to fit both versions of the page into one template. A redesign of a page that already has a dozen sections will not fit. Build it as a split-URL test instead: the new page lives on its own template and Ordinary sends half the visitors there. See A/B testing and experiments for how split-URL tests behave.

Before you press Start

For same-page tests, opening the Start or Resume confirmation checks the live pages and shows any styling warnings before you confirm. You can also use Verify on live site while the test is a draft or paused. These checks do not start the test. A styling warning is advisory: inspect the affected templates yourself, because the check does not prove that every stylesheet or layout is correct. If the check is unavailable, the confirmation says so rather than claiming a pass.

1. The markup is on every page the test targets, with the right test ID and the group Keys (not Names) from the Test groups tab.
2. Non-control groups are hidden inline and carry data-oyx-d.
3. Each group’s styling lives in its own section file, and the group looks the same on every template that shows it, on desktop and on a phone.
4. The test in Ordinary has the right page pattern. A pattern can include a query string when the test should only run on an alternate template.
5. Start the test, then open each group with the preview links on the test’s page. Preview views never count as visitors.
6. Within the first hour, check the Results tab shows visitors in every group.

Ending a test you built in your theme

Once a test has ended, each group renders exactly as it ships in your theme. A variant that ships hidden stays hidden, so visitors see the control and the leftover markup is only tidying. Markup left visible is the risk: a non-control group without the inline hiding shows next to the control from the moment the test ends, because Ordinary only hides groups while a test is active. When you end a test, Ordinary checks the page and tells you which it found: markup that is showing, or hidden markup you can remove when convenient.

End the test in Ordinary first, then make the winning version the only one in your theme: remove the other group’s markup, and if the winner shipped hidden, drop its style="display:none" and data-oyx-d. Removing a group while the test is still running leaves the visitors assigned to it without their version. If a variant isn’t hidden inline, have that theme edit ready to make straight after ending. A test ID stored in a theme setting can stay until you remove the wrappers; clear it then.

Related

• A/B testing and experiments
• Running a test you built yourself, for registering exposures from your own code