Skip to content
Alokai CMS — Dev Docs

A/B Testing

A/B testing in Alokai CMS sits on top of the existing personalization system: an experience with embedded variants on a page becomes an A/B test by setting experiment_type='ab_test', traffic weights, and a conversion event. Visitors are bucketed deterministically server-side; the storefront ships the matching variant content; conversions are tracked back to the CMS via a small client- and/or server-side hook.

Conceptual model

┌────────────────────────────────────────────────────────────────────┐
│                          Page                                      │
│  componentsAboveFold: [...base...]                                 │
│  _variants: {                                                      │
│    [experienceId]: {                                               │
│      [componentId-A]: { targetType, zoneKey, data: { ... } },      │
│      [componentId-B]: { targetType, zoneKey, data: { ... } },      │
│    },                                                              │
│  }                                                                 │
└────────────────────────────────────────────────────────────────────┘
        │
        │  experience_type='ab_test'
        │  variant_weights={"comp-A":0.5,"comp-B":0.5,"control":0.0}
        │  conversion_event='newsletter_signup'
        ▼
┌────────────────────────────────────────────────────────────────────┐
│  Delivery — per request (both /pages/:id and /pages/by-path/*):    │
│                                                                    │
│    visitorId = X-Alokai-CMS-Visitor header                         │
│    bucketingKey = ab_visitor_aliases[customer] || visitorId        │
│    assignment  = assignBucket(bucketingKey, expId, weights, alloc) │
│                                                                    │
│  Response:                                                         │
│    page._ab_assignments = { [expId]: variantId | null }            │
│    impression row written to ab_events                             │
│                                                                    │
│  Storefront middleware then:                                       │
│    page = resolveAbTestVariants(page, page._ab_assignments)        │
│    response.cookie = serializeAbAssignmentsCookie(...)             │
└────────────────────────────────────────────────────────────────────┘
        │
        ▼
┌────────────────────────────────────────────────────────────────────┐
│  Storefront                                                        │
│    - Renders the assigned variant content                          │
│    - At conversion points: tracker.track(expId, eventName)         │
│    - Server-side webhook: tracker.trackAll('purchase', metadata)   │
└────────────────────────────────────────────────────────────────────┘
        │
        ▼
┌────────────────────────────────────────────────────────────────────┐
│  POST /api/v1/ab/track  →  ab_events row                           │
│  GET  /api/ab/events/summary  →  per-variant counts + rate         │
└────────────────────────────────────────────────────────────────────┘

Lifecycle

A test’s effective state is the product of the experience’s status and the page’s publish state. Delivery only serves published pages, so an active test on a draft page sits dormant — the test “kicks in” the moment the page goes live.

Test statusPage statusBehavior
draftanyNo bucketing. Operator clicks Start test to flip to active.
activedraft / scheduledConfigured but inert. Bucketing fires automatically once the page publishes.
activepublishedLive. Visitors are bucketed; impressions and conversions are recorded.
pausedanyBucketing stops; existing assignments stay in cookies but no new events are recorded.
archivedanyConcluded. Summary is frozen at concluded_at (later events are kept but excluded from totals).

start_at and end_at add an additional time gate on active experiences. Setting start_at to the page’s scheduled_at is the recommended way to coordinate test activation with a scheduled publish — the dashboard’s “Start when the page goes live” option does this automatically.

Schema

A/B test fields extend the existing experiences table:

ColumnTypePurpose
experiment_type'personalization' | 'ab_test'Defaults to 'personalization'. Set to 'ab_test' to enable bucketing.
traffic_allocationREAL (0–1)Portion of audience-eligible visitors who enter the test. Rest are pure control.
variant_weightsTEXT (JSON)Map of variantId → weight (0..1). Weights must sum to 1.0.
conversion_eventTEXTEvent name your tracker calls (e.g. purchase, newsletter_signup).
concluded_atTEXT (ISO)Set on conclude; freezes the summary at this point.
winner_variant_idTEXTPicked by /conclude, used by /promote-winner.

Two new tables back the runtime:

  • ab_events — (id, space_id, environment_id, experiment_id, variant_id, event_type, visitor_id, occurred_at, metadata). One row per impression and per conversion.
  • ab_visitor_aliases — (session_id, customer_id, linked_at). Stitches an anonymous visitor’s bucket to their customer ID at login so the bucket survives across devices.

Variant IDs in variant_weights should match the keys you use in _variants[experienceId] on the page (typically component IDs). Add a "control" pseudo-variant with weight 0–N to bucket some visitors to the unmodified base content for tracking.

Bucketing

assignBucket() in src/server/utils/ab-bucketing.ts is a pure function:

assignBucket({
  visitorId: "session-abc",
  experimentId: "exp-1",
  weights: { "comp-A": 0.5, "comp-B": 0.5 },
  trafficAllocation: 1.0,
})
// → { variantId: "comp-A" }   (deterministic for this visitorId/expId)

Two independent FNV-1a hashes:

  1. Inclusion — incl:{expId}:{visitorId} ↔ [0,1). If ≥ trafficAllocation, returns { variantId: null, reason: 'outside_allocation' }.
  2. Variant pick — {expId}:{visitorId} ↔ [0,1), walked along the cumulative weights.

The two hashes are independent, so adjusting trafficAllocation (e.g., 50% → 100%) doesn’t reshuffle which variant in-test visitors land on — it only changes who’s in vs. out.

Visitor identity

The CMS bucketing key comes from the request header X-Alokai-CMS-Visitor, set by the storefront middleware. Recommended source order:

  1. Customer ID for logged-in users (gives consistent buckets across devices).
  2. Anonymous session cookie for logged-out users.

When a visitor authenticates mid-test, the storefront posts to /api/v1/ab/visitor-alias with (session_id, customer_id). Subsequent customer-keyed deliveries resolve to the session’s bucket via ab_visitor_aliases, so the variant survives the login boundary.

Storefront integration

The @alokai/cms-api package ships both pieces — server middleware helpers and a browser tracker — from the same install. There’s no separate dependency.

  1. Forward the visitor identity to the CMS.

    In your storefront-middleware extension (e.g. apps/storefront-middleware/sf-modules/cms-alokon/extensions/unified.ts), set the X-Alokai-CMS-Visitor header on every Delivery API call:

    // Resolve a stable visitor id, customer-first then session.
    const customerId = context.commerce?.session?.customerId ?? null;
    const sessionId =
      context.req.cookies.get("alokai-session")?.value ??
      crypto.randomUUID();
    if (!context.req.cookies.get("alokai-session")) {
      context.res.cookies.set("alokai-session", sessionId, {
        path: "/", httpOnly: true, sameSite: "lax", maxAge: 60 * 60 * 24 * 365,
      });
    }
    const visitorId = customerId ?? sessionId;
    

    // Forward on every CMS request from this middleware.
    context.req.headers.set("X-Alokai-CMS-Visitor", visitorId);

    Without this, no bucketing fires and _ab_assignments stays empty.

  2. Apply variant content + persist assignments to a cookie.

    The CMS attaches _ab_assignments: { [expId]: variantId | null } to the page payload, but it doesn’t swap variant content into the page zones — that’s the storefront’s job (mirrors how resolvePersonalizedPage works for personalization).

    @alokai/cms-api ships two helpers for this:

    import {
      resolveAbTestVariants,
      serializeAbAssignmentsCookie,
    } from "@alokai/cms-api";
    

    const page = await api.getPage({ locale, path, preview });
    

    // 1. Apply the visitor's variant data to the page zones
    const resolved = resolveAbTestVariants(page, page._ab_assignments);
    

    // 2. Persist assignments as a cookie so subsequent server routes
    //    (form posts, API endpoints, commerce webhooks) can read them via
    //    parseAbAssignmentsCookie without round-tripping through the page.
    if (page._ab_assignments) {
      response.headers.set(
        "Set-Cookie",
        serializeAbAssignmentsCookie(page._ab_assignments),
      );
    }
    

    return resolved;

    resolveAbTestVariants honors the same targetType semantics as promote-winner:

    • 'zone' → replaces the entire zone array
    • 'component' → replaces a component-by-id within a zone
    • 'props' → merges the variant’s data into the component’s props
    • 'hidden' → removes the component from the zone

    Personalization-type experiences are ignored (they’re handled by resolvePersonalizedPage, which uses audience rules, not buckets).

  3. Stitch session → customer at login.

    When an anonymous visitor authenticates, post to /api/v1/ab/visitor-alias once so the bucket survives:

    // In your auth callback / login handler:
    await fetch(`${cmsBaseUrl}/api/v1/ab/visitor-alias`, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        "Authorization": `Bearer ${deliveryToken}`,
        "X-Alokai-CMS-Space": spaceId,
        "X-Alokai-CMS-Environment": environmentId,
      },
      body: JSON.stringify({ session_id: sessionId, customer_id: customerId }),
    });

  4. Instantiate the tracker on the storefront.

    Once per page (RSC or layout), create a tracker bound to the current visitor and the page’s assignments:

    import { createTracker } from "@alokai/cms-api";
    

    const tracker = createTracker({
      baseUrl: process.env.NEXT_PUBLIC_CMS_URL!,
      deliveryToken: process.env.NEXT_PUBLIC_CMS_DELIVERY_TOKEN!,
      spaceId: "default",
      environmentId: "main",
      visitorId,                            // same id as the X-Alokai-CMS-Visitor header
      assignments: page._ab_assignments,    // from the page payload
    });

    Pass tracker (or just assignments + visitorId) down via context if multiple components need it.

Adding a tracked user action

Once steps 1–4 above are wired, adding a new tracked action is a two-step ritual: pick a name and call track().

  1. Decide on the event name.

    Conversion event names are arbitrary strings. Pick something descriptive — they show up in the dashboard summary and in ab_events.event_type.

    • Use snake_case: newsletter_signup, add_to_cart, cta_clicked, pdp_viewed.
    • One name per business outcome — don’t reuse click for everything.
    • Names are scoped per-experiment, so newsletter_signup on one test is independent of newsletter_signup on another.

  2. Set the event name on the test.

    In the CMS dashboard:

    • Open A/B Tests → New A/B Test (or an existing test’s detail page).
    • Set Conversion event to the exact string you picked.

    Or via API:

    Terminal window
    curl -X PUT "$CMS_URL/api/experiences/$EXPERIMENT_ID" \
      -H "X-Alokai-CMS-Space: $SPACE" \
      -H "X-Alokai-CMS-Environment: $ENV" \
      -H "Content-Type: application/json" \
      --cookie "alokai-cms-session=…" \
      -d '{"conversion_event":"newsletter_signup"}'

  3. Call tracker.track() at the conversion point in your storefront.

    The first argument is the experiment ID (visible at the top of the test detail page), the second is the event name. The tracker no-ops if the visitor isn’t bucketed into this experiment.

    import { useTracker } from "@/lib/cms-tracker"; // your wrapper around createTracker
    

    function NewsletterForm() {
      const tracker = useTracker();
      async function onSubmit(values: FormValues) {
        await api.subscribe(values);
        await tracker.track("exp_yhB3kQ7m", "newsletter_signup");
      }
      // ...
    }

    Per-event metadata is supported and stored in ab_events.metadata:

    await tracker.track("exp_yhB3kQ7m", "add_to_cart", {
      productId: product.id,
      price: product.price,
    });

  4. (Optional) Use trackAll for events that aren’t tied to one specific test.

    tracker.trackAll(eventName, metadata?) fires the same event for every experiment the visitor is currently in. Useful for cross-cutting events like pdp_viewed or purchase where any running test on the page might count.

    await tracker.trackAll("purchase", {
      order_id: order.id,
      order_value: order.total,
    });

Server-side conversions (commerce webhook)

For conversions detectable server-side — orders placed, subscriptions created — skip the client tracker and call the same module from middleware. The visitor’s assignments come from the cookie or the order metadata.

// In sf-modules/cms-alokon/extensions/unified.ts, on order-completed webhook:
import { createTracker, parseAbAssignmentsCookie } from "@alokai/cms-api";


export async function onOrderCompleted(req: Request, order: Order) {
  const tracker = createTracker({
    baseUrl: process.env.CMS_URL!,
    deliveryToken: process.env.CMS_DELIVERY_TOKEN!,
    spaceId: "default",
    environmentId: "main",
    visitorId: order.customer_id,
    assignments: parseAbAssignmentsCookie(req.headers.get("cookie") ?? ""),
    fetchImpl: fetch,
  });
  await tracker.trackAll("purchase", {
    order_id: order.id,
    order_value: order.total,
    item_count: order.line_items.length,
  });
}

This requires the cookie to still be present on the request that triggers the webhook — most commerce stacks pass the original request through, but verify in your setup. Falling back to looking up assignments via ab_visitor_aliases from the customer ID is also possible.

Validating the wiring

Two checks, both available from the test detail page in the dashboard:

  • “Fire impression” — runs a synthetic visitor through the real bucketer and writes one impression. Use this to confirm CMS-side configuration.
  • “Fire impression + conversion” — same plus a conversion row using the test’s configured event. Confirms the conversion path independent of the storefront.

If both buttons produce visible counts in the chart but real traffic doesn’t, the missing piece is on the storefront side — typically the X-Alokai-CMS-Visitor header isn’t being forwarded.

Concluding a test

The dashboard’s Conclude button picks the variant with the highest conversion rate, requiring at least 100 impressions per variant by default. The detail page also surfaces a chi-squared two-tailed p-value for the leader vs. runner-up; concluding while p ≥ 0.05 is allowed but flagged.

After concluding, Promote winner copies the winning variant’s data into the page’s base zone (matching targetType — zone replaces the array, component replaces by id, props merges) and archives the test. The next page render serves the winner to all visitors with no bucketing overhead.

Late-arriving events (CDN-cached page views still trickling in after concluded_at) are still accepted by /api/v1/ab/track and stored, but excluded from the summary so the result stays stable.

API reference summary

Public Delivery API (delivery-token auth):

EndpointPurpose
POST /api/v1/ab/trackRecord an event row. Used by the client tracker and the commerce webhook.
POST /api/v1/ab/visitor-aliasStitch session → customer for cross-device bucket continuity.

Admin API (session/api-key auth):

EndpointPurpose
GET /api/ab/testsList all A/B tests in the current space.
GET /api/ab/events/summary?experiment_id=…Per-variant impression and conversion counts. Frozen at concluded_at.
POST /api/ab/test-event/:idFire a synthetic impression (and optional conversion) — used by the dashboard’s “Validate” buttons.
POST /api/experiences/:id/concludePick winner, set concluded_at. Body {winner_variant_id?} overrides auto-pick.
POST /api/experiences/:id/promote-winnerCopy winner data into the page’s base zones, republish, archive.
PUT /api/experiences/:id/statusManually flip between draft / active / paused / archived.