AB Testing and Benchmarking

To ensure reliable performance comparisons, we strongly suggest letting Froomle handle the traffic splitting and user segmentation.

Benchmarking through the Froomle platform ensures a "level playing field" where user assignment is consistent, and metrics for all groups—whether powered by Froomle or your own logic—are collected and processed in the exact same way.

For frontend SDK integrations, split ownership and rendering ownership are two separate decisions.

First decide who assigns the split:

  • Froomle-managed split

  • customer-managed split

Then, when you implement a benchmarked recommendation placement, decide how that placement is rendered:

  • response-driven passthrough: render what Froomle returned and keep your own fallback/control outside the SDK contract when no items are returned

  • SDK-managed control-aware placement: keep the control content inside the SDK-managed placement so the SDK can preserve or replace it and auto-track both branches

  • application-managed external control: keep the control content outside the SDK-managed placement and own the control UI and any manual tracking yourself

Choose that rendering shape during recommendation implementation, not during the earlier page/business event rollout. For the SDK-specific ownership model and rendering shapes, see Benchmarking and A/B Testing with the SDK.

Scenario 1: Froomle-Managed Split (Recommended)

In this setup, Froomle automatically assigns users to groups based on a configured percentage. There are two common ways to run this:

Option A: Full A/B Test (Model vs. Model)

Froomle compares two different internal strategies (e.g., "Trending" vs. "Personalized").

  1. Response: You receive items for both groups.

  2. Action: Simply render the items returned. You do not need to know which model produced them.

Option B: Froomle vs. Internal Baseline

Froomle compares its recommendations (Group A) against your existing solution (Group B/Control).

  1. Response:

    • Froomle Group: API returns personalized items.

    • Control Group: API returns an empty items array.

  2. Action: Check if items were returned. If yes, render them. If no, render your fallback/baseline.

Group assignment is strictly based on the device_id. This ensures that anonymous users who later log in (or users without a user_id) remain in the same test group throughout their session on that device.

How it works (for both options):

  1. Request: Make a standard request without any group information.

  2. Tracking: Send standard impression/click events with the request_id. Froomle automatically links the event to the correct group on the backend.

// 1. Standard Request (No user_group needed)
POST /<environment>/recommendations/requests
{
  "user_id": "user-123",
  "page_type": "home",
  "lists": [
    {
      "list_name": "home_page_trending",
      "list_size": 10
    }
  ]
}

// 2. Response Example (Option B - Control Group)
// Froomle decides this user is in control, returns empty items.
{
  "request_id": "req-abc-123",
  "user_group": "control",
  "lists": [
    {
      "list_name": "home_page_trending",
      "items": []
    }
  ]
}

Scenario 2: Customer-Managed Split

Use this only if you operate your own A/B testing platform and must strictly control user assignment yourself.

How it works:

  1. Request: You determine the group and must explicitly pass the user_group parameter in the request body.

  2. Response: Froomle respects your decision.

    • If you send user_group: "froomle", it returns items.

    • If you send user_group: "customer", it returns empty items.

  3. Tracking: You must pass the same user_group back in all tracking events (impressions, clicks) to ensure attribution.

The amount and names of the user groups are to be agreed upon with your Froomle account team.

If you are unsure which benchmark setup fits your use case, contact your Froomle account team before implementing the placement. They can help define the benchmark contract, ownership model, and rollout plan.

 
// 1. Explicit Request
POST /<environment>/recommendations/requests
{
  "user_id": "user-123",
  "page_type": "home",
  "user_group": "customer", // You decided this
  "lists": [
    {
      "list_name": "home_page_trending",
      "list_size": 10
    }
  ]
}

User Group Overview

Feature Froomle-Managed Split (Recommended) Customer-Managed Split

Who assigns the group?

Froomle

Customer (You)

Content in response?

Option A (Model vs Model): Items for all groups
Option B (Vs Baseline): Items (Froomle) or Empty (Control)

Items (Froomle group)
Empty (Customer group)

Pass user_group in Request?

No

Yes (Mandatory)

Pass user_group in Events?

No

Yes (Mandatory)

Event Requirements

For the benchmark to be valid, you must send tracking events for all groups, even if Froomle returned no items and you rendered a fallback.

  • Impressions: Send an impression event for the modules you displayed.

  • Clicks: Track clicks on the items.

If you strictly follow this pattern, the Froomle dashboard will automatically show side-by-side performance metrics for both strategies.

Related Topics