Why open source A/B testing for ecommerce starts with where revenue data lives
For an online store testing checkout steps, price displays or recommendations, the deciding question is where the revenue numbers live. If orders sit in a self-hosted commerce database or warehouse, the experiment tool should analyze results where that data already is, and GrowthBook is the open source option built for that pattern. Its self-hosted Open Source plan is free with unlimited users, and its warehouse-native analysis runs SELECT queries against the store's own data source (BigQuery, Snowflake, Postgres, MySQL and others) and returns only aggregated statistics. Shopper-level order data never has to leave your infrastructure for the analysis step.
Unleash and Flagsmith fit a different job. Teams that mainly need to switch a new checkout step on for a percentage of sessions, roll back a broken recommendation widget quickly, or keep flag evaluation inside their network can run them well without a revenue analysis layer. The trade-off is that conversion, average order value and refund metrics must be computed somewhere else, usually by a data team building its own queries. Before choosing, confirm with the store team which metrics the business will sign off on, which system holds them, and whether the commerce platform writes order events in a form a warehouse can read.
Two checks settle most of the decision early. First, confirm that the storefront platform (Magento Open Source, WooCommerce, Saleor, Medusa or Sylius) can evaluate flags on the server before the page renders, since that choice drives flicker and experiment integrity. Second, confirm that GrowthBook's self-hosted Open Source edition covers the number of projects and governance you expect, because approval workflows, sticky bucketing and holdouts sit in the self-hosted Enterprise tier. Ask your partner to document both answers in writing before any experiment is configured.
Which requirements shape an ecommerce experimentation build
This table compares the requirements that most often change how an ecommerce experimentation build is designed, with the source of each requirement and what it means for the build. Read it before choosing between server-side and client-side experiments, and before deciding where shopper data and revenue metrics live.
| Requirement | Where it comes from | What it means for the build |
|---|---|---|
| Shopper attributes stay in the store's infrastructure | The store's own data rules and any customer contracts | Targeting attributes such as segment, region or loyalty tier should be evaluated on servers the store controls. Confirm which attributes the SDK sends to the flag service and whether any leave the store's network. |
| Revenue metrics tied to order records | The store's finance and analytics definitions of an order, refund and net revenue | Define conversion and revenue metrics from the order database or warehouse, not from front-end click events. GrowthBook queries the warehouse with SELECT statements and receives only aggregated statistics, so the metric SQL must match finance's definitions. |
| No flicker on price and checkout pages | The storefront's rendering model and cache layer | Client-side variants that swap prices or steps after first paint are visible to shoppers and can distort results. Assign price and checkout variants server-side or at the edge, and test the cached page path before launch. |
| Content Security Policy compatibility for self-managed Magento Open Source | Adobe's developer documentation on CSP support from version 2.3.5, with restrict mode by default on payment pages from 2.4.7 | Experiment SDK scripts, inline snippets and third-party hosts must be allowed in the policy. Restrict mode on payment pages can block a client-side experiment script, so test the policy in staging on the exact 2.4.x release in use. |
| Varnish or other full-page caching | Adobe's configuration guide, which documents Varnish for self-managed installations | A cached page can serve the same variant to every visitor. Experiment assignment must vary the cache key or move to server-side or edge assignment, and the team must confirm the cache rules with the platform partner. |
| PCI scope boundaries for checkout changes | The store's payment provider contract and its PCI DSS responsibilities, which the store should confirm with its assessor | Experiments that touch card entry, payment fields or hosted payment pages may change the systems in scope. Keep checkout test logic outside cardholder data flows where possible and confirm the boundary with the QSA before any change. |
| Licence obligations of the chosen platform | The licence of the self-hosted experimentation edition and of the storefront platform | GrowthBook code outside its enterprise directories is MIT licensed, and those enterprise directories are under a separate enterprise license. Features such as SSO, approval workflows and holdouts require the Enterprise edition, so confirm which features the store needs. |
| Sample size and test duration | The store's traffic, conversion rate and minimum detectable effect chosen by the analytics lead | Sample size depends on the store's own traffic and baseline conversion, not on a platform setting. Ask the analytics lead to set the target effect and power before launch, and confirm that the platform's statistics engine is configured to match. |
Server-side or client-side: which experiments belong where
Server-side experiments decide the variant before the storefront renders the response. This is the right place for anything that changes a price, a shipping option, a checkout step or the order of recommended products, because the shopper never sees the control version swap into the variant. The experiment service returns the assignment to the storefront or commerce backend, which then renders the matching logic.
- Price and checkout logic: Pricing rules, discount presentation and checkout step order are computed in the backend, so the shopper sees one consistent result from the first paint.
- Recommendation ranking: Ranking models or rules can be evaluated per request and the assignment logged with the session identifier.
- Flicker avoidance: No content is replaced after load, which removes the visible jump that client-side tests can produce.
- Development requirement: Engineers must add assignment calls to application code, keep the flag key stable across releases, and handle a fallback when the experiment service is unreachable.
- Cache interaction: Variants that differ per user must not be served from a shared page cache, so the cache key or the rendering path has to account for the assignment.
Before a server-side test ships, confirm with your team which cache layer sits in front of the pages involved and how it keys responses.
Client-side experiments change the page in the browser after the HTML has arrived. They suit copy, banner placement, layout tweaks and visual treatments where a short change in the first render is acceptable and the team wants to launch without a backend release. Most marketing and merchandising teams can run these with less engineering help.
- Copy and banners: Headlines, promotional banners and button labels are good candidates because a brief visual swap is unlikely to affect purchase decisions.
- Flicker risk: The control version can appear for a moment before the variant loads. Hiding the affected elements until the decision arrives reduces this, but can delay rendering.
- Caching effect: Shared HTML served from a full-page cache stays the same for every shopper, so the browser must apply the variant after load. Confirm that the assignment is not cached with the page.
- Measurement limits: Client-side events depend on scripts that ad blockers or consent settings can block, so compare event counts with order records.
- Pricing caution: Avoid changing prices or checkout steps only in the browser, since the displayed value and the charged value can then differ.
Before a client-side test ships, check the flicker behavior on slow connections and confirm how the consent banner interacts with the experiment script.
On self-managed Magento Open Source, Adobe documents Varnish as the full-page cache accelerator. Cached pages are reused across shoppers, which makes response time fast but changes how experiments must be delivered. A test that alters the rendered HTML for some shoppers needs a cache key or a rendering path that separates those shoppers, or the cache will serve one variant to everyone.
- Varnish role: The cache sits in front of the application and stores full responses, so every experiment that changes HTML must be checked against it.
- Cache keys: Variant identifiers can be carried in a cookie or header that Varnish is configured to vary on, which should be tested before launch.
- Client-side workaround: Serving a neutral cached page and applying the variant from an API call keeps the cache effective, at the cost of the flicker discussed in the client-side tab.
- Hosting difference: Adobe documents Fastly for cloud-hosted Adobe Commerce, so cache rules differ between self-managed and cloud deployments.
- Version dependence: Confirm the Varnish version and configuration against the support matrix for your Magento release before editing cache rules.
Before running any experiment that changes HTML, ask your implementation partner for a written test of cache behavior with two variants.
The choice of placement also decides who does the work. Server-side tests need developers in the loop for every experiment, while client-side tests can often be set up by merchandising or marketing staff within agreed guardrails. Governance rules should match that split, so that price and checkout changes receive the same review as other production changes.
- Developers: Own assignment calls, fallbacks, and cleanup of finished experiments in the backend.
- Merchandisers and marketers: Own copy and banner variants that stay within approved templates.
- Security and compliance staff: Review any experiment that touches payment steps, personal data or consent handling.
- Operations: Monitor cache hit rates and the experiment service during tests, and know how to switch a variant off.
- Auditors: Expect a record of who changed each experiment, when, and what the shopper saw.
Before the first test, agree in writing which experiment types need developer review and which can be launched by business teams.
Which open source platforms fit an ecommerce experimentation program
The experimentation tool and the commerce platform are separate decisions, but they meet at the checkout, the product page and the order database. Start with where revenue metrics live and who must approve a change, then pick the flag and test platform that can read that data and run inside your infrastructure.
GrowthBook, self-hosted for warehouse analysis
GrowthBook fits stores that want experiment results computed from their own order data in a warehouse, with only aggregated statistics leaving that environment. The self-hosted Open Source edition is free with unlimited users and one project, and its code outside the enterprise directories is MIT licensed. Check whether you need sticky bucketing, holdouts, approval workflows or SSO and SCIM, because those sit in the separately licensed Enterprise features and are a distinct licence decision. Confirm your warehouse is on the supported list before committing.
Unleash for flag delivery
Unleash is a fit when the main need is controlling which price display, checkout step or recommendation block a shopper sees, with a lighter analysis layer. Teams often pair it with their own analytics or the commerce platform's reporting to judge results. Confirm which experiment and statistics features your chosen edition includes, how SDKs evaluate flags in your storefront, and whether results can be joined with order data without copying customer attributes outside your network.
Flagsmith for flag delivery
Flagsmith suits programs where feature flags and remote configuration for storefront components matter more than statistical analysis inside the tool. It can supply variants to front-end and back-end code paths during a test while revenue reporting stays in the warehouse or the order system. Before choosing it, confirm which SDKs cover your storefront language and framework, how segments are built from shopper attributes, and whether the self-hosted deployment meets your data location rules.
Match the flag tool to the commerce platform
The right split depends on the commerce platform. Magento Open Source, WooCommerce, Saleor, Medusa and Sylius each expose pricing, cart and checkout differently, so server-side variant assignment and caching behave differently on each. Saleor extends through webhooks and apps rather than in-process plugins, which shapes where a variant decision is made. Confirm, for your platform, where the assignment runs, whether full-page caching can serve a variant to the wrong shopper, and how order events reach the analysis layer.
How sample size and revenue metrics get set before a test starts
This sequence turns a pricing, checkout or recommendation idea into a written experiment brief with one revenue metric, an agreed minimum detectable effect, a sample size tied to real traffic, and guardrails everyone signs off before any traffic is split.
Pull baseline order and session data
The analytics or data engineering lead queries the order database or warehouse for the last several weeks of sessions, completed orders, order value and refunds for the pages or flows the test touches. The analyst checks that event definitions match the order records, such as whether a session counts only once per visitor and whether cancelled or fraud-held orders are excluded. The step is done when the baseline numbers are reproduced by two people from the same query and written into the brief.
Define the primary revenue metric
The analyst and the product owner choose one primary metric from the order records, typically revenue per session or conversion to a completed order, and write the exact SQL logic: which order statuses count, whether discounts and taxes are netted out, and the attribution window from exposure to purchase. Revenue per session captures basket size and conversion together, while conversion alone can rise when average order value falls. The step closes when the metric definition is stored in the experimentation tool's metric catalog or referenced by a versioned query.
Set the minimum detectable effect with merchandising
The merchandising or pricing team states the smallest change in the primary metric that would justify a rollout, and the analyst checks whether that change is plausible against the baseline variance. A small expected lift needs far more traffic than a large one, so the business owner has to decide what matters before the sample is calculated, not after. The output is one agreed relative effect written into the brief, with the name of the approver and the date of the decision.
Compute the required sample from warehouse traffic
The analyst uses the baseline variance, the agreed minimum detectable effect, the chosen significance level and power, and the number of eligible sessions per day from the warehouse to estimate how many visitors each arm needs and how long the test must run to collect them. Where the experiment runs on a traffic split, the analyst also confirms that the eligible population is what the split will actually reach, for example logged-in shoppers on a specific checkout step. The step is done when the calculation, its inputs and the resulting run length are recorded in the brief for an independent reviewer to recheck.
Agree guardrail metrics and thresholds
The finance lead, customer service lead and product owner name guardrail metrics that must not get worse, usually refunds and returns, cart abandonment, page load time, payment failures and order cancellations. Each guardrail gets a direction, a threshold and the rule for what happens if it breaches, such as pausing the variant. These guardrails are computed from the same order data as the primary metric so that a revenue gain cannot hide a rise in refunds. The step closes when each guardrail has an owner and a written threshold in the brief.
Document the experiment brief and sign-off
The analyst assembles the experiment brief with the hypothesis, variants, targeting rules, exposure point, primary metric, guardrails, minimum detectable effect, sample size, planned run length and decision rules for stopping early or extending the test. The brief is reviewed by the merchandising owner, a second analyst and, where checkout or pricing is involved, the finance or compliance contact. The step is complete when all named approvers have signed the brief and the experiment is configured to match it.
Integration points between the store, flags and the warehouse
An experiment touches four places: the storefront code that asks for a variant, the flag service that answers, the warehouse that holds orders and events, and the cache layer between shoppers and the store. Each has its own failure modes, and most ecommerce experiment problems start at one of these seams rather than inside the statistics.
Storefront SDK or server hook
The commerce platform needs a place where the variant is decided before the page renders. On Magento Open Source, that is usually a server-side module or a block that reads the assignment. On Saleor, the storefront calls the GraphQL API, so the assignment should be made in the storefront server layer, not in the browser. Confirm which layer owns the decision, whether the SDK can evaluate flags locally without a network call per page view, and what the storefront does if the flag service is unreachable. A fallback value that matches the control is the usual safe default.
Flag and experiment service
GrowthBook bundles its Next.js front end, Express API and Python statistics engine in one Docker image, and stores metadata in MongoDB or a compatible database. The docs suggest a managed database service such as MongoDB Atlas or AWS DocumentDB for production. Plan the backup, restore and upgrade path for that database, since it holds the experiment definitions and history your team will rely on during an audit. Confirm whether your environment should run the Helm chart on Kubernetes, and where file uploads will be stored (local disk, Amazon S3 or Google Cloud Storage).
Warehouse and order data pipeline
Revenue metrics come from order data, not from the click stream alone. GrowthBook runs SELECT queries against your warehouse and receives only aggregated statistics, so the data pipeline must land order lines, refunds, cancellations and exposure events in a table the analysis can join on user and experiment identifiers. Confirm which warehouse holds the orders (BigQuery, Snowflake, Redshift, Postgres or another supported source), who owns the ETL job, how late-arriving refunds are handled, and whether the read account is restricted to SELECT on the required schemas.
CDN and cache layer
A full-page cache can serve one shopper the variant assigned to another. Self-managed Magento installations commonly use Varnish, while cloud-hosted Adobe Commerce uses Fastly. Any cache in front of the store must either vary on the experiment cookie or bypass caching for pages in an active test. Confirm the cache keys, the cookie names your SDK sets, the time-to-live for each page type, and whether purges during a rollout clear the right variants. Test with two browsers before launch, since a cached response can look correct to the developer and wrong to real shoppers.
Content security policy and script loading
Client-side experiments load a script that changes the page, and a strict content security policy can block it or the SDK's network calls. Magento Open Source supports Content Security Policy headers, and from 2.4.7 the policy runs in restrict mode by default on storefront and admin payment pages. Confirm which script sources and connect targets the flag service needs, whether the SDK's domain is listed, and whether the payment pages are excluded from any experiment that touches checkout. Review this with the security team before the first test goes live.
How the rollout moves from one test to a governed program
The sequence starts with one low-risk pilot and ends with a governed program, where each phase leaves behind a artifact the next phase depends on, so the store can scale experimentation without losing control of revenue data or checkout stability.
Map the store stack and data sources
DeployOpen and the store team inventory the storefront, the commerce platform, the order database, the product feed, the analytics tags and any warehouse or reporting layer that already holds orders. The team records which systems set prices, which own the cart and checkout, where the order of record lives and who can change each of them. The step closes with a written stack map and a list of candidate data sources for revenue metrics. Confirm with the commerce vendor which checkout and pricing hooks your edition exposes before anything is scoped.
Pilot one low-risk display test
The first experiment is a single display change, such as a product badge or a price format on category pages, evaluated server-side so the variant is decided before the page renders. DeployOpen configures the self-hosted GrowthBook instance, the SDK integration and a flag with a documented fallback value, and the store team agrees the primary metric in advance. The step closes when the pilot runs end to end, the assignment logs match expected traffic splits and the page shows no flicker in the browser checks.
Connect the warehouse and validate metrics
Analysts connect GrowthBook to the store's warehouse or order database using a read-only account, then define conversion, average order value and margin metrics in SQL and reconcile them against finance reports for the pilot window. Discrepancies are traced to the source tables, not patched in the analysis. The step closes with a validated metric set, a short metric dictionary with owners and definitions, and a record of the reconciliation so the numbers can be defended later.
Write a runbook for pausing during incidents
Engineering and the on-call team write a runbook that states how to pause or revert any experiment in seconds, who holds that authority during a checkout incident and how the team confirms the fallback is serving. The runbook covers the flag kill switch, the fallback values and a communication path to the merchandising and customer service leads. The step closes when the team has rehearsed the pause on a staging store and recorded the time it took, and the runbook is linked from the on-call guide.
Set approvals and naming standards
The steering group defines who can create, launch and stop experiments, which changes need a second reviewer, and a naming convention for experiments, flags and metrics so reporting stays readable across teams. Approval rules are configured where the edition supports them, and anything the free self-hosted edition lacks is documented as a manual control with a named owner. The step closes with a governance charter that the change board signs off and a sample of flag and experiment names checked against the convention.
Set the holdout and sampling policy
Analysts and the commercial owner decide how much traffic stays out of experiments for a long-term baseline, which segments are excluded from tests during peak trading, and how sample sizes are estimated before each launch. The policy names the minimum effect the business cares about and the approach used to size tests, and it is written before the next wave of experiments starts. Holdout features may depend on the edition, so confirm what your self-hosted tier supports. The step closes with an approved policy document and a sizing worksheet attached to each experiment template.
Retire finished experiments and keep records
When an experiment reaches a decision, the owner records the outcome, removes the losing variant from code, sets the winning value as the default and archives the flag and its results. Stale flags are reviewed on a regular cadence the governance group sets, so the flag inventory does not grow into an unmanaged set of code paths. The step closes when the experiment register shows each test with an owner, a decision and a retirement date, and the stale-flag report is empty or has named owners for every remaining item.
Evidence a store's auditor or finance team asks for
This table maps the evidence an auditor or finance team typically requests for ecommerce experiments to the component that produces each record. Read it as a checklist to confirm with your platform team before a test touches pricing or checkout. DeployOpen prepares this evidence, but external auditors and certification bodies are the ones who assess it.
| Evidence item | Component that produces it | What to confirm before go-live |
|---|---|---|
| Experiment register | GrowthBook experiment records with hypothesis, owner, dates and variants | Every test has a named owner, a business question and a documented start and stop date |
| Pre-registered metrics | GrowthBook metric definitions stored with the experiment, plus the warehouse SQL behind revenue metrics | Primary revenue and conversion metrics are fixed before traffic is split, with changes logged |
| Audit log of flag and experiment changes | GrowthBook audit log for flag and experiment edits, if your edition provides it | Confirm which edition includes audit logs and whether entries capture the old value, the new value and the user |
| Access records for the order database and warehouse | Warehouse query history and database access logs, scoped to the read-only account GrowthBook uses | GrowthBook runs only SELECT queries, so confirm that its service account has no write privileges and that query history is retained |
| Change approvals for checkout or pricing tests | Approval records in your ticketing or change system, linked to the GrowthBook experiment ID | Pricing and checkout changes have a second approver who is not the experiment owner |
| Permissions and segregation of duties | GrowthBook role and permission settings, with SSO and SCIM where the enterprise features apply | Confirm which roles can launch, edit or ramp a test on live traffic, and review them on a set schedule |
| Commerce platform change records | Store admin and commerce platform change logs for price, promotion and checkout configuration | Price changes made by an experiment and changes made directly in the store are both traceable to a ticket |
| Experiment results and decisions | GrowthBook result exports and a written decision record stored with the experiment | Each concluded test records what shipped, what was reverted and who signed off |
What do ecommerce teams ask before running self-hosted experiments?
What does the self-hosted Open Source edition of GrowthBook include?
The self-hosted Open Source plan is free and includes unlimited users, one project and unlimited API requests, according to GrowthBook's pricing page. The code outside its enterprise directories is MIT Expat licensed, while those enterprise directories fall under the GrowthBook Enterprise License. For most stores, the first question is whether one project is enough for the storefront, checkout and recommendations tests running side by side. Confirm current edition limits on the GrowthBook pricing page before you plan the rollout.
What does the Enterprise edition add for an online store?
According to GrowthBook's pricing page, self-hosted Enterprise adds SSO and SCIM, advanced permissioning, approval workflows, multi-armed bandits, safe rollouts, ramp schedules, sticky bucketing and holdouts. For a store, approval workflows and permissioning matter when pricing displays change under merchandising control, and holdouts matter when you want to measure long-run revenue effects. Ask GrowthBook for the current Enterprise price, since the pricing page lists it as custom, and confirm which features your edition includes before you commit to a design that relies on them.
How does the cost split between subscriptions, infrastructure, implementation and staff time?
Software licensing can be zero on the Open Source edition, so the real spend sits elsewhere. Infrastructure is recurring: hosting the application, a MongoDB-compatible database (the self-hosting documentation suggests a managed service for production), and storage. Implementation is usually one-time: installing, connecting the order database or warehouse, defining revenue metrics and wiring SDKs into the storefront. Staff time recurs for experiment design, analysis review and flag hygiene. If you need Enterprise features, add the subscription as a recurring line. Ask any partner for a quote that separates these lines so you can see what each one buys.
What makes the effort of an ecommerce experiment program go up?
Three drivers matter most: warehouse readiness, the number of metrics and checkout complexity. If order data is clean and already in BigQuery, Snowflake or Postgres, GrowthBook can run its analysis against that source with SELECT queries and receive only aggregated statistics, according to its data source documentation. If revenue lives in several systems that do not agree on order status, refunds or currency, that data engineering comes first. Each added metric needs a definition, validation and a owner. Multi-step checkout tests also need careful flicker control and event tracking across steps, which adds engineering time.
Can the store team run self-hosted experiments on its own?
A merchandising or growth team can usually run simple tests on copy, banners and recommendation ordering once the platform and data connection are in place. Server-side tests on pricing displays and checkout steps need engineers who can maintain the code paths and understand how the flag evaluation works. Whether your team can do this alone depends on the in-house skills you have for the storefront stack, whether that is Magento Open Source, WooCommerce, Saleor or another platform. Scope the first project with an implementation partner if the team has no one who can own the SDK integration and data pipeline after launch.
How long does it take to set up the first test?
The honest answer depends on the data connection and the storefront. A first client-side test on a simple banner can be set up quickly once the SDK is installed and one revenue metric is agreed. A first test that reads revenue from the order database requires access approval, a read-only analytics account and metric SQL that matches finance definitions, which usually takes longer. Confirm the timeline with your partner in a written scope, and ask which steps are in the fixed scope and which depend on your data team's availability.
What does the work need from merchandising, analytics and engineering teams?
Merchandising owns hypotheses, the variants being tested and the business rules for pricing displays and recommendations. Analytics owns metric definitions, sample size decisions, the revenue source of truth and the review of results. Engineering owns the SDK integration, server-side flag evaluation in checkout, performance and flicker checks, and the deployment of the platform itself. Each group should name one accountable person before the first experiment. Ask your implementation partner to document this split in the written scope so each handover is tested.
Questions to ask an implementation partner before committing
Start with the integration. Ask how the partner will evaluate flags on the server before the page renders, and which parts of the commerce stack will make those calls. On Magento Open Source, ask whether they have tested assignment with the full-page cache in front of the store, since self-managed installations use Varnish as the full-page cache accelerator and a cached page can serve the same variant to every visitor. Ask them to show the bucketing key they use, how a logged-in customer keeps the same variant across sessions, and how the SDK behaves when the experimentation service is unreachable. Ask which platform version they will test against, and confirm that the version is still in standard support on the dates your rollout will run, using Adobe's lifecycle policy as the reference.
Next, ask how revenue numbers will be checked before anyone trusts a result. A credible partner will reconcile the experiment's conversion and order value metrics against the order table or warehouse extract on a sample of days, and will document any gap, such as refunds, cancelled orders, or taxes counted differently. Ask who owns the SQL that defines each metric, whether it is version controlled, and how a test is excluded when a checkout bug affects one variant only. Ask them to name the approver for every pricing display or checkout step change, the person who can stop a running test, and how that approval is recorded in the change log your finance and audit teams will read.
Finally, ask how the work is bounded and what happens after go-live. Request a written statement of scope that lists the platform components, the number of experiments included, the integration points, and the handover items such as runbooks, flag naming rules and access roles. Confirm what is excluded, including changes to the storefront theme or the data pipeline. Then ask whether post-launch support is offered as a retainer, what response expectations it sets, who applies platform upgrades, and whether the partner will run the Open Source self-hosted instance in your environment or hand it over to your team. Ask for a quote built from this written scope rather than a general estimate.
Where build, partner or SaaS fits for a store's experimentation
A store can run experiments with its own engineers, with an implementation partner on a fixed scope, or with a hosted experimentation service. Each option moves a different mix of cost lines, and the right choice depends on how many experiments run, who owns the order data and how much internal capacity exists.
Building in-house
Your engineers build assignment, exposure logging and the statistics pipeline, and run the GrowthBook deployment yourselves. The main cost is internal staff time for setup, maintenance and upgrades, plus the cloud resources for the MongoDB-compatible metadata store and the analysis service. The free self-hosted Open Source plan has no approval workflows or holdouts, so check early whether those controls are needed. Build fits teams with steady experiment volume and a platform group that already runs production services.
Partner on a fixed scope
An implementation partner deploys the self-hosted platform, connects it to your order database or warehouse, and documents the metric definitions for a defined set of experiments, then hands over the runbook. Costs are one-time for the agreed scope, with recurring costs for infrastructure and any support retainer. Ask for the exact list of data sources, the metrics in scope and who approves changes after go-live before you sign a fixed-scope statement of work.
Hosted experimentation SaaS
A hosted service removes most infrastructure work and bills recurring subscription fees, typically per seat or per traffic tier. You still pay internally to connect the order database, define revenue metrics and validate sample sizes. Check where assignment data and any user attributes used for targeting are processed, since hosted evaluation may send them outside your infrastructure. SaaS is often the better choice for a small team with no data engineers and no hard data location rule.
Warehouse data engineering work
Revenue experiments depend on order, refund and cart data that already sits in your warehouse or order database. Someone must define order-level revenue, handle returns and cancellations, and agree on the attribution window before the first result. This work is usually the largest internal cost whichever option you pick, and it recurs when checkout or pricing logic changes. Ask any partner or vendor who owns those metric queries after handover.
Upgrades and operating support
Self-hosted experimentation needs monitoring, backups, version upgrades and SDK updates on the storefront, which recur for as long as the platform runs. Build keeps this work internal, partner engagements can include it under a retainer, and SaaS moves it to the vendor. Confirm the upgrade cadence, the support hours and who is on call during a seasonal sale before you compare total cost across the three options.
How DeployOpen scopes and supports ecommerce experimentation
The sequence turns an experimentation goal for your store into a fixed-scope, self-hosted GrowthBook setup that runs in your infrastructure, connects to your order data and is documented for handover. Each step closes with a check your team can verify.
Agree the experiment scope in writing
Your team and DeployOpen define the first experiments, such as pricing display, checkout steps or recommendation placements, along with the revenue and conversion metrics each one will be judged on. DeployOpen sets out the deliverables, the systems in scope, the responsibilities on each side and the exclusions in a written scope before any work starts. DeployOpen does not publish a price list, so the quote follows from this written scope. Done when both parties have signed off the scope document and named owners for product, engineering and data.
Map the store, data and decision points
DeployOpen reviews where experiments would run: the storefront, the checkout service, the ecommerce platform's APIs and the order database or warehouse that will hold revenue data. The team records which user attributes are needed for targeting and where each one originates. Your data engineers confirm which warehouse or database holds order data and who can grant read access. Done when a data flow map exists that shows every system an assignment or metric event touches.
Deploy self-hosted GrowthBook in your infrastructure
DeployOpen installs GrowthBook in your own cloud account, data center or private network, using the single Docker image or the official Helm chart for Kubernetes, with metadata stored in MongoDB or a compatible database. The setup covers file storage, network paths to the storefront and data warehouse, and configuration held in environment variables. Your team confirms the deployment is reachable only from the networks you approved. Done when the instance serves the dashboard and API from your environment and a test flag evaluates correctly.
Build server-side and client-side experiment paths
DeployOpen implements the SDK integration for each experiment type, placing pricing or recommendation tests where the decision is made on the server and using client-side evaluation only where the change is cosmetic. The team designs the assignment logic so that the same visitor sees the same variant across pages and sessions. Testers verify that no variant flickers during page load on the tested templates. Done when test cases show consistent assignment and no visible variant swap on the agreed pages.
Connect warehouse-native revenue metrics
DeployOpen connects GrowthBook to your data source, such as BigQuery, Snowflake, Postgres or another supported warehouse, using read-only credentials that run SELECT queries. Revenue, order count and checkout completion metrics are defined in SQL against order tables that your team has approved. The team confirms that only aggregated statistics leave the warehouse for analysis. Done when a metric reproduces a known order total from the source system and the query audit shows only read operations.
Set sample size and analysis rules before launch
DeployOpen and your product and analytics owners agree the minimum detectable effect, the traffic each experiment receives and the stopping rules before any test goes live, so that results are not read early. The team documents how holdouts and guardrail metrics such as average order value and refund rate will be reviewed. Where your edition does not include a feature you want, such as approval workflows or safe rollouts, the gap is recorded for a decision. Done when each planned experiment has a written sample size estimate and an approved analysis plan.
Harden, test and document the handover
DeployOpen applies access controls, secret management, network restrictions and backup procedures to the GrowthBook deployment and the store integration, then runs functional and security tests on the experiment paths. The team delivers runbooks, architecture notes, metric definitions and the configuration record your change managers and auditors may request. Your staff are trained on creating and stopping experiments. Done when the test report is accepted, the runbooks are in your document store and an operator has completed a supervised experiment.
Provide retainer support and upgrades after go-live
After handover, ongoing support, upgrades and continuous testing can be covered by a retainer agreed separately in writing. DeployOpen tracks GrowthBook releases, plans upgrades in a test environment, and reruns the integration tests before changes reach production. Your team decides which experiments continue and which are retired. Done when the retainer scope, response expectations and upgrade process are written down and the first upgrade cycle has been reviewed with your team.
