How to run a feature flag and A/B testing platform RFP
A team that needs gradual rollouts and experiments on infrastructure it controls usually shortlists GrowthBook, Unleash or Flagsmith, and sometimes a SaaS option for comparison. The questions in this topic are neutral: they apply to every candidate, and they only become useful when each vendor or implementation partner answers in writing and ties the answer to your own environment, your identity provider, your data location and your release process. Ask for the edition, the licence and the scope of each offer in writing before any price discussion. A quote for a free edition, a paid cloud plan and a self-hosted Enterprise contract can differ in features, and the feature gap matters more than the headline price. For example, GrowthBook's self-hosted Open Source plan and its self-hosted Enterprise plan list different capabilities, and Unleash and Flagsmith also gate SSO, governance and change control behind paid editions. Confirm which features each candidate includes in the edition you would deploy.
Split the evaluation into two rounds. The functional round tests what the platform does: SDK coverage for your languages and client types, targeting and percentage rollouts, server-side and client-side evaluation, statistical analysis for experiments, and whether the import path from your current tool preserves targeting rules, rollouts and prerequisites. Ask each candidate to run a proof of concept on a copy of one real flag set and report what did not carry over. The operations round tests how the platform runs after go-live: supported database and Kubernetes options, upgrade and rollback procedure, backup and restore, outbound network needs, log and audit export, and the support terms that apply once the fixed scope ends. Score the two rounds separately so that a strong feature demo cannot hide a weak operating model. Record each answer with a version number and a date, because release cycles on these projects are short and a claim about an older version may no longer hold.
Which questions to ask about platform fit, hosting and migration
Which self-hosted edition do we need, and which features only appear above the free tier?
Start by naming the edition and the features you must have, then ask the vendor to confirm in writing which edition provides them. For GrowthBook, the self-hosted Open Source plan is free with unlimited users and 1 project, while SSO and SCIM, advanced permissioning, approval workflows, holdouts and safe rollouts appear in self-hosted Enterprise. For Unleash, the free edition is limited to 1 project and 2 environments, and SSO, SCIM, custom roles and change requests sit in Enterprise. For Flagsmith, SAML single sign-on and governance features such as roles, change requests and audit logs start at the Scale-Up plan on its pricing page. A good answer names the edition, the feature and the price source, and the follow-up is whether the quote covers the Enterprise seats you will actually use.
What license obligations apply if we modify the platform or run it for others?
Ask which license covers the exact artifact you will deploy, because the answer changes by component. GrowthBook's LICENSE file states that code outside its enterprise directories is MIT Expat licensed, while those enterprise directories fall under the GrowthBook Enterprise License. Unleash moved its source code and the unleash-server npm package from Apache 2.0 to AGPLv3 starting with v8.0.0, while its official open-source Docker images remain Apache 2.0. Flagsmith states that most of its code is BSD 3-Clause, with Enterprise Edition features closed source. A good answer is a written license matrix for each component you deploy, and the follow-up is whether your legal team agrees with it if you patch the source or expose the platform to external users.
How do we migrate existing flags and experiments from LaunchDarkly or Statsig?
Each importer covers a different slice of your setup, so ask the vendor or partner to map your current objects to the target. GrowthBook's LaunchDarkly importer uses a REST API access token to pull projects, environments and flags into a preview, and its documentation warns that advanced setups may include cases it cannot handle. Its Statsig guide imports environments, tags, segments, feature gates, dynamic configs and experiments, but metric sources and metrics are in beta and limited to warehouse-native Statsig metrics. Flagsmith's built-in importer overwrites existing environments and flags in the target project and maps archived flags as archived. A good answer lists each object type, its import status and a manual check list, and the follow-up is who verifies the results and how the application switches from the old SDK.
How does Unleash environment export and import handle segments and custom strategies?
Ask for a written procedure that covers the export, the dependency order and the verification step. Unleash moves flag configurations as JSON between environments or between instances, but segments and custom strategies are exported only as references. That means they must already exist in the target before the import runs, or the flags will not evaluate as expected. A good answer lists the segments and custom strategies by name, shows the order of creation in the target, and includes a sample export file used in a test environment. The follow-up is how the team handles flags that reference a segment created in the target under a different ID.
Where does flag evaluation and experiment data live, and what leaves our network?
Ask the vendor to show a data flow diagram that marks each system, each network path and each data type. GrowthBook analyzes experiments against your own data source, such as BigQuery, Snowflake, Postgres or Trino, runs only SELECT queries and transmits only aggregated statistics. Its self-hosted deployment bundles the Next.js front end, Express API and Python statistics engine in one Docker image and stores metadata in MongoDB or a compatible database. A good answer names every service, the database, the object storage option (local, Amazon S3 or Google Cloud Storage for uploads) and any telemetry that leaves the network. The follow-up is whether SDKs evaluate flags locally, so user attributes used for targeting never reach the server.
What hosting model, database and high availability design does the vendor expect?
Ask for the reference deployment for your target platform, the supported database and the upgrade path. GrowthBook maintains an official Helm chart for Kubernetes and suggests a managed database such as MongoDB Atlas or AWS DocumentDB for production. Flagsmith's Helm chart creates its own PostgreSQL server in the cluster by default, while its documentation recommends an externally managed database in production, with optional PgBouncer. Unleash runs its server image against PostgreSQL 15 or later, with the server and admin UI on port 4242, and its open-source Edge caching layer reaches end of life on December 31, 2026. A good answer names the chart version, the database service, the backup design and the cache layer. The follow-up is who operates the database and how failover is tested.
What security evidence should the vendor or partner provide before go-live?
Ask for artifacts you can file, not verbal assurance. Useful items include the SBOM or image list for the release you will deploy, the vulnerability scan output, the authentication design with SSO and SCIM mappings for your identity provider, the role matrix, the audit log export and a documented backup and restore test. For Flagsmith, ask which of these depend on an Enterprise plan. For Unleash, ask whether the OSS Edge caching layer is in the design, given its end-of-life date. A good answer lists each artifact, its version and the person who signs it off. The follow-up is how these artifacts map to the control IDs your auditor will sample. DeployOpen prepares systems, controls and evidence for compliance assessments, but auditors and certification bodies issue the conclusions.
What support, upgrades and fixed scope do we get after go-live?
Separate the fixed-scope implementation from the ongoing support, because they are priced and written differently. DeployOpen's engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts, and ongoing support, upgrades and continuous testing are available as retainers. Ask the partner to list the deliverables, the acceptance criteria, the upgrade cadence and the exclusions, such as work on a vendor's Enterprise features that you have not licensed. Ask the platform vendor for its support tiers and whether the edition you chose includes them. A good answer is a signed statement of work with a named exit point, and the follow-up is how the retainer handles security patches between planned upgrades. DeployOpen does not publish a price list, so ask for a quote built from the written scope.
Which questions matter to security, engineering and product buyers
Security and compliance reviewers score the questions that decide whether the platform can be governed like any other production system. Their questions focus on who can change a flag, how those changes are recorded, and what evidence an auditor can sample without asking the vendor for help. Ask each bidder to show these items in the product, not only in a policy document.
- Identity and access: Ask whether single sign-on and SCIM provisioning are included in the edition quoted. In the Flagsmith FAQ, SAML and Okta integrations are listed as Enterprise features, and the GrowthBook pricing page places SSO and SCIM in self-hosted Enterprise.
- Role model: Ask which roles exist, whether project-level or custom roles are available, and whether approval workflows are part of the same edition.
- Audit logs and change requests: Ask for a sample audit record for a flag change, and confirm which editions include change requests and logs.
- Licence and component review: Ask the bidder to map which code is covered by a commercial licence, including enterprise directories and separately licensed features.
- Vulnerability handling: Ask how security fixes are published, how quickly the bidder commits to patch supported releases, and which version lines receive them.
Confirm the sign-off owner before scoring starts, typically the CISO or delegated security architect, with compliance leads reviewing evidence mapping.
Platform engineering reviewers score how the platform runs after the first installation. Their questions cover deployment target, database choice, caching, upgrades and whether the system can be installed from images that are available inside a restricted network. A feature that works on a laptop can still fail on a cluster with an externally managed database, so ask for the operating assumptions in writing.
- Kubernetes packaging: Ask whether an official Helm chart exists, what it creates by default, and whether it is supported for the edition you plan to run. Flagsmith's chart creates its own PostgreSQL server by default, while its documentation recommends an externally managed database in production.
- Database and cache: Ask which database engine is required and which versions are supported. GrowthBook stores metadata in MongoDB or a compatible database, while Unleash runs against PostgreSQL 15 or later. Ask what Redis or edge caching is needed for production latency.
- Upgrades and support windows: Ask which releases receive fixes and for how long. Confirm whether only the latest release is supported.
- Air-gapped images: Ask for the list of container images, their digests, and the process for updating them offline.
- Operational runbooks: Ask for backup, restore and failover documentation tested against the stated database.
The platform owner should sign off, with the security reviewer confirming that image sources and update paths meet internal policy.
Product and experimentation teams score whether the platform can answer their questions with data they already trust. Their concerns are SDK coverage in the languages and runtimes they use, how experiment metrics are defined, and whether analysis can run against their own warehouse. The answers also determine whether a migration from an existing tool is practical.
- SDK coverage: Ask for the list of server-side and client-side SDKs, their maturity, and whether each one evaluates flags locally.
- Metric definition and warehouse access: Ask which data sources are supported, whether analysis runs as read-only queries, and what leaves the warehouse. GrowthBook documents SELECT-only queries and transmission of aggregated statistics only.
- Statistical methods and holdouts: Ask which analysis methods are included in each edition, especially multi-armed bandits, sticky bucketing and holdouts.
- Migration of existing flags: Ask whether the vendor's importer carries over targeting rules, rollouts and prerequisites, and what the team must recreate by hand. Vendor documentation warns that advanced setups may need manual work.
- Experiment data retention: Ask how long exposure and metric data is retained and who can export it.
The experimentation lead signs off on metric definitions, while the data owner confirms warehouse access before any pilot begins.
Vendor answers that should worry the buyer
Vendor answers to an RFP often sound complete while hiding the points that decide cost, risk and time to value. The six answers below are the ones that most often cause trouble after go-live, so each one should be tested with a written follow-up before a contract is signed.
Vague edition answers
A response that says the platform "supports SSO, roles and approvals" without naming the edition is a warning sign. On self-hosted GrowthBook, SSO and SCIM, advanced permissioning, approval workflows and holdouts are Enterprise features, and the free self-hosted Open Source plan does not include them. Ask for a feature-by-feature table that marks each capability as Open Source, Enterprise or paid add-on, with the edition name and the version it applies to. Confirm the same for Unleash and Flagsmith, where SAML and governance features also sit behind paid plans.
Migration promises without a dry run
A vendor that promises to "import all your LaunchDarkly flags" has not yet shown which items will survive. Importers differ: GrowthBook warns that advanced setups may contain cases it cannot handle and that the application must still switch SDKs, and Flagsmith overwrites existing environments and flags in the target project. Ask for a dry run on a copy of your flag inventory, a written list of unsupported items, and a parallel-run plan before cutover.
Hosting answers that assume internet access
Many hosting answers quietly depend on outbound calls to SaaS endpoints, public container registries or telemetry services. In a restricted network, a platform that needs those calls at startup or for updates will fail or require exceptions. Ask which images, charts and updates must be pulled from outside the network, how they are mirrored, and whether SDK evaluation continues when the server is unreachable. Unleash Edge, which the Unleash documentation says is a caching layer between SDKs and the API, is one component to check on this point.
Security evidence offered as marketing
A statement that the platform is "secure by design" or "SOC 2 aligned" gives an assessor nothing to sample. Ask for artefacts: the current hardening guide, a sample audit log export showing who changed which flag and when, role definitions, the vulnerability disclosure process and the most recent independent test report for the release you will run. DeployOpen prepares controls and evidence for assessments but does not issue audit reports, so the vendor's documents must be checked against your own environment.
Support terms with no defined scope
A support line that promises "24/7 help" without defining what is covered leaves the buyer with disputes at the first incident. Self-hosted platforms split responsibility between the vendor, the implementation partner and your team, so the contract should state which layer each response covers, such as the server, the database, the SDK or the upgrade path. Ask for named severity levels, response targets, the supported version list and what happens when a release reaches end of life. Unleash Edge open-source reaches end of life on December 31, 2026, which is a concrete example of a date to confirm.
Fixed-scope quotes without deliverables
A fixed-scope quote that lists only a total and a timeline cannot be checked at acceptance. Ask for a written scope that names each deliverable, such as the installed and tested environments, the migrated flag set, the SSO integration, the backup and restore test, the runbooks and the handover training. Each item should have an acceptance test and an owner on both sides. A quote that does not list these items will be hard to hold the vendor to, and any change will become a billable surprise.
What the buyer must confirm in its own environment
Some RFP answers depend on systems that only the buyer controls, so vendors cannot answer them on the buyer's behalf. Confirm these internally before the RFP goes out, otherwise the responses will describe a generic deployment that does not match the estate. Start with the data warehouse. If the team plans warehouse-native analysis with GrowthBook, list the exact warehouse (for example BigQuery, Snowflake, Postgres or ClickHouse), the schemas that hold exposure and outcome events, the service account or database role that will run SELECT queries, and who owns that role. Vendors can describe what leaves the warehouse, but only the buyer knows whether the network path to it is open.
Next, settle identity and cluster standards. Decide which identity provider will issue administrator and developer logins, whether SAML or OIDC is the expected protocol, and who manages group mappings and SCIM provisioning. Record the Kubernetes distribution, the approved Helm chart workflow, the ingress and certificate model, and whether the organization requires an externally managed database such as a managed PostgreSQL service. Flagsmith's Helm chart creates its own database in the cluster by default, and Unleash expects PostgreSQL 15 or later, so the buyer's database standard will decide much of the hosting answer. Also confirm the approved backup, restore and key management tools that the vendor must work with.
Finally, decide what outbound traffic is permitted. Confirm whether SDK telemetry, usage reporting, update checks, container image pulls and licence validation may reach the internet, and whether a private registry or mirror is available. Write these answers into the RFP as fixed requirements, not as questions, so the responses can be compared on the same basis. Ask the security team and the data protection officer to sign off on the telemetry position before the RFP is issued, because a vendor's answer on this point can change the hosting and evidence scope.
How to score vendor responses
This table lists the seven areas a self-hosted feature flag and A/B testing RFP should score, what a full score requires in each, and a weighting column the buyer fills in. Weights are set by the buyer, not by DeployOpen, so the table can be reused across vendors without favoring any one platform. Use it to compare written answers, then check the highest-weighted rows against a live demo or reference environment.
| Criterion | What a full score requires | Buyer weight |
|---|---|---|
| Edition fit | Written confirmation which features the required workflows need (SSO, SCIM, roles, approvals, holdouts, audit logs) and which edition includes each one | Buyer sets |
| Licence clarity | Named licence for the core and for each enterprise directory or edition, with the vendor's statement on internal use and hosting for others | Buyer sets |
| Migration evidence | Documented importer or export path from the current tool, a list of known unsupported cases, and a plan to run both systems in parallel | Buyer sets |
| Hosting model | Supported deployment options (Docker, Kubernetes chart, managed database), named database and cache dependencies, and upgrade path inside the client's own network | Buyer sets |
| Security evidence | Written description of authentication, access control, audit logging, encryption and outbound network calls, with a sample of logs or configuration the buyer can inspect | Buyer sets |
| Support terms after go-live | Stated support window for each release, end-of-life dates, patch commitments and the process for security fixes, with any support tier named in writing | Buyer sets |
| Fixed-scope definition | Deliverables, acceptance tests, responsibilities split between vendor and client, and what is excluded, agreed in writing before work starts | Buyer sets |
How to define the fixed scope before work starts
This sequence turns the answers in your RFP into a written scope that names what will be built, migrated, tested and supported, so the quote and the timeline rest on the same facts.
Inventory flags, experiments and owners
Your product, platform and data teams count the flags and experiments in use today, mark which are stale, and name an owner for each active one. Export counts from any existing tool, and record the number of projects, environments and SDK types in use. The step is done when the inventory is signed off by the engineering lead and reconciled with the list of flags scheduled for removal.
Confirm environments and hosting boundaries
Your infrastructure team lists every environment the platform must cover, such as development, staging and production, and states where each runs: which cloud account, data center or air-gapped network. Ask the vendor or partner to confirm the database, cache and edge components each environment needs. The step is done when the environment map is agreed and any data location constraint is written against each one.
Map integrations: SDKs, CI/CD and warehouse
Engineering leads list every application language and SDK, the CI/CD pipelines that will change flags, and the observability and warehouse connections that feed experiment analysis. For each one, state whether the connection is in scope, who owns the credentials, and what data leaves the network. The step is done when each integration has a named owner and a written in-scope or out-of-scope decision.
Define migration sources and cutover rules
If you are leaving a hosted tool, list each source, the export format it supports, and which objects move: flags, segments, environments, experiments and metrics. Record known gaps, such as advanced setups an importer may not handle, and decide how parallel running and SDK switching will be sequenced. The step is done when each source has an agreed object list and a documented list of items that will be rebuilt by hand.
Write acceptance tests for each deliverable
Your team and the implementation partner write pass or fail tests for every deliverable: a flag evaluates correctly in each environment, a migrated flag matches its source, an experiment query returns the expected aggregate, and a kill switch works from the production console. Include security checks such as access roles and audit log retention where they are in scope. The step is done when the test list is attached to the scope and both parties sign it.
Set post go-live support and change terms
Decide what happens after cutover: who handles incidents, the response expectations you require in writing, how upgrades are applied, and whether ongoing support, upgrades or continuous testing will run as a retainer. Confirm which vendor release lines are covered, since support windows differ by release. The step is done when the support terms are written as a separate schedule, distinct from the build deliverables.
Issue the written scope and request the quote
The partner compiles the inventories, integration decisions, migration objects, acceptance tests and support terms into one written scope with deliverables and timelines. Your procurement and security leads review it, and any change after signature goes through a written change request. Most engagements are fixed-scope and agreed in writing before work starts, and the quote follows from that document. The step is done when the scope is signed and the quote is issued against it.
What drives cost and effort, and what to ask for in a quote
Ask every bidder to split its price into subscription, one-time work and recurring operations, because the three lines behave differently. Subscription fees are the only line a vendor publishes. Unleash lists Pay-As-You-Go self-hosted at $75 per seat per month with a 5-seat minimum, and Flagsmith lists Scale-Up at $250 per month billed yearly. GrowthBook's self-hosted Open Source plan is free, but its Enterprise features (SSO and SCIM, approval workflows, holdouts, safe rollouts) are custom priced. Confirm which features your RFP requires, because the feature list, not the seat count, usually decides which edition you must buy. Also confirm whether the vendor's per-seat or per-request rules apply to your own staff, service accounts and SDK traffic.
One-time work covers the migration of existing flags, segments and experiments, configuration of identity and roles, and testing of SDK behavior before cutover. The vendor importers do part of this. GrowthBook's LaunchDarkly importer, for example, documents that advanced setups may include scenarios it cannot handle and that the application still has to switch SDKs afterwards. Flagsmith's importer overwrites existing flags in the target project and asks for a manual check. Ask the quote to itemize each import, the SDK code changes and the parallel-run period. Recurring costs include hosting, the managed database that the vendor documentation recommends for production, upgrades for each release, and support. Your own staff time is a cost too. Ask bidders to estimate hours per line item, and state which parts your team will perform so that the fixed scope reflects the split.
How DeployOpen runs feature flag and A/B testing evaluations and implementation
DeployOpen plans, implements, integrates, hardens, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For feature flag and A/B testing RFPs, we help you turn vendor questions into a written scope, test what each candidate actually does in your environment, and hand over a platform your team can run.
How we scope
- We turn your RFP questions into a written scope covering platform edition, licence obligations and hosting model before any work starts.
- We confirm which edition features your teams need, such as SSO, SCIM, approval workflows and audit logs, and list them against each candidate.
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before work begins, and a quote follows from that scope.
How we build
- We deploy the candidate platforms in your infrastructure and test flag evaluation, experiment analysis and SDK coverage against your own applications.
- We test any migration of existing flags, segments and experiments from your current tool, documenting what imports cleanly and what needs manual rework.
- We run security testing of the deployment, covering access control, configuration and the APIs used by your SDKs, and record the results.
How we hand over
- We deliver the chosen platform with configuration, runbooks, backup and upgrade procedures, and documentation of the decisions made during evaluation.
- We prepare the evidence your compliance assessors will ask for, such as access settings, change records and audit log configuration, without issuing any audit report.
- We walk your platform and operations teams through administration, flag lifecycle and incident response before the handover is closed.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer, so your team does not have to track vendor release notes alone.
- We can keep the platform current with supported releases and re-test flag and experiment behavior after each upgrade.
- Support terms, response expectations and the scope of any retainer are agreed in writing, and should be part of your RFP questions to every vendor.
