Why teams need Open WebUI usage analytics before they scale
Once staff can reach Open WebUI, the platform lead needs answers to three questions: which models people actually use, who is using them, and how many tokens those conversations consume. The Analytics dashboard provides visibility into usage per model and per user, and arena comparisons give a way to test candidate models against the company's own tasks instead of relying on public benchmarks. Both features matter most before the deployment grows, because the choices made early about models, access and cost are the ones that become hard to change later.
Both capabilities depend on the deployment's underlying setup. Usage records live in the database that Open WebUI writes to, so a single-replica SQLite install and a multi-instance PostgreSQL deployment will not behave the same way at scale. Attribution by user also depends on the identity setup: if accounts come from SSO, LDAP or SCIM provisioning, the names and groups reported in the dashboard reflect whatever the identity provider supplies. Before relying on either feature for reporting, confirm how your chosen install method and database store this data.
Feature availability also varies by release. Open WebUI publishes new versions frequently, with v0.12.3 released on October 16, 2026, so the documented behavior of the Analytics dashboard and arena features should be checked against the exact release you plan to run, not against older screenshots or blog posts. Ask the project's documentation and release notes for the version in use, and test the dashboard on a non-production copy with realistic accounts before you present its numbers to finance or leadership.
Plan for the access rules that determine who sees what. Analytics data about other people's usage is sensitive, so decide which roles can open the dashboard, whether managers see only their own groups, and how long usage records are retained. Arena comparisons raise a similar question: prompts written by staff may contain customer or internal data, so agree on which workloads are appropriate for model evaluation and which models are approved to receive them.
Finally, decide what the data is for before the first report is produced. Token counts only translate into cost if you know which providers bill per token, which run on your own hardware, and what a fair comparison looks like across them. Write down the questions the dashboard must answer, such as adoption by team or spend by model, and confirm with the project or your deployment partner that the data needed to answer them is captured in your setup.
Analytics and evaluation options and when each fits
This table compares the four ways a team can measure Open WebUI use or model quality: the Analytics dashboard, arena comparisons, ratings and external observability, plus a row on data retention. Read each row for the decision it supports, the team that should own it and the condition under which it fits. The main takeaway is that built-in views answer adoption and comparison questions, while per-request tracing usually needs a separate system.
| Option | Decision it supports | Owner | When it fits |
|---|---|---|---|
| Analytics dashboard | Whether usage is growing and which models and users account for it. Confirm the exact metrics in the Open WebUI features documentation. | Platform team administering the instance | A small group of administrators needs a shared view of adoption and token use during rollout or cost reviews. |
| Arena comparisons | Which model answers a team's own prompts best, judged side by side before standardizing. | Use-case owner, with business reviewers who can judge answers | The team has representative prompts and reviewers available to compare outputs before choosing a default model. |
| Ratings | How models rank over time on the tasks users actually run. | Team lead, with a named person who reviews results | Many users work on similar tasks and feedback is consistent enough to show trends. Sparse feedback gives weak rankings. |
| External observability | Per-request tracing of latency, errors and prompt behavior for debugging, incidents and formal evaluation. Examples include Langfuse or OpenTelemetry pipelines. | Platform or SRE team, with security approval for stored prompts | Investigations or formal evaluations need request-level detail you cannot confirm the built-in views provide. Confirm the integration in the integrations documentation. |
| Data retention and privacy | How long prompts, responses and usage records are kept, and where each option stores them. | Security and compliance, with the data owner | Any option that sends prompts or responses to a system outside the Open WebUI instance needs a documented retention decision first. |
How arena comparisons and ratings help choose a model on your own workload
Arena comparisons and ratings let a team choose a model by testing candidates against its own work rather than against public benchmarks. The method has three parts: a fixed set of representative prompts, the same prompts sent to each candidate model through the providers already connected to Open WebUI, and structured ratings from reviewers who know the work. Done well, the result is a record that shows which model performs best for each task family in your organization, along with the evidence behind that choice. Start by building the prompt set from real internal tasks. Ask the teams that will use the platform to contribute examples of requests they actually make, such as summarizing a policy, drafting a customer reply, or explaining a configuration. Group the prompts by task family and keep a share of hard cases, including ambiguous requests and ones that require your internal terminology. Assign one owner to the set so that it changes through review rather than by ad hoc edits. Keep the number of prompts per family small enough that reviewers can finish a round in one sitting.
Human rating and automated checks answer different questions, and most teams need both. Reviewers judge tone, usefulness and whether an answer fits the context of your business, but their scores vary between people and tire over long sessions, so calibrate them on a few shared examples before the round starts. Automated checks, such as format validation, required-field presence, or regular-expression tests for prohibited content, are repeatable and cheap to run, but they cannot tell whether an answer is correct. Use automated checks to remove clearly failing outputs and reserve reviewer time for the comparisons that matter. Confirm the exact arena steps, comparison controls and rating mechanics in the Open WebUI features documentation for the version you install, because the interface and available rating options can change between releases.
Keep the prompt set free of sensitive data before any model sees it. Replace customer names, account numbers, credentials, health or personal details, and unpublished financial figures with realistic-looking placeholders that keep the structure of the task intact. Decide who is allowed to author, edit and review prompts, and record that decision in your data handling policy. Remember that prompts and responses pass through each connected provider, so the approval of a provider for production use does not automatically cover test prompts that contain copies of internal material. Where a prompt must reflect confidential content, run the comparison only against models hosted in infrastructure you control.
Record every round so that results can be reviewed later by someone who was not in the room. For each run, keep the prompt set version, the model names and versions, the provider and any configuration that affects output, the date, the reviewer identities or roles, the ratings, and the reasons for any low score. Store the record in a location with access controls and a retention period set by your records policy, not in individual chat histories. Review the results against the decision they support, such as which model to offer to a given group, and set a date to repeat the round, since provider models change and your own tasks evolve. Ask whether the platform's exports, or your own logging, can capture these fields before you rely on them.
Setting up usage analytics and model evaluation in Open WebUI
This sequence takes a new Open WebUI instance to a state where approved reviewers can see token use by model and user, access to the Analytics dashboard is limited, and model comparisons run on your own prompts. Each step ends with a check you can record. Confirm exact menu paths and permission names against the current documentation before you write them into runbooks.
Confirm the install method and release
The platform engineering lead chooses one official method: Docker, Docker Compose, Kubernetes with Helm, or pip install open-webui followed by open-webui serve. Record the release you deploy, using the GitHub releases page as the reference, so later upgrades and documentation lookups match what runs. The install method also determines how environment settings and persistent volumes are managed, so document both. This step is done when the instance starts, an administrator can sign in, and the recorded version matches the running instance. Check the current installation documentation for the exact commands.
Confirm the database backend and replica count
Decide the database before any usage is recorded, because the usage history is stored there. Multi-instance deployments require PostgreSQL, since SQLite does not handle concurrent writes from several processes well. SQLite remains workable for a single replica, evaluation or a small team, but only when the database file sits on locally attached SSD or NVMe storage. Multiple replicas also need Redis, shared file storage and an external vector database. This step is done when a restart keeps the administrator account and prior activity intact. Ask your platform team to confirm the backup schedule for this database.
Connect the model providers
Add the providers your users need: Ollama for locally served models, OpenAI, Anthropic, or any OpenAI-compatible service. Store provider keys in the secret management system your organization already uses, and limit who can edit provider connections. Open WebUI connects these providers from a single interface, so users reach every approved provider from one place. Check that each connection lists the models you expect and that a short test prompt returns a response. This step is done when each connection shows its models and a test prompt succeeds on each one. Confirm in the current documentation where keys are stored and how they are rotated.
Limit the Analytics dashboard to approved administrators
Create a group for the people who may review usage and evaluation data, and grant that group access to the Analytics dashboard and the evaluation features. Keep default permissions for ordinary users narrow, and remove broad administrator rights from anyone who does not need them. Where sign-in is federated, map this group through your identity provider or through SCIM provisioning once those are configured. This step is done when a standard user cannot see the dashboard and an approved reviewer can. Confirm the exact permission names and menu locations in the current role-based access documentation before you rely on them.
Create a test user and prompt
Create a dedicated test user who belongs to the ordinary user group, not the administrator group, so the test reflects what an employee sees. Choose one model and send a fixed, short prompt, so the expected usage is easy to compare later. Record the time, the model name, the test user and the exact prompt text in your test log. This step is done when the chat completes and the response comes from the selected model. Keep this test account for repeat checks after each upgrade.
Verify usage against the test user
Sign in as an approved reviewer and open the Analytics dashboard. Confirm the test prompt is counted once, against the test user and the model you selected, with token counts consistent with the fixed prompt. Where the provider offers its own usage record, compare the two figures and note any difference in how each side counts tokens. If the prompt is missing or attributed to the wrong model, check the provider connection and the model identifier before going further. This step is done when the dashboard entry matches your test log. Confirm current menu paths in the documentation.
Run side-by-side evaluations on your workload
Build a fixed set of prompts drawn from tasks your users actually perform, with sensitive content removed. Use the arena comparison to send the same prompt to two or more models, and record ratings the same way for every run. Keep the prompt set and the rating criteria in version control, so results can be repeated after a model or release change. This step is done when each candidate model has ratings against the same prompt set and those ratings are recorded in a form the decision owner can review. Confirm where ratings are stored and who can view them.
Protecting usage data and evaluation prompts
Usage analytics and arena ratings show who uses which model and what they asked, so access and storage need the same care as the chats themselves. The cards cover visibility, groups, automation keys, retention, backups and audit evidence. Identity and MFA are set through your identity provider and Open WebUI's SSO, OIDC, LDAP and MFA options, which should be confirmed for your deployment.
Who can see per-user usage
Per-user token counts and adoption figures are administrator data, so decide early which accounts can open the Analytics dashboard and export its data. Ordinary users should see only their own chats and settings. In a test instance, confirm which roles reach the dashboard, whether administrators can view individual chat content from the same screens, and whether exported files carry user identifiers that should be pseudonymized before they leave the platform team.
Role-based access and groups
Open WebUI supports role-based access control with groups and per-resource permissions. Use them to decide which teams can run arena comparisons, rate responses and view aggregated results, and which models each group can reach. Map groups from your identity provider only after you have tested that a user removed from a group loses dashboard and arena access. Document who holds the administrator role, since that role sees the most.
API keys used by automations
Scripts, scheduled reports and integrations often authenticate with API keys tied to a user account. Their requests then appear under that owner, so adoption figures can blend a service's traffic with an employee's. Create a dedicated service account for each automation, record its purpose, and confirm how keys are scoped, rotated and revoked in your version before you count their usage in reports.
Retention of chat and evaluation data
Chat history, arena votes and ratings persist in the application database, so retention is a database decision as well as a product setting. Decide how long each record type must exist, who may delete it, and whether deleting a user also removes their history and votes. Test deletion on a copy of the database, confirm which retention options your Open WebUI version provides, and document any gap that needs a scheduled cleanup job.
Backup handling for prompts and results
Backups copy everything in the database, including prompts, responses and evaluation votes, so they carry the same sensitivity as the live system. Encrypt them, limit restore access to named operators, and set a backup retention period that matches your chat policy. Plan separately for uploads and shared file storage, because knowledge files and attachments sit outside the database. Test a restore in a separate environment before you rely on it.
Audit needs by regulated sector
Audit expectations differ by sector. A bank or healthcare provider may need evidence of who accessed usage data, when retention jobs ran and which automations held keys, while a public body may focus on records management and periodic access reviews. DeployOpen prepares systems and evidence for assessments but does not issue audit reports. Ask your auditor which logs and reviews they expect, then confirm that Open WebUI and your database can produce them.
Keeping analytics accurate when Open WebUI runs on several replicas
Once Open WebUI runs on more than one replica, the analytics dashboard is only as trustworthy as the database and shared services underneath it. The scaling guide requires PostgreSQL instead of SQLite for multi-instance deployments, because SQLite does not handle concurrent writes from multiple processes well. Token and usage counts that several replicas write at the same time need a database built for that load, so plan the PostgreSQL instance, its backups and its access controls before you add a second replica. Confirm with the Open WebUI documentation which database settings your release expects, and test the migration from SQLite on a copy of your data before you cut over.
Multiple replicas also need Redis as a shared state store for sessions, websocket connections and application state, along with shared file storage so uploads and knowledge stay reachable from every replica. Without shared state, a user's requests can land on a replica that does not know their session, and per-user usage records can fragment across instances. Decide early which storage platform will hold uploads, who operates Redis and how it fails over. These choices shape the reliability of the usage data you report on, not only the user experience.
The default ChromaDB is not suitable for this setup. The scaling guide warns that it will crash in multi-process setups, so an external vector database is required when you run several workers or replicas. Choose a supported option from the thirteen vector databases Open WebUI lists, such as PGVector, Qdrant, Milvus or Elasticsearch, and confirm it fits your hosting and backup model. Retrieval activity tied to knowledge collections is one of the things your analytics will reflect, so an unstable vector store will distort adoption figures as well as break search.
Token revocation is a security issue that affects how you read session and user activity. Without Redis, signing out does not revoke tokens. They stay valid until they expire, which by default is 4 weeks. A user who signs out on one replica may still be able to make requests, and usage recorded after sign-out may be attributed to an account that should be closed. Treat Redis as required for any multi-replica deployment, and test sign-out and account deactivation across replicas before you rely on the dashboard for access reviews.
A single SQLite file is only appropriate for single-replica deployments, evaluation and small teams, and only when the database file sits on locally attached SSD or NVMe storage rather than network storage. Analytics built on one SQLite file should not be presented as organization-wide figures once you scale out. Before you rely on the dashboard for reporting or model selection, confirm your replica count, your storage type and the database engine in writing, and record that decision with the analytics owner so the numbers in your reports can be traced to a known setup.
Limits to confirm before you rely on the dashboard
The Analytics dashboard and arena are useful for tracking adoption and choosing models, but their outputs depend on settings, data sources and license terms in your own installation. The cards below list what to verify in your environment before you use the numbers for reporting, cost decisions or audit evidence.
Dashboard metrics and token sources
Confirm exactly which measures the Analytics dashboard reports in your installed version, such as token totals, request counts, and per-model or per-user adoption. Establish whether token counts come from the model provider's usage response or are estimated by the interface, since the two can differ for the same chat. Test both against a known set of prompts before you report usage or cost figures to finance.
Export and retention controls
Determine how analytics data leaves the system, whether through an interface export, the API, or a direct read from the application database. Check the columns, the format, and whether exports include user identifiers that map back to your identity records. Review the retention settings for chats, usage records and feedback, and confirm whether they also apply to aggregates. Pruned history may not be recoverable, so set retention before the dashboard becomes a reporting source.
Timezone and aggregation behavior
Check which timezone the dashboard uses to bucket usage by day, week and month, and whether it follows the server, the browser or a configured setting. Confirm how aggregation handles requests that cross a day boundary, retried requests, and deleted users or models. Compare one day's totals with an independent count from logs so the boundaries match your finance or reporting calendar.
Arena results for audit
Confirm whether arena comparisons, votes and ratings can be exported in a readable form, including the prompt, both model responses, the voter, the timestamp and the outcome. Without a documented export, evaluation decisions may rest on results you cannot show later. Test the export with your internal audit team before you cite arena results in a model selection record.
License branding threshold
The Open WebUI License forbids altering or removing branding in deployments that exceed 50 users in any rolling 30-day period, unless the operator holds an enterprise license or written contributor permission. Count distinct users across every rolling 30-day window, since a calendar-month count can understate peak use. Ask your legal team to review the clause against your deployment before you change interface branding.
Enterprise licensing scope
Confirm with Open WebUI enterprise sales whether an enterprise license changes features, support terms or the licensing of any components in your deployment, beyond the white-labeling and custom branding it allows. Open WebUI does not publish pricing, so the quote follows from an expected seat count you provide. Request the scope in writing, including support options, before you plan any branding change.
What does Open WebUI usage analytics setup cost and take?
What does Open WebUI usage analytics setup cost?
The cost follows from a written scope. DeployOpen does not publish a price list, so a quote is produced after the scope, deliverables and assumptions are agreed in writing. The fee covers planning, configuring analytics and evaluation, any infrastructure or database changes, identity and provider integration, testing and handover. Build work is usually one-time. Upgrades, support and continuous testing are optional retainers that recur. Ask the quote to separate build costs from recurring support, and to state which hosting, licensing and cloud charges your organization pays directly to third parties.
How long does Open WebUI analytics and evaluation setup take?
Duration depends on the scope you agree. No standard duration applies, because the timeline is set by the number of replicas, database and storage changes, model providers, identity integration and how much evaluation design is needed. A single-instance setup with one provider and existing identity is a smaller job than a multi-replica deployment with several providers and a formal evaluation program. Ask for a timeline with milestones, the dependencies on your identity and cloud teams, and the change windows your production environment requires.
Which factors change the cost of an Open WebUI engagement?
Four factors drive most of the effort. First, replicas and data layer changes: multi-instance deployments need PostgreSQL instead of SQLite, Redis for shared state and shared file storage, plus an external vector database, because the default ChromaDB is not suited to multiple workers or replicas. Second, the number of model providers, such as Ollama, OpenAI, Anthropic or OpenAI-compatible services. Third, identity integration through OIDC, LDAP or SCIM 2.0. Fourth, how much evaluation design you need, including test sets, scoring criteria and reporting. Each one adds configuration, testing and documentation.
Does Open WebUI enterprise licensing cost extra for our team?
Enterprise licensing is quoted separately and is not published. To get a price, contact Open WebUI enterprise sales with your expected seat count. Enterprise licenses allow white-labeling, custom branding and dedicated support options. The license matters when your deployment exceeds 50 users in any rolling 30-day period, because the Open WebUI License forbids altering or removing branding at that scale unless you hold an enterprise license or written contributor permission. Enterprise licensing is a separate contract from any implementation work DeployOpen performs, so confirm with the vendor which features the license covers.
Can our internal platform team set up Open WebUI analytics?
Often yes, if the team already runs the supporting platforms. Open WebUI installs with Docker, Docker Compose, Kubernetes with Helm, or pip. Multi-replica setups also require PostgreSQL, Redis, shared storage and an external vector database, and the identity integration needs an administrator who understands OIDC, LDAP or SCIM. Internal teams usually struggle with evaluation design, not installation. Check your team's experience with these components, the time they can commit, and whether they can test before go-live. Confirm the analytics and evaluation steps against the documentation for the version you run.
What does our team need to provide for this engagement?
Your team provides access, decisions and test material. Expect to name an accountable owner, provide access to the cloud accounts, data center or network where Open WebUI runs, and assign an identity provider administrator. You will also supply the current Open WebUI version and install method, the list of model providers and their approved use, your usage and retention policy, and representative prompts or tasks for model comparisons. Agree on the change windows, the test users, and who signs off acceptance. Delays in any of these items usually extend the schedule more than technical work does.
What should we ask for in a written quote?
Ask for an itemized, fixed scope before work starts. A useful quote lists each deliverable, the environment it applies to, and the assumptions it depends on, such as replica count, database choice, provider count, identity protocol and the number of evaluation criteria. It should name exclusions, acceptance criteria, the handover documentation you will receive, and the responsibilities of your team. Separate one-time build work from any retainer for upgrades or support, and ask how changes to scope are priced and approved. Confirm the Open WebUI enterprise licensing cost separately, because it is quoted by the vendor.
How DeployOpen scopes and supports Open WebUI analytics
DeployOpen plans, implements, documents and supports Open WebUI in infrastructure you control, including the analytics and model evaluation setup described on this page. Each engagement starts from a written scope that sets out what is measured, who can see it and how comparisons are run on your own workload.
How we scope
- A written scope that names the models, user groups and usage questions the analytics must answer.
- Agreed data sources, retention periods and access rules for token and adoption reporting, documented before build begins.
- Fixed-scope engagements, with deliverables and timelines agreed in writing before any work starts.
How we build
- Configure the Analytics dashboard so token use and adoption can be reviewed per model and per user.
- Set up arena comparisons and ratings on a test set built from your own prompts and workloads.
- Test every analytics and evaluation change in a staging instance before it reaches production users.
How we hand over
- Written documentation of the dashboard configuration, evaluation method and the steps to repeat a comparison.
- Access settings for administrators and reviewers, with roles and groups mapped to the people who need them.
- A walkthrough with your platform and data teams so they can review results and rerun evaluations.
How we support
- Optional retainer for upgrades, monitoring of the analytics setup and changes to evaluation sets.
- Review of Open WebUI releases and the license terms that apply to your user count before each upgrade.
- Support scoped to your environment, so questions about usage data are answered with your actual configuration.
