LLM observability for financial services

Banks and insurers need prompt, output and evaluation records that survive model validation and ICT risk reviews. This page explains what to record, who may view client data in traces, and which self-hosted platform fits.

On this page

Why LLM observability for financial services starts with records, not dashboards

A model risk or technology team in a bank or insurer needs to answer one question after an incident or validation review: what did the LLM feature actually receive, return and cost on a given day, and who could see it? Dashboards show trends, but validation reviewers and supervisors ask for the individual records behind those trends. For most teams, the answer is self-hosted Langfuse, run in the organization's own cloud account or data center, because its core features are available under the MIT license and the traces stay in storage the organization controls. Phoenix is licensed under the Elastic License 2.0, which prohibits providing the software to third parties as a hosted or managed service, so it fits only where the deployment model stays within that license. Confirm the license terms with counsel before choosing it.

Once a prompt contains client data, a trace becomes a regulated record rather than a debugging aid. Prompts often carry account details, payment information or personal data, and outputs can repeat them. That raises questions the platform must answer by configuration: how long traces are kept, which roles may open traces with client content, and whether personal data is masked before it is stored. In self-hosted Langfuse, project-level RBAC roles, data retention policies, audit logs and server-side data masking are listed as Enterprise license key features, so the licensing decision is also a control decision. Ask the platform owner to map each of these controls to a written retention and access standard before the first production trace is recorded.

DeployOpen plans, implements and documents these deployments and prepares the evidence a validation review or ICT risk assessment will ask for. It does not issue audit reports, attestations or certifications; independent auditors and certification bodies do that work. Treat any partner who promises a certificate for the observability stack as a warning sign. The practical next step is to list the LLM features in scope, decide with legal which client data may enter traces, and agree the retention and access rules before scoping the build.

Which obligations shape the trace design in a bank or insurer

This table maps each obligation to the trace settings it touches: retention, access roles, data location and vendor contracts. Read it as a list of inputs to your own legal and model validation review, not as a compliance determination. The main takeaway is that prompts and outputs become records with their own retention, access and location duties once they reach the trace store.

ObligationWhat it means for trace designEvidence a reviewer will ask for
EU AI Act Articles 12 and 19 (logging and six-month retention)High-risk systems must technically allow automatic event logging over the system's life, and logs must be kept at least six months unless other law sets a longer period. Set trace retention per system category, not one global value, and confirm with counsel which retention applies.Retention settings per system, log schema, documented retention period and how deletion is blocked before it ends
DORA Article 28(3) and 28(8) (register and exit)Each self-hosted component and any SaaS fallback must appear in the register of ICT contractual arrangements. Where tracing supports a critical or important function, document and test an exit plan that covers moving or exporting traces.Register entries for each component, tested exit or migration runbook, export format for traces
DORA Article 30(2) (contract terms)Contracts with the implementation partner and any hosted provider must state where services run and data is processed, give notice of location changes, and cover data return on termination or insolvency. Trace storage location should be written into the contract.Signed contract clauses on processing locations, change notice and data return, plus the storage region for ClickHouse and object storage
SEC ICFR IT general control areas (program changes, access)If traces support financial reporting or controls, change management for prompts, evaluators and tracing configuration and access to trace data are IT general control areas an auditor may test. Keep approvals and role assignments traceable.Change tickets for instrumentation and evaluator changes, role-based access lists, access review records
FTC Safeguards Rule (MFA for system access, 16 CFR 314.4(c)(5))Applies to non-bank institutions under FTC jurisdiction. Every individual who accesses the trace UI, API keys or storage needs multi-factor authentication unless the Qualified Individual approves an equivalent control in writing. Confirm applicability with counsel.MFA configuration for the trace platform and admin consoles, written approvals for any equivalent control
PCI DSS v4.0.1 (cardholder data in prompts)If a prompt, tool result or output carries primary account numbers or sensitive authentication data, the trace store falls inside the cardholder data environment scope. Mask or tokenize card data before ingestion, or scope the tracing platform accordingly. Confirm scope with your QSA.Data flow diagram showing where card data could enter, masking rules applied in the SDK or Collector, scoping decision
Access roles on client-data tracesFinancial services traces often hold client identifiers and account context. Restrict who can view raw traces versus aggregated scores. Some role and retention controls in self-hosted Langfuse sit in the Enterprise license, so confirm which features your firm needs before design.Role matrix, license key confirmation for needed features, sample access review
Data location and cross-border transferKeep trace storage in the jurisdiction your data policy and regulators require. Confirm with your data protection lead how GDPR storage limitation applies to prompt and output records.Storage region and hosting location per environment, transfer assessment where any vendor is involved

What ends up in traces and who may see it

Client names, national identifiers, email addresses, phone numbers and customer numbers often appear in prompts when a relationship manager or servicing agent pastes case notes into an assistant. Once a prompt is traced, those identifiers sit in PostgreSQL, ClickHouse and blob storage alongside the model output, so the trace store becomes a personal data store that needs the same controls as the source system. The main line of defense is masking before data leaves the application, using a redaction step in the SDK or an OpenTelemetry Collector processor, so the identifier never reaches storage in clear text.

  • Mask at the source: Redact identifiers in the application or in a Collector processor before export, since server-side masking in Langfuse is an Enterprise feature.
  • Restrict project access: Project-level RBAC roles, which require an Enterprise license key, limit who can open traces from a given business line.
  • Apply retention limits: Data retention policies, also gated behind Enterprise, define how long traces that contain identifiers are kept.
  • Log access: Audit logs, an Enterprise feature, record who viewed or exported traces.

Confirm with the vendor which of these features your license key covers before designing the access model around them.

Which platforms fit regulated LLM tracing and why

Financial institutions usually narrow the field by licensing, operating model and where trace data is stored, not by features alone. The options below cover a self-hosted trace and evaluation platform, a license-restricted alternative, a vendor-neutral telemetry layer and a gateway that logs model calls. Most regulated designs combine two or more of them.

Self-hosted Langfuse

Langfuse is the most common starting point for self-hosted LLM tracing and evaluation. Its core is MIT licensed, but the ee directories fall under a separate enterprise license, and self-hosted deployments run the same codebase as Langfuse Cloud. Features such as project-level RBAC, data retention policies, audit logs and server-side data masking need an enterprise license key, so confirm which controls your access and retention rules depend on before selecting it.

Arize Phoenix

Phoenix provides LLM tracing and evaluation under the Elastic License 2.0. That license prohibits providing the software to third parties as a hosted or managed service, so an internal deployment run by your own team is a different case from a provider running it for other entities. Bank operating models that involve a third-party operator, a shared service company or a managed observability provider need legal review before Phoenix is chosen.

OpenTelemetry Collector

The OpenTelemetry Collector receives, processes and exports traces under Apache-2.0, and it can run as an agent beside services or as a gateway for many sources. It is vendor-neutral, so it can remove or hash sensitive attributes before data leaves the application boundary. OpenTelemetry is not a storage or visualization backend, so a backend such as Langfuse must still be selected and operated beside it. Check component stability in each README before relying on a processor for masking.

LiteLLM gateway logging

LiteLLM sits between applications and model providers and records calls, token usage and cost across more than 100 LLM APIs. Its core is MIT licensed, but the enterprise directory has separate terms, so features your compliance team relies on should be mapped to the license that covers them. Gateway logs show which application and key called a model, which complements trace-level evaluation data. Confirm whether the logs can be exported to your retention store and how prompts are masked before they are written.

How Langfuse stores traces and what that means for ICT risk

A self-hosted Langfuse deployment is a set of components the institution runs itself, and each one is a location where client prompts and outputs can sit. Langfuse Web serves the UI and APIs, Langfuse Worker processes events asynchronously, and the data then lands in PostgreSQL, ClickHouse, Redis or Valkey and S3-compatible blob storage. For DORA purposes, each of these is part of the ICT service chain and must be mapped in the register of information under Article 28(3), including the cloud account, region and any managed service that hosts it. Confirm with your implementation partner exactly which component holds full prompt text, which holds only metadata, and where backups of each are written.

The deployment pattern matters as much as the component list. Langfuse documents Docker Compose on a single VM for testing only, with no high availability, so a regulated production trace store should be planned on the Kubernetes deployment path with Helm or Terraform that Langfuse documents for AWS, Azure and GCP. The scaling guide lists minimum resources of 2 CPU and 4 GiB for web, worker and PostgreSQL, 1 CPU and 1.5 GiB for Redis or Valkey, and 2 CPU and 8 GiB for ClickHouse, with at least 16 GiB of ClickHouse memory for larger deployments. Ask for these figures to be translated into a sized design for your expected trace volume, because they are floors and not capacity estimates.

These choices feed two contract and exit duties. Article 30(2) of DORA requires every ICT services contract to state where services are provided and data processed, give advance notice of changes, and cover data return on insolvency or termination, so the storage locations above belong in the contract schedule with your provider. Article 28(8) requires documented, tested exit strategies for ICT services that support critical or important functions, which for trace data means a tested export path from ClickHouse and blob storage and a version-pinned upgrade path. Langfuse's v3 to v4 guide requires ClickHouse 25.12 or later and advises about 3x ClickHouse disk headroom when backfilling, so ask your partner to include that headroom in the exit and upgrade plan.

How a bank rolls out LLM tracing in controlled phases

The sequence moves LLM tracing from a documented scope to a production system whose storage location, masking, access roles and retention can be shown to a validation team before any client data is recorded. Each phase closes with a written output that the next phase depends on.

  1. Scope use cases and data classes

    The model risk owner, the business sponsor and the data protection lead list each LLM use case, the applications that call models, and the data classes that prompts, outputs and evaluations may contain, including client identifiers, account data and payment card data where it applies. The team also confirms whether any in-scope system falls under PCI DSS, because cardholder data environment boundaries change where traces may sit. The step closes with a signed scope document that names the systems to instrument and the data classes each one may send to traces.

  2. Decide architecture and data location

    The architecture team chooses the hosting accounts or data centers, the network zone for Langfuse Web, Langfuse Worker, PostgreSQL, ClickHouse, Redis or Valkey and S3-compatible storage, and the path from applications through OpenTelemetry or the SDK. It also records where backups go and which ICT third parties, if any, touch the stack, so that the contractual and register-of-information duties under DORA are already mapped. The output is an architecture decision record and a data flow diagram that the validation team reviews before any build starts.

  3. Run a non-production pilot with masking tests

    Engineers deploy the stack in a non-production environment, using synthetic prompts that mirror the real data classes, and instrument one application first. Masking is tested at the SDK or OpenTelemetry Collector layer, and the team checks that known identifiers are removed before they reach ClickHouse. Langfuse documents Docker Compose on a single VM for testing only, so the pilot is not a production topology. The step closes with a masking test report showing which patterns were caught, which were missed and how each gap was fixed.

  4. Validate the design and test security

    Model risk and information security review the pilot evidence, the configuration of project roles, the audit log settings and the retention design. Security testing covers the web interface, APIs, network exposure and the authentication path, including OIDC or SSO integration for the Collector and the Langfuse login. Features that Langfuse reserves for the Enterprise license key, such as project-level RBAC, data retention policies, audit logs and server-side masking, are confirmed against the licensing decision made in the scoping step. The output is a validation pack and a list of accepted residual risks with named owners.

  5. Cut over to production with roles and retention set

    The production stack is deployed on Kubernetes using the Helm or Terraform approach that Langfuse documents for AWS, Azure or GCP, or on the bank's equivalent platform, with the web and worker containers scaled and the ClickHouse sizing taken from the scaling guide. Access roles are assigned to named groups, retention periods are configured to match the approved policy, and sensitive projects are isolated. Before go-live, the team tests that a user without the right role cannot view a trace containing client data. The output is a signed production readiness record and a tested restore of a backup.

  6. Plan the v3 to v4 migration and backfill

    Where a deployment is on Langfuse v3, the migration plan follows the v3 to v4 upgrade guide, which requires ClickHouse 25.12 or later (26.4 recommended), PostgreSQL 15 or later and Redis 7.0 or later, and moves the data model to observations first. If historic data must be backfilled, the guide advises about 3x ClickHouse disk headroom, so the storage team must confirm capacity before the backfill starts. The team rehearses the migration in a non-production copy and confirms the restore path. The output is a migration runbook with a tested rollback and an approved storage forecast.

  7. Operate upgrades and periodic reviews

    The platform team applies updates within a major version, which Langfuse describes as non-disruptive with database migrations applied at startup, after testing in the non-production environment. Major releases go through their dedicated migration guides and the same validation gate used at cutover. Access reviews, retention checks and masking regression tests run on a schedule the bank sets in its own policy, and changes are logged in the ICT change process. The output is a recurring evidence set that shows the platform is still configured as validated.

Which evidence validation reviews and auditors ask for

This table lists the evidence a model validation team, internal audit or an external assessor is likely to request from a self-hosted LLM tracing setup, and the component that produces each item. DeployOpen prepares the evidence, while the firm's validators and independent assessors decide whether it is sufficient. Use it to check what your environment can produce before the first review.

Evidence itemProduced byWhat the reviewer checks
Trace inventory of LLM applicationsLangfuse projects and tracing instrumentation in each applicationEvery production LLM use case appears, with its model, version and owning team
Retention settings and deletion logsLangfuse data retention policies (enterprise license key) plus storage lifecycle rules in object storage and ClickHouseConfigured retention period matches the policy, and deletion runs leave a record
Access role matrix for trace dataLangfuse project-level RBAC roles (enterprise license key) and SSO group mappingOnly named groups can view traces containing client data, and the matrix matches the access policy
Audit logs of trace accessLangfuse audit logs (enterprise license key) and the identity provider's sign-in logsWho viewed, exported or changed traces, with timestamps that can be traced to a person
Masking configuration for client dataClient-side masking in the SDK or OpenTelemetry Collector, and server-side masking where licensedAccount numbers, national identifiers and similar fields are masked before storage
Evaluation records per model versionLangfuse evaluation scores and datasets, linked to prompt and model versionsEach model version used in production has documented test results and sign-off
Change records for prompts and modelsPrompt versions and labels in Langfuse, plus the change ticket system of the firmEvery production prompt or model change has an approver, a reason and a rollback path
DORA register of information entryThe firm's ICT third-party register, updated for the self-hosted platform and any implementation partnerEntries cover the contractual arrangements for the tracing platform and its support provider, per Article 28(3)
Exit and data return planDocumented export of traces and datasets, tested restore procedure and the firm's transition planThe plan covers exit from the platform and return of data, as Article 28(8) and Article 30(2) require for critical or important functions

What banks and insurers ask before hiring an implementation partner

What drives the effort and cost of a self-hosted LLM observability project?

The main drivers are the number of applications to instrument, the trace volume they generate, whether you need enterprise features, and how much ClickHouse storage and compute you provision. Instrumentation is usually the largest variable because each model call, tool call and retrieval step needs to be captured consistently. Data volume determines ClickHouse sizing, and the Langfuse scaling guide gives minimum resources of 2 CPU and 8 GiB for ClickHouse, with at least 16 GiB of memory for larger deployments. Software licensing is a separate line item: all core Langfuse features are in the MIT-licensed version, but project-level RBAC, data retention policies, audit logs and server-side masking need an Enterprise license key. Ask for a quote that lists each application, the expected trace volume, the license decision and the validation documentation as separate lines.

How long does it take to get the first production trace?

The first trace depends on the readiness of your environment more than on the software. A test deployment can run on a single virtual machine with Docker Compose, but the Langfuse documentation says that setup has no high availability, so production deployments use Kubernetes with Helm or Terraform. The schedule is set in the written scope, and the main gating items are cloud or data center access, network approvals, secrets management and a decision on which applications go first. Ask the partner to show a dated plan with a first milestone that traces a single non-sensitive application in a non-production environment before any client data flows through the system.

What must our team supply before the engagement starts?

Three inputs gate most projects. First, a data classification of the prompts, outputs and files your applications handle, so the team knows which traces contain client data, account identifiers or other regulated content. Second, named validators or model risk reviewers who sign off on what must be recorded, retained and reviewed for validation. Third, network and security approvals for the hosts, database endpoints, object storage and identity provider the platform will use. Ask the partner to list these inputs in the written scope with an owner and a due date for each.

Can our internal engineers run the platform after the project?

Yes, if your team has the skills and the operating model to do so, and the partner's job is to hand over a documented system. The platform has several moving parts: Langfuse Web, Langfuse Worker, PostgreSQL, ClickHouse, Redis or Valkey and S3-compatible storage. Each needs patching, backup, capacity planning and monitoring. Upgrades differ by type: the Langfuse upgrade guide says updates within a major version are non-disruptive and apply database migrations at startup, while major upgrades follow dedicated guides and require specific ClickHouse, PostgreSQL and Redis versions. Ask whether ongoing support, upgrades and testing will be a retainer, and whether your staff will be trained to operate the system alone.

How does PCI DSS scope change if cardholder data appears in prompts?

If cardholder data or sensitive authentication data appears in prompts, outputs or traces, the tracing platform can fall within PCI DSS scope because PCI DSS applies to any system that stores, processes or transmits cardholder data or could affect the security of the cardholder data environment. The current version is PCI DSS v4.0.1, published 11 June 2024, and the 51 future-dated v4.0 requirements became effective on 31 March 2025. Scope is decided by your assessor and the payment brands, not by the platform vendor. The practical step is to prevent cardholder data from reaching traces at all, through masking in the SDK or OpenTelemetry Collector, and to confirm the scope decision with your Qualified Security Assessor before go-live.

What does DeployOpen provide, and what does it not provide?

DeployOpen plans, implements, customizes, integrates, hardens, backs up, upgrades, documents and supports self-hosted open-source platforms inside your own infrastructure, and prepares systems, controls and evidence for compliance assessments. DeployOpen does not issue audit reports, attestations or certifications. Those come from independent auditors and certification bodies, such as a Qualified Security Assessor for PCI DSS. Engagements are typically fixed-scope, with scope, deliverables and timelines agreed in writing before work starts, and DeployOpen does not publish a price list, so a quote follows from that written scope.

Which DORA duties does a self-hosted tracing platform touch?

DORA applies to financial entities in the EU from 17 January 2025, and a tracing platform that supports a critical or important function falls under its ICT risk duties. Your register of information must list the contractual arrangement for the platform, Article 28(8) expects documented, tested exit strategies for ICT services that support critical or important functions, and Article 30(2) requires each ICT services contract to state where data is processed and to cover data return on termination. Self-hosting inside your own infrastructure simplifies the data location question, but the exit plan and contract terms still apply to any partner or license you use. Confirm the classification of the function with your DORA lead.

Questions to put to a partner before signing a scope

Start with masking, because it decides what the trace store becomes. Ask the partner where client data is masked: in the application SDK, in the OpenTelemetry Collector before export, or on the Langfuse server. Ask which fields are redacted, how account numbers, national identifiers and free-text client details are detected, and how they test the masking with sample traces before production data flows. Then ask which Langfuse features they will enable and under which license. Data retention policies, audit logs and server-side data masking are listed in the license key documentation as needing an Enterprise license key, so ask for the written license position in the scope rather than assuming the open source edition covers the bank's control set. Confirm the current key requirements with Langfuse before signing. Next, ask how validation access is separated from developer access. Model validation teams usually need to read traces, scores and evaluation datasets without being able to change prompts or application code. Ask whether project-level RBAC roles will be configured, how the partner maps the validation function, the development teams and the operations team to roles, and who approves additions to the validation group. Ask how access reviews are scheduled and where the access logs for the trace store are kept. Also ask how retention is set. The partner should show how the retention period for traces meets the six-month minimum for logs in the EU AI Act (Article 19(1)) where it applies, and how that period is weighed against the bank's own model risk policy and any longer records rules. Ask who signs off the retention setting. Finally, ask about recovery and exit. Request a written backup and restore plan that covers the PostgreSQL database, the ClickHouse data and the S3-compatible blob storage, including how the three are restored to a consistent point, how a restore is tested, and what recovery time the partner commits to in writing. For the register of information under DORA Article 28(3), ask the partner to supply the description of the service, the locations where data is processed and the data return process. DORA Article 28(8) also expects a documented and tested exit strategy, so ask for a transition plan that shows how traces and evaluations leave the platform in a usable format if the arrangement ends. Ask which of these documents the partner will hand over at signing rather than at the end of the project.

How DeployOpen works on this

DeployOpen plans, implements, hardens, documents and supports self-hosted Langfuse inside the firm's own cloud accounts or data centers, so prompts, outputs and evaluations stay in storage the firm controls. We prepare the systems, controls and evidence a model risk or ICT risk review will ask for, while independent auditors and assessors make their own determinations.

How we scope

  • We agree a written, fixed scope covering the applications to instrument, the model risk and ICT risk requirements in play, and the deliverables.
  • Scope records which trace data includes client information, who may view it, and which retention periods the firm has approved.
  • We confirm with the firm which Langfuse features need an enterprise license key before the design is fixed, such as RBAC roles and retention policies.

How we build

  • We deploy Langfuse Web, Langfuse Worker, PostgreSQL, ClickHouse, Redis or Valkey and S3-compatible storage in the firm's infrastructure.
  • We instrument LLM applications and configure masking of client identifiers in the SDK or OpenTelemetry Collector before data reaches storage.
  • We configure SSO, project roles and audit logging, and test access so that only approved reviewers can open traces containing client data.

How we hand over

  • We document architecture, data flows, retention settings, access model and backup and restore procedures for the firm's validation and ICT risk files.
  • We deliver evidence packs mapping configured controls to the firm's own model risk and DORA register and exit-strategy requirements, for review by its teams and auditors.
  • We run a handover session with platform, security and model risk staff covering upgrade steps, including v3 to v4 ClickHouse requirements.

How we support

  • Ongoing support, upgrades and continuous testing are available as retainers, agreed in writing alongside the original scope.
  • Upgrades within a major version are planned with the firm, and major upgrades follow Langfuse's dedicated migration guides under a tested change process.
  • Periodic security testing of the tracing stack, including its APIs and access controls, can be included in the retainer scope for ongoing assurance.

Talk to our team.

Tell us what you're working on, whether it's a deployment, an audit, a security test or a cyber range. You'll speak with an engineer who can help you scope it.

  • 30-minute call: free, with no obligation.
  • NDA on request: we can sign before you share details.
  • Clear next steps: a scope and plan after the call.