LLM observability for NIST AI RMF measurement

How LLM observability supports NIST AI RMF measurement in production

A team with LLM features in production is usually asked one question by its risk function: how does it monitor those features against the NIST AI RMF? The short answer has three parts. NIST AI 100-1 is a voluntary framework, NIST does not certify or endorse any organization's use of it, and tracing and evaluation tools such as Langfuse can produce the records that support the Measure and Manage outcomes without satisfying them alone. The framework's Core has 4 functions, 19 categories and 72 subcategories: Govern, Map, Measure and Manage. Govern is cross-cutting, and most teams start with Map before moving to Measure or Manage. Observability sits mainly in Measure and Manage, so that is where this section focuses.

Three outcomes map most directly to production tracing. MEASURE 2.4 expects the functionality and behavior of the system, as identified in MAP, to be monitored in production, which means trace data, output samples and drift signals tied back to pre-deployment test results. MEASURE 3.1 expects approaches, personnel and documentation for tracking existing, unanticipated and emergent risks, based on intended versus actual performance. MANAGE 4.1 expects post-deployment monitoring plans that capture input from users and other AI actors, including appeal, override, incident response and change management. In Langfuse, traces and scores can cover the first two, and user feedback attached to traces can feed the third. The mapping is evidence support, not compliance: the organization still has to define its thresholds, its reviewers and its escalation path.

Before committing to a tooling design, the team should confirm three things with its risk owner: which MAP entries define the monitored behavior, who is named as TEVV and operations reviewer for each outcome, and how the organization will document the link between an alert and a decision. Those answers determine which traces, scores and feedback fields the application must record. Any traces that contain personal data also need masking and retention decisions, which the Langfuse self-hosting documentation addresses in its enterprise feature list. The NIST AI RMF Playbook offers suggested actions for each subcategory, but it is not a checklist, so the team should choose and justify which actions it adopts.

Mapping AI RMF subcategories to traces, evaluations and feedback

This table maps the NIST AI RMF subcategories most relevant to production monitoring (MEASURE 2.4, MEASURE 3.1 and MANAGE 4.1) to what each must do and how to configure Langfuse to produce the evidence. Read each row as a requirement, then the configuration that generates its records. The main takeaway is that the framework does not prescribe tools: Langfuse supplies the traces, scores, datasets and access controls, while your own procedures decide thresholds, reviewers and responses.

Requirement	What the function must do	How to configure Langfuse
MEASURE 2.4: production monitoring	Monitor functionality and behavior in production, compared with pre-deployment testing, with drift and distribution shift tests and alerts on anomalies.	Tag traces with application version, prompt version and model name. Store pre-deployment evaluation results as a versioned dataset, then compare production score trends against that baseline.
MEASURE 2.4: distribution shift and control limits	Test input and output data for distribution shifts and set control limits or confidence intervals for anomalies.	Attach trace metadata such as input category and length. Define score configs for output properties, then chart them over time. Set alert thresholds in your monitoring stack, using Langfuse exports or APIs.
MEASURE 2.4: newly collected ground truth	Assess outputs against newly collected ground truth on a defined cadence.	Route reviewed production traces into a new dataset version. Record reviewer scores through score configs so each ground truth label links back to its trace.
MEASURE 3.1: user feedback against internal metrics	Compare user and community feedback with internal performance measures to track emergent risks.	Record user thumbs or comments as scores on the originating trace. Compare these with automated evaluation scores in the same dashboard or export.
MEASURE 3.1: error response times	Measure how quickly errors are identified and answered, and feed support and incident staff needs back into the risk process.	Use trace timestamps and a tag for error status. Confirm with your incident team which fields they need, and which role may view error traces.
MEASURE 3.1: simpler-model and human baselines	Compare system risks with simpler models, human baselines and manual benchmarks.	Run the baseline model or human answers through the same evaluation score configs. Store both in one dataset so the comparison uses identical criteria.
MANAGE 4.1: post-deployment TEVV	Perform post-deployment testing, evaluation and verification for validity, bias, privacy and security.	Schedule recurring evaluation runs against a fixed dataset version. Keep each run's configuration and scores so assessors can trace results to the version tested.
MANAGE 4.1: red-team cadence	Run red-teaming at a prescribed cadence and record the results.	Store red-team prompts in a separate dataset with restricted project access. Tag each attack trace with the test campaign identifier.
MANAGE 4.1: dataset deletion and rectification	Track dataset modifications, including deletion and rectification requests.	Use dataset versioning to record each change. Confirm whether your Langfuse edition includes the data retention policies and audit logs needed to show deletions, since these are listed among features that need an Enterprise license key.
MANAGE 4.1: incident and attack sharing	Share errors, near misses and attack patterns with incident databases.	Export flagged traces with their score and tag history. Limit export rights through project-level RBAC roles, which also need an Enterprise license key according to the Langfuse license key documentation.

How a self-hosted Langfuse stack is built and sized

Langfuse documents Docker Compose on a single virtual machine for testing. This setup is useful for proving that applications send traces, that prompts and outputs are captured as expected, and that evaluation scores appear where your reviewers expect them. Langfuse notes that this topology has no high availability, so it should not hold the only copy of monitoring evidence that an assessor will rely on.

Containers on one host: Langfuse Web (UI and APIs) and Langfuse Worker (asynchronous event processing) run alongside PostgreSQL, ClickHouse, Redis or Valkey, and S3-compatible blob storage.
Resource floor: The scaling guide lists 2 CPU and 4 GiB for web, worker and PostgreSQL, 1 CPU and 1.5 GiB for Redis or Valkey, and 2 CPU and 8 GiB for ClickHouse.
No redundancy: A single host failure stops ingestion and the UI at the same time, and traces sent during an outage depend on the sending application's retry behavior.
Test data only: Use synthetic prompts for the first runs, since masking and access controls are not yet proven.
Version pinning: Record the Langfuse release used in tests so the production upgrade path can be tested against it.

Before leaving this environment, confirm which backup and restore steps your team will repeat in production.

Which open-source platforms fit this measurement work

Measuring AI risk in production needs a trace store, an evaluation layer, a point where data is masked, and a way to analyze exported metrics. The cards below cover the components that usually sit in that stack. Each fit depends on your licensing position, so confirm the license terms with your legal team before you commit.

Langfuse for traces and evaluations

Langfuse records traces, prompts, outputs, scores and cost for each LLM call, which gives MEASURE 2.4 and MEASURE 3.1 a single system of record. Its core features are under the MIT license, and the ee directories fall under a separate enterprise license. The self-hosted version uses the same codebase as Langfuse Cloud. Check which controls you need, such as project-level RBAC, data retention policies and audit logs, because several of these need an enterprise license key.

Arize Phoenix as an alternative tracer

Phoenix is an LLM tracing and evaluation tool that suits teams wanting an open-source option for experiments and evaluations. Its Elastic License 2.0 prohibits providing the software to third parties as a hosted or managed service. That restriction matters if you plan to run Phoenix for customers or in a shared managed offering. Internal use is a different question, so confirm with counsel how your deployment model is classified before you choose it.

OpenTelemetry Collector as the masking point

The OpenTelemetry Collector receives, processes and exports telemetry in a vendor-agnostic way, so it can sit between your applications and the trace store. This is where you can drop or mask fields before data reaches storage, and where you can route data to more than one backend. It is not an observability backend, so you still need a store such as Langfuse. Confirm which Collector components you need and check their stability level in each component's README, because support varies.

LiteLLM for gateway logging and cost

LiteLLM sits between applications and model providers and records requests, cost data and logs in one place. That gives you per-model and per-application usage evidence, which supports MEASURE 2.4 monitoring of production behavior. It is MIT licensed except for its enterprise directory. Confirm whether the features you need, such as particular guardrails or logging integrations, are in the open-source directories or the enterprise directory before you rely on them.

Superset or Metabase for analysis

Exported metrics, traces and evaluation scores need a dashboard layer to support drift reviews and periodic risk reporting. Apache Superset is Apache-2.0 licensed. Metabase is AGPL outside its enterprise directory, which uses a commercial license. Either tool can read from a database that you control, which keeps evaluation data inside your environment. Confirm the license that applies to your use, especially if dashboards will be shared outside the organization.

Which Langfuse controls need an enterprise license key

Do I need an enterprise license key to run self-hosted Langfuse for NIST AI RMF monitoring?

No. All core Langfuse features and APIs are available in the MIT-licensed open source version without limits, and self-hosted deployments use the same codebase as Langfuse Cloud. Traces, evaluations, scores, user feedback and prompt versions needed to measure production behavior do not depend on a license key. The license question matters only for a short list of governance features, which the rest of this section covers. Confirm current terms with Langfuse before you plan around them, because licensing can change.

Which Langfuse controls require an enterprise license key?

According to Langfuse's license key documentation, these self-hosted features need an Enterprise license key:

Project-level RBAC roles
Protected prompt labels
Data retention policies
Audit logs
Server-side data masking
UI customization, organization creators, the org management API with SCIM, and the instance management API

The list is the reference point for scoping. Confirm the exact list against the current license key page before procurement, since the feature set is maintained by the vendor.

Are RBAC roles and audit logs needed as evidence for NIST AI RMF measurement?

The NIST AI RMF does not prescribe tools, so you do not need a specific feature to show measurement. MEASURE 3.1 expects approaches, personnel and documentation to track risks over time, and MANAGE 4.1 expects post-deployment monitoring plans with change management and incident response. Project roles and audit logs help show who can view traces containing user data and who changed monitoring configuration. If your internal policy requires those controls, you can meet them with your identity provider, network controls and infrastructure logs, or you can license the Langfuse features. Decide this per control, with your governance and privacy owners.

Can I mask sensitive data in traces without buying the enterprise tier?

Yes, if you mask before data reaches Langfuse. Server-side data masking is listed as an enterprise feature, but masking in the application SDK or in the OpenTelemetry Collector runs before storage and does not depend on a Langfuse license key. This approach keeps personal data out of ClickHouse and blob storage from the start. The trade-off is engineering effort: masking rules must be written, tested and kept consistent across every instrumented application, and you should document them as a control. Test with representative prompts that include real-world personal data patterns in a non-production environment.

What does the free self-hosted Langfuse plan include?

The free plan is the open source version, which includes all core features and APIs under the MIT license for the repository, excluding the separately licensed enterprise directories. You run the stack yourself, which means Langfuse Web, Langfuse Worker, PostgreSQL, ClickHouse, Redis or Valkey and S3-compatible blob storage. Your recurring costs are infrastructure, operations, upgrades and support staff time. Langfuse does not provide a named support engineer, solutions architect access or a support SLA on this plan. Confirm these terms on the self-hosted pricing page.

What does the Langfuse Self-Hosted Enterprise plan add, and what does it cost?

Langfuse's self-hosted pricing page describes a custom-priced Self-Hosted Enterprise plan. It adds a named support engineer, solutions architect access and a support SLA, along with the enterprise features that need a license key. No price is published here, so request a quote from Langfuse directly. Ask whether the license covers your deployment model, how the key is activated in air-gapped or restricted networks, which SLA applies and whether the license terms change with the vendor's roadmap. Budget for the license as a recurring cost, separate from infrastructure and DeployOpen engagement fees.

Does DeployOpen handle the Langfuse license decision and enterprise setup?

DeployOpen plans, implements, configures, hardens, backs up, upgrades, documents and supports self-hosted platforms in your infrastructure, and it prepares systems and evidence for compliance assessments. In a fixed-scope engagement, we can define which controls you need, configure the features that your license covers, and document the masking and access design. Licensing decisions and the vendor's support terms are yours to agree with Langfuse, and DeployOpen does not issue audit reports or attestations. Scope and fees are set in writing before work begins, after the license decision is clear.

How to build the post-deployment monitoring plan in phases

The sequence turns MAP outputs into a monitoring plan that Langfuse can evidence, from system inventory through alerting, feedback intake, scheduled TEVV and change review. Each phase ends with a record the organization owns. The depth of each phase depends on the organization's own risk decisions.

Inventory LLM systems and link them to MAP outputs
Governance owners and the product team list every LLM application, model endpoint, prompt template, retrieval source and tool integration in scope. Each entry is tied to the context, intended use, known limitations and risk categories recorded under MAP, so the monitoring plan starts from documented intent rather than from what the tooling happens to log. Done means every production system has an owner, a named business process and a MAP reference that the monitoring plan cites.
Instrument traces with OpenTelemetry and Langfuse
Engineering instruments each application so that requests, model calls, retrieval steps, tool calls, prompt versions, token counts and cost are recorded as traces. Teams can use the Langfuse SDKs or OpenTelemetry instrumentation that exports to Langfuse through an OpenTelemetry Collector, which the organization can run in its own environment. Done means a sample of production traces can be opened in Langfuse, linked to a system and a prompt version, and checked against what the application actually did. Confirm masking of personal data before traces are stored.
Define evaluation and drift metrics with control limits
The TEVV and product teams choose the metrics that reflect each system's intended performance, such as groundedness, answer correctness, refusal behavior, latency and cost, and compare them with the results from pre-deployment testing. They set distribution checks on input and output data and define control limits or confidence intervals for anomalies, in line with the monitoring actions in the NIST AI RMF Playbook for MEASURE 2.4. Done means each metric has a documented baseline, a threshold, a method for scoring and a named reviewer, and the thresholds are approved before alerting is turned on.
Set alerts and response times
Operations configures alerts on drift, anomalies, error spikes, cost overruns and failed evaluations, and routes each alert to the team that can act on it. The risk owner records the response expectations for each severity, including who triages, who can disable a feature and how an incident is escalated. Measuring how long responses take gives evidence for the MEASURE 3.1 expectation that error response times are tracked. Done means a test alert reaches the right people and the response record for it is kept in the organization's own systems.
Run feedback and appeal intake
Product and support teams add user feedback channels, such as thumbs up or down, comment fields and a route for users to contest an output or request human review. Feedback is attached to the trace it refers to, and support and incident staff report on the information and staffing they need. This phase covers the MANAGE 4.1 input from users and other AI actors, including appeal and override paths. Done means feedback arrives in Langfuse linked to a trace, and a documented procedure shows how an appeal leads to an override or a fix.
Schedule red-teaming and TEVV reviews
The TEVV function sets a cadence for post-deployment testing of validity, bias, privacy and security, and for red-teaming that targets the system's known failure modes, including prompt injection and data leakage. Results are compared with production traces and evaluation scores, and new failure cases are added to datasets in Langfuse for regression testing. The organization sets the cadence itself, since NIST does not prescribe one. Done means each system has a dated schedule, a named tester and a findings log that feeds back into the metrics.
Review the plan at each change under change management
Every prompt, model, retrieval source, tool or threshold change goes through the organization's change process, and the monitoring plan is reviewed alongside it. The review confirms that metrics, alerts, feedback routes and red-team scope still match the MAP record, and it records any decommissioning or rollback decision. Langfuse prompt versions and trace metadata give the change history. Done means each change has a linked review record, and the monitoring plan carries a version number that auditors can trace.

Evidence an assessor will ask to see and where it comes from

This table maps the records an assessor is likely to request for NIST AI RMF measurement and post-deployment monitoring to the component that produces them. Read each row as a question to answer before the assessment starts, not as proof that the evidence already exists. DeployOpen prepares these records and controls, but independent assessors decide whether they are sufficient.

Evidence item	Source component	What to confirm
Trace records with versioned prompts and model metadata	Langfuse traces, prompt versions and OpenTelemetry instrumentation	Prompt versions, model names and parameters are attached to each trace, and the mapping to a deployed release is documented
Evaluation runs and score history	Langfuse evaluation datasets, scores and experiment runs	Pre-deployment test results are stored alongside production scores so MEASURE 2.4 comparisons can be reproduced
Drift and alert logs	Alerting rules in the monitoring stack fed by trace and score metrics	Control limits, alert thresholds and who received each alert are recorded and reviewed on a set schedule
Feedback and appeal records	User feedback scores in Langfuse and the ticketing system used for appeals	Feedback is linked to specific traces, and appeal and override outcomes are kept with timestamps and reviewer identity
Incident and change tickets	Ticketing system linked to trace IDs, plus release records	Each incident references the affected traces, and each prompt or model change links to the evaluation that approved it
Dataset deletion and rectification logs	Langfuse dataset management, data retention settings and change records	Deletion and correction requests are applied to stored traces and datasets, and the action is logged with date and operator
Collector configuration and access logs	OpenTelemetry Collector configuration files, Langfuse audit logs and SSO records	Configuration is version-controlled, and audit logs of who viewed or changed traces are available, noting that some audit log features need an enterprise license key
Retention settings	Langfuse retention configuration, ClickHouse and object storage lifecycle rules	Retention periods match the period set in the organization's policy and any applicable law, and deletion jobs have been tested

Where AI RMF monitoring overlaps with EU AI Act logging duties

The NIST AI RMF and the EU AI Act both ask for records of how an AI system behaves after launch, and a single trace store in Langfuse can serve both purposes if it is designed for both. Article 12(1) of the EU AI Act requires high-risk AI systems to technically allow for the automatic recording of events (logs) over the lifetime of the system. Article 19(1) requires providers to keep those logs for at least six months unless other law provides otherwise. The Article 19 text on the AI Act Service Desk is the natural next read for your legal team. Whether your system is high-risk at all is a classification question you must settle first, and it is not answered by this page.

The overlap is real in practice but not complete. MEASURE 2.4 and MEASURE 3.1 ask you to monitor production behavior, compare it with pre-deployment testing and track emergent risks, and the traces, evaluation scores and user feedback you collect for those outcomes are often the same records the AI Act treats as logs. The AI RMF does not set retention periods, and it does not set a minimum period for keeping logs. Confirm the retention period for each trace category with your legal review, then configure it in your own environment, since Langfuse's data retention policies are a feature that requires an Enterprise license key on self-hosted deployments. Confirm that feature set against Langfuse's license key documentation before you rely on it.

Decide early which records are kept for the statutory period, which are kept only for monitoring and which must be deleted sooner. Evaluation scores and feedback records may need a different schedule from raw prompts that contain personal data, and deletion or rectification requests must reach both. Ask your legal and privacy teams to sign off the retention matrix before the first production release, and make sure the monitoring plan for MANAGE 4.1 names who reviews those logs, how long they stay accessible and who can export them.

What tracing tools alone do not satisfy

Langfuse records what a model did. It does not decide what risk the organization accepts, who answers for it, or whether the records themselves are handled safely. The cards below cover the gaps that remain after tracing is in place, and each one needs an owner and a documented decision.

Risk tolerance and governance decisions

NIST AI 100-1 places Govern as a cross-cutting function that sets policies, accountability and risk tolerance before Map, Measure and Manage begin. A trace dashboard cannot say how much error is acceptable for a given use case or who may stop a model. Before go-live, confirm that a named owner has approved thresholds for each evaluation score, and that the escalation path for a breached threshold is written down and tested.

No certification path

NIST does not validate or endorse any organization's use of the AI RMF, and it describes no certification, accreditation or conformity assessment program for it. A Langfuse deployment showing monitoring data is evidence of practice, not proof of conformance. When a customer or regulator asks for AI RMF alignment, ask the reader to define which outcomes were chosen and what internal evidence supports each one, rather than presenting a certificate.

Pending revision of AI 100-1

As of 17 October 2026, NIST says AI RMF 1.0 is being revised as part of the White House AI Action Plan and has not published a revised version. The July 2025 plan recommends removing references to misinformation, diversity, equity and inclusion, and climate change. NIST expects a formal review with community input no later than 2028. Map your subcategory references to the version you adopted, and record which revision your monitoring plan was built against so later changes can be traced.

Playbook not updated since 2023

The NIST AI RMF Playbook, which supplies suggested actions and documentation prompts for each subcategory, shows its last content update as August 2023. NIST says it is neither a checklist nor a set of steps, and AIRC will update it after the framework is revised. Treat the suggested actions as a starting point. Document where your monitoring deviates from them and why, so auditors can see the reasoning rather than a gap.

Trace data as personal data

Prompts, retrieved documents and user feedback often contain personal or confidential information, so the trace store becomes a data store in its own right. It needs masking before storage, retention limits, and access rules that separate engineers debugging a model from staff reviewing individual conversations. Ask who can open raw traces, how erasure requests reach ClickHouse and object storage, and which masking happens in the SDK or OpenTelemetry Collector before data leaves the application.

Validating the evaluators

LLM-as-judge scores, rubric checks and sampled human reviews are instruments, and they drift like the system they measure. MEASURE 3.1 expects teams to compare user and community feedback with internal performance measures and to track the gap over time. Before a judge model's scores feed a drift alert or a release decision, check it against a labeled set of human-scored examples, record its agreement rate, and re-run that check after the judge model or prompt changes.

What buyers ask before hiring a partner for AI RMF monitoring

What drives the cost of AI RMF monitoring on Langfuse?

The cost depends on how many LLM systems need monitoring, how many applications and model calls each one instruments, how much trace data they generate, and how much evaluation design the risk owners ask for. The main cost components are the implementation work by a partner, the infrastructure you run (Langfuse Web, Langfuse Worker, PostgreSQL, ClickHouse, Redis or Valkey, and S3-compatible storage), and any Enterprise license for features such as data retention policies and audit logs. Infrastructure and licensing recur, while instrumentation and design are largely one-time. Ask for a quote that lists each system in scope, the instrumentation count, the storage sizing assumptions, and whether the license decision is included or priced separately. Langfuse's self-hosted pricing page describes a free open source plan and a custom-priced Self-Hosted Enterprise plan, so confirm the license scope with the vendor.

How long does an AI RMF monitoring setup take?

No supplied fact gives a duration for this work, so the timeline should come from a written scope rather than a generic estimate. The schedule is shaped by the number of LLM systems, whether they already emit OpenTelemetry traces, how many evaluations must be designed and validated, and how quickly the organization can approve retention and access rules. Ask the partner to break the timeline into phases: platform deployment, instrumentation, evaluation and alert design, and handover. Ask which phases depend on your reviewers, and what happens to the schedule if a review is delayed.

What does the work need from our staff?

The work needs named people on three sides. Risk owners define which MEASURE 2.4, MEASURE 3.1 and MANAGE 4.1 outcomes apply to each system and accept the monitoring thresholds. Data owners confirm which prompts, outputs and user feedback may be stored, how long, and who may view them. Reviewers, often from TEVV, model risk or operations, score evaluation samples and decide what happens when an alert fires. Engineers provide access to the applications and deployment environment. Agree on these roles in the written scope so that no outcome lacks an owner.

Can our team set up AI RMF monitoring without a partner?

Yes, a capable internal platform team can deploy Langfuse and instrument applications without outside help, and the open source version includes core features and APIs without limits. The harder parts are usually the governance and evaluation design: mapping traces and scores to specific NIST AI RMF subcategories, setting drift and anomaly alerts, defining post-deployment monitoring plans, and keeping evidence that auditors can follow. Also check whether you need Enterprise license features such as project-level RBAC, audit logs or server-side data masking. A useful test is whether your team can name the owner and review cadence for each monitored outcome.

Is there an AI RMF certification we can get?

No. There is no AI RMF certification. NIST states that it does not validate or endorse any individual organization or its approach to using the AI RMF, and NIST's framework pages, FAQ, Playbook and Roadmap describe no certification, accreditation or conformity assessment program. NIST also says the framework is voluntary. What you can show an assessor is evidence that you used the framework, such as documented monitoring plans, evaluation results and review records. Be cautious with any vendor or consultant who claims an AI RMF certificate. DeployOpen prepares systems, controls and evidence for compliance assessments but does not issue audit reports, attestations or certifications.

How is a partner engagement different from a SaaS tracing plan?

A SaaS tracing plan typically gives you a hosted service priced by seats, traces or spans and their retention limits, with the vendor operating the data store. A partner engagement deploys self-hosted Langfuse inside your own infrastructure, so prompts, outputs and scores stay in storage you control, and the work includes mapping those records to NIST AI RMF outcomes. The partner's scope covers architecture, instrumentation, ClickHouse and blob storage sizing, backup, evaluation design, and documentation of monitoring plans. Ask each option for its total cost, which controls you must configure yourself, and where the data physically resides.

What should we check on the Langfuse license before we commit?

Start with the license terms. Langfuse's repository is MIT licensed except the ee directories, which fall under a separate enterprise license, and Langfuse states that all core features and APIs are available in the open source version. Several controls that matter for monitoring need an Enterprise license key, including project-level RBAC roles, data retention policies, audit logs, server-side data masking and the SCIM-enabled organization management API. Decide which of these your AI RMF evidence depends on, then confirm the price and support terms with Langfuse, because its self-hosted pricing page lists the Enterprise plan as custom-priced.

How DeployOpen scopes and supports AI RMF monitoring

DeployOpen plans, builds and hands over self-hosted Langfuse monitoring in your own infrastructure, mapped to the NIST AI RMF outcomes your team has chosen to measure. Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts, and ongoing support is available as a retainer.

How we scope

Agree which AI RMF subcategories in MEASURE and MANAGE the monitoring will evidence, in writing, before build starts.
Inventory the LLM applications, frameworks and user feedback channels to be instrumented in Langfuse.
Confirm which Langfuse features need an Enterprise license key, such as RBAC and data retention policies, before the scope is fixed.

How we build

Deploy self-hosted Langfuse in your cloud account or data center, with PostgreSQL, ClickHouse, Redis or Valkey and S3-compatible storage.
Instrument applications through the Langfuse SDK or OpenTelemetry, with client-side masking of personal data before traces are stored.
Configure evaluations, drift and anomaly alerts, and feedback capture so production metrics can be compared with pre-deployment test results.

How we hand over

Document a monitoring plan that maps each trace, evaluation and feedback signal to the AI RMF outcome it supports.
Provide runbooks for alert triage, incident escalation, dataset updates from production failures and change management.
Hand over a record of configuration, access roles and retention settings that your auditors or assessors can review independently.

How we support

Run upgrades within and across Langfuse major versions with migration plans, including the ClickHouse headroom checks each major version requires.
Review the monitoring plan when NIST revises AI RMF 1.0, and adjust mappings to the revised outcomes in scope.
Offer ongoing support, upgrades and continuous testing as a retainer, with coverage terms agreed in the written scope.