LLM observability RFP questions

A capable partner names the exact frameworks and languages in your stack, such as a specific Python service calling a model SDK, a Node.js backend, or a workflow tool, and says how each one is traced. Ask for a list per application, with the instrumentation method for each: a native Langfuse SDK integration, an OpenTelemetry instrumentation library, or manual spans written by your engineers. Gaps should be written down as out of scope or as work that needs a decision. A partner who answers "all of them" has not looked at your code yet.

OpenTelemetry is a vendor-neutral framework for generating, exporting and collecting traces, metrics and logs, but it is not a storage backend, so Langfuse or another tool still has to receive and display the data. The OpenTelemetry Collector, licensed under Apache-2.0, can sit next to services as an agent or run as a gateway that receives telemetry from many sources. Ask the partner whether the Collector will redact fields, batch exports and authenticate senders, and which trace attributes they will map. Trace API stability differs by language, so confirm the SDK status for each of your languages before committing.

Yes, if the instrumentation records parent and child spans, and a good partner will demonstrate this on a request from your own application. For a single chat turn, you should see the retrieval call, the model call, any tool calls and the post-processing as nested spans under one trace. Ask how context is propagated across services and queues, because a broken parent link splits one user request into several unrelated traces. Request a sample trace review as part of acceptance, not a slide with a generic diagram.

Cost attribution depends on the model provider reporting token counts for each call and on a price table that maps model names to unit prices. Ask the partner where the price table lives, who updates it when providers change prices, and how cost is grouped by user, team, feature or tenant. If requests pass through an AI gateway such as LiteLLM, confirm whether cost is recorded at the gateway or in the application, because double counting is a common error. Ask for a reconciliation test that compares recorded cost with the provider invoice for one day of traffic.

Trace volume is estimated from request volume, the number of model and tool calls per request, the size of prompts and outputs, and the retention period you choose. The partner should ask for your current request rates per application, peak-to-average ratios and the share of requests that will be traced, then calculate storage needs for ClickHouse and blob storage from those numbers. Agents can produce hundreds of spans per request, so a single estimate for all applications is a warning sign. Ask the partner to show the formula and assumptions in writing, then measure real volume during a pilot before sizing production.

The effort depends on the number of applications, their languages, how much code must change, and whether engineers from your team will do the work. Sizing is set in the written scope, not by a generic duration, so ask the partner to break the work into applications, instrumentation tasks, testing and handover. Each application should have a named owner on your side who approves the span design and data fields. Ask what is one-time (initial instrumentation and dashboards) and what recurs (SDK upgrades, new endpoints, and price table maintenance).

A promise to instrument everything without reviewing your applications is a red flag, because LLM applications differ in framework, language, hosting and data flow. Watch for answers that skip the test plan, avoid naming the stack, or treat masking and cost tracking as later add-ons. A credible partner asks for an inventory of applications, builds a test plan with sample requests, and states what will not be traced, such as client-side code or third-party tools that cannot emit spans. Ask for the test plan and acceptance criteria in the proposal before you sign.

Masking should happen before data leaves the application, either in the SDK or in the OpenTelemetry Collector, so raw personal data never reaches ClickHouse or blob storage. Langfuse's documented server-side data masking is an Enterprise license feature, so an open source deployment depends on client-side controls you design and test. Ask the partner which layer masks each field, and why that layer was chosen for your applications. The OpenTelemetry Collector is useful when several services send traces, because one set of processors can cover them all. Confirm the Collector processors and SDK hooks against the versions you deploy, since OpenTelemetry component stability varies.

Self-hosted Langfuse documents several controls that need an Enterprise license key, including server-side data masking, data retention policies, audit logs, project-level RBAC roles and protected prompt labels. Core tracing, evaluation and APIs are available in the MIT-licensed open source version without limits. Before you commit, ask the partner to list every masking and retention requirement from your policy and map each one to either a client-side control or a licensed feature. Confirm current licensing terms directly with Langfuse, because a license decision changes the architecture and the budget.

Masking rules should live in version control as code, with each rule tied to a named data category such as email addresses, phone numbers, account numbers or national identifiers. Each change should pass an automated test suite before deployment. A good partner will show a masking test that sends sample personal data (illustrative examples are fine, such as fake names and fake card numbers) through the instrumented application, then checks that the stored trace contains only the masked form. Ask who approves new rules, how rule versions are linked to trace records, and how a failed test blocks a release.

They should, because personal data often appears in model responses, retrieved documents and tool calls, not only in the user prompt. Agent traces are the most exposed, since a single request can carry API responses containing customer records. Ask the partner to show which span attributes are masked for inputs, outputs, tool arguments and tool results, and how retrieved document chunks are handled. Masking only the prompt field is a common gap. Verify that the rules also cover metadata fields and error messages, which can echo user input.

A practical review queries a sample of recent traces in the self-hosted storage and checks each field against the masking policy, looking for unmasked personal data in inputs, outputs, metadata and logs. The review should be repeated after each rule change and on a schedule, because new features and prompts can introduce new fields. Ask the partner who performs the review, which sample size and fields are used, and where the results are recorded as evidence. The output should be a documented finding, not a verbal assurance.

Ownership should be written down before go-live: engineering maintains the rule code, a named data protection or security owner approves changes, and the platform team deploys them through the same review path as other infrastructure changes. Changes to masking should be logged, and the partner should explain how a rule change is rolled out without leaving a window where unmasked data is stored. Ask whether the partner provides ongoing rule maintenance under a retainer or hands over the rules and test suite for your team to run.

Masking adds design, rule-writing, testing and review effort on top of the base deployment, and the size depends on how many applications, data categories and languages are in scope. The main cost drivers are the number of instrumented applications, the complexity of agent traces with many tool calls, and whether your team needs enterprise controls. Ask the partner for a quote that itemizes masking design, test creation, the review cadence, handover and any Langfuse license cost, which is priced separately by Langfuse on its self-hosted pricing page. Also ask what recurs, such as rule maintenance, reviews and license renewals.

Model-based scoring uses an evaluator model to grade outputs against a rubric, which is fast and can run on every sampled trace, while human scoring uses trained reviewers who judge quality, tone and policy compliance. Most teams use both: automated scores cover volume, and reviewers calibrate the automated judge on a labeled sample so its scores can be trusted. Ask the partner which rubrics they would start with for your use case and how they would measure agreement between the judge and reviewers.

Failed or low-scoring traces are collected from Langfuse, reviewed, stripped of personal data where needed, and saved as versioned dataset items with the expected behavior written down. Each new production failure becomes a regression case, so the dataset grows with the application. Confirm how the partner handles masking before traces are copied into datasets, because a dataset is a second copy of the prompts and outputs and needs the same access and retention rules as the original traces.

Teams usually score a fixed percentage of live traces, plus every trace that meets a risk rule such as a user complaint, a guardrail trigger or a long tool-call chain. Sampling rates set the evaluator workload and the volume of stored scores, so they should be reviewed as traffic grows. Ask the partner how sampling is configured in the self-hosted deployment, who can change the rate, and how sampled and unsampled traffic is reported so the scores are not mistaken for full coverage.

Good practice names reviewers by role: a product owner accountable for quality targets, a domain expert who can judge correctness, a security or privacy reviewer for data exposure, and an engineer who owns the prompt and model configuration. Named people, with a backup for each, make it clear who signs off a change and who answers for a low score. Ask the partner to help write the reviewer roster, the review frequency and the escalation path, and to confirm that reviewer access to traces is limited to what each role needs.

A written release gate sets the minimum scores on the regression dataset, the maximum acceptable drop from the current production baseline, and the reviewer sign-off required before a prompt, model or retrieval change ships. The gate should be run automatically in the release pipeline and its results stored with the change record. Ask the partner which thresholds they recommend for your risk level, who can override a failed gate, and how an override is recorded.

Evaluation runs, judge outputs and human scores are stored in the same self-hosted Langfuse deployment as the traces, which means their data sits in the organization's own PostgreSQL, ClickHouse and S3-compatible storage. Confirm that evaluator model calls also stay inside the boundary: the partner should explain whether the judge model is hosted on the organization's own servers, such as through vLLM or Ollama, or called through an external API, because an external judge would send prompt and output content outside the infrastructure. Ask who holds the keys and how project roles separate reviewers from engineers.

A team with engineers familiar with the application can run evaluations in-house once the platform, rubrics and gate are defined, and much of the work is ongoing review rather than setup. A partner is more useful for designing the rubrics, calibrating the judge model, building the first regression dataset and writing the release gate. Ask the partner which parts they would hand over, what documentation they would provide, and whether a retainer for ongoing review is needed. The scope and deliverables should be agreed in writing before work starts.