Open source LLM observability

Decide the logging schema before you deploy Langfuse or Phoenix, because retrofitting fields later leaves gaps in the records you may need for incident review or EU AI Act record-keeping. For each request, capture the model identifier and provider version, the prompt template name and version (not only the rendered prompt), the input and output payloads, latency, input and output token counts, the derived cost, a pseudonymous user or tenant reference, evaluation scores, and any human feedback signal. Those fields let you answer which model and prompt produced an output, for which tenant, at what cost, and how it was rated. Confirm in your version of each platform whether it records template versions natively or whether your application must pass them as metadata.

Raw prompts and completions are the largest privacy exposure in an observability store, because users routinely paste customer records, health details or credentials into chat. Store a stable pseudonymous user identifier rather than names or email addresses, and treat the trace store as a system of record with the same access controls, encryption and retention rules as the source application. Where raw content is not needed for a given purpose, log hashes, lengths or classification labels instead. Decide per application whether full payloads are kept, truncated, or dropped, and document that decision as part of your data protection assessment.

Redaction and sampling reduce both risk and storage. Redaction can happen in the application before data leaves it, in a collector or proxy in front of the platform, or inside the platform through configurable masking, and each location has different coverage. Sampling reduces volume by keeping a fraction of traces, but it must not drop every trace for a flagged user, error or high-risk workflow. Ask each vendor and your implementation partner to confirm which masking, filtering and sampling features your chosen release supports, how they behave on nested fields and attachments, and whether redaction is applied before or after data is written to disk.

Storage growth for LLM traces is driven by five factors: request volume, the average size of prompts and completions, retained attachments such as uploaded documents or images, evaluation outputs (scores, judge model responses and annotations), and replication across environments or backups. Of these, token volume per request is the one teams most often underestimate, because agent workflows and retrieval-augmented prompts can carry far more context than the user's question. Before sizing anything, measure real token counts per use case in a pilot, separating interactive chat from batch evaluation runs, since the two produce very different trace profiles. Ask the platform team to confirm how the chosen tool stores full request and response bodies compared with metadata, and whether evaluation results are written to the same database or to a separate store. Those answers, not a generic estimate, should drive the storage budget.

Retention is where observability design meets legal duty. Article 12(1) of the EU AI Act requires high-risk AI systems to technically allow for the automatic recording of events over the system's lifetime, and Article 19(1) requires providers to keep the logs they control for at least six months unless other law provides otherwise. The six-month floor is a minimum, not a target, and it applies to a specific set of logs, so the first decision is to classify which traces fall under that obligation and which are operational debugging data. A practical approach is to keep a compliance-scoped log set with its own retention rule and access restrictions, and to apply shorter, data-minimized retention to full prompt bodies that may contain personal data. Confirm with your legal and compliance team whether your sector imposes longer periods, and check whether the EU AI Act Service Desk's Article 19 guidance matches your classification of the system.

Retention settings should be tested, not only documented. Run a restore and deletion test on a copy of the trace store to confirm that expiry jobs remove full bodies, attachments and evaluation outputs together, and that backups do not silently extend the retention period beyond what the policy allows. Decide who may change retention, who approves exceptions for litigation holds or incident investigations, and how that change is recorded. Ask the implementation partner to size the database, object storage and backup volumes from your pilot measurements, to state which costs are one-time (setup, schema and dashboard work) and which recur (storage growth, backup copies, compute for evaluations), and to show how the retention schema and expiry configuration will be documented as evidence for your assessment.

Start with the license, because it decides what you can run and for whom. Ask the partner to confirm, in writing, which Langfuse or Phoenix version is in scope and which license applies to every component that will be deployed. Langfuse is MIT outside its ee directories, while Phoenix is under the Elastic License 2.0, which prohibits offering the software to third parties as a hosted or managed service. If your observability platform will serve a customer-facing product or a group subsidiary, ask for a written reading of that case before the design is fixed. Also ask how the partner will track version changes after handover, since the latest releases move quickly and a license review is only valid for the version reviewed.

Next, ask how traces will be classified and redacted before they reach storage. Prompts, retrieved documents and model outputs often contain personal data, so the partner should show which fields are captured by default, which are masked or dropped, and where that logic lives, whether in the SDK instrumentation, a collector or the platform itself. Ask who approves the data classification and who tests redaction with realistic but synthetic samples. Then ask for the retention design in writing. Article 19(1) of the EU AI Act requires providers of high-risk systems to keep automatically generated logs for at least six months unless other law provides otherwise, and Article 12(1) requires the logging capability to cover the system's lifetime. Your legal team should confirm which obligations apply to your system, and the partner should map each retention period to a specific store, a deletion job and an owner.

Finally, ask about operations and money. Request a backup and restore test with documented recovery results, not only a backup schedule, and ask whether the test covers the ClickHouse and object storage tiers. Ask how upgrades are handled, including who tests a new release against your instrumentation and who decides when to move. Ask who holds administrative credentials, API keys and database secrets at handover, and whether those are stored in your vault. On cost, ask which items are one-time, such as design, licensing review and initial deployment, and which recur, such as hosting, storage growth, support and upgrades. Require that every deliverable is listed with acceptance criteria in the fixed-scope document. DeployOpen provides quotes that follow from that written scope, so the scope should be complete before a price is discussed.