How Langfuse data masking keeps personal data out of traces
Langfuse data masking changes trace content before it is stored, so personal or sensitive data in prompts and model outputs does not end up in your self-hosted Langfuse database. Teams need it when user messages, retrieved documents or model responses may contain names, contact details, account numbers or health information, and when they cannot rely on every application team to keep that data out of its traces by hand. Masking can run in the SDK, on the server, or in both places. Where you apply it depends on whether the sensitive data may leave the application at all, so settle that question with your privacy and security leads before you configure anything.
Client-side masking runs in the Langfuse SDK before data is sent to the server. Because the sensitive value is removed inside the application process, it never crosses the network to Langfuse. This is the only layer that meets a requirement that sensitive data must never leave the application. The trade-off is that every SDK integration must be configured and kept current, so a team that ships a new service without masking will send raw content to Langfuse. Confirm which masking hook your SDK version exposes for your language, and test it with a dummy identifier in a sample prompt before you rely on it in production.
Server-side ingestion masking is an Enterprise feature that needs a license key. Langfuse sends each OpenTelemetry trace object to an HTTP callback endpoint that you provide before the trace is stored in ClickHouse, and you turn it on by setting LANGFUSE_INGESTION_MASKING_CALLBACK_URL. The default callback timeout is 500 ms with one retry. Langfuse describes this layer as a centralized safety net that complements client-side masking, so it catches gaps in SDK configuration across teams. Ask the team that operates the callback for its latency and availability targets, because a slow endpoint will trigger the timeout even when the service is healthy.
The failure behavior needs the most attention. By default the callback fails open, so if it fails or times out, events are processed and stored unmasked. Setting LANGFUSE_INGESTION_MASKING_CALLBACK_FAIL_CLOSED=true drops those events instead. That protects personal data but means traces go missing during an outage of the masking service. Decide which loss is acceptable for each environment, and set an alert on callback failures so that a fail-open event does not pass unnoticed. Masking is one control among retention, access control and encryption, and it works alongside those controls rather than standing in for them.
Langfuse data masking options and when each fits
This table compares where client-side SDK masking and Enterprise server-side ingestion masking run, what each protects and when each fits. Read each row across both columns to see which layer handles data at each point in its path. The main takeaway is that server-side masking acts after data has already left the application, so it works best alongside client-side masking.
| Aspect | Client-side SDK masking | Server-side ingestion masking |
|---|---|---|
| Where masking runs | Inside the application, in the Langfuse SDK, before data is sent | In the Langfuse ingestion path, which sends each OpenTelemetry trace object to an HTTP callback before ClickHouse storage |
| License needed | No Enterprise license key is listed for client-side masking in Langfuse's license key documentation | Enterprise license key, because server-side data masking is listed as an Enterprise feature |
| Data protected before it leaves the application | Sensitive values are removed or replaced in the application before network transmission to Langfuse | None before transmission. Data reaches Langfuse first, then the callback masks it before storage |
| Failure behavior | Confirm with your SDK version how masking errors are handled before data is sent | Fails open by default, processing events unmasked if the callback fails or times out. Set fail-closed to drop them instead |
| Timeout and retry | Set in your application code and SDK configuration | Default callback timeout of 500 ms with 1 retry, configured through the callback URL setting |
| Typical reason to choose it | When sensitive data must never leave the application or its network boundary | When one masking policy must apply centrally to traces from many applications, teams or SDKs |
| Combining both | Masks at the source so sensitive values are not sent in the first place | Acts as the centralized safety net Langfuse describes, so keep client-side masking in place where data must never leave the application |
How Langfuse data masking reaches traces before storage
Langfuse data masking can act at two points in the trace pipeline, and each point sees the data at a different moment. The first is inside the application, in the Langfuse SDK, before any trace data is sent over the network. The second is inside the Langfuse server, after the trace reaches Langfuse but before it is written to storage. The application team owns the first point and the platform team usually owns the second. Decide early which fields must be masked at each point, because the answer determines where the masking logic lives and who maintains it.
Client-side masking runs in your application code. Values are transformed before the SDK sends them, so personal or sensitive data never reaches your Langfuse deployment. This is the only point that guarantees data stays out of the Langfuse network path entirely. Confirm with your SDK version how the masking function is registered and whether it applies to every instrumented path, including traces that arrive through OpenTelemetry instrumentation from frameworks and libraries your teams did not write. Test with synthetic personal data and check the stored trace in ClickHouse.
Server-side ingestion masking is an Enterprise feature. When it is enabled through LANGFUSE_INGESTION_MASKING_CALLBACK_URL, Langfuse sends each OpenTelemetry trace object to an HTTP callback that you host, and stores the object only after the callback responds. The callback receives the trace object and returns a version with sensitive content transformed. Confirm the exact request and response format in the Langfuse data masking documentation before writing the callback, because the contract defines what your service must parse and return. The default callback timeout is 500 ms with one retry, so the service must answer within that window, including any calls it makes to other systems.
Because the callback sits in the ingestion path, its failure behavior determines what gets stored. By default the masking callback fails open: if it fails or times out, Langfuse processes events unmasked. Setting LANGFUSE_INGESTION_MASKING_CALLBACK_FAIL_CLOSED=true drops those events instead. Choosing between these settings is a governance decision, weighing the risk of storing unmasked personal data against the loss of trace data. Record the decision, the owner and the alerting you will use to detect callback failures, since a fail-open setting that goes unnoticed can store unmasked data for a long period.
Langfuse describes server-side masking as a centralized safety net, not a replacement for client-side masking when sensitive data must never leave the application. Treat the two layers as complementary. For fields that must never leave your environment, mask in the SDK and use the server-side callback as a second check for data that teams did not anticipate. Before you design either layer, inventory the prompt and output fields that carry personal data, assign an owner for each masking rule, and confirm with Langfuse that your license key includes server-side masking.
Choosing fail-open or fail-closed masking for your traces
Server-side ingestion masking runs an HTTP callback before trace data reaches ClickHouse, and the failure mode decides what happens when that callback is slow or unavailable. The cards below set out the data protection trade-off, the effect on trace completeness, the questions to settle with your data owner and security team, and the load test to run before you commit.
Default fail-open behavior
By default, the masking callback fails open. If the callback fails or exceeds its timeout, Langfuse processes the events unmasked and stores them. The default callback timeout is 500 ms with one retry, so a slow masking service can expose personal data in traces without any visible error. Confirm this default in your Langfuse version before you assume it is safe for your data classes.
Fail-closed event dropping
Setting LANGFUSE_INGESTION_MASKING_CALLBACK_FAIL_CLOSED=true drops events when the callback fails or times out, so unmasked data is never written to ClickHouse. The protection is stronger, but every affected event is lost from the trace store. Set this variable in the Langfuse Web and Worker configuration you control, and test that the change takes effect in your deployment.
Effect on trace completeness
Under fail-closed, traces with gaps appear during callback outages, and a dropped event may be the one that explains a production incident. Engineers debugging a failed chain may see a partial trace with no obvious reason for the missing spans. Decide how your team will detect dropped events, such as monitoring ingestion counts or the callback's error rate, before a failure occurs.
Callback availability and latency
The masking callback is in the ingestion path, so its availability and response time directly control which mode produces losses or exposure. Load test the callback at your expected peak ingestion rate, using trace payloads that match your largest prompts and outputs. Record p95 latency, error rates and timeouts against the 500 ms default, and compare the results with your Langfuse scaling plan.
Data owner and security decisions
Settle with the data owner which data classes must never reach ClickHouse, and whether a short window of unmasked storage is acceptable under your retention and privacy commitments. Ask the security team whether the callback service is in the same trust boundary as Langfuse, who can change the timeout and failure settings, and how a change would be approved and logged.
Client-side masking as a backstop
Langfuse describes server-side masking as a centralized safety net, not a replacement for client-side masking when sensitive data must never leave the application. If data must never leave the application, mask it in the SDK before sending, and treat the server callback as a second layer. Confirm which fields the SDK masks and which the callback sees before you choose the failure mode.
Setting up Langfuse data masking step by step
This sequence moves sensitive data out of Langfuse traces in layers: a server-side safety net on the self-hosted instance, plus SDK masking where data must never leave the application. Each step ends with a check so the team can show that masking works and that failures behave as chosen.
Confirm the Enterprise license key is installed
Server-side ingestion masking is an Enterprise feature, so the platform team confirms that a valid license key is loaded into the self-hosted Langfuse instance before any callback work starts. The Langfuse license key documentation lists server-side data masking among the features that need the key. Check the license status in your deployment and record which environment and instance it covers. This step is done when the instance reports its Enterprise features as active in the configuration you will use for production.
Define sensitive fields and patterns
Security, privacy and the application owners agree on what counts as sensitive in traces. Typical candidates include names, email addresses, account or national identifiers, health or payment details, and free-text fields where users paste personal content. Map each category to the trace locations where it appears, including prompt inputs, model outputs, metadata and tool arguments. Keep the result as a written rule inventory with an owner and a change process. This step is done when the data owner has signed off on the inventory for every application that sends traces.
Build the masking callback service
The engineering team writes an HTTP service that receives each OpenTelemetry trace object from Langfuse, applies the agreed rules and returns the masked object. Confirm the request and response format in the Langfuse data masking documentation before coding, because the service must match it exactly. Keep the logic deterministic, log rule identifiers and counts rather than raw values, and keep the service stateless so retry behavior stays predictable. This step is done when the service passes tests using sample traces that contain each sensitive pattern from the inventory.
Deploy the callback inside your infrastructure
Run the service inside the same network boundary as Langfuse Web and Langfuse Worker, behind internal TLS and without a route to the public internet. Confirm that network policy allows the Langfuse containers to reach the service, and give the service its own logging and access controls. Release it on a track separate from Langfuse so that rule changes do not require a Langfuse upgrade. This step is done when a health check from the Langfuse network succeeds and the service logs reach your central log store.
Set the callback URL and timeouts
Set LANGFUSE_INGESTION_MASKING_CALLBACK_URL to the internal address of the service in the Langfuse environment, then restart the Langfuse containers using your deployment method. Langfuse uses a default callback timeout of 500 ms with 1 retry. Measure how long your rules take under realistic ingestion load, and confirm in the Langfuse data masking documentation whether the timeout can be changed before you adjust it. This step is done when the instance reads the variable and a test trace reaches the callback, visible in the service logs.
Choose fail-open or fail-closed behavior
By default, the masking callback fails open, so events are stored unmasked if the callback fails or times out. Setting LANGFUSE_INGESTION_MASKING_CALLBACK_FAIL_CLOSED to true drops those events instead. Choose based on your data obligations: if storing unmasked traces would breach your policy, fail-closed is the safer setting, and the team must accept that affected traces are lost. Record the decision, the reason and the approver in the runbook. This step is done when the chosen value is set in the environment and documented.
Add SDK-side masking for data that must never leave
Server-side masking runs after data has already left the application, so Langfuse describes it as a centralized safety net rather than a replacement for client-side masking. Where data must never reach Langfuse, implement masking inside the SDK so values are redacted before the trace is sent. Reuse the rules from the inventory and test them against the same sample set. This step is done when traces from the application show redacted values at the SDK output and captured network traffic in the test environment contains no sensitive values.
Verify masking and failure behavior
Send test traces containing each sensitive pattern and confirm in the Langfuse interface and database that stored values are masked. Then make the callback fail on purpose by stopping the service or blocking its route. Confirm the outcome matches your setting: unmasked storage under fail-open, or dropped events under fail-closed. Restore the service and retest normal masking. This step is done when the results are saved as evidence and the runbook names the owner for future rule and configuration changes.
Testing Langfuse data masking before production traffic
Test masking in a non-production Langfuse project before any production trace reaches the instance. Create the test project with its own API keys so that synthetic data never mixes with live traces, and so that failure experiments cannot affect real users. Build a set of synthetic payloads that contain personal data in every field your application sends: names, email addresses, phone numbers, account numbers and free-text notes inside prompts and outputs. Illustrative example: a payload with a fictional customer name, a made-up email address and an invented account number in the user message, repeated in the model output and in trace metadata. Send the same set through two paths: the client-side SDK with its masking function configured, and the server-side ingestion path with the masking callback enabled, using OpenTelemetry traces sent to the /api/public/otel endpoint.
Verification happens in two places, because the Langfuse UI and the database can show different things. First, open each test trace in the Langfuse UI and check input, output, metadata, tags and span names for any synthetic value that should have been masked. Second, query ClickHouse directly for the same trace identifier and inspect the stored fields. Ask your platform team for read-only ClickHouse access in the test environment only. Record every field that holds a synthetic identifier, then compare the results with your masking rules. Confirm that the callback receives each trace object and that prompts and outputs are both covered. Keep the query text and the results in the test record, because auditors and security reviewers will ask how you confirmed the outcome.
Failure tests show what happens when the masking layer is unavailable, which matters more than the happy path. Stop the callback service while traffic continues and observe the result with the default settings. Under the default fail-open behavior, the callback has a timeout of 500 ms with one retry, and events that cannot be masked are processed unmasked. Then set LANGFUSE_INGESTION_MASKING_CALLBACK_FAIL_CLOSED=true, repeat the test, and confirm that events are dropped and that the sending application and your alerting both show the drop. Also slow the callback beyond the timeout to simulate latency. Measure how long the ingestion path waits for each event and whether the sending application blocks or queues work. Confirm these settings in your own environment, because behavior can differ by version and deployment.
The test results should drive a written decision, not a default setting left in place. Fail-open keeps traces flowing during an outage but can store personal data unmasked while the callback is down. Fail-closed protects stored data but creates gaps in observability for the period of the outage. Involve your privacy, security and application owners before production, and record which behavior you chose and why. Server-side masking requires an Enterprise license key, so confirm licensing before you run these tests. Client-side masking should still cover sensitive values before they leave the application. Repeat the full test set after each Langfuse upgrade, after each change to the callback and after any change to timeout or retry settings.
Security and operations to plan around masking
Server-side masking adds a component to your Langfuse estate and changes how failures show up in your data. The cards below cover the controls the callback needs, who owns the rules, how failures are monitored, and how masking interacts with data that is already stored.
Treat the callback as a data-handling service
The masking callback receives full OpenTelemetry trace objects, so it sees the same prompts and outputs that Langfuse stores. Give it the access controls, network path, logging discipline and patch cadence you apply to Langfuse Web and Worker. Confirm that its request and error logs do not echo the trace payload, and that only the Langfuse ingestion path can reach its URL.
Restrict network path to the callback
Langfuse calls the callback URL configured in LANGFUSE_INGESTION_MASKING_CALLBACK_URL from its own network. Decide whether the callback runs inside the same cluster or VPC, and whether it needs outbound access at all. Use network policies or security groups so that only the Langfuse services can open connections to it, and verify that TLS is used on every hop that carries trace content.
Assign ownership of masking rules
Name one owning team for the masking logic and a separate approver for changes, usually from security or privacy with platform engineering as the operator. Document which fields, attributes and patterns are masked, which are deliberately left alone, and why. Without a named owner, rules drift as application teams add new trace attributes.
Review and version masking changes
Store the callback code and its rule set in version control, and require review and a tested deployment for every change. Run the callback against representative sample traces from each project before release, including traces with nested tool calls and multimodal content. Record the approver and release date so that audit evidence can show what was masked at a given point in time.
Monitor fail-open events
By default, events pass through unmasked if the callback fails or times out, so an outage on the callback is a privacy event, not only an availability event. Alert on callback errors, timeouts and retries, and on any increase in ingested events during those windows. Decide with the privacy owner whether LANGFUSE_INGESTION_MASKING_CALLBACK_FAIL_CLOSED=true is acceptable, since it drops events instead of storing them unmasked.
Find and handle unmasked events after an incident
When fail-open processing has stored unmasked content, masking the callback later does not change the stored records. Define a procedure to identify affected time windows and projects from callback logs, decide whether to delete or retain those records, and document the outcome. Confirm with your privacy team how this fits your incident response and notification duties.
Align masking with retention and deletion
Masking applies on ingestion, so it does not clean up what Langfuse already holds. Apply retention windows to the data that remains, and check how deletions reach ClickHouse, object storage and exports. Audit logs and dataset items are not removed by the same cleanup, so confirm what must be deleted for a given data subject request.
Account for backups of stored data
Backups of ClickHouse, PostgreSQL and blob storage can contain traces that were stored before masking was enabled or during a fail-open window. Decide how long backups are kept, who can restore them, and how a restore would re-expose data that a later deletion removed. Record these decisions alongside your backup and restore procedure.
Limits of Langfuse data masking to confirm in your environment
The first thing to confirm is whether your deployment is licensed for server-side masking. The ingestion masking callback is an Enterprise feature that requires a Langfuse license key. A team that sets LANGFUSE_INGESTION_MASKING_CALLBACK_URL on an unlicensed instance should not assume that its traces are being filtered. Check the license status of your instance before the callback goes into production, and confirm with Langfuse that the key covers your deployment and the features you plan to use. Record the answer in the masking design document, together with the name of the person responsible for renewal.
Next, decide what happens to data stored before masking was enabled. Masking applies to events as they are ingested, so traces written earlier stay in ClickHouse in their original form unless you remove them. Review existing projects for personal or sensitive content, and choose between deleting that history through retention policies, restricting access to it, or accepting it with a documented rationale. Also check whether exports or backups created before the change contain the same content, because those copies sit outside the masking path. Ask Langfuse whether your version offers any reprocessing option, and do not plan around one until you have confirmed it.
Then confirm the Langfuse version and the callback contract your environment uses. The callback receives each OpenTelemetry trace object over HTTP and returns a response that Langfuse applies before storage. The exact request and response format, the field names and the expected handling of status codes depend on the version you run. Check the current Langfuse data masking documentation for the precise format, and build your callback against that version. Test the callback again after every upgrade, particularly across major versions, because major upgrades change the data model. Compare your callback's own response time with the default 500 ms timeout and single retry, and set your service levels to match.
Confirm that SDK masking covers every client that sends traces. Client-side masking runs inside each SDK, so it protects only the clients that use it. Inventory every application, language and framework in use, including Python and JavaScript or TypeScript services, OpenTelemetry-instrumented frameworks sending to the /api/public/otel endpoint, direct API calls and internal scripts. Any client that bypasses the SDK relies entirely on the server-side callback. For each one, verify that masking runs before data is sent and that the SDK version in use supports the masking option you selected.
Masking does not replace client-side controls where data must never leave the application. The server-side callback acts on traces after they have been transmitted to Langfuse, so the data has already crossed your network boundary. Where a regulation or contract forbids transmission, remove or tokenize the data at its source. The callback also fails open by default, so events are stored unmasked if it fails or times out. Setting LANGFUSE_INGESTION_MASKING_CALLBACK_FAIL_CLOSED=true drops those events instead. Decide with the data owner which behavior is acceptable, monitor callback failures, and accept that dropped events mean lost traces.
Questions buyers ask about Langfuse data masking
Is client-side masking in the Langfuse SDK free in the open source version?
Yes. Langfuse states that core features and APIs are available in the MIT-licensed open source version without limits, and it lists server-side data masking among the features that need an Enterprise license key. That leaves SDK-side masking, which runs in your application before data is sent, inside the open source offering. The cost is engineering time: someone must define which fields to redact in each instrumented service, write tests that confirm the redaction, and update those rules as prompts and tool outputs change. Confirm the exact SDK functions and version against the Langfuse data masking documentation before you plan the work, because SDK interfaces change between releases.
What does Langfuse server-side masking require in license and engineering?
It requires an Enterprise license key and a callback service that you build and host. License: server-side ingestion masking is one of the features that need an Enterprise license key, so without that key your SDK masking is the only control before storage. Engineering: Langfuse sends each OpenTelemetry trace object to an HTTP callback before it is written to ClickHouse. Your team builds that service, decides how it detects and redacts personal data, deploys it in your infrastructure, and sets LANGFUSE_INGESTION_MASKING_CALLBACK_URL. The default callback timeout is 500 ms with one retry, so the service must respond quickly under peak ingestion. Plan for its logging, scaling, access control and security review.
What drives the cost and timeline of a Langfuse masking project?
The main drivers are the number of applications and SDK code paths that send traces, the categories of sensitive data in them, and whether you need client-side masking, server-side masking or both. Prompts, completions, tool arguments, user identifiers and uploaded documents each need different rules. Free text is the hardest category because detection is imperfect and must be tested against samples from your own data. Other drivers include whether an Enterprise license is already in place, how many teams must review the rules, and how many sign-off steps your governance process requires. No duration is fixed in advance. Ask for a written scope that lists each application, data category, test and approval step, and mark which items are one-time and which need ongoing maintenance.
What does a masking engagement cost, and what should a quote include?
DeployOpen does not publish a price list. Most engagements are fixed-scope, so a quote follows from the scope agreed in writing before work starts. Expect four cost components: the Langfuse license, if you need server-side masking, which Langfuse prices through its custom Self-Hosted Enterprise plan; the infrastructure that runs the callback service and its logging; DeployOpen's fee for design, build, testing and documentation; and an optional retainer for ongoing rule updates and upgrades. Ask any provider for a quote that separates one-time work from recurring work, lists each application in scope, states the test corpus and acceptance criteria, and names who owns the rules after handover.
What do our security and data owners need to provide for masking?
They must classify your data. They decide which fields in prompts, outputs, metadata and tool calls are personal, which are confidential, and which may remain in traces for debugging. They also decide the fail-open or fail-closed behavior, approve the redaction rules, and sign off on test results from representative samples. Expect them to review the callback's code, hosting and access controls, and to confirm retention and access for the masked traces. They will also need to name an owner for rule changes. Prepare a data inventory of your LLM applications, a sample of production traces with sensitive values handled under your existing controls, and the privacy, retention and security policies that apply to them.
Can our internal platform team set up Langfuse masking without outside help?
Yes, if the team has the skills and time. Client-side masking uses SDK code that your application developers already maintain, so it can sit in an internal backlog. Server-side masking adds a callback service with its own deployment, security review, scaling and on-call duties, plus an Enterprise license. Internal teams often find two parts hardest: building a test set that proves detection quality, and reaching the fail-open or fail-closed decision with security and legal owners. Consider a fixed-scope engagement if your team has not built a callback service before, if an audit or customer commitment sets a deadline, or if you want independent testing of the redaction. DeployOpen plans, implements, tests and documents this work in infrastructure you control, and the decision on who does the work stays with you.
What happens to traces if the masking callback fails or times out?
By default, Langfuse fails open: if the callback fails or times out, it processes the events unmasked. Setting LANGFUSE_INGESTION_MASKING_CALLBACK_FAIL_CLOSED to true drops those events instead. Fail open keeps tracing available but can store personal data you intended to remove. Fail closed protects that data but loses trace history during callback outages, which affects debugging and any evaluation pipeline that depends on those traces. The default timeout is 500 ms with one retry, so callback latency and availability determine how often this path runs. Decide the setting with security and product owners, document the choice, and test both paths in a non-production environment, including a deliberately slow callback.
Should masking happen in the SDK, on the server, or both?
For data that must never leave the application, use both. Client-side masking removes data before it is sent, so it covers the strongest cases. Langfuse describes server-side masking as a centralized safety net that complements client-side masking and does not replace it. The server-side layer applies one policy to every trace that reaches Langfuse, including services that do not yet use SDK redaction. Its trade-offs are the Enterprise license, a network dependency on your callback, and the fail-open decision. For most regulated teams, the practical question is which data categories must never reach the platform, and whether SDK code in each application can meet that requirement alone.
How DeployOpen scopes and delivers Langfuse data masking
DeployOpen plans Langfuse data masking as two layers: client-side masking in your applications' SDK calls, and optional server-side ingestion masking on Enterprise licensed deployments. Each engagement is fixed-scope, and the fail-open or fail-closed decision is recorded in the written scope before the build starts, so your security and compliance owners can review it.
How we scope
- The written scope names which applications send traces and which fields need client-side masking.
- It states whether server-side ingestion masking is in scope, which requires a Langfuse Enterprise license key.
- Engagements are fixed-scope, and ongoing support is available as a retainer after handover.
How we build
- Client-side masking is configured in each SDK integration before any trace data leaves the application.
- Server-side masking uses an HTTP callback that Langfuse calls with each OpenTelemetry trace object before storage in ClickHouse.
- The callback URL, the 500 ms default timeout and the single retry are reviewed against your expected payload sizes.
How we hand over
- Handover includes the masking rules, the callback code and its deployment, and the environment variable settings.
- A test log shows which sample events were masked, and which were dropped or passed through under fail-open.
- Runbooks cover callback failures, timeout alerts and how to change the fail-open or fail-closed setting safely.
How we support
- Ongoing support can cover callback monitoring, Langfuse upgrades that touch masking behavior, and changes to masking rules.
- Retainers are available for periodic review as SDK versions and Langfuse releases are published.
- Langfuse describes server-side masking as a safety net, so client-side masking stays in place for data that must never leave.
