Langfuse scheduled export to blob storage

When a Langfuse blob storage export fits your retention and analytics needs

Choose scheduled blob storage export when traces, observations and scores must outlive your retention window or must reach an analytics platform on a regular basis. Langfuse copies these records to Amazon S3, S3-compatible storage, Google Cloud Storage or Azure Blob Storage every 20 minutes, hourly, daily or weekly, and the feature is available on self-hosted deployments. This suits two common needs: keeping a longer history of LLM activity than your in-application retention keeps, and feeding the same data into warehouse, lakehouse or notebook tooling without running analytical queries against your production ClickHouse.

The export works as an archive and analytics feed. It does not serve as a restore path. If a ClickHouse table is damaged, or a project is deleted by mistake, files in your bucket do not rebuild the Langfuse instance. Recovery of the deployment is a separate job, covered by backups of each data store, so the exported files should be planned as a copy of records that your teams can read, query and retain under your own policies.

Before you scope this, confirm two things about your deployment. First, identify the exact Langfuse version you run and read the export documentation for that version, because format options and behavior can change between releases. Second, confirm with Langfuse which features your license covers. Data retention policies on self-hosted deployments require an Enterprise license key, and the retention documentation points teams that need data beyond the window to blob storage export. Get the answer in writing before you commit to a design that depends on either feature.

Your ClickHouse version also shapes the output. Parquet is the default format, but the documentation says self-hosted ClickHouse older than 25.11 should be upgraded, or you should choose CSV, JSON or JSONL, because incomplete Parquet output may not be reported. Langfuse v4 requires ClickHouse 25.12 or later, so your upgrade plan may already settle this question. If the upgrade is not scheduled, decide on the text format before the first export runs and confirm that your analytics tools read it.

A useful way to scope the work is to write down three things: the retention or analytics goal, the consumer of the files, and the failure you are protecting against. If the goal is retention beyond the in-application window, the bucket becomes part of your retention design and needs its own lifecycle, encryption and access rules. If the goal is analytics, a recurring export at a suitable interval may be enough. Either way, run one export into a non-production bucket and compare row counts and sample records with Langfuse before relying on it.

Export options and when each fits

This table compares the four blob storage destinations, the four schedule intervals and the four export formats, with the fit and main constraint for each. Read the destination and format rows first, because the Azure key scope and the ClickHouse 25.11 threshold can narrow your options before you choose an interval. The main takeaway is that your destination and ClickHouse version decide what is possible, and the interval sets how current each archive is.

Option	Fit	Main constraint
Amazon S3	Long-term archive for teams already running workloads in AWS	Confirm the credential method and bucket settings in the export documentation before building.
S3-compatible storage	Archive to the object store in your own infrastructure	Confirm endpoint and credential support with the export docs and your storage vendor, then test a write.
Google Cloud Storage	Archive in GCP where the host can use Application Default Credentials	Authentication relies on Application Default Credentials, so plan the identity the host runs under.
Azure Blob Storage	Archive in Azure when your security team accepts a storage account key	The account key cannot be scoped to a container or prefix, so it grants broader access than the export needs.
Every 20 minutes	Frequent loads for analytics pipelines that need near-current data	Shortest interval, so confirm the added load on ClickHouse and workers in your sizing.
Hourly or daily	Regular analytics loads or routine archive copies	Confirm downstream job timing and the number of files each run writes to your bucket.
Weekly	Low-change archive checkpoints and cold retention copies	Longer gaps hold more data between exports, so schedule runs before the retention window removes it.
Parquet (default)	Columnar files for analytics engines and long-term archive	Self-hosted ClickHouse older than 25.11 should be upgraded first, since incomplete Parquet output may go unreported.
CSV, JSON or JSONL	Text formats for older ClickHouse versions or tools that read text directly	Choose JSONL for line-by-line processing; confirm your ClickHouse version before choosing a text format.

How the scheduled sync moves data from ClickHouse to your bucket

Each scheduled run of the blob storage export reads traces, observations and scores from the self-hosted ClickHouse store and writes files to the container or bucket you configured. The schedule options Langfuse documents are every 20 minutes, hourly, daily or weekly, and that choice sets how far the archive can trail the live store. Decide first how stale an archive copy may be for your audit or analytics use. A shorter interval produces more, smaller files and more read load on ClickHouse, so test the cadence against your ingestion volume. Both the source store and the target container sit under your governance, which places the copy path inside your data handling scope.

The work splits into one-time setup and recurring runs. Setup covers creating the target container or bucket, granting the export identity write access, choosing the file format and configuring the schedule. Parquet is the default, with CSV, JSON and JSONL also available. Confirm the ClickHouse version before you choose Parquet: the Langfuse docs say self-hosted ClickHouse older than 25.11 should be upgraded or should use a text format, because incomplete Parquet output may not be reported. Each later run repeats the read and write without anyone changing the configuration, so recurring effort shifts to checking run status, storage growth and credential expiry. Credential choices also differ by provider. Self-hosted GCS export can use Application Default Credentials, while the Azure integration relies on a storage account key that cannot be scoped to a container or prefix, so review that exposure with your security team.

The export has to copy a record before the retention cleanup deletes it. Data retention in self-hosted Langfuse is applied per project, and data older than the window is removed by the nightly cleanup, so the schedule must run often enough that each trace is archived while it still exists in ClickHouse. Langfuse points teams that need data beyond the retention window to this export, which makes it an archiving mechanism. Backup and restore are a separate concern and need their own plan. Work backward from your retention window to pick the interval, then confirm in the Langfuse documentation whether a run can capture records that were created and expired between two consecutive runs. If it cannot, the interval must be shorter than the gap that would lose data.

Several behaviors need confirmation before you rely on the archive. Check in the Langfuse documentation whether each run is incremental, meaning it picks up only records added since the previous run, and how it handles records that change after they were exported. Confirm the file naming scheme, since downstream jobs will depend on it, and whether the same record can appear in more than one file, which is the deduplication question. Then run a test export against a non-production project, inspect the files in the bucket, and compare row counts with ClickHouse for a fixed time window. Record the results and repeat the test after any ClickHouse or Langfuse upgrade, because a changed format or schema can alter the output.

Set up a scheduled export in a self-hosted Langfuse

This sequence takes a self-hosted Langfuse from a blob storage target to recurring files of observations and scores that you can verify against ClickHouse. Platform and data engineers complete it together, and each step ends with a check that tells you whether the next step can start. Confirm the UI path and options against the current Langfuse export documentation before you begin.

Confirm ClickHouse version and blob reachability
The platform engineer checks the self-hosted ClickHouse version first. Langfuse's export documentation says ClickHouse older than 25.11 should be upgraded, or the export should use a text format, because incomplete Parquet output may not be reported. Run the version query against the ClickHouse cluster, then confirm that the Langfuse Worker can reach the target blob endpoint over the network path it will use in production. Write a small test object to a scratch prefix from the same host or pod. This step is done when the version and a successful test write are recorded in your change log.
Create the target bucket with a prefix convention
Create the bucket or container that will hold the exports, and agree on a prefix convention before any file is written. A common pattern is one prefix per Langfuse instance, then per project, then per export type, so that retention and access rules can later follow the same path. Enable the storage controls your organization requires, such as versioning, default encryption and a lifecycle rule, and record which of them you applied. This step is done when the bucket exists, the prefix layout is written down, and the test object from the previous step is removed from the production prefix.
Prepare the credential for your cloud
The cloud administrator creates the credential that Langfuse will use to write exports, and the choice depends on your provider. For self-hosted Langfuse writing to Google Cloud Storage, the documentation describes Application Default Credentials. For Azure Blob Storage, the integration uses a storage account key, and that key cannot be scoped to a single container or prefix, so plan the storage account boundary accordingly. For S3 and S3-compatible storage, use a dedicated identity limited to the export prefix. Store the secret in your secrets manager and confirm that the Langfuse Web and Worker containers can read it. This step is done when the identity can write to the test prefix and nothing else.
Configure the export schedule and format
In the project settings, open the blob storage export configuration and select the destination, the credential and the schedule. The documented schedule options are every 20 minutes, hourly, daily or weekly, so choose the interval that matches your retention and analytics needs. Choose the file format: Parquet is the default, and CSV, JSON and JSONL are also offered. If your ClickHouse version is older than 25.11, select a text format. Save the configuration and record each choice with the reason for it. This step is done when the saved configuration shows the destination, schedule and format you approved.
Run a manual or first scheduled export
Start the first export manually if the interface offers that option, or wait for the first scheduled run if it does not. Watch the Langfuse Worker logs and the export status in the project settings so that you can see the run start, process data and finish. A failed run usually shows up here first, with the error tied to the credential, the endpoint or the ClickHouse query. Note the time window that the run covered and the number of files it wrote. This step is done when the run reports completion and the expected files appear under the prefix you created.
Verify file contents and row counts
The data engineer downloads a sample of the exported files and checks their structure against the chosen format. Then run a count query in ClickHouse for the same project and time window that the export covered, and compare it with the row count in the files. Check that scores and observations both appear if your export includes them, and that timestamps and identifiers are present and readable. If the counts differ, do not treat the export as complete. Hold the schedule and investigate the gap, including whether the window was still receiving data when the run started. This step is done when the counts match for the window you tested.
Check permissions and recurring runs
The security engineer confirms that the export identity can write only to the intended prefix and that people who should not read exports cannot list or download them. Review the access logs or audit trail that your storage platform provides, and confirm that the credential is covered by your rotation process. After at least one scheduled run completes without manual intervention, check that the next run produced a new set of files and did not duplicate the earlier window. Confirm the upgrade path for ClickHouse and the export settings, since version changes can affect the Parquet behavior described earlier. This step is done when permissions, rotation and recurring runs are all documented.

Choosing Parquet, CSV, JSON or JSONL with your ClickHouse version

Parquet is the default export format for scheduled exports to blob storage, but it is not automatically the right choice for every self-hosted deployment. Langfuse's documentation says that self-hosted ClickHouse older than 25.11 should either be upgraded or be used with a text format, because incomplete Parquet output may not be reported. Before you configure the export, check the ClickHouse version your deployment runs. If it is 25.11 or later, Parquet remains a reasonable default. If it is older, you either plan the ClickHouse upgrade first or choose CSV, JSON or JSONL for the export.

The trade-off comes down to who reads the files. Parquet is a columnar format that analytics engines such as Spark, DuckDB, Trino and most data warehouses can scan efficiently, reading only the columns a query needs and compressing repetitive values well. That makes it the better fit for long-term archives that feed analytics, and for teams that will query observations and scores over many months. CSV, JSON and JSONL are simpler to inspect with standard command-line tools, spreadsheets and ad hoc scripts. JSONL, with one record per line, is usually easier to stream and resume than a single large JSON array, but it is still larger on disk and slower to scan at scale.

The version check should be a written decision, not a default. Ask the team that owns ClickHouse for the exact server version, confirm whether any Langfuse upgrade is already planned, and record which format the export will use and why. If the answer is an upgrade, plan it through the ClickHouse work separately, including backups and a test restore, and keep the export on a text format until the upgrade is verified. Also confirm with the team that will consume the files which formats their tools accept, so that a format choice does not force a second conversion step later.

Whichever format you choose, validate it before relying on the archive. Run one export to each target, then check row counts for a known time window against the number of observations and scores in ClickHouse for the same period. Record the format, the ClickHouse version and the export settings in your runbook, so that a later upgrade or a change of consumer can be assessed against the same evidence. Note that the export is an archive and analytics path, not a restore path for the live Langfuse deployment, so the format decision does not replace the ClickHouse backup plan.

How credentials differ across S3, GCS and Azure exports

Amazon S3 and S3-compatible stores, including self-hosted object gateways, authenticate the export with a principal you create and control. Langfuse writes each scheduled file as that principal, so the identity and the bucket policy together decide what a compromised export credential could reach.

Dedicated identity: Create a principal used only for exports, with write access to the export location and no list or delete rights on unrelated buckets.
Bucket policy: Confirm the policy limits writes to the export prefix and that public access is blocked at both the bucket and account level.
Encryption: Confirm whether the bucket requires server-side encryption and whether the key management service grants the export principal permission to use the key.
Endpoint and addressing: For S3-compatible storage, confirm the endpoint, region and path-style addressing your gateway expects, and test them against the export form.
Key lifecycle: Confirm how access keys are issued, where they are stored and how they are rotated without interrupting scheduled exports.

Run a test export to a non-production bucket and confirm that the principal can write to the export prefix and nowhere else before you enable the schedule.

Security and operational considerations for exported data

Exports contain the same prompts, outputs, metadata and scores as the traces they came from, so the archive inherits the sensitivity and controls of the live project. The cards cover what the bucket holds, how the export identity reaches it, how the archive is encrypted, retained and logged, who can change the schedule, and how failed runs get noticed. Confirm each setting in your own environment before go-live.

Export data sensitivity

Each export contains the same prompts, outputs, metadata and scores as the traces it came from, so the archive is as sensitive as the live project. Masking decisions carry over: data masked by the client SDK before ingestion stays masked in the file, and data that was never masked is archived in full. Server-side ingestion masking is an Enterprise feature, so confirm whether your deployment depends on it before relying on it for exports. Classify the bucket to match the most sensitive project that writes to it.

Least-privilege storage access

Grant the export identity write access only to the destination prefix, and avoid bucket-wide roles. For Google Cloud Storage, self-hosted Langfuse can use Application Default Credentials, so you can attach a workload identity with a narrowly scoped role instead of a long-lived key. The Azure integration authenticates with a storage account key, and that key cannot be scoped to a container or prefix, so it grants access to the whole account. Confirm the scoping options for S3 and S3-compatible stores with your provider, and test that a second prefix is unreachable.

Encryption and retention in the bucket

Encryption at rest is set on the bucket or storage account, so confirm default encryption and key ownership with your storage team. Langfuse retention cleanup acts on data inside Langfuse, so files already written to the bucket remain until bucket rules remove them. Set object lifecycle or retention rules on the archive, and confirm how deletion requests for personal data reach exported files as well as the live database. Document the archive window for each project.

Access logging and audit trail

Enable access logging on the bucket or storage account so reads, writes and deletions of export files are recorded where your security team can search them. Langfuse audit logs, an Enterprise feature, record configuration activity inside the Langfuse instance, so the two sources answer different questions: who changed the schedule, and who accessed the files. Confirm that bucket logs are kept long enough for your review cycle and stored where the export identity cannot alter them.

Export configuration by project role

Creating, editing and disabling a blob storage export is a project-level action, so it follows the role a person holds in that project. Map configuration rights to the people who manage the destination and its credentials, and keep Member and Viewer roles away from destination settings. Project-level role-based access control is an Enterprise feature, so confirm with Langfuse which export settings require a license key before you design the role model. Review role assignments after each team change.

Monitoring export runs for failures

Exports run every 20 minutes, hourly, daily or weekly, so a silent failure can leave a gap that surfaces only during an audit or restore. Check for a new file on each run, compare file counts and sizes with the previous window, and alert when a run produces no output. Format matters here: self-hosted ClickHouse older than 25.11 should be upgraded or the export set to a text format, because incomplete Parquet output may not be reported. Confirm your ClickHouse version before enabling Parquet.

What drives the cost and effort of an export project

The export job itself is rarely the expensive part. The cost comes from four components, and each is paid to a different party. Storage and request charges for the bucket or container land in your cloud account and are billed by the provider. Your ClickHouse cluster absorbs the query load of every export run, so it needs spare CPU and memory during those windows. Engineering time covers creating the storage identities, configuring each project, wiring alerts for failed runs and proving that files arrive. Ongoing review of the delivered files is the fourth component. It is easy to leave unowned, and it is the part a partner can describe most concretely in a written scope.

One-time work includes designing the destination and credential model, configuring the export in each project, building monitoring, running the acceptance test and any backfill of historic data. Recurring costs include storage growth, request charges for each run, ClickHouse capacity, monitoring, file review and the work of keeping the export aligned with ClickHouse and Langfuse upgrades. Several inputs raise these costs. Data volume sets file size and run duration. The interval, which can be every 20 minutes, hourly, daily or weekly, sets how many runs and object writes occur. The format, Parquet by default or CSV, JSON and JSONL, changes file size and how your analytics tools read the output. The number of projects multiplies runs, objects, credentials and review work.

Version choices can change the scope of the project. The Langfuse documentation says self-hosted ClickHouse older than 25.11 should be upgraded or should use a text format, because incomplete Parquet output may not be reported. Treat that as a decision to make before the build starts, not a footnote. If your export coincides with a v3 to v4 upgrade, the v4 upgrade guide requires ClickHouse 25.12 or later, recommends 26.4, and advises about 3x ClickHouse disk headroom when backfilling historic data. Plan that headroom before any backfill runs. Credential design also drives effort: Azure integration uses a storage account key that cannot be scoped to a container or prefix, which adds review and rotation work, while GCS export can use Application Default Credentials. Confirm the identity model for each destination in your own environment.

When you request a quote, ask for a written scope that names the projects in scope, the export interval and format for each, and every destination with its provider, bucket or container, and prefix. Ask how credentials are created, stored and rotated for each destination, and who owns that process after handover. The scope should state the ClickHouse version path, including any upgrade prerequisites and the date ranges covered by any backfill. Ask for monitoring and alert coverage, the file review responsibilities after go-live and what is excluded. The acceptance test should be written into the scope: named files delivered to each destination, a successful read in the analytics tool you will use, and a verified alert for a deliberately failed run. DeployOpen does not publish a price list, so a quote follows from that written scope.

Limits to confirm in your environment

Archive exports are only as reliable as the assumptions behind them. The checks below cover the parts of Langfuse scheduled export that depend on your Langfuse version, your ClickHouse build, the teams that read the files and your license, and each one is best confirmed with a test export in a non-production environment before the schedule is trusted.

Interval and format on your version

Confirm that the export interval you need (every 20 minutes, hourly, daily or weekly) and the format you choose (Parquet, CSV, JSON or JSONL) are offered by the Langfuse version you run. Check the export settings in the release you deploy, then trigger a test export and record the release number you tested. Each major upgrade can change what is available, so retest the schedule after every upgrade and before you change the interval or format in production.

Parquet reporting on ClickHouse

Parquet is the default format, but the Langfuse documentation says self-hosted ClickHouse older than 25.11 should be upgraded or should use a text format, because incomplete Parquet output may not be reported. Check your ClickHouse server version against that threshold before you accept the default. If you must stay on an older server, run a test export for a known time window, compare the row count with a direct ClickHouse query and keep a text format as the fallback.

Contents of exported files

Do not assume that every export contains the same records. Confirm whether each file includes traces, observations and scores, which columns are present, how IDs and timestamps are written, and whether input, output and metadata fields are included. Your analytics schema depends on these details. Open a sample file from a test export, map every field to the consumer that reads it and agree on a schema change process with that team before the schedule goes live.

Failed runs and backfill

Find out what happens when a run fails partway through. Confirm whether the next run resumes from the last successful window, reprocesses the gap or skips it. Also confirm whether the first run exports only new data or includes history, and how long a backfill takes on your data volume. Ask how duplicate records are handled if a window is exported twice, then test a deliberate failure in staging and check the resulting files.

Export after retention cleanup

Retention cleanup deletes data once it passes the retention window you configure. Confirm whether an export that runs before cleanup captures every affected record, and whether an export in progress can miss data that cleanup removes during the run. Test this with a short retention window in a non-production project. If you plan to archive data before it expires, verify the export timing against the cleanup behavior of your version.

Enterprise license requirement

Confirm whether the export feature runs on the MIT-licensed codebase or needs an Enterprise license key in your version. Langfuse states that core features are available in the open source version without limits, while data retention policies and audit logs need a license key. Check the license key documentation and the files covered by the separate enterprise license, and get the answer in writing before you plan the budget and support model.

Questions buyers ask before hiring help with Langfuse exports

Is a scheduled Langfuse export a backup?

No. A scheduled export copies traces, observations and scores to blob storage on a recurring schedule so you can keep them beyond the retention window or analyze them elsewhere. It is an archive for retention and analytics, not a recovery method for a running Langfuse instance. Recovery needs backups of each data store, planned separately. Exports also depend on the instance being healthy at each run, so a failed run leaves a gap that someone must detect. Keep ClickHouse, PostgreSQL and blob storage backups on their own schedule, and confirm with your operations team which recovery point objective applies to trace data before you choose the export frequency.

Which blob storage destination should I choose for Langfuse exports?

Choose the destination that matches where your organization already governs storage, retention and access. Langfuse exports to Amazon S3, S3-compatible storage, Google Cloud Storage or Azure Blob Storage, and each has different credential behavior. For Google Cloud Storage, self-hosted Langfuse can use Application Default Credentials. For Azure, the integration uses a storage account key that cannot be scoped to a single container or prefix, so the key grants access beyond the export location. Confirm with your security team whether an account-wide key is acceptable before choosing Azure. For S3-compatible storage, confirm endpoint and addressing settings with your storage team. Also consider where the archive must reside for legal and residency requirements.

How much does a Langfuse export project cost?

Cost depends on the scope agreed in writing. DeployOpen does not publish prices, so a quote follows from that scope. The main cost components are planning the destination and credentials, configuring the schedule and format, validating that the output loads in your analytics tools, and documenting the run for operations. Costs rise with the number of projects in scope, the volume of observations, the number of destinations, and whether ClickHouse must be upgraded first. Recurring costs include the storage you pay your cloud or data center provider for retained exports, plus any support retainer you choose. Ask for the split between one-time setup and recurring support, and the assumptions behind each line.

How long does it take to set up scheduled exports?

Duration depends mainly on your ClickHouse version, the destination's credentials, and how many projects need exports. No fixed timeline applies to every environment, so confirm the timeline against your written scope. If your self-hosted ClickHouse is older than 25.11, you must either upgrade it or use a text format, and that upgrade is often the longest dependency. Schedule options run from every 20 minutes to weekly, so selecting the frequency is quick. Validating the first runs, checking completeness and setting the storage lifecycle take longer. Ask for the planned sequence, the validation steps, and which items depend on approvals in your cloud account.

Which export format should I pick: Parquet, CSV, JSON or JSONL?

Parquet is the default format and suits analytics engines, but the ClickHouse version matters. The export documentation says self-hosted ClickHouse older than 25.11 should be upgraded or should use a text format, because incomplete Parquet output may not be reported. CSV, JSON and JSONL are text formats that avoid that risk on older versions, although they are usually larger and slower to query. Choose the format your analytics or archive consumers can read, then test a full export of a sample project and load the result. Confirm your ClickHouse version with your database team before you set the format.

Can my team set up Langfuse exports without outside help?

Yes, a team with ClickHouse, cloud storage and Langfuse administration skills can configure an export. Scheduled export to blob storage is available on self-hosted deployments, and the core features are part of the MIT-licensed open source version. The work gets harder when several areas must be handled together: upgrading ClickHouse to 25.11 or later where needed, creating credentials that meet your security policy, setting retention for exported files, and proving the archive is complete. Outside help is most useful for the design decisions, the verification steps and the runbook that auditors or operators will rely on. Check that the owner of the credentials can rotate keys on your schedule.

How does exporting Langfuse data differ from restoring it?

An export copies data out of Langfuse into files in blob storage, while a restore rebuilds a running Langfuse instance from backups of its data stores. Exports run on a schedule from every 20 minutes to weekly and produce files that analytics or archive tools read. A restore brings back the database and storage state the application needs to operate, which exports do not replace. Plan both for different purposes: exports for long-term retention and analysis, and tested backups of each data store for recovery. Confirm recovery time and recovery point objectives with your own operations team, and test a restore before you rely on it.

How DeployOpen works on this

DeployOpen scopes Langfuse scheduled export to blob storage as a fixed-scope written engagement, covering the export target, format, credentials and ClickHouse version before any configuration starts. We build and test the schedule in your infrastructure, then hand over documentation and run ongoing support as an optional retainer.

How we scope

Target choice across S3, S3-compatible storage, GCS or Azure Blob Storage, with the credential model for each
Export format decision: Parquet, CSV, JSON or JSONL, matched to the downstream analytics or archive tools
Schedule interval, retention window and ClickHouse version confirmed in the written scope before work begins

How we build

Configure the export schedule in your Langfuse instance and confirm the first run completes as expected
Check ClickHouse version against the Parquet requirement before enabling Parquet output
Set storage permissions per credential model, noting Azure keys cannot be narrowed to a container or prefix

How we hand over

Written runbook covering schedule settings, credential ownership, format choice and the ClickHouse version it depends on
Test evidence showing sample exports land in the bucket and open correctly in the target tool
Notes on retention overlap, so exported data and in-platform cleanup policies are understood together

How we support

Optional retainer for ongoing support and upgrades, including ClickHouse and Langfuse version changes
Review the export target, format and credentials again when the ClickHouse or Langfuse version changes
Ask for the support boundary, response expectations and named contacts in the written scope