Why an open source data lakehouse for manufacturing starts with the plant floor
An open source data lakehouse for manufacturing works when plant data is ingested, kept on site and combined with ERP records under a clear OT boundary. The usual starting point is a set of historians, MES and quality systems that each hold part of the story, while ERP holds orders, materials and costs. A workable first design uses Apache Iceberg tables on on-premises object storage, a catalog that all engines share, Trino or Spark for joins across sources, and ClickHouse for high-volume time series. Airflow or Dagster schedules the pipelines, Superset or Metabase serves dashboards, and DataHub or OpenMetadata records lineage so a quality figure can be traced back to the sensor and batch that produced it. The platform choices matter less than the boundary decisions around them. The function differs from generic analytics in three ways. First, the data originates on industrial networks that must stay separated from business IT, so the ingestion path from historians and MES into the lakehouse is itself a controlled conduit that needs a documented zone design, a reference architecture reviewed against the ISA/IEC 62443 series, and a decision on who may pull data across it. Second, sensor ingestion is continuous and bursty, and the team must confirm sustained and peak event rates per plant before choosing between batch loads and streaming. Third, quality and traceability records often have to survive for years, which makes retention tiers and immutable snapshots part of the design from day one, not a later clean-up task. Before procurement, ask each plant owner for the systems in scope, their data volumes and freshness needs, and which network segments may send data outward. Confirm with your OT security team whether the ingestion point sits in a demilitarized zone or in a separate data zone, and who approves changes to it. These answers determine the catalog, the engines and the access model, so they should be agreed in writing before any platform is installed.
Which regulations and standards shape the lakehouse design
This table maps the standards and regulations most likely to affect a manufacturing lakehouse to the design decisions they force. Read each row as a question to answer with your compliance and OT teams before choosing ingestion paths, storage zones and retention rules.
| Requirement | What it means for the build | Evidence an assessor will expect |
|---|---|---|
| ISA/IEC 62443 zones and conduits | Group plant networks into zones, route plant data through a documented conduit, and place the lakehouse outside the control zone with a segregated ingestion zone. | Zone and conduit diagram, data flow inventory, documented security level targets per zone |
| ISA-62443-3-2 risk assessment | Assess risk for each data flow from controllers, historians and MES into the lakehouse before design is frozen, including what happens if analytics is compromised. | Risk assessment record, residual risk decisions, sign-off by the asset owner |
| ISA-62443-3-3 system requirements | Set target security levels for the ingestion path and enforce them with authentication, least privilege and logging at each conduit. | Security level targets mapped to controls, configuration baselines, access reviews |
| NIST SP 800-82 Rev. 3 (OT security) | Design for OT performance and reliability: ingestion must not slow historians or controllers, and analytics outages must not affect production control. | Architecture showing one-way or tightly limited data paths, load testing results, failover design |
| NIS2 Annex II (manufacturing scope) | If in scope, the lakehouse and its logs feed incident detection so significant incidents can be reported with an early warning within 24 hours, a notification within 72 hours and a final report within one month. | Incident runbook with these three deadlines, log retention records, named reporting owner |
| Cyber Resilience Act and Machinery Regulation 2023/1230 | Where your products have digital elements or machinery connects to remote devices, keep product telemetry flows separate from machine control and log vulnerability and incident data for product obligations. | Product scope assessment, vulnerability handling records, machine safety assessment for connected functions |
| CMMC levels (32 CFR Part 170) | For defense suppliers, data that holds federal contract information or CUI must sit inside a boundary that meets the level named in the contract. Confirm the current status of Phase II with the contracting office before relying on it, since the Department of War suspended CMMC Phase II requirements on 13 July 2026. | Scoping statement for CUI systems, SSP-style control documentation, contracting officer confirmation of required level |
| 21 CFR Part 11 (regulated production records) | For regulated manufacturing such as pharmaceuticals or medical devices, electronic records need audit trails, access controls and controlled retention so that records can be trusted and reviewed. | Validation documentation, immutable audit trail samples, access control matrix, retention schedule |
Which plant and enterprise systems feed the lakehouse
Sensor and historian data is the highest-frequency family in a plant. Programmable logic controllers, drives, and process instruments produce readings continuously, and historians compress and store them on site. Volume and frequency depend on the number of tags, the sampling rate and how the historian stores exceptions, so ask the plant team for tag counts and sampling intervals before sizing storage. Ingestion usually runs through an edge buffer or an OPC UA connection, or through a scheduled export from the historian, so that a network interruption does not drop readings. The main quality problem is time. Controllers, gateways and historians drift apart, so the same event can carry different timestamps depending on its source. Units also differ between sites and equipment generations, for example pressure in bar at one line and kPa at another, and tag names change during retrofits.
- Edge buffering: A local store-and-forward layer keeps readings during WAN or Ethernet outages and replays them in order when the link returns.
- OPC UA or historian export: Subscriptions or scheduled historian exports move data into the lakehouse, and each choice has different gap-handling behavior.
- Clock drift: Compare device time with a site time source and record the offset per device so analysts can correct it.
- Unit mismatches: Keep the original unit and a canonical unit column, and maintain a tag dictionary that maps each tag to its engineering units.
- Tag lifecycle: Track renamed or replaced tags in metadata so long time series remain continuous across retrofits.
Before committing to a design, confirm with the controls team which historian products are in use, whether OT networks allow outbound data flows, and who approves changes to the tag dictionary.
The manufacturing execution system records what the plant did: work orders, batches, operations, machine states, operator actions and material consumption. Records are event-driven and much lower in volume than sensor streams, but they carry the business context that makes sensor data useful, such as which product ran on which line during which shift. Ingestion is usually a batch extract from the MES database or a scheduled export of transaction tables, sometimes supplemented by change data capture where the vendor supports it. Batch records often arrive after the fact, when an operator closes an order or a batch record is signed off, so the lakehouse must handle corrections and reopened orders without double counting.
- Batch extracts: Scheduled jobs pull completed orders and batch records, with a watermark on the last modified timestamp.
- Change capture: Where available, log-based capture reduces load on the MES database, but confirm support with the MES vendor and the database version.
- Late corrections: Operators can reopen or amend records, so keep the latest version and an append-only history instead of overwriting.
- Identifier alignment: Work order, batch, line and equipment identifiers must match the historian tag context and the ERP order numbers.
- Status semantics: Machine state codes often differ between lines, so document the mapping from local codes to a shared vocabulary.
Confirm the MES product and version, the database access model the MES vendor permits, and whether batch records fall under validated electronic record requirements, which affects how changes are retained.
Quality data includes inspection measurements, laboratory results, nonconformance reports, and test outcomes from coordinate measuring machines, vision systems, and lab information systems. Volume is moderate, but the data is high in value because it links process conditions to product outcomes. The defining problem is timing. A measurement may be taken minutes or days after the production event it describes, and laboratory results can arrive after a batch has moved downstream. Quality sources also use their own sample identifiers, pass or fail codes and specification limits, which change when a product revision is approved. Ingestion commonly uses file drops from inspection stations, database extracts from a lab system, or API calls where the quality system exposes them.
- File drops: Inspection stations that write CSV or XML files need a landing zone with checksums and a parser that rejects malformed files without losing them.
- Lab system extracts: Scheduled reads capture results with their sample time and analysis time, which are not always the same.
- Late-arriving results: Partition by event time and reprocess recent windows so that results arriving after a batch closes update the correct records.
- Specification versions: Store the limits in effect at the time of measurement, because revised tolerances change what counts as a pass.
- Sample linkage: Link each result to the batch, serial or lot it measured, and record any result that cannot be linked for review.
Ask the quality team how specification limits are versioned today and whether any results are used for release decisions, since that determines the audit trail the lakehouse must keep.
ERP data provides the enterprise reference points: material and product masters, bills of materials, cost centers, purchase orders, goods movements, sales orders and financial postings. Master data changes slowly but matters in every join, while transaction data grows steadily with orders and goods movements. The main ingestion options are batch extracts through the ERP's database views or standard interfaces, and in some cases change-based feeds. Extracts often need to respect the ERP's own licensing and interface rules, so confirm them before building. The quality problems are different from the plant side. Master data may contain duplicate materials after mergers, inconsistent unit of measure conversions, and effective-dated changes that must be applied to historical transactions. Transaction timing can also differ from plant timing, since a goods receipt may be posted hours after the physical move.
- Batch extracts: Scheduled extracts from views or interfaces, with full refreshes for masters and incremental loads for transactions.
- Effective-dated masters: Keep history for materials, routings and cost centers so that a historical report uses the definition in force at the time.
- Unit of measure conversions: Store conversion factors as governed reference data and apply them in one place.
- Posting delays: Compare physical and posted timestamps so plant output and ERP receipts can be reconciled.
- Duplicate entities: Maintain a cross-reference table that maps legacy and current identifiers after acquisitions or system changes.
Confirm with the ERP owner which interfaces are supported for analytics extracts, how the vendor's license treats data egress, and who owns master data decisions.
Which open source platforms fit a manufacturing lakehouse and why
A manufacturing lakehouse joins sensor time series with MES, quality and ERP records, so the table format, query engines, storage and governance have to work together on plant-controlled infrastructure. The platforms below cover one layer each, and the limits matter as much as the fit when you decide which components to standardize.
Apache Iceberg table format
Iceberg is an Apache-2.0 table format that lets Spark, Trino and Flink work on the same tables at the same time. For manufacturing, its time travel and rollback to an exact snapshot let you reproduce the data behind a released batch record, which matters when a quality investigation asks what the table looked like on a given date. Check how your MES and historian exports land in Iceberg, how late-arriving sensor data is handled, and which team owns snapshot expiry so that historical batch evidence is not removed too early.
Trino query engine
Trino's Iceberg connector reads tables through Hive metastore, AWS Glue, JDBC, REST and Nessie catalogs, selected with iceberg.catalog.type. It suits federated analysis that joins lakehouse tables with ERP or quality data where the source systems can be reached. Confirm which catalog your plant network can run without cloud identity, and test whether the joins you need to answer quality questions perform acceptably on your data volumes before committing to Trino as the main query layer.
ClickHouse for time series
ClickHouse server is Apache-2.0 and is built for high-volume analytical scans, which fits sensor and machine telemetry that is queried by time window and asset. Its ReplicatedMergeTree tables need ClickHouse Keeper with at least three nodes in production, and its BACKUP and RESTORE features do not replace restore tests. Choose between the stable line and the LTS line based on your upgrade cadence, and confirm that the version you deploy is inside the supported window before plant data depends on it.
Apache Airflow or Dagster
Airflow 3.3.2 and Dagster 1.13.26 are Apache-2.0 orchestrators that schedule ingestion from plant historians, MES extracts and ERP feeds, then run quality checks and table maintenance jobs. Airflow has a long track record for batch DAGs, while Dagster models data assets and their lineage natively, which can simplify audits of derived tables. Confirm connector coverage for your historians and PLC gateways, and decide who owns on-call for failed loads at night and across shifts.
Apache Superset or Metabase
Superset 6.1.0 is Apache-2.0 and provides plant and line dashboards on top of Trino or ClickHouse. Metabase is AGPL outside its enterprise directory, and that directory uses the Metabase Commercial License, so features you need, such as certain SSO or embedding options, may require a commercial license. Check which features your plant dashboards require, how row-level access is enforced for supplier or site users, and whether dashboards will be reachable only from the plant network.
DataHub or OpenMetadata lineage
DataHub and OpenMetadata are Apache-2.0 catalogs that record ownership, schemas and lineage from sensor feeds through to quality reports. Lineage is what lets a quality engineer trace a KPI back to the historian tag and the transformation that produced it. Both need connectors built for your source systems, and neither automatically documents custom PLC or MES mappings. Confirm which connectors your MES and historian need and who maintains the tag-to-asset mapping.
Integration points and deployment pattern for plant networks
A plant lakehouse only works if data crosses the plant boundary in a controlled way and survives outages on either side. The cards below cover the components that connect shop-floor sources to a central platform, from local buffering to backups, and the checks to run before any of them go live.
Edge buffering at each plant
Sensor gateways and MES exports should write to a local store that keeps accepting data when the WAN link or the central cluster is down, then forwards it in order once connectivity returns. For manufacturing, the risk is not only lost data but duplicated or out-of-order readings that distort OEE and quality trends. Ask the implementation partner how buffers are sized for the longest outage you plan to tolerate, how each record is deduplicated on replay, and how buffer fill levels are alerted on. Confirm the buffer capacity with your plant engineers before sizing the store.
Segmented DMZ transfer point
Data should leave the operational network through a single, documented transfer zone between the plant control networks and the enterprise network, rather than through direct connections from historians or PLC-adjacent systems to the lakehouse. The transfer point terminates connections in both directions, so the plant never accepts inbound traffic from the enterprise. For ISA/IEC 62443 programs, the zone model and conduit design should come from the asset owner's risk assessment. Check which protocols cross the boundary, who approves new flows and how they are logged.
Retention tiers on on-premises object storage
Keep recent sensor data on fast, hot storage for troubleshooting and short-term analytics, and move older raw telemetry and batch records to colder tiers with longer retention. Quality and traceability records often need longer retention than raw sensor streams, so tier rules should follow the record type, not only its age. Ask how Iceberg snapshot expiry and file compaction are scheduled per tier, and whether deletes for expired data are verified. Set tier boundaries per record class before ingestion starts.
Catalog and access tied to plant directory identity
The catalog and query engines should authenticate users and service accounts against the plant or enterprise directory, so that access to a line's data can be granted and removed in one place. Trino and ClickHouse both support external authentication, and ClickHouse documents LDAP, Kerberos, X.509 and JWT options. Map directory groups to table and schema privileges, and test how removing a user from a group affects active sessions and cached credentials. Confirm which directory your plants use before choosing the authenticator.
Replicas and Keeper for availability
ReplicatedMergeTree tables are coordinated by ClickHouse Keeper, which replaces ZooKeeper, and the replication guide recommends at least three Keeper nodes on dedicated hosts in production. A recovered replica syncs missing data from a healthy one without manual steps. Replication keeps the service available when a node fails, but it copies bad writes as well, so it is an availability control, not a recovery control. Ask which hosts run Keeper, how they are placed across failure domains and how Keeper quorum loss is alerted on.
Backups to S3-compatible storage
Use ClickHouse BACKUP and RESTORE to write full and incremental backups to S3-compatible storage that sits outside the failure domain of the cluster, ideally in a separate site or on separate media. Replication alone does not protect against human error, a bad migration or corrupted data, because those changes replicate to every copy. Schedule regular restore tests on a spare cluster, since a backup that has never been restored will probably not restore when needed. Ask for the restore runbook, the measured restore time and the retention period for backups.
How a manufacturing lakehouse rollout runs in phases
The sequence moves from agreed sources and owners to a tested, handed-over lakehouse, with each phase closing on a written output so OT, quality and IT sign off before data moves on.
Scope sources and use cases with owners
DeployOpen works with plant IT, OT engineering, quality and the ERP and MES owners to list the sources in scope, such as historian and sensor time series, MES batch records, quality results and ERP master and transactional data. Each use case is written with its owner, its freshness need and the question it answers. The step closes with a signed source inventory that names the system owner, the data owner and the network segment each source lives on. Confirm with the OT team which historians and controllers may be read from and by which route.
Design zones and data flows with security review
The team maps where data is collected, staged, stored and queried, and which zones sit inside the plant network and which sit in the company data center or cloud account. Flows crossing from OT to IT are defined as one-way or brokered paths with documented ports, identities and owners. The design is reviewed against the ISA/IEC 62443 series, including the zone and conduit concepts in ISA-62443-3-2, and NIST SP 800-82 Rev. 3 for OT-specific risks. The output is an approved zone diagram and data-flow register that the security team has accepted in writing.
Run a pilot on one line or plant
A pilot runs on one production line or one plant using representative data, including normal load, peak batch periods and known quality events, so that ingestion rates, schema drift and query patterns are measured rather than assumed. Open table formats such as Apache Iceberg are tested with the engines the team intends to use, such as Trino or Spark, and the results are compared with the business questions from the first step. The pilot closes with a written report of throughput, query behavior, data quality defects and open risks that the sponsors accept or reject.
Design the catalog and table maintenance
Based on pilot results, the team selects the catalog approach for Iceberg tables, such as a REST, Hive metastore or Glue-compatible catalog, and confirms that the chosen engines support it. Table maintenance is designed next: snapshot expiration, compaction of small files from high-frequency sensor writes, orphan file cleanup and partition strategy. Naming, ownership and lineage metadata are defined for a catalog such as DataHub or OpenMetadata if the organization adopts one. The output is a documented catalog design and a maintenance schedule with an owner for each job.
Build retention and access policies
Retention tiers are set per data class, for example high-resolution sensor data kept for a shorter hot period and aggregated quality records kept longer, with the rules tied to the quality management and record obligations the plant must meet. Access control is built on the roles agreed in scoping, using catalog and engine permissions, and authentication is connected to the organization's identity provider. Policies are tested with real user accounts before production. The output is a retention schedule and access matrix signed by the data owners and reviewed by compliance.
Cut over to production with runbooks
Production cutover moves one source at a time from the pilot configuration to the production pipelines, with rollback steps written for each. Runbooks cover ingestion restarts, failed loads, storage capacity alerts, catalog outages and credential rotation, and the on-call contacts in IT and OT are named. Each source is switched only after reconciliation checks confirm row counts and key fields match the source system. The step closes when every source runs in production, the runbooks have been walked through by the operators and the rollback path has been exercised.
Hand over with restore tests
Handover transfers operating knowledge to the internal platform and operations team. DeployOpen documents the architecture, configuration, upgrade procedure and support model in the client's own repositories. Backups are configured for both the object storage and the metadata or catalog database, and a restore test is run on a spare environment to prove the data and catalog can be recovered. ClickHouse documentation warns that a backup never test-restored will probably not restore properly, so restore testing is a required exit criterion. The handover closes with a signed restore test report and a runbook review by the internal team.
Which evidence auditors and regulators ask for from the lakehouse
This table maps each kind of evidence that a manufacturing assessor, regulator or customer auditor may request to the component that produces it in a self-hosted lakehouse. Read it as a checklist to agree before build starts, since the owner of each record determines how it is retained and restored. DeployOpen prepares this evidence for assessments, but independent auditors and regulators issue any assessment outcome.
| Evidence item | What it shows | Produced by | Gaps to confirm |
|---|---|---|---|
| Data-flow diagrams and zone definitions | How plant, MES and ERP data moves into IT and OT zones, mapped to the ISA/IEC 62443 zone and conduit model | Architecture documentation maintained by the implementer and asset owner | Whether plant networks and the lakehouse ingestion path are documented as separate zones |
| Security risk assessment records | Identified threats and target security levels for the lakehouse and its connections to the plant | Risk assessment process under ISA/IEC 62443-3-2 with asset owner sign-off | Whether the assessment covers ingestion gateways and not only the central platform |
| Incident records and NIS2 reporting timelines | Dated record of significant incidents, early warning within 24 hours, notification within 72 hours and final report within one month | Incident ticketing system and runbook, with timestamps from monitoring and logs | Whether your organization is an in-scope entity and who owns the reporting decision |
| Access logs from the catalog and query engines | Who queried or changed plant and quality tables, and when | Trino or ClickHouse query logs, catalog audit events and identity provider records | Whether logs leave the site, how long they are kept and who can alter them |
| Change records for tables and pipelines | Schema changes, pipeline deployments and upgrades to engines and catalog | Version control of pipeline code and the catalog change history | Whether production changes pass an approval step with a recorded approver |
| Iceberg snapshot history | The data state at a given time, supporting reproduction of a quality or batch-release analysis | Apache Iceberg snapshot metadata, queried through time travel and rollback features | Snapshot expiry settings, since expired snapshots no longer support historical reproduction |
| Backup and restore test records | Evidence that ClickHouse backups restore and that recovery was tested, not only scheduled | BACKUP and RESTORE jobs and restore test reports run on a spare cluster | Whether restore tests are recorded with dates, scope and results, not only backup success |
| Electronic record controls under 21 CFR Part 11 | Where batch or quality records are kept electronically, controls for access, audit trails and signatures | Application-level controls in the MES or quality system, with the lakehouse holding copies | Whether the regulated record is the source system or a derived copy, which changes the scope |
What manufacturers ask before building a lakehouse on premises
What drives the cost of an on-premises manufacturing lakehouse?
The main cost drivers are the number of data sources, the volume and frequency of sensor and historian data, how many zones and retention tiers you need, the number of plants connected, and the in-house skills you already have. Storage and compute hardware or cloud capacity recur as data grows, while design, integration, hardening and handover are mostly one-time. Ask any partner to split the quote into one-time engineering, recurring infrastructure, and recurring support, and to state which sources and volumes the price assumes. Hardware and software licensing choices also matter: the core components, including Apache Iceberg (Apache License 2.0) and ClickHouse server (Apache-2.0), are open source, but you still pay for infrastructure and people to run them.
How long does an on-premises lakehouse project take?
Duration depends on scope rather than on the platform alone. A project that connects one plant's historian and one ERP extract to a single analytics zone is far smaller than a multi-plant estate with quality, MES and ERP feeds, retention tiers and access control tied to your identity provider. Before accepting a timeline, confirm which sources are in scope, how many plants go live in the first phase, whether network changes or OT approvals are needed, and what acceptance tests close each phase. DeployOpen works from a written, fixed scope with agreed deliverables and timelines, so the schedule should be an output of that scope, not an assumption.
What must our own teams provide during the build?
A lakehouse for manufacturing needs input from three groups that the platform team cannot supply alone. OT and plant engineers must confirm which controllers, historians and MES tables produce data, where the network boundaries sit and what can be connected without affecting control systems. Quality owners must define the quality datasets, metrics and retention periods. Data stewards must agree ownership, naming and access rules, and IT must provide identity integration, certificates, backup storage and change windows. Ask partners to name these roles in the scope document, with the hours or availability expected from each, so gaps appear before go-live.
Can plant IT run the lakehouse alone after go-live?
Plant IT can operate a lakehouse alone only if it has people who understand the table format, the query engine, the catalog, backups and upgrades, and if the operating runbooks are complete. ClickHouse, for example, recommends at least three Keeper nodes on dedicated hosts for production replication, and its backup documentation warns that a backup never test-restored will probably not restore properly, so restore testing is a recurring task someone must own. A retainer is an option for ongoing support, upgrades and continuous testing. Ask what a retainer covers in writing: response expectations, upgrade cadence, backup restore tests, security patching and how changes to plant sources are handled.
What does ClickHouse's support policy mean for upgrades?
ClickHouse publishes its version support status, and the policy determines how often you must upgrade. The security policy lists versions 26.9, 26.8, 26.7 and 26.3 as currently supported with security updates, and all other versions as unsupported. The production FAQ says stable releases ship roughly monthly, LTS releases ship twice a year and are supported for a year, and you should read the changelog before upgrading because minor backward-incompatible changes can occur. Plan a pre-production environment with representative plant data, and ask your partner to schedule upgrades against plant maintenance windows.
Can a self-hosted lakehouse meet NIS2 or CRA obligations by itself?
No single platform meets NIS2 or the Cyber Resilience Act by itself. NIS2 Annex II lists manufacturing as a critical sector covering NACE Rev. 2 divisions 26 to 30, and in-scope entities must report significant incidents with an early warning within 24 hours, a notification within 72 hours and a final report within one month (Directive (EU) 2022/2555, Article 23(4)). The Cyber Resilience Act's Article 14 reporting duties have applied since 11 September 2026 under Article 71(2), and Article 64 sets maximum fines of EUR 15 million or 2.5% of worldwide annual turnover. A lakehouse can support evidence collection, such as access logs, backups and retention records, but the obligations sit with your organization and its governance processes. Confirm your classification and obligations with counsel and your competent authority.
Which industrial security standards apply to plant data platforms?
The ISA/IEC 62443 series is the main reference for securing industrial automation and control systems, and ANSI/ISA-62443-2-1-2024 sets security program requirements for asset owners. ANSI/ISA-62443-3-2-2020 covers risk assessment for system design, and ISA-62443-3-3-2013 sets system security requirements and security levels. Because a lakehouse connects to plant networks, these controls shape how you segment the data path, where ingestion runs and who can reach historians. Ask your partner to map the design to these control areas and to confirm, with your OT security team, which zones and conduits the platform may cross.
What to ask an implementation partner before signing
Start with OT zone design, because it decides what the lakehouse is allowed to touch. Ask the partner which ISA/IEC 62443 zones and conduits they would place the historian, the MES connectors and the ingestion gateways in, and how they derived those boundaries. The reference to ISA-62443-3-2-2020 covers security risk assessment for system design, so ask which risk assessment method they use and who on your side signs off on it. Ask for examples of past OT isolation work in general terms, and ask how one-way data flow from the plant network into the analytics zone is enforced, for example with a data diode or a tightly restricted pull from a DMZ. A partner who answers with a generic network diagram and no plan for OT isolation should be treated as a red flag.
Next, ask how the design handles late and out-of-order sensor data, and what the table maintenance plan looks like after go-live. Late readings from edge buffers will arrive after their partitions have been written, so ask how the ingestion job decides whether to append, merge or quarantine them, and how the Iceberg tables are partitioned for time-series access. On maintenance, ask for a written schedule covering compaction of small files, snapshot expiry that matches your retention tiers, and orphan file cleanup, along with who runs each job and where its logs are kept. Ask how the partner will prove restores work. The ClickHouse backup documentation warns that a backup never test-restored will probably not restore properly, so ask for a commitment to scheduled restore tests on a spare cluster, with the restore results delivered to you. A partner who will not put restore testing in the written scope should be a red flag.
Finally, ask how upgrades and governance are handled, then check that the written scope fixes what you receive. ClickHouse publishes stable releases roughly monthly and supports only a short list of versions with security updates, so ask how the partner will track that list, stage upgrades in a pre-production environment with representative data, and read changelogs for backward-incompatible changes. Ask the same of Iceberg and the query engines that read it. For lineage and catalog governance, ask which catalog the partner will deploy and whether DataHub or OpenMetadata will record lineage from the MES and ERP sources to the reporting tables. Insist that the scope names each deliverable, the acceptance test for it, and what is excluded, because a fixed-scope agreement is only as useful as its written deliverables.
How DeployOpen scopes and supports a manufacturing lakehouse
DeployOpen is a Chainscore Labs company that plans, implements, integrates, hardens, backs up and documents self-hosted open source platforms inside the client's own plants and data centers. For a manufacturing lakehouse, that means the work starts with a written scope that names each plant, each source system (historian, MES, quality, ERP), the object storage nodes, the catalog, the query engines and the people who will operate them. Most engagements are fixed-scope, so deliverables, acceptance criteria and timelines are agreed in writing before any build work begins. Ask any partner, DeployOpen included, to name which plant networks are in scope, who signs off on changes that touch the OT boundary, and which items are explicitly excluded.
Security testing covers the layers that carry plant data into the lakehouse. That includes the ingestion endpoints that accept data from the plant network, the API and connector credentials, the catalog and the SSO or LDAP integration used by analysts. DeployOpen runs application, API and infrastructure testing against these components before go-live and can repeat it after upgrades. Before scoping, confirm with your OT security team how the plant zones and conduits are defined under ISA/IEC 62443, since that determines where the lakehouse may be connected and which ingestion direction is allowed. DeployOpen prepares evidence for compliance assessments, such as configuration baselines, access reviews, backup restore test records and change logs, but it does not issue audit reports, attestations or certifications. Independent auditors and certification bodies perform those assessments.
After handover, operations usually need continued attention: ClickHouse or Trino version upgrades, catalog maintenance, certificate rotation, restore tests and changes to new production lines. These are offered as retainers, with the scope of upgrades, response expectations and testing cadence agreed in writing. When you request a quote, ask for the line items separately: discovery and architecture, ingestion and integration build, security testing, backup and restore setup, documentation and training, and the recurring support retainer. Ask which items are one-time and which recur, and what would change the price, such as the number of plants, sensor tag volume or the number of source systems.
