Migrate from Pentaho Data Integration to Apache Hop: what the move involves
The move is a conversion of definitions followed by a long stretch of testing and rework. Hop's importer reads PDI .ktr transformations and .kjb jobs and writes Hop .hpl pipelines and .hwf workflows. It also carries kettle.properties and shared database connections from shared.xml into Hop metadata. Everything else is work your team does: checking each converted pipeline, rebuilding anything the importer skipped, deciding how credentials and environments are handled, and proving that the output matches what PDI produced. The importer runs from the GUI (File, Import from Kettle/PDI) or from the hop-import command-line tool, so a small pilot on one non-critical transformation is a sensible first step before you commit to the full estate.
Most of this work falls to your internal data engineers, with an implementation partner brought in for specific gaps. The people who know the PDI jobs, their schedules and their downstream consumers are the ones who can judge whether a converted pipeline is correct. Review the documented import limitations before you start: same-named database connections keep only one copy, and metastore content is not imported. Plan for steps that have no direct Hop equivalent, which need a redesign rather than a mechanical swap, and for PDI schedules, which are not part of the import and must be recreated on an external scheduler because Hop Server does not schedule work. Ask any partner for a written scope that names the pipelines in it, the test method for each one, and who signs off on parity with PDI output.
The short answer on effort and risk is that the import itself is quick and the risk sits in parity testing and scheduling. The effort scales with the number of jobs, the share of custom or scripted steps, the number of duplicate connections, and how many environments and schedules you run. Confirm these counts for your own estate before you estimate anything. Run old and new pipelines side by side on the same inputs, compare row counts, checksums and load timings, and keep PDI in place until every job has matched output through at least one full business cycle. The main risk is a silent difference in results, so treat parity evidence as a deliverable, not a formality.
Which PDI features move to Hop cleanly and which need rework
This table maps common PDI features to their Apache Hop equivalents and marks each as clean, needs rework or gap. Use it to scope conversion work before you run the importer, since the importer handles some artifacts automatically and leaves others for manual review. Confirm every step type against the Hop transform list before you commit to a plan.
| PDI feature | Apache Hop equivalent | Status and what to check |
|---|---|---|
| Transformations (.ktr) | Pipelines (.hpl) via hop-import or the GUI import | Needs rework: the importer converts them, but each pipeline must be run and compared against PDI output. |
| Jobs (.kjb) | Workflows (.hwf) | Needs rework: job entries map to workflow actions, so review every entry and its error handling path. |
| kettle.properties | Hop project and environment variables | Clean: values are imported, but confirm which variables each environment expects after import. |
| Shared database connections (shared.xml) | Hop metadata database connections | Needs rework: the importer keeps only one copy of same-named connections, so reconcile duplicates by hand. |
| Metastore objects | Not imported | Gap: recreate any metastore content in Hop metadata manually and test each dependent pipeline. |
| Steps with no Hop equivalent | Confirm against the Hop transform list (250+ transforms) | Gap until confirmed: plan a replacement or custom transform for anything without a documented match. |
| Scheduling (cron, PDI Carte, external schedulers) | External scheduler or orchestrator submits work to Hop Server | Gap: Hop Server is not a scheduler, so schedules must be rebuilt in your own orchestrator. |
| Run target: native engine locally or on Hop Server | Native Hop engine, local or remote Hop Server | Clean: the closest match to a PDI run, and the usual starting point for parity testing. |
| Run target: Spark, Flink, Dataflow, Databricks | Run configurations through Apache Beam or native Spark, or Databricks | Needs rework: runtime selection is configuration, but performance and behavior must be tested on each target. |
What to export and rebuild from a PDI estate
A PDI estate is more than its transformation and job files. The hop-import tool and the GUI importer convert .ktr and .kjb files, kettle.properties and shared database connections, but schedules, logging and several external dependencies sit outside that path. Inventory each category first so that nothing reaches production by accident.
Transformation and job files
Collect every .ktr and .kjb file, including those in subfolders that no current job calls directly. The importer converts transformations to pipelines (.hpl) and jobs to workflows (.hwf), so keep the source tree in version control before you run it. Plan to review each converted file, because step-level differences are where conversion problems surface. Note which files are dead code so you do not convert them.
Shared connections and duplicate names
The importer converts shared database connections from shared.xml into Hop metadata. Documented limitations say it keeps only one copy of same-named connections, so list every connection name that exists in more than one place, such as a development and a production definition under the same label. Decide which definition wins and rename the others before import, or the converted pipelines may connect to the wrong database.
kettle.properties and variables
Export kettle.properties and any variables set in kicker, carte or the spoon startup scripts. These values become Hop configuration and environment settings, and they often carry hostnames, file paths and credentials. Confirm which variables are used by which pipelines, and replace stored passwords with a secrets mechanism your team controls rather than copying them into project files.
Metastore content
The importer does not import the PDI metastore, so any objects stored there must be rebuilt by hand in Hop. Inventory the metastore entries that pipelines reference, including connections, data sources and other reusable definitions, and map each to its Hop metadata equivalent. Test each rebuilt object against a pipeline that uses it, since a missing definition often shows up only at runtime.
Schedules, cron and Carte jobs
Job schedules held in cron entries, the PDI scheduler or a sequencing tool are not converted. Hop Server is not a scheduler, so an external scheduler or orchestrator must submit work to it. Write down each trigger time, its dependency on other jobs, and who owns it, then rebuild the trigger in your chosen orchestrator. Keep the old schedules disabled but available until parallel runs match.
Logging, error handling and external endpoints
Record the log levels, log tables, error-handling hops and alerting that PDI jobs use, because these are typically hand-configured and not part of the importer output. Also list every external source and target, including files, APIs, message queues and object storage paths, with its credentials owner. Confirm each endpoint is reachable from the new Hop runtime before you cut over.
The migration sequence from inventory to decommission
This sequence moves Pentaho Data Integration jobs and transformations to Apache Hop in an order that keeps production data flowing until each pipeline has been proven. Each step ends with a written output, so the cutover decision rests on evidence rather than memory.
Inventory every PDI asset and its schedule
Export a list of every
.ktrtransformation,.kjbjob,shared.xml,kettle.propertiesfile and external scheduler entry that starts them, including the server, user account and downstream consumer for each. Your data engineering lead and the owner of each schedule should sign off on the list. The output is an inventory sheet with one row per asset, showing its owner, source and target systems, run frequency and business criticality.Classify assets by conversion risk
Mark each asset as direct, needs review or blocked. Direct means it uses only steps and connections that Hop's importer handles. Needs review means it uses duplicate connection names, custom Java or scripting steps, or plugins with no Hop equivalent. Blocked means it depends on a feature you have not yet replaced. Confirm the step list against the Hop manual's import documentation. The output is a classified inventory that sets the pilot scope.
Pilot a small set of representative pipelines
Choose a handful of assets that cover your common patterns, such as a database-to-warehouse load, a file ingestion job and a job with error handling. Run
hop-importon them in a scratch Git branch, open the results in the Hop GUI and fix what the importer leaves behind. Record every manual change. The output is a pilot report listing each asset, what the importer produced, what you changed by hand and the open questions for the platform team.Convert the full set and resolve duplicate connections
Run the importer across the remaining inventory in the same project structure you chose during the pilot. The importer keeps only one copy of same-named database connections, so compare each connection against its source definition in
shared.xmland rename or merge the duplicates deliberately. Check that every environment-specific value fromkettle.propertiesis mapped to a Hop variable. The output is a versioned Hop project where each connection and variable has a documented owner and source.Compare Hop output with PDI output
For each pipeline, run the PDI original and the Hop version against the same input snapshot in a non-production environment. Compare row counts, column types, null handling, sort order, error rows and target table contents. Investigate any difference before accepting it. The output is a comparison log per pipeline with a pass or fail result and the reason for every failure, which becomes the acceptance record for the cutover.
Run PDI and Hop in parallel
Keep the PDI schedule active for production while the Hop version runs against a separate target or a shadow schema for an agreed observation period. Compare the two outputs on each run and log any drift. The operations team should confirm alerting covers both paths. The output is a parallel-run report showing that each pipeline matched its PDI counterpart across the observed runs, with any exceptions assigned to an owner.
Deploy Hop Server or Docker and move the schedules
Choose the runtime for production: a long-lived Hop Server, short-lived
apache/hopcontainer runs, or both. Hop Server does not schedule work, so move each job trigger to your existing scheduler or orchestrator, and deploy projects as Docker images, exported ZIP resources or from a Git checkout. Switch one schedule at a time and keep the PDI entry disabled rather than deleted until the cutover is confirmed. The output is a production runbook with the schedule map and rollback steps.Decommission PDI after the agreed exit criteria
Once every production schedule has run on Hop for the agreed period and the owners have signed off, archive the PDI projects, repositories and
kettle.propertiesfiles, revoke the service accounts and remove the PDI server or runtime. Keep the archived copies and comparison logs for your audit and records retention policy. The output is a decommission record that names the retired assets, the date of retirement and the owner who approved it.
How the migration differs by estate size and starting point
A small estate is a handful of transformations and jobs maintained by one team, usually with a few database connections and a scheduler that already triggers them. Here the migration is mostly a conversion and verification exercise. The importer can be run from the Hop GUI, and the team can review each converted .hpl and .hwf file by hand, which is usually faster than building a custom process around the work.
- Conversion scope: Run the import on every
.ktrand.kjbfile, then open each converted artifact and confirm that its steps still behave the way the original did. - Connections: Check
shared.xmlandkettle.propertiesfor duplicate names, since the importer keeps only one copy of same-named database connections. - Testing: Compare row counts and output files against PDI results for each pipeline, using the same input data for both runs.
- Run target: The native Hop engine on a single machine, run locally or on one Hop Server, is often enough. Confirm the existing scheduler can call Hop from the command line.
- Operations: Keep one owner for the converted project so changes are not split across old and new copies.
Before cutover, confirm with the team that every scheduled job has a named owner and a tested fallback to PDI.
A large estate has many jobs, shared connections across teams, and environment-specific settings spread through kettle.properties and the shared connection file. The main risk is not the individual transformation but the shared metadata. Converting in one pass without a mapping of connections, environments and owners tends to produce a project that works in one environment and fails in the others.
- Connection inventory: Before importing, list every shared database connection, the teams that use it, and which names are duplicated, so the review of duplicate connections has a clear owner.
- Environment design: Plan how development, test and production will differ, using Hop project and environment settings rather than copied files.
- Testing: Build a regression set that covers the most-used and most business-critical jobs first, and compare outputs with PDI for a full run cycle, including month-end or other peak loads.
- Run target: A shared Hop Server gives several teams one execution point, but Hop Server does not schedule work, so an external orchestrator must submit each run.
- Phased cutover: Move groups of jobs by dependency, keeping upstream and downstream jobs on the same engine during each phase.
Ask the team who owns each shared connection and each scheduled job, and confirm that they have signed off on the test results before any cutover date is set.
An estate that already runs on Spark or Databricks has a different question: where the pipelines should execute after conversion. Hop can run pipelines on Apache Spark natively or through Apache Beam, and it can run them on Databricks, selected as a runtime configuration rather than a rewrite. This means the migration can keep the cluster as the run target while changing the design tool, but the team still has to confirm that each converted step behaves the same way on the cluster as it did in PDI.
- Run target selection: Decide whether each pipeline runs on the native engine, Spark, Beam or Databricks, and document the reason for each choice.
- Step compatibility: Check every step against the Spark or Databricks runtime, since some steps and transforms may behave differently or need configuration changes on the cluster.
- Testing: Run each converted pipeline on the same cluster it will use in production, against the same sample data PDI used, and compare outputs, not just row counts.
- Cluster ownership: Confirm who manages the Spark or Databricks environment, the credentials and the job submission path, since these are often owned by a different team from the one that owns the PDI jobs.
- Cost model: Cluster time is a recurring cost that changes with run frequency and data volume, so ask the platform team to confirm the expected run profile before cutover.
Confirm the exact Spark, Beam or Databricks runtime version and Java requirements with the platform team before testing, because Apache Hop 2.19.0 requires Java 21.
What drives migration cost and duration
This table lists the factors that most change the effort of moving PDI jobs and transformations to Apache Hop, and how each one recurs or stops after cutover. Read the driver column first, then check how to reduce it before you ask any partner for a fixed-scope quote. Most of the cost sits in conversion review and testing, not in the importer run itself.
| Driver | Why it changes cost or effort | How to keep it down |
|---|---|---|
| Number of transformations and jobs | Each .ktr and .kjb needs import, a review of converted steps and a comparison of output with PDI. Effort scales roughly with the count of pipelines and workflows in scope. | Inventory every file with its owner and last run date, then retire unused jobs before import rather than converting them. |
| Custom Java or scripting steps | User Defined Java Class, JavaScript and similar steps have no direct Hop equivalent in most cases, so each one needs a rewrite and its own tests. | Count these steps early, move logic into standard transforms where possible, and confirm each replacement with the data owner. |
| Duplicate database connections | The importer keeps only one copy of same-named connections, so duplicates across projects can silently point a pipeline at the wrong database. | List every connection in shared.xml and project files, merge duplicates and rename them before import, then check each target host in the converted metadata. |
| Scheduling complexity | PDI schedules often live in cron, Carte, the PDI scheduler or an external tool. Hop Server does not schedule work, so each trigger must be rebuilt in an external scheduler or orchestrator. | Document every trigger, its parameters and its failure alerts in one table, then map each one to a named scheduler job. |
| Volume of test data available | Output comparison needs representative inputs and known-good PDI outputs. Without them, testing relies on sampling and takes longer to sign off. | Snapshot inputs and PDI outputs for each key job before cutover, and agree which differences are acceptable with the data owners. |
| Downstream integrations | Every system that reads files, tables or messages from a pipeline must be checked after the change, and each one adds coordination and retesting. | Map producers and consumers per pipeline, then run parallel loads for a fixed period before PDI is switched off. |
| One-time costs | Inventory, import review, step rewrites, comparison testing, cutover and runbooks are paid once, usually to staff or a partner under a written scope. | Ask any quote to itemize each phase separately so you can see what is conversion and what is testing. |
| Recurring costs | Hop Server hosting, the Java 21 runtime and its patching, scheduler upkeep and any bought support or retainer continue after cutover. | Confirm which Java 21 build your platform team will patch, and ask whether support is included or priced separately. |
What usually goes wrong when moving PDI to Hop
Most PDI to Hop migrations stall on a short list of predictable problems: output that looks right but differs, connections that the importer silently merges, steps that have no counterpart, schedules that nobody moved, and runtime versions that do not match. Each card names one problem and the check that prevents it before cutover.
Silent output differences
A converted pipeline can run without errors and still write different results than the PDI original. Differences often appear in null handling, string trimming, decimal precision, date parsing, sort order and row counts after joins. Run the same input through the PDI transformation and the converted Hop pipeline, then diff the output files or target tables column by column. Keep the PDI output as a frozen baseline so the comparison does not depend on the old server still running.
Same-named connections overwritten
The importer keeps only one copy of database connections that share a name. If your PDI shared.xml defines two connections called the same thing for different environments, one definition can replace the other and point jobs at the wrong database. Before importing, list every duplicate connection name, decide which definition each pipeline needs, and rename connections so each Hop metadata entry maps to exactly one target. Then confirm every converted pipeline resolves to the intended host.
Unsupported steps found late
Steps without a Hop equivalent often surface only when someone opens a converted transformation that was not part of the first test set. Build an inventory of every step type used across all .ktr files, including steps inside sub-transformations and those from third-party plugins. Mark each one as mapped, replaced or needing redesign, and assign an owner. Treat an unmapped step as a blocker for that pipeline, not a warning to resolve after go-live.
Schedules that stop at cutover
PDI jobs are usually triggered by a cron entry, a Carte server, a scheduler tool or a wrapper script that nobody documented. Hop Server is not a scheduler, so when the PDI environment is switched off, nothing submits the converted workflows. Inventory every trigger, including ones owned by other teams, and record its time, parameters and failure alerts. Run the new schedule in parallel with the old one for a defined period and confirm that alerts reach the same people.
Java 21 runtime mismatches
Apache Hop 2.19.0 was released on August 17, 2026 and requires Java 21. Hosts, build agents and containers still on an older Java version will fail to start Hop, or will run it on an unsupported JVM if the wrapper script picks up a different installation. Confirm the Java version on every Hop GUI workstation, Hop Server host and CI runner before installing. In containers, pin the apache/hop image tag so the JDK does not change under you during a rebuild.
Questions teams ask before moving from PDI to Apache Hop
How long does a migration from Pentaho Data Integration to Apache Hop take?
No duration is published for this work, so the answer depends on the size of your estate. The main drivers are the number of .ktr transformations and .kjb jobs, how many steps have no direct Hop equivalent, how many database connections are duplicated across projects, and how much test data exists to compare outputs against PDI. Start with an inventory of those four items, then run hop-import on a copy of one representative project to see how many objects need manual work. Ask any partner to base its timeline on that inventory and to state its assumptions in writing.
Can our own data engineers migrate PDI to Apache Hop without a partner?
Yes, many teams do the conversion in-house, provided they have people who know the existing PDI jobs and the target databases. The importer is available from the Hop GUI (File, Import from Kettle/PDI) or from the hop-import command-line tool, so the mechanics are not the hard part. The harder work is reviewing duplicate connection names, rebuilding steps that have no Hop equivalent, and proving that outputs match. If that review capacity is not available internally, a fixed-scope engagement can cover those parts with the scope agreed in writing first.
What does our team need to provide before a migration starts?
Plan to provide four things: access to the PDI repository or exported project files, including shared.xml and kettle.properties; named owners for each pipeline and job who can confirm expected behavior; read access to source and target systems, plus non-production credentials; and representative test data with known good outputs from PDI. Without the last item, nobody can prove that a converted pipeline behaves the same way. Confirm these requirements with your partner or internal team before the work is scheduled.
Do .ktr transformations convert to Apache Hop without any edits?
Some do, but expect manual changes in most real estates. Hop provides an importer that converts transformations (.ktr) to pipelines (.hpl) and jobs (.kjb) to workflows (.hwf), but the Hop project documentation lists limitations, including keeping only one copy of same-named database connections and no metastore import. Steps without a Hop equivalent, custom Java code, and environment-specific variables need review. Run each converted pipeline against PDI output before retiring the original. The Apache Hop manual on importing Kettle projects lists the current limitations.
How do PDI schedules move to Apache Hop?
Schedules do not move automatically, because Hop is not a scheduler. Hop Server is a long-lived process that executes pipelines and workflows, but an external scheduler or orchestrator must submit the work. Map each PDI schedule to whatever you will use, such as cron, a CI runner, or an orchestrator your team already operates, and record the owner, frequency and failure alerts for each job. Test the schedule in a non-production environment, including what happens when a run overlaps or fails. Hop can be deployed from Docker images, exported ZIP resources or a Git checkout, as described in the Hop Server deployment guide.
Which Java version does Apache Hop require?
Apache Hop 2.19.0 requires Java 21, according to the Apache Hop download page, so plan your runtime images and servers around that version. Check whether your existing PDI servers, custom plugins, JDBC drivers and monitoring agents work on Java 21 before cutting over. The official apache/hop and apache/hop-web Docker images are built on OpenJDK 21, which simplifies container-based deployments. Confirm any other driver or plugin requirement with its vendor, since those are outside Hop's own documentation.
What should a migration or support quote include?
A useful quote lists the scope in writing: the number of projects, transformations, jobs and connections in the inventory; the steps that need redesign; the test approach and who signs off outputs; the target runtime, such as local engine, Hop Server, or Spark, Flink or Databricks; and how schedules and secrets will be handled. It should also separate one-time work, such as conversion and testing, from recurring work, such as upgrades, backups and support retainers. Ask for assumptions and exclusions in writing, and for the deliverables and acceptance criteria. Since DeployOpen works on fixed-scope engagements with written scope before work starts, a quote follows from that scope rather than from a list price.
How DeployOpen scopes and runs a Pentaho to Hop migration
DeployOpen plans, implements, integrates, hardens, documents and supports self-hosted open-source platforms inside the client's own infrastructure, and a Pentaho to Apache Hop move follows the same pattern. The written scope comes first: it names which .ktr and .kjb files are in, which Pentaho environments and schedulers are affected, where the Hop projects and metadata will live, and which run targets (local engine, Hop Server, Spark, Beam or Databricks) the converted pipelines must support. Deliverables and timelines are agreed in the same document before any conversion starts. The quote follows from that scope, so the number of transformations, jobs, shared connections, custom or plugin steps and the test effort needed to compare outputs against PDI are the factors the reader should expect to see priced. Ask for each of these to be listed separately so the scope can be checked line by line.
Delivery is typically organized around the component checks described elsewhere on this page, applied to the client's own environment. Work covers running the importer on a sample set, reviewing duplicate database connections and the absence of metastore import, rebuilding scheduling and deployment with an external orchestrator because Hop Server is not a scheduler, and documenting each converted pipeline with its PDI reference output. Access is limited to what the client grants, and the client keeps the source code, the Hop projects and the run environment. Confirm in the statement of work who receives administrator access to the Hop Server host, Docker images and any Git repository holding projects, and how credentials for source and target databases are handled during testing.
Ongoing support, upgrades and continuous testing are available as retainers, which suit teams that want help keeping Hop on a current Java 21 runtime, reviewing new pipelines and rerunning regression comparisons after each upgrade. DeployOpen prepares systems, controls and evidence for compliance assessments, but it does not issue audit reports, attestations or certifications; independent auditors and certification bodies perform those. DeployOpen also is not a Pentaho or Apache Hop partner, reseller or certified provider, and any statement of that kind in a proposal should be questioned. Before signing, ask for the written scope, the deliverables list, the acceptance tests, the timeline and the retainer terms, if any, as separate items.
