CKAN migration and legacy portal upgrades

What a CKAN migration involves

A CKAN migration usually means one of two different jobs, and the reader should decide early which one applies. The first is importing datasets and metadata into CKAN from another catalog, a harvest source or a set of spreadsheets. The second is moving a legacy CKAN 2.x portal, with its custom extensions, theme and data, to a supported release on new infrastructure. The first is mostly a data mapping and loading exercise. The second is a software upgrade with a data move attached, and it carries most of the risk.

Common triggers for the second type are an old minor version that no longer receives fixes, an extension that will not run on a newer release, or hosting that has to change for reasons such as a data center exit or a new security baseline. CKAN's release policy supports the latest patch of the current minor version and the previous minor version, so a portal further behind than that is already outside the support window. Confirm which minor version and patch level your portal runs before estimating anything else.

In-house or partner? A team already familiar with CKAN internals, its plugin interface and its schema can often do a straightforward import itself, particularly from a clean DCAT export. Porting custom extensions to a newer release and moving the database, Solr and Redis to new infrastructure usually benefit from outside help, because breaking changes in minor releases, DataStore and search index rebuilds, and cutover timing all interact. Ask any partner to show how they will test extensions against the target release before they commit to a scope.

Two inventories set the scope for either path: a list of every installed extension with its source and any local patches, and a list of every dataset, resource and DataStore table that must survive the move. Missing either one is the most common reason a migration quote changes after work starts. Agree in writing which extensions are kept, replaced or retired, and who signs off on data counts after cutover.

What the work includes

The migration breaks into seven work packages that run in sequence, with overlap between design and mapping. Each one ends in a artifact your team can review, rerun or keep after the engagement closes.

Source inventory and audit

Record every dataset, resource, organization, user role, harvester, custom extension and theme on the source portal, including its CKAN version and patch level. Identify which extensions have no maintained equivalent on the target release, and which datasets depend on files that DataStore or XLoader will need to load. The audit sets the scope that every later estimate depends on.

Target design and stack

Fix the target CKAN release, metadata schema, and component layout before any data moves. Decide how DataStore and XLoader will load tabular resources, and size PostgreSQL, Solr, and Redis for the target hosting in your own infrastructure. Confirm supported Python, PostgreSQL, and Solr versions against the release documentation for the chosen CKAN version.

Dataset and metadata mapping

Map each source field to a CKAN core field, a scheming field or an extra, and record any value transformations such as date formats, licence identifiers and controlled vocabularies. Run validation on a sample batch and on the full set, flagging records that fail schema rules. The mapping document becomes the reference for every later import and reconciliation.

DCAT import with ckanext-dcat

Where the source publishes DCAT catalog records, use ckanext-dcat and its RDF harvester to import them rather than re-keying metadata by hand. Check which DCAT-AP version the source emits against the version the extension supports on your CKAN release. Test imports on a copy and compare field counts before accepting the output.

Extension compatibility and porting

Review each custom extension against the target CKAN minor version, because minor releases can include backwards-incompatible changes. Port the ones you keep, remove the ones that duplicate core features, and test each against your own datasets and organizations. Record the test results so the next upgrade starts from a known state.

Staged cutover and rollback

Run at least one full rehearsal on a staging copy with production-size data, then cut over in a planned window with a freeze on source edits. Keep a tested rollback path that restores the source portal or the prior database snapshot, and time both forward and reverse steps during rehearsal. Reconcile counts and checksums after each stage.

Redirects, link checks and handover

Map old dataset and resource URLs to new ones, publish the redirect rules, and run link checks against harvested and external references. Hand over the migration mapping document, the scripts with their run logs, the reconciliation report, an updated runbook and the tested rollback procedure. Your team should be able to rerun the import and restore the target without the implementation team.

What drives effort and duration

This table lists the migration factors that most change the effort and schedule for a CKAN move, with why each matters and how to keep it under control.

Driver	Why it matters	How to reduce it
Source systems and record counts	Each source adds mapping work, and record volume drives load and validation time.	Inventory every source and export. Profile record counts per dataset before scoping.
Metadata quality and schema mismatch	Legacy fields rarely map cleanly to the CKAN schema, and gaps need manual decisions.	Agree a field mapping early. Fix or default required fields in the source before export.
Resource files versus links	Uploaded files must be copied and checked, while linked resources may be dead or moved.	Run a link check. Decide per resource whether to re-upload, relink or retire it.
DataStore use and XLoader reloads	Tabular resources may need reloading through XLoader, and large files take longer to process.	Identify DataStore-backed resources and plan XLoader runs. Treat DataPusher as legacy.
Custom extensions and theming	Each extension and template may need rework on the target version, and older code usually needs the most.	Inventory extensions by age and owner. Retire unused ones and test the rest on the target.
Distance between source and target minor version	Minor releases may include backwards-incompatible changes, so a larger gap means more upgrade steps.	Plan stepwise upgrades where needed. Review release notes for each minor version in between.
Authentication changes	Moving to SAML2 with ckanext-saml2auth changes login, user mapping and sysadmin control.	Map identity provider attributes to organizations and roles before cutover. Test with a representative user set in staging.
Environments and cutover window	Each environment needs its own copy, and the publisher freeze window limits how much rework fits.	Agree the number of environments and rehearse the cutover. Confirm the freeze window with publishers in advance.
Compliance evidence requirements	Assessors may ask for change records, data lineage and access evidence from the migration.	Define evidence needs at scoping. DeployOpen prepares the evidence, and auditors and certification bodies decide on it.

Migration phases and outputs

The sequence moves a CKAN portal and its data to a supported release on new infrastructure through checked stages, with each phase producing a record the next one depends on.

Inventory the current portal
DeployOpen reviews the running CKAN version, installed extensions, custom code, theming, datasets, organizations, users, harvesters, DataStore resources and integrations with other systems. The output is a written inventory with each item marked keep, port, replace or retire, and a list of open questions for the portal owner to answer before design starts.
Design the target and select the version
The target architecture covers hosting, PostgreSQL, Solr, Redis, DataStore, authentication and backups. The target CKAN version is chosen against the current CKAN release policy, which supports the latest patch of the current minor and the previous minor, and the team confirms the version with the owner. The output is a signed design record listing versions, extension compatibility and the upgrade path.
Build the staging environment
A staging environment is built in the client's infrastructure with the target versions of CKAN, Python, PostgreSQL, Solr and Redis, following the install guidance for those versions. Hosting, monitoring and backup jobs are set up to match the planned production design. It is done when a clean install runs, passes health checks and restores from a test backup.
Write mapping and transformation scripts
Scripts map source metadata fields, organizations, groups, tags, resource formats and licenses to the target schema, and transform any catalog exports from other systems into CKAN records. Each script is version controlled and run against sample data. The output is a mapping specification and repeatable scripts with a log of rejected or changed records.
Port and test extensions
Each custom or third-party extension is either ported to the target CKAN version, replaced with a supported extension, or retired. Ported code gets automated tests and manual checks of the features the portal owner relies on, such as harvesting, DCAT output or single sign-on. The output is a tested extension set with a compatibility note for each one.
Run trial migrations and reconcile
The full migration runs several times against copies of production data in staging. After each run, record counts, dataset metadata, resource links, DataStore tables, users and permissions are compared with the inventory. Differences are logged and fixed in the scripts. The output is a reconciliation report that the portal owner signs off on before cutover planning.
Rehearse cutover and go live
The cutover runbook is rehearsed end to end on staging with the same timing, freeze window and rollback steps planned for production. Production cutover then follows the signed runbook, with a final data sync, DNS or routing change and smoke tests. Post-migration support covers monitoring, fixes and handover. The output is a go-live record and a handover document, and the length of each phase depends on the scope agreed in writing.

What teams keep in-house and what they outsource

This table compares each part of a CKAN migration or portal upgrade with the skills it needs and the usual split between the publishing organization's team and an implementation partner.

Work part	Skills needed	Typical split
Metadata mapping and data stewardship	Open data publishing, metadata standards, domain knowledge of each dataset	Mostly in-house, because only the publishing organization knows what its data means and who owns it
CKAN and Python extension porting	Python, CKAN plugin interfaces, reading extension code and release notes	Often outsourced, with in-house staff reviewing behavior and approving changes
PostgreSQL and Solr operations	PostgreSQL administration, Solr schema and index management, backup and restore	Shared: partner builds and tunes, in-house team takes over daily operation and monitoring
DataStore reload and XLoader checks	CKAN DataStore and XLoader behavior, PostgreSQL role setup, sampling of loaded resources	Partner runs the reload and checks; in-house staff confirm resources that matter to their users
Identity integration	Identity provider administration, SAML2 or OIDC configuration, attribute and role mapping	Shared: identity team owns the provider and attribute policy, partner configures the CKAN side
Infrastructure build and cutover	Linux, web server and process management, DNS, certificates, change control	Often outsourced for build and cutover runbooks, with in-house staff approving windows and access
Reconciliation sign-off	Record counts, sampled field comparison, publisher review, rollback decision	Mostly in-house: publishers and data owners sign off, partner supplies the evidence
Post-handover skills	Patch and upgrade procedure, extension compatibility checks, Solr and Redis care, restore testing	Must stay in-house after handover, even when a support retainer is in place

What usually goes wrong

Most CKAN migrations fail quietly rather than loudly. The risks below are the ones that show up after cutover, when fixing them costs the most, so each card names the check to run before users arrive.

Metadata loses fields silently

Source catalogs often store fields that have no direct home in the CKAN schema, and importers may drop them without an error. Map every source field to a CKAN field, a custom schema field or an explicit decision to discard it. Compare record counts and sample field values for each dataset type after a trial load.

Extensions break after a minor release

CKAN minor releases may include backwards-incompatible changes, so a custom extension that works on one minor version can fail on the next. Check each extension's compatibility with the target version in a staging copy, and pin the target version until every extension has passed its tests. Treat extension updates as separate work items with their own test cycle.

Source patch drift leaves the mapping incomplete

If the source portal receives patch releases or local fixes while the migration project runs, the mapping built against an earlier state can miss new fields or behavior. CKAN patch releases do not break backwards compatibility, but they can still change behavior that a mapping relies on. Freeze the source version for the migration window or re-run the mapping check after each patch.

Search index not rebuilt

Migrated datasets that are inserted without rebuilding the Solr index return stale or incomplete search results, and users may conclude the data is missing. Schedule a full index rebuild after the final data load and compare search hits with the database dataset count. Confirm that the Solr version matches the CKAN release you are deploying.

DataStore tables not reloaded

The DataStore holds structured resource data in its own PostgreSQL database, so a database move or metadata import does not carry those tables along. Resources can appear present in the catalog while their Data API queries fail or return nothing. Reload tables through XLoader, which is the supported loader, and test the read-only database user against each migrated table.

Resource URLs break after host change

Resource links, uploaded file paths and images often embed the old hostname or site URL, so they break once the portal moves. Search the database and configuration for the old domain, rewrite stored URLs under a tested rule, and confirm that uploaded files are reachable from the new storage location. Run a link check across a sample of every dataset type before DNS changes.

Cutover without a rollback path

A cutover that overwrites the only copy of the source data or the only working database leaves no way back when a defect appears on day one. Keep the source system read-only but running, take a verified backup of the target before the switch, and write down the steps and the decision owner for reverting DNS and data. Rehearse the rollback on the staging environment before the production date.

How the work changes by starting point

Moving from a spreadsheet or CSV is mostly a data quality and mapping exercise. There is no running system to preserve, but the files often hold inconsistent field names, merged cells, duplicate titles and free-text licenses that CKAN will not accept as they are.

Scope: map each column to a CKAN dataset field or custom extra, then load through the API or a scripted importer.
Risk: silent data loss from truncated values, mismatched organizations and unclear ownership of each row.
Decisions: the target metadata schema, who approves each dataset, and whether resource files are uploaded or only linked.
Ask the partner for: a field mapping document and a sample load in a staging portal before the full import.

What you need to provide

The migration moves data only as well as the inputs behind it. Before scoping starts, you need read access to the source portal and its database dumps, a sample of representative records and uploaded files, and a current inventory of every CKAN extension with its version and an owner who can answer questions about it. The inventory matters most for a legacy 2.x portal, because custom extensions often pin behavior to a specific minor release and decide which target version is feasible. If an extension has no owner, the plan needs a decision on whether to retire, replace or rewrite it.

Metadata quality depends on people. Name one metadata owner per publishing organization who can confirm which fields are mandatory, how legacy values map to the target schema, and which datasets should be archived rather than moved. Decide early on the target minor version, since the latest two minor releases receive fixes and a move to an older line creates a second upgrade later. Also decide on the identity provider, because a SAML2 or other SSO setup changes how users and organization roles are mapped during cutover. Confirm hosting, network paths to the source, DNS, certificates and outbound access for harvesting or XLoader jobs.

Provide a test environment sized close enough to production to run the full load, Solr reindexing and DataStore loads, and keep it available for repeated staging runs. Agree a cutover window with the business owners of the portal, including a rollback point and a time when publishers can freeze edits. Name a sign-off owner who accepts the staging results and the cutover, and make sure they can decide quickly on defects. During staging and cutover, expect the platform engineer, a CKAN developer, one metadata owner per affected organization and the sign-off owner to be available for the full window.

Frequently asked questions

What drives the cost of a CKAN migration or portal upgrade?

The main drivers are scope and condition. The number of versions the portal must cross, the number and age of custom extensions and themes, the volume and quality of existing metadata, and whether DataStore or Solr indexes must be rebuilt all change the effort. Cost is split between assessment, data mapping and transformation, extension porting and testing, building the target infrastructure (PostgreSQL, Solr, Redis and the application servers), cutover, and post-launch checks. Infrastructure hosting is paid to your provider and recurs. DeployOpen's work is usually one-time unless you add a retainer. Ask for a quote that itemizes each component.

How long does a legacy CKAN upgrade usually take?

The timeline is set by the scope you agree, not by a standard duration. The biggest factors are how many minor releases separate your version from a supported one, how many extensions need porting or replacing, how much data needs cleaning, and how many integrations such as harvesters, single sign-on and DCAT feeds must be retested. CKAN targets about two minor releases a year, and the release policy supports only the latest and previous minor versions, so an old portal may need intermediate steps. Approval cycles and the availability of your data stewards for testing also affect the schedule. A written timeline follows the scoping stage.

Can our own team migrate a CKAN portal without an outside partner?

Yes, if your team has the skills and capacity for the whole change. CKAN installs from source on Ubuntu 20.04 and 22.04 and on other Linux distributions, so the platform itself is accessible. The harder work is porting custom extensions, reconciling metadata, validating DataStore loads, and running a controlled cutover with a rollback plan. Your team needs people who can operate Python 3.10 or newer, PostgreSQL 12 or newer, Solr and Redis, plus time for testing across publishers. Staff needs include named data owners, administrator access to hosting, decision-makers for field mapping, and reviewers for sign-off. Confirm these during scoping before deciding.

Is a CKAN migration fixed-scope or billed by the hour?

Most DeployOpen engagements are fixed-scope. Scope, deliverables and timelines are agreed in writing before work starts, so you know what is included before you commit. If requirements change, the change is recorded in writing and priced before it is built. Ongoing support, upgrades and continuous testing after go-live are available as a retainer, which covers work that continues beyond the original project. Ask any provider to state in writing which items are fixed, which are estimated, and how change requests are handled.

What happens to a CKAN extension that has no maintained version?

The usual options are replacement, adoption, rewrite or retirement. First, check whether the functionality now exists in CKAN core or in a maintained extension. If not, the team can fork and maintain the extension, rewrite it against the target CKAN version, or drop the feature after agreeing that with the business owner. Check each candidate's compatibility with your target minor version. For example, ckanext-dcat is supported on CKAN 2.10 and 2.11, so a move to a newer release needs its own check. Each option carries ongoing maintenance, which should appear in the cost of ownership, not only in the migration quote.

Does migrating CKAN change our metadata standard?

The migration moves your metadata, but choosing a new standard is a separate decision. Moving data into a new CKAN version keeps your current schema unless you change it. If you need DCAT-AP, the ckanext-dcat extension provides DCAT-AP v1, v2 and v3 schemas, RDF endpoints and an RDF harvester for importing from other catalogs. Adopting it means mapping your existing fields to the target profile, deciding how to handle fields with no equivalent, and checking that harvested records still validate. Document these decisions before migration so publishers know what changed.

How DeployOpen delivers CKAN migrations

DeployOpen plans and runs CKAN migrations and portal upgrades inside infrastructure the client controls, starting from an inventory of the existing portal, its datasets and its custom extensions. Work follows a written, fixed scope with defined deliverables and a path to ongoing support.

How we scope

Inventory the current CKAN version, installed extensions, metadata schema, datasets, resources and users before any target is chosen.
Map each custom extension and theme to a supported CKAN release, noting which need rework, replacement or removal.
Agree in writing which datasets, metadata fields, DataStore tables and integrations are in scope, and what remains out of scope.

How we build

Stand up the target CKAN release on new infrastructure with PostgreSQL, Solr, Redis and the DataStore configured for your environment.
Run test migrations on copies of production data, checking dataset counts, metadata mapping, resource links and DataStore loads.
Validate extensions, authentication and API clients against the target release before any cutover is scheduled.

How we hand over

Deliver runbooks covering the installed versions, configuration, migration steps, rollback procedure and known deviations from the plan.
Provide a verification record that lists migrated record counts, exceptions and the acceptance checks the client signed off.
Transfer operational access and walk your team through patching, backups and extension updates on the new environment.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer after handover, scoped in a separate written agreement.
Retainers can cover tracking CKAN patch releases and applying them in your environment on a schedule you approve.
Ask any partner to confirm how they handle minor-release changes to extensions, Solr and Python before you sign.

CKAN migration and portal upgrades from old versions