CKAN harvesting: how a portal pulls in datasets from other catalogs
CKAN harvesting pulls datasets from other catalogs into your portal so your team does not re-enter them by hand. It runs through the ckanext-harvest extension, which stores each upstream catalog as a harvest source and runs each pass against that source as a harvest job. Consider a team that runs a regional portal and needs to show datasets from two partner catalogs (illustrative). Its first decisions are which harvester type matches each partner, how upstream fields map to CKAN fields, and how much of the upstream metadata must be preserved on import. Harvest jobs run in stages, and the queueing behind them depends on Redis, which CKAN's source install already lists as a requirement.
At a high level, a harvest source defines where the upstream catalog lives and which harvester reads it. A job then works through three stages. The gather stage collects the identifiers of the upstream records, the fetch stage retrieves the content for each record, and the import stage creates or updates the matching CKAN datasets. Each stage can fail independently, so the team should plan how a partial run is reported and retried. Confirm the exact configuration keys, the list of supported harvester types, and how stages behave on errors against the ckanext-harvest documentation, because this section does not restate those details and the extension's settings can change between releases.
The RDF harvester in ckanext-dcat is a separate path, not a setting inside the generic source model. It comes from ckanext-dcat, which also provides DCAT-AP v1, v2 and v3 schemas and DCAT RDF endpoints. Choose this path when the upstream catalog publishes DCAT RDF and the team needs the DCAT-AP mapping, and check the extension's own install and configuration steps before deciding how the two paths will coexist. The ckanext-dcat repository lists support for CKAN 2.10 and 2.11. Because CKAN 2.12.0 was released on August 26, 2026, confirm with the ckanext-dcat maintainers whether your 2.12 deployment is covered before you commit to the RDF path.
The CKAN version decision comes first. CKAN targets about two minor releases a year, and the tech team supports the latest patch of the current minor version and the previous minor version. CKAN 2.12.0 shipped alongside maintenance releases 2.11.6 and 2.10.11 on the same day. Patch releases do not break backwards compatibility, but minor releases may include backwards-incompatible changes, so harvester configuration should be retested whenever the minor version moves. Confirm which ckanext-harvest release matches your chosen CKAN minor version, since this section does not supply that compatibility table.
What a CKAN harvesting setup needs before the first job runs
Harvesting copies datasets from another catalog into your CKAN instance, so the software baseline, the job queue, the source's access terms, the network path and the administrator account all need to be ready first. The cards below cover each of these in the order you are likely to check them. Verify every item on the host and versions you plan to run, because a gap in any one of them usually appears as a failed job rather than a clear error message.
CKAN version and patch status
Choose a CKAN release that still receives fixes before you build anything. CKAN 2.12.0 was released on August 26, 2026, together with maintenance releases 2.11.6 and 2.10.11. The project supports the latest patch of the current minor version and the previous minor version, and an older patch loses support once a newer one is out. Minor releases can contain backwards-incompatible changes, so test harvest configuration on the version you intend to run. Confirm which ckanext-harvest version pairs with your CKAN release before you install it.
Python, PostgreSQL and Solr versions
A source install of CKAN expects Python 3.10 or newer, PostgreSQL 12 or newer, and Solr 9, which is the recommended search engine, or Solr 8. Harvested records are written to PostgreSQL and indexed in Solr, so an unsupported database or search version can stall imports even when the job itself starts. Check each version against the CKAN release you selected, and confirm the operating system packages or container images you plan to use for Solr and PostgreSQL are supported with that release.
Redis for the job queue
Redis is a hard dependency because harvest jobs are queued and then picked up by background workers. If Redis is unreachable, jobs can wait unprocessed while schedules still appear to run normally. Confirm that the CKAN worker processes can connect to Redis from the hosts where they run. Decide how Redis is persisted, restarted and monitored, and make sure a Redis outage raises an alert that someone on the platform team will see.
Source catalog access terms
Before you create a harvest source, confirm what the remote catalog allows. Find out whether its CKAN API or DCAT endpoints are public or require an API token, which datasets and organizations are exposed, and whether the operator sets rate limits or licensing conditions on bulk access. Ask the source owner for written permission where its terms call for it. Record the access method and a named contact for the source, so that a future failure has a clear owner.
Outbound network and TLS trust
The CKAN worker hosts must open outbound HTTPS connections to every source endpoint, so egress firewall rules and any forward proxy need to allow those hosts and ports. If the source uses a certificate issued by an internal or private certificate authority, add that root to the trust store used by the CKAN worker's Python environment. Test each endpoint from the worker host itself, since an administrator workstation often has different routes and trust settings.
Account rights for harvest sources
Creating, editing and deleting harvest sources usually requires an elevated role in CKAN. Confirm which role your ckanext-harvest version and authorization configuration require, and use a named, auditable account for this work rather than a shared login. Decide in advance who may start jobs manually, who may change schedules and who reviews job results. Write those roles down before the first job runs so that access questions do not block the first run.
How CKAN harvesting moves datasets through gather, fetch and import
A harvest source is the configuration record in your CKAN portal that points at a remote catalog. It holds the source URL, the harvester type, the owning organization, and the schedule or frequency that decides when runs are created. Each run of a source creates a harvest job, which is a database record with its own status and timestamps. The job is then placed on a Redis queue, and a CKAN worker process picks it up and runs the stages in order.
For the platform team, the important point is that the source is configuration and the job is state. Editing a source changes what later runs do, while a job that already exists keeps the settings it was created with.
- The source defines the remote catalog URL, harvester type and owning organization.
- Each scheduled or manual run creates one job record with a status.
- Redis holds the queue of pending work, and workers consume it.
- Job status and timestamps are the first place to check when a run looks stuck.
- Confirm the exact job fields and queue names against the installed ckanext-harvest version.
Before relying on any of this, check the queue and worker setup in your environment, because the stage behavior described here should be verified against the installed ckanext-harvest version.
The gather stage asks the remote catalog which datasets exist and records their identifiers. It reads the source URL and the harvester's listing logic, and it writes harvest object records that each represent one remote dataset to be processed. Gather does not download full dataset content, so its output is a list of work items rather than imported metadata.
Gather failures usually come from the remote side or the configuration: an unreachable endpoint, a changed listing format, or credentials that the source no longer accepts. Because gather runs first, a failure here means no datasets from that run are fetched or imported, and the job ends in an error state with the gather output the only record of what was found.
- Reads: the source URL, harvester settings and the remote listing.
- Writes: one harvest object per discovered remote dataset, linked to the job.
- Fails on: unreachable hosts, timeouts, authentication errors and unexpected listing formats.
- Effect of failure: the job stops early and no fetch or import work is created.
- Check: the number of objects created against the number you expect from the remote catalog.
When a run creates no objects, examine the source URL and network path before looking at the import logic, since the problem is almost always upstream of the import stage.
The fetch stage retrieves the full record for each harvest object from the remote catalog. It reads the object's remote identifier and the source settings, and it writes the raw remote content back to the object so the import stage can process it. Fetch is where most per-dataset failures appear, because each object is a separate request and one bad record does not have to stop the others.
For retries, the behavior depends on which objects failed and how the harvester recorded the error. Operators should expect that a partly failed job can contain both successful and errored objects, and that the errors on individual objects are the detail needed to decide whether to rerun the source or fix the remote data.
- Reads: the harvest object, its remote identifier and the source connection settings.
- Writes: the retrieved remote content, stored against the harvest object.
- Fails on: remote timeouts, missing records, rate limits and malformed responses.
- Effect of failure: the affected object records an error while others can still proceed.
- Check: the error text on each object and the ratio of fetched to gathered objects.
Review whether retries rerun only failed objects or the whole job, because that changes how much load you place on the remote catalog. Confirm this against the installed ckanext-harvest version.
The import stage turns fetched content into CKAN datasets. It reads the stored remote content and the harvester's mapping rules, then creates or updates datasets in the owning organization. Import is the only stage that changes the portal's dataset records, so it is where validation problems, duplicate detection and schema mismatches appear.
Import failures are often data problems rather than infrastructure problems. A record can be fetched successfully and still be rejected because a required field is missing or a value does not match the portal's schema. Those rejections should be logged against the harvest object so teams can tell remote data quality issues from platform faults.
- Reads: the fetched remote content and the source's mapping and ownership settings.
- Writes: new or updated datasets in CKAN, linked back to the harvest object.
- Fails on: validation errors, schema mismatches, duplicate or conflicting identifiers and permission problems.
- Effect of failure: the dataset for that object is not created or updated, and other objects continue.
- Check: the per-object validation errors and whether datasets changed after the run.
Treat import errors as a review queue for the remote publisher or the mapping configuration, not only as a platform incident. Confirm the exact mapping and validation behavior in the version of ckanext-harvest you run.
Choosing between a CKAN source harvester and the DCAT RDF harvester
The first decision in harvesting from another catalog is which path to build on: a CKAN source harvester through ckanext-harvest, or the RDF harvester in ckanext-dcat that imports DCAT metadata from a catalog publishing it. The choice depends mainly on what the source can serve and how much metadata you need to keep intact. Start by checking what the source actually exposes. If the other catalog is a CKAN site, the CKAN source path lets you work with its native package model, which is the closest match. If the catalog publishes DCAT RDF from a national or European portal, or from a non-CKAN platform, the RDF path is the one that reads what it offers.
Metadata fidelity is where the paths differ most. A CKAN-to-CKAN harvest carries CKAN's own fields, resources and extras across with fewer translation steps, so fewer properties are reshaped or dropped. An RDF import depends on how the source models its datasets, and every mapping from DCAT terms into your CKAN schema is a place where fields can be lost or interpreted differently. Before committing, pull a sample of records from the source and compare required and recommended properties against the profile your portal publishes, such as the DCAT-AP version your organization has adopted. Confirm that the version your ckanext-dcat installation supports covers that profile.
Version support also shapes the choice. The ckanext-dcat extension is supported on CKAN 2.10 and 2.11, and the researched facts for this topic do not confirm support on newer CKAN minor releases. If your portal runs CKAN 2.12 or plans to, confirm the ckanext-dcat compatibility for that release with the extension maintainers before choosing the RDF path, and plan the upgrade sequence so harvesting and DCAT endpoints move together. The CKAN source path depends on ckanext-harvest compatibility with your CKAN release, so check both extensions against your target version rather than assuming they upgrade in lockstep.
The sibling topic on CKAN DCAT-AP metadata and RDF endpoints covers RDF structure, the DCAT-AP v1 to v3 schemas and the RDF harvester configuration in detail, so use it for field mapping and endpoint settings once you have chosen this path. This section stays at the decision level. Record the choice with its reasons: the source type, the metadata properties you must keep, the profile version required, and the CKAN version you will run for the foreseeable future.
A practical check before you decide is a short proof of concept against one representative source. Harvest a small set, inspect the results in your portal, and confirm that ownership, licensing, resource URLs and contact details arrive as expected. Ask any implementation partner to deliver this comparison in writing with the scope, so the choice and its evidence are documented before the full build starts.
Setting up CKAN harvesting from source to first successful job
This sequence takes a CKAN instance from a working install with Redis to a harvest source that has completed one gather, fetch and import cycle and is scheduled to repeat. Each step ends with a check so the team can confirm the pipeline works before scheduling it.
Confirm the CKAN and Redis baseline
The platform engineer confirms that the CKAN instance runs on a supported Python, PostgreSQL, Solr and Redis combination for the installed version. Source installs require Python 3.10 or newer, PostgreSQL 12 or newer, Solr 9 (recommended) or Solr 8, and Redis. Check that the CKAN web interface loads, that the Redis service accepts connections from the CKAN host, and that the CKAN version matches the ckanext-harvest release you plan to install. Confirm the exact compatibility matrix in the ckanext-harvest documentation for that release.
Install ckanext-harvest into the CKAN environment
The platform engineer installs the ckanext-harvest package into the same Python environment that runs CKAN, using the version documented for the installed CKAN release. Pin the version in the deployment's requirements so the extension survives rebuilds and upgrades. The step is complete when the package is listed in the environment and CKAN starts without import errors in its logs. Confirm the install method and any extra requirements in the ckanext-harvest documentation for your version.
Enable the plugin and create harvest tables
The platform engineer adds the harvest plugin to the CKAN plugins setting, then runs the database initialization command that ckanext-harvest documents for creating its tables. Restart CKAN so the plugin loads. The check is that the harvest source pages appear in the CKAN admin area and that the database contains the harvest tables. If the tables are missing, the initialization step did not run against the correct CKAN database. Confirm the plugin key and initialization command in the extension documentation.
Start the gather and fetch consumers
The platform engineer starts the harvest background consumers that process gather and fetch jobs through Redis. These run as separate processes from the web application, so they need the same configuration and environment as CKAN. The check is a log line from the consumer process showing it connected to the queue without errors, and a queue status that shows no stuck jobs. Confirm the consumer command, process supervision approach and queue settings in the ckanext-harvest documentation for the installed version.
Create a harvest source for a test catalog
The data steward creates a harvest source in the CKAN admin interface for a test catalog, choosing a clear name, the source type that matches the catalog's metadata format, and the catalog's URL. Use a small, low-risk catalog for the first run so that errors are easy to trace. The check is that the source appears in the harvest source list with a status of new or idle and no jobs yet. Confirm the supported source types and the URL format for each type in the ckanext-harvest documentation.
Run a manual gather and fetch
The platform engineer triggers a manual harvest job for the test source from the CKAN admin interface or the documented command line option. The gather stage lists the remote dataset identifiers, and the fetch stage retrieves each record. The check is a job that moves through gather and fetch with a finished status, plus a count of gathered objects in the job details that matches what the test catalog publishes. Review the consumer log for per-record errors before moving on.
Verify the first import and dataset count
The data steward confirms that the import stage created or updated datasets in the local CKAN catalog and that the dataset count matches the job's reported result. Open several imported datasets and compare titles, resource links and metadata fields with the source catalog. The check is a successful job status, a dataset count that reconciles with the gathered objects, and no unexplained errors in the job report. Record the job identifier and the results so later runs have a baseline.
Schedule the recurring harvest job
The platform engineer configures the recurring run for the source using the scheduling method documented for the installed ckanext-harvest version, such as a periodic task managed by the deployment's scheduler. Set the frequency based on how often the upstream catalog changes and how much load the CKAN and Redis hosts can carry. The check is that the next scheduled job appears in the job list and that the first automatic run completes with a successful status and a dataset count consistent with the manual run. Document the schedule, the owner and the alert path for failed jobs.
How source catalog fields map to CKAN datasets, organizations and resources
This table lists the common fields in a remote catalog record and the CKAN target each one maps to, with the decision the team makes and the check that confirms it. Read it before writing the harvester mapping, then repeat the checks against a sample harvest from the real source. The main takeaway is that licenses, identifiers and contact data are where most mismatches appear, so they need the most verification.
| Source field | CKAN target and decision | Verification step |
|---|---|---|
| Title | Dataset title. Keep as is, but trim whitespace and check length against the CKAN field limits in your version. | Compare 20 harvested titles with source pages and confirm no truncation or encoding errors. |
| Description | Dataset notes. Keep, or transform if the source uses Markdown or HTML that renders differently in CKAN. | Open three harvested datasets and confirm formatting and links display correctly. |
| Publisher | Organization. Transform by matching to an existing CKAN organization, or fix at source if the publisher name is inconsistent. | Confirm each source publisher maps to exactly one organization and no duplicate organizations are created. |
| License | License identifier. Transform to a CKAN license ID where possible. Drop or fall back to a generic value if no match exists. | Confirm the mapping against the actual source catalog output, since source license strings often do not match CKAN's license list. |
| Keywords | Tags. Keep after normalizing case and removing characters CKAN rejects. Drop generic placeholder keywords. | Check tag counts per dataset and review the most frequent tags for unwanted values. |
| Spatial extent | Extra or custom field. Keep as a field if the source uses a geometry format, and confirm the portal's schema accepts it. | Validate that geometry strings parse and display or search as expected in the portal. |
| Temporal extent | Extra fields for start and end dates. Transform date formats to a single ISO form before import. | Sample records with missing or open-ended ranges and confirm they import without errors. |
| Contact point | Maintainer and author fields. Keep the name and email where the source provides them, and drop personal data the portal does not need to publish. | Confirm contact fields match the source record and that no unexpected email addresses are exposed. |
| Resource URLs | CKAN resources. Keep each downloadable or API URL as a resource, and fix broken or relative links at source where possible. | Check that each resource URL resolves and that format and media type values are set. |
| Identifiers | Dataset name or extra identifier. Transform into a stable, unique slug so reruns update the same dataset. | Run the harvest twice and confirm the second run updates datasets rather than duplicating them. |
Securing harvest sources, credentials and imported content
A harvester pulls metadata from systems you do not run, so every harvest source is an inbound trust decision. The controls below cover how the CKAN host reaches those sources, how credentials are stored, which accounts can change harvest jobs, and how imported records are reviewed before they reach the public portal. Each one is a design choice for your own security controls, and each should be confirmed in your environment before go-live.
Outbound-only network rules
Harvesting requires the CKAN worker hosts to open outbound connections to each source catalog, and nothing else needs to reach them from outside. Restrict egress to a named list of source hostnames and ports, and block access to internal address ranges so a misconfigured source URL cannot probe your own network. Confirm the allowlist with the owner of each source, and recheck it whenever a source moves to a new host.
TLS verification of source endpoints
A harvester that accepts unverified certificates can be fed content by anyone able to intercept the connection. Keep certificate validation enabled for every source, and do not disable it to clear a failing job. Where a source uses an internal certificate authority, add that authority to the worker trust store rather than turning verification off. Test a deliberately broken certificate in staging to confirm that the harvest fails.
Source API keys outside plain configuration
Some source catalogs require API keys or tokens, and these should not sit in CKAN configuration files, harvest source descriptions or version control. Store them in a secrets manager or the platform's encrypted secret mechanism, inject them into workers at runtime, and rotate them on a schedule you define. Confirm which secrets appear in logs or the admin interface, since harvest error messages can echo configuration fields.
Least-privilege harvest owners
Each harvest source has an owning organization and user, and that account decides what the harvest can create or overwrite. Give harvest owners only the organization roles needed for their sources, avoid sysadmin rights for routine harvest management, and keep a short list of people who can create or edit sources. Review that list when staff change roles, because a harvest source can publish content under the owner's name.
Malformed and untrusted metadata
Harvested records come from outside your governance, so they can contain oversized fields, broken identifiers, misleading titles, unexpected licenses or links to unwanted sites. Limit the risk by reviewing source quality before adding it, setting an import scope that matches the datasets you want, and treating each new source as untrusted until its first few jobs are inspected. Malformed records should fail visibly instead of being silently accepted.
Review before publication
Decide whether imported datasets go live automatically or land in a review state first. A review step lets a curator check titles, license terms, resource links and organization attribution before public release. Document who approves imports, how rejected records are handled, and how a bad batch is removed or reverted. Confirm that the CKAN version and extensions you run support the publication workflow you choose.
Separate portal sign-in from harvest access
Portal user sign-in and harvest source access are separate concerns. If your portal uses ckanext-saml2auth, that extension governs how people log in to CKAN and how their organizations and roles are assigned from SAML2 attributes. It does not control which external catalogs the harvester may contact or which credentials it uses. Plan and audit both paths independently, and confirm the boundary in your own identity design.
Testing a harvest and monitoring scheduled runs in production
Verify a harvest against a sample source before any of its output reaches the public portal, and keep monitoring scheduled runs after go-live. A harvest that imports the wrong records or silently stops is a publishing error, so the test plan should be written before the first scheduled job is enabled. Start with a single harvest source configured with a small, bounded slice of the upstream catalog, such as one organization or a limited set of datasets. Run it through all stages (gather, fetch and import) and confirm the job finishes with a clear status rather than a partial result.
Compare the output with the source catalog. Count the datasets the source reports against the datasets created or updated in your portal, and account for any that were skipped. Then open a sample of records side by side and check the title, description, organization, license, tags, resource URLs and extras. Pay particular attention to fields that depend on mapping. If you use ckanext-dcat's RDF harvester, review the mapping output for each source format, because a field that maps correctly for one upstream catalog may be empty or misplaced for another. Record the mismatches and decide, for each, whether to fix the mapping or accept the difference and document it.
Test deletions and updates on a second run. The first run only shows that records can be created. Change one or two records at the source, remove one, and run the harvest again. Confirm that the changed record is updated in place, that the removed record is handled according to the source's deletion setting, and that no duplicates appear. Repeat the check after a full re-run to confirm the process is idempotent. Use a staging instance that runs the same CKAN, extension and Redis versions as production, so the results carry over. Keep test sources and test records out of the public portal. Confirm the staging environment is not publicly reachable or indexed, and remove test datasets before any production run.
Monitor scheduled jobs from the start. Track each job's status, its start and end times, and the error log entries it produces. Failures that repeat across runs usually point to a source change, an expired credential or a network restriction, so alert on repeated failures rather than on each single error. Watch the Redis queue as well. A growing backlog means jobs are being scheduled faster than workers can process them, or that workers have stopped, and the portal will look current while showing stale data. Confirm which team receives alerts, how quickly they must respond, and who can disable a source that is misbehaving.
Confirm with your platform team and the extension documentation which job status fields, log locations and Redis queue names apply to the CKAN version you run, since these details change between releases. Ask DeployOpen to include the verification checklist, the monitoring thresholds and the alerting rules in the written scope, so the acceptance criteria for the harvest are agreed before work begins.
Failed CKAN harvesting jobs and how to diagnose them
This table pairs common ckanext-harvest failure symptoms with their likely causes and the first check to run. Read each row as a starting point: confirm the exact error text, job state and stage in the harvest logs and job list before changing the source configuration. The main takeaway is that most failures come from the source, the network path, the mapping or the worker processes, and each can be isolated with one targeted check.
| Symptom | Likely cause | First check |
|---|---|---|
| Job fails at the gather stage with a connection or timeout message | Source URL unreachable, blocked by a firewall or proxy, or the source portal is down | Confirm the source URL responds from the CKAN host with the same outbound path the worker uses, then review the gather error in the harvest logs |
| Job fails with a certificate verification message | TLS certificate on the source is expired, incomplete or issued by an authority the CKAN host does not trust | Inspect the certificate chain from the CKAN host, then check whether the internal trust store used by the worker includes the issuing authority |
| Job completes but no datasets are created or updated | Source returns a format the harvester does not parse, such as an HTML error page, a login redirect or an unexpected DCAT serialization | Fetch the source endpoint directly and confirm the response type and structure match what the configured harvester expects |
| Datasets are imported with required fields empty | Mapping gap between source metadata fields and the CKAN schema, often after the source renames or moves a field | Compare one harvested dataset against its source record and check the mapping for the fields that are blank |
| Jobs stay in a pending or running state and never finish | Redis is not reachable, or the harvest consumer process is not running on the worker | Check the Redis connection from the CKAN host, then confirm the harvest consumer process is running and review its logs for startup errors |
| Same dataset appears twice after a harvest run | Source changed the identifier of existing records, so the harvester treats them as new | Compare the source identifiers for the duplicated pair and check whether the source changed its URI or identifier scheme |
| Datasets deleted at the source remain in CKAN | Source no longer lists the record, but deletion handling is not enabled for that source or the job ended before the removal step | Confirm the source configuration covers removal of records, then check the job report for the removal step in the harvest logs |
| Job status shows an error with no clear message | Failure in the fetch or import stage that the summary does not expose | Open the full job object and the worker log for the same job ID, and record the stage where it stopped before changing any setting |
What CKAN harvesting costs and what the work needs from your team
What drives the effort of a CKAN harvesting project?
The effort depends mainly on five factors: the number of source catalogs, the quality of each source's metadata, how much mapping is needed, the CKAN version and hosting model, and the depth of security review.
Each additional source adds its own connection settings, schedule, test cases and mapping decisions. Sources with complete, consistent metadata need less cleanup than sources with missing licenses, free-text dates or ad hoc keywords. Mapping work grows when your target schema differs from the source schema or when you need DCAT-based RDF import. Your CKAN version and hosting choices set the install and test work. Security review adds effort when the portal handles sensitive data or sits under a formal control framework. Ask the vendor to show how each factor was counted in the written scope.
How is CKAN harvesting cost split between setup and ongoing work?
Setup is mostly one-time, and upkeep recurs for as long as the portal runs.
One-time work usually includes environment preparation, harvester configuration for each source, mapping rules, test harvests, monitoring setup and handover documentation. Recurring work includes CKAN patch and minor upgrades, extension updates, responses to source changes such as a new schema version or moved endpoint, and periodic review of failed harvest jobs. Recurring cost rises with the number of sources and with how often those sources change. In a quote, ask for the one-time and recurring amounts as separate line items, the upgrade responsibilities for each, and whether source-change requests are included or billed as they occur.
How long does a CKAN harvesting project take?
The duration depends on the same drivers as the cost: the number of sources, metadata quality, mapping complexity, CKAN version and hosting, and security review. No standard duration applies to every portal, so the timeline should come from the written scope.
Most engagements are fixed-scope, which means scope, deliverables and timelines are agreed in writing before work starts. Ask for the timeline broken into phases: environment readiness, source onboarding, mapping and testing, go-live and monitoring. Also ask which phases depend on client input, such as access approvals or decisions from data owners, because waiting on those inputs is a common cause of delay. A schedule that does not show these dependencies is hard to check.
How do CKAN releases affect a harvesting setup over time?
CKAN releases about two minor versions a year, and each minor release may include backwards-incompatible changes. Patch releases do not break backwards compatibility and do not include schema changes or new features.
The CKAN team supports the latest patch of the current minor version and the previous minor version for security and bug fixes. Older patch releases lose support once a newer patch is out. CKAN 2.12.0 was released on August 26, 2026. The ckanext-dcat extension, which provides the RDF harvester, is supported on CKAN 2.10 and 2.11, so confirm extension compatibility before planning any minor upgrade. Ask your provider how upgrades are tested against your harvest jobs and who is responsible for each upgrade.
What does our team need to provide for a CKAN harvesting engagement?
Your team needs to supply access, decisions and test material early, because most delays trace back to these inputs.
Provide administrative access to the CKAN instance and to the hosting environment, including PostgreSQL, Solr, Redis and the web tier. Identify the source catalogs to harvest and confirm in writing that you may harvest them. Supply test sources or sample harvests so mapping can be checked before go-live. Name the data owners who decide how fields such as licenses, publishers and keywords should map to your schema. Name one approver who can accept the source list and the acceptance criteria. Confirm the hosting location and any restrictions on where data may be stored or processed.
Can an internal team run CKAN harvesting without outside help?
Yes, an internal team can do this if it has the skills for the whole stack and the capacity to keep it current. Whether it can do so alone depends on that capacity more than on any single skill.
The install requires Python 3.10 or newer, PostgreSQL 12 or newer, Solr 9 (recommended) or Solr 8, and Redis, with uWSGI and Nginx recommended for production. The team must also be able to diagnose failed harvest jobs, manage Redis queues, handle mapping errors and perform CKAN upgrades on the release cadence. Teams that have done this before may run it alone. Teams new to CKAN often use a partner for the first build and then take over operations. Assess your team against each of these tasks before deciding.
What should we ask for in a CKAN harvesting quote?
Ask for a written scope that makes each cost driver visible, so you can compare quotes on the same basis.
Request the number of sources in scope and what each one requires, the assumed CKAN version and hosting model, the mapping rules and who approves them, and the test and acceptance criteria. Ask for the one-time and recurring costs as separate items, with the upgrade and source-change terms stated. Confirm what is excluded, such as new sources added later, extension development or security testing. Ask how timelines depend on client inputs, and which deliverables the quote includes, such as runbooks, monitoring configuration and handover sessions. A quote that omits these items is not yet comparable.
Does security review change the effort for CKAN harvesting?
Yes, when the portal or the harvested data falls under a formal control framework, because the review adds controls, evidence and testing work. Start by confirming whether any framework applies.
NIST SP 800-53 Revision 5 is the federal control catalog, and its SP 800-53B baselines select different numbers of controls at the low, moderate and high levels. FedRAMP covers cloud products that handle federal information on behalf of an agency, so a hosted CKAN portal may or may not fall within it depending on how it is offered. Controlled unclassified information on non-federal systems is governed by NIST SP 800-171 under federal rules. Ask your compliance lead which framework applies before the scope is written. DeployOpen prepares systems, controls and evidence for assessments, but independent auditors and certification bodies issue the audit reports.
How DeployOpen works on this
DeployOpen plans, builds, documents and supports CKAN harvesting from other data catalogs inside your own infrastructure. The work starts with a written, fixed scope that names the sources, harvest jobs, metadata mapping and monitoring you need, so you know what is included before any build begins.
How we scope
- Inventory every source catalog, its CKAN version, API access and harvest type before agreeing a build.
- Record which datasets, organizations and metadata fields each harvest source must bring into your portal.
- Confirm the written scope, deliverables and timeline with you in writing before work starts, since engagements are fixed-scope.
How we build
- Configure ckanext-harvest sources and jobs, with Redis queues and scheduling set up on servers you control.
- Map harvested metadata to your local schema and test the gather, fetch and import stages against real source samples.
- Check that failed jobs, duplicate datasets and removed upstream records are handled as the scope defines.
How we hand over
- Document each harvest source, its schedule, mapping rules and the steps to add or retire a source.
- Deliver runbooks for monitoring harvest job status, reading errors and rerunning failed jobs.
- Walk your operators through the configuration and confirm they can run the harvest workflow without us.
How we support
- Offer ongoing support and CKAN upgrade planning as a retainer, covering patch and minor release changes.
- Review harvest logs and source changes on a schedule agreed in the retainer scope.
- Confirm with us in advance which CKAN version and extension compatibility applies before any upgrade.
