Importing CSV, JSON and XML content into Drupal: what the move involves
Loading structured content from spreadsheets, APIs or exports into Drupal is a build task with a repeatable loading step, and the effort sits mostly in the decisions made before anything is loaded. The Migrate API in Drupal core handles the reading, transforming and saving of records, Migrate Plus adds source plugins for remote and file-based data such as JSON and XML feeds, Migrate Source CSV provides the CSV source plugin, and Migrate Tools supplies the Drush commands, such as migrate:status, that run, monitor and roll back each migration. Drupal itself has no paid edition, and Drupal and these contributed modules are licensed under GPL-2.0-or-later, so the cost of this work is people's time, hosting and testing rather than licenses. The modules are documented on Drupal.org, including Migrate Plus, Migrate Source CSV and Migrate Tools.
The loading mechanics are the smaller part of the work. Before a run, the team has to decide which Drupal content types, fields, taxonomy terms and media entities each source record becomes, how multi-value and reference fields are resolved, and which source identifier keeps records stable between runs. Those decisions are what determine the size of the job, along with the number of source files, how clean the data is, and how many records need manual correction. Ask any partner to show the field mapping for each source before quoting, and to state which fields are required, which are derived, and which will be dropped.
Plan validation as a deliverable, not a final check. Each import needs a test run on a copy of production-like data, a reconciliation of record counts against the source, spot checks of rendered pages, and a tested rollback using the Migrate Tools commands. Incremental runs, where changed source records update existing Drupal entities, need a stable unique key and a rule for deleted source records. Confirm with the partner how many test cycles the scope includes, who signs off the mapping, and who owns the source exports after go-live.
Which source structures map cleanly into Drupal and which need rework
This table compares common structures in CSV, JSON and XML exports against Drupal entities, fields, references, media and translations. Read each row as a question to answer about your own export before you write any mapping, because the rows marked for rework usually decide the content model. The main takeaway is that flat, single-valued data maps directly, while nested, multi-valued and revisioned data needs a model decision first.
| Source structure | Drupal target | Mapping fit | What to decide first |
|---|---|---|---|
| Flat CSV rows with one value per column | Content entity with scalar fields (text, number, date, boolean) | Maps directly | Confirm each column's data type and date format before the first run |
| Repeated columns such as tag1, tag2, tag3 | Multi-value field or taxonomy reference field | Maps with transformation | Decide a maximum count and how empty values are skipped |
| Nested JSON objects | Paragraphs, a separate entity or flattened fields | Needs a model decision | Choose whether the nested object is owned content or a shared reference |
| XML hierarchies with repeating child elements | Child entities linked by entity reference, or Paragraphs | Needs a model decision | Decide the parent record that owns each child and the ordering rule |
| Lookup codes or IDs pointing to other records | Entity reference field to a node, user or term | Maps only after target import | Import referenced records first and keep a stable source ID for lookups |
| Attachments or image URLs | Media entity or file field | Maps with transformation | Confirm whether files are fetched, copied or only linked, and where they are stored |
| Categories or controlled vocabularies | Taxonomy terms with a reference field | Maps with transformation | Agree which values become terms and who maintains them after launch |
| Multilingual rows or translation columns | Translated entities through the content translation modules | Needs a model decision | Confirm the language set and whether each translation is a separate row or field |
| Revision history or version columns | Entity revisions, which most imports do not recreate | Usually does not map | Decide whether only the latest version is kept and where older versions are archived |
Which documented importers handle CSV, JSON and XML sources
Migrate Source CSV is the contributed module that provides a CSV source plugin for the core migration framework. It reads a flat file row by row and hands each row to a migration definition, which maps columns to Drupal fields, entity references and taxonomy terms. It suits spreadsheet exports, simple product or contact lists and one-off loads where the file has a stable header row.
- Column mapping: Each source column becomes a source field in the migration YAML, so header names and their order must be fixed before the first run.
- Encoding and delimiters: Confirm the delimiter, quoting and character encoding of the export, since spreadsheet tools often change them silently.
- Keys for updates: Define a unique source identifier so reruns update existing entities instead of creating duplicates.
- Multi-value fields: Decide how repeated values inside one cell are split and whether the separator can appear in the data.
- Large files: Check memory and batch settings against file size before a full run.
Before you commit, confirm the Migrate Source CSV release that supports your Drupal core version and PHP version, and test one full run in a non-production copy of the site.",
Migrate Plus extends the core migration framework with additional source plugins, including sources for remote JSON and XML feeds. It lets a migration pull from an API endpoint or a file, and it supports nested data by selecting the item path that repeats for each record. It is the usual route when the source is an API export or a feed rather than a flat file.
- Item selectors: Identify the path to each record in the JSON or XML document, since nested arrays often hold the content you need.
- Pagination and authentication: API sources may need page parameters, tokens or headers. Confirm how the source handles them and how it behaves when a page fails midway.
- Namespaces in XML: XML feeds with namespaces need explicit handling in the selectors, which is a common source of empty imports.
- Remote sources: Caching and retry behavior depend on the source configuration, so test against the live endpoint and not only a saved sample.
- Schema drift: Source fields can be renamed or removed by the publisher, so add checks that fail loudly when expected fields disappear.
Confirm which Migrate Plus release is compatible with your target Drupal core release and check its issue queue for open problems affecting your source format.",
When the source is a database, a proprietary export, a SOAP service or a format no contributed plugin reads, the core Migrate API lets you write a custom source plugin in PHP. The plugin yields rows and the migration definition handles mapping, rollback tracking and incremental state, so the custom code stays focused on reading the source. This route costs more to build but gives full control over paging, error handling and data shape.
- Plugin responsibilities: The plugin must return a stable identifier for each row, which the migration map uses to track what was created.
- Custom code ownership: The plugin lives in a custom module that your team or a partner must maintain across Drupal upgrades.
- Testing: Write kernel or functional tests for the plugin so changes in the source are caught before a production run.
- Documentation: Record the source contract, including field meanings and null handling, for the team that owns the export.
- Upgrade cost: Core API changes between major versions can affect custom plugins, so schedule their review in any upgrade plan.
Ask whether the source owner can provide a stable export or API contract, because a clean contract reduces custom code more than any choice of module.",
Migrate Tools adds Drush commands for running and managing migrations. The command migrate:status lists each migration with its source and destination counts and status, which is the first check after any run. Other commands run imports, roll back imported items and reset stuck migrations, so an operator can control a load without using the interface.
- Status checks: Use
migrate:statusbefore and after every run to compare imported, unprocessed and failed counts against the source. - Imports and rollbacks: Run the import command for a named migration, and use the rollback command to remove items from a prior run when a mapping is wrong.
- Stuck migrations: The reset command clears an interrupted import state so the next run can start cleanly, which should be used only after checking why the run stopped.
- Scheduling: Run imports from cron, a deployment pipeline or a job runner, and log output so each run can be audited.
- Incremental runs: Confirm that the source and map tracking support updates and deletions, since not every source reports removed records.
Confirm the Migrate Tools release that matches your Drupal core version and verify each command's options in your environment before writing runbooks around them.
How the import runs from inventory through parallel run and cutover
This sequence turns a set of CSV, JSON or XML exports into a verified Drupal dataset, with each stage gated by a check that the next stage depends on. It ends with a cutover the business has signed off and a source system that is retired only after the parallel run agrees with it.
Inventory source files and fields
Your team and DeployOpen list every export, API endpoint or XML feed, its owner, refresh frequency, record counts and field definitions, including codes, date formats, encodings and records that reference other records. The step is done when each source has a named owner and a written field list that the business has reviewed. Ask for sample files that include known bad rows, because they show where validation rules will be needed.
Design content model and field mapping
Source fields are mapped to Drupal content types, fields, taxonomy terms, media and entity references, with a decision recorded for each transformation such as splitting, merging, lookups and default values. Stable unique identifiers from the source are kept as a migration key so reruns update existing entities instead of duplicating them. The output is a mapping specification that the content owners approve before any migration code is written.
Build migrations with Migrate Plus and Migrate Source CSV
Migrations are defined as configuration that reads the source through Migrate Plus source plugins, the CSV source provided by Migrate Source CSV, or a custom source plugin for APIs and nested XML. Code is reviewed and stored in version control with the mapping specification alongside it. The step closes when a developer can run each migration against a development copy and the mapping matches the approved specification.
Run a pilot import on a sample
A representative subset, including the known bad rows, is imported into a non-production environment. Teams check entity counts, field values, references, files and any records that failed validation, using the migrate:status command from Migrate Tools to read the state of each migration. The pilot is complete when content owners sign off on a sample of rendered pages and the failed-row report has an agreed disposition for each error type.
Execute full runs and confirm rollback
The full migration runs in a staging copy of production, with migrate:status checked before and after each run to confirm processed, imported and failed counts match the source extract. Before the first full run, the team tests the rollback command on the same migration so that a bad run can be reversed cleanly. The step ends with a recorded rollback test and a run log that the team can reproduce.
Run incremental syncs and parallel comparison
While the source stays live, incremental runs pick up changed records using the migration key and the source change marker, and each run is checked with migrate:status. A parallel run compares source records against Drupal output on agreed fields and counts, and discrepancies are traced to mapping rules or source data. The step is done when the discrepancy list is empty or each remaining item has a documented owner and decision.
Cut over and freeze the source
The source is set to read-only, a final incremental run is executed, and the parallel comparison is repeated on the final delta. Business owners confirm the cutover checklist, including DNS, integrations that consumed the old feed and the rollback plan if the new site must be reverted. Cutover is complete when the final comparison passes and the sign-off is recorded.
Retire the source after a retention period
After Drupal is live, the old export jobs are disabled and the source is kept read-only for a retention period set by your records policy, with an archived copy of the final extracts. Confirm with your compliance and legal teams what must be retained and for how long before anything is deleted. The step closes when the archive is verified and the decommission record is signed.
How rollbacks and incremental runs are designed in
Rollback and rerun behavior does not come from a single setting. It comes from five components that have to be designed together before the first import runs. Each one protects against a different failure, and most of them depend on the same decision: what identifier each source record carries.
Migration map of source IDs
The migration map (the ID map table that Migrate keeps for each migration) records which source record created which Drupal entity. Without it, a rerun cannot tell an update from a duplicate, and a rollback cannot find what to delete. Decide which source field is the unique key, confirm it never changes or gets reused in the export, and check that the key is stored as a string so leading zeros in SKUs or codes survive.
Rollback of imported entities
Rollback removes the entities a migration created, using the map to find them. It protects against a bad mapping or a wrong file reaching production, but only for entities still tracked in the map. Records edited by editors after import may be deleted along with the rest, so agree on whether rollback is allowed after content goes live. Test the rollback on a copy of the production database before the real run.
Incremental runs for changed records
An incremental run imports only records that are new or changed since the last run, which keeps regular feeds from rewriting the whole site. It depends on a reliable change marker in the source, such as a last-modified column or a changed timestamp in the API response. Ask the data owner whether deletions are exported at all, because a row that disappears from a CSV is not an instruction to remove the node.
Validation scripts before import
Validation checks the file before Migrate reads it: required columns present, identifiers unique, dates and enumerated values in the expected form, and referenced taxonomy terms or users existing. It protects against partial imports that fail halfway and leave the map inconsistent. Decide who owns the rules and whether a failing file stops the run or only flags rows for review.
Logging for reruns
Logs record each run's counts of created, updated, failed and skipped rows, along with the source file version and the Drush command used. Migrate Tools reports status through commands such as migrate:status, and failed rows should be written to a reviewable file. Keep logs with the release record so the team can explain what a rerun changed.
What drives the cost and duration of a structured import
This table compares the main drivers of import effort, whether each one is paid once or returns with every run, and what to ask for in a quote. Use it to see where a CSV, JSON or XML import will spend most of its budget before you compare partner proposals.
| Driver | Why it changes effort | One-time or recurring | What to ask for in a quote |
|---|---|---|---|
| Number of content types | Each content type needs its own migration class, field mapping and test pass, so effort grows with the count. | One-time | A list of content types with a mapping and test plan for each |
| Source data quality | Duplicate keys, mixed date formats and empty required fields add cleaning rules and manual exception review. | One-time, with recurring cleanup if the source keeps producing bad rows | Who fixes source defects, and whether cleaning happens in the file or in the migration |
| Field count and complexity | Multi-value fields, nested JSON and XML repeating elements need more transform logic and more checks. | One-time | Field-level mapping sheet showing source path, target field and transform |
| References between records | Taxonomy terms, authors and parent nodes must be imported in dependency order and matched by a stable key. | One-time | Migration order, lookup strategy and how missing references are reported |
| File and media handling | Remote images or attachments must be fetched, validated and linked, which adds storage and run time. | One-time for setup, recurring for large refreshes | Whether files are copied or linked, and where media storage will live |
| Multilingual content | Translations need their own rows, language mapping and checks on fallback and URL aliases. | One-time | Languages in scope and how translated rows are identified in the source |
| Custom transforms | Business rules, lookups against external services or custom plugins need code, review and tests. | One-time, with upkeep when Drupal or modules change | Written transform rules and the custom code included in deliverables |
| Number of incremental runs | Each recurring run needs change detection, idempotent updates and a rollback check before it is trusted. | Recurring | Expected run frequency, support window after cutover and who runs the commands |
| Editor review effort | Editors must check samples, flagged exceptions and rendered pages before sign-off, and this often takes longer than the code. | One-time per batch, recurring for each refresh | Who reviews, how many sample records they check and the sign-off criteria |
What usually goes wrong with CSV, JSON and XML imports
Most failed structured imports fail for predictable reasons: the source data was never checked against the target model, and the import was never designed to run twice. The five risks below are the ones teams should plan for before the first migration run, not after the first bad batch lands in production.
Inconsistent or duplicated source keys
A spreadsheet or API export may use the same identifier for two records, change identifier formats between extracts, or leave the key blank for some rows. Migrate needs a stable unique key per source row to map each record to exactly one Drupal entity. Before mapping, profile the source for duplicates, empty keys and mixed formats such as leading zeros, and agree with the data owner which field is authoritative. Record that key in the migration's ID map so later runs can update rather than create.
Encoding and delimiter problems
Spreadsheet exports often arrive as Windows-1252 rather than UTF-8, use semicolons or tabs instead of commas, or embed line breaks inside quoted cells. These problems can split a single record into several rows or corrupt accented characters without raising an error. Set the delimiter, enclosure and character encoding explicitly in the CSV source configuration, and test the import against a sample file that contains quoted line breaks, accented names and empty trailing columns.
Schema drift in API and XML sources
An upstream system can rename a field, add a nested object, or change a date format without notice, and the import will keep running while writing empty or wrong values. Pin the expected structure as a documented contract, validate each batch against it, and fail the run when required fields disappear. Ask the source owner for a change notice process and a version identifier in the payload where one exists.
Broken references between records
Articles that point to authors, categories or related products can be imported before their targets exist, leaving empty references or references to the wrong entity. Migration dependencies must be declared so that referenced entities import first, and every lookup should define what happens when the target is missing. Plan a report of unresolved references after each run so the data owner can decide whether to fix the source or skip the row.
Reruns that create duplicates
Running an import a second time without an update strategy can create a full second copy of the content. Incremental runs need a stable source key, an agreed rule for updated and deleted source rows, and a tested rollback that removes only what the migration created. Test a full rerun, a partial rerun and a rollback on a copy of the production database before any content goes live, and confirm the counts match the source.
Questions teams ask before importing content into Drupal
What drives the cost of a CSV, JSON or XML import into Drupal?
The main cost driver is the number of distinct source structures and how cleanly they map onto Drupal content types, fields and references. Each source file or API endpoint needs its own migration definition, field mapping, transformation logic and validation rules. Content with nested data, multiple languages, media files or references to other entities takes longer than flat records. Cost is one-time for building the migrations and any custom source plugin, plus a smaller recurring amount if feeds keep changing. Ask any provider to price each source separately, to state which Migrate Plus and Migrate Source CSV versions they target, and to list what is excluded, such as data cleansing or content rework.
How long does a structured content import take, and why does it vary?
Duration depends less on record count than on the quality of the source data and the number of unresolved decisions. Clean, consistent exports with agreed field definitions move quickly. Duplicate records, inconsistent date formats, missing references and undefined taxonomy terms add investigation and rework. Imports that run against large volumes need time for trial runs, performance tuning and verification passes. Ask for a timeline that separates discovery, mapping, trial imports, verification and cutover, and confirm which Drupal core version and migration module releases the plan assumes, because compatibility can change between releases.
What must our team provide before an import project starts?
The team needs to supply representative sample exports, not only a description of the data. Useful inputs include:
- Full field inventories with data types, allowed values and sample rows from each CSV, JSON or XML source
- Access to the API endpoints or export schedules, including credentials handled under your own secrets policy
- Named content owners who can decide how ambiguous records, duplicates and obsolete content should be treated
- The target content model in Drupal, or agreement on how it will be designed
- Expected volumes, update frequency and a sandbox environment for trial runs
Missing content owners are one of the most common reasons imports stall, because mapping questions need business decisions.
Can an internal team run a Drupal content import without a partner?
Yes, a team with Drupal development skills can build imports with the core Migrate API, Migrate Plus, Migrate Source CSV and Migrate Tools. The work requires PHP and Drupal entity knowledge, familiarity with Drush commands such as migrate:status, and disciplined testing. Teams usually struggle when source data is poorly documented, when volumes are large enough to need performance tuning, or when no one owns the verification step. A hybrid model is common, where internal staff own content decisions and a partner builds or reviews the migration code. Assess your team against those skills before deciding.
Can we roll back a Drupal import if something goes wrong?
Yes, the Migrate API tracks the content it creates, so Migrate Tools can roll back a migration and remove the entities it imported. Rollback is reliable only when the migration keeps stable source identifiers and when content has not been edited after import, because manual edits and dependent content can complicate removal. Test rollback in a staging copy before any production run, and confirm how it handles media files, taxonomy terms and references. Plan a database backup before each production run, since rollback is not a substitute for one.
How do incremental runs work during a parallel run?
Incremental runs process only records that are new or changed since the last run, based on the source identifiers and change tracking the migration defines. During a parallel run, where the legacy system stays live while Drupal is tested, you run the migration on a schedule and re-import updated rows. Decide in advance whether edits in Drupal are overwritten on re-import, and how deletions in the source are handled, because these choices determine whether editors can safely work in both systems. Confirm the exact update behavior for your chosen Migrate Plus and Migrate Tools releases in a test environment.
Which versions should we confirm before the import is designed?
Confirm the Drupal core version first, because the supported PHP version depends on it. Drupal 11 requires PHP 8.3 or later, Drupal 10 requires PHP 8.1 or later, and Drupal 12 requires PHP 8.5 or later, according to the Drupal.org PHP requirements page. Then confirm the Migrate Plus, Migrate Source CSV and Migrate Tools releases compatible with that core version on their Drupal.org project pages, and check their maintenance status and open issues before relying on them. All of these modules are GPL-2.0-or-later, so licensing does not add cost, but support contracts may still vary.
What the team provides and what it receives
The client provides the inputs that decide whether a structured import can be designed correctly the first time. The most important is a representative source export: the actual CSV files, JSON responses or XML documents from the system of record, not a hand-tidied sample. Alongside it, the client names a content owner for each content type who can answer questions about meaning, such as whether a status code or date format is authoritative. The client also decides the target content model, including which fields are required, which entity types and bundles receive the records, and how taxonomy terms or references should resolve. Finally, the client provides a non-production Drupal environment with the same core and contributed versions planned for production, plus a copy of the source data that can be refreshed. Confirm with the team which Drupal and Migrate module versions will be pinned before any mapping is written, because source plugin behavior and migration state tables depend on them.
In return, the team delivers mapping documentation that lists each source column or field path, its destination field, the transformation applied, the lookup or default used, and the owner who approved it. Migration definitions are delivered as versioned YAML configuration in the project repository, usually built on the Migrate Plus source plugins and the Migrate Source CSV plugin, with any custom process plugins documented beside them. Each test run produces a validation report comparing source record counts with created, updated and failed items, along with a sample of rendered records for content owners to sign off. The handover package includes a rollback procedure written for the specific migration IDs, the migrate:rollback steps and the order in which dependent migrations must be reversed, and notes on how incremental runs identify changed rows. Ask for these items explicitly in the statement of work so they are not treated as optional extras.
How DeployOpen scopes and delivers a structured import
The sequence turns a spreadsheet, API export or XML feed into a repeatable Migrate API import that your team can rerun, roll back and hand over. Each stage ends with an agreed artifact so the scope, the data and the sign-off stay visible to both sides.
Agree the written scope
DeployOpen and your team list the source files or endpoints, the destination content types, the volume of records and the environments involved. Scope, deliverables and timelines are written down before work starts, and a quote follows from that written scope. The step closes when both sides sign the scope document and the named source owners are confirmed.
Model the destination content
DeployOpen reviews the target content types, fields, entity references, taxonomy terms and media handling in Drupal, then agrees which source fields map to which destination fields. Decisions about required fields, duplicates and unique identifiers are recorded here, because they drive every later migration definition. The step closes with an approved field mapping sheet that your content owners have reviewed.
Build the source and migration definitions
DeployOpen configures the source plugin for each format, using Migrate Source CSV for spreadsheets and Migrate Plus source plugins for JSON and XML feeds, then writes the process pipeline and any custom transforms in a custom module. Source identifiers are mapped so that re-runs update existing records instead of creating duplicates. The step closes when each migration definition loads cleanly and the status command lists every migration as expected.
Run a pilot on a sample
DeployOpen imports a representative subset of records into a non-production copy of the site, including awkward cases such as empty cells, nested XML elements, pagination in APIs and special characters. Your content owners check a sample of nodes, media and references against the source. The step closes with a pilot report listing errors, skipped rows and the fixes applied to the mapping.
Execute full runs and incremental updates
DeployOpen runs the full import in the agreed environment and, where sources keep changing, schedules incremental runs that pick up new or updated records. Migrate Tools commands such as migrate:status show progress, and each run is logged with its counts and failures. The step closes when every run completes with its counts reconciled against the source totals.
Test rollback before go-live
DeployOpen rehearses a rollback with the Migrate Tools rollback commands on the pilot environment and confirms that only the imported entities are removed. The team checks that manually edited content is protected by the plan and that a rerun after rollback reproduces the same results. The step closes with a documented rollback procedure that your operations team has run successfully at least once.
Validate and sign off the content
Validation compares record counts, required fields, references, URL aliases and sample content against the agreed mapping and the source. Your business owners review the results and sign the acceptance checklist, which records any known gaps and how they will be handled. The step closes when acceptance is signed and every open issue has an owner.
Hand over and support after launch
DeployOpen hands over the custom code, migration definitions, runbooks and the scheduled run configuration to your team, and trains the people who will operate them. Ongoing support, upgrades and continuous testing are available as retainers if you want DeployOpen to keep the import running after launch. The step closes with a handover package accepted by your platform owner.
