When asynchronous and bulk REST APIs fit an ERP or PIM feed
If your ERP or PIM pushes thousands of price, stock or catalog changes into a Magento Open Source store, the asynchronous and bulk REST APIs are designed for that pattern. They queue writes for background processing, so the source system receives an acknowledgment and is not held waiting while the store processes each change. Adobe documents these features in its developer documentation for Magento Open Source, but endpoint names, request bodies and response behavior can differ between releases, so confirm them against the exact version you plan to run before you design the integration. The two releases you are most likely to evaluate today are 2.4.9, released in May 2026, and 2.4.8-p5, also released on May 12, 2026.
The asynchronous and bulk endpoints depend on message queue infrastructure. Writes are published to queues and processed by consumers that run in the background, so the store needs a working message broker and consumers that are started and kept running. Adobe's system requirements for 2.4.9 and 2.4.8-p5 both list RabbitMQ 4.3, which makes it the broker to plan around for either line. Adobe supports only the listed combinations of PHP, database, search, cache, broker and web server, so a different broker version or a self-built stack may fall outside support even if the endpoints appear to work in testing. Confirm the full matrix for your target release before you commit to it.
Decide early whether the feed actually needs this route. Bulk asynchronous writes suit feeds where the source system can tolerate a delay between submission and visibility on the storefront, and where it can handle a bulk identifier and later status checks. Feeds that must show a price or stock level immediately after a change, or that touch only a handful of records, may be simpler through standard synchronous calls. Scheduled file-based imports are another option for large but infrequent loads. Ask your team to write down the latency each data type can accept, because that answer drives the choice more than the record count does.
Plan the operational side before the build starts. Someone must own the queue consumers, including how they are started, monitored and restarted after a failed deployment or server reboot. Your team also needs a retry policy for rejected or failed items, a way to inspect bulk operation status, and a rule for what happens to stock or price data if the queue backs up during a peak period. Ask the Magento developer or partner to show, in a test environment, how a failed record is surfaced and how it is resubmitted without duplicating valid changes.
Which write path fits each integration pattern
This table compares five ways an integration can write catalog, price and stock data into Magento Open Source: synchronous single-item calls, asynchronous single-item calls, bulk endpoints, the built-in CSV import and a scheduled job that loads files. Read each row against your integration's update volume and how long downstream systems can wait for confirmation, because that combination usually decides the path.
| Write path | Volume it suits | Per-item result and failure surfacing | Store load and operating burden |
|---|---|---|---|
| Synchronous single-item REST call | Low volume, event-driven changes such as one product edit at a time | Caller gets an immediate response per item; a failed call is visible in the calling system at once | Each call runs in the request cycle during business hours; simple to build but multiplies load as volume grows |
| Asynchronous single-item REST call | Moderate, steady volume where the caller can accept confirmation later | Request is accepted and queued; the outcome arrives later through status checks, so the caller must track each request | Work moves to queue consumers, smoothing peaks; requires monitoring of consumers and the message queue |
| Bulk REST endpoint | High volume batches such as nightly price or stock sets from an ERP or PIM | One bulk operation returns a UUID; per-item outcomes must be read from status results, so the caller needs parsing and retry logic | Queued processing limits peak load; the integration owns batch sizing, status polling and reconciliation |
| Built-in CSV import | Periodic or one-off loads run by store staff, such as migrations or large catalog corrections | Validation and error reports are produced after the run; there is no per-item response to the calling system | Run manually from the admin, so it needs an operator; the import file format and error limits are covered on the sibling CSV import page |
| Scheduled job that loads files | Large, predictable batches that arrive as files on a fixed cycle | Outcome depends on job logs and reconciliation reports written by your own code, so failure handling is designed in-house | Runs outside peak hours when scheduled; adds a file transfer step and job monitoring, with scheduled job design covered on a sibling page |
How queue consumers and bulk UUIDs track each batch
The first thing to understand is that an asynchronous call is a handoff, not a finished update. The caller submits a request to a bulk endpoint, Magento validates the payload and accepts it, and the operations are placed on a message queue so they can be applied in the background. Consumers, which are background processes that read messages from that queue, then write each catalog, price or stock change. The caller receives an identifier at submission and uses it later to ask what happened. An accepted response does not mean the product, price or quantity has changed yet, and your ERP or PIM integration should be designed around that gap.
The bulk UUID is the handle for the whole group of operations. When a bulk request carries several items, Magento associates them with one identifier so the caller can track the batch as a unit and then examine individual items. Your integration should store that identifier alongside the source batch from the ERP or PIM, so a failed or stalled run can be traced back to the exact input file or export. Confirm in the Adobe documentation for your version how the bulk UUID is returned, how long status records are kept, and whether any cleanup process removes them before your reconciliation window closes.
Status reporting is where the design gains its detail. A status check should show whether a batch is accepted, in progress or finished. For finished batches, it should also report the outcome of each operation, including items rejected by validation and items that failed during processing. Use those per-operation results as the source of truth for reconciliation, and keep a record of them outside Magento if your audit or support process needs one. The exact status fields, endpoint paths and consumer names must be taken from the Adobe documentation for the Magento Open Source version you run, because they can change between release lines.
Queued work does not complete on its own. The consumer processes must be running on the servers that handle the queue, and they must connect to the message broker that your release supports. The system requirements for 2.4.9 list RabbitMQ 4.3, so confirm the broker version against the requirements for your own release before you plan the build. If consumers stop, messages wait in the queue, batches remain in progress and status checks never show completion. Before go-live, add monitoring for consumer process health and queue depth, and decide who is alerted when a batch stalls and who has the access to restart the processes.
Setting up async and bulk endpoints on a self-managed store
This sequence takes a self-managed Magento Open Source store from a version check to a verified bulk update that a status endpoint reports as complete. Each step ends with a check, so a failure is found at the point where it was introduced. Monitoring of the cron and consumer runners starts in the consumer step, not after go-live.
Confirm the platform stack against system requirements
Your team or implementation partner selects the Magento Open Source release and checks it against Adobe's system requirements page for that version. For 2.4.9, the listed combinations include PHP 8.5, MySQL 8.4 or MariaDB 12.3 (recommended) or 11.8, OpenSearch 3, Redis or Valkey 9, RabbitMQ 4.3, Composer 2.10 and nginx 1.30 on Linux x86-64. Adobe supports only the listed combinations, so record each component version in a build sheet. This step is done when every component matches a listed combination and the installed versions match the sheet.
Provision RabbitMQ and verify the connection
Install RabbitMQ at a listed version, then create a dedicated virtual host and a user whose permissions are limited to that virtual host. Add the broker host, port, virtual host and credentials to the queue connection block in the store's environment configuration. Confirm that the application host can open an authenticated connection to the broker using those exact settings, and check that the broker's management view lists the virtual host. Store the credentials in your secrets manager, not in the repository.
Enable modules and run setup upgrade and cache steps
Enable the asynchronous operations and message queue modules that ship with the edition. Confirm the exact module names with the module status command before enabling them, because names differ between releases. Then run the setup upgrade command, compile if your deployment mode requires it, and flush the caches. This step is done when the module status command shows the modules as enabled, the setup upgrade finishes without errors, and the logs under the var directory contain no new errors after a cache flush.
Start consumers and keep them running
List the queue consumers the installation exposes, then start them under a process supervisor with one program definition per consumer or consumer group. Add the consumers runner to the scheduled jobs so that consumers are restarted if they stop. Set up monitoring from this point: alerts on supervisor process state, on the consumers runner cron group failing to run, and on queue backlog growth. Confirm that every consumer reports as running after a restart of the supervisor and that the logs show no repeated fatal errors.
Create a dedicated integration with a scoped token
In the admin panel, create an integration for the ERP or PIM connector only. Grant it the access resources needed for the catalog, price or stock endpoints in scope and nothing else, then generate its access token. Store the token in the connector's secrets store. Check the scope by calling one in-scope read endpoint, which should succeed, and then calling a resource outside the granted scope, which should return an authorization error. Both results confirm the token is limited as intended.
Send a small test batch and check status
Submit a small illustrative batch of product or stock updates to the asynchronous endpoint, such as five records in a non-production catalog. The call should return an accepted response with a bulk identifier. Poll the bulk status endpoint with that identifier until it reports completion, then confirm the changed values in the admin panel or the storefront data. This step is done when the status endpoint reports the operations as completed and the data matches the input file.
Test a deliberate failure and read the errors
Submit one record that is intentionally invalid, such as a SKU that does not exist or a value outside an attribute's allowed options. Poll the bulk status endpoint and confirm that the failed operation is listed with an error message. Then find the same failure in the logs written by the consumer process and the exception log under the var directory. Confirm which records were applied and which failed, and document the retry behavior for your version before the ERP team builds its error handling.
Designing retries so a failed batch does not double-write prices
Asynchronous and bulk REST calls from an ERP or PIM can fail in ways that look like success to the sender. The cards below cover five failure modes that commonly lead to duplicate or stale price and stock writes, and the design response for each. Test every path in a staging store before the feed runs against production.
Caller timeouts that trigger a resend
When the ERP or PIM times out waiting for a response, the request may still have been accepted and queued by Magento. A blind resend then queues the same price changes twice. Design each payload to be idempotent, keyed by SKU or the source system's own record ID, so a repeated row updates the same value instead of creating a second change. In staging, submit a bulk request, drop the connection before the response, resend it, and confirm the final price matches the source once.
Consumer crashes mid-batch
Magento processes queued bulk operations through message queue consumers, and a consumer can stop partway through a batch after some rows are saved. Check how far the bulk operation status reports progress, how unfinished messages are redelivered, and whether a redelivered row is safe to apply again. Ask your integration team to confirm the behavior of the RabbitMQ setup in your environment. In staging, stop a consumer during a large batch and verify the restarted run leaves no row applied twice.
Partial batch failures
A single bulk request can contain valid and invalid rows, such as an unknown SKU, a malformed price or a missing store view code. Decide whether a bad row should block the whole batch or only itself, and make the sender able to tell which rows succeeded. Keep row-level results in your integration log keyed by source ID so the retry sends only the failed rows. In staging, mix valid and invalid rows in one batch and confirm the retry set contains only the rejected rows.
Duplicate messages from the source
ERP and PIM exports sometimes emit the same change more than once, and a retry layer can add duplicates on top. Add a sequence number or source timestamp to every price and stock update, and reject any update older than the value already stored for that SKU. This check belongs in the source system or in a middleware layer you control, so Magento receives only the newest version. In staging, replay an older message after a newer one and confirm the older price does not overwrite the newer one.
Stale stock and price overwriting newer values
A delayed batch can arrive after a later change and overwrite it with out-of-date stock or prices, even when no message is duplicated. Set bounded retries with increasing backoff and a cap on attempts, route messages that exhaust retries to a dead-letter or failed-message queue, and review that queue before reprocessing. Run a scheduled reconciliation report that compares Magento prices and quantities with the ERP or PIM and lists mismatches. Confirm the retry limits and report schedule with your operations team, then test a forced mismatch in staging.
Why GET calls stay synchronous in an async design
Reads stay synchronous because a storefront, a search page or a status lookup needs an answer in the same request. Asynchronous handling in Magento Open Source is aimed at writes that can wait for a message queue and a consumer to process them, such as bulk catalog, price or stock updates sent from an ERP or PIM. When you design an integration, separate the two paths on purpose. Send writes through the asynchronous or bulk endpoints and issue GET requests directly against the resources they read. Treating every call as the same kind of operation is the most common design mistake in this area, because it leads to retries and timeouts that the platform never needed.
The practical consequence is that a GET call returns the value currently stored in the database. An integration that polls GET endpoints for stock levels or product data does not wait for consumers to run, so it sees whatever the last completed write produced. That is useful for dashboards, price checks at checkout initiation and reconciliation jobs. It also means that a read made right after a queued write may still show the old value until the relevant consumers finish. The write has been accepted, but the change is not yet visible. Your integration must not assume that a successful response to a bulk submission means the data is already live.
Design the source system to tolerate that gap. The ERP or PIM should record that a change was submitted, not that it was applied, and it should not overwrite its own record with the value it read back from Magento until the status check confirms completion. Where a downstream process depends on the new price or stock quantity, make it wait for the status result or accept eventual consistency and say so in the process documentation. For high-volume loads, agree with the business owners how long a stale read is acceptable during peak update windows. Those are decisions to record before build starts, not after the first mismatch report.
Confirm, for the exact version you run, which resources expose asynchronous or bulk endpoints and which remain synchronous only. Support differs by resource and release, so check Adobe's developer documentation for your version and test each endpoint you plan to use in a non-production environment with your own data volumes. Also confirm which consumers must be running and how they are monitored, because a stopped consumer leaves GET results stale without raising an error on the read path. Document the result in your integration specification so that operations staff know what a stale read looks like and who to call when it persists.
Security and operations for the integration user and the queue
Asynchronous and bulk endpoints add a message queue, consumers and a bulk status trail to the integration. The controls below cover who can call the API, how credentials and traffic are protected, and how the team sees backlog and failures. Confirm each one against your own security policy and what your auditor will expect to see.
Least-privilege integration user
Create a dedicated integration user with an access role limited to the resources the ERP or PIM actually writes, such as products, prices and stock sources, and nothing else. Review the role whenever the feed changes, because a bulk endpoint that reaches more resources than intended widens the impact of a leaked token. Confirm in the admin role editor which resources the role grants before go-live.
Token rotation and secrets storage
Store the integration access token in the ERP or PIM secrets store, not in job scripts, configuration files or shared wiki pages. Agree a rotation interval with the owners of both systems and document how a token is revoked and replaced without stopping the feed. Test the rotation in staging so the first real rotation does not fail a live batch.
TLS on every hop
Encrypt traffic between the ERP or PIM, the web tier, the database and RabbitMQ, not only the public API endpoint. Check the certificate chain and hostname validation in each client library, since a client that skips verification defeats the protection. Confirm the RabbitMQ connection settings in the Magento environment configuration use TLS and that the broker certificate is trusted by every consumer host.
Restricted queue management access
Limit who can reach the RabbitMQ management interface, and avoid publishing it on the public internet. Use separate broker accounts for publishers, consumers and administrators, with permissions scoped to the Magento virtual host and the queues it uses. Review these accounts with the same cadence as admin users, because broker access can read or purge pending bulk messages.
Logging batch IDs and outcomes
Record each bulk UUID, the submitting integration user, the submission time and the final status returned by the bulk status endpoint. Keep these records in the ERP or PIM audit trail so a failed price or stock update can be traced to one submission. Decide the retention period with your compliance owner, since these logs can contain product and pricing data.
Queue depth and consumer health
Monitor the depth of the bulk and async queues, the age of the oldest pending message and whether each consumer process is running. A growing backlog or a stopped consumer can leave stock or prices stale on the storefront without any API error. Set alerts with thresholds agreed with the business owners of the feed, and confirm how consumers restart after a host reboot.
Admin security and CSP baseline
Magento Open Source follows the Content Security Policy and admin security practices that Adobe documents, so apply those settings to the admin and storefront as part of the same hardening pass. Check whether any payment-related data passes through the integration, because if it does, PCI DSS scoping must be reviewed before the feed goes live. Ask your auditor which of these controls they will test and how they expect evidence to be produced.
What buyers ask before building async and bulk API integrations
What drives the cost and effort of an async and bulk API integration?
The main drivers are the number of resources in scope, the state of the ERP or PIM data, the retry and reconciliation design, the testing matrix, and the operating model for RabbitMQ and consumers. Each resource type (products, prices, stock, categories) adds mapping, validation and test cases. Clean, consistently keyed source data shortens the build, while duplicate SKUs, missing units or unclear price rules add rework. Retry and reconciliation logic must handle partial failures, so it takes design time up front. Testing multiplies across store views, websites and environments. Ask any proposal to itemize each of these drivers and to state which are one-time build work and which are recurring operations, such as queue monitoring and consumer restarts.
Can our own team build an async Magento API integration in-house?
Yes, if your team already has Magento development skills, experience with a message queue such as RabbitMQ, and time to test thoroughly. The build itself is mostly custom module work with REST endpoints, message queue topics, consumers and status handling. The part teams most often underestimate is testing: bulk operations report results per item, and failures must be traced back to the source record across store views and environments. Ask whether the people who will carry the work in production can also own the consumer and queue runbooks after go-live. If any of those skills are missing, a partner can cover the gap, or your staff can work alongside one.
What does the integration need from our side to start?
You need a staging environment that matches production closely enough to test queues and consumers, sample ERP or PIM feeds that reflect real data including edge cases, and a named business owner who decides price and stock rules. That owner also needs to sign off on test results before release. Access to the ERP or PIM team matters because field definitions, update frequency and identifiers come from them. Agree on who approves changes to the mapping and who is on call for failed messages. Missing any of these usually moves the schedule, so confirm them in the written scope before work begins.
How long does an async and bulk API integration take to deliver?
Duration depends on scope, so the timeline should come from a written proposal rather than an assumption. The main variables are how many resource types and source systems are involved, how clean the source data is, how many store views and environments must be tested, and how many failure scenarios the retry design must cover. Ask the proposal to break the timeline into design, build, test, and cutover phases, and to name the client decisions that each phase depends on. A clear list of exclusions protects both sides from later disputes about what the timeline included.
What changes for the integration after each Magento platform patch?
Each patch can change core classes, queue behavior, dependency versions or system requirements, so custom async modules need regression testing after every patch cycle. Adobe's upgrade guidance recommends applying patches every cycle to stay secure, and it names customizations and third-party integrations as drivers of upgrade effort. Check the system requirements for the release you target, since the supported PHP, MySQL or MariaDB, OpenSearch, Redis or Valkey and RabbitMQ versions change between lines. Plan a staging run of the full bulk workload before each production patch, and confirm the RabbitMQ version your release supports. Ask your partner how they will handle the patch cadence in the retainer or support scope.
Why do status checks and GET calls stay synchronous?
Reads that a caller needs right away should return immediately, because a queued response would force the caller to poll before it could act. Bulk writes are accepted asynchronously and return an operation identifier that the caller uses to check status, so the integration needs a clear status model for accepted, processing, completed and failed items. Keeping GET calls synchronous avoids stale or ambiguous data in lookups. Engineers should confirm which endpoints must return current data and which can tolerate delay, and document that split in the integration design so downstream teams know what to expect.
How do we know the integration is ready for production?
Readiness means the test results match the agreed acceptance criteria, not that the code merely runs. Check that failed items are retried or reported correctly, that reconciliation reports match source counts for each run, that consumers recover after a restart, and that queue backlogs drain within the limits your business owner set. Test across the store views and environments you operate, and include a rollback path. Ask for the acceptance criteria in writing and the sign-off record from the named business owner before cutover. These checks are verification steps you can confirm; they do not replace an independent audit where one applies.
Limits to confirm in your version and environment
Asynchronous and bulk behavior depends on the exact release, the message broker, the PHP runtime, the cache layer and the distribution you run. Each item below is a limit to confirm with Adobe's documentation and a proof-of-concept run in your own environment, not an assumption carried over from a design document.
Which endpoints are asynchronous or bulk
Confirm, for your exact 2.4.x release, which REST endpoints accept asynchronous or bulk requests and which entities they cover, such as products, prices and stock. Check the endpoint list in Adobe's developer documentation against the release you will deploy, and test each write path you plan to use. Do not assume a GET or a single-entity endpoint has the same behavior as the bulk variant.
Batch size and sustained throughput
Measure the batch size and request rate your environment sustains before you commit to a feed design. Run the proof of concept against production-like data volumes, with the same consumer count, database and search indexing settings you plan to use. Record queue depth, consumer lag and failed message counts, and ask Adobe or your partner which limits are documented and which must be measured locally.
RabbitMQ and PHP version combinations
Check that your RabbitMQ and PHP versions match one of the combinations Adobe lists for your release. For 2.4.9, the system requirements list RabbitMQ 4.3 and PHP 8.5, while 2.4.8-p5 lists PHP 8.4 or 8.3 with RabbitMQ 4.3. Adobe supports only the listed combinations, so any other pairing needs written confirmation before go-live.
Varnish or Fastly and queued updates
Determine how full-page cache invalidation behaves when queued updates change product, price or stock data. Adobe documents Varnish for self-managed installations, while cloud-hosted Adobe Commerce uses Fastly. Test whether cache purges happen after the consumer commits, whether stale pages can appear during bulk runs, and how your cache layer and reverse proxy are configured in your environment.
Distribution differences such as Mage-OS
If you run Mage-OS or another community distribution instead of Adobe's Magento Open Source package, confirm which asynchronous and bulk behavior, consumer configuration and patch timing it inherits. Mage-OS describes itself as upstream compatible with Magento 2 extensions, but compatibility with a specific endpoint, queue setting or security patch must be verified by testing rather than assumed.
Extensions that touch the same entities
Inventory every extension, custom module and third-party integration in the stack that reads or writes products, prices, stock or customer records. Confirm whether any of them uses observers, plugins or direct database writes that bypass the asynchronous path. Overlapping writes can cause lost updates or conflicting values, so test concurrent ERP and extension changes in the proof of concept.
How DeployOpen scopes and delivers async API integrations
DeployOpen plans, builds, tests and documents Magento Open Source asynchronous and bulk REST API integrations inside infrastructure you control. We start from the written scope, agree the ERP or PIM data flows and queue design with your team, and hand over a documented, supported integration.
How we scope
- Map which catalog, price and stock updates move through bulk endpoints and which stay synchronous GET calls.
- Confirm the Magento Open Source release line, PHP, MySQL or MariaDB, RabbitMQ, OpenSearch and Redis or Valkey versions your store runs.
- Agree in writing the endpoints, consumers, error handling, volumes and acceptance tests before any build work starts.
How we build
- Implement the bulk request payloads, bulk UUID tracking and status check logic against your agreed data model.
- Configure and verify the queue consumers, topics and cron jobs that process messages on your RabbitMQ and Magento hosts.
- Design retry and failure handling so rejected rows are logged, isolated and reprocessed without duplicating stock or price changes.
How we hand over
- Deliver test results from staging runs that cover high-volume loads, partial failures and status polling.
- Document the endpoint contracts, consumer configuration, retry rules and the operational runbook your team needs.
- Walk your engineers through monitoring, log locations and the steps for restarting consumers after a failure.
How we support
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
- Ongoing support, upgrades and continuous testing are available as a retainer, including checks after each Magento patch release.
- Ask for a quote built from the written scope, and confirm with your team how upgrades to newer 2.4.x lines will affect the integration.
