Why streaming Dify API calls break behind proxies and load balancers
When a product team calls a self-hosted Dify app from a backend and sees responses cut off or stalled in production, the cause is usually not the model. A streamed response or a long workflow keeps one HTTP connection open for its whole run, and every hop on that path must keep it open too: the Nginx service in the Docker Compose stack, your load balancer, your Kubernetes ingress, and the calling service itself. If any one of them closes an idle connection, buffers the stream, or enforces a shorter request timeout than the run needs, the caller sees a truncated answer that looks like a Dify fault. Treat the whole chain as one system and test it as one system.
The behavior at each hop has to be confirmed against the Dify version you deploy, because streaming and workflow event handling change across releases. Dify ships minor versions roughly monthly, so a proxy configuration that worked on one release can need review after an upgrade. Ask the vendor or check the release notes for your version before you copy a timeout value from a blog post or from another team's runbook. This page does not state timeout values for Dify, and you should not assume defaults in your own proxies match what your application expects. Measure the longest run you care about, then set each hop to tolerate it with margin.
Planning for this matters most in three patterns. Chat interfaces that stream tokens to a user are sensitive to buffering, because a proxy that holds output until the response completes turns a fast stream into a long pause. Long agent runs that call tools, search knowledge bases and reason over several steps can exceed the idle limits of an intermediate device even when Dify is still working. Webhook-driven workflows add a second concern: the caller may disconnect before the workflow finishes, so you need to decide whether the result is delivered by callback, polled or resumed, and whether any payload you send is within the limits of the components in front of Dify. Decide which of these patterns you run before you size the proxy layer.
Streaming and polling options and when each fits
This table compares streamed server-sent events, blocking calls and polling or event-stream resumption for Dify API runs, with the proxy settings each depends on. Read each row as a decision: the delivery mode you pick determines which timeouts, buffering rules and retry logic your proxies and client must support. Confirm endpoint names, event types and timeout behavior against the Dify API reference for your installed version before you commit to a design.
| Delivery choice | Best fit | Proxy and load balancer settings it depends on | Main risk |
|---|---|---|---|
| Streamed SSE responses | Interactive chat where users read tokens as they arrive | Response buffering disabled, read timeout longer than the slowest expected pause, HTTP/1.1 keep-alive and no compression buffering on the event path | Idle connections closed by a proxy mid-answer, which looks like a truncated reply |
| Blocking call | Short workflows where the caller needs one final result | Read timeout that covers the full run, request size limits sized for inputs, retry policy that avoids duplicate runs | Client or gateway timeout fires before the workflow finishes, leaving the outcome unknown |
| Polling a run status | Long workflows where the caller can check back periodically | Short request timeouts are fine, rate limits on the status path, stable run identifiers stored by the caller | Excess polling load and gaps between checks that delay the user |
| Event stream resumption | Long workflows where a dropped stream must continue from the last event | Sticky or consistent routing if state lives on one node, stored last event position, reconnect logic with backoff | Lost or duplicated events if the client does not track the last position it received |
| Stopping generation | Users or systems that must cancel a response in progress | Stop request must reach the same backend that holds the task, client closes the stream cleanly | A stop request routed to a different node may not halt generation |
| Webhook callbacks | Outbound notification when a run completes | Inbound body size limit at your receiver, retries on non-2xx responses, signature checks | Payloads larger than your receiver accepts, or duplicate deliveries handled without idempotency |
How to set up and verify long-running calls end to end
This sequence takes a streamed Dify API call from a direct test against the api service through each proxy layer to a load test, so the team knows which hop cuts connections and who owns each timeout. The output is a documented, tested path that holds long runs open and stops them cleanly.
Confirm the Dify version and API reference
The platform team records the exact Dify version running in the environment and the API reference that matches it, starting from the release list on GitHub. Streaming behavior, event names and endpoints can change between minor releases, which arrive roughly monthly, so the team should not copy examples from another version. Confirm the streaming mode your apps use (blocking or streaming response mode) and the workflow event types your client will parse. Done when the version, the API reference link and the list of event types are written into the runbook with the date they were checked.
Test a streamed call directly against the api service
A developer on the platform team sends a streamed request straight to the Dify api service inside the network, bypassing Nginx, load balancers and any gateway. Use a workflow that runs longer than a few seconds and has at least one node that takes time, so the stream stays open long enough to observe. Confirm that events arrive incrementally and that the final completion event is received. Done when the raw event log shows continuous event delivery from start to the end marker, with timestamps saved as the baseline for later comparison.
Add each proxy layer one at a time
The network or platform engineer puts one hop in front of the api service at a time, starting with the internal Nginx that ships in the Docker Compose layout, then any load balancer, then the corporate reverse proxy or API gateway. After each addition, the same test call runs and the event timestamps are compared with the baseline. Turn off response buffering for the event stream route and confirm that the proxy forwards the event-stream content type without holding events. Done when each layer passes the same streamed test and any buffering or rewrite behavior is documented against that specific layer.
Set idle and read timeouts with each layer's owner
Each proxy, load balancer and gateway has a timeout that can end a silent connection, and each one is owned by a different team. The platform lead lists every hop with its owner, then agrees in writing on the idle and read timeout values for the streaming route, separate from the values used for ordinary short API calls. Owners confirm the setting names in their own product documentation, since keys differ between Nginx, cloud load balancers and gateways. Also check any Dify-side timing settings in the environment variable reference. Done when the timeout table has one row per hop with the owner, the setting name, the agreed value and the date it was changed in configuration.
Verify disconnect and stop behavior
The application team tests what happens when the client disconnects mid-stream and when a user or script asks Dify to stop generation. Close the client connection partway through a run, then check the api and worker logs and the run status in the Dify console for the expected end state. Next, send a stop request for an active run and confirm that generation ends and that no further events arrive. Repeat the disconnect test through the full proxy chain, since some proxies keep upstream connections open after the client leaves. Done when both tests produce the expected logged state, and the team has written down how an interrupted run appears to the client and to operators.
Check resumption and webhook payload limits
Before relying on resumption or webhook callbacks for long workflows, the integration team confirms what the deployed version supports, using the Dify API reference for that version rather than assumptions. If a client needs to reconnect to a workflow event stream, test the reconnect path directly against the api service, then through the proxy chain. For webhook-triggered runs, send a payload close to the size your systems produce and check where it is rejected, which may be at Nginx, the gateway or the application. Done when the team has a pass or fail result for each path, with the rejecting layer identified for any payload limit failure.
Load test with realistic run durations
The performance engineer runs a load test that uses the same workflows, run lengths and concurrency the business expects, not a short synthetic prompt. Start from the sizing in your own environment, since the minimum Dify requirements are a floor and not a capacity plan. Watch open connection counts on each proxy, worker queue depth, and the rate of connections closed by timeouts during the run. Repeat the test after any timeout change. Done when the test shows that the agreed timeouts hold for the longest expected run under the planned concurrency, and the results, settings and version are filed with the runbook for the next upgrade.
What happens when a client disconnects or must stop a run
A dropped client connection and an explicit stop request are different events, and the caller has to handle them differently. A dropped connection only ends the caller's view of the run. The server may keep generating, keep calling tools and keep consuming model tokens until the workflow finishes or fails. A stop request is a deliberate instruction to end generation. Before you rely on either behavior, confirm in the Dify docs for your exact version how a stop is issued for chat and workflow runs, what the API returns when a run is stopped, and whether a stopped run is recorded as stopped, failed or completed in the console logs. Those answers determine what your backend should expect after it sends a stop.
Reconnect logic needs an idempotency key that your backend owns, stored with the business action it triggers, such as a ticket, order or case ID. Do not assume a retried call resumes the same run. Unless the docs for your version describe a way to resume a workflow event stream by run identifier, treat a reconnect as a new request that may start new work. Record the run identifier returned at the start of each call, mark the business record as pending, and only fetch or resume results by the method your version documents. Confirm this behavior in a test environment before building the retry path.
Backend retries can duplicate both work and billing. If your service retries a call after a timeout, the first run may still be executing, so the same prompt can run twice, call external tools twice and consume model tokens twice. Tool steps that write to other systems are the highest risk. Plan a retry budget, put the retry decision in one place, and check whether your model provider and Dify usage logs let you reconcile duplicate runs against your own records. The exact cost of a duplicate depends on your model contracts and workflow design, so ask your provider and your team to size it for your workloads before setting retry limits.
Security and operational risks in streamed and webhook traffic
Streamed responses and long workflows keep connections open for longer than typical request traffic, and webhooks make Dify send requests to addresses you may not control. The cards below cover the exposure, data handling, credential and monitoring risks that come with that traffic, and the Dify settings and checks that address each one.
Exposing the API service directly
Streaming endpoints hold connections open, so an internet-facing api service accepts long-lived traffic from anywhere. Put the API behind your own reverse proxy or gateway that enforces TLS, client limits and authentication before traffic reaches Dify. Confirm which Dify services your Docker Compose layout publishes to the host, since the web and API paths both pass through the bundled Nginx by default. Review this exposure with your network team before the first production route is opened, and keep administrative paths off public listeners.
Buffering that hides events
Proxies that buffer responses can hold server-sent events until a block fills or the upstream closes, so users see nothing for long periods and clients time out. Logging full response bodies at the proxy also stores partial answers, prompts and retrieved documents in places with weaker retention controls. Disable response buffering for the streaming route, and set log formats so request paths and status codes are recorded without response payloads. Test with a real long workflow, not a short greeting.
API keys in browser code
Service API keys that appear in front-end JavaScript let anyone who opens developer tools call your Dify apps and consume model budget. Streaming makes this worse because a single leaked key can hold many open connections. Keep app keys on your backend, and have the browser call a route you control that adds the key and enforces per-user limits. Use the embedding and end-user identity guidance for the browser path, and rotate any key that has appeared in a public bundle.
SSRF from webhook targets
Workflow nodes that call webhook or HTTP targets can be pointed at internal addresses, such as cloud metadata endpoints or databases on private ranges. Dify's SSRF_PROXY settings route outbound requests through a proxy that blocks internal and private IP ranges, so confirm that the variables are set and that the webhook path actually uses them. Test with a target on a private address in a non-production instance and record the result. Be aware that proxy rules are a filter, so pair them with egress rules on the host network.
Sandboxed agent network access
Agent runs execute code and templates, and Dify 1.16.1 added agent sandbox network isolation through a Squid proxy, along with sandboxed Jinja2 template rendering. Confirm that your deployment is on 1.16.1 or later and that the sandbox proxy is in the path for agent tool calls. Review which agent tools can reach internal services, since a long-running agent is harder to notice than a short failing call. Ask the vendor or check the release notes for any remaining egress paths your version does not cover.
Timeouts and secret rotation
Timeouts at proxies, load balancers and clients can end a stream while the workflow keeps running, so monitoring shows success while users see failures. Rotating SECRET_KEY logs out all users, and restarts or key changes can cut open streams and orphan workflow runs. Align proxy idle timeouts with the longest expected run, record stream close reasons as separate metrics, and schedule secret changes in a maintenance window with a clear list of affected sessions. Confirm the behavior of your version before changing any secret in production.
Questions teams ask about streaming Dify calls
How long can a streamed Dify API call run before a proxy cuts it off?
The answer depends on the timeout values in each hop between your client and Dify, not on Dify alone. Every reverse proxy, load balancer, ingress controller and corporate gateway on the path can close an idle or long-lived connection on its own schedule. Dify's streamed responses are server-sent events, so a hop that buffers responses or enforces a short read timeout will break a stream that is still generating. Start by listing each hop and its read, idle and buffering settings, then test a long generation end to end. The setup section explains which settings to check first.
Can our own team handle streaming and timeout configuration in-house?
Often yes, if the team already runs the proxy and load balancer layer and can change its timeout and buffering settings in a test environment. The work is mostly configuration, verification and documentation, but it crosses ownership lines: the application team knows the client behavior, the network team owns the proxies, and the platform team owns the Dify deployment. The question for your organization is who can change each hop and who signs off on the change. The ownership section of this page describes how that split usually works.
What drives the effort and cost of fixing long-running Dify calls?
The main drivers are the number of hops between clients and Dify, how many teams must change settings, and whether long workflows need a resume or polling design in your application code. Simple streams that only need a longer timeout on one ingress are a small change. Workflows that run for minutes, need reconnection after a dropped client, or must stop generation on user request need more design and testing. Dify does not publish a price for this work, and DeployOpen does not publish one either. Ask for a written scope that lists each hop, each application change and each test case before comparing quotes.
What does the work need from our network team?
The network team needs to provide a current diagram of the request path, the timeout, buffering and body-size settings on each proxy or load balancer, and a change window for test and production. They also need to confirm whether TLS terminates at the edge and whether any device inspects or rewrites responses, since either can affect event streams. Ask them to check idle timeouts, response buffering and maximum request body size, because these are the settings most often behind dropped streams and rejected webhook payloads. The limits section lists the values to collect.
How should we test streaming before launch?
Test with the same proxies, load balancers and client libraries that production will use, not with a direct connection to the Dify container. Run generations that last longer than your slowest hop's idle timeout, interrupt the client mid-stream, and confirm that stop requests end generation. Include a workflow that runs long enough to need resumption and a webhook payload near your size limit. Repeat the tests after any Dify upgrade, because minor releases arrive roughly monthly and can change streaming behavior. Record the results so the release review has evidence.
Does Dify's security model change when streams pass through extra infrastructure?
Yes, in two ways. Streams carry generated content and user inputs through every hop, so the proxies and logs on that path become part of your data handling scope, and they should not log full response bodies without a decision to do so. Also, the Dify 1.16.1 release notes describe token-based API authentication between services, so a proxy that rewrites headers could affect how requests are authenticated. Confirm header forwarding and log retention with your security team before launch. The security section covers the controls to review.
Should we plan for an upgrade to Dify while streaming is in production?
Yes. Dify 1.17.0 was released on August 25, 2026 and 1.17.1 on September 10, 2026, after 1.16.0 on July 17, 2026, so minor releases arrive roughly monthly. Each upgrade should follow the upgrade guide for the target version, start with a backup, and include a check of new variables in .env.example. Add the streaming test set to every upgrade plan so a change in event behavior is caught before users see it. The operations section describes how to schedule this work.
How upgrades and minor releases can change streaming behavior
Treat every Dify upgrade as a possible change to streaming behavior. Minor releases arrive roughly monthly: the Dify releases page lists 1.16.0 on July 17, 2026, 1.17.0 on August 25, 2026 and 1.17.1 on September 10, 2026. Each release can change how the api and web containers handle streamed responses, and a proxy timeout or buffering setting that worked on the previous version may behave differently on the next. Plan upgrades as change events with a rollback path, not as image pulls.
Before any upgrade, follow the upgrade guide for the target version on the GitHub Releases page and take a backup, as the Docker Compose deployment docs advise. After the upgrade, compare .env.example in the new release with your .env file. New variables may introduce defaults that affect request timeouts, worker behavior or outbound routing, so review them before you change any proxy, Nginx or load balancer setting. Changing the proxy first can hide a regression in Dify itself.
After each upgrade, re-run your streaming and timeout tests against the same long-running workflows you use in production. Confirm that server-sent events reach the client in order, that a workflow longer than your proxy idle timeout still completes, that stop requests end generation, and that webhook calls still arrive within the payload limits you set. Record the results per version so that a future regression can be traced to a specific release. Ask your DeployOpen or internal team to keep these tests in the same repository as your deployment configuration.
How DeployOpen scopes and supports streaming work on Dify
DeployOpen plans, implements, tests, documents and supports self-hosted Dify inside your own infrastructure, including the proxy, load balancer and timeout settings that keep streamed and long-running API calls alive. Work starts from a written scope that names the routes, clients and failure cases to cover before any configuration changes.
How we scope
- Inventory the Dify apps, endpoints and clients that stream or run long workflows.
- Map every proxy, load balancer, ingress and CDN hop between callers and Dify.
- Record the response time, idle timeout and payload size limits each hop enforces today.
- Agree in writing which timeouts, retry rules and webhook payload checks are in scope.
How we build
- Configure reverse proxy and load balancer timeouts and buffering for event stream responses.
- Test resumption of workflow event streams after a dropped client connection.
- Verify that stop-generation requests reach Dify and end the active model call.
- Check webhook payload handling against the limits set in your Dify environment configuration.
How we hand over
- Deliver a written configuration record for each proxy and load balancer hop.
- Document the client retry, reconnect and stop-generation logic your teams must implement.
- Provide test scripts that reproduce long streams, interrupted connections and oversized payloads.
- Walk your platform and application teams through the runbook and open questions.
How we support
- Support is available as a retainer for changes to proxies, timeouts and Dify releases.
- Review timeout and buffering settings again after each Dify minor upgrade you schedule.
- Confirm with your network and platform teams any hop DeployOpen does not administer.
