How Dify connects to NVIDIA NIM and Triton endpoints
Your infrastructure team already runs NVIDIA inference services, and Dify needs to call them. The supported route is the langgenius model provider plugins for NVIDIA NIM and Triton Inference Server, both listed on the Dify Marketplace under the langgenius publisher. The NIM listing describes NIM as a set of inference microservices, and the Triton listing covers Triton Inference Server endpoints. Install the plugin that matches the stack you operate, then confirm the plugin version against the Marketplace listing and your Dify release before you register anything in production.
Each plugin registers the endpoint it is configured to call as a model provider inside a Dify workspace, so the models it exposes can be selected by workflows, agents and knowledge base settings. What stays with the infrastructure team is everything behind that endpoint: the NIM or Triton deployment, model packaging and versions, GPU allocation, autoscaling, TLS termination, authentication at the inference service, and the decision about which models are approved for use. Ask the infrastructure team to confirm in writing which model names and versions each endpoint serves, because Dify will only show what the endpoint and plugin expose.
Before you connect anything, confirm three points with the team that owns the inference stack: which endpoint and credential Dify should use for each environment, which models may be called from which workspace, and who approves changes to model names or versions. Keep the provider credential separate from any other workspace secret, and plan who may edit provider settings in Dify. Network placement and the outbound path from the Dify api and worker services are covered in the sections that follow.
What you need before registering a model in Dify
Registering a NIM or Triton endpoint in Dify only works when the Dify deployment, the inference service and the network path between them are all in place. The cards below cover the deployment baseline, the running endpoints, the routes Dify's services use to reach them, and the credentials your team must confirm before the first model call.
Dify deployment baseline
Community edition on Docker Compose needs Docker Compose 2.24.0 or later, at least 2 CPU cores and 4 GB of RAM. Enterprise on Kubernetes needs Kubernetes 1.24 or later and Helm 3.14 or later. Confirm the host sizing against your expected concurrency, because the model plugin adds a call path on top of the api, worker and plugin daemon services that already run.
Running NIM or Triton endpoints
Confirm which stack your infrastructure team operates: NVIDIA NIM microservices or a Triton Inference Server deployment. Record each endpoint's base URL, the model names it serves, and whether it runs chat, embedding or rerank workloads. Dify registers models by these names, so a mismatch between the served name and the registered name is a common first failure.
Network routes from the Dify api and plugin daemon
Model calls originate from the Dify api and plugin daemon, not from the browser. Confirm that both containers, or the Kubernetes pods that host them, can reach the inference endpoint on its port. Check whether the SSRF proxy in your deployment blocks the destination, since outbound requests to internal and private IP ranges are filtered by default. Agree on an allowlist entry with the network team rather than loosening the proxy broadly.
Credentials and API keys
Ask the infrastructure team whether the endpoints require an API key or token, and who issues and rotates it. Store the key in Dify's model provider settings, not in workflow prompts or exported DSL files, which do not include third-party API keys. Plan one key per environment so that development and production traffic can be separated and revoked independently.
TLS certificates and trust chain
If the endpoint is served over HTTPS with an internal certificate authority, confirm that the Dify containers trust that chain before registration. A certificate error during model registration usually points to the trust store rather than to Dify. Also confirm the certificate's hostname matches the URL you will enter, and agree on who renews it and how renewal is tested.
Vendor and environment checks
Confirm with NVIDIA or your own platform team the supported NIM or Triton versions, GPU driver and container runtime requirements, and the model licenses that apply to the models you plan to serve. These details depend on your hardware and vendor terms, so get them in writing before sizing the cluster or committing to a model list.
Registering NIM or Triton models in Dify step by step
This sequence takes a model already served by an infrastructure team's NIM or Triton endpoint through plugin installation, registration, a test call and workspace assignment, so each layer is confirmed before apps depend on it.
Confirm the endpoint and model names with the platform team
Before touching Dify, the infrastructure team confirms the base URL that Dify will reach, the protocol the server exposes, and the exact model identifier as the server reports it. For Triton, that means the model name in the model repository and the version directory in use. For NIM, it means the model name the microservice returns from its model listing. Record these values in a shared note. Done means the team can run one request against the endpoint from the same network segment Dify will use and get a valid response for the named model.
Install the NVIDIA NIM or Triton plugin from the Marketplace
An administrator opens the Dify Marketplace and installs the langgenius model provider plugin for NVIDIA NIM or for Triton Inference Server, depending on which stack runs the models. Install only the plugin that matches the server, and confirm the publisher is langgenius before installing. Done means the provider appears in the workspace model provider settings with its configuration form. Check the plugin version against the Dify release in your environment, since plugin behavior can change across Dify minor releases.
Enter the endpoint URL and model identifiers
In the provider configuration form, enter the endpoint URL reachable from the Dify api and worker containers, not from the administrator's laptop. Add each model as a separate entry with its model type (chat, embedding or rerank) and its identifier. The identifier must match the server's model repository exactly, including case, separators and version suffix where the server uses one. A mismatch usually fails only at the first call, so compare the string against the server output character by character. Done means the form saves without a validation error.
Add the credential the endpoint requires
If the endpoint requires an API key or token, add it in the credential field of the provider configuration rather than in environment files or prompt text. Use a separate key for each environment, such as development and production, so that rotating one does not break the other. If the server runs without authentication inside a private network, leave the field empty and record that decision in the change record. Done means the credential is stored and the form shows the model as configured, not as missing credentials.
Run a test chat or embedding call
Use the model's test function in Dify, or send a short prompt from a test app, and confirm a response from the registered model. For an embedding model, confirm that a short input returns a vector of the expected dimension for the model. Watch the Dify api and plugin daemon logs during the call for connection errors, timeouts or name-not-found messages. Done means one successful response is recorded for each model type you registered, and the log shows the request reached the intended server.
Assign the model to a workspace or app
Set the model as a workspace default only when several apps should share it. Otherwise, select it inside the specific app's model settings so that other apps do not inherit a model with different data handling. For knowledge bases, select the embedding and rerank models at the time of dataset creation, because changing the embedding model later requires re-indexing. Done means the app's model selection shows the NIM or Triton model, and a test conversation in that app returns an answer from it.
Verify the model after the next platform change
Repeat the test call after any change to the server, the model repository, the network path or the Dify version. A renamed model version on the server breaks registration silently for apps that reference the old identifier, so keep a short runbook with the endpoint, the identifiers and the test prompt. Ask your DeployOpen or internal team to include this check in the upgrade procedure. Done means the runbook names the test that proves the model still answers from the same identifier.
Where the inference endpoint should sit relative to Dify
The first decision is which network segment the NIM or Triton endpoint lives on, because that choice determines whether the Dify api and plugin daemon can reach it at all. Dify's outbound requests are not all equal. Model calls made by the plugin daemon and requests routed through the SSRF proxy follow different paths, and the SSRF proxy is configured to block internal and private IP ranges by default. An on-premises inference server on an RFC 1918 address is exactly the kind of destination that default is built to refuse. The reader should map every hop from the api and worker containers to the endpoint, then confirm in the Dify environment variables reference how SSRF_PROXY_HTTP_PROXY and SSRF_PROXY_HTTPS_PROXY are set and whether any allow-list setting exists for a specific internal host. Do not assume a private address will work until a test call succeeds from the same container that makes the production call.
Once the path is mapped, choose between a dedicated inference subnet and shared internal networking. A dedicated subnet suits GPU hosts that run NIM containers or Triton with their own firewall policy, limited administrator access and separate monitoring. Shared internal networking suits a team that already runs Dify and the inference servers in the same cluster or VPC and wants fewer routing exceptions. The trade-off is blast radius: a shared network makes it easier for any compromised workload to reach model endpoints, so the allowed ports, source addresses and TLS requirements should be written down before the first connection. Ask the infrastructure team who owns the firewall rules, how they are reviewed and what happens when a GPU host is replaced and its address changes.
Whichever layout is chosen, keep the Dify side narrow. Allow only the NIM or Triton service ports from the Dify plugin egress path, deny the rest, and log denied connections so a misconfigured SSRF exception is visible rather than silent. Treat any change to SSRF proxy settings as a security change with a named approver, because widening the proxy's reach changes which internal services a workflow can call. Also confirm that the endpoint certificates are trusted by the Dify runtime, since an internal certificate authority that the containers do not trust fails in a way that looks like a network fault.
How Dify model settings map to NIM and Triton configuration
This table pairs each Dify model setting with the NIM or Triton value it depends on and says where the reader must copy that value from the inference server. Read it before registering a model, then repeat the mapping in each environment because credentials do not travel with app exports.
| Dify field | NIM or Triton source | Copy from the server or confirm? |
|---|---|---|
| Provider | Dify plugin for NVIDIA NIM or Triton Inference Server, installed from the Dify Marketplace | Confirm the plugin version your Dify release supports before installing |
| Model name | Model identifier the NIM container or Triton model repository exposes | Copy from the server's model list, not from the model's public name |
| Endpoint URL | Base address of the NIM service or Triton HTTP or gRPC port, reachable from the Dify api and worker containers | Copy from the service definition, including scheme, host and port |
| Credential | Token or key your platform team issues for the endpoint, if the gateway in front of it requires one | Recreate per environment, since app exports exclude third-party API keys |
| Model type | Chat, embedding or rerank, matching the task the NIM profile or Triton model config serves | Confirm from the model card or model config, then match it in Dify |
| Context length | Maximum input and output tokens the deployed model profile supports | Confirm with the infrastructure team, since the served limit can be lower than the model's native limit |
| Generation parameters | Defaults such as temperature and maximum output tokens, set on the Dify model instead of the server | Set per app and test, since exports carry model parameters but the server does not enforce them |
| Network path | Service name, cluster DNS or private address, and any proxy the SSRF settings allow | Confirm the route with network staff, because outbound filtering may block private ranges |
Security controls for inference endpoints behind Dify
Dify holds the model provider credentials, makes the outbound calls and records what happens, so most of the security work for NIM and Triton sits in how Dify is configured and how the inference endpoints are reached. The cards below cover the Dify settings that protect stored credentials, the network paths to each endpoint, the transport security on the inference side, who may change providers and what gets logged.
Protect the SECRET_KEY and stored credentials
The SECRET_KEY setting signs session cookies and JWTs and encrypts OAuth credentials, so provider keys and endpoint credentials depend on it staying fixed. Changing it after deployment logs out all users, which makes the value a planned secret set before first use. Confirm where your team keeps it (a secrets manager or a restricted .env file), who can read it, and how it is restored from backup. Also decide whether workspace encryption keys stay local or move to Azure Key Vault.
Restrict network paths to each endpoint
Allow Dify's outbound traffic only to the NIM or Triton addresses it needs, on the ports those services listen on. The SSRF proxy blocks internal and private IP ranges by default, so an inference host on a private subnet needs an explicit, documented exception rather than a broader rule. Confirm which containers originate the calls, whether the plugin daemon sits on the same path, and whether the inference hosts accept connections from anywhere else on the network.
Verify TLS on the inference side
Requests between Dify and an inference endpoint should use TLS whenever the path crosses a network segment you do not fully control. Confirm with NVIDIA or your own platform documentation how a NIM or Triton endpoint is served over HTTPS, which certificate authority issues the certificate, and how the chain is renewed. Decide whether Dify must trust an internal CA, and test that certificate expiry produces a visible failure in Dify rather than silent fallback.
Confirm vendor authentication options
Whether a NIM or Triton endpoint can require a bearer token, mutual TLS or an upstream gateway check depends on how your team deploys it, so treat these as items to confirm rather than assumed features. Record what the Dify model provider form accepts for each endpoint, where any token is stored, and whether tokens are scoped per environment. Test that a request without valid credentials is rejected before any model call succeeds.
Limit who manages model providers
Model provider configuration determines which endpoint and credential every app in a workspace uses, so changes to it carry the same risk as a deployment change. Limit provider management to a small named group, separate from people who build apps, and record every change. Community edition is limited to a single workspace, so the separation has to come from Dify roles and your own change process. For multiple workspaces, Dify Enterprise adds SSO and workspace management.
Log inference calls and retention
Dify's LLMOps monitoring shows application activity, but confirm exactly which fields it records for each model call, including the endpoint, model name, token counts and any prompt or response content. Decide how long those logs are kept, who may read them, and whether inference-side logs on the NIM or Triton host are collected in the same place. Prompts may contain personal or regulated data, so test what appears in logs before production use.
Why registered NIM or Triton models fail in Dify
This table lists the failures teams most often hit after registering a NIM or Triton model in Dify, with the likely cause and the fix for each. Read it when a model appears in the console but calls fail, return errors or never show up in app model selectors. Start with the plugin version and network path, since those account for most failed registrations.
| Failure | Likely cause | Fix |
|---|---|---|
| Plugin missing or wrong version | The NIM or Triton plugin is not installed, or it does not support the Dify release running in your environment | Install the langgenius plugin from the Dify Marketplace on the target version, then confirm the version in the console before registering the model |
| Endpoint unreachable from the plugin daemon | The plugin daemon runs in a container with no route, DNS entry or firewall rule to the inference service | Test the endpoint from the plugin daemon container itself, not from your workstation, and fix routing, DNS or firewall rules first |
| SSRF proxy blocks the address | SSRF_PROXY settings block internal and private IP ranges, so a cluster-internal or RFC 1918 endpoint is refused | Confirm the proxy policy with your platform team, then allow only the specific inference host or path rather than widening the whole private range |
| Model name mismatch | The name registered in Dify differs from the model name or identifier the NIM or Triton server exposes | Query the server's model list, copy the exact identifier into the Dify registration, and rerun a test call |
| Credentials not recreated after an environment copy | Encrypted provider credentials and keys do not carry over with an app or DSL export, so the copied environment has no working key | Re-enter the endpoint and any API key in the target environment, and check that the export does not contain third-party keys |
| Model type set incorrectly | A chat model is registered as embedding or rerank, or the reverse, so Dify sends requests the server cannot answer | Match the model type to the served capability, delete the registration, and re-add it with the correct type |
Testing the connection before users depend on it
Test the path from Dify to each NIM or Triton endpoint before any application or user depends on it. Start with a direct call from the Dify host network to the inference endpoint, using the same address, port and protocol that the model provider plugin will use. Run it with curl or a similar client from the machine or container that runs the Dify api and worker services, not from a jump host or laptop, because a route that works from an administrator's workstation can still be blocked between the Dify containers and the inference network. Confirm that the endpoint answers without the proxy settings Dify uses, and that the request reaches the intended model rather than a default one.
Next, send one test prompt per model type you registered: a chat model, an embedding model and, if you use it, a rerank model. Check the response shape, not only the status code, and record the model name, dimension of returned embeddings and latency. If embeddings are in use, run a workflow with a knowledge base from the Dify console that ingests a small document, retrieves from it and returns an answer. This exercises the embedding path, the vector store, and the SSRF proxy rules together, which a plain model call does not. Confirm with your infrastructure team which NIM or Triton model versions are deployed so the results can be matched to them later.
Repeat this test plan after every Dify minor release, which the langgenius/dify releases page shows arriving roughly monthly, and after every update to the inference servers. Read the upgrade guide for the target version first, take a backup, and check .env.example for new variables, since a new setting can change how outbound model calls are routed. Keep the test prompts and expected outputs in version control so that a failing re-test is easy to diagnose, and treat a failed test as a stop signal for the upgrade rather than a problem to fix in production.
Questions buyers ask before connecting Dify to NIM or Triton
Can our infrastructure team connect Dify to NIM or Triton without outside help?
Yes, if your team already runs the inference endpoint, owns the Dify host and can change network rules, the registration itself is a small part of the work. The Dify side means installing the NVIDIA NIM or Triton Inference Server model provider plugin from the Dify Marketplace, registering each model with its endpoint and testing it from a workflow. The harder parts are usually outside Dify: firewall paths, credential storage, proxy behavior and testing each model against the prompts your apps use. A partner is most useful when those parts cross several teams or zones.
What makes the effort for a NIM or Triton connection grow?
Four factors move the effort most:
- The number of models and modalities (chat, embedding and rerank each need their own registration and tests)
- The number of network zones between Dify and the inference servers
- How often credentials rotate and how many Dify workspaces share a endpoint
- Whether the endpoints already have health checks, logging and capacity limits
Each factor adds configuration, review and testing. Ask any provider for a quote that lists these factors against your environment rather than a single lump sum.
What do we need to provide before the work starts?
You need to provide the inference endpoint details, the model names and versions the servers expose, a test host with network access to them, and a named owner for credentials and firewall changes. You also need a list of the Dify workspaces and apps that will use each model, plus the data classes those apps may send. Confirm who approves changes to outbound rules, since the Dify SSRF proxy blocks internal and private IP ranges by default and may need a deliberate exception.
How do Dify Community and Enterprise editions differ for this work?
The Community edition is free and limited to a single workspace, so a shared NIM or Triton endpoint can serve only one Dify workspace under that license. The Enterprise edition is custom priced and adds SSO, multiple workspaces, advanced security, a commercial license and professional support, according to the Dify pricing page. Enterprise SSO supports OpenID Connect, SAML and OAuth2 with providers such as Microsoft Entra ID and Okta, and it separates workspace access from web app access. Check these terms against your planned number of workspaces before you choose a layout.
What should we confirm with NVIDIA or the server vendor first?
Confirm with NVIDIA which NIM containers and model versions your license and hardware support, and which API surface the endpoint exposes. For Triton Inference Server, confirm the model repository layout, the protocol and port in use, and how the server authenticates clients. Ask both vendors how they handle upgrades, security patches and logging of prompts. Do not assume that a model name in Dify matches the deployed model, and confirm the supported model list for your release before you register anything.
How should credentials for the endpoints be handled?
Store endpoint keys in the Dify model provider configuration, not in workflow prompts or exported DSL files, because DSL exports exclude third-party API keys. Give development and production endpoints separate keys, and record who may change them. Dify uses SECRET_KEY to encrypt OAuth credentials, and changing that value after deployment logs out all users, so plan it as a controlled change. Rotation adds recurring work, so include it in the operations plan from the start.
Will Dify upgrades affect the NIM or Triton connection?
Dify publishes minor releases roughly monthly, for example 1.16.0 on July 17, 2026 and 1.17.0 on August 25, 2026, according to the Dify releases page. Each upgrade should be preceded by a backup and a check of the target version's upgrade guide and .env.example. After every upgrade, rerun a test workflow against each registered model, because plugin and provider behavior can change between versions.
Official sources for the NIM and Triton plugins
Start with the Dify Marketplace listings for the NVIDIA NIM and Triton Inference Server model provider plugins. Both are published under the langgenius account, and the listing is where you confirm the plugin version, the supported model types and the configuration fields it asks for before you install anything. Record the plugin version you tested against, because the plugin, the NIM container or the Triton release can each change independently of your Dify version. Dify Marketplace listing for NVIDIA NIM is the starting point for the NIM side.
Next, read the *Docker Compose deployment* guide and the environment variable reference for the Dify release you run. The Compose guide covers the services that must reach your inference endpoints, and the environment reference covers the outbound proxy settings that decide whether a private inference address is reachable at all. Dify routes outbound requests through the SSRF proxy, which blocks internal and private IP ranges by default, so confirm with your platform team how the NIM or Triton host will be allowed before you troubleshoot a connection failure as a plugin fault. Dify environment variable reference lists the settings to review.
Finally, read the LICENSE in the langgenius/dify repository before you plan any shared use. The Dify Open Source License is a modified Apache License 2.0 that prohibits operating the source code as a multi-tenant environment, where one tenant is one workspace, without written authorization, and it also prohibits removing or changing the logo and copyright information in the console. If several business units will share one NIM or Triton backend through separate workspaces, ask the licensor in writing whether that arrangement needs authorization. Dify license text is the document to bring to legal review.
How DeployOpen scopes and delivers this integration
DeployOpen plans, implements, integrates, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure. For Dify with NVIDIA NIM and Triton Inference Server, we start from the inference endpoints your infrastructure team already runs and build the Dify side around them, with engagements agreed in a fixed written scope before work begins.
How we scope
- We inventory the NIM or Triton endpoints, model names, versions and hardware your team already operates
- We agree which Dify workspaces and apps will call each endpoint and which models they may use
- We define network paths between the Dify api and worker services and the inference hosts
- We confirm credential ownership, rotation responsibilities and who may manage model providers in Dify
How we build
- We register NIM and Triton models through the Dify plugins and check each model type against its endpoint
- We place the inference traffic on the network segments and proxy rules you approve, then test reachability
- We verify chat, embedding and rerank calls from representative Dify apps before any user access
- We document configuration, model settings and rollback steps as part of the written deliverables
How we hand over
- You receive the configuration record, network diagram, credential inventory and test results for each endpoint
- Your platform and infrastructure teams receive a walkthrough of model registration and change procedures
- We provide a runbook covering endpoint health checks, model version changes and Dify upgrade steps
- Handover acceptance follows the criteria agreed in the written scope
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after handover
- Retainer scope for model endpoint changes, Dify releases and access reviews is agreed in writing
- We can retest model connections after Dify minor releases, which arrive roughly monthly
- Quotes follow from the written scope, and we confirm any dependency on your inference platform roadmap first
