Dify Ollama and self-hosted model setup guide

Why Dify with self-hosted models needs more planning than a provider key

Connecting Ollama or another OpenAI-compatible endpoint to Dify is a provider configuration step, but GPU capacity, request routing and data residency decide whether that connection holds up in production. A team that treats the endpoint as a settings field will usually discover the gaps once workflows run under load, knowledge base indexing competes for the same hardware, or a business owner asks where prompts and documents are processed. The provider key is the last step of the work, not the first. Dify is a self-hosted platform, and the Community edition runs anywhere Docker runs, with a single workspace. The Dify application services, PostgreSQL, Redis, the vector store and the sandbox all sit in your environment, but the model endpoint is a separate service with its own sizing, availability and access controls. An Ollama host or a vLLM-style gateway has its own GPU memory limits, concurrency settings, patching cycle and network exposure. Each one needs an owner, a capacity estimate based on your real prompt sizes and user counts, and a documented way to reach it from the Dify containers and nowhere else. Model behavior also differs from hosted APIs in ways that are easy to miss. Latency is usually higher and more variable on a local GPU, and it grows when several workflows share one model. Context windows are set by the model and its serving configuration, so a prompt that fits a hosted model may be truncated or rejected locally. Tool calling and structured output are the most fragile: some open models follow the OpenAI function-calling format well, others return malformed arguments or ignore tools entirely. Confirm each model's capability in your own tests, using your prompts, your tools and your expected input lengths, before any team builds on it. The practical starting point is a per-model acceptance test. For each model you plan to register, record measured latency at expected concurrency, the largest input your workflows send, pass or fail results for every tool call your agents use, and the output format stability over a set of representative cases. Keep those results with the configuration so that a model swap or a version upgrade triggers the same checks again. Ask your platform team to confirm where each endpoint runs, who can change its configuration and how traffic from Dify is authenticated.

Test

Which Dify services and settings carry model traffic

A model call in a self-hosted Dify does not go straight from the app to the model server. It passes through the api and worker services, the plugin daemon, the outbound proxy and the provider settings you enter, all started alongside the other components of the Compose stack. Knowing which piece does what tells you where to look when a model call fails or leaves your network.

The api and worker services

The api service handles requests from the web console and published apps, while the worker service runs background jobs such as document indexing. Both can initiate model calls, so a workflow test in the console and a knowledge base build can fail for different reasons. Check the logs of both services when a model call errors, and confirm which one a given feature depends on in your Dify version.

The plugin daemon

Model provider support in current Dify versions is delivered through plugins that the plugin daemon loads and runs. Ollama and OpenAI-compatible endpoints are reached through the provider plugin you install, so a missing or mismatched plugin blocks the model even when the server is healthy. Confirm the exact provider plugin names and their installation steps for your Dify version in the release notes and the app management documentation.

The SSRF proxy and private address rules

Outbound requests from Dify are routed through an SSRF proxy configured by the SSRF_PROXY settings, and that proxy blocks internal and private IP ranges by default. A model server on a private address in your own network may therefore be unreachable even when it is correct in the provider settings. Test the path from the Dify containers to the model endpoint and decide deliberately whether any private range needs an exception.

Model provider endpoints and credentials

Each model provider entry in the Dify console holds the endpoint URL, model names and any API key the server expects. Ollama usually needs only a base URL and model name, while OpenAI-compatible servers may also need a key. Record which entries hold secrets, and confirm with your team who may view or change them, because the exported app files do not include third-party API keys.

Compose services around the model path

The Compose stack starts the api, worker, web, plugin daemon, PostgreSQL, Redis, Weaviate, Nginx, a code sandbox and the SSRF proxy together. A model call therefore crosses several containers before it reaches your model server, and each one needs to be running and reachable. When a call fails, check the container health and network connections in this order before changing the model configuration.

Local models, hosted APIs or both: which option fits which workload

This table compares fully local models served through Ollama or another OpenAI-compatible server, hosted model APIs reached over outbound calls, and a mixed setup where each Dify workflow selects its model. Read each row as a question to answer for your own data and traffic, then check the operational cost on the right before choosing a default.

Decision factor	Local models through Ollama or an OpenAI-compatible server	Hosted model APIs over outbound calls	Mixed setup with per-workflow model selection
Best fit for data classification	Prompts and documents that must stay inside your network or air-gapped environment	Data your policy allows to leave the boundary under contract and logging terms you have reviewed	Restricted workflows pinned to local models, lower-sensitivity workflows allowed to call hosted APIs
Latency tolerance	Predictable when the GPU host is sized for peak concurrency, but queues build quickly under load	Depends on provider region, rate limits and network path from the Dify api and worker containers	Measure each workflow separately, since a single slow model can stall a chain of nodes
Cost per request	Mostly fixed capacity cost, so cost per request falls as utilization rises	Variable cost per token billed by the provider, so cost rises with usage	Blended cost that depends on how many workflows are routed to each side
Model quality needs	Limited to open-weight models you can run on your hardware, so benchmark them on your own tasks	Access to the provider's latest models without local hosting work	Lets quality-critical nodes use hosted models while bulk or internal tasks run locally
GPU hardware to buy and patch	Yes, including drivers, runtime versions and host OS patching on a schedule you control	No GPU purchase, but the Dify host still needs a stable network path and secrets handling	Only for the local portion, so size it from the workflows assigned to local models
Model updates to test	Each new model tag or quantization needs regression tests against your prompts and workflows	Provider-side model changes can alter outputs, so pin versions where the provider allows it	Two update streams to test, local model changes and hosted model deprecations
Egress controls to configure	Minimal outbound traffic, mainly to the internal model endpoint, which must be allowed through the SSRF proxy rules	Explicit allow rules for each provider host through the outbound proxy, with logging of destinations	Allow lists for every hosted provider plus internal routes to the local server
Credentials to govern	Usually no third-party key, but the local endpoint address and any gateway token still need rotation	One API key per provider, stored in Dify model settings and rotated under your secret policy	Duplicate credential sets across providers and local endpoints, which need one owner and one review cycle
What to measure first	Queue time, time to first token and GPU memory use at your expected concurrency	Error rates, rate-limit responses and cost per workflow run from provider usage reports	Per-workflow routing decisions, fallback frequency and the share of traffic that leaves your network

How to connect Ollama to Dify in a sequence you can verify

This sequence takes a self-hosted Dify instance from a confirmed baseline to a test app and knowledge base that answer from a local model, with a check at each step so problems surface before any real data is loaded. It is written for the platform engineers and the security reviewer who will sign off on the outbound paths.

Confirm the Dify and Compose baseline
Confirm the Dify release you intend to run, and check it against the Docker Compose deployment requirements in the Dify self-hosting docs: Docker Compose 2.24.0 or later, at least 2 CPU cores and 4 GB of RAM for the host. Record the exact Dify version and tag in your change record, since the environment variables and the SSRF proxy settings are version-sensitive. Done when docker compose version reports 2.24.0 or later on the host and the running Dify containers, including api, worker, web, plugin daemon, the SSRF proxy and the sandbox, all report healthy status.
Place the model server and map its network path
Install Ollama on a host or cluster with the GPU capacity your model set requires, and confirm the GPU is visible to the runtime before anything else. Then map the network path from the Dify api and worker containers to the model endpoint, including any firewall rules, routing or DNS names that sit between them. Your infrastructure team should confirm the port and the reachable address from inside the Dify network, not from the host alone. Done when a request from the Dify api container to the model server's API port returns a response, and the GPU utilization changes during a test generation on the server.
Choose and pin the model names and quantization
Decide which models the team will test, the exact model tags and the quantization level for each, and write them down in the test plan. Pin the tags rather than using a floating label such as latest, so that a later pull cannot change the behavior under test. Confirm with the model publisher or the Ollama model library that the license permits your intended use. Done when the test plan lists each model name, tag, quantization and license, and the same tags are shown as installed on the model server.
Register the model provider with endpoint and credential
In the Dify console, add the model provider that matches your server, using the endpoint address confirmed in step two and the model names pinned in step three. Where the provider asks for a credential, store it through Dify's model credential settings and not in a shared document or a prompt. If you use an OpenAI-compatible endpoint instead, enter its base URL and the key your gateway expects. Done when the provider test in the console returns success for each pinned model, and the provider list shows the models as available to the workspace.
Configure SSRF proxy rules for the model endpoint
Review the SSRF proxy settings in the Dify environment reference, since the proxy blocks internal and private IP ranges by default. Adjust the allow path so that only the model endpoint address and port are reachable from the api and worker containers, and leave other internal ranges blocked. Have your security team review the change because it widens an outbound path. Done when a request to the model endpoint succeeds through the configured path, and a request to a different internal address, such as a database or admin host on the same network, is still refused.
Create a test app and a test knowledge base on the local model
Build a small test chat app and a test knowledge base in the Dify console, and bind both to the local model provider rather than a hosted one. Use non-sensitive sample documents for the knowledge base so the test does not place client data in a new store. Keep the app configuration in the test workspace so it can be exported as YAML DSL later if needed. Done when the test app answers a question using the local model, and the knowledge base returns retrieval results for a question whose answer is in the sample document.
Run a workflow and confirm the model used
Run a workflow that calls the local model and inspect the run log in the Dify console. Confirm that the log records the provider and the pinned model name for each LLM node, and that no node silently falls back to a hosted provider. Repeat the run for each pinned model, then record the results in the test plan. Done when every LLM node in the run log shows the local provider and the expected model tag, and the test results are filed with the Dify version and the model tags used.

What drives GPU and integration effort for self-hosted models

This table lists the drivers that move GPU capacity, integration work and cost when Dify calls models you host, such as Ollama. Read each row for why the driver matters and the habit that keeps it manageable. The main takeaway is that a small initial model set on one OpenAI-compatible interface usually reduces effort more than any hardware choice.

Driver	Why it changes effort or cost	How to keep it down
Number of models to serve	Each model needs configuration, testing, prompt tuning and credential handling, and each must fit in GPU memory on the model host.	Start with a short initial model set and add models only for approved use cases.
Model size and context length	Larger models and longer contexts need more GPU memory and slower responses, which changes hardware and running cost.	Confirm memory needs with the model vendor and test context limits in staging before buying hardware.
Concurrent users and workflows	Parallel chats, agents and workflow runs share one model server and queue behind each other, raising latency.	Load test with expected peak workflows and set concurrency limits per model endpoint.
Endpoints and provider types	Each Ollama or OpenAI-compatible endpoint has its own URL, authentication and failure behavior to configure and monitor in Dify.	Standardize on one OpenAI-compatible interface where possible and keep the endpoint count small.
Network segmentation	Traffic from the Dify host to the model host may cross firewall zones. Dify's SSRF proxy blocks internal and private IP ranges, so private model hosts need a reviewed exception.	Allow only the model port from the Dify host and document each rule for security review.
Credential governance across workspaces	Model credentials sit in workspaces. The Community edition runs one workspace, while Enterprise adds multiple workspaces with separate access.	Assign one owner per credential, rotate keys on a schedule and remember that app exports omit third-party API keys.
Number of environments	Test, staging and production each need their own endpoints, credentials and promotion steps. Community installs may need separate deployments per environment.	Separate test from production endpoints and promote apps by YAML DSL export and import, re-entering keys in each environment.
Cluster sizing reference	The Dify Enterprise checklist production example uses six worker nodes with 8 CPUs and 32 GB RAM each. It is an HA platform reference only and does not size model servers.	Use it to plan platform redundancy, then size GPU model hosts separately from load tests on your own models.

Where the model server runs changes the design

The simplest layout runs Dify and the model server on one machine under Docker Compose. Dify's api and worker containers reach Ollama over the Compose network or the host address, so model traffic stays on the host. Setup is short, but every component shares one failure domain. A host reboot, a disk fault or a driver problem can take down chat, workflows and inference at the same time.

Network path: Point provider endpoints at an address inside the host or Compose network. Confirm that containers can reach the model port and that no model port is published beyond the intended network.
Failure domain: The web tier, database, vector store and model share one machine. Plan a restore onto a second host, because no failover exists at this level.
Update staging: Dify and the model server update on the same host, so an upgrade to one can break the other. Back up before each change and rehearse the upgrade on a copy first.
GPU scheduling: Confirm that the container runtime can see the GPU, that the vendor driver version matches the runtime, and that model memory leaves enough RAM for Dify's own services.

Before you commit, check with the GPU vendor that the driver and container runtime versions are supported together, and confirm that the Docker Compose version meets Dify's minimum of 2.24.0.

How model routing keeps regulated data on local models

The routing decision starts with classification, not with the model. Before any app, knowledge base or workflow is connected to a model, the team should assign each one a residency class: hosted models permitted, or local models only. The classification has to cover three data types separately: the prompts users or workflows send, the documents a knowledge base retrieves into those prompts, and the outputs the model returns. A customer complaint can be low sensitivity as a prompt yet contain account numbers once a retrieved record is appended. The strictest class of any of the three should decide where the call goes. Write the rule down per app, and name the owner who signs it off.

Dify's exports make the boundary easier to reason about but do not complete it. According to the Dify docs, a YAML DSL export carries app configuration, workflow nodes, model parameters, prompts and knowledge base connections, but not third-party API keys, knowledge base content or usage logs. Two consequences follow. A restricted app copied into another workspace will still reference a model provider that must be configured there, so the target environment needs its own credentials and provider list. And the documents themselves must be migrated separately, which means the residency class of a knowledge base has to follow its content into any rebuild. Confirm these export boundaries against the Dify version you deploy, since export contents change between releases.

Logs are where residency most often breaks. Prompts and responses are recorded in two places: inside Dify, in the metadata database and the monitoring data it keeps for each app, and inside the model server, such as the request logs an Ollama instance writes if logging is enabled. For a local model, the second location stays within your boundary, but only if the server's logging and any reverse proxy in front of it are configured to keep that data local. For a hosted provider, the logs sit with that provider under its own terms. Ask the team to map every log store for a restricted app, set its retention, and confirm who can read it through the Dify console and the database.

Routing is enforced by configuration and access control, not by the model server alone. An Ollama endpoint that only serves local models does not prevent a builder from selecting a hosted provider in an app that is configured for it, unless the workspace prevents that selection. In practice this means that model provider credentials are configured only in the workspaces that need them, that restricted apps are built in workspaces where hosted providers are not configured, and that the people allowed to edit model settings are a small, named group. Confirm with your Dify version which roles can add providers and change app model settings, because role behavior is a configuration question you should test rather than assume.

Document the residency decision as an artifact an assessor can follow: the app or knowledge base, its data classes, the approved model endpoints, the log locations and retention, and the approver. Then test it. Attempt to select a hosted provider in each restricted app, confirm that the option is absent or blocked, and confirm that an app export contains no provider keys. Repeat the test after every upgrade, since new .env variables and provider changes arrive with releases. A failed test is a finding to fix before go-live, not a note for later. An independent assessor can review this evidence, but the routing design and its test records remain the client's responsibility to maintain.

Securing model credentials, endpoints and outbound calls

Running Dify against self-hosted models creates a set of secrets, outbound paths and provider settings that need deliberate control. The cards below cover the signing key, encryption key storage, exported files, outbound network paths, the model server itself and who may change providers.

Treat SECRET_KEY as a signing and encryption key

In Dify, SECRET_KEY signs session cookies and JWTs and encrypts stored OAuth credentials. Generate it once with a strong random value, store it in your secrets manager, and keep it out of shared configuration files. Changing it after deployment logs out every user, so plan any rotation as a scheduled maintenance event with users informed in advance.

Decide where workspace encryption keys live

Dify can keep workspace encryption keys on the local host or store them in Azure Key Vault. Local storage keeps the key path inside your Compose or cluster host, but its protection depends on host access controls and backups. Key Vault moves custody to a managed service with its own access policies and audit trail. Confirm which option your security policy requires before data is encrypted.

Keep endpoint credentials out of DSL exports

Dify DSL exports carry app configuration, workflow nodes, model parameters, prompts and knowledge base connections, but not third-party API keys. That protects secrets in the file, yet teams still need to re-enter model credentials on import. Document where each provider credential is held, restrict who may export apps, and review exported files for embedded prompts or endpoint addresses that reveal internal topology.

Route outbound calls through the SSRF proxy

Dify's SSRF_PROXY settings send outbound requests through a proxy that blocks internal and private IP ranges. Pair it with network policy that allows egress only to the approved model endpoints, such as the Ollama host or a hosted provider's API address. Test the policy by attempting a call to an unapproved internal address from a workflow and confirming it fails and is logged.

Authenticate the local model server

An Ollama or other OpenAI-compatible endpoint that listens without authentication is reachable by any host that can route to it. Bind the model server to a private interface, place it behind a reverse proxy or gateway that requires a credential, and restrict network access to the Dify services only. Confirm the authentication method your model server version supports before relying on it, since defaults differ between tools and releases.

Restrict who can add or change providers

Adding or editing a model provider changes which endpoint and credential every app in the workspace uses. Limit provider configuration to a small set of administrators, separate that role from app builders, and review changes to endpoint addresses and keys. Record who holds administrative access in your identity system so that access reviews can cover Dify roles alongside other platforms.

How to test that Dify uses the right model and fails safely

Testing a self-hosted Dify model setup means proving two things before users rely on it: each app calls the model you intended, and the platform fails in a controlled way when that model is unavailable. Start by confirming every model registered under the model provider settings answers a fixed prompt set from the Dify host network path, not from your workstation. Run the same requests from the Dify api container's network namespace so you exercise the route the platform will really use, including any local DNS names, Docker network aliases or host gateway addresses that Ollama or another OpenAI-compatible endpoint depends on. Record the response, latency and status for each model so the baseline is usable later.

Next, run a workflow and check that the logs identify the model and provider used for each LLM node. Dify's app logs and the LLMOps monitoring views should match the model you configured in the node, not a default model left over from testing. Repeat this for every app that is meant to use local models. Then test isolation: from a restricted app or workspace that should only use the local provider, attempt to select or call a hosted provider. The expected result is that the hosted option is unavailable or the call is refused. Also confirm that outbound requests from the sandboxed code and tool paths are constrained by the SSRF proxy, because a model URL pointing at a private address should be blocked by design rather than by luck.

Measure response behavior under the concurrency your users will generate. Agree that number with the business owner before testing, and run a steady load at that level with a realistic mix of short and long prompts. Watch queue depth in the worker, GPU memory and utilization on the model host, and timeouts at the Nginx layer. Because local GPU capacity is usually the constraint, the result tells you whether the design needs more hardware, a smaller model or a routing rule that sends overflow traffic to an approved alternative. Record the numbers as a baseline, not as a guarantee.

Simulate model server downtime by stopping the model process or blocking its port during a workflow run. The workflow should return a controlled error message that the app can display, and it should not hang indefinitely or expose internal hostnames, IP addresses, stack traces or credentials in the response or in the browser. Check the api and worker logs for the same behavior. Then restart the Dify services and separately restart the model server, and confirm that both recover without manual data repair and that the workflow succeeds again. Ask the platform owner to confirm the expected behavior for each provider type, since error handling depends on the provider integration.

Keep a fixed evaluation set. Store the prompts, expected outputs or scoring criteria and the recorded responses under version control or in a controlled repository. Re-run the set after every model pull or tag change, and after any Dify upgrade or change to the .env file, because a model update can change answers even when the Dify version stays the same. Ask your model host team to confirm how model versions are pinned and whether the evaluation set must be re-approved before a change goes live.

What to monitor, back up, upgrade and document after go-live

Once a self-hosted Dify serves real users, the work shifts from building to keeping it running. The items below connect because they all protect the same thing: apps that keep answering correctly, against models whose versions and locations you can account for. Each one has a recurring owner and a recurring checkpoint.

Model endpoint and GPU health

Watch availability of each Ollama or OpenAI-compatible endpoint separately from Dify itself, since a healthy api container can still fail every model call. Track GPU memory use and request queue depth on the inference hosts, because a model that loads slowly or evicts other models will show up as workflow timeouts in Dify. Confirm with your team which alert thresholds and escalation path apply to each host.

Dify api, worker and plugin health

Monitor the api, worker, web, plugin daemon and sandbox containers from the Docker Compose stack, along with PostgreSQL and Redis. Dify's LLMOps features help with app-level traces, but container liveness and restart counts need separate infrastructure monitoring. Check the logs of the worker and plugin daemon after any model provider change, since failures there often appear only as stalled document indexing.

Database and file storage backups

Back up the metadata database (PostgreSQL by default in the Compose stack) and the file storage location used for uploaded documents and knowledge base files. Test a restore into a separate environment on a schedule your team sets, because an untested backup does not show whether apps, workspaces and documents come back intact. Confirm that the backup includes the secret key and encryption material needed to read stored credentials.

Model weights and version records

Model weights are not part of the Dify backup, so the weights on each inference host need their own copy or a documented re-pull path. Record each model name, tag and version in the change log whenever it is pulled, replaced or removed from an endpoint. Without that record, a workflow that behaved one way last month can be impossible to reproduce later.

Backup before every upgrade

Take a full backup before each Dify upgrade and follow the upgrade guide for the target version on the GitHub Releases page. Minor releases have arrived roughly monthly in the release history, so schedule upgrades on a standing cadence rather than waiting for a security notice. Keep the previous image tags so you can roll back if the upgrade fails.

Post-upgrade checks: env, plugins and providers

After each upgrade, compare your .env file with .env.example and add any new variables with values you have decided on. Re-run the model provider tests for every configured endpoint, because the plugin daemon can change between releases and a provider that worked before may fail silently. Record the results against the release version so you can show what was checked.

Model inventory and residency decision record

Keep an inventory listing each model, the host and region it runs on, the data classes it may receive and the apps that call it. Pair it with a residency decision record that states why each workload runs locally or on a hosted model and who approved that choice. Auditors and assessors typically ask for these two documents first, and DeployOpen prepares the evidence without issuing any audit opinion.

Questions teams ask before connecting Dify to Ollama or their own models

How much effort does connecting Dify to Ollama take?

The provider connection itself is a small task, but the effort lies in the surrounding work: reachable model endpoints, GPU capacity sized for your concurrency, testing each model against your prompts and documents, and deciding how local and hosted models are routed. What drives the effort:

Number of model providers and models in use
Whether the Ollama host sits on a private network that Dify's containers can reach
Number of apps and workflows that must be re-tested after each model change
Acceptance criteria for answer quality and latency

Confirm these items with your team before estimating. A quote from a partner should list each of them as a separate line in the written scope.

Can a small team connect Dify to its own models without a partner?

Yes, a small team with container and Linux skills can usually wire up a single Ollama endpoint on its own, provided someone owns the model choice and the test plan. Difficulty rises when the team must also run a GPU host, set up network rules, document data flows for a compliance review and keep the stack current. Questions to answer internally:

Who will own GPU drivers, model pulls and host patching?
Who will keep the Dify upgrade cadence and back up before each upgrade?
Who signs off on model output quality?

If those roles are already staffed, an internal build is realistic. If not, a fixed-scope partner engagement can cover the gaps in writing.

What does the integration need from our side?

Expect to provide compute, network access, a named owner for each model and representative test data. Typical inputs:

GPU hosts or accelerator capacity sized for the models and concurrency you choose, with the specification confirmed by your hardware or cloud team
Network paths from the Dify API and worker containers to each model endpoint, and an agreed egress policy for any hosted model
A named model owner who approves versions, prompts and changes
Test documents, questions and expected answers drawn from real but approved sources

The Dify environment reference describes SSRF_PROXY settings that block internal and private IP ranges, so confirm early that your Ollama host is reachable through the path you intend to allow.

How long does the integration take, and what sets the timeline?

The timeline depends on how many models you run, how fast your GPU hosts and network approvals arrive, and how much testing your acceptance criteria require. No fixed duration applies to every environment, so ask any partner for a timeline tied to the written scope and dependencies. Main timeline drivers:

Hardware lead time and capacity validation
Network and security review of the model endpoint paths
Number of apps and knowledge bases to validate
Stakeholder sign-off cycles for model choice and output quality

Ask for the dependency list and the assumptions behind any date, and compare them against your own change calendar.

Does the Community edition support self-hosted models, and what does Enterprise add?

The Community edition can connect to self-hosted model providers as part of a self-hosted Docker deployment, and it is free but limited to a single workspace. Dify Enterprise is custom priced and adds SSO, multiple workspaces, advanced security, a commercial license and professional support. Enterprise adds for high availability:

Deployment on Kubernetes with Helm, per the Dify deployment overview
External PostgreSQL, Redis and a clustered vector database, per the Enterprise resources checklist
SSO over OpenID Connect, SAML or OAuth2 with identity providers such as Microsoft Entra ID or Okta

Confirm current feature availability with Dify on its pricing and documentation pages before you decide, since editions change over time.

How do we keep evidence of data residency for model traffic?

Keep a dated record of where each model runs, which network paths Dify uses to reach it, and where documents, embeddings, logs and backups are stored. Evidence comes from configuration exports, network rules, storage locations and test results, not from a statement alone. Evidence to collect:

Architecture diagram showing every model endpoint and storage target inside your boundary
Environment configuration and proxy settings captured at each release
Egress rules and test records showing blocked or allowed destinations
Backup locations and retention settings

DeployOpen prepares systems, controls and evidence for compliance assessments, but independent auditors and certification bodies issue the reports. Ask your assessor which artifacts they expect.

What should we test before moving a Dify app to a self-hosted model?

Test each app against the same prompts, documents and acceptance criteria used for the current model, and record answer quality, latency and failure behavior. Run the tests again after every model, prompt or Dify version change. Test areas:

Output quality against approved expected answers
Knowledge base retrieval with the chosen embedding and vector settings
Error handling when the model endpoint is slow or unavailable
Confirmation that outbound calls go only to approved destinations

Keep the results as versioned records so the model owner can compare changes over time.

How DeployOpen scopes and hands over a self-hosted model integration

DeployOpen starts every self-hosted model integration with a written scope, and the scope has to name the specifics that decide whether the work succeeds. That means the Dify version to be deployed, including whether it is the Community edition or the Enterprise edition, the model providers that will be connected (for example Ollama, vLLM-style OpenAI-compatible endpoints, or a hosted API kept only for non-sensitive workloads), the GPU hosts that will serve those models with their memory and location, and the residency rules that govern which data may reach which model. Without these four items written down, the team cannot tell whether a routing rule or a sandbox setting is correct, so the scope is the first deliverable and the last point at which the client can change direction cheaply.

Configuration then happens inside the client's own infrastructure, in their cloud accounts, data centers or air-gapped network. DeployOpen works through the provider setup in the Dify console, the model credential entries, the outbound proxy settings and the network paths between the api and worker services and each model endpoint. Decisions made during the build include which models are registered per workspace, which workloads may route to a hosted model, and how SECRET_KEY and model credentials are stored. Each of these choices is recorded in the scope file so the client can see what was built and why, rather than inferring it later from the running system.

Testing follows the verification plan agreed in the scope. For this topic, that plan checks that each registered model answers from the intended host, that requests for restricted data never leave the permitted endpoints, that the outbound proxy blocks internal and private address ranges, and that the sandboxed code and agent paths behave as configured. DeployOpen delivers the results as test evidence: the test cases, the observed outputs, the configuration versions tested and any open findings with owners. This evidence is what a client's own assurance team or an independent auditor will ask to see. DeployOpen prepares systems and evidence for compliance assessments, but it does not issue audit reports, attestations or certifications, and those decisions remain with the client and the assessor.

Handover closes the engagement. The client receives the configuration as exported and versioned files, runbooks for adding or retiring a model, rotating model credentials, changing the routing rules and restoring from backup, and a model inventory that lists each provider, endpoint, GPU host, residency classification and owning team. The inventory is the document most often out of date after go-live, so the client should name the person who updates it before handover. Most engagements are fixed-scope, and a quote follows the written scope, so the client should ask the quote to itemize these handover deliverables separately from the build.

Ongoing work is optional. Retainers can cover Dify upgrades, which the Docker Compose documentation says should follow the upgrade guide for the target version after a backup, with new .env variables checked each time, and continuous security testing of the model routes and sandbox. Clients who do not need that support can run the handover package with their own team. Before signing, confirm with DeployOpen which items in the written scope are the client's responsibility, including the GPU hardware, model licensing and the residency decisions themselves.

Dify with self-hosted models such as Ollama