Connecting Dify to LocalAI or llama.cpp for lightweight self-hosted inference
A small team or edge site that wants models running on its own hardware can connect Dify to those models through one of two plugins listed on the Dify Marketplace: the langgenius LocalAI plugin, or the OpenAI-API-compatible plugin for model providers that follow OpenAI's API standard, such as LM Studio or a llama.cpp service. The plugin choice follows the server's API, not the team's preference. If the inference server exposes a LocalAI API, use the LocalAI plugin. If it exposes an OpenAI-style chat and embedding interface, use the OpenAI-API-compatible plugin and enter the base URL, model name and any API key the server requires. Confirm the exact endpoint paths and supported features of your server version with its own documentation before you register a model.
Dify's Community edition runs anywhere Docker runs, according to the Dify deployment overview, so the same Compose stack can sit on a single office server or an edge box next to the inference host. Keep the Dify host and the model server on a network segment that Dify's outbound requests can reach, and check whether the SSRF proxy settings in the environment reference allow that address. Those proxy rules block internal and private IP ranges by default, so a model server on a private address may need an explicit exception. Record that exception as a deliberate decision, since it is the point where Dify's outbound filtering and your inference server's network placement meet.
CPU-only models have clear limits for agents. Agent workflows usually need several model calls per task, tool selection that depends on reliable structured output, and acceptable latency for each step. Before you promise an agent to users, test a representative workflow end to end on the actual hardware, measure response time and failure rate, and confirm the model follows the tool-calling format your workflow expects. Simple chat, summarization or retrieval answers may be workable on modest hardware, while multi-step agents may need more capacity or a different serving tier. Ask your hardware vendor and model publisher to confirm the memory and throughput figures for the model you plan to run, because these depend on quantization, context length and concurrency.
What you need before connecting Dify to a local model server
Connecting Dify to LocalAI or a llama.cpp server touches three layers: the Dify host, the model server, and the network path between them. Confirm each layer before you install a plugin, because most first-day failures come from a version mismatch or a blocked route rather than from the model itself.
Dify host minimums
The Docker Compose deployment starts the api, worker, web, plugin daemon, PostgreSQL, Redis, Weaviate, Nginx, a code sandbox and an SSRF proxy. The documented floor is 2 CPU cores and 4 GB of RAM, with Docker Compose 2.24.0 or later. That floor is for Dify alone, so the local model server needs its own memory and CPU budget on the same host or on a separate machine.
Current Dify release
Minor releases arrive roughly monthly, and 1.17.1 is the latest listed on the GitHub releases page as of this writing. Pin the exact version you test against, read its release notes, and check the upgrade guide before moving. Plugin behavior can change between releases, so a LocalAI or OpenAI-compatible plugin that worked on one version may need retesting on the next.
Model server build and endpoint
Record the exact LocalAI or llama.cpp server build, the model file loaded, and the base URL the server exposes. Confirm that the server's OpenAI-style endpoints (such as chat completions and embeddings) respond as the plugin expects. Test with a direct request from the Dify host before you configure anything in Dify, so model problems are separated from plugin problems.
Plugin and API compatibility
Check the Marketplace listing for the LocalAI plugin or the OpenAI-API-compatible plugin, and confirm which API routes and model types that plugin version supports. LM Studio and llama.cpp servers differ in how they expose model names and features, so confirm the exact compatibility with your chosen plugin version rather than assuming it. Write down the plugin version alongside the Dify version.
Network path from api and plugin daemon
Model calls originate from the Dify api and the plugin daemon, not from your browser. Confirm that both containers can reach the model server address on the required port, and that the Compose network or host firewall permits it. The SSRF proxy blocks internal and private IP ranges by default, so a local model server on a private address may need an explicit, reviewed exception. Ask your security team to approve that path in writing.
Hardware for agent workloads
Confirm that the CPU, memory and model size on the server suit the work. Agents make several sequential model calls per task, so response time on a CPU-only server can dominate the whole workflow. Run a representative workflow on the target hardware before committing, and record latency and failure rates so the decision rests on measured results rather than a model card.
Configuring the LocalAI or OpenAI-compatible provider in Dify
This sequence connects a LocalAI or llama.cpp server to Dify through the right Marketplace plugin, verifies that the model answers before any app depends on it, and then makes it a workspace default. Each step ends with a check you can repeat after upgrades or server changes.
Confirm the server exposes an OpenAI-style API
Before touching Dify, the platform team confirms which server runs the model, whether it is LocalAI or a llama.cpp server or LM Studio, and that it answers requests in the OpenAI API format on a reachable address. Note the base URL, the port, the exact model identifiers the server reports, and whether it requires an API key. Confirm with the server's own documentation how it lists models and handles authentication, since these details vary by server version. The step is done when a request from the Dify host's network to the server's model list returns the expected model names.
Install the matching plugin from the Marketplace
In the Dify console, a workspace administrator installs the LocalAI plugin from the Dify Marketplace when the server is LocalAI, or the OpenAI-API-compatible plugin for llama.cpp and LM Studio style servers. Pick one plugin per provider entry so the settings match the server's behavior. Check that the plugin appears under the workspace's installed plugins and that its version is recorded for later upgrade checks. The step is done when the plugin is listed as installed and shows no load error in the console.
Register the base URL and credentials
The administrator adds the model provider in the plugin settings, entering the server base URL as seen from the Dify api and worker containers, not from the administrator's laptop. If the server requires a key, enter it here and store it under the workspace's credential management rather than in shared notes or chat. Because Dify routes outbound calls through its SSRF proxy, confirm that the address is allowed by the proxy configuration, since private ranges may be blocked by default. The step is done when the provider saves without a connection error.
Add model names and set the model type
For each model the server serves, the administrator adds an entry using the exact identifier reported by the server, then sets the model type to chat (LLM) or text embedding, depending on what the model does. Set context length and the maximum output tokens to values the model actually supports, because Dify uses these limits to build prompts. Avoid registering a chat model as an embedding model or the reverse, since errors appear only later in workflows. The step is done when each model appears in the model list with the correct type and limits.
Test a completion from the provider screen
Use the provider's test or debug option to send a short prompt to the chat model, then send a short text to the embedding model if one is registered. Check that the response arrives with reasonable latency and that the text is coherent for the prompt. A failure at this stage usually points to the base URL path, a mismatched model identifier or a server that is not running the model you expected. The step is done when both test calls return a normal response with no error in the Dify console.
Confirm the request on the server side
Check the inference server's logs or its request log to confirm the test traffic arrived from the Dify host and used the expected model. This closes the loop between what Dify reports and what the server actually handled, and it catches cases where a proxy or a different server answers on the same address. Record the server version and the model file used, so a later mismatch can be traced. The step is done when the server log shows the test requests with the correct model name and a successful status.
Assign the model as a workspace default
Once tests pass, the administrator sets the chat or embedding model as the workspace default only for the model types it was verified for, so new apps and knowledge bases fall back to it. Keep production and development providers separate where your team uses different servers, because defaults apply to every app in the workspace. Document which app or knowledge base uses which model, since changing an embedding model later requires re-indexing. The step is done when a new test app selects the default model without manual changes and returns a response.
Record the setup and plan the re-test
The platform team writes down the plugin version, server software and version, base URL, model identifiers, model types and the test results, and stores this with the deployment record. Set a recurring check that repeats the completion and server-log test after each Dify upgrade, each server update and any change to the network path. Confirm with your own change process how often these checks must be recorded for your compliance program. The step is done when the record exists and a re-test is scheduled with a named owner.
Mapping Dify model settings to the local server's API
This table pairs each field Dify asks for when you add a model with the matching setting on a LocalAI or llama.cpp server, and shows where each value comes from. The owner column shows who should approve or change each value in a regulated team. Use it as a checklist before you register the first model.
| Dify field | Server equivalent and value source | Owner in a regulated team |
|---|---|---|
| Base URL | The HTTP address of the OpenAI-compatible endpoint, including the path prefix the server exposes. Confirm the exact path in your server's API documentation, since LocalAI and llama.cpp builds can differ. | Platform or infrastructure team, with a review by security for internal network routing |
| API key | A token only if the server enforces one. llama.cpp and LocalAI can run without authentication, so confirm whether your build requires a key before storing one in Dify. | Security team, stored through Dify's credential handling and rotated on a schedule |
| Model name | The identifier the server returns from its model listing endpoint. For llama.cpp this is often tied to the loaded model file name, so confirm it on the running server. | Application owner, recorded in the change request for each model |
| Model type | Chat (LLM), text embedding or rerank. A chat model loaded for embeddings will fail or return poor vectors, so match the type to the model's actual purpose. | Application owner with the data governance lead for embedding choices |
| Context length | The context window set when the model was loaded, such as the context size option on the server. Dify settings above the loaded size can cause truncated or failed requests. | Platform team sets the server value, application owner confirms it fits the workload |
| Model file and quantization | Not a Dify field, but it decides output quality and memory use. Record the file name, source and quantization level so the Dify entry can be traced to a reviewed artifact. | Model risk or AI governance owner, with the source approved under your model intake process |
| Streaming and function calling | Check whether the server supports streaming responses and tool calls for the features you plan to use in agents. Unsupported features will show up as failed agent steps, not as configuration errors. | Application owner, verified in test before production |
Where CPU-only models break down for agents
A CPU-only model is usually adequate for a single chat turn but tends to strain under agent workloads. An agent that calls tools, plans across several steps and pulls large knowledge base passages into its context sends far more tokens per user request than a simple chat. Each tool call is another model round trip, and each round trip must read the full accumulated context again. Latency therefore compounds across the run, and a model that answers a single prompt acceptably can make a multi-step workflow feel stalled. Context window size matters as much as speed. Retrieved chunks, tool schemas and prior steps compete for the same window, so the model you choose must hold the whole workflow, not just the question.
Test each candidate against a real workflow before you commit to it. Export the workflow you intend to run (Dify apps can be exported as YAML DSL files, as described in the Dify docs on app management) and run it end to end against the LocalAI or llama.cpp endpoint with representative inputs. Record wall-clock time per run, the number of model calls, whether tool calls are emitted in the format the workflow expects, and whether the final answer stays grounded in the retrieved passages. Repeat with the largest knowledge base query and the longest conversation your users will produce. Ask the team that runs the server to confirm the context length and quantization configured for each model, because those settings, not the model name alone, determine whether a long run completes.
Dify's 1.16.1 security changes do not make the model faster. Agent sandbox network isolation through a Squid proxy, token-based authentication between services and sandboxed Jinja2 rendering protect the platform, but they add no inference capacity and do not change how quickly the model generates tokens. If a workflow is slow on your test host, the sandbox is not the cause and enabling those controls will not fix it. Plan for the bottleneck to sit in the inference server, so decide early whether a CPU-only tier suits the lighter workflows at a small site and whether heavier agents need GPU-backed serving elsewhere.
Securing local model endpoints in Dify
A local LocalAI or llama.cpp server is still a network service that Dify calls, stores credentials for, and trusts with prompts and documents. The cards below cover the Dify settings and network controls that decide who can reach that endpoint, how Dify signs and protects its secrets, and who can change the credentials that point to it.
Protect the SECRET_KEY
The SECRET_KEY variable signs session cookies and JWTs and encrypts OAuth credentials. Set it once, store it in your secrets manager, and keep it out of copied .env files. Changing it after deployment logs out every user, so plan the rotation as a maintenance window. Confirm that the stored provider credentials for your local endpoint survive any change you make to this value in your environment before you test it.
Resolve the SSRF proxy conflict
The SSRF_PROXY settings route outbound requests through a proxy that blocks internal and private IP ranges. A LocalAI or llama.cpp server on a private address will look like a blocked destination. Decide whether the model server sits on a private address that must be allowed through a narrow exception, or whether it is moved to an address the proxy permits. Test the exact host and port Dify uses, not only the model name, and keep the exception as narrow as the setup allows.
Token-based service authentication
Dify 1.16.1 added token-based API authentication between services. Confirm that your release includes it, and that the internal services Dify uses to reach the plugin and sandbox layers authenticate rather than trust the network. Check the release notes for your target version before you rely on this control, and record which version you tested so that later upgrades are checked against the same behavior.
Isolate the model server
Place the LocalAI or llama.cpp server on a network segment that only the Dify API and worker containers can reach. Block direct access from user networks and from the internet, and allow only the port the server listens on. An unauthenticated inference server on a shared subnet exposes prompts, documents and compute to anyone who can route to it, so treat it as an internal database rather than a utility service.
Limit who edits provider credentials
Anyone who can add or edit a model provider can change the endpoint URL and key that Dify sends prompts and retrieved documents to. Restrict provider configuration to a small administrator group, review changes to base URLs, and keep the local server address out of workspace settings that general builders can edit. Confirm in your environment which roles can open the provider settings before you grant workspace access.
Licensing limits on shared use
The Dify Open Source License forbids using the source code to operate a multi-tenant environment, where one tenant is one workspace, without written authorization. If several business units or customers will share one local model server through separate Dify workspaces, ask the licensor for written authorization before you build that arrangement. Confirm the terms with your legal team, since this section does not give legal advice.
Common failures when Dify cannot reach a local model
This table lists the failures teams most often hit when a self-hosted Dify cannot reach a LocalAI, llama.cpp or LM Studio endpoint, with the likely cause and fix for each. Read it from the symptom you see in the Dify console or logs, then work through the cause column before changing settings. The main takeaway is that most of these failures sit in network routing, URL formatting or model naming rather than in Dify itself.
| Failure | Likely cause | Fix to check first |
|---|---|---|
| Requests to a private address are blocked | Dify routes outbound model calls through the SSRF proxy, which blocks internal and private IP ranges by design. | Confirm the SSRF_PROXY settings and the proxy's allow rules, then add a narrowly scoped exception for the inference host rather than disabling the proxy. |
| Connection fails or returns not found | The base URL is wrong, often missing the /v1 path segment or carrying an extra suffix such as /chat/completions. | Enter the server root with the path the server expects, then test it with a direct request from the Dify host before retrying in the console. |
| Model not found or invalid model error | The model name registered in Dify does not match the identifier the server reports for the loaded model. | Query the server's model list endpoint and copy the exact identifier into the Dify model configuration, including case and file extension. |
| Requests time out on CPU inference | CPU-only generation is slow, and long prompts, agent loops or large contexts push responses past the client timeout. | Use a smaller quantized model, reduce context and output length, limit agent steps, and set realistic expectations for interactive use. |
| Context length exceeded | The prompt, retrieved knowledge chunks and system instructions together exceed the context window the server loaded the model with. | Raise the context size at server start within the memory available, and lower retrieval chunk counts and chunk sizes in the knowledge base settings. |
| Plugin daemon restarts or model provider disappears | The plugin daemon restarts after a failed or heavy call, or after a container resource limit is hit, which interrupts provider calls. | Check plugin daemon logs and container memory limits, confirm the plugin version is current, and review the release notes before upgrading Dify. |
Questions teams ask before connecting Dify to LocalAI or llama.cpp
What drives the cost of connecting Dify to LocalAI or llama.cpp?
Cost comes mostly from three places: the server hardware that runs the model, the model you choose, and the integration time to connect and test it in Dify. A CPU-only server is a lower hardware commitment but limits model size and response speed. Dify itself is free to self-host under the Community edition, which is limited to a single workspace, so the Dify software license is not the main cost. Ask any partner for a quote that separates one-time work (installation, the Dify provider configuration, testing) from recurring work (model updates, Dify upgrades, monitoring). Dify's Docker Compose deployment needs at least 2 CPU cores and 4 GB of RAM, and that is before the model server itself, so size the inference host separately.
Can our own team set up Dify with LocalAI or llama.cpp without a partner?
Often yes, if you have someone comfortable with Docker, Linux service management and HTTP APIs. The Dify side is a provider configuration: you install the langgenius LocalAI plugin or the OpenAI-API-compatible plugin from the Dify Marketplace, then point it at the server's base URL and model name. The harder parts are the things teams underestimate: hardening the inference host, keeping it off public networks, testing tool-calling and output quality for your workflows, and writing an upgrade and rollback routine for both Dify and the model server. A partner is most useful when those controls must be documented for an audit or when several sites need the same pattern.
What does the work need from our team?
Plan for four inputs before work starts:
- Shell or administrative access to the Dify host and the inference server, through your normal change process.
- A test workflow or agent that represents a real task, with sample inputs and the output you expect.
- A named technical owner who can approve model choices, network rules and the production cutover.
- Network details for the model endpoint, including whether it sits on a separate VLAN or host and which ports Dify may reach.
Also confirm who will hold the credentials that Dify stores, since the SECRET_KEY setting protects those stored credentials and changing it later logs out all users.
How long does a LocalAI or llama.cpp integration take?
Duration depends on scope and the environment, not on the Dify connection itself, which is usually short once the endpoint responds. The schedule grows with the number of models to evaluate, the number of Dify workspaces and apps that depend on them, whether the host must be built from scratch, and how many approvals your change process requires. Ask for a written scope with deliverables and a timeline before work starts, and ask which environment assumptions the timeline depends on, such as existing Docker hosts, network access and test data. DeployOpen's engagements are generally fixed-scope, with scope, deliverables and timelines agreed in writing first.
Does a CPU-only model work for Dify agents?
Small CPU-only models can serve simple chat, summarization and extraction, but agents need more. Agents rely on reliable tool calling, instruction following across several steps and long context, and CPU-only inference is usually slow and limited to smaller models that handle those tasks less consistently. Test the exact agent and workflow you plan to run, measure latency under realistic concurrency, and record how often the model produces malformed tool calls. Keep the expectation narrow: a CPU-only setup suits low-volume, well-bounded tasks, while multi-step agents usually need a stronger model.
When is a cloud or GPU-backed model a better fit?
Choose a cloud or GPU-backed model when the workload needs larger models, higher concurrency, faster responses or reliable multi-step agent behavior. GPU-backed serving, such as vLLM or other inference engines, is usually the better path when many users depend on the same model. Cloud models can fit when your data handling terms allow it and the provider contract covers the data you send. Keep the local option when data must stay inside your network and the tasks suit smaller models. Many teams run both, with a workspace default model for sensitive work and a larger model for approved tasks, and confirm the data path for each.
What should we check on the Dify side before going live?
Check that the provider entry points to the intended endpoint, that the model name matches what the server reports, and that the context length and output limits in Dify match the model's actual capacity. Then run each production workflow with representative data, including failure cases such as timeouts and empty responses. Review the Dify environment settings for outbound request filtering through the SSRF proxy, because it blocks internal and private IP ranges by default and may need a deliberate rule for a model server on a private address. Document the configuration so that an upgrade of Dify or the model server can be tested against the same cases.
How DeployOpen scopes and delivers a Dify model integration
DeployOpen plans, implements, documents and supports self-hosted open-source platforms inside your own infrastructure, including Dify connections to LocalAI and llama.cpp servers. We start from the models your team needs, the hardware they will run on, and the agent workloads they must serve, then build and hand over the connection in your environment.
How we scope
- Inventory which Dify apps, workflows and agents must call a local model, and which can tolerate CPU-only latency.
- Record the LocalAI or OpenAI-API-compatible endpoints, model names, context limits and hardware each workspace will use.
- Agree the written, fixed-scope deliverables, timelines and acceptance tests before any build work starts.
How we build
- Install the LocalAI or OpenAI-API-compatible plugin from the Dify Marketplace and register each model endpoint in the workspace.
- Route outbound model calls through the SSRF proxy settings so internal inference hosts are reached only where you allow.
- Test each registered model against representative prompts, tool calls and knowledge base queries before any user sees it.
How we hand over
- Document endpoint inventory, credential locations, model versions and the workspace defaults each app falls back to.
- Provide runbooks for restarting inference servers, rotating keys and recording which model served which request.
- Walk your platform, security and application owners through the configuration and the known limits of CPU-only models.
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer after handover, scoped in writing.
- Review Dify minor releases, which arrive roughly monthly, and test model connections before you adopt each one.
- Re-test model connections after inference server upgrades, model file changes or Dify plugin updates in your environment.
