Why a self-hosted AI coding assistant on your own GPUs is a different build
If your engineers cannot send source code to a hosted coding assistant, you can run one yourself: a server such as Tabby, with editor plugins, calling models served on your own GPUs through vLLM or Ollama. The hard parts are not the model choice. They are completion latency, identity integration, managing an editor fleet, and the license limits that come with the free editions. Tabby's own README describes the server as self-contained, with no DBMS or cloud service required, and its LICENSE file places most content under Apache 2.0 while carving out the ee/ directory under its own license. Read that split before you promise anyone a free deployment for a large team.
Completion is the feature developers judge within seconds, so latency is the requirement that drives GPU sizing, not total model quality. Chat and completion have different load profiles, and concurrent developers queue on the same GPUs. Ollama's default of one parallel request per model (OLLAMA_NUM_PARALLEL) is a useful reminder that defaults rarely match a team. vLLM is the stronger candidate for shared multi-user serving, but it needs its own endpoint hardening: its --api-key option does not cover every route, so a reverse proxy should expose only the endpoints developers need. Ask the implementation partner to measure time to first completion token at your expected peak concurrency before you buy hardware.
This page covers the build decisions that differ from a general chat deployment: the coding assistant server, the inference platform underneath it, identity, the editor fleet and licensing. It does not cover model training or fine-tuning, which belong to a separate engineering practice, and it does not compare hosted coding products feature by feature. Confirm current Tabby release status and the features of each pricing tier directly with the vendor, since the Tabby release page and pricing page change more often than a use-case page can track.
Requirements a code assistant must meet before engineers rely on it
This table maps each requirement for a self-hosted coding assistant to the fact that shapes it, what it means for the build, and what to confirm with the vendor. Read it as a checklist: every row is a decision you need to settle before engineers start relying on completions in their editors.
| Requirement | What the source says | What it means for the build and what to confirm |
|---|---|---|
| Code confidentiality and no egress | Ollama states that when it runs locally it does not see prompts or data, and cloud features can be turned off with OLLAMA_NO_CLOUD=1 or the server configuration file. | Set OLLAMA_NO_CLOUD=1 and block outbound traffic from the inference hosts at the network layer. Confirm with the vendor which telemetry or update calls remain for the exact version you deploy. |
| Ollama bind address | Ollama binds to 127.0.0.1 port 11434 by default, and OLLAMA_HOST changes the bind address. | Keep the default for single-host testing. For shared use, place an authenticated reverse proxy in front of it, since the local API does not require authentication. |
| Authenticated proxy in front of vLLM | The vLLM --api-key option does not cover /metrics, /health or several other endpoints, and the docs recommend a reverse proxy that allowlists only user-facing endpoints. | Publish only the completion and chat routes the assistant uses through a proxy that enforces authentication. Test that /metrics and admin routes are unreachable from developer subnets. |
| Multi-node traffic | vLLM multi-node communication is insecure by default, without authorization or encryption, and must sit on an isolated network. | If a large model spans nodes, isolate the GPU interconnect network. Confirm how Tabby's model backend connects to vLLM and whether the path is TLS terminated at the proxy. |
| LDAP authentication in Tabby | Tabby's README notes LDAP authentication arrived in v0.24.0 (February 5, 2025). | Use v0.24.0 or later for directory-based sign-in. Confirm group mapping and how disabled accounts are removed from the assistant. |
| GitLab SSO in Tabby | Tabby's README notes GitLab SSO arrived in v0.12.0 (June 6, 2024). | Use v0.12.0 or later if your code host is GitLab. Confirm which SSO protocols are supported for other identity providers before assuming coverage. |
| Community tier user limit | Tabby's pricing page lists Community as free and open source for up to 5 users. | Suitable for a small pilot team only. Confirm the current limit on the pricing page before the rollout plan depends on it. |
| Team and Enterprise tiers | Team is listed at $19 per seat per month for up to 50 users. Enterprise is custom-priced with unlimited users and enhanced security and group management (Tabby pricing page). | Team pricing is a published per-seat figure. Ask for a written Enterprise quote covering group management and security features, and confirm which features sit in the ee/ directory. |
| Licence scope of the code | Tabby's LICENSE says content in ee/ falls under ee/LICENSE, third-party components keep their own licences, and the rest is Apache 2.0. | Review the ee/ licence before using enterprise features. Keep a software bill of materials that lists third-party components and their licences. |
| EU AI Act logging | Article 12(1) requires high-risk AI systems to allow automatic event logging over their lifetime, and Article 19(1) requires providers to keep logs for at least six months unless other law provides otherwise. | These articles apply only if your use of the assistant falls in a high-risk category. For most coding assistants, confirm your classification with counsel before designing log retention. |
Which platforms fit a self-hosted code completion stack
A self-hosted code completion stack usually pairs an assistant that talks to the IDE with a model server behind it. The choices below separate the layer developers touch from the layer that does the GPU work, and an optional gateway for access control. Each one changes what your team must license, secure and operate.
Tabby as the coding assistant
Tabby is the layer developers install in their editor, with plugins for VS Code, Vim and IntelliJ and an OpenAPI interface for cloud IDEs and internal tooling. Its LICENSE splits the code: everything outside the ee/ directory is Apache 2.0, while ee/ content falls under its own license in ee/LICENSE. Confirm which features you need sit in ee/ before you standardize on the free edition. Its README also lists support for consumer-grade GPUs, which matters for sizing.
vLLM for shared throughput
vLLM is the serving engine to choose when many engineers share one GPU pool and concurrent completions must stay fast. Its server exposes OpenAI-compatible endpoints such as /v1/completions and /v1/chat/completions, and it is licensed under Apache-2.0 with no paid edition. Cost comes from the GPUs, the infrastructure and the people who run it. Check the security docs before exposing it: the --api-key option does not cover every endpoint, so a reverse proxy should allowlist only what users need.
Ollama for small teams
Ollama suits a small team or a single GPU server where simplicity matters more than peak throughput. It is MIT licensed, runs on Linux and Docker, and exposes a REST API on port 11434 with OpenAI compatibility. The local API does not require authentication and binds to 127.0.0.1 by default, so you must add a proxy with authentication before any developer machine can reach it. Concurrency is limited by OLLAMA_NUM_PARALLEL, which defaults to 1 request per model, so size the parallel setting against your team's peak usage.
LiteLLM as an optional gateway
LiteLLM is an optional layer in front of the model server when you need per-developer keys, budgets, usage logs or routing between models. It is MIT licensed for most of its code, but its enterprise directory falls under a separate commercial license, so confirm which features you rely on are outside it. Skip the gateway if one model server and your existing identity controls already meet your access requirements. Adding it adds another service to patch, back up and monitor.
How the editor, Tabby, the model server and identity connect
A pilot usually puts Tabby and a model server on one GPU host for a small team. The IDE plugin in VS Code, Vim or IntelliJ sends completion requests to Tabby's OpenAPI interface, and Tabby forwards inference calls to the model backend on the same machine. Tabby's Docker example runs with --gpus all and --device cuda, using StarCoder-1B for completion and Qwen2-1.5B-Instruct for chat, which is a reasonable starting point for checking latency with real developers.
- Request path: IDE plugin to Tabby on a single internal hostname, then to a model server bound to localhost or a private interface.
- Identity: Community edition supports up to 5 users, so the pilot can run without SSO while access is limited to named testers on the internal network.
- Reverse proxy: Put a proxy in front of Tabby with TLS, and keep the model server off the network so only Tabby reaches it.
- Metrics: Scrape the metrics that Tabby and the model server expose, and confirm the exact paths for your versions in their documentation.
- Limits: A single host has no redundancy, so a GPU fault stops completions for the whole pilot.
Before expanding the pilot, confirm the user count against the Community limit and record which endpoints are reachable from the developer subnet.
For hundreds of engineers, the model service usually moves to a shared vLLM deployment on several GPU nodes, with Tabby acting as the coding-assistant front end. vLLM serves OpenAI-compatible endpoints, and the reverse proxy decides which of them developers can reach. Identity should sit in front of Tabby, where Tabby's LDAP authentication (added in v0.24.0) or GitLab SSO (added in v0.12.0) can map users to accounts. Confirm whether your edition includes the group management you need, since Enterprise adds it.
- Reverse proxy allowlist: Expose only the endpoints meant for users, such as the chat and completion paths, and block
/metrics,/health,/pause,/resumeand/abort_requestsfrom developer networks. - API key limits: vLLM's
--api-keycovers/v1but not several operational endpoints, so do not rely on it alone. - Multi-node networking: vLLM traffic between nodes is insecure by default, without authorization or encryption, so place those nodes on an isolated network.
- Metrics: Scrape the Prometheus metrics at
/metricsfrom the monitoring network only, covering request counts, token throughput, latency and KV cache use. - Orchestration: vLLM's Kubernetes options include Helm and the production-stack, so confirm which fits your cluster.
Check that the proxy allowlist, the isolated node network and the monitoring scrape path are each tested before the cluster serves production traffic.
In air-gapped or restricted networks, nothing can be fetched at runtime. Container images, model weights, Tabby plugin packages and documentation must be transferred and verified before they reach the enclave. Identity usually stays on the internal directory, so LDAP is the more likely integration path, and SSO depends on what the internal identity provider supports. The request path stays inside the enclave: IDE plugin, Tabby, then the model server, with no outbound route for telemetry or updates.
- Offline artifacts: Stage container images, weights and plugin builds through an approved transfer process, and record checksums for each.
- Isolated nodes: Keep multi-node vLLM traffic on a segregated network, since it is unauthenticated and unencrypted by default.
- Proxy and egress: Allowlist only user-facing endpoints, and confirm that no component attempts outbound connections.
- Metrics: Keep Prometheus scraping inside the enclave and confirm that dashboards and alert routes work without external services.
- Release cadence: vLLM ships minor releases about every two weeks, so plan a repeatable offline patch process.
Ask the security team to confirm which endpoints, package sources and identity paths the enclave policy permits before the design is signed off.
How GPU sizing for completion latency works
Completions are the latency-sensitive path, so size the GPU for them first. A developer types, the editor sends the code around the cursor to the server, and a suggestion has to arrive before the developer has moved on. Chat is tolerant of a slower first token. That difference is why the Tabby Docker example pairs StarCoder-1B for code completion with Qwen2-1.5B-Instruct for chat. Two small models serving different traffic patterns are easier to place, and to measure, than one large model asked to do both. Confirm the model choice with your own engineers, because a smaller model that completes fast but suggests poorly can cost more in rejected suggestions than it saves in hardware.
Concurrency settings decide whether a shared GPU holds up when several engineers complete code at once. In Ollama, OLLAMA_NUM_PARALLEL controls how many requests a loaded model handles at the same time (its default is 1), and OLLAMA_MAX_QUEUE limits how many requests wait behind them (its default is 512). A high queue limit hides overload by making every engineer wait, which looks like a slow tool rather than a full server. On vLLM, tensor parallelism splits one model across several GPUs inside a node, which can lower per-request latency for a large model but also consumes more GPUs per replica. Test the combination you plan to run, not the defaults, and record the setting used with each result.
Measure before you buy hardware. vLLM exposes Prometheus-format metrics at /metrics, including request counts, token throughput, latency histograms and KV cache use. Watch the latency histograms under a realistic burst and the KV cache usage over the same window: sustained high cache use means requests are competing for memory, and adding concurrency will only add queueing. Run a load test that replays completion requests built from your own repositories and your own models, at your own peak hours, and set the pass criteria with the engineering leads before the test. Ask your implementation partner to deliver the test scripts and dashboards as part of the handover, since you will repeat the test each time you change the model or the engine version.
How a pilot becomes a company-wide code assistant
The sequence moves a single-team trial to a fleet-wide coding assistant in stages, with a written decision and a named deliverable at each gate. Each stage tests one risk before the next one adds users, so the organization knows what it runs and who can use it at every point.
Agree scope and the license position
The team defines which repositories, languages and developer groups are in the pilot, and which code must never reach the assistant. Legal and engineering leads confirm which Tabby edition fits: the Community edition is free and open source for up to 5 users, while the Team and Enterprise editions are commercial. The deliverable is a signed scope document with a user count, an edition decision and an owner for the license file, including the ee/ directory terms. Confirm the license text directly with the vendor before you rely on it.
Select models and run a latency test
Engineers choose a completion model and a chat model, then measure time to first suggestion on the GPUs you plan to buy or already own. The Tabby Docker example uses StarCoder-1B for completion and Qwen2-1.5B-Instruct for chat, which is a starting point for testing, not a sizing answer. Run the test with realistic prompt lengths and concurrent typists, and record p95 latency per GPU type. The deliverable is a short sizing report that names the GPU, the model and the number of simultaneous users it supports at an acceptable latency.
Deploy Tabby and the model server behind a proxy
The platform team deploys Tabby with the chosen models and runs the model server, which can be vLLM or Ollama, on an internal network segment. Only a reverse proxy is exposed to developers, and it allowlists the endpoints meant for users. The vLLM security docs warn that the --api-key option does not protect every route, so the proxy must block administrative and metrics paths from the user network. Ollama binds to 127.0.0.1 by default, so its bind address must be changed deliberately. The deliverable is a runbook and a test report showing that blocked routes return denial from a developer subnet.
Integrate SSO or LDAP authentication
Engineers connect identity to Tabby using the method your directory supports. The Tabby README lists GitLab SSO from v0.12.0 and LDAP authentication from v0.24.0, so confirm the release you deploy includes the method you need. Test group mapping with a pilot group and a removed account, and check that the removed account loses access to the editor plugin and the API. The deliverable is a test record showing login, group-based access and deprovisioning for each identity path.
Package the editor plugins for the fleet
Tabby provides plugins for VS Code, Vim and IntelliJ, and the IntelliJ plugin is listed on the JetBrains Marketplace. The platform team decides whether developers install from public marketplaces, from an internal mirror or through managed endpoint software, and pins a tested plugin version and server URL in the configuration. Confirm that the plugin's network calls go only to the internal server. The deliverable is a packaged plugin configuration, an install guide for each supported editor and a list of approved versions.
Turn on metrics and access logging
The team collects Prometheus-format metrics from the model server, which expose request counts, token throughput, latency histograms and KV cache use, and records access logs at the proxy and in Tabby. Access records should show who requested completions, from which team, and when, without storing source code in logs unless your policy allows it. Set alerts for latency and error rates tied to the sizing targets from the test. The deliverable is a dashboard, a log retention rule approved by the security team and a monitoring owner named in the runbook.
Expand past the free user limit in controlled waves
Before the Community edition's 5-user limit is exceeded, the organization either moves to a paid edition or keeps the user count within the free terms. The expansion happens in waves by team, each gated by the sizing report and the latency dashboard. Ask the vendor for current Team and Enterprise pricing and licensing terms in writing, because those figures change. The deliverable for each wave is a go or no-go record that compares measured latency and error rates with the targets, plus a licence count that matches the seats in use.
Evidence to keep for auditors and security reviews
This table lists the evidence a security review or assessor is likely to ask for, the component that produces each item and what the reader should confirm before relying on it. Use it to plan which records the coding assistant stack generates by default and which ones must be exported and retained. DeployOpen prepares this evidence, but auditors and certification bodies issue any opinion on it.
| Evidence item | Produced by | What to keep and confirm |
|---|---|---|
| Access control configuration for SSO and LDAP groups | Tabby server authentication settings | Export the LDAP or GitLab SSO configuration and group mappings, and record who can administer the server. Confirm the Tabby version that supports the chosen method. |
| Proxy allowlist configuration | Reverse proxy in front of vLLM or Ollama | Keep the allowlist file and change history showing that only completion and chat endpoints are reachable by developers. Confirm that management and metrics paths are blocked from user networks. |
| Model and engine version records | Deployment manifests, container image tags and release history | Record the Tabby, vLLM or Ollama versions and model file hashes in each environment, with the release date of each version taken from the project's release page. |
| Request and access logs | Tabby, the reverse proxy and the inference engine | Retain authentication events, request metadata and proxy access logs in central storage. Confirm what prompt and code content each component writes to disk by default. |
| Metrics history for capacity | Prometheus-format metrics at /metrics on the vLLM server, plus Ollama and host telemetry | Keep request counts, latency histograms and KV cache use over time to show completion latency against GPU capacity. Confirm the retention period with the security team. |
| License file review for each component | Component repositories and the license files in each release | Record the license of Tabby, the model weights, vLLM or Ollama and each editor plugin. Note that Tabby's ee/ directory uses a separate license, and that the free Community edition is limited to 5 users. |
| Model licence and usage terms | Model publisher's card and license text | Store the license for each model used for completion and chat. Confirm that the terms allow internal commercial use and that no code training terms conflict with company policy. |
What teams ask before building a self-hosted coding assistant
What does a self-hosted coding assistant cost to build and run?
Most of the cost sits in three places: GPU or accelerator capacity, the surrounding infrastructure, and the people who build and operate it. Tabby's Community edition is free and open source for up to 5 users, and vLLM and Ollama carry no license fee for self-hosted use, so the software line is often zero. Recurring costs are the GPUs (purchased or rented), power and hosting, and engineering time for upgrades and on-call. The main cost driver is how many engineers you serve and how many completions they trigger during peak hours. Ask any partner for a quote that separates one-time work (design, installation, integration with your identity provider, editor rollout) from recurring work (monitoring, model updates, engine upgrades, support), and that states which GPU models and counts the figure assumes.
How long does a pilot of a self-hosted coding assistant take?
The duration depends on scope, not on the software, which installs quickly. A pilot usually covers one or two GPU nodes, one completion model and a small group of engineers, and its timing depends on how quickly GPU capacity is available, how long identity and network approvals take, and how many editors you need to support. No duration is fixed until the scope is written down. Ask a partner to show the pilot's exit criteria in writing, such as completion latency measured with your own repositories, acceptance by a named group of developers, and the security review outcome, and to list which of your teams must approve each step.
What must engineers and platform staff provide before the build starts?
The project depends on a short list of inputs from your side. Platform staff need to provide GPU hosts or cloud accounts, network paths from developer workstations to the inference nodes, DNS names, TLS certificates and a decision on where logs are stored. Security needs to approve the model sources and container images and to confirm the identity provider. Engineering leads need to name the repositories, languages and editors in scope and the developers in the pilot group. Ask a partner to produce a written intake list that names an owner for each item, because delays usually come from missing approvals rather than from installation.
Can our internal team build and run this without an implementation partner?
Yes, a team with Linux, container and GPU operations experience can deploy Tabby, vLLM or Ollama using their published documentation. The harder work is operational: putting an authenticated proxy in front of the engine, wiring SSO or LDAP, sizing GPUs for completion latency, keeping the stack patched and documenting controls for a compliance review. Tabby added LDAP authentication in v0.24.0 and GitLab SSO in v0.12.0, so confirm which login method your edition supports before you plan. An internal team should estimate its own hours for these tasks. A partner is most useful when you need the platform in a fixed timeline, want a documented handover, or need evidence prepared for an audit.
Which limits of the free Tabby edition matter for our engineering team?
The Community edition is free and open source for up to 5 users, so any team beyond five developers will hit the limit. The Team plan, listed at $19 per seat per month for up to 50 users, is the next paid step, and the Enterprise plan has custom pricing with unlimited users and added security and group management. Check whether the features you need, such as LDAP, group management or enhanced security, are included in the edition you choose, because the free and paid editions differ. Also review the license for the ee/ directory, which the Tabby LICENSE file says is governed by a separate license. Ask your vendor or partner to confirm the current terms in writing before you commit.
How do we handle upgrades when vLLM and Ollama release so often?
Both engines release frequently, so upgrades need a fixed process rather than ad hoc changes. vLLM shipped minor releases about every two weeks in the period from v0.28.0 on August 26 to v0.31.0 on October 5, 2026, and Ollama published v0.40.5 on October 14, 2026. Not every release needs to reach production. A workable process pins versions, tests each candidate release against a fixed set of completion and chat prompts from your repositories, and schedules rollouts on a cadence your team can support. Ask a partner how they test engine and model updates, how they roll back a bad release, and who approves each change.
How do we secure the inference endpoint and keep source code inside our network?
The inference engine is the part most likely to be exposed by mistake. The vLLM security documentation says the --api-key option does not protect every endpoint, including /metrics and /health, and recommends a reverse proxy that allows only the endpoints meant for users. Ollama listens on 127.0.0.1 port 11434 by default and does not require authentication locally, so it needs a proxy before any developer can reach it. Keep the engine on an isolated network, disable cloud features with OLLAMA_NO_CLOUD=1 where Ollama is used, and set logging policies for prompts and completions. Ask a partner to show the exact allowlist and a network test that proves the engine is unreachable from outside the segment.
What to ask an implementation partner before signing
Start with GPU sizing, because it sets the budget and the latency promise. Ask the partner which model, quantization, context length and concurrency their sizing assumes, and ask for the load test that produced the number, not a spec sheet. A credible answer names the completion latency target per developer, the vllm serve settings used in the test and the metrics read from /metrics during the run. Be cautious if the partner sizes by seat count alone or cannot show measured time-to-first-token under concurrent typing. Ask for the same test to be repeated on the hardware you will buy, since consumer-grade GPUs are listed as supported for Tabby but their headroom differs from data center cards.
Next, ask how the endpoint is exposed. vLLM's --api-key option does not cover /invocations, /pause, /resume, /abort_requests, /metrics or /health, and the security documentation advises against relying on it alone. The right answer is a reverse proxy that allowlists only the paths developers need, blocks the rest from the network and keeps /metrics reachable only by monitoring. Ask the partner to show the proxy rules and a test that proves a blocked path returns a denial. For LDAP and SSO, ask how group membership maps to access, how disabled accounts are removed from Tabby, and whether the test covered a user who leaves the directory group. Confirm the Tabby version that supports your identity method against the Tabby GitHub releases page before accepting the design.
Finally, ask about licensing, upgrades and handover. Tabby's ee/ directory falls under a separate license, and the free Community tier is limited to 5 users, so ask the partner to name every feature in use and confirm which tier and license cover it. Ask how Tabby, vLLM and model weights are upgraded, who approves a release, and how a rollback is tested, since vLLM ships minor releases about every two weeks. Require a written handover covering runbooks, dashboards, credential ownership and a named operator for GPU failures. A partner who cannot answer these questions in writing, or who suggests a shared admin account for the server, should not be signed.
How DeployOpen scopes and builds a self-hosted coding assistant
DeployOpen plans, implements, integrates, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, including the GPU servers that host a coding assistant. For this use case we start from which editors, repositories and identity systems your engineers use, then size the model service and the editor plugins around that.
How we scope
- Written scope covers the editors, IDEs and repositories in use, the Tabby deployment and the model serving layer.
- We agree the GPU or accelerator inventory, network placement and expected number of concurrent developers before build starts.
- Scope names the SSO or LDAP integration, the license edition chosen and the user count it must support.
How we build
- We deploy Tabby and the vLLM or Ollama model backend on your hardware, with the endpoints behind a reverse proxy that allowlists only intended paths.
- We configure editor plugins for the IDEs your teams use and test completion latency against the agreed targets.
- We wire authentication, metrics collection and access logging, then run security testing of the endpoints before go-live.
How we hand over
- You receive architecture and configuration documentation, runbooks for model and engine updates, and the sizing assumptions behind the GPU choice.
- Handover covers operating the service, rotating credentials, adding models and reading the metrics dashboards.
- Compliance evidence such as access records and configuration baselines is packaged for your own auditors to review.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers, agreed separately from the initial scope.
- Retainers can cover tracking Tabby and vLLM releases, testing upgrades in a staging environment and applying them on your schedule.
- Support includes investigating latency regressions and capacity pressure as your developer count and repositories grow.
