Self-hosted LLM inference for universities and research

FERPA requires the institution to record each request for access to and each disclosure of personally identifiable information from education records, including the parties involved and their legitimate interests, and to keep that record for as long as the records are kept (34 CFR 99.32). For a shared model service, the record has to show who or what touched student data, not only that the service existed. The usual source is the gateway or application layer that authenticates users, combined with the model server's own request logs, which DeployOpen would configure to capture user identity, time and the data scope of each request. Your registrar or compliance office decides which disclosures count and how long records are kept. Confirm with counsel whether prompt logs containing student information fall under this requirement before you set retention. 34 CFR Part 99 sets the full rule.

The evidence is an access configuration plus test results showing unauthorized requests fail. vLLM's --api-key option protects the /v1 routes but not /metrics, /health, /pause or /invocations, and the vLLM security documentation advises a reverse proxy that allowlists only the endpoints meant for users. So the artifacts are the proxy allowlist, the identity provider configuration that issues credentials, a record of which endpoints are reachable from which network segments, and scan or penetration test output showing the blocked paths. Ollama binds to 127.0.0.1 on port 11434 by default and its local API requires no authentication, so the same proxy-and-network evidence applies there. DeployOpen can prepare these configurations and tests. Auditors decide whether the controls meet the institution's requirements.

The Safeguards Rule requires multi-factor authentication for anyone accessing any information system that holds customer information (16 CFR 314.4(c)(5)). For an inference service, you need to show that the identity provider enforces a second factor for administrators, operators and, where the policy covers them, end users, and that the model servers and admin consoles sit behind it. Useful artifacts are the identity provider policy export, enrollment reports, and a sample of login events showing the second factor was used. Be aware that a model endpoint called only by a backend service account has a different control story, so document service credentials separately. Every Title IV institution agreed to comply with the Safeguards Rule in its Program Participation Agreement, and the Department of Education examines GLBA compliance in annual audits (FSA Electronic Announcement GENERAL-23-09). Confirm the scope with your compliance office.

The institution needs a written retention policy that states what is stored, for how long, who can read it and how it is deleted. The policy has to cover prompts and outputs that may contain student records, server and gateway logs, and metrics. Three components produce the evidence. The gateway or chat interface controls conversation storage and deletion. The model server writes request logs and Prometheus metrics at /metrics, which hold counts and latency rather than prompt text, so their retention can differ. Observability tools such as Langfuse or Phoenix, if used, store full traces and need their own schedule. For children under 13, COPPA requires a written retention policy with a deletion timeframe (16 CFR 312.10). Ask legal counsel to set the periods. DeployOpen can implement and test the deletion jobs, then document what ran.

A partner that performs work the institution would otherwise do with employees can receive education records as a school official, but only if it is under the institution's direct control for use and maintenance of the records and follows the redisclosure limits in 99.33(a) (34 CFR 99.31(a)(1)(i)(B)). In practice, the evidence is a signed services agreement with confidentiality, data use, redisclosure and return or destruction clauses, a record of which staff can access the environment, and a data processing description. Open-source components carry no license fee to the institution's vendor file, but the implementation contract and any support retainer are contracts that should be on record. Ask counsel to confirm whether the agreement meets the school official condition. DeployOpen can supply operational documents such as access lists and change records, but the contract language is for your legal team to approve.

FERPA requires institutions to use reasonable methods, such as physical or technological access controls, so school officials see only the records in which they have a legitimate educational interest (34 CFR 99.31(a)(1)(ii)). For a shared model service, the evidence is the mapping of departments and roles to permitted data sources, enforced by the gateway or retrieval layer, and test results showing a user outside that group cannot retrieve another department's documents. If the service uses a retrieval layer, its access rules and the vector store's permissions become evidence too. Keep the role matrix under version control so you can show what it was on a given date. DeployOpen can build and test these controls, and your records office and auditors decide whether they are reasonable for the institution.

If the institution's use falls under high-risk obligations, Article 12(1) requires the system to technically allow automatic recording of events over its lifetime, and Article 19(1) requires providers to keep those logs for at least six months unless other law provides otherwise. Whether the use case is high-risk, and who is the provider or deployer, is a legal question. If it applies, the evidence is the logging configuration, a retention schedule of at least six months, proof that logs are collected from the model server and gateway, and a tested restore or export procedure. The Article 12 and Article 19 pages on the EU AI Act Service Desk set out the text. Confirm your classification with counsel before relying on these artifacts.

The artifacts come from specific components: the identity provider (MFA and user lists), the reverse proxy or gateway (endpoint allowlists and request logs), the model server (request and metrics records), the storage layer (retention and deletion jobs), and the contract file (agreements). DeployOpen plans and configures these components, runs tests and collects the output into an evidence pack, which is a support service rather than a certification. The institution owns the policies, and its auditors, regulators or the Department of Education decide whether the evidence meets the law. Ask any implementation partner to list the evidence each component produces and where it is stored before work begins.

Most of the cost sits in GPU or accelerator capacity, the surrounding servers, storage, networking and power, and the people who operate the service. The hardware is usually a one-time purchase or lease, while power, space, support contracts, staff time and engineering changes recur every year. Utilization is the biggest swing factor, because a shared service that sits idle between semester peaks still carries its full fixed cost. vLLM and Ollama have no paid editions, so software licensing is not the main line item for the engine itself, but open-source platforms still need someone to install, patch and monitor them. A quote should separate one-time work (design, installation, hardening, integration, documentation) from recurring work (upgrades, monitoring, on-call, model updates) and state which hardware is included, who pays for power and space, and what happens when demand grows. Ask for each line item to be tied to the written scope.

The timeline depends on whether the hardware is already installed, how many departments are in the pilot, and how many integrations must work on day one. A pilot usually starts with one model, one engine, one identity provider connection and a small group of users, so it can be scoped and priced before the full service is funded. The vLLM engine is built for production serving, but it still needs GPU drivers, a reverse proxy in front of it and a monitoring setup, and each of these adds steps. Ask any partner for a dated plan with milestones, a named list of what the pilot excludes, and the acceptance criteria that end it. DeployOpen agrees timelines in writing as part of a fixed-scope engagement, so the pilot dates appear in the statement of work rather than in a sales estimate.

Both teams need to supply decisions and access before engineering begins. IT usually owns network segments, identity integration, certificates, backup targets and change control, and must confirm where the servers will sit and which systems can reach them. Research computing typically owns the GPU inventory, scheduling policy, storage for model weights and the fair-share rules that decide who gets capacity during peaks. Both groups should agree on the models to serve, the expected concurrency and the context lengths users need. A written list of these inputs, with named owners and due dates, prevents the most common delays. Ask the partner to produce that list during the scoping phase and to confirm which items are the university's responsibility and which fall inside the engagement.

A single department can run a small service alone if it has GPU capacity, staff who can operate Linux servers and container platforms, and a clear owner for security and patching. The harder problems appear when several departments share the service. Shared use needs per-department quotas, authentication, usage records and an approach to support, which are central IT concerns more than research concerns. The vLLM security documentation states that its API key protects only some endpoints and recommends a reverse proxy that allowlists only the endpoints meant for users, so a department without security engineering support may leave gaps. Ask whether the department can staff on-call coverage, maintain the patch cadence of a fast-moving engine and document its controls for an audit. If those answers are uncertain, a shared service run by central IT is usually easier to govern.

FERPA applies to educational agencies and institutions that receive funds under programs administered by the Secretary of Education, and its regulations are in 34 CFR Part 99. If a prompt or output contains personally identifiable information from education records, the logs that hold it become part of the institution's records management problem. FERPA requires institutions to keep a record of each disclosure of personally identifiable information from education records, and access controls must limit school officials to records in which they have a legitimate educational interest. An outside contractor may act as a school official only under the conditions in 34 CFR 99.31(a)(1)(i)(B), including direct institutional control over the use and maintenance of the records. Decide early whether prompts are logged, for how long, who can read them and how deletion works. Confirm these choices with your registrar and general counsel rather than relying on a vendor's default settings.

A hosted API can be a sound choice for public data that carries no student, employee, research or confidential information, such as published course catalogs, public web content or open datasets. The decision turns on what the prompt contains, not on the model itself. Self-hosting makes more sense when prompts include student records, unpublished research data, export-controlled material or staff personnel files, because those flows would otherwise need contract, data processing and retention terms with a third party. Hosted services also carry usage pricing that changes with volume, so the break-even point depends on the request volume and the utilization of your own hardware. Ask the university's legal team to classify the data for each use case, then match each class to a service. Many campuses run both, with a policy that routes sensitive work to the internal service.

A quote should list the scope in writing: the models and engines to deploy, the user groups, the authentication method, the quota model and the integrations. It should separate hardware from services and state which items the university buys, which it already owns and which the partner supplies. Services should cover design, installation, hardening, testing, documentation, handover, training and any retained support or upgrade work, each with deliverables and acceptance criteria. The quote should also state the assumptions it depends on, such as rack space, power, network segments and staff availability, and what changes would trigger a new estimate. Because DeployOpen works from fixed-scope written agreements, ask for every exclusion to be listed explicitly. Compare quotes on the same scope rather than on headline totals.