EU AI Act for open-weight models: deployer duties

Does the EU AI Act still apply when we run open-weight models ourselves?

The Act still applies, mainly through the deployer role. When staff or customers call an internal endpoint that serves a downloaded open-weight model, the organization is using an AI system under its authority, which Article 3(4) of Regulation (EU) 2024/1689 treats as deployment unless the use is personal and non-professional. Open-source status narrows some duties but does not remove them. Article 2(12) excludes free and open-source AI systems only if they are not placed on the market or put into service as high-risk systems and do not fall under Article 5 or Article 50. Downloading a model does not make the organization a provider of that model, but the function should record whether it modifies the model, fine-tunes it, or offers the service under its own name, because each of those can move it toward the provider role. Confirm that boundary with counsel before launch.

The 2026 amendment changed when the heavy obligations bite. Regulation (EU) 2026/1744, published on 24 July 2026 and in force from 27 July 2026, moved the high-risk rules in Chapter III Sections 1 to 3 to 2 December 2027 for Annex III systems and to 2 August 2028 for Annex I product systems. Article 4 on AI literacy is not deferred. It has applied since 2 February 2025 to providers and deployers of all AI systems, high-risk or not, so the staff who operate the endpoint, build tools on it or supervise its use need documented training suited to their roles. The European Commission's AI literacy questions and answers is the next read for what it expects in practice.

For an assessment, the function should produce a written evidence pack: a classification of each use case against the Commission's four risk levels, a record of the license and version of each model and serving engine in use, a register of who operates the service and what training they completed, and the logging configuration for the gateway and inference server. Article 26(6) requires deployers to keep logs under their control for at least six months unless other Union or national law sets a different period, so retention should be set to that floor or longer and tested. Any change of model or engine should trigger a fresh review of the role and risk analysis, since an upgrade or a fine-tune can change the answers.

Which role you hold: deployer, provider or both

When your organization runs an open-weight model on its own servers and staff use it for internal work, you are usually a deployer. Article 3(4) defines a deployer as anyone using an AI system under its authority, except in a personal non-professional activity. An internal chat service used for drafting, summarizing or search fits that definition even though no one outside the organization ever sees the output. The Act applies to deployers established in the EU, and to non-EU deployers whose AI output is used in the EU, under Article 2.

Deployer duties still apply: Article 4 on AI literacy has applied since 2 February 2025 to providers and deployers of all AI systems, so staff who operate or rely on the service need training proportionate to their role.
Log retention: Article 26(6) requires deployers to keep logs under their control for at least six months where high-risk rules apply, unless other law sets a longer period.
Open-source exemption does not cover use: Article 2(12) excludes free and open-source AI systems, but not if they are put into service as high-risk systems or fall under Article 5 or Article 50.
Use cases decide the risk tier: An HR screening tool built on the same model may be high-risk under Annex III, even though the model is identical.
Staff tools sit on top: Chat front ends and gateways you add do not change your role, but they affect what you can log and show.

Confirm with counsel which internal uses could fall into a high-risk category before rollout.

Mapping each EU AI Act requirement to what the function must do

This table maps each EU AI Act requirement that matters for an in-house model service to the duty it creates, and to the settings in the inference server, gateway and observability stack that support it. Read each row against your own role: whether you are a deployer, or have become a provider by modifying a model or putting it into service under your own name. The main takeaway is that the open-source exemptions are narrow, and log retention and AI literacy apply to most internal deployments regardless.

Requirement	What the function must do	How to configure the inference server, gateway and observability stack
Article 4 AI literacy (applies since 2 February 2025)	Make sure staff who operate or use the model service have a sufficient level of AI literacy for their role. Record training and scope it to the people who build, run and rely on the service.	Not a technical setting. Tie gateway API keys or SSO groups to roles, so you can show which teams have access and which training they completed.
Article 3(3) provider role	Check whether you develop the system, have it developed, or place it on the market or put it into service under your own name, including free of charge. Serving a model for your own staff under your name can still make you a provider of an AI system.	Document who builds the wrapper application, prompts and fine-tunes. Keep the model card, version and build records for each deployed configuration in your change system.
Article 3(4) deployer role	Confirm when you use an AI system under your authority in a professional context. Most organizations serving open-weight models internally are deployers unless they fall under the provider definition.	Restrict who can call the endpoint through the gateway, and label each internal application with an owner and business purpose so the deployment inventory is traceable.
Article 2(12) open-source exclusion and its exceptions	The Act excludes AI systems released under free and open-source licenses, but not if they are placed on the market or put into service as high-risk systems, fall under Article 5 (prohibited practices), or fall under Article 50 (transparency). Confirm which exception applies to each use case before relying on the exclusion.	Keep license records for each model and engine (for example, the Apache-2.0 license of vLLM and the MIT license of Ollama). Tag use cases that touch Article 5 or Article 50 topics, such as chatbots disclosing AI interaction, in the gateway so they go through review.
Articles 53(2) and 54(6) open-source exemptions for general-purpose models	Open-weight models released under a free and open-source license with public weights, architecture and usage information are exempt from two documentation duties and from appointing an authorised representative. They are not exempt from the copyright policy or training data summary, and not if the model has systemic risk.	Store the model license, weight source and any published model documentation with each model version. Flag models that may meet systemic-risk criteria for legal review, since the exemption does not apply to them.
Article 19 and Article 26(6) log retention (at least six months)	Providers and deployers must keep logs under their control for a period appropriate to the intended purpose, of at least six months, unless other Union or national law requires longer or shorter. Logs of prompts, outputs and model versions are the likely evidence set.	Set gateway and observability retention to at least six months and confirm it with legal, since sector rules may set longer periods. Export logs to storage you control, and record the model version, engine version and user or key identifier with each request.
Article 12(1) automatic recording of events for high-risk systems	Where a system is high-risk, it must technically allow automatic recording of events over its lifetime. Under the 2026 amendment, Annex III high-risk rules apply from 2 December 2027, so plan for this in the design phase.	Enable structured request logging at the gateway, and trace events in the observability stack, including timestamps, input and output references, and the model and version used. Confirm log completeness with the vendor's documentation and your own tests.
Article 99 fine ceilings	Fines are EUR 35 million or 7% of worldwide annual turnover for prohibited practices, EUR 15 million or 3% for most operator obligations, and EUR 7.5 million or 1% for misleading information to authorities, whichever is higher. The ceilings are the reason to scope roles and exemptions in writing before launch.	Not a configuration item. Keep a dated scope memo, the role analysis and the exemption reasoning next to the deployment record so you can show how the decision was made.

Open-source platforms that support the compliance work

The platforms below produce much of the technical evidence an organization needs to show how it runs open-weight models: request records, access points, and traces of what was asked and answered. Each one has licence terms or default settings that change what you can deploy and what you must add yourself before the evidence is usable.

vLLM inference server

vLLM serves open-weight models behind OpenAI-compatible endpoints and exposes Prometheus-format metrics at /metrics, covering request counts and token throughput. It is Apache-2.0 with no paid edition, so the cost is the GPU capacity and the people who run it. The main gap is access control: the --api-key option does not cover /metrics or /health, so put a reverse proxy in front that allowlists only the user endpoints. Confirm which of these endpoints your deployment exposes before relying on it.

Ollama local runtime

Ollama is MIT licensed and binds to 127.0.0.1 on port 11434 by default. Its local API requires no authentication, so exposing it to staff means placing an authenticated proxy in front and changing the bind address deliberately with OLLAMA_HOST. Set OLLAMA_NO_CLOUD=1 so that no cloud features send requests outside your network, and record the model storage directory set with OLLAMA_MODELS as part of the model inventory. Test these settings on the version you deploy, since releases ship frequently.

LiteLLM gateway logging

LiteLLM sits between applications and models and can record requests, apply guardrails and track cost per caller, which gives you the audit trail that a deployer needs for logs under its control. It is MIT licensed except for the enterprise directory, so confirm which features you need fall outside that directory before you adopt them. Decide the retention period for gateway logs in advance, because the AI Act sets a minimum of six months for logs under the operator's control unless other law provides otherwise.

Langfuse trace records

Langfuse records traces of prompts, outputs, costs and evaluation scores in storage you control, which supports documentation of how a deployed model behaved over time. It is MIT licensed except for its ee directories, which carry separate terms, and its copyright is held by ClickHouse, Inc. Before you store customer or staff prompts, define who can read traces, how long they are kept, and how personal data is masked. Keep these traces separate from the gateway logs so each record has a clear owner.

Phoenix tracing and evaluation

Phoenix provides LLM tracing and evaluation, so it can support the same kind of record as Langfuse for teams that prefer it. Its Elastic License 2.0 prohibits providing the software to third parties as a hosted or managed service, so you may run it for your own staff but cannot offer it as a managed service to customers. If your partner or a third party operates it for you, have counsel confirm that the arrangement is permitted. Compare its retention and access controls with Langfuse before you choose one.

Evidence auditors and market surveillance authorities ask for

This table lists the records an assessor or market surveillance authority is likely to ask for, which component produces each one, and the retention period tied to Article 19 or Article 26(6) where it applies. Read it as a preparation list: DeployOpen prepares this evidence inside your environment, but it does not issue any audit report or attestation, and independent auditors or authorities make the determination. Retention periods are the minimum the Act sets for logs, so confirm longer periods with counsel and your own record policies.

Evidence	Produced by	Retention and notes
AI literacy training records	Learning platform or LMS, plus attendance and completion exports from the training owner	Article 4 applies to all AI systems since 2 February 2025. The Act sets no fixed retention period, so set one in your records policy and confirm it with counsel.
Role determination memo	Legal or compliance review, recorded in the AI system inventory	States whether you act as deployer or provider for each use, citing Article 3(3) and 3(4). Keep it current whenever a model, use case or customer-facing output changes.
Model licence and weight provenance	Model register, with license files, download source and checksums recorded at intake	Shows the license terms and that weights came from the publisher. Keep for as long as the model is in service, since it supports the Article 2(12) and Article 53(2) analysis.
Model card and usage information from the publisher	Publisher documentation archived with the model version in the model register	Supports the open-source exemption analysis under Article 53(2). Archive the exact version reviewed, because publishers can update cards.
Log retention policy and exports	Inference gateway and engine logs, exported to controlled storage on a schedule	Article 19(1) and Article 26(6) require logs under your control to be kept for at least six months unless other law sets a longer period. Where the system is high-risk, Article 12(1) requires automatic event logging.
Access records for the endpoint	Reverse proxy and identity provider logs covering who called which endpoint and when	Retain under the same log policy as inference logs. The vLLM docs warn that --api-key alone is insufficient, so the proxy log is often the primary evidence.
Change and upgrade records for engine releases	Change tickets, release notes and test results for each vLLM or Ollama version deployed	Shows which engine and model versions served which period. Keep the version history for the full period a log could be relevant to an inquiry.
Incident records	Incident tracker, with timelines, root cause and notifications made to authorities or customers	Covers serious incidents and malfunctions involving the system. Confirm the notification route and retention with counsel, because the Act's reporting duties depend on the classification.

Where the open-source exemption stops

The open-source carve-outs in the EU AI Act are narrower than they first look, and the first check is whether your deployment falls inside them at all. Under Article 2(12), AI systems released under free and open-source licenses are excluded from the Act, but only unless they are placed on the market or put into service as high-risk systems, or fall under the prohibited practices in Article 5 or the transparency duties in Article 50. An internal assistant that screens job applicants, for example, stays in scope regardless of the license on its weights. Map each use case against those three triggers before you rely on the exclusion.

For general-purpose models, Article 53(2) exempts models released under a free and open-source license with public weights, architecture and usage information from the technical documentation duties in points (a) and (b), and Article 54(6) lifts the requirement to appoint an authorised representative for the same models. The exemption does not cover the copyright policy or the public training data summary, and it falls away entirely for models with systemic risk. Confirm the exact license terms and whether the publisher's release actually includes all three elements, because a model published with weights only does not qualify. Document that check in your model inventory.

Two practical points follow. Article 50 transparency duties can apply to a staff chatbot or to generated content whatever the model license, so disclosure and labelling need an owner. Also check the timetable: the 2026 Digital Omnibus, Regulation (EU) 2026/1744, moved the high-risk rules in Chapter III to 2 December 2027 for Annex III systems and 2 August 2028 for Annex I product systems. Read the consolidated text of the Act as amended before fixing a compliance date, and confirm with counsel which obligations apply to your organization on those dates.

Questions buyers ask before hiring help with the EU AI Act

How much does EU AI Act compliance work cost for an open-weight model service?

No supplied figure gives a price for this work, and any quote depends on written scope. The main cost drivers are the number of AI systems in scope, whether each use case is a deployment you run or a product you place on the market, how much of the logging and access control already exists, and how many business owners must be interviewed and documented. Ask a partner to separate the one-time work (inventory, classification, documentation, logging setup) from recurring work (log review, re-assessment when a model or use case changes, and support after the 2026 amendments). Compare quotes line by line against the same written scope, and ask who in the quote performs legal review, since DeployOpen prepares evidence and does not issue legal opinions or certifications.

How long does an EU AI Act readiness engagement take?

No supplied duration applies to every organization, so the timeline depends on the written scope. It is set mainly by how many systems are in scope, how well current AI use is documented, how quickly owners can answer questions, and whether logs are already being collected from the inference endpoint. Ask for milestones tied to deliverables, such as an inventory, a role and risk classification per use case, a logging design, and an evidence pack, rather than a single end date. Also confirm which dates drive the plan: the Digital Omnibus changed the high-risk rules to apply from 2 December 2027 for Annex III systems and from 2 August 2028 for Annex I product systems, so a plan built on the older dates needs to be rebased.

What does our team need to provide for an AI Act readiness review?

Four things make the work move. First, a list of every model, application and workflow that uses the inference service, including who built it and where its output goes. Second, a named owner for each system who can answer for its purpose, users and decisions. Third, read access to the serving stack's configuration, reverse proxy rules and log storage, so logging and retention can be checked against Article 12 and Article 19 or Article 26(6), which set a minimum of six months unless other law says otherwise. Fourth, a decision on who approves the final evidence pack, since your organization remains accountable for its own classification and compliance statements.

Can our own team handle EU AI Act compliance without a partner?

Yes, many teams can prepare the inventory, the owner assignments and the logging review internally, especially if they already run an AI governance or model risk process. The work becomes harder when several business units deploy models, when the organization might become a provider by modifying or placing a model on the market under its own name, or when the evidence must be assembled from engineering logs that nobody has reviewed. Ask yourself whether someone can hold the inventory current after launch and whether they can explain the Article 4 AI literacy duty, which has applied since 2 February 2025 to providers and deployers of all AI systems. A partner can add capacity and technical checks, but the accountable decisions stay with your organization.

Does using open-source models reduce our fine exposure under the AI Act?

Open-source licensing narrows some duties but does not remove fines from the picture, and the exemptions have limits. Article 2(12) excludes AI systems released under free and open-source licenses, unless they are placed on the market or put into service as high-risk systems or fall under Article 5 or Article 50. Article 53(2) exempts certain general-purpose models with public weights, architecture and usage information from two documentation duties, but not if the model has systemic risk, and not from the copyright policy or training summary. Article 99 sets maximum fines of EUR 35 million or 7% of worldwide annual turnover for prohibited practices, EUR 15 million or 3% for most operator obligations, and EUR 7.5 million or 1% for misleading information, whichever is higher. Have counsel confirm which category each model and use case falls into.

How did the 2026 amendment change the high-risk dates?

Regulation (EU) 2026/1744, the Digital Omnibus on AI, moved the high-risk rules in Chapter III Sections 1 to 3. They now apply from 2 December 2027 for Annex III systems and from 2 August 2028 for Annex I product systems, instead of 2 August 2026 and 2 August 2027. The amendment was published on 24 July 2026 and entered into force on 27 July 2026. Other rules, including Article 4 on AI literacy, have applied since 2 February 2025, so the later dates do not postpone everything. Confirm with your counsel which obligations already apply to your current deployments, and build the inventory and logging work now so it is not rushed before the high-risk dates.

What tooling alone does not satisfy

A model server, a gateway and a logging stack can make the technical parts of EU AI Act readiness easier to meet, but they cannot decide your legal role, judge your use cases or run your policies. The cards below cover the decisions and records that remain with your organization, and the points to confirm with counsel or a notified body.

Choosing your legal role

Under Article 3, a provider is whoever develops or has developed an AI system and places it on the market or puts it into service under its own name, free of charge or not. An organization that downloads an open-weight model and serves it to staff is often a deployer, but building a product on that model under your own name can make you a provider. Decide this per service, in writing, before launch, and confirm the reading with counsel.

Running AI literacy programs

Article 4 has applied since 2 February 2025 and covers providers and deployers of all AI systems, including internal assistants. No product creates literacy. You need a training scope, role-based content for administrators, developers and business users, and records showing who was trained and when. Check the Commission's AI literacy guidance and agree what evidence your auditors will expect.

Judging high-risk use

Whether an assistant is high-risk depends on what it is used for, such as employment decisions, credit or access to essential services, not on the model underneath it. Only a documented use-case review can answer that. Annex III high-risk rules apply from 2 December 2027 under the 2026 amendment. Have counsel sign off on each use, and involve a notified body only where the system falls in a category that requires one.

Confirming licence terms per model

Open-weight does not mean free to use in every way. The exemptions in Article 53(2) apply only to models released under a free and open-source license with public weights, architecture and usage information, and never where the model has systemic risk. Check each model's license, each engine's license and each interface's terms, since some front ends restrict branding or multi-tenant use. Record the findings in your model inventory.

Retaining and reviewing logs

Article 26(6) and Article 19 require logs under the deployer's or provider's control to be kept for at least six months unless other law sets a longer period. Producing logs is not enough: you must set a retention period tied to the intended purpose, define who reviews them, how often, and what triggers escalation, and document that policy. Confirm the retention period with counsel against your sector rules.

How to start the assessment in the next few weeks

Start with a model and use inventory, because every other step depends on knowing which open-weight models are running, which engine serves them (vLLM, Ollama or llama.cpp), and which staff or customer-facing use each one supports. Record the model name, its license and whether its weights, architecture and usage information are published, the engine version, the hosting location, and the owner of each use case. Include pilots and experiments, since staff often run models on laptops or test servers that the organization still controls. Once the list exists, the scope document becomes the basis for any fixed-scope quote, and it should name each model, each use and each environment in the same terms.

For each use, decide the role the organization holds. Serving a model to staff under the organization's authority usually makes it a deployer under Article 3(4), while building or substantially modifying a system for others, or placing it on the market under its own name, can make it a provider under Article 3(3). Check each model's open-source license with the source files in hand, and confirm whether any feature, such as a commercial directory or a cloud route, falls outside the license. Ask counsel to confirm how Article 2(12) and Article 53(2) apply to each model and use, and do not assume an open-source license settles the question. Document the decision for each use so auditors can trace it.

Next, set log retention and AI literacy. Confirm which logs your serving stack writes, such as gateway, endpoint and application logs, who can read them, and where they are stored. Articles 19 and 26(6) require at least six months unless other law sets a longer period, so write the retention period into the scope and confirm it with legal counsel and any sector regulator. Finally, schedule AI literacy training for staff who build, deploy or use the service, following the Commission's questions and answers on Article 4. Agree the training owner and refresh cycle now, because both appear in the evidence an assessor will ask for.

How DeployOpen scopes and delivers EU AI Act readiness for self-hosted models

DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside the client's own infrastructure. For open-weight models, we start from the organization's role under the AI Act, then build the serving stack, logging and evidence around that role, while the compliance assessment itself stays with the organization's own auditors and assessment bodies.

How we scope

Confirm which models, engines and user groups are in scope, and whether the organization acts as a deployer, a provider or both.
Record which models carry a free and open-source license with public weights, architecture and usage information, and which do not.
Agree in writing which AI literacy, log retention and documentation deliverables the engagement covers before work starts.

How we build

Deploy the inference engine, such as vLLM or Ollama, with an authenticated reverse proxy that exposes only the endpoints users need.
Configure request and event logging so retention can be set to match the period the organization's obligations require.
Test endpoints for exposed routes and access control, then document the configuration, model inventory and change history.

How we hand over

Deliver the architecture record, runbooks, model inventory template, log configuration and access control documentation to the owners named in the scope.
Walk operations and compliance teams through the evidence set so they can show it to their own auditors or regulators.
Document the points where a legal or compliance decision remains with the client, including provider status and high-risk classification.

How we support

Engagements are fixed-scope, and ongoing support, upgrades and continuous testing are available as a retainer.
Track engine and model releases, apply updates in a test environment first, and retest the endpoint configuration after each change.
Review the evidence set when the organization's use changes, such as a new user group, a new model or a new regulatory date.

Running open-weight models in-house under the EU AI Act