Self-hosted LLMs and the EU AI Act: what applies

Does the EU AI Act apply to self-hosted LLMs?

Yes, the EU AI Act applies to a self-hosted large language model, but it applies by role and use, not by where the model runs. Running an open-weight model in your own cloud account, data center or air-gapped network does not take you outside the Act. Article 4 on AI literacy has applied since 2 February 2025 to providers and deployers of all AI systems, so staff who build, operate or rely on the model already fall within that duty. Three questions decide the rest: whether you are a provider or a deployer (Article 3 defines both by whether you develop and place the system on the market under your name, or use it under your authority), what the system is used for (a high-risk use in an Annex III area, or a transparency use such as a chatbot or generated content), and where the data and outputs flow, including to customers or other people in the EU.

Work through these questions in writing before you choose tooling. A company that fine-tunes an open-weight model and offers it to customers under its own name is likely a provider, and must check whether a general-purpose model rule or a fine-tuning duty applies to it. An internal assistant that drafts text for employees is usually a deployer with transparency and literacy duties. A self-hosted screening or scoring tool used in hiring or lending may be high-risk, which brings logging and retention duties under Articles 19 and 26(6), with at least six months as the baseline for logs under your control. Confirm your role and use classification with counsel or your compliance lead, since the Commission's guidance on these definitions is still being applied. DeployOpen prepares the systems, configuration and evidence for these checks. Your organization owns its controls and its compliance decisions, and only an independent auditor, assessor or certification body issues reports or certificates.

Which duties apply to which self-hosted use

Running an open-weight model on your own infrastructure does not switch the AI Act off, and it does not switch every duty on. Your role (provider or deployer), the purpose of the use and whether the use falls in a high-risk category decide which duties apply. The cards below cover the duties that most self-hosted LLM deployments meet first.

AI literacy for staff

Article 4 applies to providers and deployers of every AI system, including a internal assistant that only summarizes meeting notes. Staff who operate or rely on the model need a level of AI literacy suited to their role and the context in which the system is used. Check that you can show who was trained, on what, and when, and that training covers the model's known limits such as fabricated answers and data exposure through prompts. Keep training records with the system inventory.

Chatbot disclosure to users

Article 50(1) applies to providers who design an AI system intended to interact directly with people, so users must be told they are talking to an AI unless that is obvious to a reasonably well-informed person in context. If you host a model and build the chat interface yourself, you are likely the provider of that system. Check the interface text, the first message, and any voice or embedded widget where the disclosure could be missed.

Marking generated content

Synthetic text, images, audio and video produced by your self-hosted model may need machine-readable marking and, for deepfakes and certain public-interest text, disclosure to people. The Commission published a Code of Practice on marking and labelling AI-generated content on 10 June 2026, and guidelines on transparency duties followed on 20 July 2026. Confirm which outputs your use produces, which marking method your pipeline applies, and whether the output is ever published to the public.

High-risk use and log retention

If your self-hosted model supports a use in an Annex III area, such as recruitment screening or credit scoring, you may be a provider or deployer of a high-risk system. Article 19 (providers) and Article 26(6) (deployers) require logs under your control to be kept for at least six months, unless other Union or national law sets a longer period. Your tracing stack must capture inputs, outputs, and model versions, and its retention settings must be configured to hold those logs for the required period.

Annex III high-risk dates

As amended in 2026, the high-risk rules for Annex III systems apply from 2 December 2027, instead of 2 August 2026. Annex I product systems follow on 2 August 2028. Use the date to plan your inventory, risk classification and documentation, but do not treat it as the start of the work. Log retention, data governance and technical documentation take time to build, so confirm with your legal team which of your uses fall within Annex III and which deadline applies to each.

Chatbot transparency and log retention in practice

The chatbot notice has one job: make sure a natural person knows they are talking to an AI system before or during the exchange. Under Article 50(1), the duty sits with the provider that designs the system, so if your team builds the assistant on an open-weight model and exposes it to customers or staff, the notice is part of what you build and test. Evidence is usually a screenshot or recording of the first screen, the greeting text in the front-end configuration, and a test record showing the notice appears in every entry point, including the mobile app and any embedded widget. The exception is narrow: the notice can be skipped only where the AI nature is obvious to a reasonably well-informed, observant and circumspect person in that context. Document the reasoning for that call per use case rather than assuming it, and expect an assessor to ask for it.

Log retention is a separate question from the notice, and the answer depends on the intended purpose of the system. For high-risk uses, Articles 19 and 26(6) require logs under your control to be kept for at least six months unless other Union or national law sets a different period. Article 12 shapes what those logs must show, such as the period of use, the reference data checked and the identity of people who verified results, so logging a prompt and response alone may not be enough. In a self-hosted stack, an observability tool such as Langfuse can record traces with inputs, outputs, token usage and metadata, but retention windows, masking and audit trails may depend on the enterprise license features, so confirm which ones your edition includes. Check the Commission's transparency guidelines of 20 July 2026 for their current reading before fixing the notice wording or retention period.

Which open-source tools produce which AI Act evidence

Traces are the most direct source of record-keeping evidence when a self-hosted model sits inside a high-risk use. Langfuse, released under the MIT license, records each request as a trace made of observations, so a single user interaction can be followed from input through the model call and any downstream steps. Its documentation describes traces that capture LLM calls, retrieval steps, tool executions and custom logic, with timing, inputs, outputs, token usage and metadata. Those fields let a team show what the system received, what it produced and which version or configuration handled the request.

Request-level detail: Each trace links the user request to every model call, retrieval step and tool execution that followed it.
Timing and token usage: Latency and token counts per call support capacity reviews and investigations of unusual behavior.
Inputs and outputs: Stored prompts and responses form the factual record that a human reviewer or auditor can sample.
Metadata: Tags for model version, prompt version and deployment environment connect each trace to the configuration in force at that time.
Retention decisions: Traces may contain personal data, so the retention period and access rules need to be set deliberately and documented.

Confirm with the vendor which trace features are included in your license tier, and check your own log retention requirements against the Article 19 and Article 26(6) minimum of six months where those articles apply.

How to get a self-hosted LLM ready for an AI Act review

This sequence turns a self-hosted large language model into a documented set of records, tests and traces that an internal owner or independent assessor can review. Each step produces a specific artifact, and the order matters because the classification decides which duties apply to the evidence you collect.

Inventory each model and use
The system owner lists every open-weight model you run, every application or workflow that calls it, and the intended purpose of each use. For each entry, record who operates the system, whether your organization built, modified or only deployed it, and which users or affected people it touches. Done means every model and use has a named owner and a written intended purpose, and the list is kept current when a new model version or use case is added.
Classify each use against Article 6 and Annex III
Legal or compliance staff, with the system owner, test each intended use against Article 6 and the Annex III categories to decide whether it is high-risk, subject only to transparency duties, or outside the high-risk and transparency categories. Record the reasoning, including any Article 6(3) analysis. Done means each inventory entry carries a classification decision signed by the internal owner, and any use that may be high-risk is flagged for the full set of steps below.
Set AI literacy training records
Work out which staff operate, supervise or rely on each system, then deliver role-appropriate AI literacy training that covers how the model behaves, its known limits, and when to escalate. Keep attendance and content records per person and per role. Done means every person in the inventory's operator and supervisor lists has a dated training record that the compliance owner can produce on request. The Commission's AI literacy questions and answers are a useful reference for the training scope.
Configure tracing and log retention
The platform team enables request tracing for each deployment, capturing inputs, outputs, retrieval steps, tool calls, token usage and metadata that link each output to a model version and user context. Set access controls and a retention period for these logs, and confirm which logs fall under your control. For high-risk uses, keep logs for at least six months unless other Union or national law requires longer, as Articles 19 and 26(6) require. Done means a test request appears in the trace store with all expected fields and the retention setting is documented.
Run documented red-team tests
The security team runs structured adversarial tests against each model and application, covering prompt injection, data leakage, jailbreaks, hallucination and toxic output. Open-source probe tools can generate and score many attempts, and the report should show every attempt, the evaluation result, and the hits that found a weakness. Record the findings, the fixes, and the retests. Done means each test run has a versioned report, each finding has an owner and a disposition, and the results are linked to the model version tested.
Document transparency measures for users
For any system that interacts directly with people or generates synthetic content, the product owner records how users are told they are interacting with AI and how generated content is marked, following Article 50 and the current Commission guidance. Include screenshots or configuration evidence. Done means each user-facing system has a dated disclosure record that matches what users actually see.
Assemble the technical file and hand over evidence
The compliance lead gathers the classification decisions, training records, trace configuration, test reports, disclosure records and change history into one file per system, organized by the requirement it supports. The internal owner reviews the file, accepts residual risk in writing, and decides what goes to an independent assessor if one is engaged. Done means the owner has signed the file and each item is traceable to its source system. Note that only an independent auditor or notified body issues conformity findings, and the implementation partner prepares the evidence without issuing it.

What a self-hosted LLM compliance program costs and who gets paid

This table separates the cost components of an AI Act program for a self-hosted LLM by who is paid, whether the cost is one-time or recurring, and what drives it up. Read it as a checklist for requesting quotes: each row should map to a written scope item. No figures appear because none were supplied for this topic, so ask each party for its own pricing basis.

Cost component	Who is paid or who bears it	One-time or recurring	What drives the cost
Internal staff time (owner, legal, security, data)	Your own staff; no external payee, but the time has a real cost	Both: one-time for role mapping and inventory, recurring for reviews and log checks	Number of use cases, whether the model is high-risk, and how many teams deploy it
Implementation partner for tracing, testing and evidence set-up	DeployOpen or another implementation partner, under a fixed scope agreed in writing	Mostly one-time, covering trace instrumentation, test runs and evidence packs	Number of model endpoints, integrations, and whether logs must meet a retention period
Tooling and licenses	Open-source projects are free to use under their licenses; enterprise license keys are paid to the vendor	Recurring where a license key is needed, such as audit logs or RBAC features	Whether you need enterprise features like self-hosted audit logs, project-level roles or SCIM
Independent assessor, where a conformity route needs one	An independent body; DeployOpen does not issue reports or certificates	One-time per assessment cycle, with repeats when the system changes	Whether the system falls under internal control or a notified body route, and its scope
Ongoing testing, upgrades and monitoring	Retainer with the implementation partner, or staff if kept in-house	Recurring	Model version changes, new use cases, frequency of red teaming and log reviews

What must be in place before an assessor reviews the system

An assessor will not start with the model. They will start with the records that show who is responsible, what the system does, and whether the controls ran as written. The six items below are the minimum set that connects a self-hosted LLM deployment to the AI Act duties that apply to its use.

Current AI system inventory

List every self-hosted model, application, agent and vendor-hosted AI feature in use, with its owner, purpose, users, the data it touches and the environment it runs in. Confirm the inventory was updated after the last model swap, prompt change or new integration. An assessor will compare this list against network logs and deployment manifests, so gaps between the two are the first finding to expect. Record the date of the last review.

Documented role determination

For each entry in the inventory, record whether your organization is the provider or the deployer of the system, with the reasoning under Article 3. Note where you modified an open-weight model, built a product on it, or only configured a tool someone else built. A role that changes after fine-tuning is the most common blind spot, so the record should name the trigger that would prompt a review.

AI literacy records

Keep a current list of staff and contractors who operate, supervise or rely on the system, with the training each received and the date it was completed. Training content should match the role, so engineers who tune prompts and reviewers who approve outputs are not given the same material. Check that the records are refreshed when the model or its intended use changes.

Disclosure text and where it appears

Where users interact directly with the chatbot or assistant, collect the exact notice text, the interface screens or pages that display it, and the release version in which it was introduced. Check whether the notice is obvious from context. Generated content that could be mistaken for human-made material also needs a labelling approach, so record how outputs are marked in exports, emails and API responses.

Log retention settings matching the policy

If any use could be high-risk, logs under your control must be kept for at least six months unless other law requires longer. Export the retention configuration from the tracing and storage layers and compare it with the written policy. Confirm that backups and deleted-trace handling follow the same rule. Note that some retention, masking and audit features in self-hosted tools depend on an enterprise license, so confirm which edition you run.

Test records with dated runs and fixes

Store the output of each adversarial test run with its date, the model and configuration tested, and the probes used. Garak writes a full JSONL report and a hit log of successful attempts, so keep both. For each hit, record the fix, the retest date and the result. An assessor will ask whether a finding was closed, not only whether testing took place.

Who owns the controls and who prepares the evidence

Your organization remains the operator under the AI Act. Whether you are a provider or a deployer of a self-hosted model depends on facts only you know: whether you develop or substantially modify the model, whether you place it on the market or put it into service under your own name, and what use case it serves. Those role decisions, the classification of each use, and the accountability for them cannot be handed to a partner or an auditor.

Role and use decisions: Record whether you act as provider or deployer for each model and use case, and why.
Transparency and disclosure: Decide how users are told they are talking to an AI system and how generated content is labelled.
Staff AI literacy: Own the training program and records that show staff have sufficient AI literacy for their roles.
Policies and sign-off: Approve the acceptable use rules, log retention periods and incident procedures, and sign them off at the right level.
Operation and accountability: Run the system day to day, review evidence, and answer for it to customers, staff and regulators.

Before starting any work, confirm which named person signs off each role decision and where those decisions are recorded.

Questions buyers ask about self-hosted LLMs and the AI Act

Do we need a notified body for an internal chatbot?

Usually no. A notified body is involved only for certain high-risk systems, such as some biometric systems and Annex I products, and most Annex III high-risk systems use internal-control conformity assessment instead. An internal chatbot that answers staff questions is generally not high-risk on its own, so the duties that apply are mainly AI literacy under Article 4 and, if the chatbot talks directly to people, the disclosure duty in Article 50(1). The route depends on the use case, not on the model. If a chatbot is used for hiring, credit, insurance pricing or another Annex III purpose, the analysis changes, and you should confirm the classification with counsel or an assessor before you launch. Ask any consultant to put the route and its reasoning in writing.

How long must we keep logs for a self-hosted LLM?

For a high-risk use, the AI Act requires logs under your control to be kept for a period appropriate to the intended purpose, of at least six months, unless other Union or national law provides otherwise. Providers face this under Article 19 and deployers under Article 26(6). The six-month figure is a floor, not a target, and sector rules such as financial or health record laws may require longer retention. For a use that is not high-risk, the Act does not set this period, so set retention from your own risk and legal review. Confirm the retention period with your legal team before you configure deletion, because removing logs too early can break your evidence trail.

Can our own team implement this without outside help?

Yes, if your engineers have the skills and time. Self-hosting an open-weight model, configuring tracing, testing outputs and writing the evidence files are engineering tasks that internal teams can do. Effort is driven by the number of models and applications, how many integrations the logs must cover, the sensitivity of the data, whether you need a named enterprise feature, and how much evidence an assessor will expect. Governance work, such as the AI inventory, risk classification and policies, usually needs legal and compliance staff as well. An implementation partner can prepare systems and evidence under a fixed written scope, but only an independent auditor or certification body issues reports or certificates. Ask any partner for a written scope and deliverables before work begins.

What drives the cost of meeting AI Act duties for a self-hosted LLM up?

The main cost drivers are scope and the number of moving parts. More models, more use cases, more applications calling them, and more data classes all increase the work. Higher data sensitivity adds masking, access control and retention design. A use that falls into a high-risk category adds documentation, logging, human oversight and conformity assessment work that a transparency-only chatbot does not need. Costs usually fall into one-time items such as architecture, initial testing and policy writing, and recurring items such as log review, model upgrades, re-testing and retainers. No public price list applies to these services. Ask for a quote that separates one-time and recurring work, names the models and use cases in scope, and states what the partner will not cover.

Does open-weight status exempt us from AI Act duties?

No. The open-source exemptions in the Act are narrow and mainly affect providers of certain free and open-source models and systems. They do not remove deployer duties. If your organization uses an open-weight model under its own authority, AI literacy under Article 4 still applies, and transparency duties under Article 50 apply to any system that interacts with people or generates content. Your classification also depends on how you use the model, so a self-hosted model in a high-risk use faces the same high-risk rules as any other system. Whether you are also a provider, for example after fine-tuning, is a separate question. Read the open-source topic on this site for the exact exemptions and the duties that remain.

Do we need a Langfuse enterprise license for audit evidence?

It depends on the features you need. The Langfuse documentation states that self-hosted audit logs, project-level RBAC roles, data retention policies, server-side data masking, protected prompt labels and SCIM need an enterprise license key. Basic tracing of LLM calls, retrieval steps, tool executions, inputs, outputs, token usage and metadata is part of the open-source project under the MIT license. If your evidence plan depends on audit logs, retention controls or masking, a license may be needed, so confirm the current terms with the vendor before you design around a feature. Also check the license file in the project repository. Get written confirmation of which features your version and license cover.

How DeployOpen works on this

DeployOpen is a Chainscore Labs company that plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure. For self-hosted LLMs under the EU AI Act, we prepare the systems, logs and test evidence that your team and an independent assessor can review, while the organization keeps ownership of its role determinations and compliance decisions.

How we scope

Confirm which provider or deployer role each model use creates, based on your own determinations and legal review.
Inventory the models, serving stack, user-facing chatbots and generated-content flows that fall within the written scope.
Agree the written scope, deliverables and timeline before work starts, since most engagements are fixed-scope.

How we build

Deploy the chosen open-source tracing platform in your cloud account, data center or air-gapped network, with access controls you approve.
Configure trace capture for LLM calls, retrieval steps and tool executions, and set log retention to match your legal minimums.
Run garak probes against the models in scope and keep the JSONL reports and hit logs as test evidence.

How we hand over

Document the architecture, data flows, logging configuration, retention settings and test results in a handover package.
Map each piece of evidence to the AI Act article it supports, so your assessor can trace it without guesswork.
Confirm which enterprise features, such as audit logs or data masking, need a license key before you rely on them.

How we support

Offer ongoing support, upgrades and continuous testing as a retainer after the initial engagement ends.
Re-run agreed tests when models, prompts or serving configurations change, and update the evidence accordingly.
Work alongside your independent auditors or assessors, who issue any reports or certifications, not DeployOpen.

Self-hosted LLMs and the EU AI Act