Private LLM chat and the EU AI Act

Does the EU AI Act apply to a private LLM chat assistant for staff?

If you are deploying Open WebUI or LibreChat on Ollama or vLLM for staff, the first EU AI Act duty to plan for is AI literacy under Article 4, not the high-risk rules. Article 4 has applied since 2 February 2025 and covers providers and deployers of all AI systems, not only high-risk ones. A general staff chat assistant is normally a deployer situation: your organization uses the system under its authority for professional purposes. The duty is to take measures that support the AI literacy of staff and others who operate AI for you. The Commission's AI literacy questions and answers state that no certificate is required and no training format is mandatory, so the evidence you keep (training records, usage guidance, an acceptable use policy) matters more than any particular course.

Scope follows from Article 2. The Act applies to providers placing AI systems on the EU market wherever they are established, to deployers established or located in the EU, and to non-EU providers and deployers whose AI output is used in the EU. Regulation (EU) 2026/1744, the Digital Omnibus on AI, amended the Act on 8 July 2026 and entered into force on 27 July 2026. It replaced the wording of Article 4(1), which now requires measures to support AI literacy without guaranteeing any specific level for an individual. It also moved the Annex III high-risk date to 2 December 2027. Confirm the consolidated text on EUR-Lex with your legal counsel before you rely on any date, and confirm which national market surveillance authority would supervise your organization.

When building the assistant in-house makes you a provider

When you buy a chat product from a vendor and roll it out to staff under the vendor's authority, the vendor is the provider and your organization is the deployer. Article 3(3) defines a provider as the party that develops an AI system, or has one developed, and places it on the market or puts it into service under its own name or trademark. Article 3(4) defines a deployer as the party that uses an AI system under its authority outside personal non-professional use. Your core duty is AI literacy under Article 4, which has applied to deployers of all AI systems since 2 February 2025.

Provider duties stay with the vendor: For a high-risk system, design, conformity assessment, CE marking and post-market monitoring fall to the provider under Articles 16, 72 and 73.
Deployer duties sit with you: Use the system as the vendor instructs, assign human oversight and keep staff AI literacy measures current.
Purpose decides the risk level: The same chat product used to draft meeting notes differs from one used to rank job applicants, so the intended use drives the analysis.
Literacy has no fixed format: The Commission's AI literacy Q&A states that no certificate is required and no training format is mandatory, but you must show that measures were taken.

Ask the vendor for its written statement of intended purpose and whether it claims conformity for your use case.

Mapping Article 4, Article 12 and Article 19 to chat platform configuration

This table maps the EU AI Act duties most relevant to a staff chat assistant to the platform settings that support them. Read the Article 12 and Article 19 rows as conditional: they bite mainly if the assistant's use case is classified as high-risk, while the Article 4 literacy duty and the scope check apply to the deployment now. The main takeaway is that the chat platform's logging, identity and data-scope settings produce most of the evidence an assessor will ask for.

Requirement	What the function must do	How to configure it
Article 4(1) AI literacy measures	Take measures so staff and others operating the assistant have sufficient AI literacy for their context. No certificate or mandatory training format is required.	Deliver role-based training, publish an acceptable use policy and show staff guidance in the chat interface, such as a banner or system prompt notice on first login. Keep dated records of who was trained and when.
Article 2 and 3 scope check	Determine whether the organization is a provider (builds or has built the system and puts it into service under its own name) or a deployer (uses it under its authority).	Document whether an in-house build makes the organization a provider. Record the model source, the chat software and any vendor role, and confirm this with counsel before rollout.
Article 12(1) automatic event recording (high-risk only)	Technically allow automatic recording of events (logs) over the system's lifetime if the use case is high-risk.	Enable request and response logging in Open WebUI or LibreChat, and trace model calls through LiteLLM or Langfuse. Confirm which events are captured and whether prompts and outputs are stored.
Article 19(1) six-month minimum log retention (high-risk only)	Keep the logs for at least six months unless other law requires longer retention.	Set retention in the chat database and log store to at least six months and align it with GDPR storage limits. Confirm the setting with legal, since other law may require a different period.
Data scope: local-only operation	Limit where prompts, files and outputs travel, which supports the data and transfer analysis around the assistant.	Serve models with Ollama on local hardware, which binds to 127.0.0.1 port 11434 by default. Set OLLAMA_NO_CLOUD=1 to turn off cloud features and avoid cloud inference keys.
User attribution: SSO and RBAC	Tie each logged event to an identifiable user and role so logs can be reviewed and access is limited.	Connect Open WebUI to the corporate identity provider through OIDC or LDAP, use SCIM 2.0 provisioning and role-based groups, and disable shared accounts. Confirm the setup with the vendor documentation for your version.
Model and provider inventory	Keep a current record of which models and endpoints the assistant uses, supporting the Article 4 and Article 12 evidence.	Route all model calls through a single gateway or the Ollama endpoint, and record model names, versions and licenses in a register that is reviewed on each update.

Which self-hosted open-source platforms fit an AI Act evidence trail

A staff chat assistant spreads its evidence across several components: the interface that holds chat history, the runtime or serving engine that answers prompts, the gateway that records calls, and the tracing layer that stores outputs. Each one has a license, an authentication model and logging behavior that an organization must confirm before it can point to them as records.

Open WebUI as the chat record

Open WebUI holds chat history, user accounts and group permissions for the assistant, so it is the first place an auditor will look for who used the system and when. Its license includes a branding clause that applies above 50 end users in any rolling 30-day period, which affects both the interface and any white-labeled rollout. Confirm the license position before wide release. For multiple replicas, plan PostgreSQL, Redis and shared file storage, and confirm how SSO through OIDC or LDAP and SCIM 2.0 provisioning map to your identity source.

LibreChat as an MIT alternative

LibreChat is an AI chat interface under the MIT license, which carries fewer distribution conditions than Open WebUI's branding clause. It is a credible option when the organization wants a permissive license for the interface layer and is prepared to own its own configuration and access controls. Check how it authenticates users, where it stores conversations and whether its logs contain enough detail to show who used a model, on what date and with which data, since those details support a record-keeping argument under Article 12.

Ollama as the local runtime

Ollama runs models on the organization's own hardware under the MIT license, and its documentation states that local runs do not expose prompts to Ollama. The local API on port 11434 requires no authentication, so the endpoint must sit behind a reverse proxy or be bound only to a protected network before any staff tool can reach it. Confirm that cloud features are disabled where the deployment must stay offline, since they are optional and use a separate API key.

vLLM for shared serving

vLLM is an inference and serving engine under the Apache-2.0 license, suited to a shared internal model service with more concurrent users than a single workstation runtime handles well. The evidence question is operational: who may call the endpoint, which model version answered each request and how the serving configuration changed over time. Ask the implementation partner to document model version pinning and the change log for each update, since these become part of the record an assessor will request.

LiteLLM for request logs

LiteLLM sits between the chat assistant and model providers, and it offers logging, cost tracking and guardrail hooks in one place. It is MIT licensed except for its enterprise directory, so the organization must confirm which features it needs fall outside the open part before relying on them. A gateway log is often the most complete record of model calls, but it only helps if retention, access and time synchronization are set deliberately. Ask what each log entry contains and where it is stored.

Langfuse for traces and evaluation

Langfuse records traces, prompts, outputs and evaluation results for LLM applications, and it can run in the organization's own storage. It is MIT licensed except for its ee directories, which carry separate terms, so confirm which modules are in the deployed build. Traces are useful evidence of how outputs were produced and reviewed, but they also contain personal data and must be covered by the same retention and access rules as chat history. Decide early what is traced and for how long.

How to scope, configure and document the assistant before rollout

This sequence turns a staff chat assistant into a documented deployment: a classified use case, a stated provider or deployer role, a trained and governed user base, controlled identity and logging, and a pilot that produces evidence before wider access. Each step closes with an output the compliance and security owners can review.

Classify the use case and your role
Start by writing down what the assistant will do for staff, which models and document sources it will touch, and who operates it. Decide whether your organization is a deployer, which uses an AI system under its authority, or a provider, which develops the system or has it developed and puts it into service under its own name. Building the assistant in-house on a self-hosted open-source stack is the point where this question needs a deliberate answer, because the provider duties are substantially heavier. The output is a one-page use case statement signed by the business owner and the compliance lead, with the role recorded and the reasons given. Confirm the role with counsel against Article 3 of the Act before the pilot starts.
Confirm whether any use is high-risk
Compare the documented use cases against the Act's risk levels and the Annex III list of high-risk areas. A general staff assistant for drafting, summarizing and internal search is usually not high-risk, but a use that feeds employment decisions, access to essential services or other Annex III purposes can be. Record every use that is out of scope and every use that needs a further assessment, and state the reasoning. The high-risk rules in Chapter III now apply from 2 December 2027 for Annex III systems, so a deployment that starts now should still be designed with that date in view. The output is a risk screening record listing each use, its risk level and the owner who approved it.
Build the AI literacy program and acceptable use policy
Article 4 applies to providers and deployers of all AI systems, so the staff chat assistant needs an AI literacy measure even when no use is high-risk. Identify who operates or is affected by the assistant, including administrators, prompt-library authors and staff who upload files, and set training content that fits each group. The Commission's Q&A states that no certificate is required and no training format is mandated, so the program can be a short onboarding module, a policy briefing and role-specific guidance. Pair it with an acceptable use policy that states which data classes may be pasted or uploaded, which tasks are prohibited, and who to contact. The output is a training record per user group and a signed policy version that users acknowledge in the interface.
Configure identity, roles and model access
Connect the assistant to the corporate identity provider through OIDC or LDAP so that every chat is tied to a named account, and use SCIM 2.0 provisioning so that leavers lose access without a manual step. Map groups to roles and restrict which models, knowledge collections and tools each group can use. Set the model endpoints so that the chat interface reaches the inference service only through the internal network. Confirm each of these settings with the vendor documentation for your chosen release, because feature names and behavior change between versions. The output is a role matrix and a test report showing that a user outside a group cannot reach its models or documents.
Set logging, retention and the evidence trail
Decide what the platform records: user, time, model, prompt and response metadata, uploaded file references and administrative changes. Article 12 requires high-risk systems to allow automatic event recording over their lifetime, and Article 19 requires providers of high-risk systems to keep those logs for at least six months unless other law requires otherwise, so the logging design should be ready for that outcome even if your use is not high-risk. Set chat retention to match your policy, and make sure logs are written to storage you control and cannot be edited by the people they describe. Document where logs live, who can read them and how they are exported. The output is a logging specification approved by security and legal, with a test showing that a sample event appears in the log store.
Pilot with named groups and review results
Roll the assistant out first to a small set of named groups, such as one business unit and one central function, and keep the pilot under the acceptable use policy and the training record from earlier steps. During the pilot, review sampled logs for policy violations, check that restricted data classes were not uploaded, and collect user feedback on misleading or harmful output. Record each finding with its owner and the change made in response. Hold a go or no-go decision before each wider group is added. The output is a pilot report that lists the groups, the findings, the fixes and the approval for the next wave.
Keep the evidence file current
Assemble one evidence file that holds the use case statement, the role determination, the risk screening, the training records, the signed policy, the configuration and role matrix, the logging specification, and the pilot report. Assign an owner for each document and a review trigger, such as a new model, a new knowledge source, a platform upgrade or a change in the Act's dates. Keep the file in controlled storage and version it with every change. This file does not certify compliance, but it is the material that internal audit, a market surveillance authority or an independent assessor will ask to see. The output is a versioned evidence index that the compliance lead signs off each quarter.

Evidence a market surveillance authority or auditor is likely to ask for

This table lists the records a market surveillance authority or auditor is likely to request from a self-hosted staff chat assistant, what each record shows, and which component produces it. Read it as a preparation checklist: DeployOpen prepares the evidence from the implementation, while any attestation or certification comes from an independent auditor or certification body, not from DeployOpen.

Evidence item	What it shows	Which component produces it
AI literacy measures record	Training or guidance offered to staff and others operating the assistant, with dates and who took part	Organization's learning records, plus the onboarding materials delivered through the chat interface
Role and scope classification memo	Why the assistant is treated as a minimal-risk or transparency-only use, and whether any use case could reach Annex III	Written memo owned by the organization's legal and compliance team, informed by the intended use inventory
Acceptable use policy and user acknowledgment	Rules for permitted prompts, uploads and restricted data, and each user's acceptance	Policy document, with acknowledgment captured through SSO login or the chat interface
SSO and group mapping export	Which identities can reach the assistant, which groups hold which permissions, and when they changed	Open WebUI identity configuration (OIDC or LDAP, SCIM 2.0 provisioning, RBAC groups)
Chat and access logs with retention setting	Who used the assistant, when, and how long records are kept, supporting traceability	Open WebUI application logs and database, the configured retention policy, and the log store
Model inventory with versions and licenses	Which models, runtimes and interface versions run in production, with their license terms	Ollama or vLLM model registry, Open WebUI release record, and the organization's asset register
Change records for model and platform updates	When models or components were upgraded or rolled back, who approved it, and test results	Change tickets, deployment pipeline history and test reports kept by the operations team

What tooling alone does not satisfy under the AI Act

A self-hosted chat assistant gives the organization control over prompts and files, but several EU AI Act duties are about people, decisions and records that no platform produces. The cards below cover the work that stays with the organization after the software is installed.

No software delivers AI literacy

Article 4 requires providers and deployers to take measures to support the AI literacy of staff who operate AI for them. The amended text does not require a specific level for any individual, and the Commission's Q&A states that no certificate is required and no training format is mandated. Chat platforms can show a usage notice, but they cannot train people or prove competence. Ask who owns the training program, which staff groups it covers, and how completion and refresher content are recorded.

Provider or deployer is a legal judgment

Whether the organization is a provider or a deployer depends on whether it develops the assistant, or has it developed, and puts it into service under its own name. Building the assistant in-house with a self-hosted model and selling or offering it to others can move the organization into the provider role. Record the classification decision with counsel, and revisit it when the assistant's purpose, branding or distribution changes.

Conformity assessment is organizational work

If an organization is a provider of a high-risk system, it carries quality management, a conformity assessment, a declaration, CE marking, registration, post-market monitoring and serious incident reporting. The high-risk rules for Annex III systems now apply from 2 December 2027. The first harmonised standard, EN 18286:2026 on quality management, was cited in the Official Journal on 14 October 2026. Confirm with the assessor which route applies and who signs the declaration.

Branding clause is a licensing decision

The Open WebUI License prohibits altering or removing Open WebUI branding in deployments above 50 end users in any rolling 30-day period, unless the operator holds an enterprise license or written permission. This is a commercial and legal question, not a configuration setting. Decide early whether the assistant will be white-labeled for staff, and get the license position in writing before the user count grows past the threshold.

Logs must be kept and reviewed

Article 12 requires high-risk systems to technically allow automatic event logging over their lifetime, and Article 19 requires providers to keep those logs for at least six months unless other law provides otherwise. Turning on logging is only the first step. Name the people who review chat and admin logs, set a retention period that other laws allow, and document how reviews are done and what triggers escalation.

Enforcement varies by national authority

National market surveillance authorities, not the Commission's AI Office, supervise Article 4 as of 2 August 2026, and each Member State organizes its own enforcement. Fines under Article 99 can reach EUR 15 million or 3% of worldwide annual turnover for most operator obligations, whichever is higher. Because practice differs across countries, confirm with counsel which authority applies to your establishment and how it has signaled its approach.

Questions buyers ask about EU AI Act compliance for a staff chat assistant

Do we need certification for AI literacy under the EU AI Act?

No. The Commission's AI literacy questions and answers state that no certificate is required and that there is no mandatory training format. Article 4 asks providers and deployers to take measures that support the AI literacy of staff and others operating AI on their behalf, and the amended text does not require a specific level for any individual. What auditors and supervisors will look for is evidence of those measures: a documented training approach matched to the roles that use the assistant, records of who was covered, and a way to refresh content as the tool changes. National market surveillance authorities, not the AI Office, supervise Article 4 as of 2 August 2026. Confirm the expected evidence format with your legal counsel and your national authority before designing the program.

How much does an EU AI Act readiness engagement cost?

No figure can be given here, because cost depends on the written scope. The main drivers are the number of AI systems in scope, whether the assistant is built in-house or only configured from open-source components, how many user groups need tailored literacy material, how much existing policy and logging evidence you already have, and how many business units must review intended use. Most DeployOpen engagements are fixed-scope, so scope, deliverables and timeline are agreed in writing before work starts, and a quote follows from that scope. Ask for a breakdown of the readiness assessment, control evidence, documentation and any training support, and ask which items recur, such as updates after model or release changes. Keep in mind that Article 99 sets maximum fines of EUR 15 million or 3% of worldwide annual turnover for most operator obligations, whichever is higher.

How long does EU AI Act readiness work take for a staff assistant?

Duration depends on scope and on how much evidence already exists, so ask for a timeline in the written scope rather than relying on a generic estimate. The fixed dates matter more for planning than any single engagement length. Chapters I and II, including Article 4, have applied since 2 February 2025, so the AI literacy duty is already in force. The general-purpose AI rules applied from 2 August 2025, and the rest of the Act applies from 2 August 2026. Under Regulation (EU) 2026/1744, the high-risk rules in Annex III systems apply from 2 December 2027. Plan your work so AI literacy measures are in place now, and treat any high-risk classification as a later workstream with its own milestones.

What must our team provide for an EU AI Act engagement?

Your team needs to supply the facts about how the assistant is used, because documentation cannot be written accurately without them. Expect to provide:

A list of the models, interfaces and connected data sources, with the version and license of each component
Named business owners for each use area and the intended purpose of the assistant in each one
Current acceptable use rules, access groups, log retention settings and any existing AI policy
Access to the people who run the platform, so controls and logging can be checked against the deployment
Decisions on who is responsible for the provider versus deployer role, if the organization builds or substantially modifies the system

The assessment is much faster when these items are ready before work starts.

Can we handle EU AI Act readiness ourselves?

Yes, in many organizations the AI literacy program and the acceptable use policy can be run by internal teams, and the Commission does not prescribe a training format. The harder questions are role questions. Under Article 3, a provider develops an AI system or has one developed and puts it into service under its own name, while a deployer uses it under its authority. If your team builds the assistant in-house from open-source components and offers it under its own name, the provider duties may apply, and those include quality management and post-market monitoring for high-risk systems. Get a legal opinion on that classification before you decide who owns each control. DeployOpen can prepare systems, controls and evidence for compliance assessments, but it does not issue audit reports, attestations or certifications.

When does the high-risk regime apply to a chat assistant?

A general staff chat assistant is usually not a high-risk system on its own, but the answer depends on the use case and must be confirmed against the Act's Annex III categories. Under Regulation (EU) 2026/1744, the high-risk rules in Chapter III Sections 1 to 3 apply from 2 December 2027 for Annex III systems and from 2 August 2028 for Annex I product systems. If the assistant is used for a purpose listed in Annex III, such as decisions affecting employment or access to essential services, providers carry the design and market-entry duties. Article 12 requires high-risk systems to technically allow automatic event logging over their lifetime, and Article 19 requires providers to keep those logs for at least six months unless other law provides otherwise. Confirm the classification early, because it decides which controls apply.

Which EU AI Act penalties could apply to a staff chat assistant?

Article 99 sets the maximum fines, and the level depends on the breach. For prohibited practices the maximum is EUR 35 million or 7% of worldwide annual turnover. For most operator obligations it is EUR 15 million or 3%. For misleading information given to authorities it is EUR 7.5 million or 1%. In each case the higher figure applies. National market surveillance authorities enforce the Act for AI systems. The practical lesson is that the bigger exposure usually comes from prohibited uses or misleading statements to a regulator, so keep the acceptable use policy specific, record what the assistant is and is not used for, and make sure any statement about compliance is accurate before it is made.

Does the EU AI Act apply to a self-hosted assistant we run in the EU?

Usually yes, if the assistant's output is used in the EU. Article 2(1) applies the Act to deployers established or located in the EU, to providers wherever they are established, and to non-EU providers and deployers whose AI output is used in the EU. Self-hosting does not remove the obligation, because Article 4 applies to providers and deployers of all AI systems, not only high-risk ones. Hosting the model on your own infrastructure mainly changes who controls the data and the logs, which helps with other obligations, but it does not by itself settle the AI Act question. Ask your counsel to confirm your role and the scope for each use area in writing.

What DeployOpen prepares for your compliance file

DeployOpen plans, configures, hardens and documents the self-hosted chat platform inside your own infrastructure, and prepares the controls and evidence that an assessor or your own compliance team will ask for. For this topic, that means the deployment record for the assistant: which Open WebUI or LibreChat version runs, how it connects to SSO and SCIM provisioning, which groups hold which permissions, where Ollama or vLLM binds and how the inference endpoint is protected, and where chat history and uploaded files are stored. It also means the AI literacy material that accompanies rollout, such as the acceptable use guidance and the training record for staff. DeployOpen does not issue audit reports, attestations or certifications. An independent auditor or certification body makes those determinations, and no certificate is required for Article 4 compliance, according to the Commission's AI literacy questions and answers.

Work is sold as a fixed-scope engagement. Before work starts, both sides agree in writing on the systems in scope, the deliverables (for example, a configuration baseline, a logging and retention design, and the evidence index), the responsibilities of each party, and the timeline. Ask for the written scope to state which Open WebUI License obligations you are taking on, including whether your deployment is above 50 users in any rolling 30-day period, since that affects branding. Ask the partner to name who owns each control afterward, so the evidence does not depend on a single engineer. Confirm the price basis and the handover format with the partner, since DeployOpen does not publish a price list and a quote follows from the written scope.

Some evidence needs to be kept up to date after go-live. Open WebUI, Ollama and the surrounding components release often, so a retainer can cover upgrades, re-running configuration checks, reviewing logs and periodic security testing of the assistant and its endpoints. Decide early whether you want that continuity, and ask the partner how upgrade notes and changes to the evidence index are tracked. Your internal owner remains responsible for the acceptable use policy, staff training completion and the decision on which data classes may be entered into the assistant.

How DeployOpen works on this

DeployOpen implements self-hosted open-source chat assistants inside the client's own infrastructure and prepares the systems, controls and evidence that an EU AI Act assessment will look for. We do not issue audit reports or certifications, so the work is organized around documentation an independent reviewer can examine.

How we scope

We agree a written scope covering the assistant, its models, users, identity integration and the logging design before work starts.
The scope states which AI literacy measures are in and out of the engagement, and which parts remain with your HR and training teams.
We confirm with you whether the in-house build could make your organization a provider, and document that position in the scope.

How we build

We deploy the chat interface, model runtime and supporting services in your cloud accounts, data centers or air-gapped network.
We configure SSO, group-based access control and retention settings, then test them against the agreed requirements.
We enable the event logging your architecture needs and check that logs are retained for the period your legal team sets.

How we hand over

You receive architecture records, configuration documentation, access and logging descriptions and the test results from delivery.
We provide a draft acceptable use policy and an AI literacy material outline for your legal and compliance reviewers to adapt.
Handover includes a walkthrough with the owners who will run the assistant and the evidence file they will maintain.

How we support

Engagements are fixed-scope, and ongoing support, upgrades and continuous testing are available as a retainer.
Retainers can cover tracking of Open WebUI and model runtime releases, so the documented configuration stays current.
We can review changes to the assistant against the written scope and update the evidence file when the regulatory position changes.