Apache Kafka for healthcare

An ADT admission or discharge message or an order message usually starts in the EHR, passes through an integration engine that handles HL7 v2 framing, acknowledgments and routing, and then reaches a FHIR-facing producer that publishes events to Kafka. The producer is the place to translate a pipe-delimited HL7 v2 segment into a FHIR resource such as Encounter, ServiceRequest or Patient. HL7 v2 messages are event notifications carrying the fields that changed at one moment, while FHIR resources are the current state of an entity. Topic design should follow the clinical event boundary (an admission, a transfer, a lab order placed) rather than mirror the HL7 message type one to one, because consumers usually need the resource state, not the trigger. Consumers then write to a HAPI FHIR JPA server through its REST API, or feed other clinical systems, and each consumer group should keep its own offsets so a failed FHIR write does not block the others. The producer should also keep the original HL7 message identifier so every FHIR resource can be traced back to its source message.

Several details must be confirmed with the EHR vendor and the integration engine before topics are named. Ask which HL7 v2 message types and trigger events the EHR emits, whether it sends them over MLLP or through an interface engine only, and whether resend or replay is supported if a consumer falls behind. Confirm the patient identifier domain, because a Kafka key built on a local MRN can collide across facilities, and confirm whether the EHR already exposes FHIR endpoints that could make a separate producer unnecessary. Also confirm how the integration engine handles acknowledgments, so the team knows when a message is considered delivered and whether a Kafka outage can cause duplicate sends. Finally, record the version of each FHIR specification the consumers expect, since the HAPI FHIR JPA server supports DSTU2, DSTU3, R4, R4B and R5 and a resource written for one version may not validate against another.

Topic retention is a privacy decision as well as a storage decision. In Kafka, a message stays in a topic for the retention window set by the topic configuration, whether or not any consumer has read it. During that window the message can be replayed by a new consumer group, and a MirrorMaker 2 flow can copy the topic, with its configuration, to another cluster. Each of those copies is a place where electronic protected health information (ePHI) lives. Retention limits should therefore be set per topic by the team that owns the data, with the clinical purpose written down first and the storage cost considered second. The settings to review are retention.ms and retention.bytes at the topic level, and cleanup.policy if a compacted topic is used for current state. Confirm the defaults on your cluster, because a topic created without explicit settings inherits the broker defaults.

Match the retention window to the clinical need. An order or result event that a downstream system must reprocess after an outage may need days, while a bulk feed used only to refresh a read model may need hours. Where feasible, separate identifiers from clinical payloads: publish a stable pseudonymous key or an opaque reference on the replayable topic and keep the directly identifying fields in a system that the FHIR server or the record system controls and audits. This reduces what a replay exposes and makes deletion requests easier to honor. Do not assume that compaction removes personal data, since older values remain in segments until they are cleaned, and tombstones only apply to keys you have chosen to delete. Plan the access model for replay separately from production access, using ACLs on the consumer group and topic so that only named services and roles can read history, and log replay sessions.

Under the HIPAA Security Rule, retention and replay controls map to technical safeguards such as access control and audit controls, and several of the related implementation specifications are addressable rather than required. For each addressable specification, the covered entity or business associate must assess whether it is reasonable and appropriate, document the decision if it is not adopted, and implement an equivalent alternative where one is reasonable. Record the retention rationale, the replay approval process and the reasoning for any alternative in the risk analysis and in your documentation for the Kafka platform. Confirm with your privacy and security officers which data classes qualify as ePHI in your environment, and confirm the current requirements with the HHS HIPAA Security Rule guidance and your counsel.

Start with the business associate question, because it decides whether a partner can touch ePHI at all. Under 45 CFR 164.314(a)(2)(i), a business associate contract must require the partner to comply with the Security Rule, flow the same terms down to any subcontractor that handles ePHI, and report security incidents, including breaches of unsecured PHI, to the covered entity. Ask for the partner's standard business associate agreement before scope is discussed, confirm who in the partner signs it, and confirm which subcontractors (hosting, backup, support tooling) would see message payloads. If a subcontractor cannot accept the same terms, the scope needs to change before work starts. Also confirm that the partner's staff with access to your clusters are named in the contract.

Next, ask how ePHI is kept out of places it should not reach. Kafka payloads are often HL7 or FHIR resources, so the partner should explain how broker logs, Connect worker logs, debug output, dead letter topics and monitoring labels are configured to exclude message bodies and identifiers. Ask whether test data is synthetic or de-identified, who approves any copy of production messages into a lower environment, and how that copy is deleted. Ask how the security incident process runs in practice: who triages an alert, how quickly the partner notifies you, and which evidence it preserves. Confirm that the contract's reporting terms match your own incident response obligations to the covered entity.

Then ask the operational questions. Confirm who holds the cluster administrator credentials, where they are stored, and whether the partner's engineers use named accounts with authorization rules and logged sessions rather than shared secrets. Ask how upgrades will be tested against the supported 4.x releases listed on the Kafka downloads page, including rolling upgrades from the last three releases, and whether a test cluster with representative topics, consumers and ACLs will be used before production. Finally, ask for the evidence package: configuration baselines, ACL exports, TLS and SASL settings, backup and restore test results, and the access review record. DeployOpen prepares these artifacts for your assessors, but it does not issue audit reports or certifications, so confirm which independent body will review them.