Why spam and malware filtering is the first control to get right in self-hosted email
If you are moving mail in-house or running it yourself for the first time, the filtering job that your hosted provider handled quietly now belongs to your team. The filtering layer is not a single product. It is a set of components: message scoring, attachment antivirus, greylisting, DNS blocklist checks and a quarantine that users and administrators can actually work with. Each one must be selected, configured, tuned against your real mail flow and evidenced for auditors. Get this wrong and you either lose legitimate mail to the junk folder or let phishing and malware reach mailboxes, and both failures land on your users and your sender reputation before anyone notices the server is at fault.
On open-source platforms the answer usually starts with what the mail server already bundles. mailcow packages Rspamd for spam scoring, with ClamAV available as an optional scanner, so a small or mid-size deployment can run the core filtering chain from one installation. Stalwart is an all-in-one mail and collaboration server, and its filtering features sit inside the same system, which suits teams that want fewer moving parts. Neither choice settles the question on its own. Message volume, the number of domains, how much of your traffic is inbound from unknown senders and your sector's rules on retention and malware handling all change whether a bundled filter is enough or whether a commercial gateway belongs in front of it. Confirm each platform's current filtering features and supported integrations with its own documentation before you commit.
What the filtering layer must satisfy by sector and constraint
This table compares the regulatory requirements most likely to shape a self-hosted mail filtering layer and what each one means for the build. The regulations do not prescribe specific products, so read each row as a question to settle with your compliance lead before choosing Rspamd, ClamAV or a commercial gateway. The main takeaway is that filtering and quarantine are part of the evidence chain, not only a defense.
| Requirement | Source | What it means for the build |
|---|---|---|
| Protect ePHI in transit and block unauthorized access (healthcare) | 45 CFR 164.312(e)(1) of the HIPAA Security Rule | Filter inbound threats before they reach clinician mailboxes, and confirm that TLS enforcement and malicious-attachment handling are documented as technical safeguards. |
| Record and examine activity on systems holding ePHI (healthcare) | 45 CFR 164.312(b), audit controls | Keep filter, quarantine and delivery logs with timestamps and retention, and confirm who reviews them and how often. |
| Preserve business communications for three years (financial firms) | 17 CFR 240.17a-4(b)(4), SEC Rule 17a-4 | Filtering logs and quarantine must not destroy messages that must be retained, so route retained copies to an archive before any deletion or rejection decision. |
| Sender authentication and attachment handling (all sectors) | General email security practice and your own policy | Configure SPF, DKIM and DMARC checks in the filter, define which file types are blocked or sandboxed, and confirm the policy with your security team. |
| Quarantine release and false positive handling | Internal policy and records obligations | Define who may release quarantined mail, how releases are logged, and how retained messages are kept intact when a user or reviewer acts on them. |
Which filtering components fit a self-hosted mail stack
A self-hosted mail server does not filter by itself. Scoring, malware scanning, pre-acceptance checks and an optional gateway each do a different job, and the right mix depends on which mail platform you run. The cards below cover the components you will meet most often and the questions to settle before you commit to one.
Rspamd as the scoring engine
Rspamd assigns each inbound and outbound message a spam score from many weighted checks, then tells the MTA to accept, reject, quarantine or tag it. mailcow ships Rspamd alongside Postfix and Dovecot, so it is the default choice if you run that stack. Check which score thresholds you can set per domain or per user, how learning from user reports is stored, and who owns the rule updates. Test it against a sample of your real mail before you enable quarantine, because false positives on legitimate invoices and contracts are the main risk.
ClamAV for attachment scanning
ClamAV scans message content and attachments for known malware signatures. mailcow lists it as an optional component, so you decide whether to run it, and it adds memory and signature-update load to the host. Signature databases refresh on a schedule, which means a brand-new payload can pass until the update lands. Plan for encrypted archives and password-protected files that the scanner cannot open, and decide whether those are delivered, held or rejected. Confirm the update path if the server is air-gapped.
Greylisting and DNS blocklists
Greylisting temporarily rejects mail from unknown senders, which legitimate mail servers retry and most spam engines do not. DNS blocklists check the sending IP against published reputation lists before the message body is accepted. Both run before content scoring, so they reduce load but can delay first contact from new partners. Decide which blocklists you trust, whether you allow any allowlist exceptions, and how you will notify senders who are delayed. Review the lists regularly, since a listed partner IP can block your own business mail.
Stalwart's built-in filtering
Stalwart is an all-in-one mail and collaboration server that supports IMAP, JMAP, SMTP, CalDAV, CardDAV and WebDAV in one product. Confirm its built-in spam and malware controls, and whether it can call an external scanner such as Rspamd or ClamAV, before you assume a mailcow-style stack will drop in. Ask the vendor how rules are updated, how quarantined messages are reviewed, and which features need the Enterprise License. Treat the answers as the basis for your design, not the documentation of an older release.
OX App Suite and a separate filter layer
OX App Suite v8 runs on Kubernetes, is delivered as Helm charts, and requires Istio, with ingress-nginx as a basic option where Istio cannot be used. Its mail layer does not come with a mailcow-style filtering stack, so you need to design the inbound filter in front of it, typically on the edge or as a gateway service. Confirm with Open-Xchange which filter integrations are supported, how message flow is routed through the filter, and where quarantine data lives. Ask whether any filtering or antivirus modules fall under separate license terms.
Where the filter sits: single server, high availability and gateway in front
A single-server stack such as mailcow keeps the whole filtering path on one host. Postfix accepts inbound SMTP, Rspamd scores each message, and ClamAV can scan attachments when it is enabled. Because the components share one machine, filtering decisions, quarantine and logs are all local, which keeps the design easy to reason about and to back up.
- SMTP relay: Inbound mail reaches Postfix directly from the internet on port 25, and Postfix hands messages to Rspamd for scoring before delivery to Dovecot.
- Quarantine storage: Held messages live on the same disk as the mailboxes, so backup scope and retention must cover them explicitly.
- Logging export: Rspamd and Postfix logs stay on the host unless you ship them to a central collector, which is usually a separate build task.
- What scales: Throughput is bounded by one host's CPU, memory and disk. Scaling up is simpler than scaling out, but there is no built-in failover for the filter.
- Maintenance window: Updates to the filter or to ClamAV signatures restart services on the same host that carries live mail.
Confirm with the vendor and your own testing how the bundled ClamAV is enabled, how Rspamd rules are customized without being overwritten by updates, and what sizing they assume for your message volume.
OX App Suite v8 is designed for a Kubernetes cluster and is delivered as Helm charts that reference container images. Helm 3.17.0 or newer is required, and the platform depends on MariaDB with InnoDB as its primary data store, Redis 7.4.x for caching and sessions, and a distributed file system for persistent data such as PIM attachments. The filtering layer must fit into this dependency chain rather than sit beside it.
- SMTP relay: Inbound mail must be routed to the filtering service through the cluster ingress or a dedicated mail gateway, and the path has to be confirmed for each replica.
- Quarantine storage: Quarantined messages need to land on the persistent storage tier, which is NFS for smaller deployments or object storage such as S3, Ceph RadosGW or Scality RING for larger ones.
- Logging export: Container logs should go to a cluster-wide collector so that filter decisions survive pod restarts.
- What scales: Filter replicas can be added horizontally, while MariaDB, Redis and the file tier determine the real ceiling. Galera Cluster is supported for MariaDB.
- Version constraints: Kubernetes support follows N-1 and N-2 releases, and Istio is required where it is used, with ingress-nginx as the basic option.
Confirm with Open-Xchange which filtering integrations are supported in the Helm deployment, how quarantine is shared between replicas, and which Istio and Kubernetes versions the filter path has been tested against.
A gateway placed in front of the mail server accepts inbound mail first, filters it, and forwards only accepted messages to the open-source platform. The gateway may be a commercial product or an open-source relay, and it changes where spam, malware and quarantine decisions are made. Your internal Rspamd and ClamAV layer can then act as a second check instead of the only one.
- SMTP relay: Public MX records point to the gateway, which relays clean mail to the internal server over a restricted, authenticated path.
- Quarantine storage: Quarantine is usually held in the gateway's own service, so you need to know where the messages live, who can release them and how long they are kept.
- Logging export: Gateway logs often export through syslog or an API, and you should confirm the format before the mail server's own logs are used for audits.
- What scales: The gateway scales on its own terms, which can reduce load on the internal server, but it adds a service that must be patched and monitored.
- Sender identity: Recipients see the sending address from the origin, so SPF, DKIM and DMARC alignment must still be correct once a relay is added.
Confirm with the gateway vendor whether it preserves original sender details, which logs and quarantine records it can export to your archive, and what happens to mail if the gateway is unavailable.
How to tune scoring without losing legitimate mail
Tuning Rspamd scoring is a measured cycle, not a one-time configuration. Start the new or changed filter in a logging-only or tagging mode, so that messages are scored and annotated but not rejected or moved. Let that run long enough to see the normal mix of newsletters, invoices, supplier replies, automated notifications and bulk messages your organization actually receives. Then review the quarantine and the tagged messages with a small group of users from finance, operations and customer-facing teams, who can tell you quickly which legitimate business messages were flagged. Each false positive should be recorded with the sender, the recipient, the rule or symbol that fired and the business reason the message was expected. Confirm with your mail administrators which Rspamd modules and symbols your deployment exposes and how their scores are logged, because the review is only as good as the data behind it.
Once the review shows a pattern, fix it at the narrowest level that works. A per-sender or per-domain allowlist for a known supplier is usually better than lowering a global threshold, because a global change affects every message in the organization. Give every allowlist entry an owner, a business reason and an expiry date, and schedule the review before that date so that stale exceptions do not quietly accumulate. Keep ClamAV attachment decisions separate from scoring decisions: a message that scores as spam can be safely delivered without an attachment being cleared, and an allowlist for a sender should never bypass malware scanning. Move score thresholds only after several review cycles show that the change would have reduced false positives without letting through the messages you care about most, such as payment-change requests and credential phishing.
False positives are a business problem, not only an administrative one. A rejected or quarantined invoice, a legal notice or a candidate's reply can cost time, money or a missed deadline, and users often stop trusting quarantine altogether, which causes them to miss real threats. Filter changes therefore need change control: a written request describing the change, the expected effect, the test results from logging mode, the approver, the rollback step and the date it went live. Keep the record of every rule, threshold and allowlist change in a system your auditors can read, such as a version-controlled configuration repository with approvals. Confirm with your compliance and security leads which of these changes they must approve, and how long the records must be kept, since that depends on your own policies and the regulations that apply to your sector.
How a filtering rollout runs from pilot to enforcement
The sequence moves a self-hosted mail platform from observation to enforced filtering in controlled stages, so each change is measured against real mail flow before it can block or reject anything. Each step ends with a named output that the next step depends on.
Assess current mail flow and sender authentication
Before any filter is enabled, the team inventories every inbound and outbound path, including application relays, scanners, marketing platforms and partner gateways. For each sending source, they confirm SPF, DKIM and DMARC alignment and record which messages currently reach mailboxes and which are already blocked upstream. The output is a mail flow map and a sender inventory that show which legitimate senders must be allowed before enforcement starts. The step is complete when every known sender is either authenticated or on a documented exception list.
Deploy Rspamd in tagging-only mode
The mail administrator enables Rspamd scoring on a single pilot domain with actions limited to adding headers and subject tags, so no message is moved, quarantined or rejected. ClamAV and DNS blocklist checks run in the same mode so their verdicts are visible without consequence. The output is a baseline set of scoring results in the logs and a documented configuration record. The step is complete when the pilot domain has processed a representative sample of traffic and the team can explain each score contributor.
Tune scoring with user and sender feedback
Mail administrators review tagged messages with a small group of pilot users and the owners of the sending systems identified in the first step. Each false positive and missed spam is traced to the rule, list or score that caused it, and the change is recorded as a tuned policy record with the reason and the author. Greylisting thresholds, allowlists and blocklist sources are adjusted in this stage, not earlier. The step is complete when the pilot's misclassification findings have been resolved or accepted and the policy record is approved by the mail owner.
Enable quarantine and attachment scanning
The team switches high-scoring messages to a quarantine folder that administrators and users can review, and turns on ClamAV scanning for attachments with a defined response for infected or encrypted files. The quarantine procedure states who may release messages, how releases are logged and how users report a message they expected to receive. The output is a written quarantine procedure and a set of test messages, including a harmless test file, that show each action fires correctly. The step is complete when the test results match the procedure in the pilot.
Enforce rejection policies on the pilot domain
Only after the earlier stages are stable, the team moves the pilot domain to enforcement by rejecting or quarantining messages above agreed score thresholds and blocking attachment types the policy prohibits. Rejection at the SMTP layer is reviewed carefully because it can bounce legitimate mail back to senders, so the team confirms the rollback path before switching. The output is a signed-off enforcement configuration and a rollback runbook. The step is complete when the pilot has run in enforcement without unresolved sender complaints and the rollback has been tested.
Extend enforcement to remaining domains
The approved configuration is applied to the remaining mail domains in groups, with each group checked against the same sender inventory and quarantine procedure before it goes live. Domain-specific allowlists and exceptions are documented so they can be audited later. The output is an updated domain inventory with the filtering mode recorded for each domain. The step is complete when every domain is in enforcement or has a recorded reason for remaining in a lower mode.
Operate filtering with reporting and review
The operations team runs recurring reviews of filtering reports, covering quarantine volume, false positive releases, blocklist hits, ClamAV signature update status and authentication failures from DMARC aggregate reports. Rule and score changes go through the same tuned policy record used during rollout, so every change has an owner and a reason. The output is a standing report and a change log that auditors or security reviewers can inspect. The step is complete when the first review cycle has produced the agreed report and any open findings have an owner.
Evidence auditors ask for and which component produces it
This table maps the evidence an assessor typically asks to see for a mail filtering layer to the component that generates it. Read each row as a question to answer before the assessment: where the record lives, who owns it, and how long it is kept. DeployOpen prepares these artifacts, but independent auditors and certification bodies issue any assessment or certification.
| Evidence item | What it shows | Component that produces it |
|---|---|---|
| Filter configuration under change control | The scoring thresholds, rules, milterworkers and routing settings in force on a given date, and who approved each change | Version-controlled Rspamd and Postfix configuration, with change tickets linked to each commit or deployment |
| Scanning and scoring logs | Per-message verdicts, spam scores, rule hits and ClamAV results, showing the filter ran on all inbound and outbound mail | Rspamd log and history output, ClamAV daemon logs, and Postfix logs forwarded to a central log store |
| Quarantine and release records | Which messages were held, who released them, when, and on whose request | Rspamd quarantine or the mail platform's quarantine interface, with release actions captured in the audit log |
| Allowlist review records | Which senders or domains bypass scoring, why they were added, and evidence of periodic review | Rspamd local allowlist maps under change control, plus a dated review record signed off by the mail owner |
| Update and signature currency for ClamAV and rules | That antivirus signatures and spam rule sets were refreshed on a defined schedule and the dates they were applied | freshclam update logs, Rspamd rule update records, and package manager history from the host or container image |
| Retained copies showing the filter did not destroy records | That messages rejected, quarantined or tagged were either retained or handled under a documented policy, so the business record is not lost | Journaling or archive copies taken before filter actions, with quarantine retention settings and a reconciliation report |
What do buyers ask before they hire help with mail filtering?
What does mail filtering help cost, and what drives the price?
The price follows from the written scope, and no figure is published for it. The main cost drivers are the number of domains and mailboxes, daily message volume, how many tuning cycles the scope includes, whether attachment scanning and sandboxing are in scope, and what evidence a regulated sector needs to keep. Adding a commercial gateway in front of the mail server adds a license or subscription that recurs, while the open-source components themselves carry no license fee. Ask for a quote that separates one-time work (design, deployment, initial tuning, documentation) from recurring work (rule updates, signature feeds, false positive review, monitoring), and ask which of those lines the quote includes.
How long does a filtering rollout take, and what sets the pace?
The pace is set less by installation than by tuning. A filtering layer built on Rspamd, ClamAV and greylisting can be deployed quickly, but scores, thresholds and allow lists only settle after real mail has passed through them and users have reported misclassified messages. Expect the schedule to include a monitored period where mail is tagged rather than rejected, followed by a staged move to enforcement. Timelines are agreed in the written scope, so ask the partner to show the phases, the review points and what triggers the move from one phase to the next.
What does our team need to provide before filtering work starts?
Three inputs unlock most of the work. The first is mail flow data: sample headers, sender lists, current spam and quarantine statistics, and known legitimate bulk senders such as payroll, ticketing and newsletters. The second is DNS and mail host access, so SPF, DKIM, DMARC, reverse DNS and blocklist checks can be reviewed and changed. The third is a named approver who can sign off policy changes, such as rejecting a file type, changing a quarantine threshold or whitelisting a partner domain. Without a named approver, false positive disputes stall and the tuning loop slows down.
Can an internal team run spam and malware filtering alone?
A small internal team can run a self-hosted filter for a single domain or a modest mailbox count, provided someone owns it. The initial build is the smaller task. The recurring work is reviewing quarantine, handling false positives, watching blocklist status, keeping ClamAV signatures and Rspamd configuration current, and reading reports after a new phishing campaign. Ask whether your team has the hours for that work each month, and who covers it during leave or an incident. Partners often split the work into a build phase and a retainer for ongoing support and upgrades.
Do we need a commercial email security gateway in front of open-source mail?
Not always. Rspamd, ClamAV, greylisting and DNS blocklist checks can run on the mail server itself, and mailcow bundles Rspamd and optional ClamAV in its stack. A commercial gateway tends to make sense when you need features the open-source layer does not cover well for your environment, such as advanced sandboxing of attachments, managed threat intelligence, impersonation protection, or a vendor contract that an auditor or insurer expects. Confirm each capability against the product you are considering rather than assuming it, and compare the gateway subscription with the staff time it would save.
What evidence should a filtering partner deliver for a compliance review?
Ask for documented filter policies, the configuration under version control, records of rule and signature updates, quarantine and false positive review logs, and the monitoring that shows the filters are running. These support controls such as anti-malware and anti-phishing requirements in frameworks like PCI DSS and CIS Control 9, but they do not satisfy an audit by themselves. DeployOpen prepares systems, controls and evidence for compliance assessments, while independent auditors and certification bodies issue the audit results. Confirm with your assessor which artifacts they expect before the work begins.
Who is responsible when legitimate mail is blocked by the filter?
Responsibility should be written into the scope before go-live. Typically the partner handles diagnosis of blocklist listings, Rspamd symptom analysis and configuration changes, while your named approver decides whether a sender or file type is trusted. Ask how the partner measures the time from a complaint to a fix, how a blocked executive or customer message is released from quarantine, and whether the process covers weekends and incidents. Clear answers to these points matter more than any single feature of the filter.
Questions to put to an implementation partner on filtering
Start by asking which scoring and antivirus components the partner will deploy and why. For a self-hosted mail stack the expected answer names Rspamd for scoring, with ClamAV as an optional attachment scanner, and explains how each is wired into the Postfix path. Ask whether greylisting and DNS blocklist checks are part of the design and at which stage they run. Then ask how the partner will verify that ClamAV signature updates and Rspamd rule updates are applied on a schedule you can audit. A partner who cannot describe the message flow from inbound SMTP to the mailbox, step by step, has not yet designed the filter.
Next, ask how false positives are tuned and documented. The useful answer names a written change process: who proposes a score or rule change, who approves it, how it is tested against a sample of legitimate mail, and where the change record lives. Ask how the partner handles DNS blocklist listings for your outbound addresses, including who is notified, how delisting is requested, and what the partner does to the outbound reputation impact while a listing is active. Also ask who responds when the filter blocks a critical message at 2 a.m., and what the response time commitment is in the written scope. Ask for the answer in writing.
For regulated records, ask how quarantined messages are retained, who can release them, and whether quarantine is covered by the same retention and legal hold rules as the mail archive. Ask how the filter is tested before enforcement, such as running in a tag-only or report-only mode first and comparing results with a known sample. Finally, ask how signature and rule updates reach offline or air-gapped networks, including how the partner packages updates, validates them, and records each import. Confirm the update source, the validation step, and the rollback plan before you accept the scope.
How DeployOpen works on this
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source mail platforms inside your own infrastructure. For spam and malware filtering, we start from your mail flow and risk profile, build the filtering layer in your environment, and leave you with documented settings and a clear operating model.
How we scope
- Written, fixed scope listing the mail platform, domains, inbound and outbound flows and filtering components in use
- Review of current false-positive and quarantine patterns, plus any attachment types your policy must block
- Agreed decisions on whether a commercial gateway sits in front of the open-source mail server, confirmed before build starts
How we build
- Configure Rspamd scoring, ClamAV attachment scanning, greylisting and DNS blocklist checks in your own environment
- Test known-good and known-bad message samples, then tune thresholds to reduce false positives before go-live
- Document every rule, score, allowlist and exception so changes can be reviewed and traced later
How we hand over
- Handover of configuration, runbooks and quarantine procedures to your administrators, with acceptance testing against the written scope
- Clear record of which filtering controls support your assessment evidence, prepared for your auditors rather than certified by us
- Knowledge transfer on reviewing quarantine, handling blocklisting events and updating signatures and rules
How we support
- Ongoing support, upgrades and continuous testing available as a retainer after handover
- Periodic review of filtering performance, scanner signature updates and mail platform releases in your environment
- Security testing of the mail platform, including phishing and malware delivery paths, on request under separate scope
