When Zabbix high availability is worth designing and why it is harder than it looks
Monitoring that goes dark during a node failure hides the outages it exists to catch. If the server that evaluates triggers and sends alerts stops, a database problem, a saturated link or a failed application can run unnoticed until a user or customer reports it. That is the situation that makes Zabbix high availability worth designing, especially where monitoring is the only early warning for banking, healthcare, public sector or critical infrastructure services. The question is not whether to run a cluster, but whether a short gap in monitoring is acceptable for the systems you watch, and who is accountable when it happens. Zabbix server has a native high availability cluster. Several server nodes share one database, one node is active and the others stand by. When the active node exceeds a configurable failover delay, a standby takes over. The failover delay can be set from 10 seconds to 15 minutes, and that single setting determines how long monitoring can be silent before a standby becomes active. A short delay recovers faster but risks failing over on a slow node that is still working. A long delay avoids flapping but leaves a larger blind window. Set this value from your own alerting requirements, then test it against the outage durations you actually need to detect.
The cluster does not remove the database as a single point of failure. All server nodes depend on the same database, so the Zabbix server cluster is only as available as that database. Native HA covers the Zabbix server processes, not database failure, storage failure or a database failover that takes longer than your delay. Confirm which database engine and version you will run (MySQL or Percona, MariaDB, PostgreSQL, with TimescaleDB as an option), and confirm the replication, failover and backup design your team can operate for it. Also confirm who owns that layer, because database HA is often run by a separate team and its failover behavior must be tested together with Zabbix.
Proxies and agents need their own redundancy and are easy to overlook. A highly available server does not help if a remote site's proxy is down and its hosts stop reporting. From Zabbix 7.0, proxy groups can redistribute monitored hosts between proxies when one goes offline, which addresses proxy availability for suitable designs. Check that each site has enough proxy capacity to absorb another proxy's hosts, that network paths from proxies to hosts and to the server cluster are resilient, and that agents can reach more than one proxy where your design requires it. Agent-side failures are usually not solved by server HA at all, so plan them as a separate item.
Before committing to a design, ask for a written test plan covering a server node failure, a database failover, a proxy failure and a network partition between site and server. Measure how long monitoring is blind in each case, how alerts and trigger states behave on recovery, and whether duplicate or missing notifications occur. Confirm the Zabbix version you will deploy, since the supported line affects which HA and proxy group features you can rely on. The Zabbix life cycle and release policy and the high availability documentation should be checked against your chosen release before the design is signed off.
Which Zabbix components need redundancy and how each one fails
Zabbix high availability spans several layers, and each one fails differently. Native Zabbix handles some failures automatically, such as server node takeover and proxy group redistribution, while the database, the web tier and the network paths between components need infrastructure your team plans and tests. Use these cards to decide where to add redundancy and what to verify for each component.
Zabbix server nodes
The Zabbix server runs as a cluster of nodes that share one database. One node is active and processes data while the others stand by. If the active node stops responding for longer than the failover delay, which you can set from 10 seconds to 15 minutes, a standby node takes over. Native Zabbix handles this election and takeover. Check the delay against how much data gap you can accept, and confirm that every node points to the same database and runs the same configuration.
Shared database and replication
Every server node depends on one database, so it is the most important single point of failure in the design. Zabbix does not replicate the database for you. Your team must add a replication or clustering layer, such as primary and standby instances, and operate it. Confirm which engine and version your Zabbix release supports, plan how the server reconnects after a database switch, and test a failover before go-live.
Frontend and web tier
The web frontend and API are separate from the server cluster and can run on more than one host behind a load balancer. If one frontend host fails, users should reach another host, but the team must build the load balancer and its health checks to detect a dead frontend quickly. Check that every frontend host uses the same database connection and server address, and that session handling and TLS certificates behave the same on each host.
Proxies and proxy groups
Proxies collect data from remote sites and segmented networks and forward it to the server. When a single proxy goes offline, collection stops for its hosts unless a proxy group is in place. From Zabbix 7.0, proxy groups redistribute monitored hosts to the remaining proxies. Confirm that surviving proxies have spare capacity for the extra hosts and that their network paths stay open. Detailed proxy design, including group sizing and placement, is covered in the proxies topic.
Zabbix agents
Agents run on monitored hosts and have no clustering of their own. If an agent or its host fails, that host stops reporting, and the failure is visible only if you alert on missing data. Zabbix does not move an agent to another server or proxy, so the design question is which server or proxy each agent is configured to contact. Check active and passive settings, and create triggers that flag hosts whose expected data stops arriving.
Choosing the HA server layout, database replication and proxy strategy
This table compares the main layout choices for a native Zabbix HA setup: server node count, database topology and proxy design. Read each row for the situation where the option fits, the upkeep it adds after go-live, and the questions to put to your database vendor before you commit. The main takeaway is that the database layer usually sets the real recovery ceiling, so decide it first.
| Option | When it fits | Operational cost | Confirm with the vendor or Zabbix |
|---|---|---|---|
| Two server nodes | Smaller estates where one standby can take over and maintenance windows are planned | Fewer nodes to patch, but no spare standby while one node is down for maintenance | Confirm the node upgrade order for your Zabbix version in the HA documentation |
| Three or more server nodes | Estates where monitoring must continue through planned patching plus an unplanned node failure | More nodes to patch, test and keep configured identically | Confirm the supported node count and any version-specific HA behavior with Zabbix |
| Single database instance | Lab or non-critical estates where a restore from backup is an acceptable recovery path | Lowest licensing and skills load, but the database becomes the single point of failure | Confirm restore time against your backup schedule and whether the vendor offers a supported failover route |
| Replicated or clustered database | Estates where a database outage would stop all data collection, alerting and frontend access | Replication monitoring, regular failover drills and staff who can run the chosen engine | Confirm which replication or clustering features need paid editions or add-ons, and the supported failover method |
| Database engine choice | MySQL or Percona, MariaDB, or PostgreSQL with TimescaleDB for larger history volumes | Each engine has its own replication, backup, patch and tuning routine | Confirm the engine versions your Zabbix release supports on the requirements page before you choose |
| Single proxy per site | Remote sites with a small host count where a gap in monitoring during a proxy outage is accepted | One proxy to patch per site, and that site's monitoring stops when the proxy fails | Confirm how host data is buffered during a proxy outage for your Zabbix version |
| Proxy group per site | Sites where hosts must move to another proxy automatically when one goes offline (Zabbix 7.0 and later) | Capacity planning for redistributed hosts, testing of failover, and keeping proxies in sync | Confirm proxy group limits and version support with Zabbix before you design around them |
How to build a Zabbix high availability cluster step by step
This sequence takes a team from a confirmed Zabbix version and database to a server cluster, frontends and proxy groups that keep monitoring running when a node fails. Each step ends with a check you can run, so you know the cluster behaves as designed before you rely on it.
Confirm the Zabbix version and database
Choose the Zabbix line first, then check every component against the requirements page for that exact version. Zabbix 7.0 LTS has full support until 30 June 2027, and Zabbix 8.0 LTS is planned for Q4 2026, so weigh the target line against your own upgrade window. For Zabbix 7.4, the supported databases are MySQL or Percona 8.0.30 to 9.7, MariaDB 10.5 to 12.3 and PostgreSQL 13 to 18, with the TimescaleDB extension 2.13 to 2.29 optional. Record the server, database and TimescaleDB versions in your design document. This step is done when every component sits inside its supported range and the record is signed off by the platform owner.
Build the shared database with replication
Create one database endpoint that every server node will use, and configure replication so a standby copy holds the same data. Your DBA or platform team should follow the replication method your database vendor documents for production, and size storage and connection limits for the number of nodes you plan to run. Confirm that each server node can reach the primary endpoint and that a planned promotion of the replica works without manual schema changes. Check: write a test record on the primary, confirm it appears on the replica, then promote the replica and confirm the nodes reconnect.
Install server nodes with identical configuration
Install the same Zabbix server version on each node from the same packages, then apply one configuration to all of them. Keep the database connection, timezone, time synchronization and the paths for alerting scripts and external checks identical, because a standby that differs from the active node can fail in ways you never saw during normal operation. Give each node a unique node name, as the cluster requires, and start every node against the shared database. Check: open the high availability nodes list in the frontend. Each node should appear, exactly one should show as active, and the others should show as standby.
Set the failover delay deliberately
The failover delay sets how long the active node may stay unresponsive before a standby takes over. Zabbix accepts values from 10 seconds to 15 minutes. A short delay reduces the monitoring gap but can trigger a switch during brief load spikes or database latency, while a long delay keeps monitoring paused longer after a genuine failure. Choose the value from your tolerance for data gaps and the latency you measure between nodes and the database, then document the reasoning. Check: save the setting, reload the high availability view and confirm it shows the value you chose.
Point frontends at the cluster
Configure every Zabbix web frontend to use the same shared database and the same connection settings. If you place several frontends behind a load balancer, confirm that the balancing method keeps user sessions working as your team expects, and that the frontend configuration files match. Users should see the same hosts, items and problems whichever frontend they reach, and the cluster view should update when the active node changes. Check: log in through each frontend address, open the same host and confirm that the latest values and open problems are identical on every frontend.
Add proxy groups where sites need them
For remote sites or segmented networks, deploy Zabbix proxies and assign them to a proxy group, a feature available from Zabbix 7.0 onward. When a proxy goes offline, the group redistributes monitored hosts between the remaining proxies, so each proxy must be able to reach the hosts it might inherit. Size each proxy with headroom, because a surviving proxy must carry the load of a failed one. Confirm with the proxy load balancing documentation which monitoring methods and hosts are eligible for redistribution in your setup. Check: the proxy group should list every proxy as online and the host assignments should match your design.
Test failover before go-live
Test each failure you plan to survive in a window agreed with your operations team, and record the results. Stop the active server node and confirm that a standby becomes active within the failover delay plus the time the takeover takes. During the test, check that items keep collecting, triggers still evaluate and notifications still reach the intended recipients. Repeat the test for one database node and for one proxy in a group, then restore the stopped node and confirm it rejoins as standby. This step is done when each test shows the expected active node in the cluster view, the data gap stays within your tolerance and the recovery steps are written down.
How the failover delay shapes what you lose during a node failure
The failover delay controls how long the active Zabbix server node can stop responding before a standby node takes over. The setting is configurable from 10 seconds to 15 minutes. A short delay shortens the gap in data collection and alerting after a real failure, but it also raises the chance of a false failover. A node that is only slow for a moment, for example while it waits on a busy database or handles a burst of incoming data, can look dead to the cluster and hand over work it could have finished. A long delay avoids those handovers but leaves monitoring blind for longer when the active node has failed.
Choose the value from the recovery expectations of the people who depend on monitoring, not from the default you happen to find first. Ask the operations and service owners how long a silent period they can accept before an outage goes unnoticed, and which hosts and services are covered by availability triggers that must fire quickly. Critical services usually justify a shorter delay, while capacity trend monitoring can tolerate a longer gap. Write the chosen value and the expected blind period into the runbook, so the on-call team knows what a failover means for their alerts and what data will be missing from the history.
The delay interacts with network latency and database health because every server node writes to the same shared database. If the database is slow, every node slows down together, and a standby may take over a node that is healthy but waiting on storage or locks. Latency between the nodes and the database, and between the nodes and the hosts they poll, also changes how quickly the active node appears to respond. Measure database response times under peak load, review housekeeping and history write volume, and confirm with the Zabbix documentation for your version how node status is recorded and checked before you fix the value.
Test the delay instead of guessing it. Run controlled failure scenarios on a staging copy that matches production sizing: stop the active node process, cut its network connection to the database, slow the database during a peak collection window, and cause a brief network interruption between nodes. For each run, record how long the failover took, whether a standby took over when it should not have, how much history was lost, and which triggers fired. Repeat the tests after each upgrade, because the failover behavior can change between Zabbix versions. Confirm the exact behavior for your version with Zabbix before go-live.
Proxies have their own failover behavior, so test them separately. From Zabbix 7.0, proxy groups redistribute monitored hosts when a proxy goes offline, which is a different mechanism from server node failover and has its own timing to verify. When you request a quote, ask the partner to include a written test plan that covers both server nodes and proxy groups, the measurements to capture, and the acceptance criteria for the failover delay you select.
What drives Zabbix high availability effort and cost
This table lists the factors that change how much design, build, testing and handover work a Zabbix high availability cluster needs, and how each one can be kept under control. Read each row as a question to answer in your own environment before asking for a quote. The main takeaway is that the database platform, the number of environments and the integration points usually matter more than the license, which is free.
| Driver | How it raises effort or cost | How to keep it down |
|---|---|---|
| Host and item counts | More hosts and items increase the load on the server nodes and the shared database, which affects node sizing, housekeeping design and load testing. | Confirm current counts and expected growth, then size nodes and the database against a measured baseline rather than estimates. |
| Number of sites and proxies | Each remote site may need its own proxy, and proxy groups add failover rules that must be tested for every site and for proxies going offline. | Group sites with similar network paths, standardize proxy builds and limit the number of proxy groups to what the design needs. |
| Database platform and in-house skills | Replication, failover and backup of the shared database often take more design and testing effort than the Zabbix side. Teams without that experience need more hands-on work and handover time. | Reuse a database platform your team already runs and operates, and confirm which replication and backup skills are in-house before scoping. |
| Number of environments (lab, staging, production) | Each environment needs its own cluster build, configuration sync and failover tests. Production-like staging adds effort because it must mirror node count and database topology. | Use a smaller lab for design and failover rehearsal, and keep staging only as large as needed to prove the production failover behavior. |
| Load balancers and virtualization platforms | Virtual IP handling, load balancer health checks and hypervisor-level high availability can overlap with Zabbix failover. Each integration needs its own configuration, testing and ownership split. | Document which layer owns failover for each component, and test the combined behavior once instead of testing each layer separately. |
| Compliance evidence requested | Assessors may ask for failover test records, change logs, access reviews and proof of backup restores. Producing and organizing this evidence adds documentation and test time. | Agree the evidence list before build starts, so tests and logs are captured during the work. DeployOpen prepares evidence, but independent auditors decide what it proves. |
| Design, build, testing and handover cost | Design, build, testing and handover are usually paid as project work, either to your own staff time or to an implementation partner under a written scope. Each stage has its own deliverables. | Ask for a quote that lists each stage separately, with named deliverables and who performs each task. |
| Recurring costs | The Zabbix software has no license fee. Recurring costs include infrastructure for nodes and the database, storage growth, patching, monitoring of the cluster itself, and any paid Zabbix support subscription. | Decide early who owns patching and failover tests after go-live, and whether a support subscription or a retainer is needed. |
Securing the HA cluster: database access, node trust and frontend traffic
A native Zabbix server HA cluster adds trust relationships that a single server does not have. Every node reads and writes the same database, so database accounts, network paths and the replication channel become part of the monitoring trust boundary. These cards cover only what changes because of HA. Broader controls belong to the security hardening topic.
Database accounts for each node
Every server node connects to the shared database with credentials, so the account each node uses defines what a compromised node can change. Decide whether all nodes share one account or each node has its own, and grant only the privileges the server needs for its schema and housekeeping. Confirm the exact grants for your database engine and Zabbix version in the HA documentation, and plan a credential rotation that updates every node in one controlled change window.
Network paths to the shared database
Each node needs a reliable, low-latency path to the database, and a node that loses that path stops being a working standby. Restrict database ports so only server node addresses and approved administrators can reach them. Test every node individually, because one successful connection does not prove that the others can reach the database. Record the path for each node in your runbook so failover checks are repeatable.
Database replication channel
If the database runs with replication, the replication link carries full monitoring data and uses its own credentials, so it needs the same care as client connections. Zabbix node failover does not replace database failover, so confirm how your chosen engine promotes a replica and how the Zabbix server connection settings follow the new primary. Encrypt replication traffic where the engine supports it, and agree the design with your database team before go-live.
Frontend behind a load balancer
Users reach the web frontend of any node through a load balancer, so session handling, TLS termination and forwarded client addresses must behave the same on every node. Confirm whether your balancer needs session affinity for the frontend, and restrict direct access to node web servers so traffic cannot bypass the balancer and its controls. Test a node failover while a user is logged in and record what the user experiences.
Frontend authentication settings
Internal, HTTP, LDAP, SAML and MFA authentication methods apply to the Zabbix frontend, so each frontend instance must enforce the same method and login behavior. Verify that an authentication change made on one frontend takes effect on all of them before relying on it during an incident. The design of SSO, group mapping and just-in-time provisioning is covered in the Zabbix SSO topic.
How to test Zabbix high availability failover before go-live and run it afterwards
Test failover before go-live with a written plan that proves each claim the design makes, starting with the server cluster. Stop the active server node during a controlled window while a known set of items is being collected, then record when a standby node becomes active. The gap should fall within the failover delay you configured, which the Zabbix documentation allows to range from 10 seconds to 15 minutes. Choose the value deliberately. A short delay speeds recovery but can trigger a false failover when a node is only slow under load. Run the test with the shortest delay you plan to use, confirm the new active node in the frontend and in the server log, then restart the stopped node and confirm it rejoins as a standby.
Next, fail the database primary. The test should show whether the server nodes reconnect to the surviving database and resume writing history, trends and events. The recovery path depends on your database product and replication design, so confirm the details with your database administrator: which supported version of MySQL, MariaDB, PostgreSQL or TimescaleDB you run for your Zabbix release, how a replica is promoted, and whether the connection address used by the server nodes follows the new primary. Measure the interval in which Zabbix cannot write, and check the backlog of unwritten data once the database returns. Rehearse this first on a staging copy restored from a recent backup, so the promotion steps are proven before production is touched.
Then isolate a proxy to test proxy groups, which from Zabbix 7.0 redistribute monitored hosts to other proxies when one goes offline. Block the proxy's path to its hosts and to the server, wait, and confirm that the affected hosts appear on a surviving proxy in the same group and that their data resumes there. Record the host-to-proxy assignments before and after the test. Redistribution only works if the surviving proxies can reach those hosts through your network, so check routing and firewall rules for each proxy in advance. Finally, check for silent data loss by comparing values for a sample of hosts across the test window and reviewing server and proxy logs for discarded or dropped values. Confirm that a trigger you deliberately breach still produces an alert during the outage.
After go-live, monitor the cluster's health from outside the cluster. A Zabbix node cannot reliably report its own failure, so use a second tool, or a separate Zabbix instance, to watch each server node, the active node changes, the database host, replication lag, disk space and proxy availability. Confirm which internal items your version exposes for these checks and which alerts you need to build. Back up the database and the configuration separately. Ask your database and backup teams to confirm that backups are consistent, that point-in-time recovery is available if you need it, what retention applies, and what recovery point and recovery time they can meet. Keep server and frontend configuration files for every node under version control so a rebuilt node can be brought back to the same state.
Rehearse failover on a fixed schedule, and again after any upgrade, database change or topology change. Each rehearsal should repeat the server, database and proxy tests above and record the measured failover time, the write gap and any lost data, so trends are visible over time. Keep a runbook that names each node and database host, the exact commands or console steps for every failure case, the person who can declare failover, the criteria for returning to the original primary, and the checks that confirm recovery. Update the runbook after every rehearsal. If you engage a partner, ask for these tests, the runbook and the rehearsal records to be listed as written deliverables in the fixed scope.
Questions teams ask before building Zabbix high availability
How much effort does a Zabbix high availability build take?
The effort depends mostly on the database design, the number of server nodes and how much of the surrounding platform you must build. Native Zabbix server HA needs several server nodes that share one database, so the database becomes the central design decision. Effort rises when you add database replication or a clustered database that you must build and test yourself, a load balancer or shared entry point for the web frontend, separate network paths for each node, and monitoring of the cluster itself. Effort also rises with the number of proxies and remote sites, because proxy groups and agent configuration must be checked. Effort falls when your organization already runs a tested HA pattern for the database and the scope covers a single site. Ask any provider for a written scope that lists each node, the database topology, failover tests and handover documentation, then compare quotes against that list.
Can our team build Zabbix high availability without a partner?
Yes, if your team has the skills and time to build and test it. A partner is optional. Zabbix documents the server HA design, and the cluster is configured within Zabbix itself. The harder work sits around it: a highly available database, a shared frontend entry point, reliable time synchronization, certificates, backups and a failover test plan. Most teams underestimate the testing step. A failover that has never been exercised under normal load is an untested assumption. A partner can help with scoping, a design review or a fixed-scope build, but the decision should rest on whether your staff can operate the result after handover. DeployOpen works inside your own cloud accounts, data centers or air-gapped networks, so ownership of the platform stays with you either way.
What skills and infrastructure does Zabbix HA require from our staff?
Plan for Linux administration, database administration, network engineering and working knowledge of Zabbix. Your team must build the server nodes, configure the shared database and its replication or clustering, set up a frontend entry point, and manage certificates and time synchronization. On the infrastructure side, each server node needs its own host, the database environment must be able to fail over independently, and network paths must let nodes reach the database, agents and proxies. Storage must be sized for the database. Confirm supported database versions and platform requirements in the Zabbix requirements documentation before ordering hardware, because supported versions change between releases. You also need a named owner for upgrades, backup restores and on-call response to alerts about the cluster itself.
How long does a Zabbix high availability project take?
Duration depends on the drivers of the work, not on a fixed figure. The main drivers are the database topology you choose, the number of nodes and sites involved, whether the environment is new or extends an existing Zabbix installation, how many proxies and integrations must be verified, the change approval process in your organization, and how many failover tests are needed before go-live. Access to cloud accounts, data center change windows and staff availability often set the pace more than the technical work does. Ask for a timeline built on milestones: design sign-off, build in a non-production environment, failover tests, production cutover and handover. Also ask which milestones depend on your own teams, because those dates sit outside any provider's control.
Do Zabbix proxy groups replace server high availability?
No. Proxy groups and server HA solve different problems. Server HA keeps the central server and its frontend running when a server node fails, using a shared database and standby nodes that take over after the configured failover delay. Proxy groups, available from Zabbix 7.0, redistribute monitored hosts between proxies when a proxy goes offline, which protects data collection at remote sites and in segmented networks. Many designs use both: server nodes for the core and proxy groups for distributed collection. If your main risk is a remote site losing its collector, proxy groups address that risk directly. If your main risk is the central server going down, you need server HA and a resilient database. Confirm how proxy groups behave in your Zabbix version by reading the proxy load balancing and high availability documentation.
What happens to in-flight monitoring data during a Zabbix failover?
Data in transit during a failover is at risk of delay or loss, and you should confirm the exact handling in the Zabbix documentation and in your own tests. When the active node stops, the standby waits for the configured failover delay, which can be set from 10 seconds to 15 minutes, and then takes over. Values collected during that gap may be delayed, queued or lost, depending on where they sit: on an agent, on a proxy, or in server memory that has not yet been written to the database. A shorter delay reduces the gap but raises the chance of a false failover during a brief slowdown. Test by stopping the active node under normal load in a staging environment, then compare item values in the database before and after. Record the results so operators know which gaps to expect.
Does running Zabbix high availability require a paid support subscription?
No. Zabbix states that its open source software has no license fee, no device limits and no feature gates. The costs to plan for are infrastructure, database storage, the staff time to run the cluster, and any paid Zabbix technical support subscription you choose. According to the Zabbix support page, tiers run from 8x5 support with a 1 day response at Silver to 24x7 support with a 1 hour response at Global. Ask whether your on-call model needs 24x7 response before you choose a tier. Implementation work, including HA design, testing and handover, is a separate cost that follows from a written scope. DeployOpen does not publish a price list, and a quote is based on the scope you agree in writing.
How DeployOpen works on this
DeployOpen scopes Zabbix high availability as a fixed-scope engagement with a written list of nodes, database, proxies and tests before work starts. We build the cluster inside your own infrastructure, prove failover under controlled conditions, and hand over runbooks your operators can use without us.
How we scope
- Confirm which Zabbix line you run or plan to run, such as 7.0 LTS or 7.4, and the support window that applies to it
- Map server nodes, the shared database engine and its HA design, and the network paths between nodes
- Agree the failover delay value, the proxy groups for remote sites, and the failure scenarios to be tested
How we build
- Install and configure the server nodes and shared database with the exact versions your scope names, following the vendor's HA documentation
- Set up database replication or clustering where your design needs it, and verify that each node sees the same data before go-live
- Configure proxy groups for remote sites, then test node loss, proxy loss and database failover against documented expected behavior
How we hand over
- Deliver architecture diagrams, configuration records, and the failover delay and proxy group settings with their reasoning
- Provide runbooks for planned node maintenance, failover checks, and recovery after a node or proxy goes offline
- Run a handover session with your operators and record the test results against the acceptance criteria in the written scope
How we support
- Offer ongoing support and upgrade work as a retainer, with the scope and response expectations written into the agreement
- Review HA behavior after Zabbix upgrades, database changes and changes to proxy layout, and document any impact
- Confirm with Zabbix which paid technical support tier, if any, fits your response requirements, since that subscription is separate from DeployOpen's services
