Working Through vSAN Health Service Alerts and Field-Proven Fixes
VMware vSphere administrators in Australia run some of the most geographically dispersed environments in the Asia-Pacific region. From NextDC S3 in Sydney to Equinix ME1 in Melbourne and newer builds going live in Perth, the pressure to keep stretched clusters healthy has never been higher. The vSAN Health Service sits quietly inside vCenter, polling ESXi hosts and surfacing warnings long before an outage reaches the operations team in Sydney or the support desk in Brisbane.
When a vSAN health alert escalates into a cluster-wide event, the cost is rarely just technical. Australian financial services and mining firms operate under tight data sovereignty rules, and any unplanned downtime in a vSAN-backed cluster can ripple into customer-facing systems that are bound by APRA CPS 234 obligations. Understanding how the health service categorises warnings, how to automate checks through PowerCLI, and which remediation steps to apply in what order is what separates firefighting from a calm, predictable operations cadence.
| Alert Category | Sample Checks | Typical Cause | Severity |
|---|---|---|---|
| Cluster | Member hosts, master/slave, partition | Split-brain, network partition | Red |
| Network | Multicast, vmkernel, jumbo frames | MTU mismatch, blocked UDP 2233 | Red |
| Storage | Disk groups, capacity headroom, HCL | Unhealthy disks, full cache tier | Yellow to Red |
| Hardware compatibility | HCL, drivers, firmware | Unsupported device on VSAN HCL | Yellow |
| Performance | Latency, congestion, IOPS limits | Oversubscribed backend, slow disks | Yellow |
Decoding the health service architecture
The Health Service is a collection of tests grouped into categories that mirror the layers of a vSAN stack. Each category runs on a defined cadence: some checks fire every five minutes, while longer-running hardware compatibility or firmware audits might only fire once per day. Administrators monitoring environments across Australian time zones need to remember that vCenter's scheduled tasks inherit the host's local clock, so an alert that fires at 03:00 AEST will reach a Melbourne-based engineer well before their shift begins.
At the top level, the cluster tests validate master and slave roles, witness placement for stretched clusters, and the number of member hosts. Below that, network tests confirm multicast traffic and vMotion adjacency, storage tests inspect disk group health and capacity, and hardware compatibility tests compare installed components against the published VSAN HCL. Performance tests, which became richer in recent releases, look at backend congestion and component-level latency.
Understanding this layering matters because remediation rarely starts with the alert that fires first. A "Disk balance" yellow flag may actually originate from a failing cache device, which belongs to the hardware category. Walking up and down the tree before acting prevents the classic mistake of replacing a healthy disk because the wrong screen caught the eye first.
Tuning alerts for Australian datacentre realities
Datacentre operators in Sydney and Melbourne frequently run hybrid clusters that span on-premises capacity and AWS Outposts or local cloud providers such as AUCloud. The Health Service is happy with that topology, but several checks deserve close attention in this context. Network latency between sites can be flagged by the vSAN performance service even when individual disk operations are healthy, and the alert is easily misread as a cluster-wide fault.
Multicast tests are another frequent offender. Many Australian enterprises use Cisco Nexus or Arista switches with IGMP snooping enabled by default, and the Health Service will fire a warning if vSAN traffic cannot reach all hosts. Configuring a dedicated VLAN with the correct querier settings and verifying UDP 2233 and 2325 are unblocked should be the first port of call before deeper troubleshooting begins.
Capacity headroom also deserves explicit attention. Many shops still follow the vendor recommendation of keeping at least 30 percent free space, but Australian mining and retail customers running predictable workloads often trim this to 20 percent to maximise hardware spend. The Health Service's storage pool capacity check makes the trade-off visible, and operators should decide in advance what threshold triggers procurement rather than waiting for a red event.
Building a proactive workflow with PowerCLI and APIs
Manual checks do not scale across sites in Sydney, Melbourne, Adelaide, and a DR presence in Perth. A practical workflow connects the vSAN Health Service to existing monitoring stacks such as VMware Aria Operations, PRTG, or even a humble syslog server. PowerCLI remains the most accessible starting point because it does not require additional licensing and most Australian IT teams already have it installed on their management jump hosts.
The Get-VsanClusterHealth cmdlet returns a structured object that mirrors what the vSphere Client shows under Monitor, then vSAN, then Health. Pairing it with Get-VsanHealthTest and filtering on HealthTestStatus makes it trivial to push the failing checks into a Slack or Microsoft Teams channel each morning. Engineers in different states can subscribe only to categories that match their on-call rotation, which keeps the signal-to-noise ratio high during quiet periods.
For deeper integration, the vSAN Management API exposes /api/v1/health/cluster endpoints that can feed custom dashboards. A common pattern in Australian consultancies is to combine those endpoints with the AWS Health Dashboard for Outposts, giving a single pane that covers both halves of a hybrid estate without forcing teams to learn yet another portal.
Remediation patterns for the most common warnings
When the Health Service fires red, speed matters more than elegance. The most reliable remediation playbook starts with collecting a log bundle, then isolating the failing host from the cluster, and only then replacing hardware or adjusting configuration. Skipping the log step is the single most expensive mistake seen in Australian support tickets, because it forces a return visit once the root cause becomes apparent.
Disk and disk group failures are best handled by placing the host in maintenance mode with the "Ensure accessibility" option set, which migrates VMs and rebuilds components on remaining capacity. Network warnings usually require tracing the vSAN vmkernel adapter back to the physical NIC and confirming MTU 9000 is honoured end to end. Hardware compatibility warnings can only be cleared by updating drivers or firmware against the supported versions listed on the VSAN HCL, and any planned rollouts should be scheduled outside Australian business hours to limit impact.
For stretched clusters, witness appliance warnings deserve particular care. The witness lives in a third site, often Sydney for east-coast clusters, and connectivity issues frequently masquerade as cluster-level alerts. Validating the witness network path before assuming the production sites are at fault avoids the embarrassing scenario of opening a Sev1 ticket at 02:00 AEST on something that a quick ping would have ruled out.
Long-term operational practices
Sustained vSAN health comes from disciplined operations rather than heroic recoveries. Quarterly HCL audits, scheduled disk scrub windows, and a documented threshold matrix for capacity and performance warnings keep the Health Service quiet. Australian integrators such as Datacom and Logicalis often package these activities into monthly service reviews, and the same cadence works for in-house teams.
Documentation should capture not only what each alert means, but also the named owner and the expected remediation time. A runbook that pairs every red and yellow category with an Australian on-call contact, a vendor escalation path, and a rollback plan will outlast any single engineer on the team. With that scaffolding in place, the Health Service stops being a fire alarm and starts acting as the early-warning system it was always meant to be.