Deploying HashiCorp Nomad with C2DS and host volume management

HashiCorp Nomad is a practical scheduler for Australian teams that already operate Linux fleets, Docker hosts, and cloud infrastructure without adopting a full Kubernetes platform. A small control plane can place batch jobs and services on selected clients while keeping deployment definitions in version control.

The difficult parts are usually operational rather than declarative. C2DS, Nomad’s client-to-daemon services path, must remain healthy across firewalls, while host volumes need explicit permissions, backups, and cleanup rules. This guide builds a small secure deployment and shows where CSI storage should replace node-local directories.

Prepare the Linux hosts

Use supported 64-bit Linux releases and install identical Nomad versions on servers and clients. Give the Nomad user access to Docker or the selected task driver, and keep /opt/nomad, /var/lib/nomad, and TLS material on persistent storage with restrictive permissions.

For a production control plane, place three servers across availability zones in one region and set bootstrap_expect = 3; use one server only for a lab. Give every client a stable name, because host-volume placement and operational diagnostics depend on knowing which node owns the data.

Configure servers and clients

Keep role-specific files under /etc/nomad.d, then validate them with nomad validate before starting the service. A development server can begin with this configuration:

region     = "au"
datacenter = "syd"
name       = "nomad-syd-01"

server {
  enabled          = true
  bootstrap_expect = 1
}

Use the following client configuration separately, with the same region and datacenter as the servers it should join:

region     = "au"
datacenter = "syd"
name       = "nomad-client-01"

client {
  enabled = true
}

Start the servers before the clients, then confirm gossip membership and node registration. For production, replace the single-server bootstrap value with three and place Raft voters in separate availability zones; clients belong to one home region rather than joining every regional server.

Secure the C2DS path

C2DS is not a separate sidecar or listener. It rides Nomad’s RPC channel and supports operations such as nomad alloc exec, log streaming, and filesystem access, so the RPC listener and its certificates determine whether these features work.

Permit TCP 4647 from clients to servers; keep gossip on 4648 between servers, and expose 4646 only to management networks. Enable RPC mTLS on both roles, issue certificates with usable DNS or IP SANs, set verify_server_hostname = true, and account for long-lived streams when choosing network timeouts.

With ACLs enabled, issue operator tokens narrowly scoped for allocation execution and logs. Diagnose failures with nomad node status, nomad server members, and client journal entries; expired certificates, clock skew, blocked port 4647, and an overloaded load balancer commonly mimic a C2DS outage.

Define host volume policy

Create each directory before placement and align its ownership with the task user. For example, install -d -o 1000 -g 1000 -m 0750 /srv/nomad/payroll creates a path suitable for a container running as UID 1000. If the path is not present on every client, constrain placement with a node attribute identifying its owner.

A job references a named job volume, whose source is the absolute host path:

job "payroll-api" {
  datacenters = ["syd"]

  group "api" {
    task "api" {
      driver = "docker"
      user   = "1000:1000"

      config {
        image = "registry.example.com/payroll:1.4.0"
      }

      volume {
        type        = "host"
        source      = "payroll-data"
        destination = "/var/lib/payroll"
      }
    }
  }

  volume "payroll-data" {
    type   = "host"
    source = "/srv/nomad/payroll"
  }
}

Validate with nomad validate payroll.nomad, then run it with nomad job run payroll.nomad. The directory remains after allocation termination and job purge, so deletion must be a separate, audited operation. On EC2, snapshot the EBS volume only after application-level quiescing when consistency matters.

Fit the deployment to Australia

Run authoritative servers in Sydney’s ap-southeast-2 region and use Melbourne’s ap-southeast-4 for a representative lab or separately operated regional cluster. Keep traffic private, and measure cross-region latency before moving latency-sensitive allocations or replication traffic between the two cities. When choosing between an Elastic Beanstalk service and Nomad scheduling, the Beanstalk autoscaling guide provides a useful operational comparison for health checks and capacity.

Australian data-residency obligations and internal policies should decide where workloads and backups remain; keep sensitive allocation data in Australia and map controls to ACSC Essential Eight outcomes. Follow local operating customs by recording dates day-first, labelling maintenance windows AEST or AEDT, and accounting for daylight-saving changes. For procurement, price EC2, storage, and support through an Australian channel in AUD with GST included rather than assuming an overseas quotation matches local buying terms.

Troubleshoot and select storage

Start with placement evidence: inspect the allocation on its assigned node, verify the path with stat, compare host and container UIDs, then test read and write access. If nomad alloc exec fails while the job is healthy, treat it as a separate C2DS or ACL problem rather than a volume problem.

The operational split becomes clearer through this comparison:

Concern	Host volume	CSI volume
Provisioning	Directory is prepared outside Nomad	Controller and plugin provision storage
Placement	Requires the path on the selected client	Access and attachment modes guide placement
Lifecycle	Data survives allocation termination and job purge	Attachments are managed; persistence follows reclaim policy
Portability	Node-local and path-specific	Can move among compatible clients
Encryption	Disk or application controls it	Backend and plugin settings control it
Best suited to	Caches, exports, and legacy node-local data	Shared, replicated, or managed storage

Use host volumes for node-local caches, exports, and legacy applications that cannot consume CSI. Choose CSI when its backend supplies provisioning, encryption, replication, or portable storage; neither approach removes the need for tested backups.