Running Nomad workloads with Consul discovery and Connect

Nomad places and supervises workloads, while Consul maintains a service catalogue, DNS records and optional Connect proxies. Together they let an application request orders-api instead of carrying an allocation address or fixed cloud endpoint through deployments. Nomad owns the workload lifecycle; Consul describes how healthy instances are found and reached.

A reliable setup needs more than a service block. Checks, ACL tokens, namespaces, intentions, sidecar resources and update behaviour must agree across the Nomad client and Consul agents. Otherwise a job can run while its endpoints are absent, unhealthy or unreachable.

Consul discovery handles internal service lookup, while Connect adds proxy-based traffic control, identity and encrypted transport. The operating model should reflect that difference rather than applying mesh settings to every workload.

Concern	Discovery only	With Connect
Service location	DNS or catalogue APIs	A local proxy reaches healthy upstreams
Traffic control	The application handles each connection	Proxy policies and intentions govern traffic
Failure mode	A client can bypass catalogue health	Identity or intention errors can block traffic
Operations	Mostly health checks	Sidecar resources, certificates and logs

Define the service contract

Start with one logical service name per independently deployable API. Use tags for routing and metadata, such as production, orders or a release identifier. Clients should never select an instance by Nomad allocation ID, because those IDs disappear during rescheduling.

Make readiness decisive. An HTTP check against /health/ready should succeed only after the process can serve traffic, while liveness checks detect stuck processes. Consul removes failing instances from discovery, but Nomad must still restart allocations that repeatedly fail.

Treat the service definition as an API contract. Changing its name, port label, namespace or partition affects clients even when the application image is unchanged. Keep these values under version control and review changes to tags and health paths carefully.

Register services from Nomad

A Nomad service block ties the task group to Consul and lets Nomad maintain the registration as allocations start and stop. This example uses a labelled HTTP port and an explicit readiness check:

job "orders-api" {
  datacenters = ["dc1"]

  group "api" {
    network {
      port "http" {
        static = 8080
        to     = 8080
      }
    }

    service {
      name     = "orders-api"
      port     = "http"
      provider = "consul"
      tags     = ["orders", "production"]

      check {
        type     = "http"
        path     = "/health/ready"
        interval = "10s"
        timeout  = "2s"
      }
    }

    task "orders-api" {
      driver = "docker"
      config {
        image = "example/orders-api:1.8.0"
        ports = ["http"]
      }
    }
  }
}

Run nomad job run orders.nomad.hcl, then inspect job and allocation status before testing Consul. Applications can query orders-api.service.consul or the catalogue API; they should retry another healthy instance rather than pin one returned address. The Nomad client’s Consul token needs permission to register and update the service.

Use rolling updates and let new instances pass readiness checks before retiring old ones. A rollout can report task success while the service remains empty or unstable. Compare Nomad allocation state with consul catalog service orders-api during each deployment.

Add Connect for controlled traffic

Connect places a proxy beside a Nomad task. Declare an upstream in the service stanza and give the proxy a local port that the application can call:

connect {
  sidecar_service {
    proxy {
      upstreams = [{
        destination_name = "payments-api"
        local_bind_port = 9091
      }]
    }
  }
}

The application connects to 127.0.0.1:9091; the sidecar selects a healthy payments-api instance and enforces the applicable namespace, partition and intention. Destination names must match Consul registrations, not Nomad job names. A running upstream is insufficient if identity policy blocks it.

Create narrow intentions that allow only the source service to reach the destination, and give Nomad and sidecar identities least-privilege policies. Certificate issuance and renewal need reliable time and access to agents that issue Consul certificates. Include proxy CPU, memory and logs in allocation monitoring; sidecars consume resources too.

Operate the Australian footprint

AWS Sydney and Melbourne use ap-southeast-2 and ap-southeast-4. Place latency-sensitive Nomad clients, Consul agents and services in the same metropolitan region where possible; cross-region calls and Consul WAN traffic add delay and widen the failure domain. A failover can also change endpoints and data paths.

The Australian market remains strongly hybrid. Many organisations run VMware estates on premises while moving selected workloads to AWS, often with managed service providers supporting both. NBN or business-fibre connections may sit behind NAT, so Consul clients need stable outbound access and correctly configured gossip and RPC paths.

Australian teams often say “the service is up” when the process exists, but the registered endpoint and passing check provide the operational meaning. Use Australian English around authorisation and prioritisation, yet keep states unambiguous in dashboards. Clock changes across AEDT, AEST, ACST and AWST can expose brittle renewal windows, so retain margin in TTL and certificate timing.

Troubleshoot from allocation to proxy

Trace the layer that owns the symptom. Nomad status reveals failed allocations and task restarts; consul catalog service shows registration and health; agent and sidecar logs expose ACL, intention, certificate and upstream errors. DNS failures can also result from querying the wrong namespace or partition, so verify the task’s Consul agent.

Separate registration failures from routing failures. A missing service often means the Nomad client lacks a token, the task has no service block, or the check uses the wrong port. A registered but unreachable service points towards intentions, proxy configuration, security groups or local bind-port conflicts. Inspect proxy access logs before restarting allocations.

Resource contention needs evidence as well. For a JVM-based API, Java CPU profiling can distinguish application hot spots from a busy sidecar on the same host. Combine allocation CPU, memory, connection counts and latency percentiles, then identify the constrained layer before scaling. A healthy catalogue entry helps only when the complete client-to-workload path remains healthy.