Deploying Nomad with GPU passthrough for machine learning workloads

HashiCorp Nomad is a practical scheduler for machine learning teams that need predictable placement without adopting the full Kubernetes stack. With the right client configuration, Nomad can run training jobs, inference services, and batch pipelines on hosts equipped with NVIDIA GPUs.

GPU passthrough adds another layer to the design. The hypervisor must assign a physical device to a virtual machine, the guest operating system needs the correct driver and container runtime, and Nomad must advertise the accelerator as a schedulable resource.

This approach suits Australian teams running workloads across a home lab, a private VMware cluster, or cloud-connected infrastructure in Sydney and Melbourne. It also provides a useful way to keep sensitive training data within an approved region while scaling selected workloads into public cloud.

Choose the host and passthrough model

A bare-metal Nomad client is the simplest option because the operating system can access the GPU directly. Virtualised deployments are also workable, but the hypervisor must expose the entire PCI device to the guest. In VMware, this generally means enabling DirectPath I/O for the GPU, reserving memory for the virtual machine, and rebooting the ESXi host before the device becomes available.

Enable IOMMU in the server firmware before configuring PCI assignment. Intel systems commonly use intel_iommu=on, while AMD systems use amd_iommu=on. Confirm that the GPU is isolated from the host and that no display manager or competing VM has claimed it.

For a home lab in Brisbane or Perth, check power, cooling, and upstream bandwidth before installing a high-end card. A GPU can turn a small rack into a warm and noisy arvo project, particularly in summer. Data centre hosts in Sydney should also be checked for sufficient power and approved GPU hardware, rather than assuming every instance type supports passthrough.

Prepare the guest operating system

Use a supported Linux distribution on the Nomad client and install the NVIDIA driver that matches the kernel and GPU generation. After installation, nvidia-smi should show the card, driver version, temperature, and available memory. Resolve driver issues before involving Nomad; the scheduler cannot make an unavailable device usable.

Install the NVIDIA Container Toolkit when jobs will run in Docker containers. The Docker daemon should be configured with the NVIDIA runtime, and a simple test should succeed:

docker run --rm --gpus all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smi

The CUDA version in the image must be compatible with the host driver. A newer container runtime cannot compensate for an older driver, so pin tested driver and image versions in the same way you would pin application dependencies.

Install and configure Nomad

Install Nomad on the GPU machine as a client, using a server cluster elsewhere or on separate nodes. Keep server and client roles distinct for production environments. The client configuration should identify the node, enable the Docker driver, and join the correct gossip and RPC addresses.

Nomad’s device plugin discovers hardware and exposes it to the scheduler. For NVIDIA workloads, configure the NVIDIA device plugin on the client and verify that Nomad reports the GPU in its node information. The exact plugin configuration depends on the Nomad release, so keep the client and plugin versions aligned and review the generated device inventory after every driver upgrade.

A minimal client configuration typically resembles:

client {
  enabled = true
  servers = ["10.0.10.11:4647", "10.0.10.12:4647"]
}

plugin "docker" {
  config {
    allow_privileged = false
  }
}

Nomad must be able to resolve the server addresses and reach ports for RPC, HTTP, and Serf traffic. In AWS, routing and security groups matter just as much as the Nomad configuration; the practical checks described in VPC internet access are useful when clients need to pull images or contact external registries.

Request GPUs in a job specification

A batch training job should request a GPU explicitly rather than relying on a node class alone. This prevents multiple jobs from competing for the same card and allows Nomad to place work only on clients with matching capacity.

A simplified task group can use a device requirement alongside CPU and memory resources:

job "model-training" {
  type = "batch"

  group "trainer" {
    count = 1

    device "nvidia/gpu" {
      count = 1
    }

    task "pytorch" {
      driver = "docker"

      config {
        image = "registry.example.com/ml/pytorch:2.2-cuda12"
      }

      resources {
        cpu    = 4000
        memory = 16384
      }
    }
  }
}

Use constraints when a workload needs a particular GPU model, CUDA capability, or local dataset. Tags such as gpu_model = "A100" or cuda_major = "12" can separate inference nodes from training nodes. For ordinary consumer cards, a node attribute is often sufficient, but it should reflect a validated capability rather than a marketing name.

Control data, images, and model storage

Training jobs frequently need large datasets and model checkpoints. Container layers are a poor substitute for durable storage, so mount a local filesystem, NFS export, object-storage cache, or CSI volume with clear read and write boundaries. Keep checkpoints outside the container so a failed allocation can resume on another run.

For Australian organisations, storage location may affect contracts, privacy assessments, and customer commitments. A team using an AWS region in Sydney may keep source datasets and model artefacts there, while allowing non-sensitive container images to come from a global registry. Cross-region transfers to Singapore or the United States can introduce both latency and governance concerns.

Use immutable image tags or digests, and authenticate to private registries through Nomad variables or a secrets manager. Avoid placing registry passwords directly in a job file committed to Git. Large model downloads should be cached locally where possible, especially for sites connected through variable NBN services.

Schedule safely and monitor utilisation

GPU memory is often the limiting resource, not CPU. A job can appear healthy while failing with CUDA out-of-memory errors because the model, batch size, and concurrent processes exceed the device capacity. Set application-level limits and use one allocation per GPU unless the workload has been tested with sharing.

Monitor nvidia-smi, Docker task statistics, Nomad allocation events, and host temperatures. Export metrics to Prometheus or another monitoring platform, recording GPU utilisation, memory consumption, ECC errors, fan speed, and allocation restarts. Alert on a client that remains registered but has lost its device, since Nomad may otherwise keep attempting placement.

MIG-capable cards can divide a supported GPU into hardware-partitioned instances, but this requires compatible hardware, driver versions, and device-plugin behaviour. Treat MIG profiles as separate resources and validate scheduling after reboots. Ordinary time-slicing is less isolated and should not be presented as equivalent to dedicated GPU assignment.

Troubleshoot failed allocations

Start with the host: confirm PCI passthrough, inspect dmesg, run nvidia-smi, and test the container runtime independently. If the host sees the card but the container does not, inspect Docker’s runtime configuration and the NVIDIA Container Toolkit. If the container works but Nomad cannot place the job, inspect the client’s device inventory and the allocation evaluation.

Common failures include an IOMMU setting missing from the bootloader, a GPU still attached to a virtual display, mismatched CUDA and driver versions, and insufficient memory reserved for a VM. A client can also be healthy from Nomad’s perspective while its GPU plugin has failed, so check client and plugin logs rather than relying only on node status.

When a workload is stuck in pending state, run nomad job status, inspect evaluation details, and compare the requested device name with the resource advertised by the client. In hybrid environments, firewall rules, DNS, and asymmetric routing between Melbourne, Sydney, and an on-premises site can produce symptoms that look like scheduling faults but are actually connectivity problems.

Operate upgrades and recovery

Drain a GPU client before changing drivers, the hypervisor, or the device-plugin configuration. Nomad can then stop allocations cleanly and prevent new work from landing on the node. After rebooting, validate the PCI device, NVIDIA driver, container test, Nomad client registration, and a small disposable GPU job in that order.

Keep a record of firmware, ESXi, Linux kernel, NVIDIA driver, CUDA image, Docker, and Nomad versions. This version matrix makes rollback practical when a previously stable training pipeline fails after an upgrade. It also helps Australian managed-service teams document which components are covered by a vendor agreement and which are maintained internally.

With passthrough correctly configured, Nomad provides a straightforward control plane for GPU-aware workloads: the hypervisor owns device assignment, Linux owns the driver, the container runtime exposes the accelerator, and Nomad places jobs according to declared capacity.