HashiCorp Vault Agent with AWS Auto Scaling for Secrets Injection

In a tradie's ute, tools get tossed in the back and grabbed when needed. HashiCorp Vault Agent on EC2 instances works similarly — secrets are fetched on demand and cached close to the workload, rather than baked into AMIs or hardcoded into user data. For shops running Auto Scaling groups across Sydney and Melbourne, this approach removes friction when fleets grow and shrink throughout the day.

Australian IT teams have adopted elastic compute quickly, particularly fintechs in Sydney and ASX-listed enterprises running batch jobs out of Melbourne. Traditional secrets management breaks down when instances spin up faster than a barista during morning rush at a Bondi cafe. Static credentials go stale, IAM keys get copied onto golden images, and rotation becomes a manual nightmare nobody wants to own.

This walkthrough covers configuring Vault Agent with the AWS authentication method, integrating it into Auto Scaling lifecycle hooks, and handling token leases as instances terminate. It assumes familiarity with Vault, EC2, and IAM, focusing on practical patterns that hold up in production.

Why dynamic credentials matter for elastic fleets

Secrets sprawl hits hard when infrastructure is elastic. An Auto Scaling group launching twenty instances during a peak period needs twenty sets of database credentials, API tokens, and TLS certificates. Hardcoding these into AMIs is a security antipattern — a compromised golden image exposes every credential baked into it. APRA and the Essential Eight framework both push for ephemeral credentials that expire quickly and rotate automatically.

Vault generates short-lived credentials on demand. Rather than distributing long-lived secrets, instances authenticate and receive a token with a lease duration. When the lease expires or the instance terminates, the credentials cease to exist. This model aligns with the ephemeral nature of Auto Scaling groups, where instances come and go without manual intervention.

The challenge is making authentication transparent to the application. Developers do not want Vault API calls in every service. Vault Agent handles the authentication dance, renews tokens, and renders secrets to files or environment variables that legacy code can consume.

Configuring the AWS auth method

The AWS authentication method lets EC2 instances authenticate using their IAM role signature. Each instance presents a signed GetCallerIdentity request, which Vault validates against AWS before issuing a token. Setup starts with enabling the auth method and creating a role that maps to an IAM role attached to the Auto Scaling group's launch template.

Configuration involves defining a bound IAM principal ARN, the type of EC2 authentication, and token policies. The bound_iam_principal_arn parameter accepts the role used by instances, while token_policies attaches Vault policies granting access to specific secret paths. For a deployment in ap-southeast-2 (Sydney) or ap-southeast-4 (Melbourne), the IAM role must exist in the same partition as Vault's configured AWS account.

The launch template needs an instance profile with the appropriate IAM role. The role requires ec2:DescribeInstances and sts:GetCallerIdentity permissions. Many Australian shops use AWS Organizations with separate accounts for production and non-production, so Vault roles reference the production account's role ARN explicitly.

Vault Agent configuration and templates

The agent.conf file controls how Vault Agent operates on each instance. Configuration specifies the Vault address, auth method, token sink file, and templates that render secrets to disk. Templates use Consul Template syntax, allowing conditional logic for complex rendering scenarios.

A typical configuration points to the regional Vault cluster, configures the AWS auth method with the matching role, and renders secrets to /etc/myapp/secrets/. The template block pulls from secret paths and writes structured files that applications read on startup. Cache directives reduce API calls when multiple processes need the same secret.

For Auto Scaling integration, the agent runs as a systemd service that starts before the application. Health checks ensure that if Vault becomes unreachable, the agent retries without blocking startup indefinitely. This matters in Australia where NBN dropouts or cross-region latency between Sydney and Melbourne can briefly interrupt connectivity.

Lifecycle hooks and graceful termination

Lifecycle hooks pause instance launch or termination, giving external systems time to react. The instance-launch hook waits until the instance signals success after bootstrapping — the ideal place to verify Vault Agent has authenticated and rendered secrets. Without this, an instance might enter the InService state before secrets are available, causing application startup failures.

For scale-in events, the instance-terminating lifecycle hook gives Vault Agent a window to revoke its token and clean up lease IDs. AWS sends a notification to SNS or EventBridge, triggering a Lambda that calls Vault's revoke endpoint with the instance's token. This prevents dangling leases from accumulating in Vault's token store.

Token cleanup matters for quota management and audit hygiene. Australian compliance teams — particularly in banking and government — care deeply about token lifecycle visibility. Stale tokens from terminated instances create blind spots during APRA audits and complicate forensic investigations.

Handling high churn and regional failover

Auto Scaling groups with aggressive scale-in policies — common during EOFY batch processing — generate dozens of instances per hour. Each authenticates to Vault, producing API load that must be planned for. Performance replication in the Sydney region handles this well, but dual-region setups between Melbourne and Sydney should configure the agent with multiple Vault addresses and failover logic.

Cross-region latency from Perth or Darwin to the nearest cluster in Sydney can exceed 60ms, which compounds during token renewal cycles. Edge workloads in Western Australia, often near mining operations, run on patchy connectivity. Caching and renewal intervals need tuning to match the network reality rather than the default five-minute interval.

Practical recommendations for production rollouts

• Use a dedicated IAM role per Auto Scaling group rather than sharing one, so Vault policies can be scoped tightly to specific workloads
• Configure token TTLs shorter than the average instance lifetime, so natural expiration handles cleanup even if lifecycle hooks fail
• Deploy Vault in performance standby mode in a second region (typically Melbourne if Sydney is primary) and configure the agent for automatic failover
• Monitor token counts and lease activity through Prometheus, with alerts on sudden spikes that could indicate a compromised instance
• Test scale-in events regularly in pre-production using AWS Fault Injection Simulator to validate the revoke path actually fires