ctx: prod
back to posts
$kubectl describepost/eks-cross-az-cost

The half of your EKS bill nobody reads

Karpenter v1.13.0 shipped on June 10, and my feed has been wall-to-wall EKS cost content ever since: consolidate your nodes, move stateless workloads to Spot, switch to Graviton, right-size everything. All of it is correct. All of it is about compute.

And compute is the right place to start. It's usually the biggest line on the bill and the one you have the most direct control over. But it's only half the invoice. The other half is data transfer, and it hides in plain sight: split across a dozen tiny rows nobody scrolls down to read. So it survives every cost review, quietly, for years.

This is the order I actually work a noisy EKS bill, and why the second half is the one that gets left on the table.

The 60-second version — flip through the deck8 slides · swipe →
The premise

Your EKS bill has two halves

Everyone tunes compute. Almost nobody reads data transfer.

The quiet one

The chatter tax

Two pods across AZs cost $0.01/GBeach way. Spread "for HA" pays it on every hop.

The rates

What each byte costs

Cross-AZ $0.01/GB · NAT processing $0.045/GB · egress ~$0.09/GB.

The order

Compute → cross-AZ → NAT

Start with compute. Don't stop there.

takeaway

Compute is the number you watch

Data transfer is the number that watches you.

Half the invoice is compute. You already know that half.

Karpenter, Spot, and Graviton are genuinely the highest-leverage moves on most clusters, and I reach for them first:

  • Karpenter consolidation. Bin-pack pods onto fewer, cheaper nodes and let idle capacity terminate. v1.13.0's changes are mostly operational (subnet-level IP tracking, configurable AMI/subnet refresh, custom IAM paths) rather than new cost levers, but the consolidation engine that makes Karpenter worth running has been there all along.
  • Spot for stateless. Anything that tolerates interruption runs at a fraction of on-demand.
  • Graviton where the image is multi-arch. ARM instances are cheaper per vCPU, and if your containers already build for arm64, the migration is mostly a node-pool change.

Do all of it. Then look at the number sitting one row down.

The half nobody reads

Data transfer inside AWS isn't one charge. It's several small ones that only look small until you multiply them by "all your traffic, all the time."

Compute is the number you watch. Data transfer is the number that watches you.

Here's what each byte actually costs on an EKS cluster in us-east-1:

Line itemRateWhere it comes from
Cross-AZ transfer$0.01/GB each directionpods, services, or storage talking across Availability Zones
NAT Gateway processing$0.045/GBany outbound byte routed through a NAT, on top of egress
Internet egress~$0.09/GB (first 10 TB)traffic leaving AWS to the internet

The trap is that these compound. A pod that pulls an image from the internet through a NAT Gateway in a different AZ can pay cross-AZ and NAT processing and egress on the same bytes.

The chatter tax

The most common silent cost is intra-cluster traffic crossing AZ boundaries. You spread your workloads across three Availability Zones for resilience, which is a good instinct. But every time a pod in us-east-1a calls a service whose endpoint happens to land in us-east-1b, that round trip costs you $0.01/GB out and $0.01/GB back.

what a single cross-AZ round trip costs
Pod in AZ-amakes a call
crosses the AZ line$0.01/GB out
reply comes back$0.01/GB back

For a chatty microservice architecture (service mesh, sidecars, high-QPS internal APIs) this adds up to a real line on the bill. And most of it is invisible, because no single call is expensive. It's the aggregate that hurts.

The order I actually work a bill

first to last
1 · ComputeKarpenter consolidation, Spot, Graviton. Biggest single lever — do it first.start here
2 · Cross-AZMeasure intra-cluster traffic that crosses zones. Keep hot paths in-zone where safe.the quiet one
3 · NATRoute S3/ECR/DynamoDB through VPC endpoints so image pulls and API calls skip the NAT meter.easy win

The NAT step is often the fastest money you'll find. Gateway VPC endpoints for S3 and DynamoDB are free, and they take that traffic off the NAT Gateway entirely:

# S3 traffic through a Gateway VPC endpoint:
# no NAT processing charge, no cross-AZ charge.
resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.us-east-1.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = aws_route_table.private[*].id
}

For ECR, interface endpoints do the same for image pulls. On a cluster that scales nodes up and down all day, that's a surprising amount of traffic to be paying NAT rates on.

Where I've landed

A lot of the cross-AZ spread is there for "resilience" nobody actually measured. Sometimes the multi-AZ topology is load-bearing and the transfer cost is the price of staying up. Often it's a Helm chart default, a topologySpreadConstraint copied from a blog, and a bill nobody ever traced back to a specific decision.

I'm not arguing against multi-AZ. I'm arguing for knowing what it costs you, so it's a decision instead of an accident. Turn on cost allocation by AZ, look at the data-transfer line for one week, and see how much of it is workloads talking to themselves across a zone boundary they didn't need to cross.

Compute is where you start. It's just not where the bill ends.


When an EKS bill spikes, where do you look first: compute or the network? And what's the biggest data-transfer surprise you've traced back? I'll start: an internal service mesh spread across 3 AZs "for HA" that was paying cross-AZ on every single hop, all day, for traffic that never needed to leave the zone.