Somewhere in a cluster I look after, an autoscaler had provisioned a full compute node to run two small controller pods. Nothing else would ever schedule onto it. Nothing else could.
The cause was two lines of YAML that had been copied, correctly, from a cluster where they cost nothing.
Two lines, copied correctly
We run the same ingress controller on four clusters. Two of them run a meaningful amount of
arm64 workload and keep arm64 capacity around permanently. On those two, someone had pinned
the controller to arm64 — a nodeSelector on the architecture label, plus the matching
toleration for the arm64 taint. Sensible: the capacity is there, it’s cheaper per unit, the
pods fit in the gaps.
When the controller was rolled out to the other two clusters, the values file was copied. Of course it was. That’s how you get consistency, and the diff between the four files looked right in review precisely because the lines matched.
But those two clusters have no arm64 workloads. The node pool exists — the platform configuration is uniform — and until that moment nothing had ever asked it for anything.
What a scheduling constraint means to an autoscaler
Here’s the part that’s easy to say and easy to forget under review.
To the Kubernetes scheduler, a nodeSelector is a filter: it narrows the set of existing
nodes a pod may land on. If none match, the pod stays Pending. That’s the mental model most
people carry, and in a fixed-size cluster it’s the whole story — you notice immediately,
because the pod doesn’t run.
To a provisioning autoscaler, the same nodeSelector is an order. A pending pod with
unsatisfiable constraints is not an error condition; it’s a work item. The autoscaler reads
the constraint, computes the cheapest instance that satisfies it, and buys one.
So the failure mode inverts. In a fixed cluster, an over-constrained pod is loudly broken. In an autoscaled cluster, it is silently expensive. Everything comes up green. The deployment reports Available, the controller serves traffic, no alert fires. The only evidence is a node in the list with one workload on it and an instance-hours line on the bill.
In our case one cluster got a general-purpose arm64 node and the other got a memory-optimized one — roughly four vCPU and thirty-odd gigabytes of RAM — to host two pods whose combined requests were a rounding error against that.
Nothing needed it
The controller image publishes both linux/amd64 and linux/arm64 in its manifest list.
There is no native extension, no compiled sidecar, no architecture-specific behaviour. The
pin wasn’t buying compatibility or performance; it was an artifact of where the manifest was
first written.
Deleting the two lines rescheduled both replicas onto existing nodes, and the autoscaler deprovisioned the now-empty node in each cluster. That’s the whole fix. No image change, no version bump, no migration.
Why review didn’t catch it
This is the bit worth generalizing, because the review was not lazy.
The correctness of a scheduling constraint is a property of the cluster, not of the
manifest. You cannot evaluate nodeSelector: {kubernetes.io/arch: arm64} by reading it.
The same characters are free on one cluster and cost you a node on another, and the file
gives you no way to tell which. Review compares the change against its neighbours, the
neighbours match, and matching is exactly the wrong signal here.
The related conflation, which I’ve now watched several people make including myself:
- A toleration is permission. It says “I’m allowed onto nodes with this taint.” Adding one to a pod that never encounters the taint changes nothing and costs nothing.
- A nodeSelector (or a required node affinity) is demand. It says “I will not run anywhere else.” Adding one to a cluster that can’t satisfy it obliges someone to go get capacity.
They’re usually written adjacent to each other, in the same block, by the same person, in a single copied hunk. One of them is inert and one of them spends money.
The same asymmetry applies to anything else that narrows placement: zone pins, dedicated
node-pool labels, GPU or accelerator selectors, instance-family requirements, and
topologySpreadConstraints with whenUnsatisfiable: DoNotSchedule — which can force a new
node purely to satisfy a spread rule.
The question to ask
When copying a workload’s manifest to another cluster, go through every placement constraint and ask one thing:
Does this cluster already run something that satisfies this? If not, I am not writing a preference — I am placing an order.
If the answer is no and the workload doesn’t actually require the constraint, delete it. If the workload does require it, that’s fine, but now you know you’re adding a node and you can size the decision honestly.
And after you remove one: check that the node actually goes away. Consolidation isn’t instant, and anything else that drifted onto that node — or any pod on it carrying a “don’t disrupt me” annotation — will keep it alive and keep you paying for a fix you think you already landed.
Takeaways
- In an autoscaled cluster, a scheduling constraint is a purchase order. The scheduler filters; the provisioner procures. Same YAML, opposite consequences.
- Over-constrained pods fail loudly on fixed clusters and quietly on elastic ones. If your intuition was formed on fixed clusters, it is now inverted and nothing will tell you.
- Tolerations are free, selectors are not. They travel together in copied blocks. Treat them differently.
- A constraint can’t be reviewed in isolation from the cluster it lands on. “It matches the other environments” is the specific reasoning that produces this bug.
- Copying manifests between clusters needs a per-constraint pass, not a per-file comparison — the file being identical is the thing that hides it.
- Verify the node disappears after you unpin. Otherwise you’ve paid for the fix twice: once in the incident and once in the node that never left.