Kubernetes not distributing pods evenly on all the nodes of the cluster after adding nodes in the cluster.
- Platform9 Managed Kubernetes - All Versions
During the scoring of the nodes for scheduling, the number of pods already running on the nodes is not considered. The pod scheduling in Kubernetes depends on the node scoring done by the scheduler which does not account for the number of pods on the node. This is why the nodes can have an uneven number of pods and it is by design.
- The scheduler is assigning the score to the nodes based on the resources available. You can see the resource utilization running the `kubectl top nodes` command.
- We noticed some of the nodes were having higher resource utilization and were having a higher number of pods on them and it was due to the fact that that these nodes were older and added to the cluster prior to the node it was being compared with. Below is one such example
Number of pods
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
XX.XX.XX.XX 679m 2% 25026Mi 4%
XX.XX.XX.XX 865m 3% 57925Mi 11%
XX.XX.XX.XX Ready worker 100d v1.XX.X
XX.XX.XX.XX Ready worker 599d v1.XX.X
- The pods are not rescheduled when a new node is added to the cluster. So the pods already scheduled on a node will not get drained in case a new node is added and scoring will be done for all the nodes according to resource utilization see in kubectl top command.
- In case we have to change the scheduling behavior and limit the pods scheduled to a node, it will have to be done during the filtering phase with the help of node affinity, taints, or max pods per node. Once the nodes are filtered and sent to the scoring phase it will all depend on the scheduler logic.