How to Better Load Balance OpenShift Nodes

An example via spreading out DB2 Pods that do not require specific or especially configured Nodes to run

Background

In dense enterprise environments — such as those supporting the IBM Maximo Application Suite (MAS) — it is common to run multiple database instances or data warehouses concurrently. By default, the standard Kubernetes scheduler evaluates pods on an isolated, local namespace basis.

When several DB2 instances (e.g., p01-db2oltp, p01-db2dwh, p02-db2oltp) are installed into separate namespaces, the scheduler treats them as completely unrelated workloads. Consequently, it stacks them onto the same worker nodes based on image caching or historical weight — resulting in heavy resource imbalances, memory hotspots, and uneven node utilization while leaving other worker hosts virtually idle.

To combat this, we implement a two-pronged strategy:

• Reactive: the Red Hat Kube Descheduler Operator to gently evict stacked workloads.
• Proactive: declarative cross-namespace Pod Anti-Affinity constraints within the Db2uCluster Custom Resources (CR).

We also address an infrastructure edge case where database engines lose their communication flags during automated pod migrations, ensuring persistent application availability.

Pre-requisites

Install the Kube Descheduler Operator from the official Red Hat registry (registry.redhat.io/kube-descheduler-operator). Select the stable Update Channel within a dedicated namespace — by default openshift-kube-descheduler-operator — using Manual updates to maintain control over scheduling cycles.

Step 1 — Create the KubeDescheduler Custom Resource

Once the operator is active, submit the following YAML to create the live descheduler instance. This manifest enables the core balancing profiles required to remediate resource stacking:

Step 2 — Declare Anti-Affinity and Fix the DB2COMM Registry

Modifying the descheduler alone is insufficient — upon evicting a pod, the default scheduler will re-schedule it onto the same node unless an explicit rule prevents it. Because zone-locked ReadWriteOnce block volumes (such as ibmc-vpc-block-10iops-tier) are permanently tied to a single availability zone, we use a Soft (Preferred) anti-affinity layout rather than a hard constraint. This avoids leaving databases in a permanent Pending state when a zone runs low on alternate worker nodes.

Each Db2uCluster CR manifest must be amended under the spec: block as follows:

Step 3 — Bulk Automation with a Patch Script

To safely apply this modification across all active Db2uCluster custom resources in bulk — without manual text editing per namespace — use the following structured Bash loop from an administration terminal. It queries OpenShift for every instance and applies a precise, non-disruptive merge patch:

The operator immediately picks up the manifest update. Thanks to the DB2U configuration management engine, it dynamically injects the registry parameters into the container environment profiles on the fly (visible via log lines like Waiting for configmaps... change detected) without requiring an immediate forced pod restart.

Step 4 — Verification and Cluster Status Validation

Track progression across the cluster and wait until all instances complete their background reconciliation loops and return to a healthy state:

Next, connect directly to the underlying engine pods and confirm databases are activated and accepting active client connections from the Maximo application layer:

The command should return a healthy runtime response similar to:

The Applications connected currently value greater than zero certifies that external secure connection pooling over port 50001 has successfully re-stabilized.

Final Topology and Scheduling Results

To extract the resulting placement mapping, run a wide pod inspection filtered by DB2 engine pod names:

The final state demonstrates an ideal, high-performance pod separation across the infrastructure:

Pairing a transactional database (OLTP) with an analytical warehouse (DWH) on the same node is a meaningful architectural win. Analytical engines spike heavily on long, I/O-intensive disk sweeps, while OLTP workloads demand steady, low-latency CPU and memory block operations — their resource profiles are complementary, not competing.

With the descheduler and anti-affinity architecture permanently declared, any future node failure or cluster maintenance event will automatically maintain this balanced, high-performance topology.