Hi!

Thanks for the feedback!

The issue was VMs with big "imbalances" of a lot of RAM and little CPU and vice versa.
What metrics does the CRS take into account? Both memory and CPU? How does it weigh between those (or does it do any weighting at all)?

The load balancer takes both memory and CPU in account. Ad weighing, see the next paragraphs.

There are two modes the CRS can run in, TOPSIS and "Brute Force" with the latter one being the default. What are the differences in practice between the two modes? Are there scenarios where you should choose one over the other?

The load balancer can score the balancing migrations by either one of these methods.

The brute-force method (as in 'greedily find the best balancing migration') does currently weigh average CPU load and memory usage as equal. Though it might be weighed differently in the future as well, it is a well-balanced starting point as both resources (CPU and memory) can cause pressure and therefore degradation in resource utilization over time.

The TOPSIS method does weigh memory as more important than the CPU load: a 5:1 ratio for average CPU/memory usage and a 10:5 ratio for CPU/memory peaks to signify that memory is a truly limited resource while high CPU pressure does 'only' degrade the processing time. This is already the method we used for scoring nodes to start new HA resources (if rebalance-on-start is enabled).

The TOPSIS method might be helpful for more memory-bound applications, though as for many applications an equal balance for both resources is useful as well as cpu pressure often being the more common problem, the brute force method is the current default.

Hope this helps!

PS: There is a patch series in review, which overhauls the CRS section itself and adds documentation for the new load balancing system here [0]. External feedback on these patches are also very welcome if things could be made clearer or certain things should be elaborated on more!

[0] https://lore.proxmox.com/pve-devel/20260415091635.162224-20-d.kral@proxmox.com/

This is on a PVE 9.1.9 (enterprise repo) server. `pve-ha-manager` is installed as version 5.1.3 though? The patch is for 5.2.0+

Yes, some recent security fixes for the pve-manager package forced us to ship the package, which already includes the load balancer options in the web interface, earlier to all repositories, while we're still waiting to move pve-ha-manager 5.2.0 and pve-cluster 9.1.2 to the enterprise repositories as well.

See this post [0] for a little more information.

[0] https://forum.proxmox.com/threads/183143/#post-850798

As for the modes, I think we will just need to try the modes and see what happens. TOPSIS sounds best for our production clusters as they are usually memory limited.

One thing I would have as feedback is that it is way more "expensive" to do live migrations of VMs with vGPU resources as a migration halts them for extended periods of time (8gb takes ~6s for us, 24gb ~20s etc.). So ideally we would move those last as long as there are better options for shuffling VMs around.
For now I guess I can create an affinity group to "pin" them to one host with higher priority to not get them to balance except for in the case of host failures.

Thanks for the feedback!

We also thought about including more terms to the "cost function" of a migration, though we mainly focused on the core feature first.

The current load balancing implementation focuses on reducing the imbalance between the nodes as much as possible. The larger the imbalance, usually the more 'expensive' the balancing migrations will be (in terms of memory, etc. and therefore migration time) to minimize the total amount of migrations that are needed to rebalance the HA resources within the cluster. Without knowing the rest of the cluster, in these situations it might be the best option to move these 'heavy' HA resources first. Though it might be expensive if this becomes a transient state, where these HA resources are moved quite often. Does this occur for you?

One more question about the calculations done: I know the dynamic CRS only migrates resources managed by the HA Manager. But how does it calculate the host resource usage? Does it only take the ha managed resources into account or the "general" host CPU/Memory usage which also contains non-ha VMs or other processes like Ceph?

I might have missed it, but can you explain the difference between static and dynamic load-balancing regarding the new rebalancing feature? I cant find it in the mailing-list or current docs?

I'll answer both questions in one, because they're quite related: The scheduling mode (basic, static-load, dynamic-load) specifies which measure is used to compare nodes and HA resources against each other:

• For the basic mode, it's simply how many running guests are on the nodes.
• For the static-load mode, it's how much CPU and memory the running guests have allocated statically (i.e., the quotas set in the guest configs with cores, sockets, memory, etc.).
• For the dynamic-load mode, it additionally accounts for the actual current CPU and memory usage of each running guest and the node itself.

This scheduling mode is used for the load balancer, as well as other HA-related scheduling points, such as initial placement rebalancing (while starting HA resources), HA resource recovery, maintenance mode, etc.

The automatic rebalancing system can only be used in conjunction with the static-load and dynamic-load mode. So with the static-load mode, the system will actively rebalance according to the quotas set in the config (which might also change while guests are running if CPU/memory hotplugging is enabled). While with the dynamic-load mode, the system actively rebalances according to the current actual usage.

The rebalancing migrations are chosen by picking from the pool of migratable HA resources with their possible targets and predicting which will lower the imbalance the most in the forseeable future. In dynamic-load mode, the node usages currently also include the CPU/memory usages for Ceph, ZFS, etc., to accomodate these usage requirements as well.

please could you explain how dynamic load CRS imbalance is calculated?
Value is displayed in "Datacenter / HA / Status / master" when dynamic CRS is active.

Hi!

The cluster imbalance is derived from the standard deviation and mean of the individual node loads and is normalized to a 0-100% range. [0]

In more detail, each node's current cpu and memory usage is normalized with its own capacity (i.e. cpu_load = current_cpu / max_cpu and memory_load = current_memory / max_memory), where the node's current load is then a equal-weighted average of that (i.e. (cpu_load + memory_load) / 2.

The cluster imbalance is then the normalized coefficient of variation, which is the standard deviation of the current node loads divided by the average mean of the current node loads. To normalize the coefficient of variation, this is divided once more by the square root of the cluster size minus 1 (i.e. (std / mean) / sqrt(n - 1)).

[0] https://pve.proxmox.com/pve-docs/chapter-ha-manager.html#_crs_load_balancer

No worries, and the manual can always be improved though I didn't put in the technical explanation from above as it felt too specific. If it's useful there as well, I can certainly find a way to put it there without being too distracting from the main functionality

If it's useful there as well, I can certainly find a way to put it there without being too distracting from the main functionality

This is absolutely needed and has to be very precise. I.e. which exact metric is used for both CPU and Memory "loads": CPU load vs CPU usage vs CPU pressure, Mem usage vs Mem pressure.

Currently testing CRS in lab and it seems to allow a node to be 100% CPU usage while another is sitting idle with no VMs. Understanding the details will help predict the behavior, figure out corner cases and report improvements.