Skip to content

fix: Reevaluate and restart the run when there is a connection - #135

Open
gyptazy wants to merge 4 commits into
mainfrom
fix/134-reevaluate-on-api-error
Open

gyptazy wants to merge 4 commits into
mainfrom
fix/134-reevaluate-on-api-error

Conversation

@gyptazy

@gyptazy gyptazy commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator

fix: Reevaluate and restart the run when there is a connection related API issue.

Fixes: #134

@gyptazy gyptazy self-assigned this Sep 16, 2026
@gyptazy gyptazy added the bug Something isn't working label Sep 16, 2026
@gyptazy
gyptazy force-pushed the fix/134-reevaluate-on-api-error branch from 3d2e2bf to c334e30 Compare September 16, 2026 13:10
@gyptazy

gyptazy commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator Author

Testrun by killing a single node during runtime:

During API connection handling

Sep 16 14:54:10 pve01 systemd[1]: Stopping proxlb.service - ProxLB - A loadbalancer for Proxmox clusters...
Sep 16 14:54:10 pve01 systemd[1]: Stopped proxlb.service - ProxLB - A loadbalancer for Proxmox clusters.
Sep 16 14:54:10 pve01 systemd[1]: Started proxlb.service - ProxLB - A loadbalancer for Proxmox clusters.
Sep 16 14:54:12 pve01 python3[2633]: 2026-09-16 14:54:12,541 - ProxLB - INFO - Using config path: /etc/proxlb/proxlb.yaml
Sep 16 14:54:12 pve01 python3[2633]: 2026-09-16 14:54:12,547 - ProxLB - DEBUG - Set to debug level
Sep 16 14:54:12 pve01 python3[2633]: 2026-09-16 14:54:12,547 - ProxLB - DEBUG - Starting: get_service_delay.
Sep 16 14:54:12 pve01 python3[2633]: 2026-09-16 14:54:12,547 - ProxLB - DEBUG - Service delay not active. Proceeding without delay.
Sep 16 14:54:12 pve01 python3[2633]: 2026-09-16 14:54:12,547 - ProxLB - DEBUG - Finished: get_service_delay.
Sep 16 14:54:12 pve01 python3[2633]: 2026-09-16 14:54:12,547 - ProxLB - DEBUG - Starting: ProxmoxApi initialization.
Sep 16 14:54:12 pve01 python3[2633]: 2026-09-16 14:54:12,547 - ProxLB - DEBUG - Starting: api_connect.
Sep 16 14:54:12 pve01 python3[2633]: 2026-09-16 14:54:12,547 - ProxLB - DEBUG - Starting: validate_config.
Sep 16 14:54:12 pve01 python3[2633]: 2026-09-16 14:54:12,547 - ProxLB - DEBUG - Finished: validate_config.
Sep 16 14:54:12 pve01 python3[2633]: 2026-09-16 14:54:12,547 - ProxLB - DEBUG - Starting: api_connect_get_hosts.
Sep 16 14:54:12 pve01 python3[2633]: 2026-09-16 14:54:12,548 - ProxLB - DEBUG - Starting: test_api_proxmox_host.
Sep 16 14:54:12 pve01 python3[2633]: 2026-09-16 14:54:12,548 - ProxLB - DEBUG - Starting: get_host_port_from_string.
Sep 16 14:54:12 pve01 python3[2633]: 2026-09-16 14:54:12,548 - ProxLB - DEBUG - 10.10.10.222 has IPv4.
Sep 16 14:54:12 pve01 python3[2633]: 2026-09-16 14:54:12,548 - ProxLB - DEBUG - Starting: test_api_proxmox_host_ipv4.
Sep 16 14:54:13 pve01 python3[2633]: 2026-09-16 14:54:13,549 - ProxLB - WARNING - Host 10.10.10.222 is unreachable on IPv4 for tcp/8006 (errno 11).
Sep 16 14:54:13 pve01 python3[2633]: 2026-09-16 14:54:13,549 - ProxLB - DEBUG - Finished: test_api_proxmox_host_ipv4.
Sep 16 14:54:13 pve01 python3[2633]: 2026-09-16 14:54:13,549 - ProxLB - DEBUG - Finished: test_api_proxmox_host (unreachable).
Sep 16 14:54:13 pve01 python3[2633]: 2026-09-16 14:54:13,549 - ProxLB - WARNING - Attempt 1/1 failed for host 10.10.10.222. Retrying in 1 seconds...
Sep 16 14:54:14 pve01 python3[2633]: 2026-09-16 14:54:14,549 - ProxLB - DEBUG - Starting: test_api_proxmox_host.
Sep 16 14:54:14 pve01 python3[2633]: 2026-09-16 14:54:14,550 - ProxLB - DEBUG - Starting: get_host_port_from_string.
Sep 16 14:54:14 pve01 python3[2633]: 2026-09-16 14:54:14,550 - ProxLB - DEBUG - 10.10.10.223 has IPv4.
Sep 16 14:54:14 pve01 python3[2633]: 2026-09-16 14:54:14,550 - ProxLB - DEBUG - Starting: test_api_proxmox_host_ipv4.
Sep 16 14:54:14 pve01 python3[2633]: 2026-09-16 14:54:14,550 - ProxLB - DEBUG - Host 10.10.10.223 is reachable on IPv4 for tcp/8006.
Sep 16 14:54:14 pve01 python3[2633]: 2026-09-16 14:54:14,550 - ProxLB - DEBUG - Finished: test_api_proxmox_host_ipv4.
Sep 16 14:54:14 pve01 python3[2633]: 2026-09-16 14:54:14,550 - ProxLB - WARNING - SSL certificate validation to host 10.10.10.223:8006 is deactivated.
Sep 16 14:54:14 pve01 python3[2633]: 2026-09-16 14:54:14,644 - ProxLB - DEBUG - Using username/password authentication.
Sep 16 14:54:14 pve01 python3[2633]: 2026-09-16 14:54:14,644 - ProxLB - INFO - API connection to host 10.10.10.223:8006 succeeded.

During runtime where a nods becomes unavailable

Sep 16 15:15:13 pve01 systemd[1]: Stopping proxlb.service - ProxLB - A loadbalancer for Proxmox clusters...
Sep 16 15:15:13 pve01 systemd[1]: Stopped proxlb.service - ProxLB - A loadbalancer for Proxmox clusters.
Sep 16 15:15:13 pve01 systemd[1]: Started proxlb.service - ProxLB - A loadbalancer for Proxmox clusters.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,229 - ProxLB - INFO - Using config path: /etc/proxlb/proxlb.yaml
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,235 - ProxLB - DEBUG - Set to debug level
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,235 - ProxLB - DEBUG - Starting: get_service_delay.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,235 - ProxLB - DEBUG - Service delay not active. Proceeding without delay.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,235 - ProxLB - DEBUG - Finished: get_service_delay.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,235 - ProxLB - DEBUG - Starting: ProxmoxApi initialization.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,235 - ProxLB - DEBUG - Starting: api_connect.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,235 - ProxLB - DEBUG - Starting: validate_config.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,235 - ProxLB - DEBUG - Finished: validate_config.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,235 - ProxLB - DEBUG - Starting: api_connect_get_hosts.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,235 - ProxLB - DEBUG - Starting: test_api_proxmox_host.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,235 - ProxLB - DEBUG - Starting: get_host_port_from_string.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,235 - ProxLB - DEBUG - 10.10.10.222 has IPv4.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,235 - ProxLB - DEBUG - Starting: test_api_proxmox_host_ipv4.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,235 - ProxLB - DEBUG - Host 10.10.10.222 is reachable on IPv4 for tcp/8006.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,236 - ProxLB - DEBUG - Finished: test_api_proxmox_host_ipv4.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,236 - ProxLB - DEBUG - Starting: test_api_proxmox_host.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,236 - ProxLB - DEBUG - Starting: get_host_port_from_string.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,236 - ProxLB - DEBUG - 10.10.10.223 has IPv4.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,236 - ProxLB - DEBUG - Starting: test_api_proxmox_host_ipv4.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,236 - ProxLB - DEBUG - Host 10.10.10.223 is reachable on IPv4 for tcp/8006.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,236 - ProxLB - DEBUG - Finished: test_api_proxmox_host_ipv4.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,236 - ProxLB - WARNING - SSL certificate validation to host 10.10.10.223:8006 is deactivated.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,324 - ProxLB - DEBUG - Using username/password authentication.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,325 - ProxLB - INFO - API connection to host 10.10.10.223:8006 succeeded.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,325 - ProxLB - DEBUG - Finished: api_connect.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,325 - ProxLB - DEBUG - Starting: test_api_user_permissions.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,333 - ProxLB - DEBUG - Finished: test_api_user_permissions.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,333 - ProxLB - DEBUG - Finished: ProxmoxApi initialization.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,334 - ProxLB - DEBUG - Starting: is_node_ha_manager.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,334 - ProxLB - DEBUG - Starting: get_ha_manager.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,339 - ProxLB - DEBUG - HA manager node: {'timestamp': 1789564512, 'node_status': {'pve01': 'online', 'pve02': 'online', 'pve03': 'online'}, 'service_status': {'ct:100': {'state': 'started', 'uid': '3XAlCXXxxbXy1PfXJ6fe7w', 'running': 1, 'node': 'pve03'}}, 'master_node': 'pve01'}
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,339 - ProxLB - DEBUG - Finished: get_ha_manager.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,339 - ProxLB - DEBUG - Cluster Master is: pve01 | We are: pve01) | This node is the HA manager.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,339 - ProxLB - DEBUG - Finished: is_node_ha_manager.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,339 - ProxLB - DEBUG - This node is the HA manager. Continuing.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,339 - ProxLB - DEBUG - Starting: apply_maintenance_nodes_schedule.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,339 - ProxLB - DEBUG - Runtime maintenance nodes: ['virt66.example.com']
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,339 - ProxLB - DEBUG - Finished: apply_maintenance_nodes_schedule.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,339 - ProxLB - DEBUG - Starting: get_nodes.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,345 - ProxLB - DEBUG - Starting: set_node_ignore.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,345 - ProxLB - DEBUG - Finished: set_node_ignore.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,345 - ProxLB - DEBUG - Starting: get_node_rrd_datasets.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,446 - ProxLB - DEBUG - Getting average RRD data from node: pve02.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,553 - ProxLB - DEBUG - Getting spike RRD data from node: pve02.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,564 - ProxLB - DEBUG - Finished: get_node_rrd_datasets.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,564 - ProxLB - DEBUG - Starting: get_node_pve_version.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,665 - ProxLB - DEBUG - Trying to get PVE version for node: pve02.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - Got version 9.1.1 for node pve02.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - Finished: get_node_pve_version.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - RRD data (spike: False) for pressurecpusome from node: pve02: 0.00037878787878787835
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - RRD data (spike: True) for pressurecpusome from node: pve02: 0.0227272727272727
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - RRD data (spike: False) for pressureiosome from node: pve02: 0.0
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - RRD data (spike: False) for pressureiofull from node: pve02: 0.0
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - RRD data (spike: True) for pressureiosome from node: pve02: 0
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - RRD data (spike: True) for pressureiofull from node: pve02: 0
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - Starting: apply_resource_reservation
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - CRITICAL - Configured default reservation for node pve02 of type memory with 4 GB is higher than available resource value 3.82 GB. Not applying...
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - RRD data (spike: False) for pressurememorysome from node: pve02: 0.0
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - RRD data (spike: False) for pressurememoryfull from node: pve02: 0.0
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - RRD data (spike: True) for pressurememorysome from node: pve02: 0
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - RRD data (spike: True) for pressurememoryfull from node: pve02: 0
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - Starting: set_node_maintenance.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,679 - ProxLB - DEBUG - Node: pve02 is not in maintenance mode by ProxLB config.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,682 - ProxLB - DEBUG - Finished: set_node_maintenance.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,682 - ProxLB - DEBUG - Starting: set_node_ignore.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,683 - ProxLB - DEBUG - Finished: set_node_ignore.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,683 - ProxLB - DEBUG - Starting: get_node_rrd_datasets.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,783 - ProxLB - DEBUG - Getting average RRD data from node: pve01.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,891 - ProxLB - DEBUG - Getting spike RRD data from node: pve01.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,903 - ProxLB - DEBUG - Finished: get_node_rrd_datasets.
Sep 16 15:15:15 pve01 python3[7965]: 2026-09-16 15:15:15,904 - ProxLB - DEBUG - Starting: get_node_pve_version.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,004 - ProxLB - DEBUG - Trying to get PVE version for node: pve01.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - Got version 9.1.1 for node pve01.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - Finished: get_node_pve_version.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - RRD data (spike: False) for pressurecpusome from node: pve01: 0.11422222222222235
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - RRD data (spike: True) for pressurecpusome from node: pve01: 4.44166666666667
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - RRD data (spike: False) for pressureiosome from node: pve01: 0.02106944444444445
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - RRD data (spike: False) for pressureiofull from node: pve01: 0.021055555555555557
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - RRD data (spike: True) for pressureiosome from node: pve01: 0.250833333333333
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - RRD data (spike: True) for pressureiofull from node: pve01: 0.250833333333333
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - Starting: apply_resource_reservation
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - CRITICAL - Configured default reservation for node pve01 of type memory with 4 GB is higher than available resource value 3.82 GB. Not applying...
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - RRD data (spike: False) for pressurememorysome from node: pve01: 0.001972222222222217
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - RRD data (spike: False) for pressurememoryfull from node: pve01: 0.001972222222222217
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - RRD data (spike: True) for pressurememorysome from node: pve01: 0.118333333333333
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - RRD data (spike: True) for pressurememoryfull from node: pve01: 0.118333333333333
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,018 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,019 - ProxLB - DEBUG - Starting: set_node_maintenance.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,019 - ProxLB - DEBUG - Node: pve01 is not in maintenance mode by ProxLB config.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,022 - ProxLB - DEBUG - Finished: set_node_maintenance.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,022 - ProxLB - DEBUG - Starting: set_node_ignore.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,022 - ProxLB - DEBUG - Finished: set_node_ignore.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,022 - ProxLB - DEBUG - Starting: get_node_rrd_datasets.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,122 - ProxLB - DEBUG - Getting average RRD data from node: pve03.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,231 - ProxLB - DEBUG - Getting spike RRD data from node: pve03.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,243 - ProxLB - DEBUG - Finished: get_node_rrd_datasets.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,243 - ProxLB - DEBUG - Starting: get_node_pve_version.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,344 - ProxLB - DEBUG - Trying to get PVE version for node: pve03.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - Got version 9.1.1 for node pve03.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - Finished: get_node_pve_version.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - RRD data (spike: False) for pressurecpusome from node: pve03: 0.020441666666666674
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - RRD data (spike: True) for pressurecpusome from node: pve03: 0.803
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - RRD data (spike: False) for pressureiosome from node: pve03: 0.02812499999999994
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - RRD data (spike: False) for pressureiofull from node: pve03: 0.02697499999999995
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - RRD data (spike: True) for pressureiosome from node: pve03: 1.08933333333333
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - RRD data (spike: True) for pressureiofull from node: pve03: 1.08733333333333
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - Starting: apply_resource_reservation
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - CRITICAL - Configured default reservation for node pve03 of type memory with 4 GB is higher than available resource value 3.82 GB. Not applying...
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - RRD data (spike: False) for pressurememorysome from node: pve03: 0.0
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - RRD data (spike: False) for pressurememoryfull from node: pve03: 0.0
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - RRD data (spike: True) for pressurememorysome from node: pve03: 0
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - Starting: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - RRD data (spike: True) for pressurememoryfull from node: pve03: 0
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,352 - ProxLB - DEBUG - Finished: get_node_rrd_value.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,353 - ProxLB - DEBUG - Starting: set_node_maintenance.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,353 - ProxLB - DEBUG - Node: pve03 is not in maintenance mode by ProxLB config.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,356 - ProxLB - DEBUG - Finished: set_node_maintenance.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,356 - ProxLB - DEBUG - Node metrics collected: {'pve02': Node(name='pve02', pve_version='9.1.1', pressure_hot=False, maintenance=False, cpu=Metric(total=2, assigned=0, used=0.0287292817679558, free=1.9712707182320441, assigned_percent=0.0, free_percent=98.5635359116022, used_percent=1.43646408839779, pressure_some_percent=0.00037878787878787835, pressure_full_percent=0.0, pressure_some_spikes_percent=0.0227272727272727, pressure_full_spikes_percent=0.0, pressure_hot=False), disk=Metric(total=14513811456, assigned=0, used=4101591040.0, free=10412220416.0, assigned_percent=0.0, free_percent=71.74008321360407, used_percent=28.259916786395934, pressure_some_percent=0.0, pressure_full_percent=0.0, pressure_some_spikes_percent=0.0, pressure_full_spikes_percent=0.0, pressure_hot=False), memory=Metric(total=4106768384, assigned=0, used=1697337344.0, free=2409431040.0, assigned_percent=0.0, free_percent=58.66975720829938, used_percent=41.33024279170062, pressure_some_percent=0.0, pressure_full_percent=0.0, pressure_some_spikes_percent=0.0, pressure_full_spikes_percent=0.0, pressure_hot=False)), 'pve01': Node(name='pve01', pve_version='9.1.1', pressure_hot=False, maintenance=False, cpu=Metric(total=2, assigned=0, used=0.0328685258964144, free=1.9671314741035857, assigned_percent=0.0, free_percent=98.35657370517929, used_percent=1.64342629482072, pressure_some_percent=0.11422222222222235, pressure_full_percent=0.0, pressure_some_spikes_percent=4.44166666666667, pressure_full_spikes_percent=0.0, pressure_hot=False), disk=Metric(total=14513811456, assigned=0, used=4659023872.0, free=9854787584.0, assigned_percent=0.0, free_percent=67.89937718204295, used_percent=32.100622817957046, pressure_some_percent=0.02106944444444445, pressure_full_percent=0.021055555555555557, pressure_some_spikes_percent=0.250833333333333, pressure_full_spikes_percent=0.250833333333333, pressure_hot=False), memory=Metric(total=4106764288, assigned=0, used=1793499136.0, free=2313265152.0, assigned_percent=0.0, free_percent=56.32816957036907, used_percent=43.67183042963093, pressure_some_percent=0.001972222222222217, pressure_full_percent=0.001972222222222217, pressure_some_spikes_percent=0.118333333333333, pressure_full_spikes_percent=0.118333333333333, pressure_hot=False)), 'pve03': Node(name='pve03', pve_version='9.1.1', pressure_hot=False, maintenance=False, cpu=Metric(total=4, assigned=0, used=0.3213131061924896, free=3.6786868938075106, assigned_percent=0.0, free_percent=91.96717234518776, used_percent=8.03282765481224, pressure_some_percent=0.020441666666666674, pressure_full_percent=0.0, pressure_some_spikes_percent=0.803, pressure_full_spikes_percent=0.0, pressure_hot=False), disk=Metric(total=14513811456, assigned=0, used=4102033408.0, free=10411778048.0, assigned_percent=0.0, free_percent=71.73703530298913, used_percent=28.262964697010872, pressure_some_percent=0.02812499999999994, pressure_full_percent=0.02697499999999995, pressure_some_spikes_percent=1.08933333333333, pressure_full_spikes_percent=1.08733333333333, pressure_hot=False), memory=Metric(total=4106063872, assigned=0, used=1757634560.0, free=2348429312.0, assigned_percent=0.0, free_percent=57.1941739146916, used_percent=42.8058260853084, pressure_some_percent=0.0, pressure_full_percent=0.0, pressure_some_spikes_percent=0.0, pressure_full_spikes_percent=0.0, pressure_hot=False))}
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,357 - ProxLB - DEBUG - Finished: get_nodes.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,357 - ProxLB - DEBUG - Starting: validate_any_non_pve9_node.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,357 - ProxLB - DEBUG - Finished: validate_any_non_pve9_node. Result: False
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,357 - ProxLB - DEBUG - Starting: get_pools.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,360 - ProxLB - DEBUG - Finished: get_pools.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,360 - ProxLB - DEBUG - Starting: get_ha_rules.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,360 - ProxLB - DEBUG - Cluster running Proxmox VE 9 or newer, proceeding with HA rule retrieval.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,364 - ProxLB - DEBUG - Finished: ha_rules.
Sep 16 15:15:16 pve01 python3[7965]: 2026-09-16 15:15:16,364 - ProxLB - DEBUG - Starting: get_guests.

(!!!! NEW ACTION !!!!)

Sep 16 15:15:26 pve01 python3[7965]: 2026-09-16 15:15:26,375 - ProxLB - CRITICAL - ProxLB: A Proxmox API call failed: HTTPSConnectionPool(host='10.10.10.223', port=8006): Read timed out. (read timeout=10). Reconnecting to the Proxmox API and restarting balancing run from scratch.
Sep 16 15:15:27 pve01 python3[7965]: 2026-09-16 15:15:27,375 - ProxLB - DEBUG - Starting: get_config.
Sep 16 15:15:27 pve01 python3[7965]: 2026-09-16 15:15:27,375 - ProxLB - INFO - Using config path: /etc/proxlb/proxlb.yaml
Sep 16 15:15:27 pve01 python3[7965]: 2026-09-16 15:15:27,381 - ProxLB - DEBUG - Set to debug level
Sep 16 15:15:27 pve01 python3[7965]: 2026-09-16 15:15:27,381 - ProxLB - DEBUG - Starting: ProxmoxApi initialization.
Sep 16 15:15:27 pve01 python3[7965]: 2026-09-16 15:15:27,381 - ProxLB - DEBUG - Starting: api_connect.
Sep 16 15:15:27 pve01 python3[7965]: 2026-09-16 15:15:27,381 - ProxLB - DEBUG - Starting: validate_config.
Sep 16 15:15:27 pve01 python3[7965]: 2026-09-16 15:15:27,381 - ProxLB - DEBUG - Finished: validate_config.
Sep 16 15:15:27 pve01 python3[7965]: 2026-09-16 15:15:27,381 - ProxLB - DEBUG - Starting: api_connect_get_hosts.
Sep 16 15:15:27 pve01 python3[7965]: 2026-09-16 15:15:27,381 - ProxLB - DEBUG - Starting: test_api_proxmox_host.
Sep 16 15:15:27 pve01 python3[7965]: 2026-09-16 15:15:27,381 - ProxLB - DEBUG - Starting: get_host_port_from_string.
Sep 16 15:15:27 pve01 python3[7965]: 2026-09-16 15:15:27,381 - ProxLB - DEBUG - 10.10.10.222 has IPv4.
Sep 16 15:15:27 pve01 python3[7965]: 2026-09-16 15:15:27,381 - ProxLB - DEBUG - Starting: test_api_proxmox_host_ipv4.
Sep 16 15:15:28 pve01 python3[7965]: 2026-09-16 15:15:28,383 - ProxLB - WARNING - Host 10.10.10.222 is unreachable on IPv4 for tcp/8006 (errno 11).
Sep 16 15:15:28 pve01 python3[7965]: 2026-09-16 15:15:28,383 - ProxLB - DEBUG - Finished: test_api_proxmox_host_ipv4.
Sep 16 15:15:28 pve01 python3[7965]: 2026-09-16 15:15:28,383 - ProxLB - DEBUG - Finished: test_api_proxmox_host (unreachable).
Sep 16 15:15:28 pve01 python3[7965]: 2026-09-16 15:15:28,383 - ProxLB - WARNING - Attempt 1/1 failed for host 10.10.10.222. Retrying in 1 seconds...
Sep 16 15:15:29 pve01 python3[7965]: 2026-09-16 15:15:29,383 - ProxLB - DEBUG - Starting: test_api_proxmox_host.
Sep 16 15:15:29 pve01 python3[7965]: 2026-09-16 15:15:29,383 - ProxLB - DEBUG - Starting: get_host_port_from_string.
Sep 16 15:15:29 pve01 python3[7965]: 2026-09-16 15:15:29,383 - ProxLB - DEBUG - 10.10.10.223 has IPv4.
Sep 16 15:15:29 pve01 python3[7965]: 2026-09-16 15:15:29,383 - ProxLB - DEBUG - Starting: test_api_proxmox_host_ipv4.
Sep 16 15:15:29 pve01 python3[7965]: 2026-09-16 15:15:29,384 - ProxLB - DEBUG - Host 10.10.10.223 is reachable on IPv4 for tcp/8006.
Sep 16 15:15:29 pve01 python3[7965]: 2026-09-16 15:15:29,384 - ProxLB - DEBUG - Finished: test_api_proxmox_host_ipv4.
Sep 16 15:15:29 pve01 python3[7965]: 2026-09-16 15:15:29,384 - ProxLB - WARNING - SSL certificate validation to host 10.10.10.223:8006 is deactivated.
Sep 16 15:15:29 pve01 python3[7965]: 2026-09-16 15:15:29,471 - ProxLB - DEBUG - Using username/password authentication.
Sep 16 15:15:29 pve01 python3[7965]: 2026-09-16 15:15:29,471 - ProxLB - INFO - API connection to host 10.10.10.223:8006 succeeded.
Sep 16 15:15:29 pve01 python3[7965]: 2026-09-16 15:15:29,471 - ProxLB - DEBUG - Finished: api_connect.
Sep 16 15:15:29 pve01 python3[7965]: 2026-09-16 15:15:29,471 - ProxLB - DEBUG - Starting: test_api_user_permissions.
Sep 16 15:15:29 pve01 python3[7965]: 2026-09-16 15:15:29,482 - ProxLB - DEBUG - Finished: test_api_user_permissions.
Sep 16 15:15:29 pve01 python3[7965]: 2026-09-16 15:15:29,482 - ProxLB - DEBUG - Finished: ProxmoxApi initialization.
Sep 16 15:15:29 pve01 python3[7965]: 2026-09-16 15:15:29,482 - ProxLB - DEBUG - Starting: is_node_ha_manager.
Sep 16 15:15:29 pve01 python3[7965]: 2026-09-16 15:15:29,482 - ProxLB - DEBUG - Starting: get_ha_manager.
Sep 16 15:15:29 pve01 python3[7965]: 2026-09-16 15:15:29,487 - ProxLB - DEBUG - HA manager node: {'timestamp': 1789564524, 'service_status': {'ct:100': {'uid': '3XAlCXXxxbXy1PfXJ6fe7w', 'state': 'started', 'running': 1, 'node': 'pve03'}}, 'master_node': 'pve01', 'node_status': {'pve02': 'online', 'pve01': 'online', 'pve03': 'online'}}
Sep 16 15:15:29 pve01 python3[7965]: 2026-09-16 15:15:29,487 - ProxLB - DEBUG - Finished: get_ha_manager.
Sep 16 15:15:29 pve01 python3[7965]: 2026-09-16 15:15:29,487 - ProxLB - DEBUG - Cluster Master is: pve01 | We are: pve01) | This node is the HA manager.

@gyptazy

gyptazy commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator Author

The second part is now also included where it checks if the node count changed during migrations which could lead into imbalances and starts the calculations from scratch:

Sep 17 08:24:06 pve01 python3[178646]: 2026-09-17 08:24:06,412 - ProxLB - DEBUG - Balancing: Job ID UPID:pve01:0002B9F9:0060AAAC:6AAB875D:vzmigrate:102:root@pam: (guest: test03) is a standard migration job. Proceeding with status check.
Sep 17 08:24:06 pve01 python3[178646]: 2026-09-17 08:24:06,412 - ProxLB - DEBUG - Balancing: Job ID UPID:pve01:0002B9F9:0060AAAC:6AAB875D:vzmigrate:102:root@pam: (guest: test03) for migration is still running...
Sep 17 08:24:11 pve01 python3[178646]: 2026-09-17 08:24:11,512 - ProxLB - DEBUG - Starting: _get_rebalancing_job_status.
Sep 17 08:24:11 pve01 python3[178646]: 2026-09-17 08:24:11,522 - ProxLB - DEBUG - Balancing: Job ID UPID:pve01:0002B9F9:0060AAAC:6AAB875D:vzmigrate:102:root@pam: (guest: test03) is a standard migration job. Proceeding with status check.
Sep 17 08:24:11 pve01 python3[178646]: 2026-09-17 08:24:11,523 - ProxLB - CRITICAL - Balancing: Job ID UPID:pve01:0002B9F9:0060AAAC:6AAB875D:vzmigrate:102:root@pam: (guest: test03) went into an error! Please check manually.
Sep 17 08:24:11 pve01 python3[178646]: 2026-09-17 08:24:11,524 - ProxLB - DEBUG - Finished: _get_rebalancing_job_status.
Sep 17 08:24:11 pve01 python3[178646]: 2026-09-17 08:24:11,524 - ProxLB - CRITICAL - Balancing: Job ID UPID:pve01:0002B9F9:0060AAAC:6AAB875D:vzmigrate:102:root@pam: (guest: test03) for migration went into an error! Please check manually.
Sep 17 08:24:11 pve01 python3[178646]: 2026-09-17 08:24:11,524 - ProxLB - DEBUG - Starting: check_nodes_available.
Sep 17 08:24:12 pve01 python3[178646]: 2026-09-17 08:24:12,421 - ProxLB - WARNING - Helper: Node pve03 did not respond to a status check and may no longer be available: 595 Errors during connection establishment, proxy handshake: No route to host - {'errors': b''}
Sep 17 08:24:12 pve01 python3[178646]: 2026-09-17 08:24:12,421 - ProxLB - DEBUG - Finished: check_nodes_available.

(!!!!)

Sep 17 08:24:12 pve01 python3[178646]: 2026-09-17 08:24:12,421 - ProxLB - CRITICAL - Balancing: At least one cluster node is no longer reachable via the Proxmox API. Aborting the current balancing run.
Sep 17 08:24:12 pve01 python3[178646]: 2026-09-17 08:24:12,421 - ProxLB - CRITICAL - ProxLB: A Proxmox API call failed or a cluster node became unavailable: At least one cluster node became unavailable during balancing.. Reconnecting to the Proxmox API and restarting balancing run from scratch.
Sep 17 08:24:13 pve01 python3[178646]: 2026-09-17 08:24:13,421 - ProxLB - DEBUG - Starting: get_config.
Sep 17 08:24:13 pve01 python3[178646]: 2026-09-17 08:24:13,421 - ProxLB - INFO - Using config path: /etc/proxlb/proxlb.yaml
Sep 17 08:24:13 pve01 python3[178646]: 2026-09-17 08:24:13,426 - ProxLB - DEBUG - Set to debug level
Sep 17 08:24:13 pve01 python3[178646]: 2026-09-17 08:24:13,426 - ProxLB - DEBUG - Starting: ProxmoxApi initialization.
Sep 17 08:24:13 pve01 python3[178646]: 2026-09-17 08:24:13,426 - ProxLB - DEBUG - Starting: api_connect.
Sep 17 08:24:13 pve01 python3[178646]: 2026-09-17 08:24:13,426 - ProxLB - DEBUG - Starting: validate_config.
Sep 17 08:24:13 pve01 python3[178646]: 2026-09-17 08:24:13,426 - ProxLB - DEBUG - Finished: validate_config.
Sep 17 08:24:13 pve01 python3[178646]: 2026-09-17 08:24:13,426 - ProxLB - DEBUG - Starting: api_connect_get_hosts.
Sep 17 08:24:13 pve01 python3[178646]: 2026-09-17 08:24:13,426 - ProxLB - DEBUG - Starting: test_api_proxmox_host.
Sep 17 08:24:13 pve01 python3[178646]: 2026-09-17 08:24:13,426 - ProxLB - DEBUG - Starting: get_host_port_from_string.
Sep 17 08:24:13 pve01 python3[178646]: 2026-09-17 08:24:13,426 - ProxLB - DEBUG - 10.10.10.221 has IPv4.
Sep 17 08:24:13 pve01 python3[178646]: 2026-09-17 08:24:13,426 - ProxLB - DEBUG - Starting: test_api_proxmox_host_ipv4.
Sep 17 08:24:13 pve01 python3[178646]: 2026-09-17 08:24:13,427 - ProxLB - DEBUG - Host 10.10.10.221 is reachable on IPv4 for tcp/8006.
Sep 17 08:24:13 pve01 python3[178646]: 2026-09-17 08:24:13,427 - ProxLB - DEBUG - Finished: test_api_proxmox_host_ipv4.
Sep 17 08:24:13 pve01 python3[178646]: 2026-09-17 08:24:13,427 - ProxLB - WARNING - SSL certificate validation to host 10.10.10.221:8006 is deactivated.
Sep 17 08:24:13 pve01 python3[178646]: 2026-09-17 08:24:13,515 - ProxLB - DEBUG - Using username/password authentication.
Sep 17 08:24:13 pve01 python3[178646]: 2026-09-17 08:24:13,515 - ProxLB - INFO - API connection to host 10.10.10.221:8006 succeeded.
Sep 17 08:24:13 pve01 python3[178646]: 2026-09-17 08:24:13,515 - ProxLB - DEBUG - Finished: api_connect.

Comment thread proxlb/utils/helper.py Outdated
Comment thread proxlb/models/balancing.py Outdated
Comment thread proxlb/__main__.py
Comment thread proxlb/__main__.py
# every connect) and reconnect from scratch, re-running host failover.
proxlb_config = config_parser.get_config()
logger.set_log_level(proxlb_config.service.log_level)
proxmox_api = ProxmoxApi(proxlb_config)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming someone has only one endpoint configured, and that endpoint is rebooting. Is ProxLB supposed to sys.exit(2) then?

And with multiple endpoints, it is possible that ProxmoxApi exits e.g. because of SSLError of a single random endpoint.

@gyptazy

gyptazy commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator Author

Hey @dlucredativ,

thanks for your time and efforts you took into this review. Good catches! I've adjusted the mentioned things and revalidated them on my cluster.

Cheers,
gyptazy

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: failover to other hosts on connect

2 participants