VM migration between hosts, plus detection of migrations done outside the API - #351
Merged
Conversation
… has
`vm.host_id` is aimed at by every lifecycle operation there is — start,
stop, reinstall, firewall sync, state polling — so a VM that moves
without the database being told breaks all of them at once, silently. A
VM migrated by hand in the Proxmox UI is exactly that case, and one such
VM is out of sync in production today.
Both halves work off the same rule: the host is the source of truth for
where a VM lives. `ReconcileVmHosts` runs every 10 minutes, lists the VMs
on every reachable host, and re-points host_id/disk_id at whichever host
actually has each VM, recording a `migrated` history entry with
`detected: true` and notifying admins. A host that could not be polled is
left out rather than treated as empty, and a VM seen on two hosts (the
signature of a leftover copy on the source node) is reported but never
reconciled, so the record cannot flap between them.
`POST /api/admin/v1/vms/{id}/migrate` performs the move. Pre-flight
refuses a target in another region — IP assignments are not rewritten, so
the VM would come up unreachable — as well as a different hypervisor kind
or CPU architecture, a disabled host, or one without free CPU, memory or
disk. The destination pool is the one whose name matches the source where
it exists, which on shared or mirrored storage means nothing is copied;
otherwise the emptiest pool with room, copied across. host_id/disk_id
change only after the hypervisor confirms the move, and the firewall
ruleset is re-applied on the destination.
Placement moves through a new `update_vm_host`, not `update_vm`, which
has never written host_id: where a VM lives is owned by the hypervisor,
so it must not change as a side effect of saving an unrelated edit.
Migration is implemented for Proxmox; other host kinds return a clear
"not supported" rather than half-moving a VM.
Fixes #66
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #66
vm.host_idis what every lifecycle operation aims at — start, stop, reinstall, firewall sync, state polling — so a VM that moves without the database being told breaks all of them at once, with no signal that anything is wrong. A VM migrated by hand in the Proxmox UI is exactly that case, and one such VM is out of sync in production today.Detection and repair
WorkJob::ReconcileVmHostsruns every 10 minutes: it lists the VMs on every reachable host and re-pointsvm.host_id/vm.disk_id(matching the storage pool it was found on) at whichever host actually has each VM. Each correction is recorded in VM history as the newmigratedaction withdetected: true, and raises an admin notification — nobody asked this API for the move, so it is worth surfacing.Two rules keep it from doing harm:
Migration
POST /api/admin/v1/vms/{id}/migratewith{ target_host_id, live?, reason? }(permissionvirtual_machines::update) queues aMigrateVmjob and returns ajob_id. Pre-flight refuses:The destination pool is the one whose name matches the source where such a pool exists — on shared or mirrored storage that means nothing is copied — otherwise the emptiest pool with room, copied across with
with-local-disks/targetstorage.live: falsestops a running VM first and starts it again on the destination.host_id/disk_idare updated only after the hypervisor confirms the move, so a failed migration leaves the VM on, and pointed at, the source host; the firewall ruleset is re-applied on arrival.Placement moves through a new
LNVpsDb::update_vm_hostrather thanupdate_vm, which has never writtenhost_id: where a VM lives is owned by the hypervisor and must not change as a side effect of saving an unrelated edit to aVm.Implemented for Proxmox. Other host kinds return a clear "not supported" error instead of half-moving a VM.
Tests
14 new tests, all green with
cargo test --workspace --exclude lnvps_e2e -- --test-threads=1:migrate_vmandreconcile_vm_hostsoverMockDband the dummy hypervisor, asserting the database follows the host and the history entry is written;with-local-diskswhen the disk does not have to move);409/404refusals.ADMIN_API_ENDPOINTS.mdandAPI_CHANGELOG.mdare updated.