Add LCOW live migration source and destination APIs by rawahars · Pull Request #2803 · microsoft/hcsshim

rawahars · 2026-07-02T05:30:06Z

Summary

Introduce an end-to-end live-migration workflow for LCOW sandboxes, letting a running sandbox be handed off from a source shim to a destination shim.

Add a migration Controller that sequences a single session through a source or destination lifecycle: the source prepares and exports an opaque sandbox snapshot (VM + per-pod state), while the destination imports it, patches each container onto its new IDs, builds the destination VM, transfers memory over a shared socket, and finalizes with a resume or stop.
Expose the migration gRPC/ttrpc surface on the LCOW shim service (PrepareAndExportSandbox, ImportSandbox, PrepareSandbox, TransferSandbox, FinalizeSandbox, Cancel, Cleanup, CreateDuplicateSocket, Notifications) and register it alongside the task and sandbox services.
Wire migration into the task lifecycle: route destination-side CreateTask for rehydrated containers into a patch path, and reject task mutations (state, delete, kill) while a session is in progress.
Add duplicated transport-socket adoption, a subscriber-based progress notification stream, and proto/HCS conversion helpers for migration options and events.

This PR needs #2790 to be merged.

marma-dev

Overall looks good, doc.go / state.go diagrams are excellent and match the code — nice 👍 .
I would like run.Transfer and service-owned maps comments addressed (or explained). Rest are nice to have.

marma-dev · 2026-07-08T13:59:09Z

+		c.state = StateTransferring
+
+		// Run the transfer; a failure marks the session failed for subscribers.
+		if err := c.runTransfer(ctx); err != nil {


It seems the background transfer goroutine does c.mu.Lock(); defer c.mu.Unlock() and then calls c.runTransfer(ctx) inside that critical section. runTransfer issues several HCS calls (StartLiveMigrationOnSource / StartWithMigrationOptions / StartLiveMigrationTransfer).

While those calls are in flight, State() (RLock) and especially Cancel() (Lock) will block. If the goal is that the caller returns immediately and cancellation happens out-of-band, being unable to acquire the lock to Cancel during the transfer kickoff partly defeats that goal.

If any of those HCS calls can block for a meaningful duration, this serializes all controller access behind them.

Should we consider transitioning to StateTransferring under the lock, releasing it, then running the HCS calls, and re-acquiring only to set the terminal state?
I am trying to confirm whether those Start* calls are fire-and-forget (completion arriving via notifications) — if so, the impact is smaller, but the lock-held-across-HCS pattern still seems fragile.

marma-dev · 2026-07-08T14:03:24Z

+		opts.PodControllers[importedPod.PodID()] = importedPod
+
+		// Source container IDs must be unique across pods so the later patch
+		// lookup is unambiguous.
+		for containerID := range importedPod.ListContainers() {
+			if _, dup := pending[containerID]; dup {
+				return fmt.Errorf("duplicate source container id %q across imported pods: %w", containerID, errdefs.ErrInvalidArgument)
+			}
+
+			// Track the container as awaiting a patch and map it to its pod.
+			pending[containerID] = struct{}{}
+			opts.ContainerPodMapping[containerID] = importedPod.PodID()
+		}


If a later pod.Import (or a duplicate-container check) fails, the function returns an error but leaves earlier pods/containers already inserted into the service maps, while c.state stays StateIdle.

I think that even if the controller is never mutated on the basis of a half-specified request, it only holds for the controller struct, not for the borrowed maps. Building into local maps first and committing them only on success would keep failures atomic

marma-dev · 2026-07-08T14:07:33Z

+	wantSize := int(unsafe.Sizeof(windows.WSAProtocolInfo{}))
+	if len(protocolInfo) < wantSize {
+		return fmt.Errorf("protocol info is %d bytes, want at least %d: %w", len(protocolInfo), wantSize, errdefs.ErrInvalidArgument)
+	}
+
+	// Decode the opaque caller-supplied bytes into the socket descriptor
+	// used to recreate the duplicated socket.
+	var info windows.WSAProtocolInfo
+	if err := binary.Read(bytes.NewReader(protocolInfo), binary.LittleEndian, &info); err != nil {
+		return fmt.Errorf("decode WSAProtocolInfo: %w", err)
+	}
+


Looks like we validate the buffer with int(unsafe.Sizeof(windows.WSAProtocolInfo{})) but decode with binary.Read(..., &info). binary.Read consumes binary.Size(info) bytes (its own reflection-based, unpadded layout), which is not guaranteed to equal unsafe.Sizeof if the struct has any alignment padding.

I believe for WSAProtocolInfo the fields are 4-byte-aligned so they likely match today, but relying on that coincidence is brittle.

Could we consider validating against binary.Size(windows.WSAProtocolInfo{}) (and/or decoding via the same mechanism the caller serialized with) so the size check and the decode agree?

jiechen0826 · 2026-07-09T01:58:49Z

+			// Reattach each migrated pod's containers and processes on this host.
+			for podID, podCtrl := range c.podControllers {
+				if err := podCtrl.Resume(ctx, c.vmController, events, true /* isDestination */); err != nil {
+					return fmt.Errorf("resume pod %q from migration: %w", podID, err)


On the RESUME path, if one podCtrl.Resume fails partway through the loop, Finalize returns before setting StateFinalized, so the session stays in StateTransferCompleted with the VM already resumed and some pods live. Since finalizeSandboxInternal just propagates the error to the caller, what's the intended recovery here, is the caller expected to retry Finalize, or fall back to a STOP?

It seems like a retried RESUME would call vm.Resume again on an already-running VM, which vm.Resume rejects, so I wasn't sure how the partial-failure case is meant to unwind. I could be missing where that's handled upstream.

rawahars · 2026-07-09T17:04:11Z

Rebased onto the mainline. No code changes from the last review were made.

rawahars · 2026-07-23T23:52:43Z

Rebased onto the mainline. No code changes from the last review were made.

rawahars · 2026-07-24T03:15:42Z

@marma-dev @jiechen0826 All your review suggestions have been incorporated in 166ff74. Please take a look now.

Introduce an end-to-end live-migration workflow for LCOW sandboxes, letting a running sandbox be handed off from a source shim to a destination shim. - Add a migration Controller that sequences a single session through a source or destination lifecycle: the source prepares and exports an opaque sandbox snapshot (VM + per-pod state), while the destination imports it, patches each container onto its new IDs, builds the destination VM, transfers memory over a shared socket, and finalizes with a resume or stop. - Expose the migration gRPC/ttrpc surface on the LCOW shim service (PrepareAndExportSandbox, ImportSandbox, PrepareSandbox, TransferSandbox, FinalizeSandbox, Cancel, Cleanup, CreateDuplicateSocket, Notifications) and register it alongside the task and sandbox services. - Wire migration into the task lifecycle: route destination-side CreateTask for rehydrated containers into a patch path, and reject task mutations (state, delete, kill) while a session is in progress. - Add duplicated transport-socket adoption, a subscriber-based progress notification stream, and proto/HCS conversion helpers for migration options and events. Signed-off-by: Harsh Rawat <harshrawat@microsoft.com>

Give the live-migration session predictable, caller-facing behavior: - Cancellation & wind-down: a session can be aborted from any active state and always finalizes then cleans up back to idle, even after a failure or a partly-applied finalize. - Predictable retries: resource snapshots can be retaken and resumes are idempotent, while duplicate or out-of-order session calls are rejected instead of silently reapplied. - Typed errors: calls report invalid-argument, failed-precondition, or already-exists so callers can branch on them. Backed by unit tests and a documented state machine. Signed-off-by: Harsh Rawat <harshrawat@microsoft.com>

rawahars · 2026-07-27T06:42:56Z

Rebased onto the mainline

marma-dev

LGTM!

Copilot

Pull request overview

Introduce an end-to-end LCOW live migration workflow in the lcow-v2 shim by adding a migration session controller (source + destination lifecycles), wiring it into the shim service surface (ttrpc), and updating underlying controllers to support retry-safe save/import/patch/resume semantics and richer error classification.

Changes:

Add an internal/controller/migration controller implementing a session state machine, duplicated-socket adoption, and subscriber-based progress notifications.
Extend VM/pod/container/process/network/device controller migration save/import/patch/resume flows (including idempotent retries) and wrap key failures with containerd/errdefs.
Register and implement migration APIs in cmd/containerd-shim-lcow-v2/service, including a destination CreateTask patch path for rehydrated containers.

Reviewed changes

Copilot reviewed 45 out of 47 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
pkg/migration/parse.go	Adds helpers to map HCS migration events/origin/results into wire notification fields.
pkg/migration/parse_test.go	Adds unit tests for migration proto parsing/mapping and blackout-exited details parsing.
internal/vm/vmmanager/migration.go	Adds `UtilityVM.CancelLiveMigration` wrapper.
internal/hcs/v2/migration.go	Adds `System.CancelLiveMigration` entry point (currently stubbed).
internal/controller/vm/vm.go	Ensures migrating flag cleared during VM termination to release bridge earlier.
internal/controller/vm/vm_migration.go	Broadens finalize state-acceptance and adds VM-level cancel API.
internal/controller/vm/save_lcow.go	Tightens migration state preconditions and wraps errors with `errdefs`; adds resume idempotency.
internal/controller/process/save.go	Makes Save retry-safe from `StateSourceMigrating`, adds resume idempotency, wraps errors with `errdefs`.
internal/controller/process/save_test.go	Updates/extends tests for new Save/Resume semantics and error wrapping.
internal/controller/pod/save_lcow.go	Makes Resume idempotent and improves error classification for Import/Patch.
internal/controller/pod/save_lcow_test.go	Updates tests for new Resume/Patch behavior and `errdefs` wrapping.
internal/controller/network/save.go	Makes Save retry-safe from `StateSourceMigrating`, adds resume/reset idempotency, wraps errors with `errdefs`.
internal/controller/network/save_test.go	Updates tests for new Save/Resume/Reset behavior and `errdefs` wrapping.
internal/controller/migration/types.go	Defines migration controller interfaces/options used by service + tests.
internal/controller/migration/state.go	Defines migration session state machine and stringification.
internal/controller/migration/socket.go	Implements duplicated transport socket adoption and validation.
internal/controller/migration/socket_test.go	Adds end-to-end tests for socket descriptor adoption and guards.
internal/controller/migration/notifications.go	Adds subscriber fan-out + replay notification stream over VM migration events.
internal/controller/migration/notifications_test.go	Adds tests for subscription lifecycle, replay, fan-out, and controller Subscribe integration.
internal/controller/migration/mocks/mock_lcow_migration.go	Adds gomock for migration controller’s VM interface.
internal/controller/migration/doc.go	Documents the migration controller lifecycle, failure/cancel paths, and notifications.
internal/controller/migration/controller.go	Implements core transfer/finalize/cancel/cleanup sequencing for a session.
internal/controller/migration/controller_source.go	Implements source-side prepare + export snapshot workflow.
internal/controller/migration/controller_source_test.go	Adds source-side unit tests.
internal/controller/migration/controller_destination.go	Implements destination import + per-container patch + prepare destination VM.
internal/controller/migration/controller_destination_test.go	Adds destination-side unit tests.
internal/controller/linuxcontainer/save.go	Makes container save retry-safe, adds resume idempotency, wraps errors with `errdefs`.
internal/controller/linuxcontainer/save_test.go	Updates tests for new save/resume semantics and error wrapping.
internal/controller/device/vpci/save.go	Makes VPCI Save a no-op when empty; wraps unsupported state with `errdefs`.
internal/controller/device/vpci/save_lcow_test.go	Updates VPCI Save tests to assert `errdefs` wrapping.
internal/controller/device/scsi/save.go	Wraps validation/precondition failures with `errdefs`; tightens import validation comments.
internal/controller/device/scsi/save_test.go	Updates SCSI tests for `errdefs` wrapping and new invalid-argument cases.
internal/controller/device/scsi/mount/save_lcow.go	Wraps mount save precondition failures with `errdefs`.
internal/controller/device/scsi/mount/save_lcow_test.go	Updates mount save tests for `errdefs` wrapping.
internal/controller/device/scsi/disk/save.go	Wraps disk save precondition failures with `errdefs`.
internal/controller/device/scsi/disk/save_test.go	Updates disk save tests for `errdefs` wrapping.
internal/controller/device/plan9/save.go	Makes Plan9 Save a no-op when empty; wraps unsupported state with `errdefs`.
internal/controller/device/plan9/save_test.go	Updates Plan9 Save tests to assert `errdefs` wrapping.
cmd/containerd-shim-lcow-v2/service/types.go	Extends service vmController interface to satisfy migration controller needs.
cmd/containerd-shim-lcow-v2/service/service.go	Instantiates migration controller and registers migration service; adds migration-in-progress guard.
cmd/containerd-shim-lcow-v2/service/service_task_internal.go	Routes migrated CreateTask into patch path and rejects task ops during migration.
cmd/containerd-shim-lcow-v2/service/service_task_internal_test.go	Updates task API tests to reflect migration/VM-running guard changes.
cmd/containerd-shim-lcow-v2/service/service_sandbox_internal_test.go	Updates service test construction to include migration controller.
cmd/containerd-shim-lcow-v2/service/service_migration.go	Adds instrumentation-layer migration service methods and registration plumbing.
cmd/containerd-shim-lcow-v2/service/service_migration_internal.go	Implements migration service business logic (prepare/export/import/prepare/transfer/finalize/notifications/cancel/cleanup).
cmd/containerd-shim-lcow-v2/service/mocks/mock_service.go	Updates mocks to include migration-related vmController methods.

Files not reviewed (2)

cmd/containerd-shim-lcow-v2/service/mocks/mock_service.go: Generated file
internal/controller/migration/mocks/mock_lcow_migration.go: Generated file

Comments suppressed due to low confidence (1)

internal/controller/migration/controller.go:486

Cleanup() resets c.socketReady to a new channel without closing the old one. If Transfer() previously started and is still blocked waiting on the old socketReady (e.g., canceled/failed before socket registration), that goroutine will keep waiting until timeout, causing a long-lived goroutine leak.

	// Reset all session-scoped state so the controller can host a new session.
	c.sessionID, c.sandboxID, c.origin = "", "", ""
	c.vmController = nil
	c.podControllers = nil
	c.containerPodMapping = nil
	c.pendingPatches = nil
	c.socketReady = make(chan struct{})
	c.state = StateIdle

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+	// TODO: wire this to the HcsCancelLiveMigration compute-core API once it is
+	//  available. Until then this is a stub that reports success without contacting HCS.
+
+	return nil


+	// Enter the cancelling state before the abort so a concurrent transfer yields
+	// instead of completing; a successful abort then marks the session cancelled.
+	c.state = StateCancelling
+
+	if err := c.cancelLiveMigration(ctx); err != nil {
+		// Abort did not take hold; mark failed so the caller can retry Cancel.
+		c.state = StateFailed
+		return fmt.Errorf("cancel live migration: %w", err)
+	}
+
+	c.state = StateCancelled
+


+	if c.state != StateSourcePrepared {
+		return nil, fmt.Errorf("export requires state %s or %s (current: %s): %w", StateSourcePrepared, StateSourceExported, c.state, errdefs.ErrFailedPrecondition)
+	}


+	// Live-migration destination path: the container is already rehydrated
+	// from the imported snapshot. Delegate to the migration helper, which
+	// patches host-side resource paths, fixes bookkeeping, and publishes the
+	// TaskCreate event without driving the creation/start lifecycle.
+	if _, ok := spec.Annotations[annotations.LiveMigrationSourceContainerID]; ok {
+		return s.patchMigratedContainerInternal(ctx, request, spec)
+	}


rawahars requested a review from a team as a code owner July 2, 2026 05:30

rawahars added this to the Add Live Migration support to LCOW V2 shim milestone Jul 2, 2026

rawahars force-pushed the lm_service branch 2 times, most recently from 4a16358 to 68e2037 Compare July 2, 2026 06:41

marma-dev requested changes Jul 8, 2026

View reviewed changes

jiechen0826 reviewed Jul 9, 2026

View reviewed changes

rawahars force-pushed the lm_service branch from 68e2037 to 17b31d6 Compare July 9, 2026 17:03

rawahars force-pushed the lm_service branch from 17b31d6 to d1b1b67 Compare July 23, 2026 23:52

rawahars added 2 commits July 27, 2026 12:11

rawahars force-pushed the lm_service branch from 166ff74 to c47d48a Compare July 27, 2026 06:42

marma-dev requested a review from Copilot July 27, 2026 09:59

marma-dev approved these changes Jul 27, 2026

View reviewed changes

Copilot started reviewing on behalf of marma-dev July 27, 2026 10:00 View session

Copilot AI reviewed Jul 27, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add LCOW live migration source and destination APIs#2803

Add LCOW live migration source and destination APIs#2803
rawahars wants to merge 2 commits into
microsoft:mainfrom
rawahars:lm_service

rawahars commented Jul 2, 2026

Uh oh!

marma-dev left a comment

Uh oh!

marma-dev Jul 8, 2026

Uh oh!

marma-dev Jul 8, 2026

Uh oh!

marma-dev Jul 8, 2026

Uh oh!

jiechen0826 Jul 9, 2026

Uh oh!

rawahars commented Jul 9, 2026

Uh oh!

rawahars commented Jul 23, 2026

Uh oh!

rawahars commented Jul 24, 2026

Uh oh!

rawahars commented Jul 27, 2026

Uh oh!

marma-dev left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

rawahars commented Jul 2, 2026

Summary

Uh oh!

marma-dev left a comment

Choose a reason for hiding this comment

Uh oh!

marma-dev Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

marma-dev Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

marma-dev Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

jiechen0826 Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

rawahars commented Jul 9, 2026

Uh oh!

rawahars commented Jul 23, 2026

Uh oh!

rawahars commented Jul 24, 2026

Uh oh!

rawahars commented Jul 27, 2026

Uh oh!

marma-dev left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants