|
| 1 | +# Upgrading from dstack 0.5.x to 0.6.0 |
| 2 | + |
| 3 | +Existing apps keep their identity through the upgrade: the same app ID, the |
| 4 | +same v0 keys, and the same encrypted data disk. What changes is almost every |
| 5 | +measurement, the service configuration, and the SDK names, so the upgrade is a |
| 6 | +planned rollout rather than a package update. Upgrade the KMS first, then the |
| 7 | +Gateway, then the VMM hosts, and move app CVMs to the new guest image last. |
| 8 | + |
| 9 | +The full list of changes is in the [changelog](../CHANGELOG.md#060---2026-09-28). |
| 10 | + |
| 11 | +## What's new |
| 12 | + |
| 13 | +- **Guest API v1.** A versioned API, `dstack.guest.v1`, with domain-separated |
| 14 | + key derivation, GPU evidence, and one attestation call that works on every |
| 15 | + platform. The unversioned API is frozen at v0.5.11, so existing apps keep |
| 16 | + working. See [guest-api-v1.md](./guest-api-v1.md). |
| 17 | +- **More platforms.** One attestation model covers Intel TDX, AMD SEV-SNP, |
| 18 | + AWS Nitro Enclaves, AWS NitroTPM, and GCP Confidential VMs. |
| 19 | +- **A new guest OS.** Guest images are built with Debian and mkosi, are |
| 20 | + reproducible, and ship a kernel build tree for out-of-tree modules. Yocto |
| 21 | + images are deprecated. |
| 22 | +- **Networking.** The VMM manages bridge and macvtap networking through |
| 23 | + `netd`, with vhost-net multiqueue and libvirt network filters. |
| 24 | +- **Gateway.** Application health checks keep unhealthy instances out of load |
| 25 | + balancing, CVMs can connect to several Gateway clusters, and |
| 26 | + `dns-persist-01` issues certificates without DNS credentials |
| 27 | + (experimental). |
| 28 | +- **Storage.** Data disks return deleted blocks to the host, and dm-verity |
| 29 | + volumes provide integrity-protected read-only data. |
| 30 | +- **Hardening.** Every component had a security pass: bounded parsing of |
| 31 | + untrusted input, owner-only secrets, and authentication on every |
| 32 | + administrative API. |
| 33 | + |
| 34 | +## Compatibility at a glance |
| 35 | + |
| 36 | +| Combination | Works? | |
| 37 | +|---|---| |
| 38 | +| 0.5.x app, unchanged, on a 0.6.0 guest image | Yes, through the frozen v0 API | |
| 39 | +| 0.5.8 or later KMS onboarding a 0.6.0 KMS | Yes | |
| 40 | +| 0.5.5 or earlier KMS onboarding a 0.6.0 KMS | No; onboard a 0.5.8+ KMS first | |
| 41 | +| 0.5.x KMS serving 0.6.0 guests | Yes, with limits (see [KMS](#1-kms)) | |
| 42 | +| 0.5.x and 0.6.0 Gateway nodes in one cluster | Serve traffic, but do not sync | |
| 43 | +| 0.5.x guest image on a 0.6.0 VMM | Only without `key_provider_id` (see [VMM](#3-vmm-hosts)) | |
| 44 | +| 0.5.x SDK against a 0.6.0 guest | Yes | |
| 45 | +| 0.6.0 SDK `DstackClient` against a 0.5.x guest | No; it now means v1 | |
| 46 | + |
| 47 | +## Before you start |
| 48 | + |
| 49 | +Every 0.6.0 guest image has a new `os_image_hash`, and the new KMS and Gateway |
| 50 | +CVMs have new measurements. Register them before you deploy anything: |
| 51 | + |
| 52 | +- add the 0.6.0 `os_image_hash` from the |
| 53 | + [release notes](https://github.com/Dstack-TEE/dstack/releases/tag/mkosi-os-v0.6.0) |
| 54 | + with `addOsImageHash` (see [onchain-governance.md](./onchain-governance.md)) |
| 55 | +- add the new KMS CVM's `mrAggregated` with `addKmsAggregatedMr`, and any new |
| 56 | + device IDs if your apps use device allowlists |
| 57 | + |
| 58 | +The contracts need no redeploy. |
| 59 | + |
| 60 | +Component images moved from Docker Hub to `ghcr.io/dstack-tee`, for example |
| 61 | +`ghcr.io/dstack-tee/dstack-kms:0.6.0`. The 0.5.x images stay on Docker Hub. |
| 62 | + |
| 63 | +The KMS and Gateway admin APIs now fail closed: with the admin API enabled but |
| 64 | +no `auth_token` or `htpasswd_file`, the service refuses to start. Set a token |
| 65 | +before you restart them. Old configuration files still parse, and |
| 66 | +removed keys are ignored, so check the per-component sections below for |
| 67 | +settings that silently stopped taking effect. |
| 68 | + |
| 69 | +## 1. KMS |
| 70 | + |
| 71 | +A KMS seals its root key to its own measurements, so you cannot upgrade one in |
| 72 | +place. Deploy a 0.6.0 KMS CVM next to the old one and onboard it, which copies |
| 73 | +the root keys over RA-TLS. The procedure is in |
| 74 | +[deployment.md](./deployment.md#adding-kms-replicas-onboarding), with one |
| 75 | +difference for a 0.5.x source: it has no admin listener, so leave |
| 76 | +`source_token` empty and point `source_url` at its public RPC port. |
| 77 | + |
| 78 | +```bash |
| 79 | +curl -X POST "http://<new-kms>:9203/prpc/Onboard.Onboard?json" \ |
| 80 | + -H "Content-Type: application/json" \ |
| 81 | + -d '{"source_url": "https://<old-kms>:9201/prpc", "source_token": "", "domain": "kms.example.com"}' |
| 82 | +``` |
| 83 | + |
| 84 | +Both sides must authorize each other first: the new KMS allows the old |
| 85 | +`mrAggregated`, and the old KMS allows the new `mrAggregated` and the 0.6.0 |
| 86 | +`os_image_hash`. After onboarding, the KMS CA public key must match the old |
| 87 | +one. The CA certificate itself is reissued, so compare keys, not PEM files. |
| 88 | + |
| 89 | +The source must be 0.5.8 or later. Older sources reject the 0.6.0 certificate |
| 90 | +format; onboard a 0.5.8+ KMS from them first, then onboard 0.6.0 from that. |
| 91 | + |
| 92 | +Keep the old KMS running until every client has moved. While a 0.5.x KMS still |
| 93 | +verifies 0.6.0 guests: |
| 94 | + |
| 95 | +- it can only measure 0.6.0 guests with exactly 2 GiB or at least 2816 MiB of |
| 96 | + memory, and it must be able to download the 0.6.0 image |
| 97 | +- apps using v1 `IssueCert`, and anything verifying v1 attestations, need |
| 98 | + 0.5.9 or later |
| 99 | + |
| 100 | +Configuration changes that affect an upgrade: |
| 101 | + |
| 102 | +- `[rpc.tls.mutual]` and `core.admin_token_hash` are gone. Admin RPCs such as |
| 103 | + `ClearImageCache` now live on the `[core.admin]` listener. |
| 104 | +- Key release on AMD SEV-SNP, AWS NitroTPM, and AWS Nitro Enclaves is off |
| 105 | + until you set `sev_snp_key_release`, `aws_nitro_tpm_key_release`, or |
| 106 | + `nitro_enclave_key_release`. |
| 107 | +- Root-key handover is also served as `Admin.GetKmsKey`. Keep |
| 108 | + `core.onboard.public_key_handover = true` while any KMS still onboards |
| 109 | + through the public port, and turn it off afterwards. |
| 110 | +- Auto-bootstrap refuses to overwrite existing root keys. Restore lost key |
| 111 | + files instead of letting the KMS regenerate them. |
| 112 | + |
| 113 | +If you use `auth-simple`, check your policy before switching over, because |
| 114 | +three defaults now deny boots that 0.5.x allowed: |
| 115 | + |
| 116 | +- an empty `devices` list denies every device unless `allowAnyDevice` is true |
| 117 | +- `allowedTcbStatuses` defaults to `["UpToDate"]` |
| 118 | +- `allowedAdvisoryIds` defaults to `[]`, so any advisory ID on a quote denies |
| 119 | + |
| 120 | +`auth-mock` refuses to start with an unknown `MOCK_POLICY`. `auth-eth-bun` |
| 121 | +accepts `ETH_CHAIN_ID` and `ETH_FINALITY_CONFIRMATIONS`. |
| 122 | + |
| 123 | +## 2. Gateway |
| 124 | + |
| 125 | +Upgrade the Gateway after the KMS cutover. The new Gateway image changes the |
| 126 | +Gateway app's compose hash, so whitelist the new hash on its app contract |
| 127 | +first. Then roll the cluster one node at a time: stop a node, update its |
| 128 | +compose to the 0.6.0 image, and start it again. |
| 129 | +Its WaveKV data directory is migrated in place, so registrations, |
| 130 | +certificates, and DNS credentials carry over, and traffic keeps flowing |
| 131 | +through the other nodes. |
| 132 | + |
| 133 | +0.5.x and 0.6.0 nodes do not sync with each other. During the rollout, avoid |
| 134 | +admin changes and expect a CVM that registers with one version to be unknown |
| 135 | +to the other until every node runs 0.6.0. Finish the rollout promptly. |
| 136 | + |
| 137 | +Configuration changes: |
| 138 | + |
| 139 | +- Set `core.admin.auth_token` (or `DSTACK_GATEWAY_ADMIN_TOKEN`). The admin API |
| 140 | + had no authentication in 0.5.x. |
| 141 | +- `core.debug.insecure_skip_attestation` is gone, and a Gateway without a |
| 142 | + guest agent no longer starts. |
| 143 | +- A certificate configured in `[core.proxy]` (`cert_chain`, `cert_key`) is |
| 144 | + ignored. Install it with `Admin.ImportCert` instead |
| 145 | + (see [dstack-gateway.md](./dstack-gateway.md)). |
| 146 | + |
| 147 | +0.5.x CVMs register with a 0.6.0 Gateway unchanged, and 0.6.0 CVMs can still |
| 148 | +register with a 0.5.x Gateway during the rollout. |
| 149 | + |
| 150 | +## 3. VMM hosts |
| 151 | + |
| 152 | +Install the 0.6.0 `dstack-vmm` and restart it. Running CVMs are not touched, |
| 153 | +but each CVM picks up the new behavior on its next start. |
| 154 | + |
| 155 | +**Check 0.5.x apps that set `key_provider_id` before you upgrade a host.** A |
| 156 | +0.6.0 VMM writes those apps' `MRCONFIGID` in a newer format that 0.5.x guest |
| 157 | +images reject, so the CVM fails early in boot and restarts in a loop. Move |
| 158 | +such apps to the 0.6.0 guest image when you upgrade their host. Apps without |
| 159 | +`key_provider_id` keep booting 0.5.x images. |
| 160 | + |
| 161 | +Configuration changes in `vmm.toml`: |
| 162 | + |
| 163 | +- `qemu_hotplug_off` now defaults to `true`, which changes the ACPI tables and |
| 164 | + RTMRs of every CVM on its next start, whatever its image. Update any |
| 165 | + allowlist that pins MRs. A host without GPUs can set it to `false` to keep |
| 166 | + its current measurements; a GPU host needs `true`. |
| 167 | +- Bridge and macvtap networking now runs through `netd`, a privileged helper |
| 168 | + started as `dstack-vmm netd`. Install and start it before the VMM (see |
| 169 | + [bridge-networking.md](./bridge-networking.md) and |
| 170 | + [libvirt-network-filter.md](./libvirt-network-filter.md)). User-mode |
| 171 | + networking needs nothing new. |
| 172 | +- `[auth]` now guards the whole API and web UI. `vmm-cli` takes `--token` or |
| 173 | + `DSTACK_VMM_TOKEN`. |
| 174 | +- KMS and Gateway URLs are shuffled per CVM by default |
| 175 | + (`shuffle_kms_urls`, `shuffle_gateway_urls`). |
| 176 | +- `image.registry` is gone; install guest images into the image directory. |
| 177 | + |
| 178 | +Keep `gateway_urls` while any 0.5.x CVM runs on the host. The new |
| 179 | +`gateway_clusters` setting is read only by 0.6.0 guests. |
| 180 | + |
| 181 | +The 0.6.0 web UI writes `manifest_version: "3"` when you move an app to a |
| 182 | +0.6.0 image, and 0.5.x images cannot read it. To roll such an app back to a |
| 183 | +0.5.x image, recreate it with `vmm-cli`. |
| 184 | + |
| 185 | +## 4. Apps |
| 186 | + |
| 187 | +Moving an app CVM to the 0.6.0 guest image keeps its app ID, its v0 keys, and |
| 188 | +its data disk. The disk key is derived exactly as before, and the LUKS and ZFS |
| 189 | +formats are unchanged. Back up the data anyway: this path is not covered by |
| 190 | +the release tests. |
| 191 | + |
| 192 | +To upgrade, change only the image. The compose hash stays the same as long as |
| 193 | +the app-compose bytes do, so nothing new needs whitelisting beyond the |
| 194 | +`os_image_hash`. |
| 195 | + |
| 196 | +Behavior an app may notice after the move: |
| 197 | + |
| 198 | +- Data disks now release deleted blocks to the host, and the first boot starts |
| 199 | + a one-time `zpool trim`. Set `storage_discard: false` to opt out, which |
| 200 | + changes the compose hash. |
| 201 | +- `EmitEvent` always fails, because runtime RTMR3 events are system-owned. |
| 202 | + Bind app data through `report_data` instead. |
| 203 | +- `GetQuote` answers on Intel TDX only. Use `Attest` on other platforms. |
| 204 | + |
| 205 | +## 5. SDKs |
| 206 | + |
| 207 | +Apps keep working with their 0.5.x SDK. When you update to the 0.6.0 SDK, the |
| 208 | +unsuffixed `DstackClient` means the v1 client, which needs a 0.6.0 guest. To |
| 209 | +stay on the v0 API, rename it: |
| 210 | + |
| 211 | +```python |
| 212 | +from dstack_sdk import DstackClientV0 |
| 213 | + |
| 214 | +client = DstackClientV0() |
| 215 | +``` |
| 216 | + |
| 217 | +```ts |
| 218 | +import { DstackClientV0 } from '@phala/dstack-sdk' |
| 219 | + |
| 220 | +const client = new DstackClientV0() |
| 221 | +``` |
| 222 | + |
| 223 | +Deep imports of the v0 modules gain a `_v0` suffix, for example |
| 224 | +`dstack_sdk.dstack_client_v0`. The Python v0 client emits a |
| 225 | +`DeprecationWarning`, which fails test suites that turn warnings into errors. |
| 226 | +JavaScript v0 methods now reject on failure instead of resolving to an empty |
| 227 | +object, and the wallet adapters reject TLS keys. |
| 228 | + |
| 229 | +**Moving to v1 changes your keys.** v1 `GetKey` derives different keys from |
| 230 | +v0 for the same name. Derive the v1 key, move any assets with a transaction |
| 231 | +signed by the v0 key, and only then switch. See |
| 232 | +[the v1 migration guide](./guest-api-v1.md#migration-from-the-unversioned-api). |
| 233 | + |
| 234 | +**Recheck compose hashes computed with the Go or Python SDK.** Before 0.6.0, |
| 235 | +those helpers dropped some app-compose fields, and Go escaped `<`, `>` and |
| 236 | +`&`, so the hash you whitelisted may not match what the CVM measures. Recompute |
| 237 | +it with the 0.6.0 SDK, whitelist the correct value, and remove the stale one. |
0 commit comments