Model Version support

AGENTS.md

@@ -66,6 +66,7 @@ Harmony is the substrate other apps sync through, so a bug here corrupts every c
 - **Hash chain.** `Commit.Hash` chains over `ParentHash` in commit order; `AlwaysValidateCommits` (on in tests) checks it after every write.
 - **Serialized changes are forever.** Commits written by old clients must still deserialize. Keep every change type's `$type` discriminator and JSON property names stable; a rename or removal needs a migration path. `PeekThenConcreteChangeConverter` owns `IChange` discrimination and needs `$type` first; unknown types become `OpaqueChange` when configured.
 - **JSON options come from `HarmonyConfig.JsonSerializerOptions`**, never a fresh `JsonSerializerOptions`, so registered change and object types resolve.
+- **Model versions are append-only.** Adding a change type, or changing how one applies, means appending a model version to `ModelVersionBuilder` that lists the type; a change is stored with the number of the authoring app's latest version, listed or not, and options validation fails while a registered type is unlisted. Listing a type invalidates changes authored at the new version or later, and the snapshots are rebuilt from the checkpoint before the oldest of them; pass `invalidateFrom` when older changes must be invalidated too, including after a release that changed a type without listing it. A released version, the sample's version 0 included, stays untouched: reconcile throws on an edited one. Harmony's own change types aren't versioned yet: changing how one applies needs sillsdev/harmony#141 first. A test that needs a different history calls the internal `ModelVersionBuilder.Clear()`, as `ModelVersionTests` does.
 
 ## Review
 

README.md

@@ -102,8 +102,10 @@ Once you have created your change types, you need to tell Harmony about them. Ag
 ```C#
 services.AddCrdtData<SampleDbContext>(config =>
 {
-// add the following line
+// add the following lines
     config.ChangeTypeListBuilder.Add<SetWordTextChange>();
+    //every change type must be listed by a model version, see Model versions below
+    config.ModelVersionBuilder.Add(0, "Before model versions", v => v.Change<SetWordTextChange>());
     config.ObjectTypeListBuilder
         .Add<Word>()
         .Add<Definition>()
@@ -178,6 +180,47 @@ await dataModel.SyncWith(remoteModel);
 ```
 It's that easy. All the heavy lifting is done by the interface which is fairly simple to implement.
 
+### Model versions
+Clients on different app versions sync with each other, so an older app receives changes a newer app authored. It skips the ones whose type it doesn't know (stored as `OpaqueChange` with `UnknownChangeHandling.Fallback`) and applies the rest with its old code. When the older app upgrades, those changes must be replayed.
+
+Model versions make that possible. Declare them in order, numbered from 0, each listing the change types it added or changed. Version 0 lists every change type that existed before model versions. Every change is stored with the number of the authoring app's latest version, whether or not that version lists the change's type, so a change authored by a newer app carries a number the older app doesn't have yet. Startup fails if no versions are declared, a registered change type isn't listed, or remote resources are enabled without `RemoteResourceChanges`. Only add versions at the end, never modify a released one:
+```C#
+services.AddCrdtRemoteResources<MediaMetadata>();
+services.AddCrdtData<AppDbContext>(config =>
+{
+    config.ChangeTypeListBuilder
+        .Add<NewWordChange>()
+        .Add<SetWordTextChange>()
+        .Add<SetWordNoteChange>();
+    config.ModelVersionBuilder
+        .Add(0, "Before model versions", v => v
+            .Change<NewWordChange>()
+            .Change<SetWordTextChange>()
+            //the change types AddCrdtRemoteResources registers
+            .RemoteResourceChanges<MediaMetadata>())
+        .Add(1, "Word notes", v => v.Change<SetWordNoteChange>())
+        //a major version regenerates all snapshots on upgrade
+        .AddMajor(2, "Reference rewrite", v => v.Change<SetWordTextChange>());
+});
+```
+Call `ReconcileModelVersions` when you open a database, before you use it. A database with nothing stored is at version 0. For each type listed by a version newer than the stored one, it invalidates the changes authored with that version or later, and rebuilds the snapshots from the checkpoint before the oldest of them. A major version, or a downgrade, regenerates all snapshots. It throws if an applied version was modified. Querying changes by type is supported on SQLite only.
+```C#
+await dataModel.ReconcileModelVersions();
+```
+Listing a type invalidates only the changes authored by apps on the new version or later. When the new behavior must also apply to older changes, pass `invalidateFrom` with the model version to invalidate from. That covers a released version that changed how a type applies but forgot to list it: its apps still stamped their changes with its own number, so invalidating from that version replays only the changes those apps and later ones authored:
+```C#
+config.ModelVersionBuilder
+    .Add(0, "Before model versions", v => v.Change<SetWordNoteChange>().Change<SetTagChange>())
+    .Add(1, "Word notes", v => v.Change<SetWordNoteChange>())
+    //changed how SetTagChange applies but forgot to list it, its apps authored tags as version 2
+    .Add(2, "Notes again", v => v.Change<SetWordNoteChange>())
+    .Add(3, "Fix tags", v => v.Change<SetTagChange>(invalidateFrom: 2));
+```
+To make sure a released version is not modified by accident, add a snapshot test of `HarmonyConfig.DescribeModelVersions()` to your app. A new version only adds a line at the end of the snapshot.
+
+> [!NOTE]
+> Model versions add an `AuthoredModelVersion` column to `ChangeEntities` and a `LocalState` table, so an EF migration is required.
+
 ## Development
 
 ### SemVer commit messages

src/SIL.Harmony.Benchmarks/BenchmarkWorkloadBuilders.cs

@@ -20,7 +20,14 @@ public static Commit NewCommit(Guid clientId, DateTimeOffset dateTime, params IC
         };
         for (var i = 0; i < changes.Length; i++)
         {
-            commit.ChangeEntities.Add(DataModel.ToChangeEntity(changes[i], i, commit.Id));
+            //the sample declares no model versions, so every change is version 0
+            commit.ChangeEntities.Add(new ChangeEntity<IChange>
+            {
+                Change = changes[i],
+                CommitId = commit.Id,
+                EntityId = changes[i].EntityId,
+                Index = i
+            });
         }
         return commit;
     }

src/SIL.Harmony.Core/ChangeEntity.cs

@@ -13,4 +13,9 @@ public ChangeEntity()
     public required Guid CommitId { get; set; }
     public required Guid EntityId { get; set; }
     public required TChange Change { get; set; }
+    /// <summary>
+    /// The model version the change was authored under: the number of the authoring app's latest model version,
+    /// whether or not that version lists the change's type. Changes from before model versions existed are version 0.
+    /// </summary>
+    public int AuthoredModelVersion { get; set; }
 }

src/SIL.Harmony.Sample/CrdtSampleKernel.cs

@@ -62,6 +62,26 @@ public static void ConfigureSample(HarmonyConfig config)
             .Add<DeleteChange<Example>>()
             .Add<DeleteChange<Tag>>()
             ;
+        config.ModelVersionBuilder
+            .Add(0, "Before model versions", v => v
+                .Change<NewWordChange>()
+                .Change<NewDefinitionChange>()
+                .Change<NewExampleChange>()
+                .Change<EditExampleChange>()
+                .Change<SetWordTextChange>()
+                .Change<SetWordNoteChange>()
+                .Change<SetAntonymReferenceChange>()
+                .Change<AddWordImageChange>()
+                .Change<SetOrderChange<Definition>>()
+                .Change<SetDefinitionPartOfSpeechChange>()
+                .Change<SetTagChange>()
+                .Change<TagWordChange>()
+                .Change<DeleteChange<Word>>()
+                .Change<DeleteChange<Definition>>()
+                .Change<DeleteChange<Example>>()
+                .Change<DeleteChange<Tag>>()
+                //registered by AddCrdtRemoteResources
+                .RemoteResourceChanges<MediaMetadata>());
         config.ObjectTypeListBuilder.DefaultAdapter()
             .Add<Word>(builder =>
             {

src/SIL.Harmony.Tests/Adapter/CustomObjectAdapterTests.cs

@@ -155,6 +155,9 @@ public async Task CanAdaptACustomObject()
             {
                 config.AlwaysValidateCommits = true;
                 config.ChangeTypeListBuilder.Add<CreateMyClassChange>().Add<CreateMyClass2Change>();
+                config.ModelVersionBuilder.Add(0, "Before model versions", v => v
+                    .Change<CreateMyClassChange>()
+                    .Change<CreateMyClass2Change>());
                 config.ObjectTypeListBuilder
                     .CustomAdapter<IMyCustomInterface, MyClassAdapter>()
                     .Add<MyClass>(builder => builder.HasKey(o => o.Identifier))

src/SIL.Harmony.Tests/DataModelTestBase.cs

@@ -62,13 +62,19 @@ public DataModelTestBase(SqliteConnection connection, bool alwaysValidate = true
     }
 
     public DataModelTestBase ForkDatabase(bool alwaysValidate = true)
+    {
+        //the fork has to be configured like its source, otherwise it replays under different settings
+        return ForkDatabase(_configure, alwaysValidate);
+    }
+
+    /// <summary>a copy of this database opened with a different config, like a newer or older version of the app</summary>
+    public DataModelTestBase ForkDatabase(Action<IServiceCollection>? configure, bool alwaysValidate = true)
     {
         var connection = new SqliteConnection("Data Source=:memory:");
         connection.Open();
         if (DbContext.Database.GetDbConnection() is not SqliteConnection existingConnection) throw new InvalidOperationException("Database is not SQLite");
         existingConnection.BackupDatabase(connection);
-        //the fork has to be configured like its source, otherwise it replays under different settings
-        var newTestBase = new DataModelTestBase(connection, alwaysValidate, _configure, _performanceTest);
+        var newTestBase = new DataModelTestBase(connection, alwaysValidate, configure, _performanceTest);
         newTestBase.SetCurrentDate(CurrentDate);
         return newTestBase;
     }
@@ -82,6 +88,8 @@ internal CrdtRepository CreateRepository() =>
 
     internal HarmonyConfig CrdtConfig => _services.GetRequiredService<IOptions<HarmonyConfig>>().Value;
 
+    internal ModelVersionService ModelVersionService => _services.GetRequiredService<ModelVersionService>();
+
     public async ValueTask<Commit> WriteNextChange(IChange change)
     {
         return await WriteChange(_localClientId, NextDate(), change);

src/SIL.Harmony.Tests/DbContextTests.VerifyModel.verified.txt

@@ -36,6 +36,9 @@
     Properties: 
       CommitId (Guid) Required PK FK AfterSave:Throw
       Index (int) Required PK AfterSave:Throw
+      AuthoredModelVersion (int) Required ValueGenerated.OnAdd
+        Annotations: 
+          Relational:DefaultValue: 0
       Change (IChange)
         Annotations: 
           Relational:ColumnType: jsonb
@@ -51,6 +54,19 @@
       Relational:TableName: ChangeEntities
       Relational:ViewName: 
       Relational:ViewSchema: 
+  EntityType: LocalStateEntry
+    Properties: 
+      Key (string) Required PK AfterSave:Throw
+      Value (string) Required
+    Keys: 
+      Key PK
+    Annotations: 
+      Relational:FunctionName: 
+      Relational:Schema: 
+      Relational:SqlQuery: 
+      Relational:TableName: LocalState
+      Relational:ViewName: 
+      Relational:ViewSchema: 
   EntityType: ObjectSnapshot
     Properties: 
       Id (Guid) Required PK AfterSave:Throw ValueGenerated.OnAdd

src/SIL.Harmony.Tests/ModelVersionTests.DescribeModelVersions.verified.txt

@@ -0,0 +1,5 @@
+0 Before model versions: AddWordImageChange, EditExampleChange, NewDefinitionChange, NewExampleChange, NewWordChange, SetAntonymReferenceChange, SetDefinitionPartOfSpeechChange, SetTagChange, SetWordNoteChange, SetWordTextChange, TagWordChange, create:pendingUpload, create:remote-resource, delete:Definition, delete:Example, delete:RemoteResource, delete:Tag, delete:Word, set:remote-resource-metadata, setOrder:Definition, uploaded:RemoteResource
+1 Notes: SetWordNoteChange
+2 major Rewrite: 
+3 Tags and notes: SetTagChange, SetWordNoteChange
+4 Fix tags: SetTagChange invalidates from 1, SetWordNoteChange