Skip to content

BacDive and MediaDive serialize edge provenance as Python list text (4,274,895 merged rows) #1070

Description

@realmarcin

Evidence: data/review-20260914.cvXJSA/REVIEW.md, snapshot data/merged/20260914/merged-kg.tar.gz, SHA-256 0fb80d5d865d0da503d9c759286c91216b3dab5f5cc5df0a8e759cd4e233ec81 (3,353,182 nodes / 15,295,064 edges). Counts describe the immutable reviewed archive, not a rebuilt output.

The full archive has 4,274,895 primary_knowledge_source values containing Python-list strings, including 21,964 with outer pipe unions. Both producers already emit the malformed strings (BacDive 4,238,128; MediaDive 55,254). This is not the actual-list-object merge regression. See provenance-review.md for source/merged examples.

Replace Python repr serialization in both producers with consistent KGX pipe-token serialization, retaining infores:bacdive and the contributing bacdive record IDs. Update the #688/#741 test that deliberately pins the old representation, preserving its cross-transform equivalence intent. Do not eval untrusted graph values or silently erase record-level attribution. Reconcile with #1061 so scalar evidence fields are not pooled into invalid enum lists.

Acceptance: exact writer tests, BacDive/MediaDive byte-equivalent shared assertions, round-trip through real KGX preserves distinct provenance tokens with no repr fragments; offline fixtures; rebuild verification separately records counts.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions