Skip to content

feat: cursor-based streaming for CSV export at scale (100K+ rows) #126

Description

@kvithayathil

Problem

_generate_csv_rows materializes all match results via list(yield_per(...)) before generating rows. This works at current scale (<10K) but will hit memory limits at 100K+ rows.

Approach

  • Replace materialized list with cursor-based pagination through _generate_csv_rows
  • Stream CSV rows via StreamingResponse with backpressure
  • Yield rows in batches to avoid unbounded memory growth
  • Preserve existing export format and behavior

Sub-tasks

  • Design streaming API contract (chunk size, cursor state)
  • Implement cursor-based iteration in _generate_csv_rows
  • Wire StreamingResponse in campaign router export endpoint
  • Integration test with 100K+ rows (in-memory SQLite benchmark)
  • Verify memory stays bounded under load

Context

Identified as deferred follow-up from PR #124 (cursor-based keyset pagination). Low priority — current scale is <10K rows.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions