mapsight-pulp is a lightweight PHP ETL library for geo and traffic data — not a full application backend. Mapsight frontends fetch GeoJSON from static files (or CMS URLs) that pulp produces on a schedule.
| Scenario | CMS-only | Pulp | Platform |
|---|---|---|---|
| Static GeoJSON files edited in CMS | ✓ | — | — |
| Merge CMS GeoJSON + external HTTP feed | — | ✓ | — |
| Domain transforms (Concert, TIC, KML→GeoJSON, CSV) | — | ✓ | — |
| Time-series stations, Count Aggregator API, admin imports | — | — | ✓ |
Rule of thumb: pulp when you need fetch → transform → write pipelines without standing up a database-backed API. Use the optional data platform when you need structured imports, station models, and authenticated admin.
flowchart LR
Ext["External feeds APIs files"]
CMS["CMS GeoJSON"]
Cron["systemd timer or cron"]
Pulp["pulp CLI script"]
Files["Static GeoJSON on disk"]
Web["Apache nginx"]
App["Mapsight embed or SPA"]
Ext --> Pulp
CMS --> Pulp
Cron --> Pulp
Pulp --> Files
Web --> Files
App -->|" HTTP GET static file "| Web
Pulp uses a Gulp-inspired stream pipeline: fetch handlers → transform handlers → write output. Domain logic
lives in Composer packages (pulp-geojson, pulp-concert, pulp-tic, pulp-geocsv, etc.).
Default (production):
- Install
mapsight/pulpvia Composer in a host directory. - Add per-use-case CLI scripts (e.g.
standplan.php,events.php) that run the pipeline and write GeoJSON to disk. - Schedule with systemd timer or cron (interval depends on feed — minutes to daily).
- Serve written files as static assets from Apache/nginx (same maps host or CMS path).
- Reference the stable URL in Mapsight
featureSourcesor loader config.
Benefits: predictable load, simple caching, no PHP worker per browser request, secrets stay off the request path.
Avoid for production: exposing pulp scripts directly as live HTTP endpoints that transform on every request. That pattern was used only to stand up demos quickly — it adds latency, complicates caching, and runs ETL in the web tier.
From the embed/SPA perspective, pulp output is ordinary static GeoJSON over HTTP:
- Configure URL in declarative Redux JSON / Zod-validated embed config
- Refresh when the scheduled job updates files (or on page load — no live ETL round-trip)
- Same-origin or CORS as for any static GeoJSON asset
Pulp does not run inside the browser or the Node monorepo build.
For polling clients that use Mapsight’s opt-in feature-source document cache, prefer a
features-stable weak ETag plus Cache-Control: no-cache so unchanged feature
collections can 304 without depending on file Last-Modified. See
FEATURE_SOURCES.md.
| Concern | Guidance |
|---|---|
| Scheduling | systemd timer or cron; monitor job success/failure |
| Output paths | Stable URLs for embed config; atomic writes (temp file + rename) where possible |
| Caching | Long cache headers on versioned or timestamped files; shorter cache if URLs are fixed |
| Errors | Failed jobs should alert ops; last good file often remains served |
| Secrets | API keys only in server-side pulp config — never in embed JSON |
Pulp fits as a specialized transform job in a CIVITAS stack — not a replacement for NiFi/Redpanda-style pipeline
engines. See the upstream mapsight-pulp monorepo for
civitas-core-pulp-integration.
- GitHub: github.com/open-mapsight/mapsight-pulp
- Package: Composer
mapsight/pulp - License: See upstream
LICENSE
Monorepo packages do not wrap pulp — integration is HTTP to static GeoJSON at the deployment boundary.
- FEATURE_SOURCES.md — xhr-json cache, ETag / If-None-Match contract
- TILE_PROXY.md — basemap tiles (companion PHP service)
- DATA_BACKEND.md — when CMS + pulp are not enough
- Integration overview
- Ecosystem
- Decision 003 — GeoJSON-first