Progressive, lossless RGB image refinement over binary IIBIN WebSocket streams. MIT licensed, with Python encoding and TypeScript decoding.
Watch the demonstration on YouTube.
Requires Python 3.12+, a C compiler for the vendored IIBIN extension, and Node.js 22+.
npm ci
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/python demo/prepare.py
npm run build
.venv/bin/python demo/server.py --port 5270Open http://localhost:5270. Windows users can use the corresponding .venv/Scripts/python executable. Preparation encodes the bundled source images and can take several minutes.
Preparation runs once and produces reusable streams, lossless reference PNGs and a measured manifest under ignored assets/generated/. Stop the demo with Ctrl+C. The server listens only on loopback. An isolated aiohttp server serves the prepared PNGs and the binary IIBIN WebSocket at /ws on the same port. No production routes, database, installer, registry or proxy changes are needed.
The browser needs Canvas 2D and service workers; use a current Chromium browser on localhost. The service worker is confined to this demo's origin. It observes each native image request once, forwards the original bytes and records prefixes. Requests identify their method and stage, so preview and full-image bytes are summed without mixing the comparison columns or downloading references twice.
The compact workbench shows two images side by side for the selected preset:
- Left: iiStreamImage. A genuinely downscaled PNG of at most 20 KiB loads first. It is displayed immediately and also seeds the decoder. The WebSocket stream then carries only correction data that progressively sharpens the image; it sends no coarse color fills or quadtree approximations.
- Right: PNG · HTTP. The full PNG uses the browser's normal image-loading path.
The selector offers one large 4K architecture image and two smaller landscape/portrait PNGs just below 2 MiB. The left-hand byte total includes its preview and correction stream. Both variants finish with identical full-resolution RGB pixels. First-visible on the left refers to the real loaded preview, not an artificial blur or placeholder.
No image data is fetched and no WebSocket opens before Start. Start launches both methods from a shared timestamp. Reset closes the image socket, aborts native image requests, clears images, metrics and snapshots, disposes buffers and revokes URLs. Changing image, bandwidth or method resets without downloading. Each side also supports an independent run.
The stream uses eight progressively increasing WebSocket messages: an immediate 1 KiB start packet, then seven larger packets with approximately doubling sizes. Byte pacing applies the selected rate; large messages do not get a free initial burst. The metric separates WebSocket packets from internal correction operations.
Click any image to open the left-side inspection modal. Three columns and two rows show received-data checkpoints of 1 KiB, 20 KiB, 100 KiB, 250 KiB, 2 MiB and 5 MiB. The suffix KB in the original request is interpreted as binary KiB. The left-hand budgets include preview bytes, then correction-stream bytes. A screenshot combines the actual preview with the corrected regions that have arrived. Decoder state includes only complete correction frames within each prefix. A frame crossing the byte boundary is retained but not applied. Snapshot display is downsampled to 768 × 432; decoding and quality measurement use the selected image’s full-resolution pixels. PNG prefixes are shown only if the browser can decode them. A non-decodable prefix is reported, never replaced with a fabricated blurred image.
First visible on the codec side records the real preview becoming visible. PNG's first paint and decoder CPU time are not exposed reliably by native <img> and are shown as unavailable. Completion is observed separately for each method. “Decode wall” measures time inside awaited decoder calls, including decompression scheduling; it is not a CPU profiler measurement. Snapshot canvas capture, rendering, event-loop contention and network time remain included in end-to-end completion. PNG encoding for stored snapshots runs asynchronously and does not hold up the stream. Codec PSNR in the live gallery refers to the last reached correction-stream checkpoint, excluding preview bytes; it describes the deterministic full-resolution decoder buffer. The final target is pixel equality for both formats.
All demo files and data endpoints send Cache-Control: no-store; every run also uses a new identifier. Server-side preparation is independent of HTTP/browser caching. No external image requests, inference or telemetry occur during tests.
python/ii_stream_image/analysis.py: integral-image RGB means and squared error; adaptive rectangle statistics.python/ii_stream_image/encoder.py: deterministic quadtree frontier prioritized by error reduction divided by actual serialized operation bytes. Candidate costs are rechecked when sequence varint widths change. The default weighting is uniform squared RGB error. The optionalpriority(gain, byte_count, rectangle)callback allows spatial weighting or alternative policies. Residual tiles are ordered by the same benefit-per-byte policy, rather than raster order.frame-schema.jsonandprotocol.py: versioned IIBIN frame schema and bounded record iterator.typescript/decoder.ts: transport-independent incremental decoder; persistent RGBA state; bounds, sequence and decompression checks.typescript/renderer.ts: Canvas 2D renderer coalescing changed rectangles into one dirty region per animation frame.demo/prepare.py,server.py: preparation and isolated delivery.demo/inflate.ts: a bounded synchronous inflater adapter using Pako, without decompression scheduling per tile. The library also exposes its nativeDecompressionStreambackend; callers can inject either through the decoder/renderer constructor.typescript/socket.ts: binary WebSocket consumer with IIBIN start/acknowledgement messages and bounded receive queue.demo/main.ts,sw.js: paired workbench, byte-prefix capture, native browser image observer.
/ws upgrades to a WebSocket only when Start is pressed. The first message is binary IIBIN using control-schema.json: op=1, image ID, rate in KiB/s (zero for unrestricted), and format (codec or codec8). The regular stream uses at most 16 KiB per message. codec8 sends 1 KiB immediately, then divides the remaining exact codec bytes into seven progressively larger messages (approximately doubling sizes), each bounded to 8 MiB. WebSocket message boundaries need not match codec frame boundaries. The existing incremental decoder assembles them normally.
After decoding each 128 KiB, and after the last byte, the browser sends a binary IIBIN acknowledgement (op=2, cumulative consumed offset). The regular stream allows at most 512 KiB of unacknowledged payload; the eight-message stream allows a byte window twice the largest message, so a slow decoder cannot accumulate the entire image in a WebSocket event queue. Reset closes the socket and cancels its sender. Normal completion waits for the final acknowledgement and closes with code 1000. Malformed controls, unknown images and invalid acknowledgement offsets close with code 1008. Per-message WebSocket compression is disabled because residuals are already compressed. Shutdown closes active demo sockets.
There is no HTTP .iisi image route and no HTTP fallback for the left-hand image. PNG references use the normal HTTP route. All paths use the selected transfer rate; the eight-message variant differs in message size. Metrics count image payload bytes; HTTP/WebSocket headers and client-to-server controls are outside that count on both comparisons. The demo uses the included IIBIN codec and schema APIs without requiring another application.
Each record is encoded with IIBIN Schema({'frames': bytes, repeated: true}). Its one frames value is another IIBIN message described by frame-schema.json. Records are concatenated; the incremental reader recognizes the existing IIBIN field tag and varint length before passing each frame to the existing schema decoder. This is IIBIN framing, not a second image transport. Python and TypeScript share the exact JSON field definition.
op and seq are required. Sequence starts at zero and increases by one. IMAGE_INIT (1) declares version 2, integer dimensions and RGB8/sRGB; FILL (2) fills an integer rectangle with three RGB bytes; SPLIT (3) fills four dyadic children with twelve RGB bytes in TL, TR, BL, BR order; PATCH (4) replaces a rectangle with raw RGB; RESIDUAL (5) inflates zlib RGB differences, reverses the declared spatial predictor and adds modulo 256 to the existing reconstruction; IMAGE_END (6) completes the image. Version 2 deliberately has no separate gradient operation. Quadtree means, then residuals, provide the continuous path to lossless quality.
The demo emits version 3, seeded by its separately loaded preview. The low-resolution preview is an opaque, palette-quantized PNG, so Python and browser decode the same RGB values. The seed expands deterministically with integer nearest-neighbor coordinates: source floor(x × preview_width / width), floor(y × preview_height / height). INIT declares version 3; the stream contains only INIT, RESIDUAL/PATCH and END. The decoder rejects FILL/SPLIT in seeded streams and rejects a missing, malformed or transparent seed. The preview is not duplicated in the WebSocket payload.
The native preview remains beneath a transparent canvas. Only completed correction rectangles are drawn onto the canvas. Dirty rectangles are kept separate, so gaps between corrected regions do not expose an uncorrected buffer. END renders the full exact pixel state, including regions that needed no residual. The library retains its version-2 adaptive-quadtree encoder for independent comparison tests; the demo always uses the seeded path.
Residual frames declare predictor: 0 for no spatial filter, 1 for Sub (left RGB pixel), 2 for Paeth (left, upper and upper-left RGB values). Predictor state resets at each tile boundary. The encoder compares actual zlib sizes and chooses the smallest result per tile. Tiles default to 256 × 256 instead of 64 × 64, reducing decompression calls by roughly sixteen times. Version 2 introduced predicted residuals; version 3 adds the external preview baseline.
Limits: dimensions ≤8192 each, ≤12 million pixels, frame payload ≤1 MiB, inflated patch ≤1 MiB, decoder input chunk ≤2 MiB. Call feed() sequentially, split larger transport chunks, then call finish() to detect truncation. Repeated init, unknown operations, malformed envelope, sequence gaps, invalid bounds, residual size mismatch and trailing bytes are errors. Dispose a failed or completed instance. Independent instances share no pixel state. Residual tiles are nonoverlapping and sent only after all quadtree operations, preserving their exact baseline. No application checksums are used.
RGBA state costs four bytes per pixel, plus the canvas backing store and transient frame/decompression data. A renderer releases its retained pixel buffers after completion or cancellation; the visible canvas remains. The demo bounds snapshot dimensions and revokes their URLs on restart. Quadtree analysis uses larger float64 integral images only during offline preparation.
The regression source set has six freshly generated subjects covering landscape, portrait, architecture, detailed nature, interior and graphic/text content. With explicit user approval, generated 1672 × 941 images are upscaled using Lanczos to 3840 × 2160. Upscaling does not add native photographic detail. Every losslessly compressed source PNG exceeds 5 MiB; no padding or artificial noise is added. Exact sizes and transformations are recorded in assets/sources/provenance.json. PNG references and codec streams contain the same RGB pixels. The tiny previews are physically resized and palette-quantized to a 20 KiB PNG budget. The two approximately 2 MiB demo presets are derived offline from the landscape and portrait sources with Lanczos resizing; the highest integer width fitting the actual compressed PNG byte budget is selected. They are smaller images, not padded files.
See the measured benchmark, raw browser results and checkpoint screenshot. This prototype prioritizes early recognizability; its filtered block residuals currently cost slightly more bytes than optimized PNG, and its JavaScript reconstruction takes longer than native PNG decoding on an unrestricted local connection. Neither compression savings nor speedups are assumed. Source upscaling, parallel browser activity and local hardware affect the measurements.
npm run build
npm run typecheck
npm test
# With the demo running and Chrome installed:
npm run test:browser
.venv/bin/python demo/benchmark.pySet CHROME to your Chrome/Chromium executable (default /opt/google/chrome/chrome), and II_STREAM_URL to the running demo URL if using another port. The browser test uses Node's built-in WebSocket implementation. Set II_STREAM_PYTHON to override the test runner's Python executable.
Tests cover deterministic flat/gradient/detailed/odd-size images, detail priority, all six 4K lossless round trips, actual Python → IIBIN → TypeScript interoperability, bytewise chunking, malicious frames and residual inflation, truncation, parallel instances and disposal. The real-browser check verifies the two-image layout, compact header and all three presets, no image requests before Start or after selecting a preset, native PNG byte accounting, the actual <=20 KiB preview, transparent correction overlay, actual WebSocket 101 upgrades, binary image and control messages, absence of HTTP codec downloads, exact checkpoint budgets, pixel equality at full resolution, Reset during a run and protection against late callbacks, a single-side run, the 3×2 left modal, mobile width, matching throttled transfer limits, no-store headers and uncaught errors. Six separate asynchronous server tests verify identical codec bytes for all presets and both packetization modes, including exactly eight increasing messages and an immediate small first packet under throttling, acknowledgement-window backpressure, malformed controls, invalid offsets and native PNG/HTTP codec-route behavior. The runner includes them through the demo venv (override with II_STREAM_PYTHON). The browser test writes its report and screenshot. Browser timing thresholds are intentionally not treated as codec correctness assertions.
MIT. The minimal IIBIN implementation is included under vendor/iibin/ with its original MIT license. No API key or external service is required. The image codec is experimental; the included measurements show its current tradeoffs against native PNG.