Skip to content

feat: BLE scanner + GATT client + MCP WebSocket transport - #9

Open
AdelinoGP wants to merge 15 commits into
navado:mainfrom
AdelinoGP:feat/ble-scanner-mcp
Open

AdelinoGP wants to merge 15 commits into
navado:mainfrom
AdelinoGP:feat/ble-scanner-mcp

Conversation

@AdelinoGP

Copy link
Copy Markdown

Summary

Adds BLE advertising capture for reverse-engineering broadcast formats, exposed over the MCP interface, plus the WebSocket transport that makes the MCP server reachable on the port it advertises.

The GATT client is a hardware-verified state machine built directly on the raw esp_gattc API, replacing the Arduino BLE library's untimeoutable blocking calls.

Highlights

  • BLE advertising scanner — Captures raw payloads from nearby devices, with per-device payload history limited to 32 devices and 8 distinct payloads per device. This makes broadcaster state changes, such as button presses or level changes, visible as payload deltas. Active scans request scan responses to capture advertised names and service data.

  • GATT client — Implements ble/connect, ble/disconnect, and two-phase ble/services/start, ble/services/results, and ble/services/stop enumeration using a raw esp_gattc state machine. Per-operation deadlines are enforced:

    • Open: timeoutMs
    • Discovery: 10 seconds
    • Reads: 5 seconds
    • Notification capture: 3 seconds
    • Close: 2 seconds
  • Async connect — ble/connect returns immediately, while ble/connect/results polls the outcome. The requested timeoutMs is genuinely enforced.

  • MCP WebSocket transport — MCPServer::begin() now starts a WebSocket server on port 9000, which mDNS was already advertising. JSON-RPC frames are routed through processMessage(), and notifications are pushed to subscribed clients.

Note: BLE is compiled for the ESP32-S3 target only. On 4 MB-flash boards, the BLE stack cannot fit alongside the WebSocket transport and OTA. Therefore, the ble/* methods are not built for those boards.

New MCP methods

Method Description
ble/scan Start a non-blocking scan using durationMs.
ble/scan/results Stop the scan and return captured devices and payload history.
ble/scan/stop Stop a continuous scan early.
ble/connect Begin an asynchronous GATT connection using mac and timeoutMs.
ble/connect/results Poll the asynchronous connection outcome.
ble/disconnect Drop the GATT link.
ble/services/start Start two-phase GATT enumeration with notification capture.
ble/services/results Retrieve GATT enumeration results.
ble/services/stop Stop GATT enumeration.

Fixes included

  • lwIP initialization crash — The board previously crashed in a reboot loop before joining Wi-Fi because AsyncServer::begin() called tcpip_api_call() before the TCP/IP stack was initialized. WiFi.mode() now runs first.

  • WDT starvation under BLE load — CORE_DEBUG_LEVEL=5 flooded the UART at 115200 baud. Under heavy radio activity, verbose per-advertisement logging stalled BTC_TASK on core 0, starving async_tcp and triggering a watchdog abort. The default log level is now INFO, and AsyncTCP runs on core 1.

  • Heap budget for BLE — BusHistory previously reserved freeHeap - 64 KB, leaving insufficient space for BLE controller and scan allocations and causing bad_alloc in the scan callback. The heap cap is now configurable through BOARD_BUS_HISTORY_HEAP_CAP:

    • ESP32-S3: 160 KB
    • Other boards: 128 KB
    • 0: Uncapped
  • Payload history never populated — wantDuplicates=false caused the library to discard repeated advertisements before invoking the callback, limiting each device's history to one entry. Scans now request duplicates, while the library still frees each device after the callback. History logic was moved into natively tested core code with eviction and reappearance handling.

  • GATT client defects found on hardware — The raw state machine now:

    • Uses the library's setCustomGattcHandler extension point because esp_ble_gattc_register_callback is one-shot.
    • Registers the GATT application once and verifies REG_EVT status.
    • Treats connection ID 0 as valid by gating on state instead of connection ID.
    • Transitions out of Discovering on SEARCH_CMPL_EVT, allowing the cache walk to run.
  • 4 MB-flash boards — The original BLE commit did not compile on DevKitC, C3, HUZZAH32, or M5Stack because of a FreeRTOS.h include-path issue. BLE, transport, and OTA also exceeded the 1.25 MB OTA application slot. The include path is fixed, BLE is disabled on those boards, and the three classic ESP32 boards now use larger OTA slots through partitions_4mb_ota.csv:

    • 1.86 MB application slots
    • 128 KB SPIFFS
    • Coredump retained
  • JSON-RPC error shape — ble/connect now returns a proper -32602 error envelope when mac is missing.

  • Pre-existing CI blocker — test_mcp_server failed to build because std::find was used without including <algorithm> in the shared mock header. The firmware matrix is gated behind this test.

Testing

  • pio test -e native — 173/173 passing

    • Includes 19 new BLE/GATT core tests covering hex encoding/decoding, connectable-flag heuristics, payload-history deduplication, bounds and eviction, and GATT state, deadline, UUID, and property logic.
  • Full ESP32 matrix passes:

    • ESP32-S3
    • DevKitC V4
    • ESP32-C3
    • HUZZAH32
    • M5Stack
  • nRF52 builds are unaffected.

  • Verified live on hardware over the MCP WebSocket using an ESP32-S3-WROOM:

    • Six-device scan with payload history
    • Asynchronous connection to a Galaxy Watch Ultra in under 2 seconds
    • Five-service GATT enumeration with readable device name, model, and firmware values
    • Clean disconnect
    • Successful reconnect

Documentation

The README now includes a Bluetooth (BLE) section under the MCP API Reference, covering:

  • Request and response examples
  • Asynchronous connection and ble/connect/results
  • Enforced per-operation timeouts
  • Payload-history semantics, including the bounded eight-entry history and retained-entry count
  • The ESP32-S3-only BLE limitation

Checklist

  • Native tests pass: 173/173
  • All ESP32 board builds pass
  • Verified live on hardware: scan, connect, enumerate, disconnect, and reconnect
  • nRF52 builds unaffected

Attribution

This implementation was developed using Agentic AI with human decision-making, OpenCode, and DeepSeek V4 Flash 0731 with Max reasoning.

AsyncTCP calls tcpip_api_call() in AsyncServer::begin(), which asserts
with "Invalid mbox" if the TCP/IP stack (esp_netif_init) has not been
initialized. WiFi.mode() is what triggers the low-level init in the
Arduino core; previously the web server was started before any WiFi
call, crashing the board in a reboot loop before it could join the
network.
std::find() is used in MockWebSocketServer::removeClient() but <algorithm>
was never included; the native test_mcp_server suite failed to build, and
CI's firmware matrix is gated behind the native tests.
CORE_DEBUG_LEVEL=5 floods the UART at 115200 baud; on a busy BLE radio
the verbose per-advertisement logging stalls BTC_TASK on core 0 while it
waits for the TX FIFO to drain, starving the async_tcp task and tripping
the watchdog. INFO is the new default (opt-in for targeted debugging),
and AsyncTCP runs on core 1 so BLE can never starve it.
The auto-config budget (freeHeap - 64KB) leaves Bluetooth no heap
headroom, causing bad_alloc in the scan callback and BTU_StartUp
failures.  Each ESP32 board now declares BOARD_BUS_HISTORY_HEAP_CAP
(S3: 160KB, others: 128KB); 0 means uncapped (legacy behaviour) via the
board_config.h fallback.
MCPServer::begin() now starts a WebSocketsServer on port 9000 (the port
mDNS was already advertising), handleClient() pumps the event loop, and
text frames are routed through processMessage().  Notifications are
pushed to subscribed clients via sendFunc_.  All ESP32-specific code is
NATIVE_TEST-guarded so the native test suite stays hardware-free.
Passive BLE capture for reverse-engineering broadcast formats:
- ble/scan / ble/scan/results / ble/scan/stop: non-blocking scans with
  per-device payload history (32 devices x 8 distinct payloads)
- ble/connect / ble/disconnect: GATT client
- ble/services/start|results|stop: two-phase GATT enumeration on a
  dedicated FreeRTOS task so a slow peer can never block the MCP task
- payloadIsConnectable + hex helpers in BLEScannerCore.h (natively
  unit-tested); gated by BOARD_HAS_BLE and excluded from nRF52/native
  builds via build_src_filter
Features-table row plus a 'Bluetooth (BLE)' section under the MCP API
Reference covering ble/scan, ble/scan/results, ble/connect and the
two-phase ble/services/* flow, with request/response examples and
notes for reverse-engineering broadcast payloads.
wantDuplicates=false made the BLE library drop repeat advertisements
for a known MAC before the callback fired, so recordPayload() never
saw them and per-device payload history was capped at one entry — the
state-change signal the feature exists for was never captured.

- scan with wantDuplicates=true (the library frees each advertised
  device after the callback, so the heap stays bounded)
- move BLEAdvReport/MAX_DEVICES/MAX_PAYLOADS and recordPayload() into
  BLEScannerCore.h so the history logic is natively unit-testable
- evicted payloads are removed from the seen-set, so a re-broadcast of
  an evicted payload is recorded again and the set stays bounded
- native regression tests: distinct append, duplicate skip, 8-entry
  eviction, evicted-payload reappearance
…machine

The 2.0.17 Arduino BLE library has no timeouts on its discovery/read
semaphores, so a slow or uncooperative peer blocked the MCP task
forever; the 10 s cap only bounded the waiter while the task itself
could hang, and disconnect() could free the client underneath it.

- drive esp_gattc directly on a dedicated task with per-operation
  deadlines: open (connect timeoutMs), discovery (10 s), reads (5 s),
  notify capture (3 s), close (2 s)
- event-driven: Bluedroid callbacks only record state under the
  mutex; the task is the only issuer of controller calls, and never
  holds the mutex across them; events are scoped to our app id /
  interface / conn id
- failed lifecycles route through Closing so a late OPEN_EVT cannot
  leak the link; the task exits only from Idle
- async connect: ble/connect returns immediately, ble/connect/results
  polls the outcome (timeoutMs is now genuinely enforced)
- notify subscription via CCCD writes; reads via get_db cache walk
  with 128/32/16-bit UUID and properties formatting
- new BLEGattCore.h holds the pure state/deadline/format helpers with
  a native test suite (test_gatt_core)
…d timeouts

- 'Passive' -> 'Active' (the scanner uses active scans, which request
  scan responses; the docs promised passive capture the code never did)
- document ble/connect/results and the async connect semantics, and
  that timeoutMs is enforced
- clarify payloadHistory/count: bounded to 8 retained entries, oldest
  dropped on eviction, count is the number currently retained
- document the per-operation bounds: 10 s discovery cap, 5 s per-read,
  notifyCaptureMs capture window, safe teardown via services/stop
…ards

The BLE stack (Bluedroid/NimBLE) plus the MCP WebSocket transport and
OTA cannot fit the 1.25 MB default OTA app slot on 4 MB-flash boards
(devkitc, c3, huzzah32, m5stack): the original BLE commit never even
compiled there (FreeRTOS.h include path) and the branch overflowed by
~800 KB with BLE linked.

- include <freertos/FreeRTOS.h> instead of <FreeRTOS.h> so classic
  ESP32 SDKs resolve it (S3 SDK adds the extra include dir, others do
  not) — the board matrix now builds everywhere
- gate BLE off on the 4 MB boards (BOARD_HAS_BLE 0 + exclude
  BLEScanner.cpp from their builds, same pattern as nRF52) since the
  hardware supports it but the flash budget does not; S3 keeps BLE
- custom partitions_4mb_ota.csv for devkitc/huzzah32/m5stack: 1.86 MB
  OTA slots (was 1.25 MB) + 128 KB spiffs + coredump, fitting the
  transport at 70% with headroom; c3 keeps the default layout (RISC-V
  libs are smaller and it already fits)
- guard bleScanner.begin() with BOARD_HAS_BLE in main.cpp
- README: board matrix and BLE section reflect the S3-only BLE build
…d on hardware

Hardware testing against a Galaxy Watch Ultra exposed four real defects
that unit tests could not catch:

- esp_ble_gattc_register_callback() silently fails after BLEDevice::init
  already registered the Bluedroid handler (one-shot registration), so
  the scanner never saw REG_EVT/OPEN_EVT and every connect timed out.
  Hook via the library's official extension point
  (BLEDevice::setCustomGattcHandler) instead.
- The GATT app registration survives across connections; re-registering
  on the second connect returned ESP_GATT_ERROR and the open was never
  issued. Track appRegistered_ and only register once; verify
  REG_EVT status before accepting the interface.
- conn_id 0 is legitimate on this stack (first GATT connection), so
  gating discovery/close on connId_ != 0 deadlocked the state machine
  at Connected. Gate on state instead.
- SEARCH_CMPL_EVT set loadPending_ but never left Discovering, so the
  task loop only ever exited via the 10 s discovery timeout with empty
  results. Transition to Connected so the load runs.

Verified on hardware over MCP WebSocket: scan (6 devices), async
connect + poll, 5-service GATT enumeration with readable values,
disconnect, and reconnect all pass end-to-end.
ble/scan/results serialised the whole capture as one JSON response while
the scanner stored up to 32 devices x 8 payloads unbounded.  Under a busy
BLE radio the response grew past available heap, std::bad_alloc escaped
the MCP task through the WebSockets library, and std::terminate aborted
the board - dropping every client with 'ESP32 connection closed' and
empty follow-up scans (reproduced live: abort() on core 1, backtrace
into the ble/scan/results handler).

- BLEScannerCore: bound capture storage to 12 devices x 4 payloads,
  cap payloads at 64 bytes (truncated before dedupe so the history and
  the serialised response stay small even under flood).
- MCPServer::dispatch: catch handler exceptions and reply with a
  JSON-RPC -32603 error instead of letting the exception escape and
  reboot the device.

Verified on hardware: the flood test that crashed the old firmware now
returns clean captures across repeated scan cycles with no reboots.
Raise the per-device history cap from 4 to 64 distinct payloads so a full
command set (levels x channels + heater + patterns) can be captured in a
single scan, and keep the unfiltered ble/scan/results response small by
serialising only the newest 4 payloads per device (count reports the true
distinct total).

- ble/scan/results?mac=<addr> (case-insensitive) returns ONLY that device
  with its FULL payload history; unseen MACs return an empty devices array.
- Storage: 12 devices x 64 payloads x 64-byte cap (~98 KB pathological
  worst case, a few KB realistic) — the existing -32603 OOM backstop
  covers the worst case without rebooting.

Verified on hardware: unfiltered (7 devices, newest-4 view), mac fetch
(full history), case-insensitive match, and unseen-MAC all behave as
specified.
… scan

The phone rotates its BLE MAC every ~1s, so per-device payload history
never accumulates (each rotated MAC carries 1 payload).  The new registry
collapses all broadcasters into the actual command sequence:

- ble/scan/payloads: MAC-independent distinct payload registry, first-seen
  order, each entry with count + firstSeen/lastSeen (ms relative to scan
  start).  Recorded in onAdv so it survives device FIFO eviction.  Verified
  on hardware: 6 distinct commands captured across 8 rotated MACs in one
  45s scan.
- ble/scan/poll: stateless live read of the registry without stopping the
  scan (diff by firstSeen); pairs with continuous scans.
- ble/scan: durationMs 0 = continuous until ble/scan/stop; already-running
  scans report {started:false, scanning:true, remainingMs}.
- ble/scan/results: AND-combined response filters serviceUuid /
  manufacturerId / macPrefix (never affect capture); results carry
  scanStartedMs/durationMs.

Storage: registry bounded to 64 payloads x 64 bytes (~9KB worst case).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant