feat(3jane): monitor Accountable Proof of Solvency

monitoring.yaml

@@ -43,6 +43,10 @@ protocols:
         description: "Alert-once when IS_PAUSED is true; re-armed after a monitoring gap >3 hours"
       - name: "Borrower Default Watch"
         description: "Envio-backed MorphoCredit borrower watch using live ProtocolConfig timing; MEDIUM alerts when unpaid obligations become delinquent after grace or reach default"
+      - name: "Proof of Solvency"
+        description: "Accountable collateral ratio <95% (CRITICAL) or <99% (HIGH)"
+      - name: "Proof of Solvency Freshness"
+        description: "MEDIUM when a short-cadence report/source misses 2 periods or a >1h cadence misses 1; HIGH on first unavailable run, CRITICAL on the second consecutive miss"
       - name: "Timelock"
         description: "CallScheduled events from 24h and 7-day TimelockControllers via Envio"
 

protocols/3jane/README.md

@@ -14,6 +14,7 @@
 - **Nominal sUSD3 Backing Floor:** `ProtocolConfig.config(keccak256("SUSD3_NOMINAL_BACKING_FLOOR"))` vs cached prior. Alerts on any change (governance lever). Separate alert-once when the floor exceeds sUSD3's USD3 holdings valued in USDC — sUSD3 redemptions can be blocked while floor > backing.
 - **Protocol Pause:** `ProtocolConfig.config(keccak256("IS_PAUSED"))`. Alert-once on transition to true. Distinct from per-vault `isShutdown()` — pauses the underlying credit market.
 - **Borrower Default Watch:** optional Envio-backed borrower default risk feed. The Envio indexer maintains `ThreeJaneBorrowerMarket` rows from MorphoCredit events, and the monitor computes the current delinquent/default status at runtime. Alerts are **MEDIUM only** and deduped per borrower/cycle/default milestone.
+- **Proof of Solvency:** [Accountable](https://accountable.3jane.xyz/) collateral ratio (reserves / liabilities). Alerts **CRITICAL below 95%** and **HIGH below 99%**, plus freshness and availability alerts. See [Proof of Solvency](#proof-of-solvency) below.
 
 ## Key Contracts
 
@@ -41,6 +42,10 @@
 | Nominal floor breach | Floor > sUSD3 backing valued in USDC (alert-once) | MEDIUM |
 | Protocol paused | `IS_PAUSED` transitions to true (alert-once) | CRITICAL |
 | Borrower delinquent/default watch | New milestone: delinquent, ≤14d, ≤7d, ≤3d, ≤1d, default | MEDIUM |
+| Accountable collateral ratio | < 95% (band transition) | CRITICAL |
+| Accountable collateral ratio | < 99% (band transition) | HIGH |
+| Accountable feed stale | Short cadence >2 periods; long cadence >1 period (alert-once) | MEDIUM |
+| Accountable feed unavailable | First consecutive miss HIGH; second CRITICAL (then quiet until recovery) | HIGH / CRITICAL |
 | Monitoring run failure | Uncaught exception in `main()` | LOW |
 
 ## Cache Freshness
@@ -82,13 +87,41 @@ The indexer should populate/update that entity from `SetCreditLine`, `Borrow`, `
 
 The current countdown and alert bucket are intentionally computed in this monitoring script, not in Envio, because they depend on wall-clock time and governance-controlled timing. This also prevents an indexer environment value from drifting away from the live contract configuration.
 
+## Proof of Solvency
+
+[Accountable](https://docs.accountable.capital/accountable-documentation/proof-of-solvency) publishes a TEE-attested Proof of Solvency dashboard for 3Jane (feed id `100000026`). The human-readable UI is at `https://accountable.3jane.xyz/` (override with `THREE_JANE_ACCOUNTABLE_MESSAGE_URL`); the JSON report is at `https://accountable.3jane.xyz/dashboard` (override with `THREE_JANE_ACCOUNTABLE_URL`). No API key is required.
+
+The client lives in [`utils/accountable.py`](../../utils/accountable.py) and is keyed by data feed id (DFID), so other Accountable feeds can be added without a rewrite. The request is URL/type-based and neither sends nor echoes the DFID, so feed identity is bound explicitly in config.
+
+### Ratio is recomputed, not read
+
+The API rounds `collateralization` to six decimals. Near the alert boundary that is a missed-critical-alert path: a true ratio of `0.9499996` would present as `0.95` and pass a `< 0.95` test. The monitor therefore computes the ratio from `total_reserves / total_supply` at full precision and uses the reported field only as a consistency cross-check (tolerance ≥1e-6, since the server's own rounding sets the floor).
+
+`net` and `collateralization` are defined against *liabilities*, which equal `total_supply` only for a USD-pegged feed. The client asserts `total_supply.fx == 1` when the field is present. The live response currently omits it, so that path independently derives liabilities from `total_reserves - net` and requires them to match raw supply; a non-pegged feed still fails loudly instead of silently comparing against the wrong denominator.
+
+### Freshness is per source, not global
+
+A fresh aggregate timestamp does not prove every input is fresh, and this matters more than usual here: `reserves_split` is essentially all "Morpho Credit", of which the bulk is off-chain loan receivables priced by manually uploaded document reports. Those routinely run past their declared cadence.
+
+The aggregate report and each required source use their declared cadence. Cadences of one hour or less get one missed-period allowance and become stale after two periods; longer cadences become stale as soon as the first expected update is late. The aggregate cadence comes from `reserves.interval`; source cadences come from each source's `frequency`. This means `15 MIN` becomes stale after 30 minutes, hourly after 2 hours, daily after 24 hours, and weekly after 7 days. A source whose `lastUpdated` is in the future is treated as unusable rather than clamped to "fresh", which would defeat the check. Unknown additional sources with an unrecognised cadence are skipped rather than flagged, so a schema addition on Accountable's side cannot spuriously page us.
+
+The 3Jane dashboard UI declares `Slope - Forward Flows` as weekly, while older `/dashboard` JSON responses reported it as daily. The feed configuration therefore binds that source to `WEEKLY`; stale alerts display the effective cadence used by the monitor.
+
+The four known 3Jane sources are required, and a missing or malformed freshness record for one of them makes the feed **stale**, not unavailable. Freshness can no longer be established, but the collateral ratio itself is unaffected — so the report is still returned and the sub-95% check still runs. An upstream source rename degrades the feed to a MEDIUM staleness alert; it cannot silently disable the CRITICAL solvency check.
+
+### Ratio alerts
+
+HIGH fires once when the ratio drops below 99%; CRITICAL fires once on the first reading below 95%. Each severity stays quiet until the ratio recovers above its threshold. Recovering from below 95% into the 95–99% band re-arms CRITICAL without a second HIGH. Re-polling a frozen report cannot re-alert.
+
+The 95%/99% bands are temporary test thresholds while Accountable's report excludes 3Jane idle funds. Recalibrate both thresholds when idle funds are included in the reported reserve totals.
+
 ## Alert dispatch
 
-Alerts use the structured `send_alert` path. HIGH and CRITICAL alerts invoke the default emergency-dispatch hook after Telegram delivery, and `3jane` is enabled in `utils.dispatch.DISPATCHABLE_PROTOCOLS`.
+Alerts use the structured `send_alert` path. HIGH and CRITICAL alerts invoke the default emergency-dispatch hook after Telegram delivery, and `3jane` is enabled in `utils.dispatch.DISPATCHABLE_PROTOCOLS`. Accountable ratio, freshness, and availability alerts use the same `3jane` protocol key as the onchain checks — a second Accountable feed is a new `AccountableFeedConfig` with that protocol's key, not a special-case name.
 
 The sender posts a signed `emergency_withdrawal` webhook using protocol key `3jane`. Dispatch requires `LIQUIDITY_WEBHOOK_SECRET`, is skipped in `LOG_LEVEL=DEBUG`, and has a 60-minute per-protocol cooldown. The receiving liquidity-monitoring deployment must independently map `3jane` to the vaults, collateral names, and markets whose caps should be zeroed.
 
-Only HIGH and CRITICAL alerts dispatch. LOW and MEDIUM alerts—including insurance-fund outflows—remain Telegram/database alerts only.
+Only HIGH and CRITICAL alerts dispatch. LOW and MEDIUM alerts—including insurance-fund outflows and Accountable staleness—remain Telegram/database alerts only.
 
 ## Governance
 

protocols/3jane/main.py

@@ -19,6 +19,9 @@
 - Debt cap changes — alerts when ProtocolConfig debt cap is modified
 - Nominal sUSD3 backing floor — alerts on change and when floor > sUSD3 backing
 - Protocol-wide pause — alerts once when ProtocolConfig IS_PAUSED flips to true
+- Accountable Proof of Solvency — collateral ratio thresholds plus feed freshness
+  and availability. HIGH and CRITICAL use the same ``3jane`` protocol key as the
+  onchain checks, so they dispatch the same way.
 """
 
 import json
@@ -27,11 +30,13 @@
 import urllib.request
 from dataclasses import dataclass
 from datetime import datetime, timezone
+from decimal import Decimal
 from typing import Any
 
 from web3 import Web3
 
 from utils.abi import load_abi
+from utils.accountable import AccountableFeedConfig, AccountableReport, AccountableStatus, fetch_report
 from utils.alert import Alert, AlertSeverity, send_alert
 from utils.cache import (
     HOURLY_CACHE_STALE_AFTER_SECONDS,
@@ -89,6 +94,11 @@
 CACHE_KEY_BORROWER_DEFAULT_WATCH_PREFIX = "3JANE_BORROWER_DEFAULT_WATCH"
 CACHE_KEY_USD3_OC_ALERTED = "3JANE_USD3_OC_ALERTED"
 CACHE_KEY_WITHDRAW_LIMIT_ALERTED = "3JANE_WITHDRAW_LIMIT_ALERTED"
+CACHE_KEY_ACCOUNTABLE_HIGH_ALERTED = "3JANE_ACCOUNTABLE_HIGH_ALERTED"
+CACHE_KEY_ACCOUNTABLE_CRITICAL_ALERTED = "3JANE_ACCOUNTABLE_CRITICAL_ALERTED"
+CACHE_KEY_ACCOUNTABLE_FAILURE_STREAK = "3JANE_ACCOUNTABLE_FAILURE_STREAK"
+CACHE_KEY_ACCOUNTABLE_HEALTH_ALERTED = "3JANE_ACCOUNTABLE_HEALTH_ALERTED"
+CACHE_KEY_ACCOUNTABLE_STALE_ALERTED = "3JANE_ACCOUNTABLE_STALE_ALERTED"
 
 # --- ProtocolConfig keys (keccak256 of the string label) ---
 CFG_KEY_SUSD3_NOMINAL_BACKING_FLOOR = Web3.keccak(text="SUSD3_NOMINAL_BACKING_FLOOR")
@@ -103,6 +113,28 @@
 INSURANCE_FUND_OUTFLOW_THRESHOLD = 50_000  # USDC
 WITHDRAW_LIMIT_THRESHOLD = 4_000_000  # USDC, alert when USD3 availableWithdrawLimit falls below
 
+# --- Accountable Proof of Solvency ---
+ACCOUNTABLE_FEED = AccountableFeedConfig(
+    dfid="100000026",
+    protocol=PROTOCOL,
+    dashboard_url=os.getenv("THREE_JANE_ACCOUNTABLE_URL", "https://accountable.3jane.xyz/dashboard"),
+    message_url=os.getenv("THREE_JANE_ACCOUNTABLE_MESSAGE_URL", "https://accountable.3jane.xyz/"),
+    dashboard_type="three-jane",
+    required_sources=(
+        "LendSwift - Warehouse Senior Note",
+        "USD3 Minted Liabilities",
+        "Slope - Forward Flows",
+        "USD3 On-Chain Reserves",
+    ),
+    # The dashboard UI declares Slope as Weekly, while older JSON responses
+    # reported Daily. Bind the operator-confirmed cadence to avoid false alerts.
+    source_frequency_overrides=(("Slope - Forward Flows", "WEEKLY"),),
+)
+# TODO: Recalibrate both thresholds after Accountable includes 3Jane's idle
+# funds in the reported reserve totals. These values are temporary test bands.
+ACCOUNTABLE_CRITICAL_RATIO = Decimal("0.95")
+ACCOUNTABLE_HIGH_RATIO = Decimal("0.99")
+
 THREE_JANE_BORROWER_DEFAULT_WATCH_QUERY = """
 query GetThreeJaneBorrowerDefaultWatch($limit: Int!, $offset: Int!) {
   ThreeJaneBorrowerMarket(
@@ -166,8 +198,8 @@ def get_cache_int(key: str) -> int:
             return 0
 
 
-def set_cache_value(key: str, value: int | float) -> None:
-    """Write a numeric value to cache."""
+def set_cache_value(key: str, value: int | float | str) -> None:
+    """Write a value to cache (numeric, or a label such as an alert band)."""
     write_last_value_to_file(CACHE_FILENAME, key, value)
 
 
@@ -907,10 +939,186 @@ def check_protocol_paused(is_paused: bool) -> None:
     set_fresh_cache_value(CACHE_KEY_IS_PAUSED, float(is_paused))
 
 
+def _accountable_alert(severity: AlertSeverity, message: str) -> None:
+    """Send an Accountable alert using the feed's owning protocol key."""
+    send_alert(Alert(severity, message, ACCOUNTABLE_FEED.protocol))
+
+
+def _clear_accountable_ratio_alerts() -> None:
+    """Re-arm HIGH/CRITICAL ratio alerts after recovery above the warning threshold."""
+    if get_cache_int(CACHE_KEY_ACCOUNTABLE_HIGH_ALERTED):
+        set_cache_value(CACHE_KEY_ACCOUNTABLE_HIGH_ALERTED, 0)
+    if get_cache_int(CACHE_KEY_ACCOUNTABLE_CRITICAL_ALERTED):
+        set_cache_value(CACHE_KEY_ACCOUNTABLE_CRITICAL_ALERTED, 0)
+
+
+def _format_accountable_report(report: AccountableReport) -> str:
+    """Render the shared report body used by every Accountable alert."""
+    return (
+        f"📊 Collateral ratio: {report.collateralization:.4%}\n"
+        f"💰 Reserves: {format_usd(float(report.total_reserves))} | "
+        f"Liabilities: {format_usd(float(report.total_supply))}\n"
+        f"🧮 Net: {format_usd(float(report.net))} | Verifiability: {report.verifiability}%\n"
+        f"🕒 Report: {report.report_timestamp:%Y-%m-%d %H:%M:%S UTC} "
+        f"({format_duration(report.report_age_seconds)} old)"
+    )
+
+
+def _alert_accountable_high(report: AccountableReport) -> None:
+    """Alert once while ratio is below the HIGH threshold."""
+    if get_cache_int(CACHE_KEY_ACCOUNTABLE_HIGH_ALERTED):
+        return
+    message = (
+        f"🚨 *3Jane Proof of Solvency Low*\n"
+        f"{_format_accountable_report(report)}\n"
+        f"⚠️ Collateral ratio below the {ACCOUNTABLE_HIGH_RATIO:.0%} warning threshold\n"
+        f"🔗 [Accountable dashboard]({ACCOUNTABLE_FEED.message_url})"
+    )
+    _accountable_alert(AlertSeverity.HIGH, message)
+    set_cache_value(CACHE_KEY_ACCOUNTABLE_HIGH_ALERTED, 1)
+
+
+def _alert_accountable_critical(report: AccountableReport) -> None:
+    """Alert once while ratio is below the critical threshold."""
+    if get_cache_int(CACHE_KEY_ACCOUNTABLE_CRITICAL_ALERTED):
+        return
+    message = (
+        f"🚨 *3Jane Proof of Solvency CRITICAL*\n"
+        f"{_format_accountable_report(report)}\n"
+        f"⚠️ Collateral ratio below the {ACCOUNTABLE_CRITICAL_RATIO:.0%} critical threshold\n"
+        f"🔗 [Accountable dashboard]({ACCOUNTABLE_FEED.message_url})"
+    )
+    _accountable_alert(AlertSeverity.CRITICAL, message)
+    set_cache_value(CACHE_KEY_ACCOUNTABLE_CRITICAL_ALERTED, 1)
+    # Avoid a follow-up HIGH once CRITICAL clears but ratio is still under the HIGH threshold.
+    set_cache_value(CACHE_KEY_ACCOUNTABLE_HIGH_ALERTED, 1)
+
+
+def check_accountable_collateral(report: AccountableReport) -> None:
+    """Alert when Accountable collateral ratio breaches thresholds.
+
+    HIGH when ratio < 99%; CRITICAL on the first reading below 95%. Each
+    severity alerts once until the ratio recovers above its threshold.
+
+    Args:
+        report: Validated Proof of Solvency report.
+    """
+    ratio = report.collateralization
+    logger.info("Accountable collateral ratio: %.6f%%", ratio * 100)
+
+    if ratio < ACCOUNTABLE_CRITICAL_RATIO:
+        _alert_accountable_critical(report)
+        return
+
+    if ratio < ACCOUNTABLE_HIGH_RATIO:
+        if get_cache_int(CACHE_KEY_ACCOUNTABLE_CRITICAL_ALERTED):
+            set_cache_value(CACHE_KEY_ACCOUNTABLE_CRITICAL_ALERTED, 0)
+        _alert_accountable_high(report)
+        return
+
+    _clear_accountable_ratio_alerts()
+
+
+def check_accountable_staleness(report: AccountableReport, reason: str) -> None:
+    """Alert once when the Accountable feed or one of its sources goes stale.
+
+    A fresh aggregate timestamp does not prove every input is fresh, and the
+    reserves side leans heavily on manually uploaded document reports. Deduped
+    until the feed recovers.
+
+    Args:
+        report: Report the staleness was detected on.
+        reason: Human-readable description of what is stale.
+    """
+    logger.warning("Accountable feed %s is stale: %s", ACCOUNTABLE_FEED.dfid, reason)
+
+    if get_cache_int(CACHE_KEY_ACCOUNTABLE_STALE_ALERTED):
+        return
+
+    message = (
+        f"⚠️ *3Jane Proof of Solvency Stale*\n"
+        f"{_format_accountable_report(report)}\n"
+        f"🕳️ {escape_markdown(reason)}\n"
+        f"⚠️ Collateral ratio may not reflect current positions\n"
+        f"🔗 [Accountable dashboard]({ACCOUNTABLE_FEED.message_url})"
+    )
+    _accountable_alert(AlertSeverity.MEDIUM, message)
+    set_cache_value(CACHE_KEY_ACCOUNTABLE_STALE_ALERTED, 1)
+
+
+def check_accountable_availability(reason: str) -> None:
+    """Track feed failures: HIGH on the first miss, CRITICAL on the second.
+
+    ``fetch_report`` has already exhausted bounded retries before reporting a
+    request failure. A success clears the streak. Further misses after
+    CRITICAL stay quiet until recovery.
+
+    Args:
+        reason: Why the feed was unusable this run.
+    """
+    streak = get_cache_int(CACHE_KEY_ACCOUNTABLE_FAILURE_STREAK) + 1
+    set_cache_value(CACHE_KEY_ACCOUNTABLE_FAILURE_STREAK, streak)
+    logger.warning("Accountable feed unusable (%d consecutive): %s", streak, reason)
+
+    if streak == 1:
+        severity = AlertSeverity.HIGH
+    elif streak == 2:
+        severity = AlertSeverity.CRITICAL
+    else:
+        return
+
+    message = (
+        f"⚠️ *3Jane Proof of Solvency Unavailable*\n"
+        f"📡 Retrieval failed after all retry attempts ({streak} consecutive run"
+        f"{'s' if streak > 1 else ''})\n"
+        f"❌ {escape_markdown(reason)}\n"
+        f"⚠️ Collateral ratio is not being monitored\n"
+        f"🔗 [Accountable dashboard]({ACCOUNTABLE_FEED.message_url})"
+    )
+    _accountable_alert(severity, message)
+    set_cache_value(CACHE_KEY_ACCOUNTABLE_HEALTH_ALERTED, 1)
+
+
+def check_accountable_solvency() -> None:
+    """Fetch and evaluate the Accountable Proof of Solvency feed.
+
+    Runs in its own failure boundary: any error here is logged and swallowed so
+    the onchain 3Jane checks always complete.
+    """
+    try:
+        result = fetch_report(ACCOUNTABLE_FEED)
+
+        if result.status is AccountableStatus.UNAVAILABLE or result.report is None:
+            check_accountable_availability(result.reason or "unknown error")
+            return
+
+        # Reachable and parseable: clear any outstanding availability alert.
+        if get_cache_int(CACHE_KEY_ACCOUNTABLE_FAILURE_STREAK):
+            set_cache_value(CACHE_KEY_ACCOUNTABLE_FAILURE_STREAK, 0)
+        if get_cache_int(CACHE_KEY_ACCOUNTABLE_HEALTH_ALERTED):
+            set_cache_value(CACHE_KEY_ACCOUNTABLE_HEALTH_ALERTED, 0)
+
+        report = result.report
+        if result.status is AccountableStatus.STALE:
+            check_accountable_staleness(report, result.reason)
+        elif get_cache_int(CACHE_KEY_ACCOUNTABLE_STALE_ALERTED):
+            set_cache_value(CACHE_KEY_ACCOUNTABLE_STALE_ALERTED, 0)
+
+        # The ratio is still evaluated on a stale report: a low-ratio
+        # reading matters even when the inputs behind it have aged.
+        check_accountable_collateral(report)
+    except Exception as e:
+        logger.error("Error during Accountable Proof of Solvency check: %s", e)
+
+
 def main() -> None:
     """Run all 3Jane monitoring checks."""
     logger.info("Starting 3Jane monitoring...")
 
+    # Runs before the onchain reads and inside its own failure boundary, so the
+    # solvency feed is checked even when RPC access is degraded.
+    check_accountable_solvency()
+
     client = ChainManager.get_client(Chain.MAINNET)
     usd3_vault = client.eth.contract(address=USD3_ADDRESS, abi=ABI_VAULT)
     susd3_vault = client.eth.contract(address=SUSD3_ADDRESS, abi=ABI_VAULT)