error: ⚠ Error: t.content.filter is not a function when using a Mod to attempt add vision capabilities

[dthyresson]

Summary

I made a mod to attempt to add Image vision capabilities to any model by detecting an image and then delegating to a cheap vision capably model then send that back into the original prompt flow.

But, when I used and tried within deepseek pro, I encountered the following error:

 describe what you see in this image [Image #1]

◼ [vision-bridge] Describing 1 image(s) via MiniMaxAI/MiniMax-M3…

⚠ Error: t.content.filter is not a function

  Type "continue" to try again. If the issue persists, contact support: https://commandcode.ai/discord
  Trace ID: 76ee1a3ecc68b48a9b9f04420863cebf

❯ continue

⚠ Error: t.content.filter is not a function

  Type "continue" to try again. If the issue persists, contact support: https://commandcode.ai/discord
  Trace ID: 00e918dd06ff0fdfd52ff1a662d19162

Here is the mod - not Sur if an issue with the mod, Command Code flow, or the vision model.

/**
 * Vision Bridge — describe/OCR images for models that lack vision.
 *
 * When a prompt includes an image but the current model can't see images,
 * this mod saves the image to a temp file and shells out to `cmd -p`
 * with a cheap vision model to describe it, then injects the description
 * in place of the image blocks.
 *
 * Cheap vision models (both available in Command Code):
 *   stepfun/step-3.7-flash     — $0.20/M in, $1.15/M out (fastest: 400 tok/s)
 *   MiniMaxAI/MiniMax-M3       — $0.30/M in, $1.20/M out (higher intel)
 *
 * Flags:
 *   vision-model     (string)  Model ID for vision (default: stepfun/step-3.7-flash)
 *   vision-prompt    (string)  Prompt sent to vision model for each image
 */

import type { ModApi } from "@commandcode/harness";
import { createHash } from "node:crypto";
import { writeFile, unlink } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";

const DEFAULT_MODEL = "MiniMaxAI/MiniMax-M3";
const FALLBACK_MODEL = "stepfun/step-3.7-flash";
const DEFAULT_PROMPT =
  "Describe this image in thorough detail. If it contains text, transcribe it verbatim (OCR). Include key visual elements, layout, colors, and context.";

interface ImageBlock {
  type: "image";
  source: { type: "base64"; media_type: string; data: string };
}

interface ContentBlock {
  type: string;
  text?: string;
  source?: unknown;
}

function hashData(data: string): string {
  return createHash("sha256").update(data).digest("hex").slice(0, 16);
}

function extForType(mediaType: string): string {
  if (mediaType.startsWith("image/")) return mediaType.slice(6);
  return "png";
}

export default function (cmd: ModApi): void {
  cmd.addFlag("vision-model", {
    type: "string",
    default: DEFAULT_MODEL,
    description: `Vision model ID (default: ${DEFAULT_MODEL})`,
  });
  cmd.addFlag("vision-prompt", {
    type: "string",
    default: DEFAULT_PROMPT,
    description: "Prompt sent to vision model for each image batch",
  });

  const cache = new Map<string, string>();

  async function describeImages(
    modelId: string,
    prompt: string,
    images: ImageBlock[],
    signal?: AbortSignal,
  ): Promise<string | null> {
    const tmpPaths: string[] = [];
    try {
      for (let i = 0; i < images.length; i++) {
        const img = images[i];
        const ext = extForType(img.source.media_type);
        const path = join(tmpdir(), `cmd-vision-${Date.now()}-${i}.${ext}`);
        await writeFile(path, Buffer.from(img.source.data, "base64"));
        tmpPaths.push(path);
      }

      const fileList = tmpPaths.join(" ");
      const fullPrompt = `${prompt}\n\nImage files: ${fileList}\n\nRead each image file above and describe its contents thoroughly. Output ONLY the descriptions, no extra commentary or introduction.`;

      const { stdout, stderr, code } = await cmd.exec({
        command: "cmd",
        args: [
          "-p",
          fullPrompt,
          "-m",
          modelId,
          "--max-turns",
          "3",
          "--no-session",
        ],
        signal,
      });

      if (code !== 0 && import.meta.env.DEV) {
        console.error(
          `[vision-bridge] cmd -p exited ${code}:`,
          stderr.slice(0, 300),
        );
      }

      return stdout?.trim() || null;
    } finally {
      for (const p of tmpPaths) {
        try {
          await unlink(p);
        } catch {
          // best effort
        }
      }
    }
  }

  cmd.hooks({
    transformContext: async ({ messages, signal }) => {
      let hasImages = false;
      for (const msg of messages) {
        if (
          msg.role === "user" &&
          Array.isArray((msg as { content: unknown }).content)
        ) {
          const blocks = (msg as { content: ContentBlock[] }).content;
          if (blocks.some((b) => b.type === "image")) {
            hasImages = true;
            break;
          }
        }
      }
      if (!hasImages) return messages;

      const model = (cmd.getFlag("vision-model") as string) || DEFAULT_MODEL;
      const visionPrompt =
        (cmd.getFlag("vision-prompt") as string) || DEFAULT_PROMPT;

      const hashToResult = new Map<string, string>();
      const uncached: ImageBlock[] = [];
      const batchHashes: string[] = [];

      for (const msg of messages) {
        if (
          msg.role !== "user" ||
          !Array.isArray((msg as { content: unknown }).content)
        )
          continue;
        for (const block of (msg as { content: ContentBlock[] }).content) {
          if (block.type !== "image") continue;
          const img = block as unknown as ImageBlock;
          const h = hashData(img.source.data);

          if (cache.has(h)) {
            hashToResult.set(h, cache.get(h)!);
          } else if (!hashToResult.has(h)) {
            uncached.push(img);
            batchHashes.push(h);
            hashToResult.set(h, "__PENDING__");
          }
        }
      }

      if (uncached.length > 0) {
        cmd.ui.notify(`Describing ${uncached.length} image(s) via ${model}…`);

        let desc: string | null = null;

        try {
          desc = await describeImages(model, visionPrompt, uncached, signal);
        } catch (err) {
          if (import.meta.env.DEV) {
            console.error("[vision-bridge] primary model failed:", String(err));
          }

          if (model !== FALLBACK_MODEL) {
            cmd.ui.notify(
              `Vision failed, trying ${FALLBACK_MODEL}…`,
              "warning",
            );
            try {
              desc = await describeImages(
                FALLBACK_MODEL,
                visionPrompt,
                uncached,
                signal,
              );
            } catch (err2) {
              if (import.meta.env.DEV) {
                console.error(
                  "[vision-bridge] fallback model failed:",
                  String(err2),
                );
              }
              cmd.ui.notify("Vision bridge: both models failed.", "warning");
            }
          } else {
            cmd.ui.notify("Vision bridge failed.", "warning");
          }
        }

        if (desc) {
          for (const h of batchHashes) {
            cache.set(h, desc);
            hashToResult.set(h, desc);
          }
        } else {
          for (const h of batchHashes) {
            hashToResult.delete(h);
          }
        }
      }

      if (hashToResult.size === 0) return messages;

      const out: Record<string, unknown>[] = [];
      for (const msg of messages) {
        if (
          msg.role !== "user" ||
          !Array.isArray((msg as { content: unknown }).content)
        ) {
          out.push(msg as Record<string, unknown>);
          continue;
        }

        const blocks = (msg as { content: ContentBlock[] }).content;
        if (!blocks.some((b) => b.type === "image")) {
          out.push(msg as Record<string, unknown>);
          continue;
        }

        const textParts: string[] = [];
        const descParts: string[] = [];

        for (const block of blocks) {
          if (block.type === "text" && block.text) {
            textParts.push(block.text);
          } else if (block.type === "image") {
            const img = block as unknown as ImageBlock;
            const h = hashData(img.source.data);
            const d = hashToResult.get(h);
            if (d) {
              descParts.push(
                `[Image description (${img.source.media_type}): ${d}]`,
              );
            }
          }
        }

        let finalText = "";
        if (textParts.length > 0) finalText += textParts.join("\n");
        if (descParts.length > 0) {
          if (finalText) finalText += "\n\n";
          finalText += descParts.join("\n\n");
        }

        if (finalText) {
          out.push({ role: "user", content: finalText });
        }
      }

      return out as typeof messages;
    },
  });
}

Expected Behavior

The mod to work ads described:

• detect image
• use smaller, cheap vision model to do vision
• send that back to main prompt model to be used.

Actual Behavior

Errors:

 describe what you see in this image [Image #1]

◼ [vision-bridge] Describing 1 image(s) via MiniMaxAI/MiniMax-M3…

⚠ Error: t.content.filter is not a function

  Type "continue" to try again. If the issue persists, contact support: https://commandcode.ai/discord
  Trace ID: 76ee1a3ecc68b48a9b9f04420863cebf

❯ continue

⚠ Error: t.content.filter is not a function

  Type "continue" to try again. If the issue persists, contact support: https://commandcode.ai/discord
  Trace ID: 00e918dd06ff0fdfd52ff1a662d19162

Steps to reproduce the issue

• create mod
• ask/prompt with an image in a model that does not support vision like deep seek pro

Command Code Version

1.4.2

Operating System

macOS

Terminal/IDE

ghostty

Shell

zsh

Additional context

No response

[dthyresson]

I do wonder if the headless invocation also detects image and gets stuck, but when composing the mode it considered this case but said was not an issue