[Bug] Premature Math.floor in truncateToTokenBudget causes character loss and empty results on fractional budgets

[lokopoko9090]

Environment

• SDK: @mysten-incubation/memwal@0.1.7
• File: dist/tokens.js (lines 62, 133)

Summary

In truncateToTokenBudget(), applying Math.floor(maxTokens) before multiplying by CHARS_PER_TOKEN (4) leads to unnecessary character truncation and completely drops valid text when 0 < maxTokens < 1. Similarly, in applyTokenBudget() with per-hit-cap, Math.floor(maxTokens / count) collapses to 0 for fractional per-hit shares, dropping all results.

Proof / Reproduction

import { truncateToTokenBudget, applyTokenBudget } from "@mysten-incubation/memwal/dist/tokens.js";

// Test 1: Fractional budget loses valid characters
const text = "1234567890abcdef"; // 16 chars
const res = truncateToTokenBudget(text, 3.75); // 3.75 * 4 = 15 chars expected
console.log(res.length); // Actual: 12 (Lost 3 characters!)

// Test 2: Sub-token budget yields empty string
const res2 = truncateToTokenBudget("Hello World", 0.75); // 0.75 * 4 = 3 chars ("Hel") expected
console.log(res2); // Actual: "" (Empty)

// Test 3: per-hit-cap collapses to empty array
const hits = [{ text: "Item 1" }, { text: "Item 2" }];
const res3 = applyTokenBudget(hits, 1.5, "per-hit-cap");
console.log(res3.results.length); // Actual: 0 (All items dropped)

Root Cause

In dist/tokens.js (line 62):

const cap = Math.min(chars.length, Math.floor(maxTokens) * CHARS_PER_TOKEN);

Proposed Fix

Compute floor after scaling to characters:

const cap = Math.min(chars.length, Math.floor(maxTokens * CHARS_PER_TOKEN));

[github-actions]

Thank you for opening this issue, a team member will review it shortly.