TaskLoom is a Flutter application that leverages Gemini AI function calling to plan and execute multi-step goals across local device tools. Built with Clean Architecture and BLoC, it incorporates a strict Human-in-the-Loop Confirmation Gate to guarantee that sensitive device actions (like scheduling calendar events or notifications) never execute without explicit user approval.
- Multi-Step Agentic Planning: Uses Gemini function calling in an iterative loop (up to 8 turns) to break complex user prompts into sequential tool calls rather than single-shot text answers.
- Human-in-the-Loop Confirmation Gate: Sensitive operations (e.g., creating reminders or scheduling events) pause the agent's execution loop and prompt the user with an interactive approval card. No sensitive code runs on the device without explicit approval.
-
Real Device Integration: Schedules real Android/iOS system notifications via
flutter_local_notificationsand creates native device calendar events viadevice_calendar. -
Observe-and-Replan: Automatically checks the user's native calendar for existing event overlaps (
checkCalendarConflicts) before proposing a reminder, adapting its proposed time if a conflict is found. -
RAG Personal Notes Tool: Includes a personal notes feature powered by text chunking, Gemini text embeddings (
gemini-embedding-001), and an ObjectBox HNSW vector database (@HnswIndex(dimensions: 3072)). The agent can search stored notes via thesearchNotestool. -
LLM-as-Judge & Automated Eval Suite: Includes a developer test harness in
test/evals/with 6 real test scenarios, LLM-as-Judge qualitative grounding checks, and automated Markdown report generation intest/evals/results/. -
Cost & Latency Optimizations:
-
Temporal Context Priming: Pre-injects current device date, time, and timezone offset into Gemini's
systemInstruction, reducing API round-trips by 50% for time-based queries. -
Implicit Cache Structuring: Orders request payloads (
systemInstruction$\rightarrow$ tools$\rightarrow$ generationConfig$\rightarrow$ contents) to trigger Gemini's automatic implicit context caching (~90% token discount on tool schemas). -
Output Bounding & Token Logging: Caps output at
maxOutputTokens: 512and logs real-time input/cached/candidate token usage.
-
Temporal Context Priming: Pre-injects current device date, time, and timezone offset into Gemini's
The diagram below illustrates the complete multi-step execution loop, from user prompt submission through tool calls, observe-and-replan cycles, confirmation gate pause/resume, and final response rendering:
sequenceDiagram
autonumber
actor User
participant UI as AgentPage UI
participant Bloc as AgentBloc
participant Repo as AgentRepositoryImpl
participant Gemini as Gemini API (Cloud)
participant Tools as Local Tools / Device OS
User->>UI: Enters prompt ("Set a reminder for team sync tomorrow at 3pm")
UI->>Bloc: Dispatches AgentPromptSubmitted
Bloc->>Repo: Calls askAgent() stream
loop Multi-Step Loop (Up to 8 Iterations)
Repo->>Gemini: POST /generateContent (System Instruction + Tools + History)
alt Option A: Non-Sensitive Tool Call (e.g. checkCalendarConflicts)
Gemini-->>Repo: functionCall("checkCalendarConflicts", {dateTime: "..."})
Repo->>Tools: Execute CalendarConflictTool locally
Tools-->>Repo: Returns conflict check results
Repo->>Repo: Appends functionResponse to conversation turns
else Option B: Sensitive Tool Call (e.g. createReminder)
Gemini-->>Repo: functionCall("createReminder", {title: "...", dateTime: "..."})
Repo->>Bloc: Emits AgentPendingConfirmationUpdate & awaits Completer<bool>
Bloc->>UI: Renders Confirmation Card
User->>UI: Taps "Approve" or "Decline"
UI->>Bloc: Dispatches AgentActionConfirmed / AgentActionDenied
Bloc->>Repo: Completes Completer<bool> with true / false
alt Approved
Repo->>Tools: Execute ReminderTool (Local Notification & Device Calendar)
Tools-->>Repo: Event & Notification Created
else Declined
Repo->>Repo: Skip execution & send decline turn to Gemini
end
else Option C: Final Answer Complete
Gemini-->>Repo: Text answer response
Repo->>Bloc: Yields AgentAnswerUpdate
Bloc->>UI: Displays Agent Answer & Steps Taken
end
end
The sequence diagram below specifically details the asynchronous callback/Completer pause-and-resume mechanism used for human-in-the-loop authorization:
sequenceDiagram
autonumber
participant Agent as AgentRepositoryImpl
participant Bloc as AgentBloc
participant UI as AgentPage UI
participant User as User
Agent->>Agent: Identifies sensitive tool call (ReminderTool)
Agent->>Bloc: Emits PendingConfirmation & awaits Completer<bool>
Bloc->>UI: Emits AgentAwaitingConfirmation state
UI->>User: Displays Confirmation Card (Approve / Decline)
alt User Approves
User->>UI: Taps "Approve"
UI->>Bloc: Dispatches AgentActionConfirmed
Bloc->>Agent: Completes Completer<bool> with true
Agent->>Agent: Executes ReminderTool & sends result to Gemini
else User Declines
User->>UI: Taps "Decline"
UI->>Bloc: Dispatches AgentActionDenied
Bloc->>Agent: Completes Completer<bool> with false
Agent->>Agent: Skips tool execution & reports decline turn to Gemini
end
TaskLoom includes a developer evaluation harness located in test/evals/ designed to run test scenarios directly against the live Gemini AI pipeline, capturing tool selection, confirmation gate interactions, and response quality.
eval-1-single-tool(Single Tool Query): Validates date calculation via temporal context injection and single-step tool execution.eval-2-multi-step(Multi-Step Planning): Verifies multi-turn tool chain reasoning combining calendar range checks, math calculation, and temporal reasoning.eval-3-notes-retrieval(RAG Personal Notes): Verifies note chunking, vector embedding viagemini-embedding-001, and ObjectBox HNSW vector search retrieval.eval-4-confirmation-approve(Confirmation Approve Path): Verifies that sensitive tool requests pause for user confirmation and execute upon approval.eval-5-confirmation-decline(Confirmation Decline Path): Verifies that declining sensitive actions skips tool execution and cleanly returns a decline turn to Gemini.eval-6-grounding-no-hallucination(Grounding & LLM-as-Judge): Uses Gemini as a qualitative AI Judge (test/evals/llm_judge.dart) to evaluate whether the agent refrains from hallucinating data (e.g., weather forecasts) when no relevant tool exists for the user request.
After each evaluation run, a detailed Markdown report is generated in test/evals/results/ (e.g. eval_report_YYYY-MM-DD_HHmmss.md and latest_report.md) summarizing pass/fail rates, tool execution histories, and LLM judge rationale.
flutter test test/evals/TaskLoom follows Clean Architecture principles structured by feature (agent and notes), maintaining a strict boundary between data layer models and pure domain entities.
lib/
βββ core/
β βββ database/ # ObjectBox store setup
β βββ errors/ # AppException & Failure hierarchies (fpdart)
β βββ network/ # Dio client & ApiKeyInterceptor (.env masking)
β βββ routes/ # GoRouter configuration
β βββ usecase/ # Generic UseCase interface
βββ features/
β βββ agent/ # AI Agent feature
β β βββ data/ # Gemini models, tool definitions, & AgentRepositoryImpl
β β βββ domain/ # Agent entities, repository contract, & AskAgent use case
β β βββ presentation/# AgentBloc (Bloc<Event, State>) & AgentPage UI
β βββ notes/ # RAG Notes feature
β βββ data/ # ObjectBox NoteChunkModel & NotesRemoteDataSourceImpl
β βββ domain/ # NoteChunk domain entity & vector search use cases
β βββ presentation/# NotesBloc & NotesPage UI
βββ test/
βββ evals/ # Agent Evals Harness, LLM Judge, & Report Generator
- State Management: Pure BLoC (
flutter_bloc,Bloc<Event, State>). Cubit is strictly avoided to maintain explicit event-driven trace logs. - Dependency Injection:
GetItservice locator manages singletons and factories (dio,objectbox, data sources, repositories, BLoCs). - Error Handling:
fpdartreturningEither<Failure, T>for functional error handling, mapping data sourceAppExceptionvariants to UIFailurestates. - Local & Vector Database: ObjectBox (
objectbox_flutter_libs) for fast local storage and HNSW vector index nearest-neighbor search.
This project was built and verified with:
- Flutter SDK:
3.41.1(stable channel) - Dart SDK:
3.11.0
git clone https://github.com/your-username/task_loom.git
cd task_loomflutter pub getCreate a .env file in the root directory of the project:
GEMINI_API_KEY=your_gemini_api_key_hereYou can obtain a free Gemini API key from Google AI Studio.
flutter runflutter test test/evals/Note: Creating reminders requires calendar permissions on Android/iOS. The app will prompt for
READ_CALENDARandWRITE_CALENDARpermissions at runtime upon first use.