Skip to content
vinittailorPublic

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

Β 

History

3 Commits

Folders and files

Repository files navigation

TaskLoom 🧡

TaskLoom is a Flutter application that leverages Gemini AI function calling to plan and execute multi-step goals across local device tools. Built with Clean Architecture and BLoC, it incorporates a strict Human-in-the-Loop Confirmation Gate to guarantee that sensitive device actions (like scheduling calendar events or notifications) never execute without explicit user approval.


🌟 Key Features

  • Multi-Step Agentic Planning: Uses Gemini function calling in an iterative loop (up to 8 turns) to break complex user prompts into sequential tool calls rather than single-shot text answers.
  • Human-in-the-Loop Confirmation Gate: Sensitive operations (e.g., creating reminders or scheduling events) pause the agent's execution loop and prompt the user with an interactive approval card. No sensitive code runs on the device without explicit approval.
  • Real Device Integration: Schedules real Android/iOS system notifications via flutter_local_notifications and creates native device calendar events via device_calendar.
  • Observe-and-Replan: Automatically checks the user's native calendar for existing event overlaps (checkCalendarConflicts) before proposing a reminder, adapting its proposed time if a conflict is found.
  • RAG Personal Notes Tool: Includes a personal notes feature powered by text chunking, Gemini text embeddings (gemini-embedding-001), and an ObjectBox HNSW vector database (@HnswIndex(dimensions: 3072)). The agent can search stored notes via the searchNotes tool.
  • LLM-as-Judge & Automated Eval Suite: Includes a developer test harness in test/evals/ with 6 real test scenarios, LLM-as-Judge qualitative grounding checks, and automated Markdown report generation in test/evals/results/.
  • Cost & Latency Optimizations:
    • Temporal Context Priming: Pre-injects current device date, time, and timezone offset into Gemini's systemInstruction, reducing API round-trips by 50% for time-based queries.
    • Implicit Cache Structuring: Orders request payloads (systemInstruction $\rightarrow$ tools $\rightarrow$ generationConfig $\rightarrow$ contents) to trigger Gemini's automatic implicit context caching (~90% token discount on tool schemas).
    • Output Bounding & Token Logging: Caps output at maxOutputTokens: 512 and logs real-time input/cached/candidate token usage.

πŸ”„ End-to-End Agent Sequence Diagram

The diagram below illustrates the complete multi-step execution loop, from user prompt submission through tool calls, observe-and-replan cycles, confirmation gate pause/resume, and final response rendering:

sequenceDiagram
    autonumber
    actor User
    participant UI as AgentPage UI
    participant Bloc as AgentBloc
    participant Repo as AgentRepositoryImpl
    participant Gemini as Gemini API (Cloud)
    participant Tools as Local Tools / Device OS

    User->>UI: Enters prompt ("Set a reminder for team sync tomorrow at 3pm")
    UI->>Bloc: Dispatches AgentPromptSubmitted
    Bloc->>Repo: Calls askAgent() stream
    
    loop Multi-Step Loop (Up to 8 Iterations)
        Repo->>Gemini: POST /generateContent (System Instruction + Tools + History)
        
        alt Option A: Non-Sensitive Tool Call (e.g. checkCalendarConflicts)
            Gemini-->>Repo: functionCall("checkCalendarConflicts", {dateTime: "..."})
            Repo->>Tools: Execute CalendarConflictTool locally
            Tools-->>Repo: Returns conflict check results
            Repo->>Repo: Appends functionResponse to conversation turns
            
        else Option B: Sensitive Tool Call (e.g. createReminder)
            Gemini-->>Repo: functionCall("createReminder", {title: "...", dateTime: "..."})
            Repo->>Bloc: Emits AgentPendingConfirmationUpdate & awaits Completer<bool>
            Bloc->>UI: Renders Confirmation Card
            User->>UI: Taps "Approve" or "Decline"
            UI->>Bloc: Dispatches AgentActionConfirmed / AgentActionDenied
            Bloc->>Repo: Completes Completer<bool> with true / false
            
            alt Approved
                Repo->>Tools: Execute ReminderTool (Local Notification & Device Calendar)
                Tools-->>Repo: Event & Notification Created
            else Declined
                Repo->>Repo: Skip execution & send decline turn to Gemini
            end
            
        else Option C: Final Answer Complete
            Gemini-->>Repo: Text answer response
            Repo->>Bloc: Yields AgentAnswerUpdate
            Bloc->>UI: Displays Agent Answer & Steps Taken
        end
    end
Loading

πŸ” Confirmation Gate Sequence Diagram

The sequence diagram below specifically details the asynchronous callback/Completer pause-and-resume mechanism used for human-in-the-loop authorization:

sequenceDiagram
    autonumber
    participant Agent as AgentRepositoryImpl
    participant Bloc as AgentBloc
    participant UI as AgentPage UI
    participant User as User

    Agent->>Agent: Identifies sensitive tool call (ReminderTool)
    Agent->>Bloc: Emits PendingConfirmation & awaits Completer<bool>
    Bloc->>UI: Emits AgentAwaitingConfirmation state
    UI->>User: Displays Confirmation Card (Approve / Decline)
    
    alt User Approves
        User->>UI: Taps "Approve"
        UI->>Bloc: Dispatches AgentActionConfirmed
        Bloc->>Agent: Completes Completer<bool> with true
        Agent->>Agent: Executes ReminderTool & sends result to Gemini
    else User Declines
        User->>UI: Taps "Decline"
        UI->>Bloc: Dispatches AgentActionDenied
        Bloc->>Agent: Completes Completer<bool> with false
        Agent->>Agent: Skips tool execution & reports decline turn to Gemini
    end
Loading

πŸ§ͺ Agent Evaluation Suite & LLM-as-Judge

TaskLoom includes a developer evaluation harness located in test/evals/ designed to run test scenarios directly against the live Gemini AI pipeline, capturing tool selection, confirmation gate interactions, and response quality.

Eval Scenarios Covered

  1. eval-1-single-tool (Single Tool Query): Validates date calculation via temporal context injection and single-step tool execution.
  2. eval-2-multi-step (Multi-Step Planning): Verifies multi-turn tool chain reasoning combining calendar range checks, math calculation, and temporal reasoning.
  3. eval-3-notes-retrieval (RAG Personal Notes): Verifies note chunking, vector embedding via gemini-embedding-001, and ObjectBox HNSW vector search retrieval.
  4. eval-4-confirmation-approve (Confirmation Approve Path): Verifies that sensitive tool requests pause for user confirmation and execute upon approval.
  5. eval-5-confirmation-decline (Confirmation Decline Path): Verifies that declining sensitive actions skips tool execution and cleanly returns a decline turn to Gemini.
  6. eval-6-grounding-no-hallucination (Grounding & LLM-as-Judge): Uses Gemini as a qualitative AI Judge (test/evals/llm_judge.dart) to evaluate whether the agent refrains from hallucinating data (e.g., weather forecasts) when no relevant tool exists for the user request.

Automated Evaluation Reports

After each evaluation run, a detailed Markdown report is generated in test/evals/results/ (e.g. eval_report_YYYY-MM-DD_HHmmss.md and latest_report.md) summarizing pass/fail rates, tool execution histories, and LLM judge rationale.

Running the Evals Suite

flutter test test/evals/

πŸ—οΈ Architecture & Design Patterns

TaskLoom follows Clean Architecture principles structured by feature (agent and notes), maintaining a strict boundary between data layer models and pure domain entities.

lib/
β”œβ”€β”€ core/
β”‚   β”œβ”€β”€ database/        # ObjectBox store setup
β”‚   β”œβ”€β”€ errors/          # AppException & Failure hierarchies (fpdart)
β”‚   β”œβ”€β”€ network/         # Dio client & ApiKeyInterceptor (.env masking)
β”‚   β”œβ”€β”€ routes/          # GoRouter configuration
β”‚   └── usecase/         # Generic UseCase interface
β”œβ”€β”€ features/
β”‚   β”œβ”€β”€ agent/           # AI Agent feature
β”‚   β”‚   β”œβ”€β”€ data/        # Gemini models, tool definitions, & AgentRepositoryImpl
β”‚   β”‚   β”œβ”€β”€ domain/      # Agent entities, repository contract, & AskAgent use case
β”‚   β”‚   └── presentation/# AgentBloc (Bloc<Event, State>) & AgentPage UI
β”‚   └── notes/           # RAG Notes feature
β”‚       β”œβ”€β”€ data/        # ObjectBox NoteChunkModel & NotesRemoteDataSourceImpl
β”‚       β”œβ”€β”€ domain/      # NoteChunk domain entity & vector search use cases
β”‚       └── presentation/# NotesBloc & NotesPage UI
└── test/
    └── evals/           # Agent Evals Harness, LLM Judge, & Report Generator

Key Technical Choices

  • State Management: Pure BLoC (flutter_bloc, Bloc<Event, State>). Cubit is strictly avoided to maintain explicit event-driven trace logs.
  • Dependency Injection: GetIt service locator manages singletons and factories (dio, objectbox, data sources, repositories, BLoCs).
  • Error Handling: fpdart returning Either<Failure, T> for functional error handling, mapping data source AppException variants to UI Failure states.
  • Local & Vector Database: ObjectBox (objectbox_flutter_libs) for fast local storage and HNSW vector index nearest-neighbor search.

πŸ“‹ Requirements

This project was built and verified with:

  • Flutter SDK: 3.41.1 (stable channel)
  • Dart SDK: 3.11.0

πŸš€ Setup & Installation

1. Clone the Repository

git clone https://github.com/your-username/task_loom.git
cd task_loom

2. Install Dependencies

flutter pub get

3. Configure Gemini API Key

Create a .env file in the root directory of the project:

GEMINI_API_KEY=your_gemini_api_key_here

You can obtain a free Gemini API key from Google AI Studio.

4. Run the Application

flutter run

5. Run the Evals Suite

flutter test test/evals/

Note: Creating reminders requires calendar permissions on Android/iOS. The app will prompt for READ_CALENDAR and WRITE_CALENDAR permissions at runtime upon first use.

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages