🧠 Open-Source AI Book Proofreading System

A self-learning AI proofreader that corrects English grammar while preserving meaning and adapts to your writing style through user feedback.

✨ Features

Grammar Correction: Uses T5-based models for high-quality grammar and fluency corrections
Semantic Validation: Ensures corrections preserve meaning using sentence embeddings
User Feedback Learning: Adapts to your style through accepted/rejected corrections
Offline-First: Runs locally without external dependencies
CLI & GUI Interfaces: Choose your preferred way to interact
Docker Support: Containerized deployment for reproducibility

🎯 Project Overview

🚀 GPU Acceleration (NEW!)

Supercharged Performance on Mac M1/M2/M3 and NVIDIA GPUs!

The system now supports automatic GPU acceleration for dramatically improved performance:

⚡ Performance Benefits

Mac M1/M2/M3: 2-3x faster processing with Metal Performance Shaders (MPS)
NVIDIA GPUs: 3-5x faster with CUDA acceleration
Intelligent Optimization: Auto-adjusts batch sizes and worker counts
Memory Management: Smart GPU memory handling and cleanup

🎯 Quick GPU Setup

# Automatic GPU detection and acceleration (enabled by default)
python3 cli/batch_proofread.py sample_docs/ --output-dir corrected/ --show-device-info

# Mac M1/M2 optimized processing
python3 cli/batch_proofread.py book_chapters/ --workers 8 --batch-size 8 --show-device-info

# Compare GPU vs CPU performance
python3 cli/batch_proofread.py docs/ --output-dir gpu_test/ --show-device-info
python3 cli/batch_proofread.py docs/ --output-dir cpu_test/ --disable-gpu

🛠️ Advanced GPU Configuration

# GPU control options
--enable-gpu              # Enable GPU acceleration (default: enabled)
--disable-gpu             # Disable GPU acceleration (force CPU only)
--enable-mps              # Enable Apple Silicon MPS (default: enabled)
--disable-mps             # Disable MPS acceleration
--workers 8               # Parallel workers (optimized for GPU)
--batch-size 8            # GPU-optimized batch size
--show-device-info        # Display GPU/CPU information

See docs/GPU_Acceleration_Guide.md for complete GPU documentation and optimization guides.

This project implements a complete AI proofreading pipeline based on the comprehensive plan outlined in [docs/Master Project Guide.md](docs/Master Project Guide.md:1). It provides:

Text preprocessing and normalization
Grammar correction using transformer models
Semantic similarity validation
SQLite-based feedback collection
Optional model retraining
Web-based GUI for interactive review

🚀 Quick Start

Installation

Clone the repository

git clone <repository-url>
cd grammar-proofreader

Install dependencies

# Create virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install requirements
pip install -r requirements.txt

# Download language model
python -m spacy download en_core_web_sm

Run CLI

python cli/proofread.py --input my_text.txt --export corrected.txt

Run GUI (optional)

python gui/app.py
# Opens at http://localhost:7860

Using Docker

# Build image
docker build -t proofreader .

# Run CLI
docker run --rm -v $(pwd):/data proofreader --input /data/input.txt --export /data/output.txt

# Run GUI
docker run --rm -p 7860:7860 proofreader python gui/app.py

📖 Usage

Command Line Interface

# Process a file
python cli/proofread.py --input sample.txt --export corrected.txt

# Process text directly
python cli/proofread.py --text "I have went to the store." --export corrected.txt

# Get detailed results
python cli/proofread.py --input sample.txt --export-json results.json --stats

# Adjust similarity threshold
python cli/proofread.py --input sample.txt --similarity-threshold 0.9

CLI Options:

--input, -i: Input file path
--text, -t: Input text directly
--export, -o: Output file for corrected text
--export-json, -j: Output file for detailed results (JSON)
--similarity-threshold, -s: Semantic similarity threshold (0.0-1.0)
--model-name: Grammar model to use
--batch-size: Batch size for inference
--stats: Show processing statistics
--quiet, -q: Suppress output
--verbose, -v: Increase verbosity

Graphical User Interface

The GUI provides an interactive interface for:

Text Input: Paste or upload your text
Real-time Processing: See corrections as you type
Accept/Reject: Manually review each suggestion
Statistics: Track processing metrics and feedback
Adjustable Threshold: Fine-tune semantic similarity requirements

🏗️ Architecture

The system follows a modular architecture as defined in tech/Technical_Spec.md:

Text Input
    ↓
Text Preprocessor (src/preprocessor.py)
    ↓
Grammar Model (src/inference.py)
    ↓
Semantic Validator (src/validator.py)
    ↓
Feedback Database (src/db.py)
    ↓
[Optional] Model Retraining

Core Components

Pipeline (src/pipeline.py): Orchestrates the entire workflow
Preprocessor (src/preprocessor.py): Normalizes and splits text
Grammar Model (src/inference.py): Handles T5-based grammar correction
Semantic Validator (src/validator.py): Validates meaning preservation
Feedback DB (src/db.py): SQLite database for user feedback

📊 Requirements & Dependencies

Python: 3.9+
CUDA: Optional, for GPU acceleration
Memory: 4GB+ RAM recommended
Storage: 2GB+ for models and dependencies

Key Dependencies

torch>=1.13.0: Deep learning framework
transformers>=4.30.0: Pre-trained models
sentence-transformers: Semantic similarity
gradio: Web interface
spacy: Natural language processing
sqlite3: Database (built-in)

See requirements.txt for complete list.

🧪 Testing

The project includes comprehensive testing as defined in tests/Test_Plan.md:

# Run all tests
pytest

# Run with coverage
pytest --cov=src

# Run specific test types
pytest tests/unit/          # Unit tests
pytest tests/integration/   # Integration tests

📈 Analytics & Feedback

The system includes analytics as outlined in analytics/Analytics_Plan.md:

Acceptance Rate: Track user acceptance of corrections
Semantic Similarity: Monitor meaning preservation
Processing Statistics: Performance metrics
User Feedback: Accepted/rejected corrections for learning

Access statistics via CLI:

python cli/proofread.py --stats --input sample.txt

🔒 Security & Privacy

See security/Security_Compliance.md for complete security documentation.

Local-First: All processing happens locally
Data Privacy: User content stays on your machine
Optional Encryption: Database encryption available
No Telemetry: No data sent to external services by default

🐳 Deployment

Docker Deployment

See docs/Repo_Scaffold.md for deployment details.

# Production build
docker build -t proofreader:latest .

# Run with volume mount
docker run -v /path/to/data:/app/data proofreader:latest

Environment Variables

MODEL_CACHE_DIR: Custom path for downloaded models
DB_PATH: Custom path for SQLite database
LOG_LEVEL: Logging level (DEBUG, INFO, WARNING, ERROR)

📋 Roadmap & Development

See shards/shard_backlog.md for detailed development plan.

Current Status

✅ Completed:

🔄 In Progress:

Model retraining automation
Enhanced GUI features
Performance optimizations

📅 Planned:

Multiple model support
Batch processing improvements
Web API interface
Advanced analytics dashboard

🤝 Contributing

Development Setup

Follow the installation instructions above

Install development dependencies:

pip install black flake8 pytest pytest-cov

Run code formatting:
```
black src/ cli/ gui/ tests/
```
Run linting:
```
flake8 src/ cli/ gui/ tests/
```

Project Structure

├── src/                 # Core modules
│   ├── pipeline.py     # Main orchestration
│   ├── preprocessor.py # Text preprocessing
│   ├── inference.py    # Grammar model
│   ├── validator.py    # Semantic validation
│   └── db.py          # Feedback database
├── cli/                # Command-line interface
│   └── proofread.py
├── gui/                # Web interface
│   └── app.py
├── tests/              # Test suite
├── docs/               # Documentation
├── requirements.txt    # Dependencies
├── Dockerfile         # Containerization
└── README.md          # This file

Contributing Guidelines

Follow the architecture defined in the tech spec
Write tests for new features
Update documentation for user-facing changes
Use semantic versioning for releases
Keep dependencies updated regularly

📚 Documentation

[docs/Master Project Guide.md](docs/Master Project Guide.md:1): Project overview and roadmap
[prd/PRD - Product Requirements.md](prd/PRD - Product Requirements.md:1): Product requirements
tech/Technical_Spec.md: Technical architecture
tests/Test_Plan.md: Testing strategy
analytics/Analytics_Plan.md: Analytics and metrics
security/Security_Compliance.md: Security documentation
shards/shard_backlog.md: Development backlog

📄 License

This project is licensed under the MIT License. See LICENSE file for details.

🙏 Acknowledgments

Based on the comprehensive plan in plan for grammar correction.md
Uses T5-based grammar correction models from Hugging Face
Semantic validation powered by SentenceTransformers
GUI built with Gradio
Architecture inspired by modern MLOps practices

📞 Support

For questions, issues, or contributions:

Check the documentation
Search existing issues
Create a new issue with detailed description
Join our discussions

Built with ❤️ for writers, editors, and anyone who values clear, correct English prose.

🧠 Learning Verification

The system includes a comprehensive learning verification system that tracks AI improvement over time:

python3 cli/proofread.py --learning-report

Key Features:

Learning Patterns - Shows most frequently corrected patterns
Quality Distribution - Breaks down corrections by similarity scores
Learning Trends - Week-over-week improvement tracking
Source Analytics - Usage patterns by interface (CLI/GUI/API)
Training Readiness - Indicates when enough data is collected for retraining
Performance Metrics - Acceptance rates, similarity improvements, daily activity

This enables you to verify that the system is truly learning from itself and improving over time.

🎯 Example Learning Report

🧠 AI LEARNING REPORT
==================================================

📊 OVERALL STATISTICS
Total Feedback: 2
Accepted Corrections: 2
Rejected Corrections: 0
Acceptance Rate: 100.0%
Recent Activity (24h): 2
Average Similarity (accepted): 0.638

🎯 QUALITY DISTRIBUTION
Low: 2/2 accepted (100.0%)

📈 LEARNING TRENDS
Last 7 days: 2 corrections
Previous 7 days: 0 corrections
Week-over-week change: +2 corrections
Similarity improvement: +0.638

🔄 MOST LEARNED PATTERNS
1. '"Huuu….' → 'Huuu....'
   Frequency: 1 | Similarity: 0.726
2. '……' → '......'
   Frequency: 1 | Similarity: 0.55

📝 SOURCE USAGE PATTERNS
Api: 2 corrections (avg similarity: 0.638)

📅 DAILY PERFORMANCE TREND
2025-11-03: 2 corrections, 100.0% accepted

⚠️  LEARNING STATUS: Gathering feedback (need 8 more corrections)
🔄 TRAINING PREPARATION: 2 pairs (need 8 more for training)

📁 Batch Processing

The system supports batch processing for multiple files simultaneously:

# Process entire directory of files
python3 cli/batch_proofread.py sample_docs/ --output-dir corrected_docs/

# Process with summary report and parallel processing
python3 cli/batch_proofread.py *.txt --output-dir batch_output/ --summary-file report.json --workers 4

# Sequential processing for debugging
python3 cli/batch_proofread.py sample_docs/ --sequential

Batch Features:

Multiple file formats: .txt, .md, .rst files
Parallel processing: Configurable worker threads
Progress tracking: Real-time progress bars
Summary reporting: JSON export of batch statistics
Error handling: Continues processing even if individual files fail
Output organization: Preserves original file structure and extensions

Batch Processing Output:

📊 BATCH PROCESSING SUMMARY
==================================================
Total files: 3
Successful: 3
Failed: 0
Success rate: 100.0%
Total processing time: 135.9s
Total sentences processed: 35
Auto-applied corrections: 35
Requiring review: 0
Average processing rate: 0.3 sentences/sec
==================================================

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
analytics		analytics
cli		cli
docs		docs
gui		gui
prd		prd
sample_docs		sample_docs
security		security
shards		shards
src		src
tech		tech
tests		tests
.gitignore		.gitignore
Dockerfile		Dockerfile
README.md		README.md
plan for grammar correction.md		plan for grammar correction.md
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation