AAC Communication System for Non-Verbal Users
A personal Augmentative and Alternative Communication (AAC) system that helps non-verbal users communicate through a symbol-based board. The system learns how a specific user communicates over time and predicts what they want to say next using a 3-layer AI cascade that improves nightly.
Non-verbal users rely on AAC devices to communicate, but most commercial tools are static — they don't learn individual patterns, context, or preferences over time. Users must manually build every phrase from scratch, and generic suggestions rarely match their actual communication needs.
ConnectAble combines a 66-symbol communication board with a personalized AI prediction system that learns nightly from the user's phrase history. A 3-layer cascade — bigram lookup, ChromaDB vector search, and LLM fallback — ensures fast, relevant suggestions that improve the more the system is used.
A 66-symbol grid organized into 5 categories (food, feelings, actions, places, people) lets users build sentences by tapping buttons. The top bar displays real-time AI phrase suggestions that update as each symbol is selected.
Phrase predictions run through a cascade designed for speed and accuracy. Layer 1 is an instant bigram lookup from a local vocabulary store. Layer 2 does semantic vector search over past phrases via ChromaDB. Layer 3 falls back to Ollama's phi3 LLM when the phrase database is sparse.
Every night at 2 AM, the system retrains from logged phrase history — rebuilding the bigram map and re-embedding phrases into ChromaDB. The model improves continuously without user intervention.
Supports two TTS modes: fully offline using pyttsx3 / macOS say, or high-quality streaming audio via ElevenLabs for users who need a more natural voice.
A dedicated agent interface lets users send natural language messages that are classified into intents — make a call, order food, set a reminder, or general chat — and dispatched to tool handlers automatically.
All phrase data is stored locally in an AES-256 encrypted SQLite database (SQLCipher). The LLM runs locally via Ollama. No user data leaves the device unless ElevenLabs TTS is explicitly enabled.
Putting the cheapest operation first (bigram lookup) and the most expensive last (LLM inference) keeps suggestions fast for common phrases while maintaining quality for novel inputs.
For assistive technology, communication history is highly sensitive. Designing for offline-first with local encryption made privacy a core feature rather than an afterthought.
The nightly training pipeline means the system improves gradually. Seeding the database with starter phrases (via seed_phrases.py) was critical for useful predictions from day one.
Building the frontend to work standalone with fallback data when the backend is offline ensured the communication board was always usable, even during development or connectivity issues.
I'm always open to discussing assistive technology, AI projects, or internship opportunities.