Inbox Triage / AI engineering take-home
Automatic triage for a shared inbox.
I built a service that automatically classifies email, selects actions and drafts replies when confidence is high enough. Below the threshold, it asks a person to decide. Sending and other external writes require confirmation.
The project starts from a public Go Fig AI exercise, which supplies eight emails and a simulated client API.
The demo replays responses recorded from the running service; the model does not run in the browser.
- Service
- FastAPI · Docker
- Decision model
- OpenDecider-nano
- Traces and flow
- LangSmith · LangGraph
- Verification
- 55 CI tests
From email to action
A shared info@ inbox receives billing requests, bug reports, sales leads and spam. When classification clears the threshold, the service selects actions from a table and drafts any replies. Uncertain or suspicious messages remain for classification or review.
- Incoming emailThe model returns four probabilities and checks for instructions addressed to the system.
- Automatic triageWith sufficient confidence, the service selects actions; uncertain cases go to a person.
- Draft and confirmationReplies are drafted automatically; a person can approve, edit or reject each action.
- Write and traceWrite access is created only after approval; LangSmith records the path.
Classification and drafts
OpenDecider-nano chooses among four categories and returns a probability for each. At or above the initial 0.90 threshold, the service prepares the corresponding actions; below it, a person classifies the email. A separate check looks for instructions aimed at the system.
The generative model drafts replies without access to external systems. The text can be edited before sending.
From proposal to execution
Replies, CRM records and engineering alerts each require a separate verdict. The code that reads and classifies mail has read access only; the write client is created at execution time, after approval. A ledger prevents duplicate sends.
Traces and experiments
LangSmith keeps the model's decision, the draft and the human verdict in the same email trace. I also evaluated the decision model as a dataset experiment, checking both expected outcomes and errors that propose actions for messages that should be held.
I implemented the same path as a LangGraph graph. Studio shows where it pauses for approval and which saved state it resumes from.
Tests and results
GitHub Actions runs 55 tests on every push: no write without approval, no early write access, no duplicate send and a defined outcome for failures. I deliberately removed the approval check on a trial branch; one test failed, as expected.
In one run, all eight exercise emails produced the expected outcome. Twelve additional probes revealed false alarms and attacks the detector missed. A flagged message is therefore kept for review, and safety does not rest on the injection detector alone.
These checks demonstrate the flow and its boundaries. Twenty emails, each evaluated once, cannot measure accuracy on a real inbox. The client's mail, CRM and Slack systems are simulated.
Demo and code
The public demo lets visitors try the decision and approval page with recorded responses. The repository contains the two-container service, tests and design decisions needed to run the full system.