Late Shipment Exception Agent
Autonomous decision intelligence with human-in-the-loop governance for e-commerce fulfillment. Resolves 80% of delayed orders in under 2 seconds while preserving strict enterprise safety and human oversight.
1. What Is This?
This is a working prototype of an AI agent that autonomously handles late-shipment exceptions in e-commerce order fulfillment. When a customer's order is delayed, the agent investigates across multiple internal systems (OMS, CRM, inventory, carrier tracking), decides on the best resolution (wait, reship, upgrade shipping, or refund), and either executes it automatically or escalates it to a human, all within company-defined guardrails.
Portfolio Prototype Scope: This prototype is built with simulated JSON data sources (orders, customers, inventory, rate cards), demonstrating how an AI agent operates within a real operations team. While data is mock, the 8-step architecture, deterministic cost calculator, and human-in-the-loop guardrails are fully production-grade.
2. The Problem: Why This Exists
The Business Pain
Late shipments are the single largest source of customer support tickets in e-commerce. When an order misses its promise date, a human operations agent must manually:
- Look up the order in the Order Management System (OMS)
- Check customer tier and history in the CRM
- Search for available replacement stock in nearby warehouses
- Decide whether to wait, reship, upgrade shipping, or refund
- Calculate and compare the cost of each action
- Execute the cheapest acceptable action
- Draft and send a customer email
- Log the case for tracking
This takes 15-30 minutes per exception and is highly repetitive. During peak retail events (Diwali, Black Friday, Big Billion Days), thousands of exceptions pile up, leading to support backlogs and team burnout.
The Opportunity
An AI agent can handle 80% of these cases autonomously in under 2 seconds, while escalating the remaining 20% (high-value, ambiguous, or policy-edge cases) to human reviewers with a pre-filled recommendation. This creates:
- Faster resolution: From 30 minutes → 2 seconds for standard cases.
- Lower operational cost: Dramatic reduction in manual overhead for routine delays.
- Proactive customer experience: Customers are notified and resolved before they need to file a complaint ticket.
- Strict human governance: The agent never acts outside human-tuned financial and policy limits.
3. Architecture & Tech Stack
| Layer | Technology | Design Rationale |
|---|---|---|
| Backend | Python 3.11, FastAPI | High-performance asynchronous execution; native Pydantic data schemas |
| Frontend | Vanilla HTML / CSS / JS | Zero build pipeline; clean single-page app served directly by FastAPI |
| LLM | Google Gemini (google-genai SDK) |
Enforces structured JSON output and fast inference for qualitative reasoning |
| Data Layer | JSON files on disk | Lightweight, portable simulation for prototype evaluation |
| Hosting | Railway | One-click deployment with environment variable management |
Project Structure
exception-agent/ ├── app/ │ ├── main.py ← FastAPI routes (API layer) │ ├── models/ │ │ └── schemas.py ← Pydantic data models (TraceStep, RunTrace, Order) │ ├── services/ │ │ └── agent.py ← The agent engine (8-step pipeline) │ └── data/ │ ├── orders.json ← 3 mock delayed orders │ ├── crm.json ← 3 mock customers (Standard, Plus, Business) │ ├── inventory.json ← Warehouse stock levels per item │ ├── rate_card.json ← Cost of each action (wait, reship, upgrade, refund) │ ├── limits.json ← Decision guardrails (editable via Settings tab) │ ├── sop.md ← Standard Operating Procedure (fed to the LLM) │ └── golden_set.json ← 10 evaluation test cases ├── static/ │ ├── index.html ← Single-page app shell │ ├── css/style.css ← Design system │ └── js/app.js ← Frontend logic ├── requirements.txt ├── Procfile ← Railway deployment config └── README.md
4. The Agent Workflow, 8 Steps, Fully Traced
Every delayed order execution runs through an 8-step deterministic and probabilistic pipeline. Each step is recorded as a TraceStep object with inputs, outputs, tool invocations, and millisecond-level execution time.
Reads orders.json and validates if the order status is "Delayed". Non-delayed orders terminate the pipeline immediately to prevent unnecessary compute.
Retrieves the complete order record (destination city, promised delivery date, delay days) and links it with the customer profile (tier: Standard, Plus, or Business). Customer tier directly dictates downstream autonomy limits.
Queries inventory.json across all fulfillment centers for the delayed item. Returns a boolean closer_stock flag to verify if a warehouse closer to the destination can fulfill a reshipment.
Reads rate_card.json to load deterministic costs: Wait: ₹0, Upgrade Shipping: ₹500, Reship: ₹1,200, Refund: ₹2,500. Critical Rule: The LLM is never allowed to calculate financial numbers. It only reads pre-computed values.
Injects sop.md and order parameters into a Gemini prompt using response_mime_type="application/json". Returns decision, reason, and confidence score. If Gemini is unreachable or output fails schema validation, the pipeline falls back gracefully to a deterministic rule engine.
Applies company guardrails from limits.json to categorize the outcome into: Auto-resolved 🟢, Escalated 🟡, or Blocked 🔴.
Generates a personalized customer notification email. Standard tier auto-resolutions are dispatched immediately; Business/Plus tiers or escalated cases hold the draft for human agent approval.
Appends the complete RunTrace object (inputs, outputs, tool invocations, execution durations) to the observability store for real-time auditability.
5. The Decision Limits System
Decision limits represent the governance backbone of the architecture, defining what the AI agent is allowed to execute autonomously without human confirmation.
| Limit Parameter | Default Value | Operational Purpose |
|---|---|---|
| Max Action Cost | ₹1,500 | Escalates any proposed action exceeding this financial exposure |
| Min Confidence | 0.75 | Escalates when the LLM exhibits semantic hesitation or ambiguity |
| Max Refund | ₹2,000 | Caps automatic customer refunds; larger sums require human authorization |
| Max Tracking Silence | 48 Hours | Escalates carrier tracking gaps regardless of computed action |
The Three Immutable "Never" Rules
Hard-coded at the engine level and cannot be overridden through settings:
- Never cancel an order: The agent may wait, reship, upgrade shipping, or refund, but never cancel.
- Never issue refunds > 1.5x policy limit: Triggers an immediate Blocked 🔴 state.
- Never promise an unconfirmed delivery date: Carrier confirmation is strictly mandatory.
6. Simulated Scenarios & Data
| Order ID | Item | Destination | Delay | Customer Tier | Tracking Silence | Outcome |
|---|---|---|---|---|---|---|
| ORD-001 | Wireless Headphones | Bangalore, KA | 1 day | Standard | 24 hrs | 🟢 Auto-resolved (Wait, Cost: ₹0) |
| ORD-002 | Ergonomic Keyboard | Delhi, DL | 3 days | Plus | 50 hrs | 🟡 Escalated (Tracking silent > 48 hrs) |
| ORD-003 | 4K Monitor | Mumbai, MH | 5 days | Business | 96 hrs | 🟡 Escalated (Business Tier & Refund > ₹2k) |
7. Standard Operating Procedure (SOP) as Data
A fundamental architectural choice was treating company policy as plain-text data rather than hardcoded logic. The SOP file is injected fresh into every LLM prompt:
1. If the shipping delay is 1 day, the required action is "wait". 2. If the shipping delay is 4 or more days, the required action is "refund". 3. If the shipping delay is 2 or 3 days: a. Check if there is stock closer to the customer's destination. b. If closer stock exists, and the reship cost is lower than the refund cost, the required action is "reship". c. If no closer stock exists, the required action is "upgrade_shipping".
This allows product managers and operations leaders to update business policies instantly by modifying a markdown file without requiring code refactoring or deployments.
8. The 6-Tab Interface
Production-Grade UI Architecture
- Exceptions (Home): Two-pane view with real-time KPI cards (Open Exceptions, Avg Resolution Time, Auto-Resolved %, Impacted Customers) and order investigation chat.
- Queue: Human review inbox displaying escalated orders with pre-filled context, options, costs, and Approve / Reject actions.
- Traces: Deep execution audit trail listing all 8 steps per run with tool calls, inputs, outputs, and millisecond timings.
- Dashboard: Operational analytics computing override rates, decision source shares (LLM vs Rule Fallback), and rejection reasons.
- Evaluations: Interactive test bench running the 10-case golden set with pass/fail telemetry and regression validation.
- Settings: Editable decision limits (Max Cost, Min Confidence, Max Refund, Max Silent Hours) with a one-click demo reset.
9. The Golden Set, 10 Evaluation Cases
Automated regression testing ensures prompt tweaks or policy adjustments never break existing branches of the decision tree:
| ID | Case Name | Delay | Tier | Condition Tested | Expected State |
|---|---|---|---|---|---|
| EV-1 | Standard Wait | 1 day | Standard | Basic 1-day wait SOP | 🟢 Auto-resolved |
| EV-2 | Standard Refund | 4 days | Standard | 4+ day delay refund rule | 🟢 Auto-resolved |
| EV-3 | Reship Closer | 2 days | Standard | Closer stock + cheaper reship | 🟢 Auto-resolved |
| EV-4 | Upgrade No Stock | 2 days | Standard | No closer stock → upgrade | 🟢 Auto-resolved |
| EV-5 | Business Tier | 2 days | Business | Business tier VIP escalation | 🟡 Escalated |
| EV-6 | Silent Tracking | 1 day | Standard | 50h silence > 48h ceiling | 🟡 Escalated |
| EV-7 | Refund Over Limit | 4 days | Standard | ₹2,200 refund > ₹2,000 max | 🟡 Escalated |
| EV-8 | Refund Blocked | 4 days | Standard | ₹3,500 refund > 1.5x limit | 🔴 Blocked |
| EV-9 | Cost Limit | 2 days | Standard | ₹1,600 action > ₹1,500 max | 🟡 Escalated |
| EV-10 | Reship 3 Days | 3 days | Standard | 3-day delay with closer stock | 🟢 Auto-resolved |
10. LLM Integration Principles
Four Pillars of Responsible AI Product Design
- LLM for Judgement, Never Calculation: The LLM reads deterministic costs; it is never asked to perform arithmetic, eliminating hallucinated finances.
- Structured JSON Output: Enforced through Gemini's
response_mime_type="application/json"schema:{"decision": "wait|refund|reship|upgrade_shipping", "reason": "...", "confidence": 0.95} - Graceful Degradation: If the model fails, times out, or produces invalid output, the agent falls back instantly to deterministic rules.
- The LLM Never Has the Final Say: Step 6 validates LLM recommendations against hard policy thresholds, customer tier locks, and financial caps.
11. Human-in-the-Loop Escalation & Feedback Loop
The escalation queue is designed as an operational learning loop. When human reviewers reject an agent's recommendation and log an override reason, that data surfaces in the Observability Dashboard.
The override rate (rejections ÷ escalations) functions as the primary diagnostic signal for when policy SOPs or limit thresholds require tuning.
12. Key PM Takeaways Demonstrated
What This Demonstrates for AI Product Management
- Systems Design Thinking: Designed an 8-step agentic pipeline separating data retrieval, probabilistic reasoning, and policy execution.
- Tiered Autonomy Governance: Built an Auto-Resolve / Escalate / Block model with tunable thresholds, the exact pattern required by enterprise e-commerce leaders.
- AI Safety & Financial Risk Mitigation: Separated arithmetic calculation from linguistic reasoning to eliminate hallucinations.
- Evaluation-Driven Development: Built a 10-case golden benchmark covering edge conditions to validate agent reliability before rollout.
- End-to-End Observability: Implemented millisecond-level step traces, tool attribution, and override analytics for operational auditability.
Built by Keerttik Kumar • October 2026
keerttik.in • SAFe® 6 Certified AI Product Manager