Late Shipment Exception Agent

Autonomous decision intelligence with human-in-the-loop governance for e-commerce fulfillment. Resolves 80% of delayed orders in under 2 seconds while preserving strict enterprise safety and human oversight.

Launch Live Application View GitHub Repo Back to Portfolio
30m → 2s
Resolution time per exception
80%
Autonomous resolution rate
(on simulated order data )
8 Steps
Fully traced audit pipeline
10 / 10
Golden set eval cases pass

1. What Is This?

This is a working prototype of an AI agent that autonomously handles late-shipment exceptions in e-commerce order fulfillment. When a customer's order is delayed, the agent investigates across multiple internal systems (OMS, CRM, inventory, carrier tracking), decides on the best resolution (wait, reship, upgrade shipping, or refund), and either executes it automatically or escalates it to a human, all within company-defined guardrails.

Portfolio Prototype Scope: This prototype is built with simulated JSON data sources (orders, customers, inventory, rate cards), demonstrating how an AI agent operates within a real operations team. While data is mock, the 8-step architecture, deterministic cost calculator, and human-in-the-loop guardrails are fully production-grade.

2. The Problem: Why This Exists

The Business Pain

Late shipments are the single largest source of customer support tickets in e-commerce. When an order misses its promise date, a human operations agent must manually:

  1. Look up the order in the Order Management System (OMS)
  2. Check customer tier and history in the CRM
  3. Search for available replacement stock in nearby warehouses
  4. Decide whether to wait, reship, upgrade shipping, or refund
  5. Calculate and compare the cost of each action
  6. Execute the cheapest acceptable action
  7. Draft and send a customer email
  8. Log the case for tracking

This takes 15-30 minutes per exception and is highly repetitive. During peak retail events (Diwali, Black Friday, Big Billion Days), thousands of exceptions pile up, leading to support backlogs and team burnout.

The Opportunity

An AI agent can handle 80% of these cases autonomously in under 2 seconds, while escalating the remaining 20% (high-value, ambiguous, or policy-edge cases) to human reviewers with a pre-filled recommendation. This creates:

  • Faster resolution: From 30 minutes → 2 seconds for standard cases.
  • Lower operational cost: Dramatic reduction in manual overhead for routine delays.
  • Proactive customer experience: Customers are notified and resolved before they need to file a complaint ticket.
  • Strict human governance: The agent never acts outside human-tuned financial and policy limits.

3. Architecture & Tech Stack

Layer Technology Design Rationale
Backend Python 3.11, FastAPI High-performance asynchronous execution; native Pydantic data schemas
Frontend Vanilla HTML / CSS / JS Zero build pipeline; clean single-page app served directly by FastAPI
LLM Google Gemini (google-genai SDK) Enforces structured JSON output and fast inference for qualitative reasoning
Data Layer JSON files on disk Lightweight, portable simulation for prototype evaluation
Hosting Railway One-click deployment with environment variable management

Project Structure

exception-agent/
├── app/
│   ├── main.py              ← FastAPI routes (API layer)
│   ├── models/
│   │   └── schemas.py       ← Pydantic data models (TraceStep, RunTrace, Order)
│   ├── services/
│   │   └── agent.py         ← The agent engine (8-step pipeline)
│   └── data/
│       ├── orders.json       ← 3 mock delayed orders
│       ├── crm.json          ← 3 mock customers (Standard, Plus, Business)
│       ├── inventory.json    ← Warehouse stock levels per item
│       ├── rate_card.json    ← Cost of each action (wait, reship, upgrade, refund)
│       ├── limits.json       ← Decision guardrails (editable via Settings tab)
│       ├── sop.md            ← Standard Operating Procedure (fed to the LLM)
│       └── golden_set.json   ← 10 evaluation test cases
├── static/
│   ├── index.html            ← Single-page app shell
│   ├── css/style.css         ← Design system
│   └── js/app.js             ← Frontend logic
├── requirements.txt
├── Procfile                  ← Railway deployment config
└── README.md

4. The Agent Workflow, 8 Steps, Fully Traced

Every delayed order execution runs through an 8-step deterministic and probabilistic pipeline. Each step is recorded as a TraceStep object with inputs, outputs, tool invocations, and millisecond-level execution time.

Step 1: Trigger Check Rule

Reads orders.json and validates if the order status is "Delayed". Non-delayed orders terminate the pipeline immediately to prevent unnecessary compute.

Step 2: OMS / CRM Lookup Lookup

Retrieves the complete order record (destination city, promised delivery date, delay days) and links it with the customer profile (tier: Standard, Plus, or Business). Customer tier directly dictates downstream autonomy limits.

Step 3: Inventory Stock Search Lookup

Queries inventory.json across all fulfillment centers for the delayed item. Returns a boolean closer_stock flag to verify if a warehouse closer to the destination can fulfill a reshipment.

Step 5: Pre-Computed Cost Calculator (Executed before Step 4) Calculation

Reads rate_card.json to load deterministic costs: Wait: ₹0, Upgrade Shipping: ₹500, Reship: ₹1,200, Refund: ₹2,500. Critical Rule: The LLM is never allowed to calculate financial numbers. It only reads pre-computed values.

Step 4: LLM Judgement (or Rules Fallback) Judgement

Injects sop.md and order parameters into a Gemini prompt using response_mime_type="application/json". Returns decision, reason, and confidence score. If Gemini is unreachable or output fails schema validation, the pipeline falls back gracefully to a deterministic rule engine.

Step 6: Guardrail Limits Enforcement Action

Applies company guardrails from limits.json to categorize the outcome into: Auto-resolved 🟢, Escalated 🟡, or Blocked 🔴.

Step 7: Context-Aware Email Notification Drafting

Generates a personalized customer notification email. Standard tier auto-resolutions are dispatched immediately; Business/Plus tiers or escalated cases hold the draft for human agent approval.

Step 8: Millisecond Trace Logging Audit

Appends the complete RunTrace object (inputs, outputs, tool invocations, execution durations) to the observability store for real-time auditability.

5. The Decision Limits System

Decision limits represent the governance backbone of the architecture, defining what the AI agent is allowed to execute autonomously without human confirmation.

Limit Parameter Default Value Operational Purpose
Max Action Cost ₹1,500 Escalates any proposed action exceeding this financial exposure
Min Confidence 0.75 Escalates when the LLM exhibits semantic hesitation or ambiguity
Max Refund ₹2,000 Caps automatic customer refunds; larger sums require human authorization
Max Tracking Silence 48 Hours Escalates carrier tracking gaps regardless of computed action

The Three Immutable "Never" Rules

Hard-coded at the engine level and cannot be overridden through settings:

  • Never cancel an order: The agent may wait, reship, upgrade shipping, or refund, but never cancel.
  • Never issue refunds > 1.5x policy limit: Triggers an immediate Blocked 🔴 state.
  • Never promise an unconfirmed delivery date: Carrier confirmation is strictly mandatory.

6. Simulated Scenarios & Data

Order ID Item Destination Delay Customer Tier Tracking Silence Outcome
ORD-001 Wireless Headphones Bangalore, KA 1 day Standard 24 hrs 🟢 Auto-resolved (Wait, Cost: ₹0)
ORD-002 Ergonomic Keyboard Delhi, DL 3 days Plus 50 hrs 🟡 Escalated (Tracking silent > 48 hrs)
ORD-003 4K Monitor Mumbai, MH 5 days Business 96 hrs 🟡 Escalated (Business Tier & Refund > ₹2k)

7. Standard Operating Procedure (SOP) as Data

A fundamental architectural choice was treating company policy as plain-text data rather than hardcoded logic. The SOP file is injected fresh into every LLM prompt:

1. If the shipping delay is 1 day, the required action is "wait".
2. If the shipping delay is 4 or more days, the required action is "refund".
3. If the shipping delay is 2 or 3 days:
   a. Check if there is stock closer to the customer's destination.
   b. If closer stock exists, and the reship cost is lower than the refund cost, the required action is "reship".
   c. If no closer stock exists, the required action is "upgrade_shipping".

This allows product managers and operations leaders to update business policies instantly by modifying a markdown file without requiring code refactoring or deployments.

8. The 6-Tab Interface

Production-Grade UI Architecture

  • Exceptions (Home): Two-pane view with real-time KPI cards (Open Exceptions, Avg Resolution Time, Auto-Resolved %, Impacted Customers) and order investigation chat.
  • Queue: Human review inbox displaying escalated orders with pre-filled context, options, costs, and Approve / Reject actions.
  • Traces: Deep execution audit trail listing all 8 steps per run with tool calls, inputs, outputs, and millisecond timings.
  • Dashboard: Operational analytics computing override rates, decision source shares (LLM vs Rule Fallback), and rejection reasons.
  • Evaluations: Interactive test bench running the 10-case golden set with pass/fail telemetry and regression validation.
  • Settings: Editable decision limits (Max Cost, Min Confidence, Max Refund, Max Silent Hours) with a one-click demo reset.

9. The Golden Set, 10 Evaluation Cases

Automated regression testing ensures prompt tweaks or policy adjustments never break existing branches of the decision tree:

ID Case Name Delay Tier Condition Tested Expected State
EV-1 Standard Wait 1 day Standard Basic 1-day wait SOP 🟢 Auto-resolved
EV-2 Standard Refund 4 days Standard 4+ day delay refund rule 🟢 Auto-resolved
EV-3 Reship Closer 2 days Standard Closer stock + cheaper reship 🟢 Auto-resolved
EV-4 Upgrade No Stock 2 days Standard No closer stock → upgrade 🟢 Auto-resolved
EV-5 Business Tier 2 days Business Business tier VIP escalation 🟡 Escalated
EV-6 Silent Tracking 1 day Standard 50h silence > 48h ceiling 🟡 Escalated
EV-7 Refund Over Limit 4 days Standard ₹2,200 refund > ₹2,000 max 🟡 Escalated
EV-8 Refund Blocked 4 days Standard ₹3,500 refund > 1.5x limit 🔴 Blocked
EV-9 Cost Limit 2 days Standard ₹1,600 action > ₹1,500 max 🟡 Escalated
EV-10 Reship 3 Days 3 days Standard 3-day delay with closer stock 🟢 Auto-resolved

10. LLM Integration Principles

Four Pillars of Responsible AI Product Design

  1. LLM for Judgement, Never Calculation: The LLM reads deterministic costs; it is never asked to perform arithmetic, eliminating hallucinated finances.
  2. Structured JSON Output: Enforced through Gemini's response_mime_type="application/json" schema:
    {"decision": "wait|refund|reship|upgrade_shipping", "reason": "...", "confidence": 0.95}
  3. Graceful Degradation: If the model fails, times out, or produces invalid output, the agent falls back instantly to deterministic rules.
  4. The LLM Never Has the Final Say: Step 6 validates LLM recommendations against hard policy thresholds, customer tier locks, and financial caps.

11. Human-in-the-Loop Escalation & Feedback Loop

The escalation queue is designed as an operational learning loop. When human reviewers reject an agent's recommendation and log an override reason, that data surfaces in the Observability Dashboard.

The override rate (rejections ÷ escalations) functions as the primary diagnostic signal for when policy SOPs or limit thresholds require tuning.

12. Key PM Takeaways Demonstrated

What This Demonstrates for AI Product Management

  • Systems Design Thinking: Designed an 8-step agentic pipeline separating data retrieval, probabilistic reasoning, and policy execution.
  • Tiered Autonomy Governance: Built an Auto-Resolve / Escalate / Block model with tunable thresholds, the exact pattern required by enterprise e-commerce leaders.
  • AI Safety & Financial Risk Mitigation: Separated arithmetic calculation from linguistic reasoning to eliminate hallucinations.
  • Evaluation-Driven Development: Built a 10-case golden benchmark covering edge conditions to validate agent reliability before rollout.
  • End-to-End Observability: Implemented millisecond-level step traces, tool attribution, and override analytics for operational auditability.

Built by Keerttik Kumar • October 2026

keerttik.in • SAFe® 6 Certified AI Product Manager