The Problem
Client kami — fashion e-commerce dengan ~15,000 SKU dan 200–400 customer inquiries sehari — ada support team 3 orang yang handle semua inquiry melalui WhatsApp Business dan email. Peak period (sale, raya, harbolnas) boleh cecah 800+ inquiries sehari. Masa response average: 4.2 jam. Customer satisfaction score: 3.1/5.
Breakdown inquiry mengikut kategori (dari 2-minggu sample sebelum project bermula):
| Inquiry Type | % of Volume | Avg. Handle Time |
|---|---|---|
| Order status / tracking | 34% | 3 min |
| Product availability / sizing | 28% | 5 min |
| Return & exchange policy | 18% | 4 min |
| Payment & promo queries | 11% | 6 min |
| Complaints & escalations | 9% | 22 min |
91% dari volume — order status, product availability, return policy, payment queries — adalah repeatable, structured problems. Tu use case yang kuat untuk AI agent.
9% yang remaining (complaints) memerlukan empathy, negotiation, dan sometimes compensation authority. Itu kena kekal dengan human.
Our Approach — Architecture Decision Log
Decision 1: LangGraph untuk orchestration, bukan simple chain
Pilihan pertama yang kami kena buat: guna simple LLM chain (prompt → response) atau full agent dengan state management. Kami evaluate tiga option:
- Simple chain — laju nak deploy, tapi tak boleh handle multi-step resolution (e.g., check order status → check warehouse → update customer). Eliminated.
- LangGraph dengan single agent — yang kami pilih. Boleh define explicit state graph, tool calling yang clean, dan escalation edges yang well-defined.
- Multi-agent supervisor pattern — over-engineered untuk use case ni. Reserve untuk Log #003.
LangGraph bagi kami explicit control over state transitions — penting bila dealing dengan customer data dan financial transactions. Kalau agent dalam ambiguous state, kami boleh define clearly apa yang perlu jadi seterusnya.
Decision 2: RAG untuk static knowledge, live API call untuk dynamic data
Mistake yang ramai orang buat: embed semua benda dalam vector store. Product descriptions — okay. Inventory levels, order status, pricing — jangan sekali.
Kami bina dua layer:
# Layer 1: RAG untuk static/semi-static knowledge # - Product descriptions, sizing guides, material info # - Return & exchange policies (update weekly) # - FAQ database (300+ curated Q&A pairs) # - Promo terms & conditions vector_store = pgvector_store( embedding_model="text-embedding-3-small", table="product_knowledge", similarity_threshold=0.78 # tuned after 200 manual evals ) # Layer 2: Live tool calls untuk dynamic data # - Order status via order management API # - Real-time stock check via inventory API # - Customer tier & purchase history via CRM API tools = [ check_order_status, # → order management system check_inventory, # → warehouse API (real-time) get_customer_profile, # → CRM (loyalty tier, history) initiate_return, # → returns portal (write access) escalate_to_human, # → CRM ticket creation + notify Slack ]
Separation ni critical. Inventory yang embedded semalam dah stale. Agent yang bagi wrong stock info kepada customer adalah worse than no agent at all.
Decision 3: Intent classification sebelum routing
Sebelum agent decide apa nak buat, dia perlu classify intent dengan betul. Kami bina explicit intent classifier sebagai first step dalam graph:
# Intent categories — berdasarkan 2-minggu historical data INTENT_MAP = { "order_inquiry": ["where is my order", "tracking", "bila sampai"], "product_query": ["ada saiz", "available tak", "color lain"], "return_exchange": ["nak return", "salah saiz", "exchange"], "payment_promo": ["promo code", "payment failed", "refund"], "complaint": ["teruk", "kecewa", "tak puas", "complain"], "ambiguous": [] # fallback — clarify before proceeding } # Ambiguous intent → ask clarifying question, don't guess # This was the biggest fix in week 3 (see failures section)
Decision 4: HITL escalation design
Kami define empat escalation triggers yang hard-coded — agent tak boleh override ini:
- Complaint intent detected — auto-escalate, no exceptions
- Refund request > RM150 — perlu human approval
- Agent confidence < 0.72 selepas 2 clarification attempts — route to human
- Customer explicitly asks for human — immediate escalation, no questions asked
Bila escalate, agent handoff dengan context summary — human agent dapat full conversation history plus structured summary. Masa onboarding untuk human agent drop dari average 3 minit ke 45 saat.
Technical Stack
| Layer | Technology | Why |
|---|---|---|
| LLM | Claude (claude-sonnet-4-6) | Best instruction following; Malay language quality significantly better than alternatives we tested |
| Agent framework | LangGraph ≥0.2 | Explicit state graph, clean tool integration, built-in checkpointing |
| API layer | FastAPI + async Python | Lightweight, handles concurrent WhatsApp webhooks cleanly |
| Vector store | pgvector on PostgreSQL | Client already on PostgreSQL — avoid new infra overhead |
| Embeddings | text-embedding-3-small | Cost-efficient for this retrieval task; overkill for large-batch embedding |
| Channel integration | WhatsApp Business API via 360dialog | Client requirement; 360dialog has cleaner webhook reliability vs direct Meta API |
| Deployment | AWS EC2 t3.medium + RDS PostgreSQL | Client infra preference; ample for this load |
What Broke in Production
Ini bahagian yang paling penting dalam log ni. Benda yang work dalam testing selalu ada surprise bila real users masuk.
Failure 1: Ambiguous queries yang agent buat assumption salah
Dalam minggu pertama production, kami notice pattern yang concerning: agent kadang-kadang jawab confident tapi salah bila query customer ambiguous. Example real dari logs (anonymised):
Customer: "Nak tanya pasal baju tu"
Agent buat assumption ia referring to the last item the customer viewed on the website (dari session data yang kami pass dalam context). Tapi customer actually referring to a different item dari purchase 2 bulan lepas. Agent bagi wrong product info dengan confident tone.
Fix: Tambah mandatory clarification step bila entity reference ambiguous. "Boleh korang confirm nama atau order number untuk item yang dimaksudkan?" sebelum proceed. Conversion rate dari clarification step: 89% — customer lebih prefer clarification dari wrong answer.
Failure 2: Malay-English code-switching dalam customer messages
Kami underestimate betapa creatively Malaysian customers code-switch. Messages macam "Baju i dah 2 weeks tak received lah, tracking pun taktahu nak bagi apa" — campuran BM, English, dan Manglish — kadang confuse intent classifier yang kami tune pakai lebih formal text.
Fix: Retrain intent classifier dengan actual production messages (minggu 2). Tambah normalization layer untuk common Manglish patterns. Selepas fix, intent accuracy improved dari 81% ke 94%.
Failure 3: Inventory API timeout cascade
Warehouse inventory API ada uptime issue — occasional 30-second timeouts. Bila agent kena call inventory API untuk product availability query dan API timeout, agent stuck dalam waiting state. Customer dapat no response. Lepas 60 seconds, WhatsApp timeout, customer perlu message semula.
Fix: Implement circuit breaker pattern dengan fallback response. Kalau inventory API timeout selepas 8 seconds, agent respond: "Kami tengah semak ketersediaan stok untuk korang — boleh kami WhatsApp balik dalam 10 minit?" Ini human-sounding fallback yang better daripada silence.
Results
CSAT breakdown by resolution type selepas 4 minggu production:
| Resolution Type | Volume % | CSAT Score |
|---|---|---|
| Fully automated resolution | 73% | 4.2/5 |
| AI + HITL escalation (proper handoff) | 19% | 4.6/5 |
| Direct human (complaint queue) | 8% | 3.9/5 |
Unexpected finding: Customers yang properly di-escalate ke human (dengan full context handoff dari agent) bagi higher satisfaction score daripada fully automated resolution. Transition yang smooth — where the human agent clearly understood the context — is more important than automation percentage.
Untuk client: support team 3 orang tu sekarang handle 27% dari volume instead of 100%. Mereka fokus sepenuhnya pada complaints dan complex cases. Overtime hours drop 68%. Mereka lebih happy, customer lebih happy.
Apa yang Kami Akan Buat Lain Kali
- Start dengan intent classification evaluation earlier. Kami spend terlalu banyak masa fine-tuning RAG sebelum verify yang intent classifier robust. Kalau intent classification salah, semua downstream logic salah juga. Test ini dulu.
- Build circuit breakers dari day 1. External API dependency adalah failure point yang predictable. Kami learnt ini di production — sepatutnya design untuk ia sebelum launch.
- Instrument everything from the start. Kami tak ada sufficient logging untuk trace exactly kenapa specific messages got wrong intents. Took us 3 days to debug sesuatu yang would have been 30 minutes dengan proper observability dari awal.
- Customer-facing language needs real Malaysian customer data. Synthetic test data atau formal BM tidak represent macam mana customer sebenar type. Get real historical messages approved for use in training/testing seawal mungkin.
Takeaway untuk Business Yang Similar
Kalau business korang ada customer support operation dengan volume yang predictable dan inquiry yang mostly follow repeatable patterns — ini adalah strong AI agent candidate.
Dua prerequisite yang non-negotiable sebelum start:
- Clean, accessible product/policy data. Kalau product catalogue korang ada 40% incomplete descriptions, agent korang akan hallucinate untuk fill the gaps. Fix data first.
- Live API access untuk dynamic data. Order status, inventory — kena ada proper API. Kalau masih manual lookup, agent tak boleh close the loop.
Kalau dua benda tu dah ada, 73% deflection rate bukan luar biasa. Kami expect itu jadi baseline untuk well-scoped implementations macam ni.