Agentic AI
Program-level details: See program/curriculum.md
| Credits: 4 | Course Number: BADM 590 (temporary, year 1) | Term: Spring 2027 (Weeks 9-16, POT B, starts Mar 15) | Instructor: Vishal Sachdev (Lead), Co-Instructor TBD (adjunct hire pending — per AY26-27 schedule 2026-06-24) |
Status: Draft Instructor confirmed. Pending T&L onboarding. Rebuilt to 8-week format 2026-07-15.
Course Vision
Students move beyond using AI to building, evaluating, and governing agentic AI systems for analytics work. The course teaches the agent loop from first principles — before any framework — then grounds agents in organizational data, connects them to enterprise systems, and makes systematic evaluation the discipline that separates a demo from a system worth trusting. By course end, teams can design, implement, evaluate, and document an agent-based analytics workflow for a real business problem, and every student can defend the design choices orally.
Design commitment — 3-year concepts, 1-year tools. Learning outcomes, milestones, and rubrics are written entirely in terms of durable concepts and methods (the agent loop, design patterns, retrieval grounding, interoperability protocols, evaluation methodology, governance). Named products and frameworks appear only in the Technology Stack (Lab Layer) and inside lab assignments, and are reviewed before each cohort. No tool name appears in a learning outcome or rubric.
Learning Outcomes (L-C-E Framework)
Literacy:
- L1: Explain how large language models generate output (token prediction, context windows) and why their two structural limitations — lack of judgment and lack of domain expertise — make human evaluation irreducible
- L2: Distinguish workflows from agents, and describe the agent loop and the four durable agentic design patterns (reflection, tool use, planning, multi-agent collaboration)
- L3: Recognize what builders owe when deploying agentic systems: documentation, risk assessment, security threat awareness, and accountability that cannot be delegated to the system
Competency:
- C1: Implement an agent from first principles — a model, a tool loop, and structured outputs — without an orchestration framework
- C2: Ground an agent in organizational data through retrieval (chunking, embedding, vector search) and measure retrieval quality
- C3: Connect agents to external systems (databases, APIs, analytics tools) through standard interoperability protocols
- C4: Design and run systematic evaluations of an agentic system — error analysis, test suites, and human-judgment calibration
- C5: Apply context engineering to direct agent behavior: instructions, grounding materials, tool selection, and memory within a bounded context window
Expertise:
- E1: Architect multi-step and multi-agent analytics workflows that combine agents, retrieval, and human judgment at deliberate control points
- E2: Evaluate architectural trade-offs — workflow vs. agent, retrieval vs. fine-tuning, framework vs. first-principles loop — for a given business problem
- E3: Deliver a governed agentic system: evaluated, documented, security-reviewed, and defensible to a professional audience
Week-by-Week Breakdown
| Week | Topic | Lectures | Project Work | Studio Session | Assessment |
|---|---|---|---|---|---|
| 1 | LLM foundations + the agent loop | How LLMs generate; judgment & expertise gaps; workflow vs. agent; the loop (model → tool call → observation) built from first principles | Team formation; problem scoping | Peer-review calibration (15 min) + Arena Round 1: prompt strategies for competing negotiation agents | Lab 1: raw-API agent loop (no framework) |
| 2 | Context engineering + tool use | From prompt to context: instructions, grounding, structured outputs; function calling and tool schemas from first principles | M1: Problem statement + data sources | Arena Round 2: adversarial prompting, persona attacks, jailbreaks → AI-safety debrief | Lab 2: tool-using agent for SQL/Python analytics tasks |
| 3 | Agentic retrieval (RAG) | Embeddings, chunking strategies, vector search, retrieval evaluation; when retrieval beats fine-tuning | Ground the team agent in project data | RAG architecture peer critique of team prototypes | Lab 3: retrieval pipeline + retrieval-quality measurement · Individual explainer video 1 |
| 4 | Interoperability: connecting agents to systems | Tool protocols (MCP); connecting to databases, APIs, analytics libraries; least-privilege tool design | M2: Grounded agent prototype (retrieval + ≥1 external tool) | Live build: wire the team agent to an external system via a standard protocol | Lab 4: agent + protocol-connected tool · Peer-eval formative pulse (2%) |
| 5 | Evaluation I: judging agentic work | Error analysis; defining “good”; eval suites for agent outputs; benchmark thinking; why evals are the core discipline of 2026 agent engineering | Build the team eval suite | Analyst vs. Auditor: one team’s agent produces an analysis, another’s audits it — scored both ways | Lab 5: eval harness with ≥15 test cases + error taxonomy |
| 6 | Evaluation II + orchestration frameworks | Human-judgment calibration; regression testing agents; then — and only then — one orchestration framework as an instance of the patterns already built by hand | M3: Evaluation report (eval results, error analysis, iteration evidence) | Framework build sprint: re-implement the Week 1 loop in the framework; compare · (Optional enrichment: Fine-Tuning Showdown) | Lab 6: framework re-implementation + trade-off memo · Individual explainer video 2 |
| 7 | Multi-agent systems + governance & security | Orchestrator and handoff patterns; shared-state conflicts; prompt injection and agent security (OWASP agentic risks); builder governance: model documentation, risk assessment (NIST AI RMF as reference) | Governance package: model card, risk assessment, security review | Source of Truth Wars: competing agents write to a shared knowledge base; teams design synchronization protocols | Governance package due (feeds final deliverable) |
| 8 | Production hardening + delivery | Cost/latency awareness; failure modes and human-in-the-loop control points; from course project to portfolio piece | Final deliverable: team agentic analytics system | Team presentations + individual oral defenses | Final system + team presentation · Individual oral defense · Peer-eval summative (8%) |
Assessments (Framework v3 — 8-week, 4-credit)
Weights follow the four program-wide buckets (assessment_strategy.md §2). Faculty-chosen split within each bucket shown below.
Individual Coursework Assignments — ICA (30%)
- Labs 1-6 (Weeks 1-6): scaffolded builds — agent loop, tool use, retrieval, protocol integration, eval harness, framework re-implementation. Each lab is individually submitted and feeds a skill the team project needs that week.
- Live Session engagement (Canvas discussions, included here per program commitment): watch the week’s Live Session (live or recording), post one insight/question/application, respond to one peer.
Project — Individual Deliverables, including oral defense (30%) Internal split: explainer videos 5% each, oral defense 20% (within the §0 “significant portion” guidance):
- Explainer video 1 (Week 3, 5%): 3-minute individual walkthrough of the retrieval design and one deliberate trade-off
- Explainer video 2 (Week 6, 5%): 3-minute individual walkthrough of eval results and what they changed
- Individual oral defense (Week 8, 20%, live in Studio preferred): 10-minute defense + Q&A; any member must be able to answer questions on any part of the system
Project — Team Deliverables (30%) One major team project (teams of 3) scaffolded across all 8 weeks. Studio output is graded here, not separately:
- M1 (Week 2): Problem statement + data sources
- M2 (Week 4): Grounded agent prototype — retrieval over project data + at least one protocol-connected external tool
- M3 (Week 6): Evaluation report — eval-suite results, error taxonomy, evidence of iteration
- Final (Week 8): Agent-based analytics workflow — multi-step (or multi-agent) system with retrieval, tool integration, eval suite, governance package (model card, risk assessment, security review), team presentation, and repository with documentation
Engagement — peer evaluation of teammates (10%) Program-wide commitment via Peerceptiv: Week 4 formative pulse (2%, completion-graded) + Week 8 summative rating (8%) on contribution / reliability / communication / collaboration.
Assessment Summary
| Component | Weight | Timing |
|---|---|---|
| Individual Coursework Assignments (Labs 1-6 + Canvas Live Session engagement) | 30% | Weeks 1-6 |
| Project — Individual Deliverables (2 explainer videos + oral defense) | 30% | Weeks 3, 6, 8 |
| Project — Team Deliverables (M1-M3 + final system) | 30% | Weeks 2, 4, 6, 8 |
| Engagement (peer evaluation via Peerceptiv) | 10% | Weeks 4, 8 |
No traditional exam. Project-based with AI systems focus. Total = 100%.
AI Usage Levels (AIAS)
| Assessment | AIAS Level | AI Permitted |
|---|---|---|
| Labs 1-6 | 4 | AI is the subject — students build, evaluate, and critique AI systems |
| Team milestones + final system | 4 | AI is the subject — students design, implement, and evaluate agentic systems |
| Individual explainer videos | 2 | AI for drafting/rehearsal only; the explanation must be the student’s own |
| Oral defense | 0 | No AI |
| Studio sessions (not graded separately) | 3 | AI as collaborator — full integration for hands-on experimentation; output folds into team deliverables |
| Live Session engagement (Canvas discussions, in ICA) | 1 | AI for ideation only when reflecting on the week’s content |
| Peer evaluation | 0 | No AI — personal judgment of teammates |
Rubric (5 dimensions, final team system)
| Dimension | Excellent (A) | Proficient (B) | Developing (C) |
|---|---|---|---|
| Agent System Design | Deliberate architecture; patterns chosen and justified against alternatives; handles failure modes and edge cases | Functional agent workflow; basic pattern use; some justification | Minimal agent functionality; architecture unexamined |
| Data Grounding (Retrieval) | Well-architected retrieval; measured quality; chunking/embedding choices justified against the data | Functional retrieval; adequate quality; partial rationale | Basic retrieval; quality unmeasured or poor |
| Evaluation Rigor | Systematic eval suite; error taxonomy; evidence that evaluation drove iteration | Eval suite exists; results reported; limited iteration evidence | Ad hoc or missing evaluation |
| Governance & Security | Complete model card, risk assessment, and security review (injection/tool-permission analysis); accountability clearly assigned | Adequate documentation; partial risk/security coverage | Minimal or missing governance |
| Oral Defense (individual) | Explains any part of the architecture clearly; handles questions confidently; articulates trade-offs and what the evals changed | Adequate explanation; answers most questions | Cannot explain choices or attribute design decisions |
Competitive Agent Studio Exercises
Agent-vs-agent competition teaches prompting, evaluation, and AI-safety concepts through gameplay. Inspired by Manzoor (2026) at Cornell (haggleforme.computer). Scheduled in the week-by-week table above.
Arena Rounds 1-2: Procurement Negotiation Arena (Weeks 1-2)
- Students write prompt strategies for AI buyer/seller agents negotiating supplier contracts
- Agents compete in round-robin tournaments; leaderboard tracks surplus
- Pedagogical progression: simple prompting (Week 1) → adversarial prompting, persona prompts, jailbreak attempts (Week 2)
- Debrief: students discover AI safety/ethics concerns organically through gameplay
- AIAS Level 4: AI is the subject of analysis
Analyst vs. Auditor (Week 5 — anchors the evaluation unit)
- One team’s agent generates an analytics report (with intentional methodology choices)
- Another team’s agent audits it (finds errors, biases, missing context)
- Competitive scoring: analysts earn points for convincing reports; auditors earn points for legitimate catches
- Connects to: Collier & Powell (2026) shift from “technical creators to AI auditors” — and doubles as live evaluation practice
Fine-Tuning Showdown (Week 6, optional enrichment)
- Teams provide 20-50 training examples to shape a negotiation persona
- Instructor fine-tunes models overnight; fine-tuned agents compete next studio
- Students learn: data quality matters more than data quantity; subtle training-data corruption changes behavior
- Debrief: Betley et al. (2025) “90 Wolf Facts” — how benign-looking data can corrupt models
Source of Truth Wars (Week 7 — anchors the multi-agent unit)
- Teams deploy multiple agents writing to a shared knowledge base simultaneously — each agent draws on a different data source or perspective
- Students observe how competing versions of “truth” emerge and compound; downstream agents consume contradictory facts without knowing it
- Challenge: design a synchronization protocol that lets agents write independently while maintaining coherent shared state
- Pedagogical frame: Hayek’s dispersed knowledge problem applied to AI systems — no single agent holds all relevant information, so how do organizations design principled contact points for integration?
- Connects to: Tonk Labs “source of truth problem is getting worse with agents” — append-only, conflict-free data structures as a structural solution vs. heuristic conflict checks
- Debrief: students articulate the difference between a stack (rigid, centralized, enclosed) and a substrate (malleable, distributed, conflict-free merge) — and why it matters as agent proliferation accelerates
- AIAS Level 4: students are designing and auditing multi-agent systems, not just using them
Implementation note: These exercises can be built with AI coding agents using a SPEC.md — see design/faculty_resources.md for the workflow. The instructor doesn’t need web development skills.
Technology Stack (Lab Layer)
Reviewed before each cohort. These are the current instances of the durable concepts taught in the course; the concepts (agent loop, retrieval grounding, interoperability protocol, evaluation methodology) are the commitment, the products below are not. Refresh is a scheduled part of course prep, not a syllabus change.
- Models/APIs: Claude API, OpenAI API (agent loop built raw against these first)
- Interoperability: MCP (Model Context Protocol) — the current standard tool-connection layer
- Orchestration framework (one, taught Week 6 as an instance): LangGraph or a vendor agent SDK — selected at course prep
- Retrieval: ChromaDB (primary); embedding model selected at course prep
- Environment: VS Code + Copilot / Jupyter Notebooks (program-standard tools, program/tools.md)
- Evaluation: first-principles eval harnesses (pytest-style); eval tooling selected at course prep
- Governance references: NIST AI RMF; OWASP agentic-application risk list
- In-house assets: candidate lab scaffolds and worked examples from Gies-built projects (canvas-mcp for the MCP week, ai-model-evaluation as the eval-harness base, K-ai as the Week 7 governance case study, fine-tune-open-llm for the optional showdown) — see
reference/agentic-tooling-inventory.md(internal), evaluated per cohort like every other lab-layer choice
Prerequisites
By Week 9 of Spring 2027, students have completed BADM 554, BDI 513, FIN 550, and Quantum Approaches, with BADM 558 in progress. This course assumes:
- Python and SQL working fluency (BADM 554, FIN 550)
- Supervised-ML concepts: train/test discipline, overfitting, evaluation metrics (FIN 550)
- Analytics workflow experience: framing a business problem as a data problem
Pre-course setup (Week 0, self-paced checklist): API-key setup for course-provided model access, environment verification, and a 30-minute “first agent loop” walkthrough — cleared before Week 1 studio.
| Course Sequence: ← BADM 558 — Big Data Infrastructures | Next: Quantum Computing for Better Business Decision Making → |