Using Grounded Theory for Agent Behavior Analysis at Scale

arXiv · Zhuoran Lu, Yangyang Yu, Zhuoyan Li, Yibo Meng, Nan Jiang, Chengxi Zang, Jie Gao, Ziang Xiao · 2026-09-05

This is no longer on the current shelf — shelves rotate as new material clears the bar. The analysis below is unchanged. See what is featured now.

A scalable grounded‑theory pipeline that turns massive logs of LLM‑agent actions into actionable behavioral taxonomies.

What it does

The paper adapts grounded theory—a qualitative research method—to automatically extract, code, and organize patterns from billions of agent‑action logs. It introduces three components: (1) a high‑throughput log parser that normalizes diverse action schemas, (2) a clustering‑plus‑human‑in‑the‑loop coder that iteratively refines emergent categories, and (3) a visualization dashboard that surfaces causal hypotheses. The authors validate the pipeline on OpenAI‑style chat agents and on a fleet of autonomous web‑browsing bots, uncovering 12 new failure modes such as “prompt drift” and “resource leakage.”

Why it matters

LLM agents are black boxes; without systematic behavior analysis, hidden bugs surface only in production outages. This framework gives engineers a repeatable, data‑driven way to surface, prioritize, and remediate failure modes before they hit users. It also creates a living taxonomy that can feed into prompt libraries, safety filters, and automated testing suites.

How it applies

Run the log parser on your telemetry pipeline to produce a normalized action stream. Use the provided clustering scripts (built on FAISS + HDBSCAN) to generate initial categories, then schedule short human coding sprints to refine them. The resulting taxonomy can be exported as JSON and plugged into your evaluation harness to automatically flag regressions in the identified failure modes.

The short version

You can’t fix what you can’t see—so make your agents’ blind spots visible.

My take

In my own work on agentic systems, I’ve spent countless hours chasing down “why did the bot keep opening the same tab?” problems that only surfaced after a rollout. The grounded‑theory pipeline turns those late‑night debugging sessions into a repeatable process—extract, cluster, codify, and then bake the findings back into the system. It’s the kind of meta‑engineering that lets you scale agents without scaling the firefighting crew.

How it connects

• Echoes the industry move toward observability‑first AI (telemetry + automated analysis).
• Provides a bridge between qualitative research methods and quantitative ML pipelines.
• Sets the stage for automated “behavioral regression” testing, a missing piece in current LLM CI/CD.

Bottom line

Integrate the grounded‑theory pipeline into your agent telemetry stack now to catch systemic bugs before they reach users.

Takeaways

• Grounded theory can be automated at web‑scale, turning raw agent logs into meaningful behavioral categories.
• The three‑stage pipeline (parse → cluster → human‑in‑the‑loop) balances scalability with interpretability.
• Discovered 12 novel failure modes across two very different agent fleets, proving cross‑domain relevance.
• Taxonomies generated can be fed directly into safety filters, prompt libraries, and regression tests.
• Open‑source tooling (FAISS/HDBSCAN integration, dashboard) lowers the barrier to adoption.

Topics: agent-behavior, analysis, grounded-theory, LLM-evaluation