Beyond the Model: A Systems Theory of Modern AI Quality
Moon, Dongsun
Synopsis "Beyond the Model: A Systems Theory of Modern AI Quality"
Your model got better. Your product did not. This book is about why. This is a diagnostic method for teams who are already shipping AI and already stuck. It is deliberately framework-agnostic, because its subject is where to intervene, not how to implement the fix. When an AI feature degrades, the reflex is to reach for a different model. Most of the time the model was not the problem, and the swap changes everything except the cause. Beyond the Model replaces that reflex with an order of operations: before you ask which model, ask which layer. And when the model really is the limit, the same method tells you so. Tools and models turn over fast. The places where production quality is won or lost change far more slowly, and those are what this book maps. The argument rests on 109 cited sources, from classical learning theory to 2026 papers at ICLR and ACL. You will learn to Find the binding layer. Use the Quality Locus, six surfaces around the model, to see which one is holding quality below the contract right now. Measure the context you actually get. Separate the window on the model card from the context your system can use, and measure the gap. Design memory, not just retrieval. Decide freshness, provenance, and the authority to forget per task, instead of arguing RAG versus long context. Run agents as closed loops. Read an agent as a feedback controller, and keep proposal quality and disposal quality apart. Trust your evaluations. Build golden sets, account for the four kinds of non-determinism and LLM-judge bias, and stop acting on gaps that sit inside their own variance band. Know when to stop answering. Use calibration and the reject option, and run the counterfactual test that shows when the model really is the bottleneck. A worked diagnosis takes two failing systems through the same seven steps to opposite conclusions, false starts left in. Who it is for. Anyone who has built and operated a production system and now answers for an AI feature running for real users: senior, staff, and principal engineers, architects and consultants, platform and SRE teams, and the engineering leaders who get paged when it degrades. It assumes production engineering experience, not a machine learning background. It is about diagnosis rather than implementation, so it does not teach a framework or walk through application code. 224 pages, 30 diagrams, a glossary, and a full index.
