Back to Blog

The Composable AI Stack: Why the Real AI Revolution Is in the Glue

9 min read

Every week brings another announcement of a model that beats another benchmark. GPT-5.5 tops the leaderboard. Claude Opus 4.7 gets a new capability. Qwen 3.6 drops at 27 billion parameters and somehow outperforms models ten times its size. But if you spend your days actually using AI — not benchmarking it, not writing about it, just trying to get work done with it — a different picture emerges. The models are astonishing. The systems are frustrating.

That gap — between capability and reliability — is where the real action is. And it's not happening where most people are looking.

The Composable Turn

Three weeks ago, a developer posted to LocalLLaMA about giving local models a fair shot for coding work. They used Qwen 27B and Gemma 4 31B — two of the best open-weight models available. They forced themselves to use nothing else for non-work tasks. Their verdict after weeks of testing: the productivity loss wasn't worth it. The specific culprit? Tool calls and decision-making quality. The models could reason beautifully. They couldn't execute reliably.

That same week, a Cornell research team published a sweeping survey on agentic AI in finance and reached a stark conclusion: agentic AI is fundamentally different from generative AI in three ways — goal-oriented autonomy, contextual reasoning through memory systems, and multi-agent coordination. These aren't incremental improvements. They're a different kind of system. And the infrastructure to support them is almost entirely missing.

Meanwhile, Kimi K2.6 launched with 300 parallel agents and open-source access. In internal benchmarks, it matched or beat GPT-5.4 and Claude Opus 4.6 on coding tasks. The model-level race continues unabated.

But here's what got less attention: Kimi K2.6's headline feature wasn't a new capability. It was that it could run 300 agents in parallel, maintain 4,000 connected task steps, and keep going for up to 5 days without human intervention. The breakthrough isn't the model. It's the orchestration layer.

Where the Field Is Actually Splitting

Two fundamentally different visions for how to fix unreliable AI are now colliding in the research literature — and they're pulling the field in opposite directions.

The first vision doubles down on the LLM-as-orchestrator approach. This is what most agent frameworks do: the model receives a task, reasons about it, calls tools, gets feedback, and loops. It's flexible, general-purpose, and easy to prototype. GitHub trending this week is littered with repos building exactly this: free-claude-code clones, computer-use agent frameworks, context-mode systems that sandboxes tool output with 98% reduction. The ecosystem is exploding because this architecture is easy to build. You don't need to understand the data. You just need a good model and a loop.

The second vision looks at that same landscape and says: this is all wrong. A paper from MIT, TUM, and AWS published this week made this argument explicitly. Their system, RUBICON, treats enterprise agentic AI as fundamentally a data integration problem, not a reasoning problem. They observe that when you deploy LLM-centric agents against real enterprise systems — not toy benchmarks — text-to-SQL accuracy drops by more than 50%. The benchmarks that show 85%+ accuracy (Spider, Bird) use clean schemas, simple queries, and data that's already in "the pile." Real enterprise data warehouses have materialized views, site-specific jargon, role-based access controls, and institutional knowledge that LLMs have never seen. The result: a technology that works beautifully in demos fails spectacularly in production.

Their alternative: explicit query languages (AQL — Agentic Query Language), wrapper-based connectors that present relational views of arbitrary data sources, and a query optimizer that treats LLM calls as a last resort rather than a first instinct. Every intermediate result is visible. Every step is auditable. The system is slower to set up but dramatically more reliable once deployed.

This is the same split you see in the practitioner world. The developer building "NeuralSwitch" — a dynamic LLM routing gateway — described the architecture that actually works in production: a fast regex-based classifier intercepts every prompt, estimates complexity, routes to the cheapest model that can handle it, and semantically caches results to eliminate redundant calls entirely. Their stack: FastAPI, Redis for caching, LiteLLM for orchestration, Neon Postgres for persistence. None of this is glamorous. All of it is production-hardened.

The Infrastructure That's Actually Winning

Look at what's trending on GitHub this week and you see the pattern clearly. The biggest repos aren't new models — they're composition layers:

context-mode (11K stars this week) sandboxes tool output and achieves 98% context reduction for coding agents. Instead of feeding every tool result into the context window, it selectively preserves only what matters. The insight: agents don't need all their tool output. They need the relevant parts.

cua from trycua (15K stars, 1.3K this week) is open-source infrastructure for computer-use agents — sandboxes, SDKs, and benchmarks for evaluating agents that control full desktops across macOS, Linux, and Windows. This is what "native computer use" looks like when you build it as infrastructure rather than a model feature.

GenericAgent (8K stars) grows a skill tree from a 3.3K-line seed file and achieves "full system control" with 6x less token consumption than comparable approaches. The agent starts minimal and becomes capable through structured self-improvement, not architectural complexity.

ml-intern from HuggingFace (7K stars, 6.4K this week) is literally an open-source ML engineer that reads papers, trains models, and ships ML models. It's not a toy demo — it's a system designed to do real ML work autonomously.

What these share: they're all about composing AI capabilities, not building bigger models. They're glue, scaffolding, routing logic, and memory systems. They're the unsexy infrastructure that makes the sexy demos actually work.

The Practical Stack That's Emerging

Across the conversations in LocalLLaMA and the practitioner discussions on X, a coherent picture of how reliable AI stacks actually look is emerging.

The winning pattern is layered intelligence. Simple tasks — summaries, formatting, basic extraction — get handled by cheap local models running on Apple Silicon or a $600 Mac Mini. Zero API costs, zero latency, zero data leakage. Only genuinely hard reasoning tasks route to Opus 4.7-class models in the cloud. One developer calculated the savings: $206/month for all-Opus vs. $35/month for a hybrid stack with local Ollama. That's 74-83% cost reduction without sacrificing capability for the tasks that matter.

The architecture that makes this work is what one practitioner called "dynamic model routing" — a fast classifier that inspects each prompt, estimates complexity, and sends it to the cheapest capable model. The key insight is that most prompts don't need a frontier model. They need something fast and good enough. The routing layer figures out which is which.

This is fundamentally different from "use the best model for everything." It's about building systems where the model is one component among many — not the entire value proposition.

Why This Matters More Than the Next Model Drop

Here's the thing about the composable turn: it's not a temporary workaround. It's a different architecture for AI deployment, and it has compounding advantages that pure model scaling doesn't.

When you build reliability into the infrastructure layer — through explicit state machines, auditable query plans, semantic caching, and multi-model routing — you get systems that are debuggable, predictable, and incrementally improvable. You can inspect why a routing decision was made. You can cache a result and verify it's still correct. You can swap one model for another without rebuilding the entire system.

When you rely entirely on model capability, you're betting that the next generation will be more reliable than the last. That bet has historically paid off — but the rate of improvement at the frontier is slowing, and the specific failure modes that matter for production (tool calling reliability, consistent decision-making, enterprise data integration) don't improve automatically just because benchmarks go up.

The researchers building RUBICON understood something important: enterprise AI isn't a reasoning problem. It's a data plumbing problem. And plumbing doesn't get solved by better reasoning models. It gets solved by better pipes.

The same is increasingly true for the broader AI stack. The models are extraordinary. They're also not enough. The real engineering frontier — the place where the next 10x improvement in usable AI is coming from — is in composition, routing, caching, orchestration, and explicit structure around the probabilistic core.

The model drops will keep coming. The benchmark wars will keep raging. But if you want to see where AI is actually going, look at what's trending in GitHub repos you've never heard of, in the routing logic developers are building to make models production-ready, in the wrapper layers that present structured data to probabilistic systems.

The real AI revolution, increasingly, is in the glue.

Sources

Academic Papers

  • An Alternate Agentic AI Architecture (It's About the Data) — arXiv, Apr 26, 2026 — RUBICON from MIT/TUM/AWS argues LLM-centric agentic architectures fail in enterprise settings; text-to-SQL accuracy drops 50%+ on real data warehouses vs. benchmarks; proposes explicit AQL query language and wrapper-based integration as the correct architecture
  • Agentic AI in Finance: A Comprehensive Survey — arXiv, Apr 27, 2026 — Cornell comprehensive survey showing agentic AI differs fundamentally from generative AI through goal-oriented autonomy, contextual reasoning with memory systems, and multi-agent coordination; analyzes deployment challenges in financial markets
  • Agentic AI: Architectures, Taxonomies, and Evaluation of LLM Agents — arXiv, Jan 18, 2026 — Melbourne/Anna University taxonomy decomposing agents into Perception, Brain, Planning, Action, Tool Use, and Collaboration; documents shift from linear reasoning to hierarchical multi-agent systems and from fixed API calls to open standards like MCP

Hacker News Discussions

Reddit Communities

X/Twitter

  • @CryptoKong59483 — @CryptoKong59483, Apr 28, 2026 — Multi-model architecture commentary: local models for exploration, expensive models only for final synthesis
  • @Charles_Haworth — @Charles_Haworth, Apr 27, 2026 — Hybrid local/cloud stack saves 74-83% vs. all-Opus; documents real production routing architecture
  • @sathishkumaratr — @sathishkumaratr, Apr 26, 2026 — Office Mac Mini serving whole team locally with Qwen 3.5 and Gemma 4; zero API costs, zero data leaving building
  • @assist1124 — @assist1124, Apr 21, 2026 — Kimi K2.6 analysis: 300 parallel agents, open-source release targeting Cursor's $50B valuation moat

GitHub Projects

  • context-mode — GitHub, Apr 2026 — Context window optimization for AI coding agents; 98% tool output reduction; 11K stars
  • cua — GitHub, Apr 2026 — Open-source computer-use agent infrastructure; sandboxes, SDKs, benchmarks for desktop control across macOS/Linux/Windows; 15K stars
  • GenericAgent — GitHub, Apr 2026 — Self-evolving agent with 6x token reduction; skill tree growth from seed file to full system control; 8K stars
  • ml-intern — HuggingFace, Apr 2026 — Open-source ML engineer that reads papers, trains models, and ships; 7K stars this week
  • free-claude-code — GitHub, Apr 2026 — 18K stars; open-source Claude Code alternative