Alibaba's SkillWeaver Cuts AI Agent Token Use by 99% With Smart Tool Routing

AI agents are powerful tools for automating complex workflows, but as they scale to handle hundreds of tools and APIs, a familiar problem emerges: they choke. Stuff every tool description into the context window and you burn through tokens at an alarming rate -- sometimes hundreds of thousands per query. Alibaba researchers have a new answer, and it cuts token consumption by over 99%.

The Tool Routing Problem

Modern enterprise AI agents don't just answer questions. They orchestrate multi-step workflows: downloading datasets, transforming data, generating visual reports, calling APIs, and more. Each of these steps requires a specific tool or skill. But when an agent has access to thousands of tools -- as is increasingly common with ecosystems like the Model Context Protocol (MCP) -- figuring out which tool to use for each step becomes a serious bottleneck.

The naive approach is to dump every tool description into the LLM's prompt and let it pick. This works poorly. Context windows get bloated, costs skyrocket, and accuracy actually drops. In Alibaba's experiments, a massive model flooded with all tool options only picked the right tool category 21% of the time. The model gets overwhelmed by choice, not empowered by it.

Enter SkillWeaver

SkillWeaver is a new framework from Alibaba's research team that tackles what they call "compositional skill routing" -- the problem of breaking a complex request into sub-tasks, finding the right tool for each, and wiring them into an executable plan.

The framework operates in three stages:

• Decompose: An LLM breaks the user's complex request into a sequence of atomic sub-tasks, each requiring one skill.
• Retrieve: An embedding model compares each sub-task against a vectorized library of tools, pulling a shortlist of top candidates.
• Compose: A planner evaluates the candidates for inter-skill compatibility and builds a Directed Acyclic Graph (DAG) execution plan with dependencies mapped out.

The Secret Sauce: Skill-Aware Decomposition (SAD)

The key innovation is something called Iterative Skill-Aware Decomposition, or SAD. The problem it solves is subtle: when an LLM breaks down a task, it tends to use generic language that doesn't match the technical vocabulary of actual tools in the library. A model might say "process the data" when the available tool is called "csv-parser" or "etl-pipeline."

SAD fixes this with a feedback loop. The LLM drafts an initial plan, the system does a preliminary search for matching tools, and then feeds those tool names back to the LLM as hints. The LLM rewrites its decomposition using the actual vocabulary of the tools that exist. This alignment step turns out to be more impactful than simply using a bigger model.

The Results: Smaller Models, Bigger Accuracy

The numbers from Alibaba's experiments are striking. Using a lightweight 7-billion parameter model (Qwen2.5-7B-Instruct) with the SAD feedback loop, decomposition accuracy jumped from 51% to 67.7%. With a larger model, accuracy reached 92%. On hard tasks requiring four to five distinct skills, SAD improved accuracy by 50%.

Perhaps the most surprising finding: larger models can actually perform worse without guidance. A 14-billion parameter model tested without SAD saw its accuracy drop below the smaller 7B model because it tended to over-decompose tasks into unnecessary microscopic steps. Once SAD was introduced, the retrieved tool hints anchored the model back to reality.

On token consumption, the improvement is staggering. The brute-force approach of feeding all tools into a large model's prompt consumed an estimated 884,000 tokens per query. SkillWeaver's targeted approach reduced that to roughly 1,160 tokens -- a 99.9% reduction that translates directly to lower API costs and faster response times.

Why This Matters for Enterprise AI

For teams building AI agents in production, SkillWeaver highlights a shift in thinking. The instinct has been to throw bigger models at harder problems. But this research suggests that smarter routing architecture often beats raw model size. A 7B model with good tool alignment can outperform a massive model drowning in options.

The implications for cost are enormous. Enterprise agent deployments that currently burn through context windows with every tool description loaded will see dramatic savings. Faster response times, lower API bills, and better accuracy -- all from rethinking how agents discover and select tools.

Building Your Own Skill Router

While Alibaba hasn't released the SkillWeaver source code yet, the framework is built entirely on off-the-shelf components that can be reproduced. The SAD feedback loop is essentially clever prompt engineering combined with a retrieval step. The researchers used the open-source all-MiniLM-L6-v2 embedding model for semantic search and a FAISS index for retrieval -- both freely available.

For teams using orchestration libraries like LangChain or LlamaIndex, implementing a similar decompose-retrieve-compose pipeline is straightforward. The key steps:

• Vectorize your tool library and build a FAISS index (the researchers embedded 2,209 skills in 15 seconds).
• Use a lightweight LLM for task decomposition with a feedback loop that retrieves tool candidates and feeds them back as hints.
• Add a composition stage that checks for inter-skill compatibility before building the execution plan.
• Consider a secondary cross-encoder or LLM-based reranker to improve top-1 accuracy from roughly 37% to something more reliable.

The Missing Piece: Error Recovery

One limitation the researchers acknowledge: SkillWeaver handles routing and planning, but not execution failures. If an API call fails in step two of a five-step chain, the whole pipeline breaks. For production deployments, teams will need to build their own error recovery, retry logic, and fallback mechanisms on top of the compose stage.

This is a common gap in AI agent frameworks right now. Planning is the glamorous part; error handling is the unglamorous part that determines whether something actually works in production. SkillWeaver solves the first problem elegantly -- but the second remains open territory.

The Bigger Picture

SkillWeaver is part of a broader trend in AI agent architecture: moving away from monolithic, context-stuffed prompts toward structured, modular pipelines. As tool ecosystems continue to grow -- MCP servers, API marketplaces, internal tool libraries -- the ability to efficiently route tasks to the right tools will only become more critical.

The lesson for practitioners is clear: don't just scale up your model. Scale up your architecture. A well-designed routing layer with proper task decomposition and tool alignment can deliver better results than a frontier model with a bloated context window. And it'll cost you 99% less.