Abstract AI: A Proposed Routing Layer for Language Models

I have a business idea but not enough time to build it, so I'm sharing it in the hope someone else will. I call it Abstract AI: an abstraction layer above language models for companies building AI products and organizations deploying AI internally. The aim is to make it easier to choose and operate models efficiently.

My premise is that many developers send every request to one frontier model through a single provider, even when a smaller model might suffice. They may pay more than necessary and become exposed to changes in that provider's prices, policies or availability. I estimate that 90–95% of use cases do not require a frontier model, but that is my judgment, not a measured universal statistic.

The proposed product targets latency, cost and flexibility. In my view, developers care less about a model's name than whether its responses are consistently good enough, fast enough and affordable at scale. Even modest savings could matter when an organization processes very large numbers of tokens, though actual results would depend on workload and infrastructure.

Developers also have a growing menu of techniques, including model routing, chain-of-thought prompting and mixtures of agents. Implementing and evaluating them takes work, and the field changes quickly. I envision a service that maintains these choices so application teams can focus on their own products.

Abstract AI would be a drop-in API endpoint connected to proprietary, open-weight and local models. For each request, a router would choose a model and, where appropriate, a workflow according to the customer's priorities for quality, speed and cost. These goals can conflict; no routing layer can guarantee the cheapest, fastest and best answer simultaneously for every prompt.

I cite RouteLLM as evidence that routing can reduce cost while preserving much of a stronger model's quality. The [RouteLLM researchers](https://www.lmsys.org/blog/2024-07-01-routellm/) report over 85% cost reduction at 95% of GPT-4 performance on MT Bench in their tested setup, with smaller reductions on other benchmarks. Those results are benchmark- and model-pair-specific, not a promise of the same savings for every application.

A company might run a small model on employee devices and send harder requests to private or external cloud models. I speculate that most prompts could be handled locally, potentially reducing external API use and data transfers. The actual proportion, quality, hardware cost, latency and privacy depend on the workload and implementation; local inference is not literally cost-free.

I compare the idea with Apple's proposed combination of on-device processing, Private Cloud Compute and, with permission, external model assistance. The analogy illustrates a routing pattern, not proof that Abstract AI would provide Apple's privacy guarantees or use the same decision rules.

The routing component could be trained or calibrated for a company's particular tasks and available models. I also imagine combining it with agent frameworks, mixtures of agents or chain-of-thought techniques. These methods may help particular tasks but can add calls, cost and latency, and need evaluation rather than an assumption that they always improve quality.

A customer could tell the API whether speed or answer quality matters most, and the router could adjust accordingly. It would use a frontier model when the estimated benefit justifies it. The difficult engineering problem is to measure those tradeoffs and monitor them as models, prices and user needs change.

Built-in benchmarking would test both response quality and consistency against a company's own examples. Each candidate model or workflow could be evaluated before being added to routing. This makes the promised optimization measurable for a real use case instead of relying only on generic leaderboards.

I also propose caching repeated requests so a previous answer can be returned without another model call, improving speed and consistency and potentially lowering cost. Whether reuse is appropriate depends on context, freshness, permissions and the desired variability; a cached answer can be stale or wrong, so caching alone does not guarantee quality.

Because the service would sit in the path of many AI requests, it could later expand into prompt management: user and group permissions, company rules, prompt versions and benchmarking those versions. That would make the API layer a broader operational tool, although it would also need to handle sensitive data and access controls carefully.

This is a rough product idea rather than a finished system. I think there could be a valuable product or business here, and I would like to hear from anyone who decides to build it. The proposed savings and quality improvements need to be established with real workloads and careful measurement.