A business automation may need to interpret a document, find evidence, modify code, and review a decision. Using the same model with the same context for everything tends to hide important differences. Multi-model orchestration aims to assign each task to a tested configuration while keeping a common contract for its results.
Separate capability, model, and runtime
The process requests a capability: extract an invoice, review a change, or analyze a source. That capability combines instructions, tools, and a permissions policy. The model performs inference; the runtime organizes tool calls and execution. Changing providers should not silently alter an agent's authorization or the structure the next step expects.
Our systems already combine specialized agents and models configurable by module. A shared gateway architecture for projects adds job contracts, versions, and policies; some of those extensions need a pilot before becoming shared operations. Distinguishing what is implemented from what is designed avoids committing to availability based on a proposal.
Evaluate complete configurations
We compare model, prompt, tools, and runtime with real cases. We look at accuracy, schema compliance, use of evidence, latency, and total cost, including retries. A cheap model that forces the task to be redone can end up more expensive. A powerful model adds no value if the bottleneck is a poorly connected API.
Selection keeps versions and results so the comparison can be repeated. Fallbacks are enabled only when they meet the same data and quality restrictions. A provider outage does not grant permission to send confidential documents to any alternative.
Memory with boundaries
There is session context, project documentation, and persistent memories. Each needs rules for freshness, provenance, and access. Retrieval must filter permissions before adding information to the prompt. Correction and deletion are also needed: an incorrect memory repeated across many tasks multiplies the problem. In Jarvis we have applied explicit capture and retrieval; extending it to multiple identities requires verifying isolation.
Budget the complete job
We record consumption by step and run, time limits, concurrency, and failure signals. The budget covers searches, embeddings, inference, and external work. The organization can choose which tasks need human review and which configurations offer the best balance between quality and cost.
The result is an operational policy that can evolve. Models change; acceptance criteria, permissions, and evidence remain the responsibility of the system and its team.
