A router picks a model for each input instead of using one model for everything. Evaligo trains it on a finished experiment, because an experiment already holds what a router needs: every input, every model’s output score, and every model’s measured cost. The trained router then runs as a model in experiments and in deployed flows.

What you need

• A completed experiment on one flow with at least two model variants.
• At least 8 inputs scored for every variant. The router can only learn from inputs that every candidate model handled.
• A score per output, from a judge, a judge panel, a reference answer or an in-flow evaluator, and the cost per output that Evaligo records for every run.

Prompt variants are not candidates. The router chooses between models on the same prompt, so if the experiment also compared prompts, only the variants that differ by model take part.

How training works

1. 1

   Each input is turned into a vector The text of each input is embedded with the all-MiniLM-L6-v2 sentence encoder. This runs on the Evaligo worker’s CPU.

2. 2

   Each model gets a utility per input Utility = judge score − cost weight × cost. Cost is scaled so that a cost weight of 1 means one full point of score is worth the price of the most expensive candidate. A cost weight of 0 routes on quality alone.

3. 3

   Nearest neighbours vote For a new input, the K most similar training inputs each vote for the model that had the highest utility on them. The model with the most votes wins. K is 1, 3, 5, 7 or 9.

4. 4

   Every setting is measured, not predicted For each cost weight and each K, every training input is routed using the other inputs as the reference set (leave-one-out) and credited with the score and cost that the chosen model actually got on it. Error bars are 95% bootstrap intervals from 1,000 resamples.

The method is the KNN router from LLMRouter, an open-source library from the University of Illinois (MIT licence), re-implemented inside Evaligo so that nothing leaves the platform. Training takes under a minute for experiments of a few hundred inputs.

Train one

1. 1

   Open the finished experiment On the Experiments page, open an experiment whose status is completed.

2. 2

   Choose Train router The button sits above the scoreboard. Evaligo checks the requirements above and reports what is missing if a check fails.

3. 3

   Wait for the chart The dialog shows progress while inputs are embedded and every setting is evaluated, then switches to the quality-versus-cost chart.

Read the chart and pick a working point

The chart plots judge score against cost per input. Each fixed model is one point. The router is a curve: one point per cost weight, from quality only on the left to cheapest on the right. A slider moves along that curve and a row of buttons switches K.

For the selected point the panel shows:

• Score with its 95% interval.
• Cost per input with its 95% interval.
• Difference from the best fixed model and from the cheapest fixed model, in score points.
• Where inputs go: the share of inputs sent to each model at this setting.
• Sample-size check: how many more inputs would bring the interval to ±3 score points, and whether the router’s interval is separated from the best fixed model’s interval.

The default working point is the K with the highest quality-only score, and on that curve the cheapest cost weight whose score stays within 0.01 of the best. You can move it and save a different one at any time; the saved point is what the router uses when it runs.

A measured example

Four models localized 63 product listings into seven languages, scored by a blind three-judge panel. A router was trained on those 63 listings and then run, together with two of the fixed models, on 28 listings none of them had seen.

The full run, including the 63-listing training scoreboard and the per-language results, is in the listing localization case study.

Run the router as a model

• In an experiment: add a variant and choose the router under Routers in the model list. It runs on the new inputs like any model. The scoreboard shows its score, cost and latency, plus the share of inputs each model received.
• In a flow: open a prompt node and choose the router under Routers in the model list. A deployed flow then routes every request. Each node run records which model handled it.

Routing adds about 150 milliseconds per input once the encoder is loaded. The first routed call after a worker starts takes about ten seconds while the encoder loads.

Limits

• A router knows only the inputs and models of the experiment it was trained on. Inputs unlike the training set are routed by their nearest neighbours regardless of how far away they are.
• It cannot route to a model that was not a variant in the experiment. To add a model, run a new experiment that includes it and train again.
• Scores come from the experiment’s judge. A router trained on one rubric optimises for that rubric.
• Costs are the per-output costs measured during the experiment at that time’s prices.