Comparing configurations

Send one message to two configurations at once and read the replies side by side — then do the same for a whole set of prompts.

Two lanes

A compare turn runs on two lanes, A and B. Each lane can carry its own sampling overrides — temperature, top-p, top-k, and the rest — so you can ask, for instance, "does raising temperature actually change this answer" without re-typing the prompt twice. Your message is sent once; each lane continues its own conversation from there, so a later turn in lane A never sees lane B's reply as if it were history.

Sequential, not parallel

The two lanes run one after another, not concurrently. That keeps the readout — tokens per second, load time — meaningful for each lane on its own, rather than two requests contending for the same GPU and both numbers being wrong.

Why the seed is pinned

Left alone, sampling is randomized per request, so two runs of the exact same configuration can read differently for reasons that have nothing to do with the configuration. When neither lane sets its own seed, Remuda pins one seed and uses it for both — so a compare measures the configurations, not sampling noise. Setting a seed on one lane deliberately is treated as an intentional per-lane difference and unpins the pair.

Regenerate · same seed reruns both lanes on the seed that was pinned for them. It's disabled when no seed was ever pinned — Remuda won't invent one just to offer the button.

Benchmark: one prompt set, several lanes

A compare answers "which of these is better for this prompt". A benchmark asks it of a whole set of prompts at once. You keep the prompts, define a lane for each configuration you want to test — a model plus a Modelfile — and press Run all. The result is a table: one row per prompt, one column per lane.

Two lanes on two different models put those models against each other. Two lanes on the same model with different Modelfiles is just as normal, and it is how you read what a Modelfile actually changed across more than one example.

Build a benchmark from the BENCHMARKS group in the sidebar, or send a prompt straight to one with Add to benchmark in the menu under any chat message.

One model load per lane

A run is grouped by lane, not by prompt: Remuda loads lane 1's model, answers every prompt with it, and only then loads lane 2. Interleaving would mean a full model load per prompt, which on a 20 GB model is the difference between two loads and twenty. The run bar names the model it is loading and which lane it belongs to, because that wait is real and a progress bar that hides it just looks hung.

One seed for the entire run

The same pinned seed covers every lane and every prompt in a run, for the reason above: two lanes on two seeds would measure sampling noise instead of the configurations. The seed is shown beside the run so you can see which one produced the table.

Different is a difference, not a verdict

Expand a row and Remuda word-diffs each lane against the first, so you can see exactly where the answers parted company. That is as far as it goes. No lane is scored, no lane is called the winner, and the table is never sorted by anything but the order of your prompts — judging the answers is your job, and a tool that guessed at it would be one whose verdicts you had to check. A cell that failed stays in the table with its error, because that is a result too.