Rethinking LLM Ensembling from the Perspective of Mixture Models

1. Introduction

Model ensembling in LLMs combines multiple models’ next-token distributions to improve quality, robustness, and reliability across tasks. The downside is cost: conventional ensembling runs every model at every decoding step, which makes latency scale linearly with the number of models. Ensembling heterogeneous models also requires vocabulary alignment and careful weighting.

We address this with a mixture-model interpretation of ensembling and a Mixture-model-like Ensemble (ME) algorithm. ME samples from the same ensemble distribution while invoking only one model per step, keeping the output distribution identical to conventional ensembling but reducing compute. Experiments show that ME matches CE performance and delivers 1.78x-2.68x speedups.

2. Mixture-model-like Ensemble

2.1 Conventional LLM Ensemble

Unlike standard decoding, which samples the next token from a single model, conventional LLM ensembling combines the next-token distributions predicted by multiple models. Let the input prefix be \(x_{\le t}\) and let \(\mathcal{M}_i\) denote the \(i\)-th model with weight \(\lambda_i\), where \(\lambda_i \ge 0\) and \(\sum_i \lambda_i = 1\). At every decoding step, each model performs a forward pass and produces its own distribution over the vocabulary. These distributions are then aggregated into an ensemble distribution:

The next token is sampled from this weighted distribution, appended to the prefix, and the same procedure repeats until generation ends. This direct implementation is simple and often improves quality over a single model, but it requires \(n\) model evaluations for every generated token. Even if the models are evaluated in parallel, the method still needs to load and execute every model at each step, so inference cost grows with the number of ensemble members.

2.2 Mixture-model-like Ensemble

ME treats the ensemble distribution as a mixture model and moves the randomness one step earlier. Instead of explicitly computing every model’s distribution and averaging them, ME samples in two stages: (1) draw a model index \(m \sim \mathrm{Cat}(\lambda)\) according to the ensemble weights; (2) run only \(\mathcal{M}_m\) and sample the next token from \(\mathcal{M}_m(y \mid x_{\le t})\).

This produces the same marginal next-token distribution as conventional ensembling:

Thus, ME preserves the ensemble sampling distribution while evaluating only one model per token. The practical implication is that the generation quality should match conventional ensembling under the same weights, while decoding becomes much closer to single-model inference.

2.3 Lazy Key-Value Cache Synchronization

Sampling only one model per step introduces a KV-cache issue. In autoregressive decoding, each model normally caches the keys and values of previous tokens. If ME selects model \(\mathcal{M}_i\) at one step and later switches to \(\mathcal{M}_j\), then \(\mathcal{M}_j\) may not have cached the tokens generated while it was inactive. Naively updating every model’s cache at every step would recover correctness, but it would also bring back the overhead that ME is designed to avoid.

ME handles this with lazy synchronization. Each model maintains its own cache asynchronously and is synchronized only when it is selected again. If \(\mathcal{M}_i\) was last used at step \(t-k\) and is selected at step \(t\), the missing tokens \(x_{t-k+1:t}\) are first processed in one forward-extend, or prefill, operation to update its cache; then \(\mathcal{M}_i\) generates \(x_{t+1}\). Because LLM decoding is often memory-bandwidth bound, processing a short accumulated span in one prefill pass has latency comparable to a decoding step, making the synchronization overhead small in practice.

2.4 Ensembling with Heterogeneous Vocabularies

When models use different vocabularies, their probability distributions cannot be averaged directly. Let the models have vocabularies \(V_1,\dots,V_n\) and next-token distributions \(P_1,\dots,P_n\). A vocabulary alignment function \(\mathcal{F}_i\) maps each \(P_i\) from its native vocabulary \(V_i\) to a distribution \(\tilde{P}_i\) over a unified vocabulary \(U\). The ensemble distribution can then be written as:

ME integrates this naturally: after sampling model \(\mathcal{M}_i\), it applies \(\mathcal{F}_i\) to the selected model’s distribution and samples the next token from the aligned distribution. In our experiments we use a top-\(k\) alignment strategy inspired by UniTe, but the framework is compatible with other alignment methods.

2.5 Connection to Token-level Routing

ME also gives a simple view of the relationship between LLM ensembling and token-level routing. A token-level router chooses one expert model at each generation step; ME can be interpreted as the simplest possible router, where the routing decision is random and follows the fixed ensemble weights. Conversely, conventional ensembling can be viewed as marginalizing over this random router. This connection highlights the tradeoff: ME is training-free and plug-and-play, while learned routers may achieve stronger routing decisions at the cost of additional training.

3. Experiments

3.1 Experimental Setup

We evaluate ME on GSM8K, MMLU, BBH, and ARC. Accuracy results are averaged over five runs, and speed is measured in tokens per second (H100 GPU unless stated). We test similar-model ensembles (Qwen-3B + Qwen-Math-1.5B, and Openchat/Nous-Hermes/OpenHermes) and a heterogeneous ensemble (Openchat + Deepseek-7B + Mistral-7B). We compare ME against conventional ensembling (CE) in both sequential and parallel settings, using the same weights and top-\(k\) vocabulary alignment.

3.2 Main Results

ME consistently matches CE accuracy across both similar and heterogeneous ensembles, confirming the distributional equivalence of the two approaches. The tables below report performance; values in parentheses denote the gain or drop relative to the best single model.

Performance comparison on ensembling similar models.

Performance comparison on ensembling heterogeneous models.