Evaluating large language models (LLMs) is challenging. Traditional ground-truth-based benchmarks fail to capture the comprehensiveness and nuance of real-world queries, while LLM-as-judge benchmarks suffer from grading biases and limited query quantity. Both of them may also become contaminated over time. User-facing evaluation, such as Chatbot Arena, provides reliable signals but is costly and slow. In this work, we propose MixEval, a new paradigm for establishing efficient, gold-standard LLM evaluation by strategically mixing off-the-shelf benchmarks. It bridges (1) comprehensive and well-distributed real-world user queries and (2) efficient and fairly-graded ground-truth-based benchmarks, by matching queries mined from the web with similar queries from existing benchmarks. Based on MixEval, we further build MixEval-Hard, which offers more room for model improvement. Our benchmarks' advantages lie in (1) a 0.96 model ranking correlation with Chatbot Arena arising from the highly impartial query distribution and grading mechanism, (2) fast, cheap, and reproducible execution (6% of the time and cost of MMLU), and (3) dynamic evaluation enabled by the rapid and stable data update pipeline. We provide extensive meta-evaluation and analysis for our and existing LLM benchmarks to deepen the community's understanding of LLM evaluation and guide future research directions.

TL;DR: We introduce MixEval, a ground-truth-based dynamic benchmark derived from off-the-shelf benchmark mixtures, which evaluates LLMs with a highly capable model ranking (i.e., 0.96 correlation with Chatbot Arena) while running locally and quickly (6% the time and cost of running MMLU), with its queries being stably and effortlessly updated every month to avoid contamination.

MixEval is an approach that bridges the gap between real-world user queries and ground-truth-based evaluation by leveraging user queries mined from the web and matching them with similar queries from existing benchmarks. MixEval is also the proposed benchmark built with this approach.

MixEval-Hard is the hard version of MixEval, designed to enhance the benchmark's ability to distinguish strong models. It is sampled from MixEval based on model evaluation results, with a higher probability of selecting harder queries. To address distribution deviation, we introduce a rejective sampling process to ensure that the distribution of MixEval-Hard aligns with that of wild queries.

Dynamic evaluation is introduced to mitigate the contamination issue. We periodically update the data points in MixEval and MixEval-Hard using our fast, stable pipeline, which performs benchmark mixture with a different batch of wild queries from the same distribution, showing low variance (0.36 Std. on a 0-100 scale) and significant version difference (85% unique query ratio).

The model scores of MixEval and MixEval-Hard scale linearly with Chatbot Arena Elo. By fitting the data points, the Arena Elo score can be roughly estimated given a model score on MixEval or MixEval-Hard. ρ and e denote the Spearman’s ranking correlation and the root mean square error of the linear fit respectively.

MixEval offers five significant advantages for practitioners: (1) accurate model ranking, demonstrated by a 0.96 correlation with Chatbot Arena1 , (2) fast, cheap and reproducible execution, requiring only 6% the time and cost of MMLU and with no dependence on human input, (3) dynamic benchmarking enabled by low-effort and stable updating mechanism, (4) a comprehensive and less biased query distribution, as it bases queries on a large-scale web corpus, and (5) a fair grading process without preference bias, ensured by its ground-truth-based nature.

Our approach improves the correlation with Arena Elo and Arena Elo (En) for all the main splits of MixEval and outperforms benchmark-level and uniform mixture.

MixEval is effective as (1) MixEval and MixEval-Hard achieve the highest correlation with Arena Elo and Arena Elo (En) among all benchmarks. (2) MixEval improves the correlation with Arena Elo and Arena Elo (En) across all its main benchmark splits. (3) MixEval outperforms both benchmark-level and uniform mixtures. (4) MixEval effectively maps real-world user queries to ground-truth-based benchmarks.

Query topic summarization for the query distribution map. The plot aggregates all queries and divides them into 16 regions. From each region, 100 queries are uniformly sampled and analyzed by GPT-4 for topic summarization. A clear trend is observed, with topics transitioning from non-technical at the bottom to technical at the top.