Chain-of-thought prompting is emerging as a powerful and effective design pattern for LLM-based apps and agents. The basic idea of chain-of-thought prompting is to let a model generate a step-by-step solution (“reasoning trace”) before answering a question or taking a decision. With the Open CoT Leaderboard we’re tracking LLMs’ ability to generate effective chain-of-thought traces for challenging reasoning tasks.
Unlike most performance based leaderboards, we’re not scoring the absolute accuracy a model achieves on a given task, but the difference between the accuracy with and without chain-of-thought prompting:
accuracy gain Δ = accuracy with CoT – accuracy w/o CoT.
This allows us to truly inspect the impact that chain-of-thought has on model accuracy.
Note: without CoT prompting, we use the loglikelihood accuracy to score the model on multiple choice evaluation.
Chain-of-thought prompting is a universally applicable prompting strategy that may improve explainability and accuracy of LLM-based apps and agents (see, e.g., this collection for recent research and implementations)). With frameworks like Langchain or LMQL, it’s straightforward to insert sophisticated reasoning chains in your apps. But even if you’ve never heard about chain-of-thought before, you may have noticed, while using a ChatBot, that it tends to proceed step by step before answering your query. So, a systematic, up-to-date comparison of LLMs’ ability to generate effective chain-of-thought traces may inform the decisions of builders and users when choosing a model.
Over time, static "accuracy-based" benchmarks risk becoming less informative: does a model score well because of its superior skill, because it has seen the correct answers during training, or because it has been developed in a competitive context that is governed by this very benchmark? These widely acknowledged issues are addressed by recent eval approaches such as ChatBot arenas, the use of LLMs as judges, or dynamic benchmarks with programmatically generated tasks. We hope the Open CoT Leaderboard contributes to these efforts, notably by being more robust to training data contamination: knowing the answer to a question doesn’t ensure that one can reason effectively about it.
The Open CoT Leaderboard evaluates LLMs’ ability to generate effective chain-of-thought reasoning traces for the following tasks:
Except for the original version of LogiQA, all these tasks are part of the AGIEval benchmark, and have been re-published as logikon-bench.
We’ve chosen these tasks because they
All tasks are rendered as multiple-choice problems, with the answer options being enumerated in the prompt.
We use the following prompt template for assessing baseline and CoT accuracies – the reasoning traces (starting with Reasoning) are only added in the case “with CoT”:
Answer the following question about the given passage. Base your answer on the reasoning below.
Passage: <passage>
Question: <question>
A. <choice1>
B. <choice2>
…
Reasoning: <reasoning>
Answer:
[#cot-generation]
The literature on chain-of-thought prompting has exploded in the last year, and the wealth of prompting strategies for CoT (involving, e.g., decoding, prompt wording, prompt chains, decomposition, aggregation and revision of reasoning traces) has become sheerly mindblowing.
To cope with this variety, the evaluation pipeline behind the Open CoT Leaderboard implements CoT generation through modular and easily extensible prompt chains. So far, we have implemented two fairly simple prompting strategies:
Let’s have a look at an example. The following problem is from the LSAT logical reasoning benchmark.
Editorial: The structure of the present school calendar was established to satisfy the requirements of early-twentieth-century agricultural life. In those days, farmers needed their children to have long breaks during which they could remain at home and help with the harvest. The contemporary school year is thus made up of periods of study interspersed with long breaks. But agricultural life no longer occupies most of our citizens, so we can now make changes that serve the interests of children. Therefore, long breaks should be removed from the school calendar.
Which one of the following is an assumption on which the editorial's argument depends?
A) A change in the present school calendar that shortened breaks would serve the interests of agricultural life.
B) Children of farmers need to continue observing a school calendar made up of periods of study interspersed with long breaks.
C) Long breaks in the school calendar should be replaced with breaks that are no longer than workers' average vacations.
D) A school calendar made up of periods of study without long breaks would serve the interests of children more than a school calendar with long breaks.
E) During long breaks children have a tendency to forget what they have learned.
Mixtral-8x7B-Instruct-v0.1 is doing fairly well in generating the following chain-of-thought traces:
| With prompting strategy Classic | With prompting strategy Reflect |
|---|---|
| The editorial argues that the current school calendar, with long breaks, was designed to accommodate agricultural life but should now be changed because agricultural life no longer occupies most of our citizens. It concludes that long breaks should be removed from the school calendar.
The assumption that the argument depends on is the belief that removing long breaks from the school calendar would serve the interests of children more than the current calendar. This is not explicitly stated in the editorial, but it is the underlying assumption that justifies the proposed change. Therefore, the correct answer is: D) A school calendar made up of periods of study without long breaks would serve the interests of children more than a school calendar with long breaks. |
The decision problem in abstract terms is to identify the assumption on which the editorial's argument depends.
Common mistakes for this kind of problem include:
A plan for how to solve this problem could be:
Solving the problem step by step, following the plan:
Therefore, the assumption on which the editorial's argument depends is D) A school calendar made up of periods of study without long breaks would serve the interests of children more than a school calendar with long breaks. |
Each of our two prompting strategies—Classic and Reflect—is combined and run with multiple decoding parameters:
This gives us, in combination, six “CoT generation regimes.” When we evaluate a model, we let it generate, for every example in the test datasets, one chain-of-thought trace for each regime. The generated traces are then plugged into the prompt template shown above, which we use to score the models. We obtain, accordingly, six different numerical scores for accuracy with chain-of-thought (namely one for each regime). In the Open CoT Leaderboard, we report (for every model/task) the best marginal accuracy gain achieved under any regime.
We’re gradually extending the Open CoT Leaderboard by evaluating more and more models, but current results (model count=30) already suggest some interesting insights.
We’re planning to move ahead in different directions. And contributions to all these efforts are more than welcome.
First, we’d love to evaluate your models! You can 📬 submit any open LLMs for evaluation on the Open CoT Leaderboard space, using the Submission tab!
Then, we’d love some help on the following coding and data analysis tasks.
Here’s where we can exchange our ideas and collaborate:
Looking forward to hearing from you!
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。