Skip to content
Marko Ivanovski

Prompting techniques compared across language models

Compares four prompting techniques across three language models on a single dataset, measured by performance.

category
academic
year
2026
status
completed
role
One of three on the project
  • Hugging Face Transformers
  • Python

What it is

A research project for the Natural Language Processing course, with two colleagues. Three open models, Llama 3.1 8B, Mistral 7B v0.3 and Gemma 4 E4B, answered the same 100 graduate-level reasoning questions from facebook/natural_reasoning under four prompting styles: zero-shot, few-shot, and both again with chain-of-thought. Every model ran 4-bit quantised on a free Colab T4.

Answers were scored seven ways, from exact match to BERTScore. Exact match stayed under 1% and contains match under 9% for every model, while BERTScore sat between 0.80 and 0.82. The answers sounded right far more often than they were right.

figure 1 · ROUGE-L overlap with Llama 3.3 70B's answer, mean over 82 questionsThe prompting technique moved the score by at most 0.057. Zero-shot scored best, or within 0.002 of best, for all three models, so the examples and reasoning steps bought nothing measurable here.
The numbers
GroupMethodROUGE-L
Llama 3.1 8BZero-shot0.286
Llama 3.1 8BFew-shot0.265
Llama 3.1 8BFew-shot CoT0.246
Llama 3.1 8BZero-shot CoT0.287
Mistral 7B v0.3Zero-shot0.279
Mistral 7B v0.3Few-shot0.222
Mistral 7B v0.3Few-shot CoT0.249
Mistral 7B v0.3Zero-shot CoT0.266
Gemma 4 E4BZero-shot0.256
Gemma 4 E4BFew-shot0.251
Gemma 4 E4BFew-shot CoT0.255
Gemma 4 E4BZero-shot CoT0.245

Cross-model comparison notebook, facebook/natural_reasoning sample, seed 42.

  • Four techniques: zero-shot, few-shot, and each of those again with chain-of-thought.
  • Llama 3.1 8B, Mistral 7B v0.3 and Gemma 4 E4B over the same dataset, with DeepSeek 7B added afterwards.

What I learnt

  1. At 7 to 8 billion parameters the prompting technique barely mattered. Contains match moved only between 0.065 and 0.073 across all four, which is within noise.
  2. Chain-of-thought did not improve correctness at this size, which matches the papers that find its gains in far larger models.
  3. The real differences were operational. Mistral followed the answer format most reliably, and Gemma was fastest but mostly ignored it. For anything that parses the output, that matters more than accuracy.
  4. Exact match is close to useless for open-ended answers. BERTScore was the most informative of the seven metrics.

What I would do next

Test models above roughly 100 billion parameters for a direct comparison with the papers, which a free Colab cannot run.

Implement Logical Thoughts, a verify-and-revise loop over each reasoning step.

Use more than 100 questions, and add an LLM judge as a metric.