Large Language Models and Gaelic
LLM Scoreboard
| 1 | Gemini 3 Pro Prev | 83.3% |
| 2 | Gemini 3 Flash Prev | 79.2% |
| Average fluent Gaelic speaker | ||
| 3 | GPT-5 | 69.2% |
| 4 | Gemini 2.5 Flash | 61.7% |
| 5 | Claude Opus 4.6 | 59.2% |
| 6 | GPT-4o | 50.0% |
| 7 | Claude Haiku 4.5 | 47.5% |
| 7 | GPT-5 Mini | 47.5% |
| 8 | DeepSeek R1 | 45.0% |
| 8 | Llama 4 Maverick | 45.0% |
| 9 | GPT-5.2 | 43.3% |
| 10 | GPT-4.1 | 42.5% |
| 11 | GPT-5 Nano | 36.7% |
| 12 | GPT-4o Mini | 34.2% |
| 13 | GPT OSS 120B | 33.3% |
| 14 | GLM 4.7 | 32.5% |
| 15 | GPT-4.1 Nano | 30.0% |
| 16 | GPT OSS 20B | 29.2% |
| 17 | GPT-4.1 Mini | 24.2% |
As part of ÈIST's ongoing evaluation work, we've developed a benchmark to test how well large language models (LLMs) — including ChatGPT, Gemini and Claude — actually understand Scottish Gaelic idioms and grammar.
The test, called GaelEval, uses 120 multiple-choice questions designed by a Gaelic language expert. Each question targets a specific grammatical feature — from noun case and verb forms, to idiomatic expressions and relative clauses — drawing on difficult examples that resist simple word-for-word translation from English.
Crucially, we didn't just test the models. We also asked 30 fluent Gaelic speakers to sit the same test, giving us a human benchmark to compare against.
The results are striking. Google's Gemini 3 Pro Preview scored 83.3%, actually exceeding the fluent-speaker average of 78.1%. Other leading models performed far less well, and open-weight (freely available) systems lagged well behind proprietary ones. Models were generally strongest on structural grammar but weaker on idiomatic and conversational usage — areas where fluent speakers still have the edge.
The graphic above breaks down performance by grammatical category, comparing the top-performing models against our human baseline.
This benchmarking work was carried out by members of the CLARIN Knowledge Centre for Digital Resources for the Languages in Ireland and Britain (DR-LIB), funded by EPSRC, alongside ÈIST, which is supported by the Scottish Government and Bòrd na Gàidhlig.
*Read the full paper: GaelEval: Benchmarking LLM Performance for Scottish Gaelic