Mar phàirt de dh'obair mheasaidh leantainneach ÈIST, tha sinn air slat-tomhais a chruthachadh gus dearbhadh dè cho math 's a tha modailean cànain mòra (LLMs) — Chat GPT, Gemini agus Claude nam measg — a' tuigsinn ghnàthasan-cainnte agus gràmar na Gàidhlig. Cleachdaidh an deuchainn seo, ris an canar *GaelEval*, 120 ceist ioma-roghnach a chaidh a dhealbhadh le eòlaiche cànain Gàidhlig. Bidh gach ceist ag amas air feart gràmadaigeach sònraichte — bho thuiseal ainmearan is cruthan ghnìomhairean, gu gnàthsan-cainnte agus clàsan dàimheach — a' tarraing air eisimpleirean doirbh nach gabh eadar-theangachadh facal air an fhacal bhon Bheurla. Tha e cudromach nach robh sinn dìreach a' cur nam modailean fo dheuchainn. Dh'iarr sinn cuideachd air 30 fileantach Gàidhlig an aon deuchainn a dhèanamh. Tha seo a' toirt dhuinn inbhe-thomhais dhaonna airson coimeas a dhèanamh rithe. Tha na toraidhean annasach. Fhuair Gemini 3 Pro Preview aig Google 83.3%, a' dol thairis air cuibheas ('average') nam fileantach fhèin de 78.1%. Cha do rinn na modailean adhartach eile cho math idir, agus bha siostaman fosgailte-cuideam (an fheadhainn a tha rim faighinn an-asgaidh) fada air dheireadh air an fheadhainn fo sheilbh. San fharsaingeachd, bha na modailean na bu treasa ann an gràmar structarail ach na bu laige ann an gnàthasan-cainnte agus cainnt conaltraidh — raointean far a bheil làmh an uachdair aig mac-an-duine fhathast. Anns a' ghraf shuas, tha coimeas eadar mar a rinn na modailean anns gach roinn ghràmadaigeach, a' cur nam prìomh mhodailean an coimeas ris an inbhe-thomhais dhaonna. Chaidh an obair seo a dhèanamh le buill de dh'Ionad Eòlais [CLARIN](https://www.clarin.eu) airson Goireasan Digiteach do Chànanan ann an Èirinn is Breatainn (DR-LIB), le maoineachadh bho EPSRC, còmhla ri ÈIST, aig a bheil taic bho Riaghaltas na h-Alba agus Bòrd na Gàidhlig. *Leugh am pàipear slàn: [GaelEval: Benchmarking LLM Performance for Scottish Gaelic](https://arxiv.org/abs/2604.02135)*

As part of ÈIST's ongoing evaluation work, we've developed a benchmark to test how well large language models (LLMs) — including ChatGPT, Gemini and Claude — actually understand Scottish Gaelic idioms and grammar. The test, called GaelEval, uses 120 multiple-choice questions designed by a Gaelic language expert. Each question targets a specific grammatical feature — from noun case and verb forms, to idiomatic expressions and relative clauses — drawing on difficult examples that resist simple word-for-word translation from English. Crucially, we didn't just test the models. We also asked 30 fluent Gaelic speakers to sit the same test, giving us a human benchmark to compare against. The results are striking. Google's Gemini 3 Pro Preview scored 83.3%, actually *exceeding* the fluent-speaker average of 78.1%. Other leading models performed far less well, and open-weight (freely available) systems lagged well behind proprietary ones. Models were generally strongest on structural grammar but weaker on idiomatic and conversational usage — areas where fluent speakers still have the edge. The graphic above breaks down performance by grammatical category, comparing the top-performing models against our human baseline. This benchmarking work was carried out by members of the [CLARIN](https://www.clarin.eu) Knowledge Centre for Digital Resources for the Languages in Ireland and Britain (DR-LIB), funded by EPSRC, alongside ÈIST, which is supported by the Scottish Government and Bòrd na Gàidhlig. *Read the full paper: [GaelEval: Benchmarking LLM Performance for Scottish Gaelic](https://arxiv.org/abs/2604.02135)