ru-prop-management-eval
GitHub

How do modern LLMs handle real questions from Russian property managers?

loading…
Как это проверяется
A question from the dataset: «Who moderates the official building chats, and what if someone posts banned content there?»
Closed book
The model gets only the question: no laws, no hints, no reference answer
↓
It answers from memory, like in a normal chat
↓
Judge jev checks the answer against the expert reference and scores it: solved, legal, facts right
🚧 Oracle — coming soon
Relevant legal texts and similar reference answers are pulled in with the question
↓
The model answers grounded in the documents, not memory
↓
The same judge scores it, showing the ceiling a retrieval harness can reach
123 questions, one run per model, one judge. Numbers are preliminary: directional, not final.
Main findings
  • Closed-book leader: gpt-5.6-sol·nr at 52.7%. Best Russian model: GigaChat-3-Ultra at 44.9%, nearly free on its freemium quota.
  • No model is production-ready without context: the best mean score across all questions is 52.7%. Real use needs retrieval over actual legal documents.
  • Judges flag answers as illegal far less often than as factually wrong. The real risk is confident advice citing norms that do not exist.
  • Reasoning adds ~2–3 points for 2–4× the tokens. The noreason variants are the cost-efficient pick.
–
Models
–
Questions
–
Mean score
–
Est. total cost
Mean score by model
Cost vs quality (avg per question)
Answers judged illegal
Factually incorrect answers
Hardest questions
Easiest questions
Suspicious answers
answers load when you pick a model