Google launches MKA benchmark for multimodal AI agents
Google and Artificial Analysis have launched the MKA benchmark for evaluating multimodal AI agents on 200 real-world tasks across 83 professions. The Gemini 4 Carbon model achieved 2nd place in quality at 6x lower cost than the best model.
Google, in cooperation with Artificial Analysis, has launched the MKA (Multimodal Knowledge Agent) benchmark, which evaluates how well AI agents handle work tasks based on visually complex materials — drawings, diagrams, tables, or presentations. The set contains 200 tasks divided into 10 industry verticals and 83 professions, created in cooperation with practitioners (architects, biologists, insurance underwriters, data scientists, and others). According to Google, each task corresponds to the scope of work that would take an experienced professional three to eight hours.
Evaluation is carried out according to an expert rubric with approximately 70 checkpoints, of which 75% of the weight is factual accuracy and 25% is the visual design of the output. The results are evaluated by an automated "autorater" built on a model from a company other than Google, which, according to Google, agrees with human expert assessments in 96% of cases. All models run in the same open-source Stirrup harness with identical tools, sandbox, and turn budget, so that the comparison is consistent. The 200 tasks themselves remain non-public at Artificial Analysis, so that they cannot leak into training data; separately, a set of 10 public sample tasks was published on Hugging Face.
According to Google, the technical report on the benchmark shows large differences in efficiency between models — the tested models used on average 30 to 130 turns to complete a single task. The Gemini 4 Carbon model achieved, according to the results, 2nd place in quality among the tested models and completed tasks at 6x lower average cost compared to the top-rated model, which, according to Google, placed it on the efficiency frontier between quality and price.
Why it matters
The benchmark provides a standardized way to measure how AI agents handle visually demanding office work (drawings, tables, reports) instead of purely text-based tasks that do not correspond to real-world practice. For developers and companies deploying AI agents, this offers a tool for comparing model quality and cost efficiency before selecting one for production.
Relevant practical impact
What this means
For a business
Companies developing or deploying multimodal AI agents gain an independent benchmark for comparing model quality and cost on real work tasks, which influences model selection for production deployment.
DevelopmentCheck the original
Event sources
only one source so far · 1 publisher, 0 independent. We count feeds from the same owner only once.