Vector Institute aims to clear up confusion about AI model performance

All 11 models also struggled with agentic benchmarks designed to assess real world problem-solving abilities around general knowledge, safety, and coding. Claude 3.5 Sonnet and o1 ranked the highest in this area, particularly when it came to more structured tasks with explicit objectives. Still, all models had a hard time with software engineering and other tasks requiring open-ended reasoning and planning.

Multimodality is becoming increasingly important for AI systems, as it allows models to process different inputs. To measure this, Vector developed the Multimodal Massive Multitask Understanding (MMMU) benchmark, which evaluates a model’s ability to reason about images and text across both multiple-choice and open-ended formats. Questions cover math, finance, music and history and are designated as “easy,” “medium,” and “hard.”

In its evaluation, Vector found that o1 exhibited “superior” multimodal understanding across different formats and difficulty levels. Claude 3.5 Sonnet also did well, but not at o1’s level. Again, here, researchers found that most models dropped in performance when given more challenging, open-ended tasks.