Evolving how LLMs are measured for Android: the next era of Android Bench

Posted by Zoe Lopez-Latorre, Senior Developer Relations Engineer, Android Back in March, we introduced Android Bench —our LLM leaderboard for real-world Android development tasks. Our goal was to provide transparency around model capabilities in Android development and to encourage model improvements, to give you more helpful AI options for your everyday workflow. Since then, we have enhanced the benchmark based on your feedback, including evaluating open-weight models and adding cost and efficiency dimensions to the leaderboard.

But AI capabilities are ever-evolving, and measurement needs to follow suit. As part of our July release, we have adopted the Harbor framework , which includes an updated version of the benchmarking agent used to evaluate models. 7 Max ) to the leaderboard.

We’re also sharing opportunities for you, the Android developer community, to contribute to the benchmark. Upgrading our methodology with the Harbor framework When we designed Android Bench, we anchored our methodology on leading industry standards available at the time. We used mini-swe-agent v1, a general-purpose benchmarking agent, and adapted it to the nuances of Android development to provide a baseline measurement for the capabilities of models for common Android development tasks.

To continue providing you with state-of-the-art evaluations that accurately measure the latest model capabilities on Android development, we are standardizing our benchmark to the Harbor framework . Harbor defines standards and integrations that make it easy for anyone to run the benchmark, evaluate their preferred set-up, or share results – providing you with additional transparency and visibility. This upgrade enables us to more rigorously evaluate models and their capabilities, and we re-ran the benchmark on all models to establish an updated baseline.

This means there is a minor shift in scoring, but you will still be able to view historical scores within the archive on our website. We want to ensure Android Bench is helpful for you, so we will continuously update it as our evaluations and the industry mature. 7 Max to the Android Bench leaderboard.

2. 4. You can check out model performance and efficiency metrics on the updated leaderboard to see how these new and previous models navigate Android-specific challenges like Jetpack Compose migrations, wearable networking, and platform API updates.

Opening Android Bench to community contributions From the beginning, we’ve valued an open and transparent approach, which is why we made our original methodology and test harness publicly available on GitHub. You’ve asked for a way to provide feedback on our dataset, so now we’re taking collaboration a step further by giving you, the Android developer community, a chance to shape Android Bench. Starting today, you can contribute to Android Bench in two ways: Design and submit your own Android development tasks to evaluate how models handle the scenarios that matter to you.