SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? - John Yang 002; Carlos E. Jimenez; Alex L. Zhang; Kilian Lieret; Joyce Yang; Xindi Wu; Ori Press; Niklas Muennighoff; Gabriel Synnaeve; Karthik R. Narasimhan; Diyi Yang; Sida Wang 001; et al. (ICLR, 2025)

KernelBench: Can LLMs Write Efficient GPU Kernels? - Anne Ouyang; Simon Guo 004; Simran Arora; Alex L. Zhang; William Hu; Christopher Ré; Azalia Mirhoseini (CoRR, 2025)

VideoGameBench: Can Vision-Language Models complete popular video games? - Alex L. Zhang; Thomas L. Griffiths 001; Karthik R. Narasimhan; Ofir Press (CoRR, 2025)