跳到正文
Sebastian Raschka· @rasbt · X·· 2026-09-15AI 评分22
AI 导读

Sebastian Raschka 用 Paint UI 复刻中心图像的任务对比了 GPT-5.6 Astra 与 Qwen3.8 Max 的计算机使用能力:Astra 默认用几何图形分层绘制,Qwen 则逐像素绘制,后者结果天然更接近原图、得分会更高。但他认为不能据此断定 Qwen 的计算机使用能力或视觉理解更强,也不代表其泛化更好,这凸显了仅比较最终结果的基准测试有多不可靠。

正文 · 原文

Some food for thought when designing benchmarks...

So, here's a little computer-use (visual) comparison between GPT-5.6 Astra and Qwen3.8 Max.

The task here was to recreate the image in the center using the Paint UI.

Super interesting how the two different LLMs+Harnesses approached this totally differently by default.

I.e., Astra tried to approach this by drawing and layering geometric shapes. Qwen approached this pixel by pixel. (Of course, the pixel-by-pixel result looks closer to the original, it's essentially a low-res version of that by nature.)

So, the Qwen-generated image would surely score higher in the sense that it's closer to the original.

But I wouldn’t conclude from this example that one LLM generalizes better than the other on other tasks. Also, I wouldn't say Qwen has better compute-use capabilities or better visual understanding than Astra.

But it highlights an interesting point about how slippery benchmarks are when they only compare final results.

来源:Sebastian Raschka · x.com