Artificial Analysis: nearly 1 in 4 serverless endpoints score below the reference on its Endpoint Accuracy Index v1.1

Artificial Analysis released v1.1 of its Endpoint Accuracy Index, which scores each serverless endpoint of an open-weights model as a percentage of its own self-hosted reference deployment of the official weights. The score weighs tool calling (BFCL-500), scientific reasoning (HLE-250) and long-context recall (AA-LCR-25) equally. Across the 100+ endpoints it benchmarked, it says nearly 1 in 4 score below the reference.

What changed in v1.1, according to Artificial Analysis:

• AA-LCR-25 moves to AA-LCR v1.1, which corrects answer-key errors and moves grading to GPT-5.6 Luna (medium). Each question now runs 25 times instead of 10, cutting the average 95% confidence interval from ±4.7 to ±2.6 points.
• Coverage grows to six models: GLM-5.3 Flash (max) and Meta's Muse Spark Open (xhigh) join DeepSeek V4.1 Flash 1017 (max), GLM-5.3 (max), Kimi K3 (max) and gpt-oss-120b (high).

Its findings:

• GLM-5.3 Flash: 16 of 22 endpoints are statistically in line with the reference and one scores above it (104.2). Two endpoints capped at about 32K output tokens cut off roughly half of HLE-250 responses and score 22% and 29% against 55% for the reference. Two others produce about 30% fewer reasoning tokens and score 48%.
• Muse Spark Open: three of five endpoints match the reference (99.5 to 99.9), including one serving FP4 weights. The other two lose accuracy only on tool calling, returning arguments the tool defines as text, such as booking IDs, as numbers (50% and 51% on BFCL-500 against 59%).
• GLM-5.3: 26 endpoints, scoring 67.2 to 99.3; 16 are in line with the reference and 10 below. Artificial Analysis says the largest gaps come from output token caps and reasoning loops, and that quantization is not a cause, with five of six FP4 endpoints in line with the FP8 reference.
• DeepSeek V4.1 Flash 1017: all 22 endpoints are in line (97.5 to 101.4), a 3.9-point range that makes it the most consistent model in the index.