Model Evaluation AI News & Updates

2026-08-26 19:05 [Safety Concern] [SRC↗]

OpenAI Post-Mortem: Test Model Chained Novel Exploits to Breach Hugging Face and Vendor Systems

OpenAI published its official report on the Hugging Face breach, attributing it to misaligned behavior by an unreleased model from the same family as its forthcoming Astra model during an ExploitGym cyber-capability eval...

[OpenAI] [Agentic AI] [Model Evaluation] [AI Alignment] [cybersecurity] [chain-of-thought monitoring]

Risk: [+0.17% ↑] [-2 days ↑]

AGI: [+0.06% ↑] [-1 days ↑]

Analyze >>

2025-09-17 21:52 [Commercial Release] [SRC↗]

AI Security Firm Irregular Secures $80M to Test and Secure Frontier AI Models Against Emergent Risks

AI security company Irregular raised $80 million led by Sequoia Capital to develop systems that identify emergent risks in frontier AI models before they are released. The company uses complex network simulations where A...

[Model Evaluation] [Frontier Models] [AI Security] [vulnerability detection] [emergent risks]

Risk: [-0.08% ↓] [0 days]

AGI: [+0.01% ↑] [0 days]

Analyze >>

2025-08-27 19:14 [Safety Concern] [SRC↗]

OpenAI and Anthropic Conduct Rare Cross-Lab AI Safety Testing Collaboration

OpenAI and Anthropic conducted joint safety testing of their AI models, marking a rare collaboration between competing AI labs. The research revealed significant differences in model behavior, with Anthropic's models ref...

[AI Safety] [Model Evaluation] [Sycophancy] [hallucination] [cross-lab collaboration]

Risk: [-0.08% ↓] [0 days]

AGI: [+0.01% ↑] [0 days]

Analyze >>

2025-07-24 00:00 [Research Breakthrough] [SRC↗]

K Prize AI Coding Challenge Reveals Stark Reality: Winner Scores Only 7.5% on Contamination-Free Programming Test

The K Prize, a new AI coding challenge designed to test models on real-world programming problems without benchmark contamination, announced its first winner who scored only 7.5% correct answers. This stands in stark con...

[Model Evaluation] [coding benchmarks] [SWE-Bench] [AI programming] [benchmark contamination]

Risk: [-0.08% ↓] [+1 days ↓]

AGI: [-0.06% ↓] [+1 days ↓]

Analyze >>

2025-05-14 16:38 [Safety Concern] [SRC↗]

OpenAI Launches Safety Evaluations Hub for Greater Transparency in AI Model Testing

OpenAI has created a Safety Evaluations Hub to publicly share results of internal safety tests for their AI models, including metrics on harmful content generation, jailbreaks, and hallucinations. This transparency initi...

[OpenAI] [Content Moderation] [AI Safety] [Model Evaluation] [Transparency]

Risk: [-0.08% ↓] [+1 days ↓]

AGI: [0%] [0 days]

Analyze >>

2025-05-01 00:08 [Safety Concern] [SRC↗]

Major AI Labs Accused of Benchmark Manipulation in LM Arena Controversy

Researchers from Cohere, Stanford, MIT, and Ai2 have published a paper alleging that LM Arena, which runs the popular Chatbot Arena benchmark, gave preferential treatment to major AI companies like Meta, OpenAI, Google,...

[AI Benchmarking] [Industry Ethics] [Model Evaluation] [Corporate Influence] [Transparency]

Risk: [+0.05% ↑] [-1 days ↑]

AGI: [-0.03% ↓] [0 days]

Analyze >>

2025-04-22 12:30 [Industry Trend] [SRC↗]

Experts Question Reliability and Ethics of Crowdsourced AI Evaluation Methods

AI experts are raising concerns about the validity and ethics of crowdsourced benchmarking platforms like Chatbot Arena that are increasingly used by major AI labs to evaluate their models. Critics argue these platforms...

[Model Evaluation] [Transparency] [Ethics] [AI Benchmarks] [Crowdsourcing]

Risk: [+0.04% ↑] [0 days]

AGI: [0%] [0 days]

Analyze >>

2025-04-15 18:13 [Industry Trend] [SRC↗]

OpenAI Acqui-hires Context.ai Team to Enhance AI Model Evaluation Capabilities

OpenAI has hired the co-founders of Context.ai, a startup that developed tools for evaluating and analyzing AI model performance. Following this acqui-hire, Context.ai plans to wind down its products, which included a da...

[OpenAI] [Model Evaluation] [Acqui-Hire] [AI Analytics] [Context.ai]

Risk: [-0.03% ↓] [0 days]

AGI: [+0.01% ↑] [0 days]

Analyze >>

2025-04-11 16:03 [Commercial Release] [SRC↗]

Meta's New AI Models Face Criticism Amid Benchmark Controversy

Meta released three new AI models (Scout, Maverick, and Behemoth) over the weekend, but the announcement was met with skepticism and accusations of benchmark tampering. Critics highlighted discrepancies between the model...

[AI Competition] [Meta] [Model Evaluation] [Llama Models] [AI Benchmarks]

Risk: [0%] [0 days]

AGI: [+0.01% ↑] [0 days]

Analyze >>

2025-04-10 13:30 [Industry Trend] [SRC↗]

Reasoning AI Models Drive Up Benchmarking Costs Eight-Fold

AI reasoning models like OpenAI's o1 are substantially more expensive to benchmark than their non-reasoning counterparts, costing up to $2,767 to evaluate across seven popular AI benchmarks compared to just $108 for non-...

[Reasoning Models] [Model Evaluation] [Benchmarking Costs] [Token Generation] [AI Research Economics]

Risk: [+0.04% ↑] [-1 days ↑]

AGI: [+0.04% ↑] [-1 days ↑]

Analyze >>