AI Safety AI News & Updates

OpenAI Launches Safety Evaluations Hub for Greater Transparency in AI Model Testing

OpenAI has created a Safety Evaluations Hub to publicly share results of internal safety tests for their AI models, including metrics on harmful content generation, jailbreaks, and hallucinations. This transparency initiative comes amid criticism of OpenAI's safety testing processes, including a recent incident where GPT-4o exhibited overly agreeable responses to problematic requests.

xAI Fails to Deliver Promised AI Safety Framework by Self-Imposed Deadline

Elon Musk's AI company xAI has missed its May 10 deadline to publish a finalized AI safety framework, which was promised in February at the AI Seoul Summit. The company's initial draft framework was criticized for only applying to future models and lacking specifics on risk mitigation, while watchdog organizations have ranked xAI poorly for its weak risk management practices compared to industry peers.

Google's Gemini 2.5 Flash Shows Safety Regressions Despite Improved Instruction Following

Google has disclosed in a technical report that its recent Gemini 2.5 Flash model performs worse on safety metrics than its predecessor, with 4.1% regression in text-to-text safety and 9.6% in image-to-text safety. The company attributes this partly to the model's improved instruction-following capabilities, even when those instructions involve sensitive content, reflecting an industry-wide trend of making AI models more permissive in responding to controversial topics.

Anthropic Sets 2027 Goal for AI Model Interpretability Breakthroughs

Anthropic CEO Dario Amodei has published an essay expressing concern about deploying increasingly powerful AI systems without better understanding their inner workings. The company has set an ambitious goal to reliably detect most AI model problems by 2027, advancing the field of mechanistic interpretability through research into AI model "circuits" and other approaches to decode how these systems arrive at decisions.

Former Y Combinator President Launches AI Safety Investment Fund

Geoff Ralston, former president of Y Combinator, has established the Safe Artificial Intelligence Fund (SAIF) focused on investing in startups working on AI safety, security, and responsible deployment. The fund will provide $100,000 investments to startups focused on improving AI safety through various approaches, including clarifying AI decision-making, preventing misuse, and developing safer AI tools, though it explicitly excludes fully autonomous weapons.

Google's Gemini 2.5 Pro Safety Report Falls Short of Transparency Standards

Google published a technical safety report for its Gemini 2.5 Pro model several weeks after its public release, which experts criticize as lacking critical safety details. The sparse report omits detailed information about Google's Frontier Safety Framework and dangerous capability evaluations, raising concerns about the company's commitment to AI safety transparency despite prior promises to regulators.

OpenAI Implements Specialized Safety Monitor Against Biological Threats in New Models

OpenAI has deployed a new safety monitoring system for its advanced reasoning models o3 and o4-mini, specifically designed to prevent users from obtaining advice related to biological and chemical threats. The system, which identified and blocked 98.7% of risky prompts during testing, was developed after internal evaluations showed the new models were more capable than previous iterations at answering questions about biological weapons.

OpenAI Updates Safety Framework, May Reduce Safeguards to Match Competitors

OpenAI has updated its Preparedness Framework, indicating it might adjust safety requirements if competitors release high-risk AI systems without comparable protections. The company claims any adjustments would still maintain stronger safeguards than competitors, while also increasing its reliance on automated evaluations to speed up product development. This comes amid accusations from former employees that OpenAI is compromising safety in favor of faster releases.

Sutskever's Safe Superintelligence Startup Valued at $32 Billion After New Funding

Safe Superintelligence (SSI), founded by former OpenAI chief scientist Ilya Sutskever, has reportedly raised an additional $2 billion in funding at a $32 billion valuation. The startup, which previously raised $1 billion, was established with the singular mission of creating "a safe superintelligence" though details about its actual product remain scarce.

Safe Superintelligence Startup Partners with Google Cloud for AI Research

Ilya Sutskever's AI safety startup, Safe Superintelligence (SSI), has established Google Cloud as its primary computing provider, using Google's TPU chips to power its AI research. SSI, which launched in June 2024 with $1 billion in funding, is focused exclusively on developing safe superintelligent AI systems, though specific details about their research approach remain limited.