Mercor Built An AI Agent To Automate SRE. Here’s What Happened
AI coding tools may have sped up development but SRE has remained a challenge for many teams. However, password management provider 1Password and Mercor, a startup that helps AI labs like OpenAI and Anthropic train and evaluate AI models, today announced the results of a project in which the pair built and deployed an AI agent in 1Password’s environment to help automate SRE investigations.
The agent, which uses Codex as an execution layer, is designed to automate the investigation of engineering incidents. Mercor built an evaluation suite based on real engineering workflows, drawing on more than 300 historical incidents recorded across 1Password’s systems, alongside its deployment history and source code, to develop evaluation material that connects alerts to root causes and remediation actions.
Whenever a monitoring alert activates or an engineer reports an incident, the agent begins a Codex investigation, using its long horizon capabilities to identify the affected service, the responsible team, the most likely cause and the first action a human engineer should consider. According to Nancy Wang, CTO of 1Password, the agent has achieved a 95% accuracy across 78 different evaluation runs, and demonstrates a 57% lower median time to remediate.
More broadly, the project highlights how agents can be used to address reliability issues more efficiently, which has been a challenge across the industry. For instance, in 2025, software development lifecycle platform Harness surveyed 900 engineers, platform leaders and technical managers and found that while 63% of organizations are shipping code to production faster, 45% of all deployments linked to AI-generated code lead to problems, such as increased security and quality risk.
Many organizations are looking to AI to support the SRE function, with a Dynatrace survey of 919 IT leaders finding that half of SREs now use AI-powered capabilities for automating incident response.
Mercor’s approach to automating the SRE involved focusing on evals. Adarsh Hiremath, cofounder and co-CEO of Mercor, a startup that now holds an estimated $20 billion valuation , told me in a video interview that the “core thesis” behind the project was that “every enterprise is going to need to invest in evals because enterprises shouldn’t deploy AI agents that they can’t measure.”
Hiremath told me that Mercor had a team of forward deployed engineers working with 1Password to map investigation patterns across historic incidents, capturing relationships between systems, documentation and services in a context graph. The graph recorded how engineers investigate incidents and attempted to replicate the work a human engineer would perform during an investigation.
Hiremath describes Mercor as “fundamentally an evals company,” which works with different AI labs to understand how models are performing on different benchmarks across different domains and tasks. It also develops scores that are then used to rank the models. “The way that we build evals is we will build them based on how a specific company does the work."
“We built an SRE agent with 1Password on OpenAI’s Codex so when an alert fires, it just like investigates across monitoring code, deployments, past incidents, and then it posts kind of like a evidence-backed first read in under four minutes for under $2, on what exactly happened," Hiremath said.
While this is a single project, Hiremath confirmed that Mercor also has its own SRE agent automating similar processes internally. The startup also makes heavy use of AI coding tools, with Hiremath confirming that most of Mercor’s code is AI-generated with human oversight.
Using AI to automate SRE has historically been a challenge for many teams. Back in 2025, IBM Research evaluated AI agents across real world IT automation tasks and found that agents powered by state of the art models resolved only 13.8% of SRE scenarios. Now that frontier AI models have evolved, many organizations are looking to automate alert response and remediation, and have access to tools that can potentially reduce toil.
“We are seeing others create SRE agents that can investigate incidents. These agents are useful because they can help enrich alerts with additional context identify the root cause of the incident and propose next steps for remediation,” Allie Mellen, a Forrester principal analyst on cybersecurity, geopolitics and AI told me via email.
Mellen notes that many agents are trained to investigate, plan and take constrained actions in SRE settings as well as other functions like incident response. That being said, she adds the biggest challenge Forrester sees is that at times agents are developed that are too generic and not built for a specific business or use case. “Having agents built with the business’s processes in mind makes the agents more productive and useful," Mellen said.
Like Hiremath, Mellen points to the importance of evaluating agents, arguing that having a detailed view of the evaluation methodology used when developing an agent is helpful for organizations looking to build their own AI agents. She also says metrics like escalation rate and business outcome can help measure the effectiveness of such agents.
AI agents have the potential to be significantly impactful for engineers, particularly when it comes to assisting with triaging alerts and streamlining time consuming investigations across multiple systems. "What I want to index on as a CTO is feature velocity because the less time your engineers spend on fixing stuff, the more time they can spend on building new stuff," Wang said.
Wang explains that 1Password created exams to run the agent, using Harbor to come up with a scoring rubric that scored the accuracy of the agent. She notes that the agent “progressively got better," after each run. “Now we’re actually at the point where we’re getting ready to roll this agent out into production, which means that we can save human hours from having to wake up at two, three am to go and look at incidents."
That being said, Wang highlights some challenges during the evals process. “Creating the exams was actually probably the hardest part of this exercises because you essentially had to freeze everything as it were in time," in this case. At the same time, deploying agents in SRE operations also creates new risks.
As OpenAI’s HuggingFace breach highlighted, AI agents can act in unexpected ways. 1Password is attempting to mitigate risk by controlling permissions, only giving the agent permissions to make calls to approved tools with read but not write access. It also has fail safes built into the agent to reach out to a human operator when required. Wang also says that 1Password isn’t at the point where its allowing the agent to make changes to production databases.