Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More
A customer service chatbot confidently describes a product that doesn’t exist. A financial AI invents market data. A healthcare bot provides dangerous medical advice. These AI hallucinations, once dismissed as amusing quirks, have become million-dollar problems for companies rushing to deploy artificial intelligence.
Today, Patronus AI, a San Francisco startup that recently secured $17 million in Series A funding, launched what it calls the first self-serve platform to detect and prevent AI failures in real-time. Think of it as a sophisticated spell-checker for AI systems, catching errors before they reach users.
Inside the AI safety net: How it works
“Many companies are grappling with AI failures in production, facing issues like hallucinations, security vulnerabilities, and unpredictable behavior,” said Anand Kannappan, Patronus AI’s CEO, in an interview with VentureBeat. The stakes are high: Recent research by the company found that leading AI models like GPT-4 reproduce copyrighted content 44% of the time when prompted, while even advanced models generate unsafe responses in over 20% of basic safety tests.
Advertisement
The timing couldn’t be more critical. As companies rush to implement generative AI capabilities — from customer service chatbots to content generation systems — they’re discovering that existing safety measures fall short. Current evaluation tools like Meta’s LlamaGuard perform below 50% accuracy, making them little better than a coin flip.
Patronus AI’s solution introduces several innovations that could reshape how businesses deploy AI. Perhaps most significant is its “judge evaluators” feature, which allows companies to create custom rules in plain English.
“You can customize evaluation to exactly [meet] your product needs,” Varun Joshi, Patronus AI’s product lead, told VentureBeat. “We let customers write out in English what they want to evaluate and check for.” A financial services company might specify rules about regulatory compliance, while a healthcare provider could focus on patient privacy and medical accuracy.
From detection to prevention: The technical breakthrough
The system’s cornerstone is Lynx, a breakthrough hallucination detection model that outperforms GPT-4 by 8.3% in detecting medical inaccuracies. The platform operates at two speeds: a quick-response version for real-time monitoring and a more thorough version for deeper analysis. “The small versions can be used for real-time guardrails, and the large ones might be more appropriate for offline analysis,” Joshi told VentureBeat.
Advertisement
Beyond traditional error checking, the company has developed specialized tools like CopyrightCatcher, which detects when AI systems reproduce protected content, and FinanceBench, the industry’s first benchmark for evaluating AI performance on financial questions. These tools work in concert with Lynx to provide comprehensive coverage against AI failures.
Beyond simple guard rails: Reshaping AI safety
The company has adopted a pay-as-you-go pricing model, starting at $10 per 1000 API calls for smaller evaluators and $20 per 1000 API calls for larger ones. This pricing structure could dramatically increase access to AI safety tools, making them available to startups and smaller businesses that previously couldn’t afford sophisticated AI monitoring.
Early adoption suggests major enterprises see AI safety as a critical investment, not just a nice-to-have feature. The company has already attracted clients including HP, AngelList, and Pearson, along with partnerships with tech giants like Nvidia, MongoDB, and IBM.
What sets Patronus AI apart is its focus on improvement rather than just detection. “We can actually highlight the span of the specific piece of text where the hallucination is,” Kannappan explained. This precision allows engineers to quickly identify and fix problems, rather than just knowing something went wrong.
Advertisement
The race against AI hallucinations
The launch comes at a pivotal moment in AI development. As large language models like GPT-4 and Claude become more powerful and widely used, the risks of AI failures grow correspondingly larger. A hallucinating AI system could expose companies to legal liability, damage customer trust, or worse.
Recent regulatory moves, including President Biden’s AI executive order and the EU’s AI Act, suggest that companies will soon face legal requirements to ensure their AI systems are safe and reliable. Tools like Patronus AI’s platform could become essential for compliance.
“Good evaluation is not just protecting against a bad outcome — it’s deeply about improving your models and improving your products,” Joshi emphasizes. This philosophy reflects a maturing approach to AI safety, moving from simple guard rails to continuous improvement.
The real test for Patronus AI isn’t just catching mistakes — it will be keeping pace with AI’s breakneck evolution. As language models grow more sophisticated, their hallucinations may become harder to spot, like finding increasingly convincing forgeries.
Advertisement
The stakes couldn’t be higher. Every time an AI system invents facts, recommends dangerous treatments, or generates copyrighted content, it erodes the trust these tools need to transform business. Without reliable guardrails, the AI revolution risks stumbling before it truly begins.
In the end, it’s a simple truth: If artificial intelligence can’t stop making things up, it may be humans who end up paying the price.
VB Daily
Stay in the know! Get the latest news in your inbox daily
From streamlining operations to automating complex processes, AI has revolutionized how organizations approach tasks – however, as the technology becomes more prevalent, organizations are discovering the rush to embrace AI may come with unintended consequences.
A report by Swimlane reveals while AI offers tremendous benefits, its adoption has outpaced many companies’ ability to safeguard sensitive data. As businesses deeply integrate AI into their operations, they must also contend with the associated risks, including data breaches, compliance lapses, and security protocol failures.
AI works with Large Language Models (LLMs) which are trained using vast datasets that often include publicly available information. These datasets can consist of text from sources like Wikipedia, GitHub, and various other online platforms, which provide a rich corpus for training the models. This means that if a company’s data is available online, it will likely be used for training LLMs.
Data handling and public LLMs
The study revealed a gap between protocol and practice when sharing data in large public language models (LLMs). Although 70% of organizations claim to have specific protocols to safeguard the sharing of sensitive data with public LLMs, 74% of respondents are aware that individuals within their organizations still input sensitive information into these platforms.
This discrepancy highlights a critical flaw in enforcement and employee compliance with established security measures. Furthermore, there is a constant barrage of AI-related messaging which is wearing down professionals and 76% of respondents agree that the market is currently saturated with AI-related hype.
Advertisement
This overexposure is causing a form of AI fatigue and over half (55%) of those surveyed reported feeling overwhelmed by the persistent focus on AI, signalling that the industry may need to shift its approach to promoting the technology.
Interestingly, despite this fatigue, experience with AI and machine learning (ML) technologies is becoming a crucial factor in hiring decisions. A striking 86% of organizations reported that familiarity with AI plays a significant role in determining the suitability of candidates. This shows how ingrained AI is becoming, not just in cybersecurity tools but in the workforce needed to manage them.
Sign up to the TechRadar Pro newsletter to get all the top news, opinion, features and guidance your business needs to succeed!
In the cybersecurity sector, AI and LLMs have had a positive impact, as the report claims 89% of organizations credit AI technologies for boosting the efficiency of their cybersecurity teams.
Agatha All Along is one of the most widely liked titles that Marvel Studios has released in, well, a while. The WandaVision spin-off premiered in late September and did a lot to win over even some of the Marvel Cinematic Universe’s more skeptical fans across its nine episodes. While Agatha All Along does set up some exciting future possibilities for several of its characters, though, its finale doesn’t include a single post-credits scene.
According to Agatha All Along creator Jac Schaeffer, that isn’t because she didn’t have any ideas for one. When asked about the series’ lack of a post-credits tag, Schaeffer told Variety, “That’s a Marvel decision. I know nothing more than that.” The writer and showrunner went on to reveal that she actually wrote multiple potential post-credits scenes for Agatha All Along, none of which were ultimately used because of behind-the-scenes decision-making by Marvel.
“I wrote a number of tags, because you always do on every Marvel everything. I love writing tags. I think some of my best writing is in the tags that were never made. I should have a little binder of my tags. They’re so fun to write, because you’re writing the promise without having to deliver on anything. They’re the best,” Schaeffer commented. “But there are so many things that factor into those. And I was told that we weren’t going to do a tag on this show.”
Schaeffer, of course, has experience writing post-credits scenes. After all, her previous Marvel show, 2021’s WandaVision, ends with a brief scene that directly sets up the events of 2022’s Doctor Strange in the Multiverse of Madness and specifically Wanda Maximoff’s (Elizabeth Olsen) villainous actions throughout it. The absence of a similar post-credits tag at the end of Agatha All Along, therefore, came as a shock for a number of reasons.
Marvel’s thinking behind this absence may have to remain a mystery for the time being to viewers. Fortunately, Agatha All Along does go out of its way to set up more adventures for, at the very least, Billy Maximoff (Joe Locke). The character teams up with the ghost of Agatha Harkness (Kathryn Hahn) to go searching for his missing brother’s soul in the Agatha All Along finale, but fans will have to wait to learn more about Marvel’s plans for Agatha, Billy, and the other members of the Maximoff family.
Bose is without a doubt one of the top contenders for active noise-cancelling headphones with its QuietComfort model, and right now Amazon has a pretty good deal on them worth looking into.
Normally the QuietComfort headphones would retail for $349 at their full price. However, Amazon currently has them on sale for $199. This is a great price for these headphones and it’s the lowest we’ve seen them. It’s also the lowest price for these headphones in the last 30 days. With the average price sitting at around $285.20.
The main feature here is the active noise cancellation. It’s part of why these have been so popular over time. Because you can wear them and drown out almost everything so you can enjoy your music or whatever else you’re listening to. They’re great for travel in this sense. If you fly a lot, then these are perfect for taking on flights to make sure you don’t have to hear everyone else on the plane.
They also sound pretty good. Battery life is great too, lasting up to 24 hours on a single charge. While that isn’t the longest battery life we’ve ever seen, it’s more than enough for most people and will get you through a few days before you need to plug them in. A feature that we really love is that they fold up and they come with a protective travel case. Whether you use the case or not, they’re easily packable. Bose also sells these in several different colors. This includes Cypress Green, Moonstone Blue, Black, Blue Dusk, Chilled Lilac, Sandstone, and White.
Advertisement
Plus, all of these colors are on sale for the discounted $199 price tag. There’s an EQ in the companion app as well if you want to tune the sound to your personal liking.
The FBI issued a statement on Saturday about deceptive videos circulating ahead of the election, saying it’s aware of two such videos “falsely claiming to be from the FBI relating to election security.” That includes one claiming the FBI had “apprehended three linked groups committing ballot fraud,” and one about Kamala Harris’ husband. Both depict false content, the FBI said.
Disinformation — including the spread of political deepfakes and other forms of misleading videos and imagery — has been a major concern in the leadup to the US presidential election. In its statement posted on X, the FBI added:
Election integrity is among our highest priorities, and the FBI is working closely with state and local law enforcement partners to respond to election threats and protect our communities as Americans exercise their right to vote. Attempts to deceive the public with false content about FBI operations undermines our democratic process and aims to erode trust in the electoral system.
Just a day earlier, the along with the Office of the Director of National Intelligence (ODNI) and the Cybersecurity and Infrastructure Security Agency (CISA) said they’d traced two other videos back to “Russian influence actors,” including one “that falsely depicted individuals claiming to be from Haiti and voting illegally in multiple counties in Georgia.”
Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More
The introduction of ChatGPT has brought large language models (LLMs) into widespread use across both tech and non-tech industries. This popularity is primarily due to two factors:
LLMs as a knowledge storehouse: LLMs are trained on a vast amount of internet data and are updated at regular intervals (that is, GPT-3, GPT-3.5, GPT-4, GPT-4o, and others);
Emergent abilities: As LLMs grow, they display abilities not found in smaller models.
Does this mean we have already reached human-level intelligence, which we call artificial general intelligence (AGI)? Gartner defines AGI as a form of AI that possesses the ability to understand, learn and apply knowledge across a wide range of tasks and domains. The road to AGI is long, with one key hurdle being the auto-regressive nature of LLM training that predicts words based on past sequences. As one of the pioneers in AI research, Yann LeCun points out that LLMs can drift away from accurate responses due to their auto-regressive nature. Consequently, LLMs have several limitations:
Limited knowledge: While trained on vast data, LLMs lack up-to-date world knowledge.
Limited reasoning: LLMs have limited reasoning capability. As Subbarao Kambhampati points outLLMs are good knowledge retrievers but not good reasoners.
No Dynamicity: LLMs are static and unable to access real-time information.
To overcome LLM’s challenges, a more advanced approach is required. This is where agents become crucial.
Agents to the rescue
The concept of intelligent agent in AI has evolved over two decades, with implementations changing over time. Today, agents are discussed in the context of LLMs. Simply put, an agent is like a Swiss Army knife for LLM challenges: It can help us in reasoning, provide means to get up-to-date information from the Internet (solving dynamicity issues with LLM) and can achieve a task autonomously. With LLM as its backbone, an agent formally comprises tools, memory, reasoning (or planning) and action components.
Components of AI agents
Tools enable agents to access external information — whether from the internet, databases, or APIs — allowing them to gather necessary data.
Memory can be short or long-term. Agents use scratchpad memory to temporarily hold results from various sources, while chat history is an example of long-term memory.
The Reasoner allows agents to think methodically, breaking complex tasks into manageable subtasks for effective processing.
Actions: Agents perform actions based on their environment and reasoning, adapting and solving tasks iteratively through feedback. ReAct is one of the common methods for iteratively performing reasoning and action.
What are agents good at?
Agents excel at complex tasks, especially when in a role-playing mode, leveraging the enhanced performance of LLMs. For instance, when writing a blog, one agent may focus on research while another handles writing — each tackling a specific sub-goal. This multi-agent approach applies to numerous real-life problems.
Role-playing helps agents stay focused on specific tasks to achieve larger objectives, reducing hallucinations by clearly defining parts of a prompt — such as role, instruction and context. Since LLM performance depends on well-structured prompts, various frameworks formalize this process. One such framework, CrewAI, provides a structured approach to defining role-playing, as we’ll discuss next.
Advertisement
Multi agents vs single agent
Take the example of retrieval augmented generation (RAG) using a single agent. It’s an effective way to empower LLMs to handle domain-specific queries by leveraging information from indexed documents. However, single-agent RAG comes with its own limitations, such as retrieval performance or document ranking. Multi-agent RAG overcomes these limitations by employing specialized agents for document understanding, retrieval and ranking.
In a multi-agent scenario, agents collaborate in different ways, similar to distributed computing patterns: sequential, centralized, decentralized or shared message pools. Frameworks like CrewAI, Autogen, and langGraph+langChain enable complex problem-solving with multi-agent approaches. In this article, I have used CrewAI as the reference framework to explore autonomous workflow management.
Workflow management: A use case for multi-agent systems
Most industrial processes are about managing workflows, be it loan processing, marketing campaign management or even DevOps. Steps, either sequential or cyclic, are required to achieve a particular goal. In a traditional approach, each step (say, loan application verification) requires a human to perform the tedious and mundane task of manually processing each application and verifying them before moving to the next step.
Each step requires input from an expert in that area. In a multi-agent setup using CrewAI, each step is handled by a crew consisting of multiple agents. For instance, in loan application verification, one agent may verify the user’s identity through background checks on documents like a driving license, while another agent verifies the user’s financial details.
Advertisement
This raises the question: Can a single crew (with multiple agents in sequence or hierarchy) handle all loan processing steps? While possible, it complicates the crew, requiring extensive temporary memory and increasing the risk of goal deviation and hallucination. A more effective approach is to treat each loan processing step as a separate crew, viewing the entire workflow as a graph of crew nodes (using tools like langGraph) operating sequentially or cyclically.
Since LLMs are still in their early stages of intelligence, full workflow management cannot be entirely autonomous. Human-in-the-loop is needed at key stages for end-user verification. For instance, after the crew completes the loan application verification step, human oversight is necessary to validate the results. Over time, as confidence in AI grows, some steps may become fully autonomous. Currently, AI-based workflow management functions in an assistive role, streamlining tedious tasks and reducing overall processing time.
Production challenges
Bringing multi-agent solutions into production can present several challenges.
Scale: As the number of agents grows, collaboration and management become challenging. Various frameworks offer scalable solutions — for example, Llamaindex takes event-driven workflow to manage multi-agents at scale.
Latency: Agent performance often incurs latency as tasks are executed iteratively, requiring multiple LLM calls. Managed LLMs (like GPT-4o) are slow because of implicit guardrails and network delays. Self-hosted LLMs (with GPU control) come in handy in solving latency issues.
Performance and hallucination issues: Due to the probabilistic nature of LLM, agent performance can vary with each execution. Techniques like output templating (for instance, JSON format) and providing ample examples in prompts can help reduce response variability. The problem of hallucination can be further reduced by training agents.
Final thoughts
As Andrew Ng points out, agents are the future of AI and will continue to evolve alongside LLMs. Multi-agent systems will advance in processing multi-modal data (text, images, video, audio) and tackling increasingly complex tasks. While AGI and fully autonomous systems are still on the horizon, multi-agents will bridge the current gap between LLMs and AGI.
DataDecisionMakers is where experts, including the technical people doing data work, can share data-related insights and innovation.
If you want to read about cutting-edge ideas and up-to-date information, best practices, and the future of data and data tech, join us at DataDecisionMakers.
CareYaya, a platform that matches people who need caregivers with healthcare students, is working to disrupt the caregiving industry. The startup, which exhibited as part of the Battlefield 200 at TechCrunch Disrupt, is looking to enhance affordable in-home support, while also helping students prepare for their future healthcare careers.
The startup was founded in 2022 by Neal Shah, who came up with the idea for the startup based on his own experience as a caregiver for his wife after she became ill with cancer and various other ailments. During this time, Shah was a partner at a hedge fund and had to wind down his fund to become a full-time caregiver for two years.
To get additional care for his wife, Shah hired college students who were studying healthcare to be caregivers for his wife. Shah learned that other families were doing the same thing informally by posting flyers at local campuses to find someone who was qualified to look after their loved one.
“I was like, wouldn’t it be nice to just build a formal system for them to do it, where you don’t have to go to your local nursing school or your local undergrad campus and post flyers,” Shah told TechCrunch. “This is what I was doing. So we were like, if you can bring that into a formal capacity through a tech platform, you can make a big impact.”
Advertisement
Fast-forward to 2024, and the platform now has over 25,000 students on its platform from numerous schools, including Duke University, Stanford, UC Berkeley, San Jose State, University of Texas at Austin, and more.
CareYaya performs background checks on students who want to join the platform and then completes video-based interviews with them. On the user side, people can join the platform and then detail the type of care their loved one needs. CareYaya then matches students to families, whether it’s for one-off sessions or continuous care. After the first session, both parties can leave ratings.
The startup says it can help families save thousands of dollars on recurring senior care. While at-home care costs an average of $35 per hour in the U.S., CareYaya charges between $17 and $20 per hour.
Since the students providing the care are tech savvy, CareYaya is equipping them with AI-powered technology to recognize and track disease progression in patients with Alzheimer’s and dementia. The company recently launched an LLM (large language model) that integrates with smart glasses to gather visual data to help students provide better real-time assistance and conduct early dementia screening.
In terms of the future, CareYaya wants to explore expanding beyond the United States, as the platform has seen interest from people in places like Canada, Australia, and the United Kingdom.
You must be logged in to post a comment Login