Anthropic, a leading artificial intelligence company, launched its new Message Batches API on Tuesday, allowing businesses to process large volumes of data at half the cost of standard API calls.

This new offering handles up to 10,000 queries asynchronously within a 24-hour window, marking a significant step towards making advanced AI models more accessible and cost-effective for enterprises dealing with big data.

The AI economy of scale: Batch processing brings down costs

The Batch API offers a 50% discount on both input and output tokens compared to real-time processing, positioning Anthropic to compete more aggressively with other AI providers like OpenAI, which introduced a similar batch processing feature earlier this year.

This move represents a significant shift in the AI industry’s pricing strategy. By offering bulk processing at a discount, Anthropic is effectively creating an economy of scale for AI computations.

This could lead to a surge in AI adoption among mid-sized businesses that were previously priced out of large-scale AI applications.

The implications of this pricing model extend beyond mere cost savings. It could fundamentally alter how businesses approach data analysis, potentially leading to more comprehensive and frequent large-scale analyses that were previously considered too expensive or resource-intensive.

From real-time to right-time: Rethinking AI processing needs

Anthropic has made the Batch API available for its Claude 3.5 Sonnet, Claude 3 Opus, and Claude 3 Haiku models through the company’s API. Support for Claude on Google Cloud’s Vertex AI is expected soon, while customers using Claude through Amazon Bedrock can already access batch inference capabilities.

The introduction of batch processing capabilities signals a maturing understanding of enterprise AI needs. While real-time processing has been the focus of much AI development, many business applications don’t require instantaneous results. By offering a slower but more cost-effective option, Anthropic is acknowledging that for many use cases, “right-time” processing is more important than real-time processing.

This shift could lead to a more nuanced approach to AI implementation in businesses. Rather than defaulting to the fastest (and often most expensive) option, companies may start to strategically balance their AI workloads between real-time and batch processing, optimizing for both cost and speed.

The double-edged sword of batch processing

Despite the clear benefits, the move towards batch processing raises important questions about the future direction of AI development. While it makes existing models more accessible, there’s a risk that it could divert resources and attention from advancing real-time AI capabilities.

The trade-off between cost and speed is not new in technology, but in the field of AI, it takes on added significance. As businesses become accustomed to the lower costs of batch processing, there may be less market pressure to improve the efficiency and reduce the cost of real-time AI processing.

Moreover, the asynchronous nature of batch processing could potentially limit innovation in applications that rely on immediate AI responses, such as real-time decision making or interactive AI assistants.

Striking the right balance between advancing both batch and real-time processing capabilities will be crucial for the healthy development of the AI ecosystem.

As the AI industry continues to evolve, Anthropic’s new Batch API represents both an opportunity and a challenge. It opens up new possibilities for businesses to leverage AI at scale, potentially increasing access to advanced AI capabilities.

At the same time, it underscores the need for a thoughtful approach to AI development that considers not just immediate cost savings, but long-term innovation and diverse use cases.

The success of this new offering will likely depend on how well businesses can integrate batch processing into their existing workflows and how effectively they can balance the trade-offs between cost, speed, and computational power in their AI strategies.

When you talk to a chatbot like ChatGPT, Claude, or Gemini, your responses are largely linear. You often have to ask follow-up questions to get more information. For instance, when you are planning a trip, you might first ask the chatbot about the place and follow it up with questions on meal planning, activities, tips for the trip, and a packing list.

A team of former Google Maps engineers is building an infinite board interface called Cove. This design should help you generate several responses based on one question or prompt to an AI bot.

The startup founded by Stephen Chau, Andy Szybalski, and Mike Chu has raised $6 million in a seed round from Sequoia Capital, Elad Gil, Homebrew, Adverb, Scott Belsky, Lenny Rachitsky along with other angel investors.

The team has previously worked on some well known Google Maps features like Street View. Both Chau and Szybalski later joined Uber and were part of the team that worked on launching Uber Eats. Jess Lee, who is the lead investor for Sequoia for this round was also a colleague of Cove co-founders at Google Maps.

When they started working on this product, the trio felt that AI could be much more helpful in a collaborative space, especially for tasks like planning, writing, storyboarding, and research. On a call with TechCrunch, Chau said that thinking is a messy process, and you constantly iterate in your head while working on something. Your brain progressively creates branched-out thoughts.

“A lot of inspiration for Cove is really thinking about how we are used to human-to-human collaboration. When we work with other folks, we have a shared space, like a room where there is a shared context, and then we can communicate together on something. That’s what we wanted to build with Cove,” said Chau, expanding on why the startup chose to have an infinite canvas as an interface.

He opined that a chat interface for AI is very linear and not editable. For the most complex projects, having to deal with multiple conversation threads can be cumbersome.

The product

Cove has an infinite board interface on the web with a chatbot window in the bottom right corner. The bot shows you some suggestions for the projects you can work on through prompts so that you are not just staring at the blank space, thinking about how to get started. These examples include typical prompts like “Help me pick a birthday gift for my mom who likes gardening,” or “Help me plan a week-long trip to Yosemite for a family of four.”

When you click on any of these prompts, Cove will generate multiple response cards with tables and lists related to the prompt. For instance, if you ask the chatbot about trip planning, it can generate cards for meal planning, packing essentials, general tips, a list of hikes, and even create an itinerary.

You can edit any part of these cards by selecting some text and prompting it. You can also create a spin-off card or add more points, lists, or table elements to existing cards.

Users can add PDFs, links through a built-in browser, images, or a new blank card to add more context. They can also ask Cove questions about specific elements by referencing different cards and mentioning them.

Since this is a collaborative board, you can share the link with anyone. Just like in Google Docs, you can share it as a “view only” board or with editing abilities, depending on your needs. When people are collaborating on one board, all of them can invoke AI separately.

The startup has also released a Chrome extension. When you are browsing a website, you can invoke the extension, ask the AI bot about the site and content on screen, and add that to one of your projects or cards.

Opportunity and limitations

There are plenty of whiteboarding tools for collaborations, including Miro, TLDraw, Kosmik, and Visual Electric (which is also backed by Sequoia).

Cove feels that its AI features can attract a different kind of audience to use its product. The startup said that it is using a variety of models — including models from OpenAI, Anthropic, Meta (Llama) and Perplexity — to balance out costs of querying and deliverying quick answers.

Currently, Cove only works on the web interface, so the best way to use it is on the desktop. While you can access the boards on mobile, the small screen is not an ideal way to view them.

The founders said that people are using the tool for planning home renovation, business growth, and finding new sales projects. The company itself has used Cove to see customer feedback and generate better support articles.

“There are products where AI does the work for you and there are products where you do all the work. Cove is designed in a way where AI is more of your thought partner and collaborator. Think of it as an assistant that helps you rather than an oracle that tells you what to do,” Sequoia’s Lee said.

Cove is free to use with a limit of 100 cards. To unlock unlimited cards and get early access to new features, users will have to pay $10 a month.

MediaTek has formally announced its new flagship mobile chipset, the Dimensity 9400. It has the year-over-year spec bumps we’d expect to see, along with a few future-looking features just to cover all the bases.

The 9400 is built on a 3nm process and is “up to 40 percent more power-efficient” than its predecessor, the 9300. It comprises one Arm Cortex-X925 core running at 3.62GHz, along with three Arm Cortex-X4 and four Cortex-A720 cores, both of which were announced at last year’s Computex. MediaTek says that this combination results in 35 percent faster single-core performance and 28 percent faster multi-core performance compared to the 9300. The chipset also includes Arm’s new 12-core Immortalis-G925 GPU with 40 percent faster ray tracing.

That’s the basic stuff. On the more futuristic side, there’s MediaTek’s own eighth-generation NPU with support for training certain kinds of lightweight AI models on-device, with “80 percent faster large language model prompt performance.” It also supports AI video generation and provides a developer framework for creating agentic applications, which is AI that can actually do things for you. In theory, that’s the next big turn in AI, with everyone from Apple to Rabbit working out how to make it a reality.

In all likelihood, the Dimensity 9400 will be ready long before the most futuristic features it supports; MediaTek says that the 9400 will be available in the market in Q4 of this year. The company’s high-end chips tend to appear in flagship phones from Chinese OEMs like Vivo and Oppo. As such, the 9400 may not make it to the US, where Qualcomm chipsets dominate the popular Android flagships.