Improving the capabilities of large language models (LLMs) in retrieving in-prompt information remains an area of active research that can impact important applications such as retrieval-augmented generation (RAG) and in-context learning (ICL).

Microsoft Research and Tsinghua University researchers have introduced Differential Transformer (Diff Transformer), a new LLM architecture that improves performance by amplifying attention to relevant context while filtering out noise. Their findings, published in a research paper, show that Diff Transformer outperforms the classic Transformer architecture in various settings.

Transformers and the “lost-in-the-middle” phenomenon

The Transformer architecture is the foundation of most modern LLMs. It uses an attention mechanism to weigh the importance of different parts of the input sequence when generating output. The attention mechanism employs the softmax function, which normalizes a vector of values into a probability distribution. In Transformers, the softmax function assigns attention scores to different tokens in the input sequence.

However, studies have shown that Transformers struggle to retrieve key information from long contexts.

“We began by investigating the so-called ‘lost-in-the-middle’ phenomenon,” Furu Wei, Partner Research Manager at Microsoft Research, told VentureBeat, referring to previous research findings that showed that LLMs “do not robustly make use of information in long input contexts” and that “performance significantly degrades when models must access relevant information in the middle of long contexts.”

Wei and his colleagues also observed that some LLM hallucinations, where the model produces incorrect outputs despite having relevant context information, correlate with spurious attention patterns.

“For example, large language models are easily distracted by context,” Wei said. “We analyzed the attention patterns and found that the Transformer attention tends to over-attend irrelevant context because of the softmax bottleneck.”

The softmax function used in Transformer’s attention mechanism tends to distribute attention scores across all tokens, even those that are not relevant to the task. This can cause the model to lose focus on the most important parts of the input, especially in long contexts.

“Previous studies indicate that the softmax attention has a bias to learn low-frequency signals because the softmax attention scores are restricted to positive values and have to be summed to 1,” Wei said. “The theoretical bottleneck renders [it] such that the classic Transformer cannot learn sparse attention distributions. In other words, the attention scores tend to flatten rather than focusing on relevant context.”

Differential Transformer

To address this limitation, the researchers developed the Diff Transformer, a new foundation architecture for LLMs. The core idea is to use a “differential attention” mechanism that cancels out noise and amplifies the attention given to the most relevant parts of the input.

The Transformer uses three vectors to compute attention: query, key, and value. The classic attention mechanism performs the softmax function on the entire query and key vectors.

The proposed differential attention works by partitioning the query and key vectors into two groups and computing two separate softmax attention maps. The difference between these two maps is then used as the attention score. This process eliminates common noise, encouraging the model to focus on information that is pertinent to the input.

The researchers compare their approach to noise-canceling headphones or differential amplifiers in electrical engineering, where the difference between two signals cancels out common-mode noise.

While Diff Transformer involves an additional subtraction operation compared to the classic Transformer, it maintains efficiency thanks to parallelization and optimization techniques.

“In the experimental setup, we matched the number of parameters and FLOPs with Transformers,” Wei said. “Because the basic operator is still softmax, it can also benefit from the widely used FlashAttention cuda kernels for acceleration.”

In retrospect, the method used in Diff Transformer seems like a simple and intuitive solution. Wei compares it to ResNet, a popular deep learning architecture that introduced “residual connections” to improve the training of very deep neural networks. Residual connections made a very simple change to the traditional architecture yet had a profound impact.

“In research, the key is to figure out ‘what is the right problem?’” Wei said. “Once we can ask the right question, the solution is often intuitive. Similar to ResNet, the residual connection is an addition, compared with the subtraction in Diff Transformer, so it wasn’t immediately apparent for researchers to propose the idea.”

Diff Transformer in action

The researchers evaluated Diff Transformer on various language modeling tasks, scaling it up in terms of model size (from 3 billion to 13 billion parameters), training tokens, and context length (up to 64,000 tokens).

Their experiments showed that Diff Transformer consistently outperforms the classic Transformer architecture across different benchmarks. A 3-billion-parameter Diff Transformer trained on 1 trillion tokens showed consistent improvements of several percentage points compared to similarly sized Transformer models.

Further experiments with different model sizes and training dataset sizes confirmed the scalability of Diff Transformer. Their findings suggest that in general, Diff Transformer requires only around 65% of the model size or training tokens needed by a classic Transformer to achieve comparable performance.

The researchers also found that Diff Transformer is particularly effective in using increasing context lengths. It showed significant improvements in key information retrieval, hallucination mitigation, and in-context learning.

While the initial results are promising, there’s still room for improvement. The research team is working on scaling Diff Transformer to larger model sizes and training datasets. They also plan to extend it to other modalities, including image, audio, video, and multimodal data.

The researchers have released the code for Diff Transformer, implemented with different attention and optimization mechanisms. They believe the architecture can help improve performance across various LLM applications.

“As the model can attend to relevant context more accurately, it is expected that these language models can better understand the context information with less in-context hallucinations,” Wei said. “For example, for the retrieval-augmented generation settings (such as Bing Chat, Perplexity, and customized models for specific domains or industries), the models can generate more accurate responses by conditioning on the retrieved documents.”

Zepto is in advanced stages of talks to raise $100 million in new investment, its third in the last six months, as the leading Indian quick commerce startup looks to rope in more domestic investors, sources familiar with the talks told TechCrunch.

The Mumbai-headquartered startup, which delivers grocery items and office stationery to customers’ doorsteps in 10 minutes in multiple Indian cities, is raising the new investment from Indian family offices and high net worth individuals.

Motilal Oswal, the asset management giant that earlier invested $40 million in Zepto, is running the mandate for the new funding deliberation, the sources said, requesting anonymity as the matter is private. The financial services firm has already received commitments for more than half of the allocation, according to another source familiar with the situation.

The new investment values Zepto at a $5 billion post-money valuation, the same value at which it recently closed a $340 million financing round in August. Zepto has raised more than $1 billion in the last six months and all of it remains in its bank.

Zepto is planning to go public next year and the new fundraise is aimed at expanding the base of domestic investors on its cap table. Zepto counts Avra, Lightspeed, Nexus, StepStone Group, YC Continuity, Glade Brook and Contrary among its backers.

Even as quick commerce startups are retreating, consolidating or shutting down in many parts of the world, the model is showing encouraging signs in India. Quick commerce startups are on track to do a sale of more than $6 billion this year, according to TechCrunch’s analysis.

In response to the fast rise of quick commerce, which is increasingly shaping the consumer behavior in India, many e-commerce incumbents — including Flipkart, Myntra and Nykaa have been forced to scramble ways to lower the time they take to deliver items to their customers.

Shares of Dmart, which runs one of the largest brick-and-mortar retail chains in India, fell this week after the firm confirmed that it was losing some business to quick commerce startups.

“We believe Quick Commerce players are expanding cities, categories, SKUs, AOVs and discounts, and creating parallel commerce for convenience-seeking customers,” analysts at Morgan Stanley wrote in a note this week.

Zepto – which competes with Zomato-owned BlinkIt, Prosus-backed Swiggy’s Instamart, and Tata’s BigBasket – has grown its annualized net runrate considerably in recent months, according to sources and an internal document reviewed by TechCrunch.

Zepto co-founder and chief executive Aadit Palicha told a group of investors in August that the startup projects to grow at 150% in the next 12 months, TechCrunch earlier reported.

DJI tells The Verge that it currently cannot freely import all of its drones into the United States — and that its latest consumer drone, the Air 3S, won’t currently be sold at retail as a result.

That’s not because the United States has suddenly banned DJI drones — rather, DJI believes the import restrictions are “part of a broader initiative by the Department of Homeland Security to scrutinize the origins of products, particularly in the case of Chinese-made drones,” according to DJI.

DJI recently sent a letter to distributors with one possible reason why DHS is stopping some of its drones: the company says US Customs and Border Protection is citing the Uyghur Forced Labor Prevention Act (UFLPA) as justification for blocking the imports. In the letter, which has been floating around drone sites and Reddit for several days, DJI claims it doesn’t use any forced labor to manufacture drones.

Reuters reported on the letter earlier today; DJI spokesperson Daisy Kong confirmed the letter’s legitimacy to The Verge as well.

In a just-published official blog post, DJI is calling this all a “misunderstanding,” and writes that it’s currently sending documentation to US Customs to prove that it doesn’t manufacture anything in the Xinjiang region of China where Uyghurs have been forcibly detained, that it complies with US law and international standards, and that US retailers have audited its supply chain. DJI claims it manufacturers all its products in Shenzhen or Malaysia.

US Customs and Border Protection didn’t reply to a request for comment.

While the US House of Representatives did pass a bill that would effectively ban DJI drones from being imported into the US, that ban would also need to pass the Senate. Last we checked, the Senate had removed the DJI ban from its version of the must-pass 2025 National Defense Authorization Act (though it did get reintroduced as an amendment and could potentially still make it into the final bill).

DJI says the “customs-related issue” has “primarily impacted” the company’s enterprise and agricultural drones, but has also now “limited us from offering the Air 3S to US customers beyond DJI.com.”

“We are actively working with U.S. Customs and Border Protection to resolve this issue and remain hopeful for a swift resolution,” writes DJI.

The US government has cracked down on DJI drones before, but not in a way that would keep stores from buying them, consumers from purchasing them, or individual pilots from flying them in the United States. Primarily, the US Department of Commerce’s “entity list” keeps US companies from exporting their technology to the Chinese company, and the US has sometimes restricted certain government entities from purchasing new DJI drones.

Even if DJI imports do get banned by Congress, the proposed law suggests existing owners could still use their drones — but the FCC could no longer authorize DJI gadgets with radios for use in the United States, which would effectively block all imports.

A whopping 69% of organizations have reported paying ransoms this year, according to research by Cohesity, with 46% handing over a quarter of a million dollars or more to cybercriminals. It is hardly the picture of resiliency that is often painted by industry. Clearly, there is a disconnect between cyber resiliency policy and operational capability that urgently needs addressing. 

With the advent of Ransomware-as-a-Service platforms and the current global geopolitical situation, organizations face a huge existential threat through destructive cyber attacks that could put them out of business. This gap between confidence and capability needs to be addressed, but in order to do so, those organizations need to recognize there is a problem in the first place.

Up to now, airfare search results on Google Flights have prioritized a combination of price and convenience over cost alone. This means you might be missing the cheapest options and spending more than you need to.

Via an update to its online tool rolling out gradually over the next couple of weeks, Google is finally making it easier to surface the best-priced airfares for your journey, though you may be sacrificing a bit of convenience if you opt for one of them.

“For example, there could be a third-party booking site offering a lower price than the airline itself,” Google explained in a blog post announcing the update. “Or you might be able to save by flying back to a different airport that’s in the same city you departed from — like flying out of New York’s LaGuardia and returning to JFK.”

The update puts a new “Cheapest” tab on Google Flights’ search page, while the most convenient options (according to Google Flights, at least) will appear beside it under the “Best” tab.

For sure, you might find some of the options under the “Cheapest” tab showing longer layovers, self-transfers, or a requirement to purchase different legs of the trip through multiple airlines or booking sites. But as Google points out in its post, “For those times when cost matters more than convenience, the new tab gives you an easy way to see the lowest prices available,” leaving you to decide which option works best for you.

Google has been gradually improving its Flights tool since launching it in 2011. In a more recent update, the tech giant added a feature that suggests the best time to book a cheap flight. Google Flights also offers price tracking, which, once set up, automatically alerts you if a fare for your route sees a notable fall.