On Thursday, Tesla CEO and noted fancy leaper Elon Musk will take the stage to showcase the company’s robotaxi plans, a la the “Cybercab.” Although the vehicle we see isn’t expected to be a fully functional product, investors and Tesla fans are hoping for a working prototype or other signs the company can navigate the technological and regulatory obstacles it will face. You can watch the event on an X livestream at 10PM ET.

With the Cybercab, Musk is aiming for a fully autonomous vehicle that runs on a Tesla ridesharing network. Owners will also reportedly be able to make their cars available on the network to run as autonomous cabs, likened to a “combination of Airbnb and Uber.”

The CEO began talking about the robotaxi plan years ago. However, it took on greater importance earlier this year when Tesla reportedly shelved its plans for an entry-level EV — often called “Model 2” — in favor of the autonomous ridesharing project.

Unlike competitors Waymo, Cruise and Zoox, Tesla’s current automation relies on cameras and AI. Reuters notes that Musk will aim to improve the tech rapidly enough to “crack” the highly regulated industry. No matter what is promised at the event, you may want to take the CEO’s promises with several grains of salt (if not Cybertruckloads). In April 2019, Musk said, “If you fast forward a year, maybe a year [and] three months, we’ll have over a million robotaxis on the road.”

Here we are in 2024 — which you may have noticed isn’t, in fact, 2020 — and Tesla doesn’t have a single robotaxi on the road.

Wired notes that the company doesn’t have an autonomous permit in California and reportedly hasn’t contacted the state’s AV regulators about testing. Waymo, Zoox, the beleaguered Cruise and Apple (no longer in the self-driving car business) have logged thousands of miles testing their self-driving vehicles in the Golden State. Waymo is the only company in the US currently operating robotaxis commercially.

You can tune in to the livestream to see what Musk and company have in store at 10PM ET / 7PM ET on Thursday.

Retrieval-augmented generation (RAG) has become a popular method for grounding large language models (LLMs) in external knowledge. RAG systems typically use an embedding model to encode documents in a knowledge corpus and select those that are most relevant to the user’s query.

However, standard retrieval methods often fail to account for context-specific details that can make a big difference in application-specific datasets. In a new paper, researchers at Cornell University introduce “contextual document embeddings,” a technique that improves the performance of embedding models by making them aware of the context in which documents are retrieved.

The limitations of bi-encoders

The most common approach for document retrieval in RAG is to use “bi-encoders,” where an embedding model creates a fixed representation of each document and stores it in a vector database. During inference, the embedding of the query is calculated and compared to the stored embeddings to find the most relevant documents.

Bi-encoders have become a popular choice for document retrieval in RAG systems due to their efficiency and scalability. However, bi-encoders often struggle with nuanced, application-specific datasets because they are trained on generic data. In fact, when it comes to specialized knowledge corpora, they can fall short of classic statistical methods such as BM25 in certain tasks.

“Our project started with the study of BM25, an old-school algorithm for text retrieval,” John (Jack) Morris, a doctoral student at Cornell Tech and co-author of the paper, told VentureBeat. “We performed a little analysis and saw that the more out-of-domain the dataset is, the more BM25 outperforms neural networks.”

BM25 achieves its flexibility by calculating the weight of each word in the context of the corpus it is indexing. For example, if a word appears in many documents in the knowledge corpus, its weight will be reduced, even if it is an important keyword in other contexts. This allows BM25 to adapt to the specific characteristics of different datasets.

“Traditional neural network-based dense retrieval models can’t do this because they just set weights once, based on the training data,” Morris said. “We tried to design an approach that could fix this.”

Contextual document embeddings

The Cornell researchers propose two complementary methods to improve the performance of bi-encoders by adding the notion of context to document embeddings.

“If you think about retrieval as a ‘competition’ between documents to see which is most relevant to a given search query, we use ‘context’ to inform the encoder about the other documents that will be in the competition,” Morris said.

The first method modifies the training process of the embedding model. The researchers use a technique that groups similar documents before training the embedding model. They then use contrastive learning to train the encoder on distinguishing documents within each cluster. 

Contrastive learning is an unsupervised technique where the model is trained to tell the difference between positive and negative examples. By being forced to distinguish between similar documents, the model becomes more sensitive to subtle differences that are important in specific contexts.

The second method modifies the architecture of the bi-encoder. The researchers augment the encoder with a mechanism that gives it access to the corpus during the embedding process. This allows the encoder to take into account the context of the document when generating its embedding.

The augmented architecture works in two stages. First, it calculates a shared embedding for the cluster to which the document belongs. Then, it combines this shared embedding with the document’s unique features to create a contextualized embedding.

This approach enables the model to capture both the general context of the document’s cluster and the specific details that make it unique. The output is still an embedding of the same size as a regular bi-encoder, so it does not require any changes to the retrieval process.

The impact of contextual document embeddings

The researchers evaluated their method on various benchmarks and found that it consistently outperformed standard bi-encoders of similar sizes, especially in out-of-domain settings where the training and test datasets are significantly different.

“Our model should be useful for any domain that’s materially different from the training data, and can be thought of as a cheap replacement for finetuning domain-specific embedding models,” Morris said.

The contextual embeddings can be used to improve the performance of RAG systems in different domains. For example, if all of your documents share a structure or context, a normal embedding model would waste space in its embeddings by storing this redundant structure or information. 

“Contextual embeddings, on the other hand, can see from the surrounding context that this shared information isn’t useful, and throw it away before deciding exactly what to store in the embedding,” Morris said.

The researchers have released a small version of their contextual document embedding model (cde-small-v1). It can be used as a drop-in replacement for popular open-source tools such as HuggingFace and SentenceTransformers to create custom embeddings for different applications.

Morris says that contextual embeddings are not limited to text-based models can be extended to other modalities, such as text-to-image architectures. There is also room to improve them with more advanced clustering algorithms and evaluate the effectiveness of the technique at larger scales.

Thoughtly co-founder Casey Mackrell had a big week. First, he got married. Then, he went viral.

At his wedding reception, Mackrell needed to quickly give a colleague access to code that could only be unblocked from his laptop. His fellow co-founder Torrey Leonard seized the moment by taking a photo to capture Mackrell wrapping up a pull request, staring at his computer in a ballroom as his friends and family danced in the background, floral arrangements and fairy lights abound.

Leonard posted the photo of his co-founder on LinkedIn with a reverent caption. The image of a founder coding at his own wedding went viral, sparking both awe and outrage.

Some commenters declared that they would never work with Thoughtly, since the image proved that the founders can’t delegate; others said that Mackrell’s wife should file for an annulment. But the actual story behind the image isn’t as bad as it seems.

“In this very moment that the picture was taken, Casey needed to push one thing to a server. There was a code on his laptop that a colleague needed to access,” Leonard told TechCrunch. “For 30 seconds, Casey was clicking a button: He logged in, clicked a button, done. And you can see in the picture, too, people are laughing.”

Leonard doesn’t provide the context that Mackrell was on his computer for less than one minute. But that’s what made his post so clicky: The idea of a founder spending hours coding at his own wedding is maddening. What actually happened isn’t as heinous.

Leonard’s post generated so much discussion in the startup community because it’s an extension of the existing discourse around “founder mode,” a concept coined by Paul Graham, a founding partner of Y Combinator. And beyond Silicon Valley, the post made for great rage bait.

“Last year, we would spend time in SF, and I’d be talking to friends at a restaurant or a bar, and there Casey would be on his laptop — and other people as well, because it’s SF, right?” said Leonard. “This founder mode mentality, it’s very inspiring to I think a lot of people in the tech space.”

But beyond the tech bubble, what founders view as dedication can be considered a lack of work-life balance.

According to Graham, you can run a company in one of two ways: in founder mode, or in manager mode. In founder mode, the founder should be hands-on with everything the company does. Founders transition into manager mode when they start delegating, which Graham argues can make a startup less successful.

Both Graham’s essay and Leonard’s LinkedIn post were met with mixed feedback. While some found Mackrell’s embodiment of “founder mode” to be motivational, others were appalled at this lack of work-life balance.

“Publicly, all of the comments that we’ve received are super negative … We were on 4chan, we were on Reddit, and obviously, people who represent communities outside of tech just, frankly, didn’t like it,” Leonard said.

Viral LinkedIn posts, which range from satirical to delusional, usually end up on other platforms, divorced from their context. One particularly successful post, in which a founder declares what proposing to his fiance taught him about b2b sales, was posted as a joke, though the post has blown up into a new meme in its own right.

Dozens of commenters said that Mackrell’s wife should immediately annul their marriage; other people said that they would avoid working with Thoughtly because they disagree with what working at a wedding represents.

“Meanwhile, I’ve received at this point thousands of emails, LinkedIn DMs, texts from founders that I know, unicorn founders that I don’t know, Fortune 500 CEOs, and the top investors in the world across Silicon Valley that have said, ‘Let’s go, I’m on your side,’” Leonard said.

Mackrell is now on his honeymoon with his wife, so he couldn’t be reached for comment. But his wife wasn’t bothered by her husband pulling out his laptop at their wedding, according to Leonard. Still, the company should probably figure out how to avoid a situation like this in the future, where only one person in a company of 15 employees can solve a particular problem. Paul Graham would probably disagree, though.

Amazon’s Prime Big Deal Days is well into its second day, and as was the case yesterday, it remains an excellent time to pick up good deals on great smart home gear. We scoured through deals on video doorbells, robot vacuums, smart lights, smart speakers, and more to gather up some of the best bargains that caught our eye. Check out a full list of all the deals in our main post, and read on for smart home deals. Just know the sale runs through the end of today, October 9th.

Update, October 9th: Adjusted pricing and added several new deals, including those for the Ring Stick Up Cam Battery and Eero Max 7 router.