Zyphra Technologies, the company working on a multimodal agent system combining advanced research in next-gen SSM hybrid architectures, long-term memory and reinforcement learning, just released Zyda-2, an open pretraining dataset comprising 5 trillion tokens. 

The offering comes as the successor of the original Zyda dataset. It is five times larger in size and covers a vast range of topics and domains to ensure a high level of diversity and quality – which is critical for training robust and competitive language models. 

But, that’s not the user profile of Zyda-2. There are many open datasets on Hugging Face for training cutting-edge AI models.

What makes this dataset unique is that it has been distilled to possess the strengths of the top existing datasets and eliminate their weaknesses.

This gives organizations a way to train language models that show high accuracy even when operating across edge and consumer devices on a given parameter budget.

The company trained its Zamba2 small language model using this dataset and found it to be performing much better than those trained with other state-of-the-art open-source language modeling datasets on HF.

What does Zyda-2 bring to the table?

Earlier this year, as part of the effort to build highly powerful small models that could automate a range of tasks cheaply, Zyphra went beyond model architecture research to start constructing a custom pretraining dataset by combining the best permissively licensed open datasets – often recognized as high-quality within the community.

The first release from this work, Zyda with 1.3 trillion tokens, debuted in June as a filtered and deduplicated mashup of existing premium open datasets, specifically RefinedWeb, Starcoder C4, Pile, Slimpajama, pe2so and arxiv. 

At the time, Zyda performed better than the datasets it was built upon, giving enterprises a strong open option for training. But, 1.3 trillion tokens was never going to be enough. The company needed to scale and push the benchmark of performance, which led it to set up a new data processing pipeline and develop Zyda-2.

At the core, Zyphra built on Zyda-1, further improving it with open-source tokens from DCLM, FineWeb-Edu and the Common-Crawl portion of Dolma v1.7. The original version of Zyda was created with the company’s own CPU-based processing pipeline, but for the latest version, they used Nvidia’s NeMo Curator, a GPU-accelerated data curation library. This helped them reduce the total cost of ownership by 2x and process the data 10x faster, going from three weeks to two days.

“We performed cross-deduplication between all datasets. We believe this increases quality per token since it removes duplicated documents from the dataset. Following on from that, we performed model-based quality filtering on Zyda-1 and Dolma-CC using NeMo Curator’s quality classifier, keeping only the ‘high-quality’ subset of these datasets,” Zpyphra wrote in a blog post.

The work created a perfect ensemble of datasets in the form of Zyda-2, leading to improved model performance. As Nvidia noted in a separate developer blog post, the new dataset combines the best elements of additional datasets used in the pipeline with many high-quality educational samples for logical reasoning and factual knowledge. Meanwhile, the Zyda-1 component provides more diversity and variety and excels at more linguistic and writing tasks. 

Distilled dataset leads to improved model performance

In an ablation study, training Zamba2-2.7B with Zyda-2 led to the highest aggregate evaluation score on leading benchmarks, including MMLU, Hellaswag, Piqa, Winogrande, Arc-Easy and Arc-Challenge. This shows model quality improves when training with the distilled dataset as compared to training with individual open datasets.

“While each component dataset has its own strengths and weaknesses, the combined Zyda-2 dataset can fill these gaps. The total training budget to obtain a given model quality is reduced compared to the naive combination of these datasets through the use of deduplication and aggressive filtering,” the Nvidia blog added.

Ultimately, the company hopes this work will pave the way for better quality small models, helping enterprises maximize quality and efficiency with specific memory and latency constraints, both for on-device and cloud deployments. 

Teams can already get started with the Zyda-2 dataset by downloading it directly from Hugging Face. It comes with an ODC-By license which enables users to train on or build off of Zyda-2 subject to the license agreements and terms of use of the original data sources.

TechCrunch Disrupt 2024 is just a week away. Our beefed-up Startup Battlefield 200 is a major highlight. Two hundred companies have been hand-selected by the TechCrunch editorial staff to grace the expo hall, 20 of which will launch their company for the first time live on our stage.

Nail-biting pitches live on the Disrupt Stage, make or break hardware and software demos. It’s where legends are made. Throughout it all, we’re incredibly grateful for the expertise of some of the top VCs in the world to tease out the cream of the crop. After an initial, thorough evaluation, five companies are selected to pitch once again in the Startup Battlefield Final on October 30. For this, we enlist the sharpest, most experienced venture capitalists we can find, all of which run a huge fund.

Learn more about these investors below and don’t miss the chance to learn from their expert insights and discover the crucial traits that lead to startup success, only at Disrupt 2024, taking place at Moscone West in San Francisco from October 28-30.

Navin Chaddha, Managing Partner, Mayfield Fund

Navin Chaddha, Managing Partner, leads the firm and drives its Cognition-As-A-Service (CaaS) and GenAI cognitive plumbing focus. This approach emphasizes investing in the infrastructure layers behind AI-driven companies and technologies and draws from his deep experience in previous paradigm shifts as founder of VXTreme /Microsoft Windows Media in the Web era and investor in infrastructure leader Hashicorp in the cloud era.

Under his leadership, Mayfield has raised eight U.S. funds and guided over 80 companies to positive outcomes. During his venture capital career, Navin has invested in over 60 companies, of which 18 have gone public (including Lyft, Poshmark and SolarCity) and 27 have been acquired.

Navin has been named a Young Global Leader by the World Economic Forum and has ranked on the Forbes Midas List of Top 100 Tech Investors fifteen times, including being named in the Top Five in 2020, 2022, and 2023. Navin’s investments have created over $120 billion in equity value and over 40,000 jobs.

An alumnus of Stanford and IIT Delhi, Navin is a philanthropist supporting various causes and a serial entrepreneur with notable startups like VXtreme and iBeam Broadcasting.

Chris Farmer, CEO, Partner, and Co-founder, SignalFire

Chris Farmer is the CEO, Partner, and co-founder of venture fund SignalFire, a $2.1 billion AI-native early-stage venture firm.

A decade ago, Farmer pioneered the use of data and AI would let VCs source better investments while scalably helping portfolio companies. So Farmer built Beacon, SignalFire’s in-house AI platform, with the help of Google veterans, Stanford AI PhDs, and feedback from 500 founders on what they really needed. Now Beacon crunches a half trillion data points to rank over 650 million people in the tech ecosystem on talent and hireability to assist portfolio companies with recruiting and go-to-market.

At SignalFire, Farmer has led investments in companies like Grammarly, Frame.io, ClassDojo, and Stampli. The firm writes lead checks from seed to Series B with a focus on sectors including health and life sciences tech, developer tools, cybersecurity, and vertical AI SAAS.

Previously, Farmer was a successful entrepreneur, spearheading the turnaround of wireless-enabled SaaS company Skybitz, preparing it for its acquisition by Telular Corporation. Farmer built one of the first technologies for startup investment sourcing while at Bessemer Venture Partners in 2007 to track the new Apple App Store. He then joined General Catalyst, where he led seed investing in iconic companies like Alation, Coinbase, Discord, Fivetran, Getaround, Segment, Stripe, and Zapier.

Most recently, Farmer launched SignalFire’s AI Lab to handpick high-potential founders and partner them with top corporations in their sector as design partners, pilot customers, and training data providers.

Dayna Grayson, Co-founder & General Partner, Construct Capital

Dayna Grayson is Co-founder and General Partner of Construct Capital, an early-stage venture firm that invests in extraordinary founders building technology to transform the most foundational industries of our economy from manufacturing to mobility. Dayna was one of the first venture capitalists to turn her attention to transforming these sectors of our economy through software-based models. She backed companies creating new advances in manufacturing, automation, and vertically integrated consumer brands. During her time as a partner at NEA (2012-2020), she was the lead investor from the earliest stages and was on the board of companies including Desktop Metal (2015 – today), Tulip (2017 – 2020), Onshape (acqd by PTC) (2013-2019), and Framebridge (acqd by Graham Holdings) (2014-2020) among others. She also led investments in Guideline, Formlabs, Evenly and Neuralmagic.

At Construct, along with her co-founder Rachel Holt, Dayna is exclusively focused on early stage investments and investing behind the accelerating changes in foundational industries that together make up half our economy’s GDP are failing to meet customer expectations. Some of their investments include Copia, Veho, Hadrian, The Rounds, and Verve Motion.

Dayna started her career in product development and led design efforts at Blackbaud [BLKB], the leading global provider of software to nonprofit organizations, as the company grew to over $130 million in revenue and completed a successful public offering and was an investor at North Bridge Venture Partners from 2007-2012.

She is a graduate of the University of Virginia (Systems Engineering) and Harvard Business School, where she serves as a venture partner today.

Ann Miura-Ko, Co-founding Partner, Floodgate

Known for her inception stage investments that range from marketplaces to highly technical companies, Ms. Miura-Ko’s investments have earned her repeat appearances on the Forbes Midas List and The New York Times list of top twenty venture capitalists worldwide. At Stanford University—where she received her Ph.D. for her work on mathematical modeling of cybersecurity—she is a lecturer in engineering, teaching on topics from blockchain to intelligent growth for startups. In addition, she is a co-director of the Stanford Mayfield Fellows Program, which helps train undergraduates to become technology leaders.

Ms. Miura-Ko is a co-founding member of AllRaise, an organization dedicated to increasing the success and prevalence of female funders and founders. Ms. Miura-Ko received a B.S. in electrical engineering from Yale. As an alumna, she has served on the School of Engineering & Applied Science Leadership Council and as a non-trustee member of the Corporation Committee on Investments. She was elected an alumni fellow in 2019.

Hans Tung, Managing Partner, Notable Capital

Hans Tung is a Managing Partner at Notable Capital and has invested in early-stage investments across the global digital economy. He has partnered with top founders from everywhere in e-commerce and fintech where the US is a key end-user market. Hans’ consumer internet portfolio has included notable companies like Airbnb, Coinbase, Ibotta, Peloton, Poshmark, Quince, StockX, and TikTok.

Don’t miss the battle at Disrupt 2024!

The Startup Battlefield winner, who will walk away with a $100,000 equity-free prize, will be announced at Disrupt 2024 — the startup epicenter. Join 10,000 attendees to witness this groundbreaking moment and see the next wave of tech innovation.

Register here and secure your seat to witness this epic startup battle. Bring a +1 and get the limited-time Expo+ 2 for 1 Pass for half the rate of one.

Instagram is launching several new features designed to protect teens from sextortion scams, which occur when scammers threaten to share intimate images of victims unless they receive a payment or more photos.

One guardrail that’s rolling out soon will prevent people from screenshotting or screen recording disappearing images or videos sent in a private message. If the sender enables replays of the image or video, Instagram will block people from opening them on the web. This won’t stop scammers from capturing the image or video by recording it with another device, however.

Starting today, Instagram will begin using certain indicators, like how new an account is, to detect scammy behavior as well. The platform will then prevent these accounts from sending follow requests to teens by blocking their request or moving it to the teen’s spam folder.

It’s also testing a safety notice in Instagram and Messenger that will alert teens if the person they’re talking to is located in a different country, as sextortion scammers often lie about their location.

In addition, Instagram will now start blocking suspicious accounts from viewing the following or followers lists of their victims, which sextortion scammers can use for blackmail. Instagram will similarly prevent suspect accounts from seeing the lists of accounts that have liked a target’s posts, the photos they’re tagged in, and other users tagged in their photos.

To protect kids from viewing obscene photos, Instagram is launching a feature that will automatically detect and blur nude images for users under 18. Instagram started testing this filter in April, and it will be enabled for teens globally by default. Other safety measures coming to the platform include an option to chat with the Crisis Text Line in the US if users report sextortion or child safety issues. It will also show an educational video to teens in the US, UK, Canada, and Australia to spread awareness about sextortion scams, which are on the rise.

This suite of features comes as part of Meta’s broader efforts to make its platforms safer for kids. Last month, Instagram announced that it would start putting all teens into more private accounts with certain safety settings enabled by default, like restricted DMs and Sleep Mode to silence notifications at nighttime.

Unity has announced that Unity 6, the latest version of its cross-platform game engine, is now available worldwide.

Unity 6 can be downloaded here, and is the company’s “most stable and best-performing version of Unity” yet that has been built, tested, and refined in partnership with game developers around the world.

Getty Images

What is sextortion?

‘Built-in protections’