Connect with us
DAPA Banner
DAPA Coin
DAPA
COIN PAYMENT ASSET
PRIVACY · BLOCKDAG · HOMOMORPHIC ENCRYPTION · RUST
ElGamal Encrypted MINE DAPA
🚫 GENESIS SOLD OUT
DAPAPAY COMING

Tech

AWS customer learns the hard way how even the smallest oversight can be mission-critical

Published

on

off-prem

An expired card, overzealous spam filter, and broken authenticator create havoc with a very clear moral 

Nothing can ruin the end of a week like finding out that all of the websites and hosted email you’re responsible for are offline, which is exactly what happened to Christopher Bradbury and his web design and development firm, Digital Takumi last week. AWS has since resolved the situation, but Bradbury’s mistakes can serve as a useful lesson to others of what not to do.

As Bradbury explained to The Register in an email, he noticed last Thursday, July 16, that all the websites and Google Workspace email accounts connected to domains he manages were offline. All of those sites are hosted through Route 53, AWS’ combined DNS/hosting service, and none of them were resolving. 

Advertisement

Bradbury went digging through his emails to figure out if there was anything to point to the failures, and sure enough: In his spam folder were numerous messages from AWS telling him a payment card on file had expired, and warning him that the account used to host all those customer websites through Route 53 was going to be suspended unless he took action. He didn’t obviously, because he didn’t realize there was an issue.

“AWS had been sending billing notifications, but unfortunately they had been filtered into a spam folder and, in some cases, were being delivered to an employee who had since left the business,” Bradbury told us. 

“I accept responsibility for missing those notifications,” he added, but that didn’t help his customers’ websites get back online. 

The situation could have been resolved sooner, but Bradbury had made some other mistakes as well:

Advertisement

“The root account had MFA enabled using a software authenticator that had been stored on an older laptop which has since suffered a motherboard failure,” Bradbury explained. Without access to that authentication code generator, he was unable to get into the account. Rather than getting the dead-laptop-with-a-critical-authenticator-on-it problem resolved, he was just relying on MFA emails instead – not the best idea.

“I’ve been bypassing the device key for quite a while by just using the recovery MFA via my email,” Bradbury told us. “It works, I get access. However if I just had my Passkey up to date it would have let me right in and I could have solved this.”

And then there was the email address where those MFA codes were going:

“The AWS recovery process required email verification using the registered root email address,” Bradbury told us. “That email address belonged to one of the domains whose DNS was hosted in the suspended AWS account, meaning I couldn’t receive the verification email.”

Advertisement

The account recovery tango

Bradbury’s next option was contacting AWS from a different email address, which didn’t go anywhere. He created another AWS account and purchased business support access on someone’s recommendation, but the support engineers he spoke to using that method wouldn’t discuss the other account until he verified he owned the one in question. 

“Over the following days I spoke with several AWS teams, including Billing and Account Recovery. I was transferred between teams multiple times, but nobody was able to complete the ownership verification or restore access to the account,” Bradbury told us.

He wanted to pay AWS, Bradbury told us, but it took a while for them to be able to take his money. “The practical consequence is that I cannot log into the AWS account, cannot receive email at the registered root address, cannot access the MFA device, cannot update the expired payment method, and therefore cannot pay the outstanding invoices from inside the account.” 

While we were working on this story, after speaking to both Amazon and Bradbury, he contacted us to say that access to the sites had been restored, and that he had logged in, paid the back invoices, updated his payment method, reset his MFA keys, and generally taken care of all the stuff he had been putting off until all this happened. 

Advertisement

It’s unfortunate that someone has to go through AWS billing hell to serve as an example to others, so let this be a warning to anyone else managing client websites through AWS.

“Firstly pay your AWS bills,” Bradbury said in an email, adding that anyone running an AWS hosting account also makes sure their emergency recovery email isn’t on the same domain as one of the sites they manage through that profile. That, and “stop using shortcuts” when it comes to MFA. 

“This isn’t a big infrastructure account, we run a single company marketing website and some domains through Route 53,” Bradbury explained. “It just shows that even the most modest instances of AWS can be absolutely business critical and you need to use proper practices and processes.” ®

Source link

Advertisement
Continue Reading
Click to comment

You must be logged in to post a comment Login

Leave a Reply

Tech

The Pickle Is In, the Squint Is Out: Here Are the Newest Approved Emoji

Published

on

It’s official: We’re getting a pickle emoji next year. On Tuesday, July 14, the Unicode Consortium announced the nine new emoji you’ll see on your device in 2027. One new emoji is the pickle, and that means the eggplant’s reign as the go-to emoji for male anatomy could be coming to an end.

Here are the nine new emoji you will see on your device next year. 

  • A cracking face
  • A leftward thumb sign gesture
  • A rightward thumb sign gesture
  • A monarch butterfly
  • A pickle
  • A lighthouse
  • A meteor
  • An eraser
  • A net with a handle

The Unicode Consortium originally proposed a squinty face emoji in 2025. However, the Unicode Emoji Standard & Research Working Group recommended changing that emoji to a cracking face in January.

Honestly, the cracking face emoji feels more relevant to today than the squinty face emoji. The squinty face read as suspicious while the cracking face emoji screams, “I’m putting on a brave face in these trying times.” If that’s not the most relatable sentiment these days, I don’t know what is. 

While the Unicode Consortium approved these emoji, companies like Apple and Samsung still need to design and then implement their versions of the emoji. Those companies usually add the emoji to devices as part of a software update in the spring, so you likely won’t see these emoji on your phone until 2027.

Advertisement

Before these emoji land on your device, the Unicode Consortium will begin fielding proposals for the next round of new emoji. Anyone can submit an idea for a new emoji, and the Unicode Consortium usually uploads proposed new emoji to a public document late in the year. For example, the upcoming emoji were originally accepted in October, 2025

For more on emoji, here’s how to decipher every emoji and the newest emoji on your iPhone and Android.

Source link

Advertisement
Continue Reading

Tech

PS Plus July 2026 Games Are Packed With Zombies, Aliens and Nostalgia

Published

on

It’s officially the second half of 2026, which is going to be all about Grand Theft Auto 6. Until then, Sony has a new selection of games coming to PS Plus in July that includes Avatar: Frontier of Pandora, Dying Light and a couple of classic PlayStation 2 games.

PS Plus, which is Sony’s version of Xbox Game Pass, offers a large, constantly expanding library of games. Subscribers can choose from the Essential, Extra and Premium tiers, each with unique perks and benefits. Starting at $11 a month, the plans give subscribers access to games and rewards, and each month, all subscribers can play a handful of new games at no additional charge. 

If you’re a PlayStation Plus Extra or Premium subscriber, you can grab these games starting July 21.


Avatar: Frontiers of Pandora

Advertisement

Avatar: Frontiers of Pandora came out in 2023, a year after Avatar: The Way of Water released in theaters. Developed by Massive Entertainment – a division of Ubisoft – the open-world action-adventure puts players in the role of a Na’vi raised by the Resources Development Administration, or RDA, before returning to defend Pandora. Set in the previously unseen Western Frontier, the game emphasizes exploration, aerial traversal on an ikran and combat using both traditional Na’vi weapons and human firearms. While its gameplay shares many similarities with the Far Cry series, its lush environments and faithful recreation of Pandora earned praise from critics.

Avatar: Frontiers of Pandora will be available for PS Plus Extra and Premium subscribers starting July 21.


Rise of the Ronin

Team Ninja went a different direction than its usual Ninja Gaiden style with 2024’s Rise of the Ronin. Set during Japan’s turbulent Bakumatsu period in the late 19th century, the open-world action RPG lets players shape the story by siding with competing political factions. Combat combines fast-paced swordplay with firearms and other period weapons with the challenging gameplay the studio is known for. Its blend of historical events, exploration and player-driven choices helped distinguish it from Team Ninja’s previous action games.

Advertisement

Rise of the Ronin is now available for PS Plus Extra and Premium subscribers.


Firefighting Simulator: Ignite

Firefighting Simulator: Ignite puts players in the role of a firefighter responding to emergencies across a large fictional city in the American Midwest. Whether playing solo with AI teammates or cooperatively with up to three friends, players battle dynamic fires, rescue civilians and use authentic firefighting equipment to contain increasingly dangerous blazes. Powered by Unreal Engine 5, the game features real-time fire, smoke and heat simulation designed to create a more realistic firefighting experience.

Firefighting Simulator: Ignite will be available for PS Plus Extra and Premium subscribers starting July 21.

Advertisement

Mighty Morphin Power Rangers: Rita’s Rewind

Mighty Morphin Power Rangers: Rita’s Rewind is a true nostalgia bomb of a game. The side-scrolling beat-’em-up reimagines the classic 1990s TV series with pixel art visuals, cooperative multiplayer and familiar villains, including a robotic version of Rita Repulsa. Players battle through stages inspired by the show before piloting Dinozords and the Megazord in 3D action sequences that break up the traditional brawler gameplay.

Mighty Morphin Power Rangers: Rita’s Rewind will be available for PS Plus Extra and Premium subscribers starting July 28.


Dying Light

Advertisement

It’s been more than a decade since Dying Light was released, but it’s still a title that gamers fawn over. Players explore the zombie-infested city of Harran using a fast-paced parkour system that makes traversing rooftops as important as combat. Scavenging for supplies, crafting weapons and surviving a dynamic day-night cycle keep the tension high as more dangerous infected emerge after dark. Its fluid movement and intense survival mechanics helped make it one of the most acclaimed zombie games of its generation.

Dying Light will be available for PS Plus Extra and Premium subscribers starting July 21.


Citizen Sleeper 2: Starward Vector

Citizen Sleeper 2: Starward Vector builds on its predecessor by sending players across a lawless star system as an escaped android searching for freedom. Rather than focusing on traditional combat, the game emphasizes dialogue, dice-based skill checks and resource management as players assemble a crew and take on dangerous contracts. Its choice-driven storytelling and tabletop-inspired mechanics make every decision feel meaningful, with multiple paths that shape how the story unfolds.

Advertisement

Citizen Sleeper 2: Starward Vector will be available for PS Plus Extra and Premium subscribers starting July 28.


Snow Bros. Wonderland

Snow Bros. Wonderland revives the long-running arcade franchise with a fresh 3D isometric look while staying true to its classic action-platforming roots. Players freeze enemies into snowballs before kicking them across each stage to defeat other foes and rack up combos. Cooperative play, colorful environments and larger-than-life boss battles help modernize the series without losing the charm that made the original games fan favorites.

Snow Bros. Wonderland will be available for PS Plus Extra and Premium subscribers starting July 28.

Advertisement

Psi-Ops: The Mindgate Conspiracy

Psi-Ops: The Mindgate Conspiracy was first released for the PS2 in 2004 and had a mix of interesting physics gameplay that was a big trend in gaming at the time. Players control Nick Scryer, a secret agent who combines traditional gunplay with psychic abilities like telekinesis, mind control and pyrokinesis to take down enemies. The game’s physics-based powers encouraged creative problem-solving and helped it stand out from other action shooters of its era, earning it a cult following years after its release.

Psi-Ops: The Mindgate Conspiracy will be available for PS Plus Premium subscribers starting July 21.


Indigo Prophecy

Advertisement

Before it made a name for itself with the games Heavy Rain and Detroit: Become Human, developer Quantic Dream first experimented with making the point-and-click adventure gameplay style a bit more action-oriented with Indigo Prophecy. The supernatural thriller follows several interconnected characters as they investigate a string of mysterious murders while uncovering a conspiracy that threatens humanity. Its cinematic presentation, branching narrative and quick-time events helped lay the foundation for the studio’s later interactive dramas.

Indigo Prophecy will be available for PS Plus Premium subscribers starting July 21.


PlayStation Game

PlayStation Plus subscriptions start at $10 a month

Advertisement

For more on PlayStation Plus, here’s what to know about the service. You can also check out other games on PlayStation Plus and games on Xbox Game Pass.

The Best Customizable Controllers and Gear for Adaptive Gaming

Source link

Continue Reading

Tech

Protesters confront Microsoft CSO over carbon goals and AI, disrupting climate event

Published

on

Melanie Nakagawa, Microsoft chief sustainability officer, left, speaking with GeekWire reporter Lisa Stiffler at a fireside chat at Seattle City Hall on July 17. (PNW Climate Week / Fer Sagastume Photo)

Microsoft Chief Sustainability Officer Melanie Nakagawa faced a barrage of pointed questions from the audience Friday during a session at the annual Pacific Northwest Climate Week in Seattle.

Protesters challenged Nakagawa through most of the 30-minute session held in a conference room at Seattle’s City Hall, calling out the company’s use of fossil fuel energy sources to power its AI data centers and challenging Microsoft’s commitment to climate goals set years ago.

As a reporter covering sustainability issues for GeekWire, I moderated the session. Many of the issues raised by the crowd were on my list of questions for Nakagawa. The disruptions also included chants from protesters seated among attendees, at times going beyond climate issues to condemn Microsoft’s technology deals with Israel.

Security guards ultimately ushered some protesters out of the space, while others remained. Interruptions from the audience continued for all but the final 10 minutes of the session.

The event capped off Pacific Northwest Climate Week, which included conversations around the city and region about climate change solutions, policies and innovations.

Advertisement

Microsoft has for many years been viewed as an environmental corporate leader, setting an ambitious goal in 2020 to become carbon negative within a decade. It created an internal carbon tax — one of the corporate world’s largest — that charges individual Microsoft divisions for emissions from sources like air travel to fund climate-friendly initiatives. The company is credited with helping create and sustain the carbon dioxide removal sector, among other roles.

But the rapid expansion of AI data centers and their huge energy demands are undercutting Microsoft’s standing. The company recently released its annual sustainability report, disclosing that its carbon footprint grew 25% last year, moving it further from its 2030 target.

Microsoft CSO Melanie Nakagawa, left, and GeekWire reporter Lisa Stiffler before a fireside chat was derailed by protesters. (PNW Climate Week / Fer Sagastume Photo)

One protester’s question was about a deal announced earlier this year in which Microsoft is partnering with Chevron to build a 2.7 gigawatt natural gas facility to power a data center campus in Texas. I asked Nakagawa how the company defends the agreement, and she pointed to the 4.7 gigawatts of renewable energy that Microsoft has supported in the state. I followed up by asking about the Redmond, Wash.-based company’s commitment to carbon dioxide removal (CDR) projects given recent reports about a pause on new deals.

Nakagawa was unable to answer before the crowd drowned her out with a call-and-response chant: “Microsoft, you can’t hide. We can see your dirty side.”

Another protester criticized the escalating pursuit of AI. “You’re selling us a product that we don’t even need, and we never should ask for,” he said. “No one wants AI. You’re destroying the climate with AI.”

Advertisement

I brought up legislation proposed earlier this year in Washington to mandate clean energy use and bring transparency to data center impacts in the state. Microsoft opposed and helped defeat the bill, though the company says it wants to work with lawmakers to pass rules next year. I asked what needed to change in the legislation for Microsoft to support it.

Nakagawa didn’t provide specifics, but noted that this year, for the first time, the company shared facility-level information in its annual report on electricity and water use for data centers worldwide.

“People want to know more about the data, and we believe you can have an honest and candid conversation with transparency and access to that information and data,” she said.

Given the obvious public concerns, I asked Nakagawa, “Do you really honestly believe that by 2030, the company can hit that carbon-negative goal?”

Advertisement

Nakagawa pointed to wide-ranging initiatives that are starting to help curb specific emissions, including investments to make Xbox devices lower carbon and financial support for the recent opening of a production plant in Moses Lake, Wash., for sustainable aviation fuel company Twelve.

“There are a couple areas where we’re seeing a lot of promising progress,” she said. “Look, this is going to be a hard target. We’ve not been at all shying away from the fact that this is a difficult goal.”

Source link

Advertisement
Continue Reading

Tech

Apple Music is the latest streaming service to get hit by a price rise

Published

on

Apple has quietly increased the price of Apple Music in the US, with individual subscriptions now costing $11.99 per month instead of $10.99.

The change also affects Family and Student plans. In addition, several Apple One bundles are impacted. The price rise comes almost four years after Apple last increased Apple Music subscription fees in October 2022.

The biggest jump is for the Apple Music Family plan, which now costs $19.99 per month, up from $16.99. The Student plan has also increased by $1, bringing the monthly fee to $6.99.

Apple has also adjusted pricing for two of its Apple One bundles. While the Individual plan remains unchanged, the Family tier now costs $27.95 per month, up by $2. Meanwhile, the Premier plan has increased to $39.95 per month. Apple has also confirmed similar price increases in Brazil.

Advertisement

In a statement to Music Business Worldwide, Apple said the latest changes are “the result of rising licensing costs,” suggesting that higher payments to rights holders and music labels are behind the increase.

Advertisement

The price hike means Apple Music now sits closer to many of its biggest streaming rivals. However, the service continues to distinguish itself with features such as Lossless Audio, Hi-Res Lossless, Spatial Audio with Dolby Atmos, live radio stations and a catalogue of more than 100 million songs.

For existing subscribers, the increase is relatively modest on the Individual plan, but families will notice a more significant jump. Moreover, an extra $3 each month adds up to $36 more per year. This makes it one of the largest increases Apple has introduced for the service.

Advertisement

Spotify, YouTube Music and other services have all introduced price increases in recent years. As a result, premium music subscriptions are becoming steadily more expensive across the board.

If you’re already subscribed, the updated pricing should appear on your next billing cycle. New customers signing up from today will pay the new rates immediately. Meanwhile, anyone considering Apple’s wider ecosystem may want to compare whether an Apple One bundle now offers better overall value than paying for Apple Music on its own.

Source link

Advertisement
Continue Reading

Tech

Echolocation For Drones | Hackaday

Published

on

Bats are remarkable creatures, able to fly at night or inside the confines of caves without light to guide their way. A team of researchers at Worcester Polytechnic Institute (WPI) has determined how to use low power ultrasonic sensors to guide drones in obscured environments.

While radar, lidar, and GPS are all great for navigation and sensing, they can run into issues when light is obscured or can take too much power to be practical for the limited battery life of a drone. The researchers found that a dual sonar array could be used to implement a much lower power sensing system for a drone that performs well in environments that would stymie a computer vision system.

A shield placed behind the array cuts down on the sound of the propellers that would otherwise drown out the signal, and further signal analysis via a neural net separates the echoes of objects in front of the drone from the background. The prototype could navigate in various simulated environments like forests, smoke, and snow. It looks like it even got a chance to go for a flight in the actual woods. All the code and hardware designs are Open Source, so have at it!

Advertisement

We’ve covered mosquito-inspired drone sensors before, and if you want to get into echolocation yourself, apparently humans can learn to do it too.

via Entertainment Engineering.

Advertisement

Source link

Continue Reading

Tech

Trump’s latest AI czar has already resigned

Published

on

Chris Fall, the director of the Center for AI Standards and Innovation (CAISI), has resigned, the agency confirmed to multiple news outlets.

He was appointed just three months ago after the last appointee, Collin Burns, left in less than a week, The Washington Post reported at the time. Burns was reportedly “pushed out” of the job in April because he previously worked for Anthropic and the Trump administration had been battling with the company, sources told the Post.

No reason was given for Fall’s departure. Prior to leading CAISI, Fall was the director of the Department of Energy’s Office of Science during the first Trump administration and had been the acting director of the DOE’s Advanced Research Projects Agency-Energy. He worked in the DOE’s Office of Naval Research (ONR) prior to that.

Before Burns and Fall, the agency was led by venture capitalist David Sacks, whose title at the time was White House AI and crypto czar. Sacks stepped down in March.

Advertisement

CAISI, which operates under the National Institute of Standards and Technology, is the primary organization for developing technical standards and testing methods for AI models as well as assessing cybersecurity risks. Yet it was not the agency at the center of the most recent model-risk brouhaha.

That occurred in June when the U.S. Commerce Department invoked an obscure export control directive that effectively forced Anthropic to pull its Mythos and Fable models from the market. The ban was lifted by the end of the month, when Secretary of Commerce Howard Lutnick said he was satisfied with Anthropic’s safety plans.

Earlier this month, the White House also signed an executive order for a new AI safety oversight program called “Gold Eagle” that creates a clearinghouse for cybersecurity vulnerability coordination. A host of federal organizations were named as part of the program, including the Commerce Department and Department of Homeland Security. But, as CNBC pointed out, CAISI was not among the federal organizations mentioned.

Meanwhile, after Anthropic’s models were freed from the ban, Google DeepMind CEO Demis Hassabis began calling for the creation of an independent, industry-run standards body to regulate frontier AI modeled after FINRA — the same sort of mission that CAISI was formed to tackle.

Advertisement

Fall’s resignation also follows this weekend’s handwringing over Chinese AI lab Moonshot’s new version of its open model Kimi, which performed competitively against flagship frontier models. The administration was weighing efforts to somehow ban Chinese open models, Axios reported. This sparked immediate debate and outrage over the weekend, including from Sacks, who argued that regulations shouldn’t be used as a protectionism strategy for U.S. proprietary AI labs.

While CAISI has released a few reports on the capabilities of Chinese open-weight models Z.ai’s GLM-5.2 and DeepSeek V4 Pro, it hasn’t talked much about its processes for testing. (Open weight means these models can be publicly downloaded and run locally, but its training code and datasets are not available). Since July 9, TechCrunch has sent multiple inquiries to both the DoC and NIST about how its LLM evaluations work and has not received a response.

When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.

Source link

Advertisement
Continue Reading

Tech

NYT Connections hints and answers for Tuesday, July 21 (game #1136)

Published

on

Looking for a different day?

A new NYT Connections puzzle appears at midnight each day for your time zone – which means that some people are always playing ‘today’s game’ while others are playing ‘yesterday’s’. If you’re looking for Monday’s puzzle instead then click here: NYT Connections hints and answers for Monday, July 20 (game #1135).

Good morning! Let’s play Connections, the NYT’s clever word game that challenges you to group answers in various categories. It can be tough, so read on if you need Connections hints.

What should you do once you’ve finished? Why, play some more word games of course. I’ve also got daily Strands hints and answers and Quordle hints and answers articles if you need help for those too, while Marc’s Wordle today page covers the original viral word game.

Advertisement

Source link

Continue Reading

Tech

Google Maps Killed The Restaurant Star

Published

on

We all know that Google and other big players pick and choose what information people see, but we sometimes overlook it outside of the search and social media space. [Lauren Leek] decided to take a look at how Google Maps picks winners and losers in the restaurant scene in London.

Building a machine learning model to determine a new restaurant recommendation (as one does), [Leek] uncovered interesting, and perhaps concerning, elements of how Google Maps ranks restaurants. Broken down by relevance, proximity, and prominence, many new restaurants face the issue of not drawing traffic without reviews and vice-versa causing a vicious cycle. Relevance and proximity are fairly straightforward, but what goes into “prominence?”

[Leek] found that “it is not just what people think of a place – it is how often people interact with it, talk about it, and already recognise it.” This leads to chains and high foot traffic areas awash in reviews while more out-of-the-way places find it more difficult to draw traffic. Some of this is expected and would be happening even when word of mouth was the primary way to find out where to eat, but as with many things, the algorithm amplifies this, along with the undisclosed paid placement of restaurants in Maps results.

While still in its infancy, [Leek] built a public dashboard where people can sort restaurants in the city. The machine learning algorithm is designed to identify places that are hidden gems that punch above their Google Maps weight and may make you look like the trendy one (if you live in London).

Advertisement

Zooming out further, [Leek] found larger clusters that revealed restaurant “diversity, in other words, is not just about taste. It is about where families settled, which high streets remained affordable long enough for a second generation to open businesses, and which parts of the city experienced displacement before culinary ecosystems could mature.”

If you want to step outside the algorithm mayhem, how about a good old-fashioned Web Ring? We’ve also addressed what’s an AI versus an algorithm, and Cory Doctorow advised us on how to reverse course on the current wave of enshittification.

Source link

Advertisement
Continue Reading

Tech

Adobe Firefly AI video editor review

Published

on

Why you can trust TechRadar


We spend hours testing every product or service we review, so you can be sure you’re buying the best. Find out more about how we test.

Adobe is a popular brand when it comes to all things creative, including AI content generation and editing. With its newly improved Firefly tool, Adobe has taken AI content generation a step further with realistic, real-time outputs.

In addition to being highly accurate, Firefly is also one of the most affordable tools on the market today. But is it the best AI video editor for your needs? Read on to find out. In this article, we’ve put Firefly to the test, examining its features, pricing, ease of use, and overall value for money.

Advertisement

Adobe Firefly: Plans and pricing

Source link

Advertisement
Continue Reading

Tech

Google Cloud outage shows it’s still hard to understand hyperscalers’ real resilience regimes

Published

on

off prem

Single datacenter and just three services taken down by ‘upstream’ power problem, while the rest of a zone and region kept humming

Google Cloud last week experienced an outage that analysts say demonstrates that not all promises of cloudy resilience are created equal.

Google’s incident report explained that three services – the VMware Engine (GCVE), NetApp Volumes, and Bare Metal Solutions (BMS) – experienced a 15-hour outage due to a cooling failure in its europe-west4-a zone.

Advertisement

The report includes the following detail: “The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure.”

The important detail there is that Google uses a discrete datacenter for those three services.

Another notable element of the incident report is the admission that “An electrical fault occurred on the utility grid upstream of the datacenter, disrupting the electrical distribution gear and cooling equipment.”

Google hasn’t explained how an upstream failure caused that disruption but did say it “proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment.”

Advertisement

Whenever your correspondent talks to hyperscalers or datacenter operators about how they ensure resilience, they tell me about their use of multiple redundant pieces of energy infrastructure, plus on-site generation capabilities that can keep a datacenter powered for days if necessary.

We’ve asked Google if it had generators or other energy sources at this site, and if so, why it nonetheless had to turn down workloads. We’ve not received a response at the time of writing. Google told us its incident analysis “is currently ongoing” and promised to follow up once it is available.

Hidden dependencies

We also asked Google if it advertises the fact that some of its services are tied to a single datacenter, a matter of interest because like other hyperscalers it divides its cloud into “regions” that typically comprise multiple “zones” spread across a city or other locale. Like its hyperscale peers, Google recommends placing workloads across different zones and regions to ensure resilience. Yet this incident shows some services can be tied to a single datacenter in a zone – and that those single datacenters can experience problems while the rest of the zone keeps working.

Analysts told The Register the outage shows organizations need to dig into clouds’ promises of resilience.

Advertisement

“The real issue is transparency: customers are generally told to use multiple zones and regions for resilience but are rarely given visibility into whether a particular managed service has a single-datacenter dependency within a zone,” said Biswajeet Mahapatra, principal analyst at Forrester. “As a result, many organizations assume the cloud abstraction provides more facility-level redundancy than may actually exist for specialized services.”

“The underlying architecture is not necessarily unusual,” he added. “AWS, Azure, and Google all operate services that rely on dedicated hardware, storage platforms, or tightly coupled infrastructure that may not be distributed across multiple facilities in the same way as core compute and storage services.”

Gartner Director Analyst Adrian Wong reminded The Register of the 2023 outage at Google Cloud’s europe-west9-a region, the cause of which was a water leak that Google said “originated in a non-Google portion of the facility.”

Google uses a tool called “Spanner” to replicate data across zones, but in the flooded zone Google’s Spanner configuration didn’t work once one building became unavailable.

Advertisement

“It is very hard to figure out how an individual region is architected,” Wong said. “Our customers are often surprised by that,” he added.

The incident report for last week’s outage includes an apology.

“We know how much you rely on Google Cloud, and we regret the impact on your productivity,” the document states, before promising a final incident report will detail “preventative actions.”

But as this incident shows, knowing how Google plans to avoid future incidents of this sort won’t arm customers with the knowledge to understand if those mitigations will address hidden design issues that can reduce resilience. ®

Advertisement

Source link

Continue Reading

Trending

Copyright © 2025