Technology

59594 readers

3376 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related content.
Be excellent to each another!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, to ask if your bot can be added please contact us.
Check for duplicates before posting, duplicates may be removed

Approved Bots

founded 1 year ago

MODERATORS

[email protected]

1191

Major IT outage affecting banks, airlines, media outlets across the world (www.abc.net.au)

submitted 4 months ago* (last edited 4 months ago) by [email protected] to c/[email protected]

564 comments fedilink hide all child comments

All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It's all very exciting, personally, as someone not responsible for fixing it.

Apparently caused by a bad CrowdStrike update.

Edit: now being told we (who almost all generally work from home) need to come into the office Monday as they can only apply the fix in-person. We'll see if that changes over the weekend...

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 40 points 4 months ago (4 children)

It's just amatuer hour across the board. Were they testing in production? no code review or even a peer review? they roll out for a Friday? It's like basic level start up company "here's what not to do" type shit that a junior dev fresh out of university would know. It's like "explain to the project manager with crayons why you shouldn't do this" type of shit.

It just boggles my mind that if you're rolling out an update to production that there was clearly no testing. There was no review of code cause experts are saying it was the result of poorly written code.

Regardless if you're low level security then apparently you can just boot into safe and rename the crowdstrike folder and that should fix it. higher level not so much cause you're likely on bitlocker which...yeah don't get me started no that bullshit.

regardless I called out of work today. no point. it's friday, generally nothing gets done on fridays (cause we know better) and especially today nothing is going to get done.

[–] [email protected] 11 points 4 months ago

explain to the project manager with crayons why you shouldn't do this

Can't; the project manager ate all the crayons

[–] [email protected] 3 points 4 months ago (2 children)

Why is it bad to do on a Friday? Based on your last paragraph, I would have thought Friday is probably the best week day to do it.

[–] [email protected] 21 points 4 months ago

Most companies, money included, try to roll out updates during the middle of start of a week. That way if there are issues the full team is available to address them.

[–] [email protected] 5 points 4 months ago (1 children)

Because if you roll out something to production on a friday whose there to fix it on the Saturday and Sunday if it breaks? Friday is the WORST day of the week to roll anything out. you roll out on Tuesday or Wednesday that way if something breaks you got people around to jump in and fix it.

[–] [email protected] 2 points 4 months ago

And hence the term read-only Friday.

[–] [email protected] 1 points 4 months ago (1 children)

Was it not possible for MS to design their safe mode to still “work” when Bitlocker was enabled? Seems strange.

[–] [email protected] 3 points 4 months ago

I'm not sure what you'd expect to be able to do in a safe mode with no disk access.

[–] [email protected] 1 points 4 months ago

rolling out an update to production that there was clearly no testing

Or someone selected "env2" instead of "env1" (#cattleNotPets names) and tested in prod by mistake.

Look, it's a gaffe and someone's fired. But it doesn't mean fuck ups are endemic.