Linux

Debian sid a few years ago: Uninstalled Python2, system became unusable and couldn't neither reinstall from APT neither recompile it

Man, this was a few months back. I’ve got fedora asahi Linux (Linux on an ARM Mac) and I was trying to install Pycharm to play a bit with Python. Unfortunately, they did not have it packaged for arm, so I had to download a pre compiled tar or zip folder. I test it, see that it is an assortment of bin folders and alike, and decide to put it all elsewhere so it wouldn’t get lost. So I put it on the root and merge the folders. I think immediately “wait this is stupid” and decide to get Pycharm out of there. (I was on nautilus with root privileges), so i simply Ctrl-Z outa there. It shows a warning whether I wanted to delete 4000 files, but because I am an idiot, I didn’t realise what rhay meant. So I did it. I then continue on with my life, and find myself unable to open apps. I was fairly confused, as the apps I already had open still worked. I decide to try to restart the laptop. It is when I see that there is no restart button anymore that I realise what I did, and I just think to myself. I’ll be dammed if this survives a restart, im already screwed so it doesn’t matter. (It didn’t survive the reboot, had to install from scratch. At least an excuse to use the K desktop environment)

For me, it was a simple enabling of AUR im manjaro, twice Now I use arch, lol.

It was my first time using a Linux GUI. I was comfortable with CLI, but it was my first time having it installed on a laptop instead of just sshing into a server somewhere.

So naturally, instead of learning how the GUI worked, I tried changing it to be exactly like Windows. I was doing things like making it so I could double click shell scripts and other code files and they would run instead of opening them up in an editor. I think you see where this is going, but I sure as hell didn't.

Well, one of my coworkers comes over and asks me to run this code on this device we were developing. We were still in the very early stages of development, we didn't even have git set up, so he brought the code over on a USB stick. I pop it into my laptop. I went to check it once by opening it in an editor by double clicking on it... Only it ran the code that was written for our device on my laptop instead of opening in an editor.

To this day, I have no idea what it did to fuck my laptop so bad. I spent maybe an hour trying to figure out what was wrong, but I was so inexperienced with Linux, that I decided to just reinstall the OS. I had only installed it the day before anyway, so I wasn't losing much.

One time I rebooted. The system never recovered.

ran chown -R www-data: ./ from /var instead of /var/www.

Built a new desktop, backed up everything on my old laptop, next step was to format an Arch installer USB. Instead of formatting the USB, I formatted my laptop's /boot partition. No big loss since I had the backup and was done with that old toaster, but oops.

I've literally done the rm -rf / thing. I thought I was in a different subdirectory, but I was in / and did rm -rf .

When it didn't return after half a second, I looked at the command again and hit CTRL+C about 20 times in the span of 3 seconds.

I had to rebuild the install, but luckily didn't lose anything in /home.

Ubuntu GUI/apt fail

Back when I used ubuntu, Unity was stuck with old gnome packages. This meant that the version gnome-terminal packaged with ubuntu (up to at least 18.04) didn't have text reflow on window size changes.

You could add the upstream sources, upgrade the specific text reflow package only, and then disable the sources.

I forgot to disable the sources, or typed dist-upgrade (this happened multiple times...). Broke the whole desktop/lightdm setup with half upgraded packages, and half removed packages (for preparation to install new versions). Way easier to reinstall the os than to disentangle. Unity was a mess then anyway.

Moral: Actually read the package change summaries when doing updates/removes/installs, and [ y/N ] means actually check what the fuck you think you're agreeing to.

BtrFS snapshots for idiots

I've also run automated snapshots on my btrfs partition, then run out of space doing multi-hop system upgrade on fedora (dnf has a plugin that creates a snapshot every time it kicks in.

You can imagine there were many changes happenning per snapshot, and I effectively could have rolled back 4 major fedora versions... Til I ran out of space.

I couldn't get a replacement drive in time, and I had an hour to rebuild my laptop before needing to be on a customer site, so sadly I couldn't preserve my drive for later investigation. My best guess is the high-water-mark was configured incorrectly, and somehow it was able to 'write' data past the extents of the filesystem.

Rollback did work for my home partition, but I had to mount it from another OS to get it to work - so no data loss!

By that time I'd already reinstalled the os to the root partition/subvolume however, so I couldn't determine the exact cause of failure :(

Moral: Snapshots are not backups, and 'working' is not 'tested'

sudo usermod -a cdrom

Forgot the -G and wasn't sudo anymore...

I did recover eventually, but it was not nice.

I ran firejail config or something, which replaces a lot of home directory app files. Not sure if binaries or desktop entries.

But things broke, randomly, screenshots not working, not even inside firefox etc. I reinstalled the system and imported the home folder... and it was there again!

I once did an apt-get upgrade in the middle of when debian testing was recompiling all packages and moving to a new gcc version. I get it, using testing invites stuff like this. But come on, there should at least be a way to warn people beforehand.

I was trying to extract some files from a a Linux image of one of those ARM boards. It was packed into the cpio format, and I had never used the format before. Of course I was trying to extract to a root owned directory and I sudo'ed it. I effed up the command and overwrote all my system directories (/bin, /usr, /lib, etc...). Thankfully I had backed up my system recently and was able to get it working again.

I set up a progressive backup of my home folder... to my home folder. By the time I got home that day it was impossible to log in because there was no room to create a login record. Had to fix that by deleting the backup file using a live CD.

@fl42v I have thousands from my early days, but my only recent-ish one was pretty funny.

On an Arch install that hadn't been updated for a while, in a rush, had an app that needed OpenSSL 3. Instead of updating the whole system, I just updated the openssl package.

*Everything* broke immediately. Turns out a lot of stuff depends on openssl. Who knew?

To fix, booted to the arch installer, chrooted into my env, and reverted to the previous version of the package — then updated properly.

I was new to Linux, I made the not so calculated decision to use manjaro as my daily, deleted xorg to in an attempt to reinstall xorg to then hopefully fix the stuttering. Everything went wrong, no display obviously, /boot/ files where corrupt. I now use arch and am wiser

I had issues with a new version of glibc that prevented me from working on music in Ardour on Manjaro. I then proceeded to force-downgrade glibc (in the hopes of letting me get back to work) and that broke sudo and some other things, which I found out after rebooting. That was an interesting learning experience. Now I snapshot before I do stupid stuff. :]

It was only in a container on a Chromebook, but I'll share it anyway. One time, I had installed Android Studio but found it mildly annoying that I got a line when using apt about Android Studio and some error on a certain line of this one file. I believe the file was something related to dpkg, and after changing some things within the file, I seemed to have broken apt. Luckily, I had a backup, but it was a few days old, so I had to reinstall some apps.

Accidentally executed a JPEG (on an NTFS partition) and the shell started going crazy. reboot was not successful =[

sudo rm -r /run/timeshift

Once I succumbed to a proprietary software's allure, post-usage, I felt like a digital pariah! To rid myself of the taint, I wiped my system clean – reinstall time!

I cant remember anymore... Let me explain ... My first computer was with at-the-time-very-new windows xp, using primary for games, after some time it got bloated with stuff so i had to reinstall again and again over time. Then i discovered redhat,centos and debian... I started heavily distro hopping. My passion for software grew to the point that I was installing new software on daily basis, just to explore new things. But nothing seemed stable enough, ubuntu, fedora, sabayon, gentoo, arch... And their derivatives all broke under my fingers to the point that i had to do more fixing than discovering new software, I took it as a challenge and continue. At around the time of university I discovered NixOS, as with any new technology I went head on with it. It took a lot of trial and error since at the time there were no documentation for any of it. I spent months reading the code, but I never gave up, since what I have found was a gem. I found the OS that is resistant to my curiosity, I just cant seem to be able to break it. Now I use NixOS everywhere that I can, even on my work computer. I do not need to reinstall after initial installation. Well... only when hardware fails...

I've had the typical disasters with partition tables and boot loader mixups, but the one I keep coming back to is updating my Nvidia drivers too eagerly. Whether something gets messed up with an external monitor, or the laptop starts resisting switching away from the integrated GPU, or an electron app I use regularly that makes heavy use of 3D acceleration breaks, or I just need to bump the driver version in a reproducible system state record... it's just bad news.

I installed python one time

Accidentally deleted system Python, which on GNOME meant my DE was toast as well. Luckily very freshly set up, so no harm done.

Related note, add this in your shell profile:

bash
export PIP_REQUIRE_VIRTUALENV=true

proper scripting language
set PIP_REQUIRE_VIRTUALENV true

A regular update I guess...

Not really a "braking my linux setup", but still fun as hell! Back in university, a friend of mine got a new notebook at a time... we spent the night at the university hacking and they wanted to set the notebook up in the evening. They got to the point where they had to setup luks via the cryptsetup CLI. But they got stuck, it just wouldn't work. They tried for HOURS to debug why cryptsetup didn't let them setup LUKS on the drive.

At some point, in the middle of the night (literally something like 2 in the morning) they suddenly JUMPED from their seat and screamed "TYPE UPPERCASE 'YES' - FUCK!!!"

They debugged for about six hours and the conclusion was that cryptsetup asks "If you are sure you want to overwrite, type uppercase 'yes'". ... and they typed lowercase. For six hours. Literally.

The room was on the floor, holding their stomach laughing.

I used to work at this place that had a gigantic QNX install. I don't know if QNX that we used back then had any relation to q&x now They certainly don't look very close.

It was in the '90s and they had it set up so that particular nodes handled particular jobs. One node to handle boot images and serve as a net boot provider, one node handled all of the arcnet to ethernet communication, one node handled all the serial to mainframe, a number of the nodes were main worker nodes that collected data and operated machinery and diverters. All of these primary systems were on upper-end 386s or 486s ,they all had local hard disks.

The last class of node they called slave nodes. They were mainly designed for user data ingest, data scanning stations, touch screen terminals, simple things that weren't very high priority.

These nodes could have hard discs in them, and if they did, they would attempt to boot from them saving the net boot server a few cycles.

If for some reason they were unable to boot from their local hard drive, They would netboot format their local hard drive and rewrite their local file system.

If they were on able to rewrite their local file system they could still operate perfectly fine purely off the net boot. The Achilles heel of the system was that you had no idea that they had net booted unless you looked into the log files. If you boot it off your local hard drive of course your root file system would be on your local hard drive. If you had net booted, and it could not rebuild your local file system, your local root file / was actually the literal partition on the boot server. Because of the design of the network boot, nothing looked like it was remotely mounted.

SOP for problems on one of the slave nodes was to wipe the hard disk and reboot, in the process it would format the hard drive and either fix itself or show up as unreliable and you could then replace the disc or just leave the disc out of it. Of course If the local disk had failed and the box had already rebooted off netboot without a technician standing there to witness it, rm -Rf would wipe out the master boot node.

I wasn't the one that wiped it, but I fully understand why the guy did.

Turns out we were on a really old version of QNX, we were kind of a remote warehouse mostly automated. They just shut us down for about a week. Flew a team out. Rebuilt the system from newer software, and setup backups.