this post was submitted on 23 Jan 2024
35 points (94.9% liked)

Linux

48090 readers
765 users here now

From Wikipedia, the free encyclopedia

Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).

Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.

Rules

Related Communities

Community icon by Alpár-Etele Méder, licensed under CC BY 3.0

founded 5 years ago
MODERATORS
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
35
Troubleshooting errorless system hang on kernel 6.7 (AMDGPU) (social.rocketsfall.net)
submitted 9 months ago* (last edited 9 months ago) by [email protected] to c/[email protected]
18 comments fedilink hide all child comments
 

Edit: See update at end of post for more details (no solutions, but things to consider.)

As of Linux 6.7 I'm getting hard freezes that require a power cut to reset (sysrq doesn't work.) Happens at both idle and load anywhere from 5 minutes in to an hour. Running journalctl --follow and dmesg -w (both as root) reveal nothing at the time of the crash. Kernel version 6.6 continues to be 100% stable.

System:

  • Distro/Kernel: Arch Linux 6.7.arch3-1
  • CPU: AMD Ryzen 5 2600X
  • GPU: AMD RX580 8GB via AMDGPU
  • RAM: Some configuration of 16GB at 2667 MT/s.
  • WM: SwayWM

I'm unsure how to go about properly reporting a bug if no errors are being generated.

Any advice?

I'm not alone on this apparently (warning, it's reddit.)

Update (01-27-2024)

I've spent the last 5 days or so bisecting the kernel from stable 6.6 to 6.7 while also touching on linux-next and 6.8rc1. I've experienced hangs on each kernel after 6.6 but under different conditions. In some cases sys-rq can rescue the system, but other times it's a hard (still errorless) crash. I believe all of these crashes can be blamed on AMDGPU given all other user reports (see: reddit thread above) mention having an AMD card.

List of similar issues

Note: Some of these are from earlier Kernel versions, but they're included since they present the same way.

Patched/Unpatched 6.8rc1 attempts

This bug report is the closest I can find on the issue. which links off to this report, which includes a patch. The patch in question is for 6.8rc1 and allows the system to stay up longer, but frequently "trips," meaning the system begins to stumble and halt for tenths of a second with accelerated video playback in my browser (qutebrowser) has to contend with load elsewhere on the system (gaming, etc.) Hangs under this patch and kernel can be rescued. However, with video acceleration disabled in the browser (and the browser not even running,) hard crashes can still occur. So either there's two new issues being brought into the fold (one to do with video accel. and the original issue mentioned in this post), or the original issue is just manifesting in different ways.

Bisecting 6.6 to 6.7

This process was taking forever because there's no reproducible situation in which the system halts. My method was to build the kernel (8 minutes) then reboot and let the system idle on a Sway session while I'm off doing something else (2 to 4 hours.) If I come back to a hard lock, then I mark the version as failed and repeat. This method let me try 2 versions of the kernel a day, not nearly enough to have this fixed quickly or easily. Due to the amount of time it takes to detect a crash, it's also possible to mark a bad commit as good (meaning it didn't crash in 2 hours, but would have in 4 or 5, etc.) I won't be continuing this.

The state of AMDGPU in general

It seems I'm lucky to have not had these issues before. A little bit of time reading issues like this show that people have been encountering problems for a while now (pre-kernel 6.7.) There are issues that have been open for years that still appear to be problems on modern systems and hardware. I'm not passing blame to anyone here, just stating that it's a miracle any of this stuff works at all given how complex the hardware is that even those who appear to be spending huge amounts of time dealing directly with it can't properly untangle what causes these faults.

top 18 comments
sorted by: hot top controversial new old
[–] [email protected] 8 points 9 months ago (1 children)

If you can't sysrq then you're down to bisecting kernel releases to find the patch that introduced the issue. You could also review for any new features that are enabled by default in 6.7

Have you upgraded all bios / fw versions?

[–] [email protected] 4 points 9 months ago (1 children)

I was afraid of that. Since I'm not the only one, maybe someone else is doing it already. But if it's still an issue in a few weeks, maybe I'll take it on as a weekend project. As for the motherboard, I believe the latest version is currently on it (2022 or 2023.)

[–] [email protected] 3 points 9 months ago* (last edited 9 months ago)

Also, can try moving the ssd to another pc to rule out hardware.

[–] [email protected] 7 points 9 months ago (1 children)

TLDR: do memtest on your RAM

I recently had an issue for quite some time where my computer would occasionally just hard crash. When it first started happening I tried many of the common tests including memcheck but found nothing. For a while it wasnt super common so I just lived through it. I thought it was an OS thing but it occurred on a different Linux distro and even on the ancient Windows 10 install I have but rarely use. I was just about to pull the trigger on replacing mobo and maybe even CPU+RAM. Before I did that I followed someone's suggestion to do a mem test. I could have at least sworn that I already did that and it came clean but it was an easy enough test to run, so why not.

Sure enough, found an error. I isolated the faulted DIMM, pulled it out and I haven't had a crash since. Crazy since I'm all but certain I did both memtest from a Linux live iso and the Windows memory checking utility.

In short, test your RAM. Do multiple passes. Maybe even just try swapping out single DIMMs and running on that for a reasonable ammount of time to see if you can isolate a culprit. It was my first thought when the issue first occurred because it's usually what causes stuff like that. When the tests came up clean originally I assumed it had to be something else. I was wrong.

[–] [email protected] 4 points 9 months ago

This is what I'll try next. I do think memory is the problem now that I've had a few more hours of research. Kernel 6.7 has issues with elevated RAM usage, so it's absolutely doing something funky with memory that might be exposing underlying hardware issues. I also realized my stable kernel was a version or two away from 6.6.13 (6.6.10), so I'm running it now to see if the issue was introduced late in the 6.6 release cycle, which would be easier to bisect than 6.7.

[–] [email protected] 5 points 9 months ago* (last edited 9 months ago) (1 children)

In the comments of the web link you shared (The link you wrote didn't work for me but I looked up the original and adding it here so that others can choose to use their preferred libreddit or teddit) at least three comments mention that 6.7 zen kernel works fine for them. Care to try that ?

[–] [email protected] 4 points 9 months ago

Fixed the link. Thanks!

I've also tried linux-tkg, which I believe rolls in the Zen patches. If it doesn't, I'll definitely try it.

[–] [email protected] 5 points 9 months ago

I recently had an issue with my computer freezing occasionally on a Deb12 Linux 6.1 where no errors showed up in syslog after a force reboot.

The way I finally found out about the issue was having dmesg open on a different monitor and waiting for the freeze to happen. Just before the freeze did happen, a number of error logs were spewed to dmesg - enough for me to catch a glimpse of the issue: intel WiFi.

I'm not saying that intel WiFi is your issue. I'm suggesting you keep dmesg -w open in another monitor (if you can) and go about your normal activity until a freeze happens.

[–] [email protected] 5 points 9 months ago (1 children)

Start eliminating hardware. Also try memtest86+.

[–] [email protected] 2 points 9 months ago

This will be my last resort mostly because I'm fairly certain it's a kernel issue, but yes, I've never ran an extended memtest on this build and should probably let it run overnight at some point just to make sure.

[–] [email protected] 3 points 9 months ago (1 children)

I've never done it, but I would try reproducing this in a VM like qemu.... I would be googling at this point but I think you can debug a kernel crash from there somehow.

[–] [email protected] 3 points 9 months ago (1 children)

That's an interesting idea. I'll have to look into whether it's a viable option first, though.

[–] [email protected] 4 points 9 months ago (1 children)

I did this recently and it was extremely quick to bisect and debug, but I was lucky enough to have a simple repro that worked in the emulator.

I think if I were you I'd try to repro on bleeding edge first. Then if it's still broken, I'd try to get the repro time down as much as possible and automate it. Then I'd either bisect on qemu if possible, or bare metal.

[–] [email protected] 2 points 9 months ago (1 children)

Yeah, the qemu idea was brought up earlier in the thread and it's very interesting. Glad you confirmed you could repro real issues there in the test environment, so it's at least a little likely I'll be able to do the same. Makes sense that it would work and is way better than letting the real system crash and burn. My kernel compile time is pretty short so it shouldn't be too bad to bisect, I'm just not sure how many commits separate my stable kernel from the bugged 6.7. TBH I'm not that familiar with kernel dev., so maybe it's way simpler than that.

[–] [email protected] 3 points 9 months ago (1 children)

The one I was able to test on qemu was a reliable failure of memory management syscalls triggered by a certain usage pattern. Unfortunately yours sounds like it's probably hardware dependent. People in that Reddit thread mentioned video decoding, so you could try hammering that.

The nice thing about bisecting is that it's mostly logarithmic, so doubling the commits should only take one extra step. I'd be surprised if you had to do more than a 10-12 steps.

You may already have a good kernel config, but for this sort of thing I usually use make localmodconfig. That'll build all the modules that are loaded when you run it, which can cut down on compile time massively.

[–] [email protected] 2 points 9 months ago (1 children)

I'm fresh off ruling out the RAM via memtest. I'll let it do a longer soak overnight to see if anything fails then, but I'm now on to bisecting the kernel from what I believe is the last release of 6.6 (6.6.13) to hopefully whatever the offending commit is. Been a while since I've had to mess around with manually building the kernel without the aid of linux-tkg, but I'm off to learn it anyway. Thanks for the help!

[–] [email protected] 2 points 9 months ago

Good luck! Sounds like you got it under control, but I'm happy to help if you run into trouble. I'm curious what you'll find.

[–] [email protected] 2 points 9 months ago

I used to have the same issue. Turns out, it was fixed by a firmware update on my motherboard.