Kernel panic! Error while loading shared libraries: libcrypto.so.3

[lum1nuss]

I'm not sure whether to put this in just issues & assistance or to put it in newbies, so if this isn't the right place, please feel free to move it

This issue began after updating yesterday, and a kernel panic happened upon shutdown. Afterwards, no matter what kernel I used (tried Zen, Hardened and LTS), the laptop wouldn't boot to the SDDM greeter - instead, it'd complain again about a missing libcrypto.so.3, and then go to a grey screen where 3 small white dots would show up, and it'd get stuck on that forever. Uh oh

Please excuse the Github Gist links - for some reason, PrivateBin refuses to save my pastes so
Upon shutdown, I was given this kernel panic message: libcrypto.so.3 kernel panic · GitHub
I also grabbed a log of what was happening during update, since some interesting errors were popping up: 6-11-22_sys_update · GitHub
It gets interesting at around line 353, which mention libcrypto.so.1.1 having the same error even though the one triggering a kernel panic is libcrypto.so.3.

So far, I've tried (without success):

• Installing lib32-libxcrypt-compat as per recommendations on StackExchange
• Looking into journalctl -b -2 and other numbers, but they don't seem to have anything related (will post if asked ). In case it's relevant, I ran journalctl after rolling back, not from a live USB
• Reinstalling all packages via Garuda Assistant - no dice, got the same kernel panic on shutdown due to the same missing file/directory
• Installing libxcrypt-compat and then update, but kernel panic still happens (I suspect this solution was offered for cases where it was a package dependency issue rather than 'oops can't boot')
• Running ldconfig and then refreshing initramfs with mkinitcpio -P linux-zen after the update. No effect

I also tried this:

sudo lsinitcpio /boot/initramfs-linux-zen.img | grep crypto 
opt/dell/dcc/libcrypto.so.3

Not sure if there should only be one occurrence?
(dcc is dell-command-configure, which I use to set the maximum battery charging level %)

Here's some other links from when I was searching around:

• kernel panic when try to shutdown / Kernel & Hardware / Arch Linux Forums
• [SOLED] Early kernel panic, another "cannot load libcrypto.so" problem / Kernel & Hardware / Arch Linux Forums
• [SOLVED]Boot failed, missing libssl.so.1.1 and libcrypto.so.1.1 / Newbie Corner / Arch Linux Forums (I avoid partial upgrades however, so ???)
• Kernel wont load after system update

A lot of other links I stumbled upon were more about the libcrypto files missing when running other things.

I've never had to deal with kernel-level issues before, so I'm not entirely sure where to start or what to investigate. Any help appreciated! ^^
For now, I've just rolled back to a btrfs snapshot from before the update. I'd like to figure out why this is happening though, since I've never had issues with system updates before this.

garuda-inxi:

System:
  Kernel: 6.0.6-zen1-1-zen arch: x86_64 bits: 64 compiler: gcc v: 12.2.0
    parameters: BOOT_IMAGE=/@/boot/vmlinuz-linux-zen root=UUID=c9f99daa-f49a-4d9f-a961-e0361c675eed
    rw rootflags=subvol=@ quiet quiet splash rd.udev.log_priority=3 vt.global_cursor_default=0
    resume=UUID=8d72c892-6d9c-4514-b2d6-398dc499ca0c loglevel=3
  Desktop: KDE Plasma v: 5.26.2 tk: Qt v: 5.15.7 info: latte-dock wm: kwin_x11 vt: 2 dm: SDDM
    Distro: Garuda Linux base: Arch Linux
Machine:
  Type: Laptop System: Dell product: XPS 15 9560 v: N/A serial: <superuser required> Chassis:
    type: 10 serial: <superuser required>
  Mobo: Dell model: 05FFDN v: A00 serial: <superuser required> UEFI: Dell v: 1.24.0
    date: 08/10/2021
Battery:
  ID-1: BAT0 charge: 65.7 Wh (79.4%) condition: 82.7/97.0 Wh (85.3%) volts: 11.9 min: 11.4
    model: LGC-LGC8.33 DELL 5XJ28 type: Li-ion serial: <filter> status: not charging
CPU:
  Info: model: Intel Core i7-7700HQ bits: 64 type: MT MCP arch: Kaby Lake gen: core 7 level: v3
    note: check built: 2018 process: Intel 14nm family: 6 model-id: 0x9E (158) stepping: 9
    microcode: 0xF0
  Topology: cpus: 1x cores: 4 tpc: 2 threads: 8 smt: enabled cache: L1: 256 KiB desc: d-4x32
    KiB; i-4x32 KiB L2: 1024 KiB desc: 4x256 KiB L3: 6 MiB desc: 1x6 MiB
  Speed (MHz): avg: 3419 high: 3567 min/max: 800/3800 scaling: driver: intel_pstate
    governor: powersave cores: 1: 3477 2: 3546 3: 3461 4: 3472 5: 2800 6: 3489 7: 3545 8: 3567
    bogomips: 44798
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx
  Vulnerabilities:
  Type: itlb_multihit status: KVM: VMX disabled
  Type: l1tf mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable
  Type: mds mitigation: Clear CPU buffers; SMT vulnerable
  Type: meltdown mitigation: PTI
  Type: mmio_stale_data mitigation: Clear CPU buffers; SMT vulnerable
  Type: retbleed mitigation: IBRS
  Type: spec_store_bypass mitigation: Speculative Store Bypass disabled via prctl
  Type: spectre_v1 mitigation: usercopy/swapgs barriers and __user pointer sanitization
  Type: spectre_v2 mitigation: IBRS, IBPB: conditional, RSB filling, PBRSB-eIBRS: Not affected
  Type: srbds mitigation: Microcode
  Type: tsx_async_abort status: Not affected
Graphics:
  Device-1: Intel HD Graphics 630 vendor: Dell driver: i915 v: kernel arch: Gen-9.5
    process: Intel 14nm built: 2016-20 ports: active: eDP-1 empty: DP-1, DP-2, HDMI-A-1, HDMI-A-2
    bus-ID: 00:02.0 chip-ID: 8086:591b class-ID: 0300
  Device-2: NVIDIA GP107M [GeForce GTX 1050 Mobile] vendor: Dell driver: nvidia v: 520.56.06
    alternate: nouveau,nvidia_drm non-free: 515.xx+ status: current (as of 2022-10) arch: Pascal
    code: GP10x process: TSMC 16nm built: 2016-21 pcie: gen: 1 speed: 2.5 GT/s lanes: 16 link-max:
    gen: 3 speed: 8 GT/s bus-ID: 01:00.0 chip-ID: 10de:1c8d class-ID: 0302
  Device-3: Sunplus Innovation Integrated_Webcam_HD type: USB driver: uvcvideo bus-ID: 1-12:7
    chip-ID: 1bcf:2b95 class-ID: 0e02
  Display: x11 server: X.Org v: 21.1.4 with: Xwayland v: 22.1.5 compositor: kwin_x11 driver: X:
    loaded: modesetting,nvidia unloaded: nouveau alternate: fbdev,intel,nv,vesa dri: iris gpu: i915
    display-ID: :0 screens: 1
  Screen-1: 0 s-res: 3840x2160 s-dpi: 168 s-size: 580x326mm (22.83x12.83")
    s-diag: 665mm (26.19")
  Monitor-1: eDP-1 model: Sharp 0x1476 built: 2016 res: 3840x2160 hz: 60 dpi: 282 gamma: 1.2
    size: 346x194mm (13.62x7.64") diag: 397mm (15.6") ratio: 16:9 modes: 3840x2160
  OpenGL: renderer: Mesa Intel HD Graphics 630 (KBL GT2) v: 4.6 Mesa 22.2.1 direct render: Yes
Audio:
  Device-1: Intel CM238 HD Audio vendor: Dell driver: snd_hda_intel v: kernel bus-ID: 00:1f.3
    chip-ID: 8086:a171 class-ID: 0403
  Sound API: ALSA v: k6.0.6-zen1-1-zen running: yes
  Sound Server-1: PulseAudio v: 16.1 running: no
  Sound Server-2: PipeWire v: 0.3.59 running: yes
Network:
  Device-1: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter vendor: Rivet Networks
    Killer Wireless-n/a/ac 1535 driver: ath10k_pci v: kernel pcie: gen: 1 speed: 2.5 GT/s lanes: 1
    bus-ID:
 02:00.0 chip-ID: 168c:003e class-ID: 0280 temp: 38.0 C
  IF: wlp2s0 state: up mac: <filter>
Bluetooth:
  Device-1: Qualcomm Atheros QCA61x4 Bluetooth 4.0 type: USB driver: btusb v: 0.8 bus-ID: 1-4:4
    chip-ID: 0cf3:e300 class-ID: e001
  Report: bt-adapter note: tool can't run ID: hci0 rfk-id: 0 state: down bt-service: disabled
    rfk-block: hardware: no software: yes address: N/A
Drives:
  Local Storage: total: 476.94 GiB used: 589.07 GiB (123.5%)
  SMART Message: Unable to run smartctl. Root privileges required.
  ID-1: /dev/nvme0n1 maj-min: 259:0 vendor: Toshiba model: KXG50ZNV512G NVMe 512GB
    size: 476.94 GiB block-size: physical: 512 B logical: 512 B speed: 31.6 Gb/s lanes: 4 type: SSD
    serial: <filter> rev: AADA4106 temp: 30.9 C scheme: GPT
Partition:
  ID-1: / raw-size: 459.61 GiB size: 459.61 GiB (100.00%) used: 151.02 GiB (32.9%) fs: btrfs
    dev: /dev/nvme0n1p2 maj-min: 259:2
  ID-2: /boot/efi raw-size: 300 MiB size: 299.4 MiB (99.80%) used: 10.2 MiB (3.4%) fs: vfat
    dev: /dev/nvme0n1p1 maj-min: 259:1
  ID-3: /home raw-size: 459.61 GiB size: 459.61 GiB (100.00%) used: 151.02 GiB (32.9%) fs: btrfs
    dev: /dev/nvme0n1p2 maj-min: 259:2
  ID-4: /var/log raw-size: 459.61 GiB size: 459.61 GiB (100.00%) used: 151.02 GiB (32.9%)
    fs: btrfs dev: /dev/nvme0n1p2 maj-min: 259:2
  ID-5: /var/tmp raw-size: 459.61 GiB size: 459.61 GiB (100.00%) used: 151.02 GiB (32.9%)
    fs: btrfs dev: /dev/nvme0n1p2 maj-min: 259:2
Swap:
  Kernel: swappiness: 133 (default 60) cache-pressure: 100 (default)
  ID-1: swap-1 type: zram size: 15.48 GiB used: 0 KiB (0.0%) priority: 100 dev: /dev/zram0
  ID-2: swap-2 type: partition size: 17.03 GiB used: 0 KiB (0.0%) priority: -2
    dev: /dev/nvme0n1p3 maj-min: 259:3
Sensors:
  System Temperatures: cpu: 48.0 C pch: 45.0 C mobo: 41.0 C
  Fan Speeds (RPM): cpu: 0 fan-2: 0
Info:
  Processes: 263 Uptime: 10m wakeups: 1 Memory: 15.48 GiB used: 2.86 GiB (18.5%) Init: systemd
  v: 251 default: graphical tool: systemctl Compilers: gcc: 12.2.0 Packages: pm: pacman pkgs: 1781
  libs: 498 tools: octopi,pamac,paru Client: shell wrapper v: 5.1.16-release inxi: 3.3.22
Garuda (2.6.9-1):
  System install date:     2022-10-19
  Last full system update: 2022-11-05
  Is partially upgraded:   No
  Relevant software:       NetworkManager
  Windows dual boot:       Probably (Run as root to verify)
  Snapshots:               Snapper
  Failed units:            intel-undervolt.service systemd-vconsole-setup.service 

(I wanted to get rid of intel-undervolt after the update since Plundervolt had undervolting locked anyways, but after reverting, I haven't done it ^^; lol)

[filo]

This is not necessarily helping solve your issue, but I noticed you have several AUR packages that might need to be rebuilt due to updated depending shared libraries.
Maybe you could try a

paru --rebuild

After this, I'd try updating again.
Remember to always use

garuda-update -a

Or use the relevant configuration, to update also your AUR packages.

[filo]

Remove that dcc until it gets fixed, I'd say:
https://bbs.archlinux.org/viewtopic.php?id=280992
See also comments in the AUR
https://aur.archlinux.org/packages/dell-command-configure

[lum1nuss]

Lol. classic Dell causing issues :')
I've rebuilt all my AUR packages and am now trying an update. I actually didn't know there was an -a flag, so thank you for mentioning that!
If this is all it is, I'm gonna be so happy lol. This was definitely one heck of a learning experience for sure!

[lum1nuss]

Yep, this was it. After uninstalling it and finishing the system update, I rebooted the laptop without issues.
TIL the effects of having another libcrypto file hanging around, and how to figure out which one is in use
Will be keeping an eye on that repo in the meantime. Thanks again, @filo !

Sometimes the smallest things cause the biggest problems huh

[filo]

With rolling releases these things happen, and even more with AUR packages, where in the end you are the maintainer…
You look smart, some interesting reading as final takeover

An AUR package stopped working after an update

The normal reasons Generally, there will be two reasons why an AUR package has “stopped working”: A repo package was updated and the AUR package was built against an older (and incompatible) version of that repo package; The AUR package was updated and requires a repo package version not present on your system. The solutions to these: Rebuild the AUR package. Update your system fully. The other reasons The other, less frequent reasons, are The package file (PKGBUILD) has an issue and nee…

Garuda Update

https://wiki.archlinux.org/title/Pacman/Rosetta
(to learn searching eg to which package a file belongs to)
But none of these could have avoided the problem in this case…

[lum1nuss]

Ahh icic
Thanks for the links! Lol there is no smart here, only pea brain how did I miss the garuda-update wiki page HAHAHA
Lol this time the package made my system stop working after an update and not quite the other way around, but now I know what to look for (⁠^ - ⁠^ )⁠ゞ