High single core CPU usage due to large interrupts

For a long time (years), I've been plauged with low performance and crashes. Kernel 5.18 was the first to actually be fairly stable for me. I always noticed there was 80%+ usage on one CPU core going back though LTS kernel and on other distros (this is also likely why some distros like Fedora will just crash within minutes of booting).

At first I looked at htop but no processes (even kernel processes) were taking more than a few percent with total generally coming out to less than 10% when not doing anything yet a single CPU core was still running at more than 80% which didn't make sense. Looking online everyone said to look at the interrupts, so I did.

System:
  Kernel: 5.19.0-rc6-1-mainline arch: x86_64 bits: 64 compiler: gcc v: 12.1.0
    parameters: BOOT_IMAGE=/@/boot/vmlinuz-linux-mainline
    root=UUID=a55c4a54-0d20-415e-93d9-23fd6938b438 rw rootflags=subvol=@ splash
    rd.udev.log_priority=3 vt.global_cursor_default=0
    resume=UUID=f2faf51c-db74-457c-bb9d-eaf707594e6f loglevel=3 ibt=off
  Desktop: KDE Plasma v: 5.25.3 tk: Qt v: 5.15.5 wm: kwin_x11 vt: 1 dm: SDDM
    Distro: Garuda Linux base: Arch Linux
Machine:
  Type: Convertible System: ASUSTeK product: ZenBook UX564EH_Q528EH v: 1.0
    serial: <superuser required>
  Mobo: ASUSTeK model: UX564EH v: 1.0 serial: <superuser required>
    UEFI: American Megatrends LLC. v: UX564EH.312 date: 03/09/2022
Battery:
  ID-1: BAT0 charge: 72.9 Wh (80.6%) condition: 90.5/96.0 Wh (94.3%) volts: 11.7 min: 11.7
    model: ASUSTeK ASUS Battery type: Li-ion serial: N/A status: not charging cycles: 4
  Device-1: hid-0018:04F3:2C26.0001-battery model: ELAN9009:00 04F3:2C26 serial: N/A charge: N/A
    status: N/A
CPU:
  Info: model: 11th Gen Intel Core i7-1165G7 bits: 64 type: MT MCP arch: Tiger Lake built: 2020
    process: Intel 10nm family: 6 model-id: 0x8C (140) stepping: 1 microcode: 0xA4
  Topology: cpus: 1x cores: 4 tpc: 2 threads: 8 smt: enabled cache: L1: 320 KiB
    desc: d-4x48 KiB; i-4x32 KiB L2: 5 MiB desc: 4x1.2 MiB L3: 12 MiB desc: 1x12 MiB
  Speed (MHz): avg: 2814 high: 4686 min/max: 400/4700 scaling: driver: intel_pstate
    governor: powersave cores: 1: 4686 2: 3223 3: 1371 4: 2952 5: 4292 6: 400 7: 2792 8: 2800
    bogomips: 44864
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx
  Vulnerabilities:
  Type: itlb_multihit status: Not affected
  Type: l1tf status: Not affected
  Type: mds status: Not affected
  Type: meltdown status: Not affected
  Type: mmio_stale_data status: Not affected
  Type: spec_store_bypass mitigation: Speculative Store Bypass disabled via prctl
  Type: spectre_v1 mitigation: usercopy/swapgs barriers and __user pointer sanitization
  Type: spectre_v2 mitigation: Enhanced IBRS, IBPB: conditional, RSB filling
  Type: srbds status: Not affected
  Type: tsx_async_abort status: Not affected
Graphics:
  Device-1: Intel TigerLake-LP GT2 [Iris Xe Graphics] vendor: ASUSTeK driver: i915 v: kernel
    arch: Gen12.1 process: Intel 10nm built: 2020-21 ports: active: HDMI-A-1,eDP-1 empty: DP-1,DP-2
    bus-ID: 00:02.0 chip-ID: 8086:9a49 class-ID: 0300
  Device-2: NVIDIA TU117M vendor: ASUSTeK driver: nvidia v: 515.57 alternate: nouveau,nvidia_drm
    non-free: 515.xx+ status: current (as of 2022-06) arch: Turing process: TSMC 12nm built: 2018-22
    pcie: gen: 3 speed: 8 GT/s lanes: 4 link-max: lanes: 16 bus-ID: 58:00.0 chip-ID: 10de:1f99
    class-ID: 0302
  Device-3: IMC Networks USB2.0 HD UVC WebCam type: USB driver: uvcvideo bus-ID: 3-5:3
    chip-ID: 13d3:56eb class-ID: fe01 serial: <filter>
  Display: x11 server: X.Org v: 21.1.4 with: Xwayland v: 22.1.3 compositor: kwin_x11 driver: X:
    loaded: modesetting,nvidia gpu: i915 display-ID: :0 screens: 1
  Screen-1: 0 s-res: 3840x1080 s-dpi: 96 s-size: 1016x285mm (40.00x11.22")
    s-diag: 1055mm (41.54")
  Monitor-1: HDMI-A-1 mapped: HDMI-1 pos: right model: Philips PHL 246E9Q serial: <filter>
    built: 2022 res: 1920x1080 hz: 60 dpi: 93 gamma: 1.2 size: 527x296mm (20.75x11.65")
    diag: 604mm (23.8") ratio: 16:9 modes: max: 1920x1080 min: 720x400
  Monitor-2: eDP-1 pos: primary,left model: BOE Display 0x07d8 built: 2018 res: 1920x1080 hz: 60
    dpi: 142 gamma: 1.2 size: 344x194mm (13.54x7.64") diag: 395mm (15.5") ratio: 16:9
    modes: 1920x1080
  OpenGL: renderer: Mesa Intel Xe Graphics (TGL GT2) v: 4.6 Mesa 22.1.3 direct render: Yes
Audio:
  Device-1: Intel Tiger Lake-LP Smart Sound Audio vendor: ASUSTeK driver: sof-audio-pci-intel-tgl
    alternate: snd_hda_intel,snd_sof_pci_intel_tgl bus-ID: 3-2.4.1:6 bus-ID: 00:1f.3
    chip-ID: 046d:0a8f chip-ID: 8086:a0c8 class-ID: 0300 class-ID: 0401
  Device-2: Logitech H390 headset with microphone type: USB
    driver: hid-generic,snd-usb-audio,usbhid
  Sound Server-1: ALSA v: k5.19
.0-rc6-1-mainline running: yes
  Sound Server-2: sndio v: N/A running: no
  Sound Server-3: PulseAudio v: 16.1 running: no
  Sound Server-4: PipeWire v: 0.3.55 running: yes
Network:
  Device-1: Intel Wi-Fi 6 AX201 driver: iwlwifi v: kernel bus-ID: 00:14.3 chip-ID: 8086:a0f0
    class-ID: 0280
  IF: wlo1 state: down mac: <filter>
  Device-2: TP-Link UE300 10/100/1000 LAN (ethernet mode) [Realtek RTL8153] type: USB
    driver: r8152 bus-ID: 4-2.3:3 chip-ID: 2357:0601 class-ID: 0000 serial: <filter>
  IF: enp0s20f0u2u3 state: up speed: 1000 Mbps duplex: full mac: <filter>
Bluetooth:
  Device-1: Intel AX201 Bluetooth type: USB driver: btusb v: 0.8 bus-ID: 3-10:7 chip-ID: 8087:0026
    class-ID: e001
  Report: bt-adapter ID: hci0 rfk-id: 3 state: up address: <filter>
Drives:
  Local Storage: total: 504.19 GiB used: 91.51 GiB (18.1%)
  SMART Message: Unable to run smartctl. Root privileges required.
  ID-1: /dev/nvme0n1 maj-min: 259:0 vendor: Intel model: HBRPEKNX0202A size: 476.94 GiB
    block-size: physical: 512 B logical: 512 B speed: 15.8 Gb/s lanes: 2 type: SSD serial: <filter>
    rev: G002 temp: 35.9 C scheme: GPT
  ID-2: /dev/nvme1n1 maj-min: 259:4 vendor: Intel model: HBRPEKNX0202AO size: 27.25 GiB
    block-size: physical: 512 B logical: 512 B speed: 15.8 Gb/s lanes: 2 type: SSD serial: <filter>
    rev: K5110440 temp: 50.9 C scheme: GPT
Partition:
  ID-1: / raw-size: 459.78 GiB size: 459.78 GiB (100.00%) used: 91.5 GiB (19.9%) fs: btrfs
    dev: /dev/nvme0n1p2 maj-min: 259:2
  ID-2: /boot/efi raw-size: 300 MiB size: 299.4 MiB (99.80%) used: 576 KiB (0.2%) fs: vfat
    dev: /dev/nvme0n1p1 maj-min: 259:1
  ID-3: /home raw-size: 459.78 GiB size: 459.78 GiB (100.00%) used: 91.5 GiB (19.9%) fs: btrfs
    dev: /dev/nvme0n1p2 maj-min: 259:2
  ID-4: /var/log raw-size: 459.78 GiB size: 459.78 GiB (100.00%) used: 91.5 GiB (19.9%)
    fs: btrfs dev: /dev/nvme0n1p2 maj-min: 259:2
  ID-5: /var/tmp raw-size: 459.78 GiB size: 459.78 GiB (100.00%) used: 91.5 GiB (19.9%)
    fs: btrfs dev: /dev/nvme0n1p2 maj-min: 259:2
Swap:
  Kernel: swappiness: 133 (default 60) cache-pressure: 100 (default)
  ID-1: swap-1 type: zram size: 15.33 GiB used: 0 KiB (0.0%) priority: 100 dev: /dev/zram0
  ID-2: swap-2 type: partition size: 16.86 GiB used: 0 KiB (0.0%) priority: -2
    dev: /dev/nvme0n1p3 maj-min: 259:3
Sensors:
  System Temperatures: cpu: 60.0 C mobo: N/A
  Fan Speeds (RPM): cpu: 3200
Info:
  Processes: 288 Uptime: 3h 1m wakeups: 3882 Memory: 15.33 GiB used: 4.52 GiB (29.5%)
  Init: systemd v: 251 default: graphical tool: systemctl Compilers: gcc: 12.1.0 clang: 14.0.6
  Packages: pacman: 1621 lib: 360 Client: shell wrapper v: 5.1.16-release inxi: 3.3.19
Garuda (2.6.4-2):
  System install date:     2022-05-15
  Last full system update: 2022-07-13 ↻
  Is partially upgraded:   No
  Relevant software:       NetworkManager
  Windows dual boot:       No/Undetected
  Snapshots:               Snapper
  Failed units:            
File: /proc/interrupts
CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
1:          0          0          0          0       3214          0          0          0  IR-IO-APIC    1-edge      i8042
8:          0          0          0          0          0          0          0          0  IR-IO-APIC    8-edge      rtc0
9:          0         21          0          0          0          0          0          0  IR-IO-APIC    9-fasteoi   acpi
14:          0          0          0          0          0          0          0          0  IR-IO-APIC   14-fasteoi   INT34C5:00
16:          0          0        360          0          0          0          0          0  IR-IO-APIC   16-fasteoi   intel_ish_ipc, i801_smbus
27:          0          0          0          0          0       1302          0      58819  IR-IO-APIC   27-fasteoi   idma64.0, i2c_designware.0
28:  310868415          0         93          0          0          0          0     377636  IR-IO-APIC   28-fasteoi   idma64.1, i2c_designware.1
29:          0          0          0          0          0          0          0          0  IR-IO-APIC   29-fasteoi   idma64.2, i2c_designware.2
30:          0          0          0          0          0          0          0          0  IR-IO-APIC   30-fasteoi   idma64.3, i2c_designware.3
43:          0          0          0          0          0          0       3150          0  IR-IO-APIC   43-fasteoi   ELAN9009:00
57:          0      34765          0          0          0          0          0          0  IR-IO-APIC   57-fasteoi   ELAN1206:00
120:          0          0          0          0          0          0          0          0  DMAR-MSI    2-edge      dmar2
121:          0          0          0          0          0          0          0          0  DMAR-MSI    1-edge      dmar1
122:          0          0          0          0          0          0          0          0  DMAR-MSI    0-edge      dmar0
123:          0          0          0          0          0          0          0          0  DMAR-MSI    3-edge      dmar3
124:          0          0          0          0          0          0          0          0  IR-PCI-MSI 114688-edge      PCIe PME, pciehp
125:          0          0          0          0          0          0          0          0  IR-PCI-MSI 116736-edge      PCIe PME, pciehp
126:          0          0          0          0          0          0          0          0  IR-PCI-MSI 458752-edge      PCIe PME
127:          0          0          0          0          0          0          0          0  IR-PCI-MSI 466944-edge      PCIe PME
128:          0          0          0          0          0          0          0          0  IR-PCI-MSI 475136-edge      PCIe PME
129:          0          0          0          0          0          0          0          0  IR-PCI-MSI 479232-edge      PCIe PME
130:          0          0    4629683       1437       4380          0          0          0  IR-PCI-MSI 32768-edge      i915
131:          0          0          0          0          0          0          0          0  IR-PCI-MSI 212992-edge      xhci_hcd
132:          0    1840046          0          0        890          0      16281          0  IR-PCI-MSI 327680-edge      xhci_hcd
133:          0         19          8          0          0          0          0          0  IR-PCI-MSI 47185920-edge      nvme1q0
134:          0          0          0          8          0          1          0         21  IR-PCI-MSI 46661632-edge      nvme0q0
135:       9848          0          0          0          0          0          0          0  IR-PCI-MSI 46661633-edge      nvme0q1
136:          0      27554          0          0          0          0          0          0  IR-PCI-MSI 46661634-edge      nvme0q2
137:          0          0      31042          0          0          0          0          0  IR-PCI-MSI 46661635-edge      nvme0q3
138:          0          0          0      41896          0          0          0          0  IR-PCI-MSI 46661636-edge      nvme0q4
139:          0          0          0          0      30681          0          0          0  IR-PCI-MSI 46661637-edge      nvme0q5

You can see IRQ 28 has millions of interrupts. Watching it with 1/2 second interval it goes up by tens of thousands each refresh when I am doing nothing. I looked up that driver (i2c designware) and some are saying it has to do with the touchpad (I have it disabled in KDE as I don't normally use it). So I tested touching the touchpad and noticed IRQ 43 going up only when I touched it. When I touch the touchsreen, IRQ 27 goes up.

Anyone know how to figure out what IRQ 28 is and maybe how to get it to stop slamming one of the CPU cores?

Does hibernation or suspend play a role?

This thread seems similar in some ways: Touchpad Spams Interrupts : archlinux

The solution ended up being to blacklist that designware driver, which was slightly complicated by the fact that it is built into the kernel (not just a module). linux - How do I disable I2C Designware support when it's not built as a module? - Unix & Linux Stack Exchange

1 Like

I tried blacklisting the driver. It disabled the touchpad and touchscreen, but doesn't completely fix the problem. It now uses a single core at 50% (a bit over half what it was).

The interrupts are still happening quite often on 28:

File: /proc/interrupts
CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
1:          0          0          0          0        877          0          0          0  IR-IO-APIC    1-edge      i8042
8:          0          0          0          0          0          0          0          0  IR-IO-APIC    8-edge      rtc0
9:          0         21          0          0          0          0          0          0  IR-IO-APIC    9-fasteoi   acpi
14:          0          0          0          0          0          0          0          0  IR-IO-APIC   14-fasteoi   INT34C5:00
16:          0          0        207          0          0          0          0          0  IR-IO-APIC   16-fasteoi   intel_ish_ipc, i801_smbus
27:          0          0          0          0          0          0          0          0  IR-IO-APIC   27-fasteoi   idma64.0
28:   15037187          0         83          0          0          0     858211          0  IR-IO-APIC   28-fasteoi   idma64.1
29:          0          0          0          0          0          0          0          0  IR-IO-APIC   29-fasteoi   idma64.2
30:          0          0          0          0          0          0          0          0  IR-IO-APIC   30-fasteoi   idma64.3
120:          0          0          0          0          0          0          0          0  DMAR-MSI    2-edge      dmar2
121:          0          0          0          0          0          0          0          0  DMAR-MSI    1-edge      dmar1
122:          0          0          0          0          0          0          0          0  DMAR-MSI    0-edge      dmar0
123:          0          0          0          0          0          0          0          0  DMAR-MSI    3-edge      dmar3
124:          0          0          0          0          0          0          0          0  IR-PCI-MSI 114688-edge      PCIe PME, pciehp
125:          0          0          0          0          0          0          0          0  IR-PCI-MSI 116736-edge      PCIe PME, pciehp
126:          0          0          0          0          0          0          0          0  IR-PCI-MSI 458752-edge      PCIe PME
127:          0          0          0          0          0          0          0          0  IR-PCI-MSI 466944-edge      PCIe PME
128:          0          0          0          0          0          0          0          0  IR-PCI-MSI 475136-edge      PCIe PME
129:          0          0          0          0          0          0          0          0  IR-PCI-MSI 479232-edge      PCIe PME
130:          0          0          0       1254       4126      35335          0          0  IR-PCI-MSI 32768-edge      i915
131:          0          0          0          0          0          0          0          0  IR-PCI-MSI 212992-edge      xhci_hcd
132:        907          0          0      17266          0          0       8450          0  IR-PCI-MSI 327680-edge      xhci_hcd
133:          0         19          8          0          0          0          0          0  IR-PCI-MSI 47185920-edge      nvme1q0
134:          0          0          0          8          0          1          0         21  IR-PCI-MSI 46661632-edge      nvme0q0
135:       2340          0          0          0          0          0          0          0  IR-PCI-MSI 46661633-edge      nvme0q1
136:          0       5223          0          0          0          0          0          0  IR-PCI-MSI 46661634-edge      nvme0q2
137:          0          0       4132          0          0          0          0          0  IR-PCI-MSI 46661635-edge      nvme0q3
138:          0          0          0       3567          0          0          0          0  IR-PCI-MSI 46661636-edge      nvme0q4
139:          0          0          0          0       2883          0          0          0  IR-PCI-MSI 46661637-edge      nvme0q5
140:          0          0          0          0          0       4638          0          0  IR-PCI-MSI 46661638-edge      nvme0q6
141:          0          0          0          0          0          0       2435          0  IR-PCI-MSI 46661639-edge      nvme0q7
142:          0          0          0          0          0          0          0       2884  IR-PCI-MSI 46661640-edge      nvme0q8

I then did rmmod intel_lpss_pci and the interrupts stopped and CPU use is now down to low single digits. Granted, the touchpad and touch screen are no longer operable.

I am wondering what would happen if I were to re-enable the designware and disable IRQ 28 (is that even possible?). I don't know what 28 is that it causes so much of a problem.

I also found something strange... after re-enabling the i2c driver, when I place a finger on the touchpad (not pressed, just touching), the CPU usage drops to about 30%. But when I pull my finger away is goes back up above 80%.

Looking around the internet, it appears that older versions of this hardware required a special "knock" to get the hardware to stop spamming interrupts. I suppose they just haven't added whatever knock this newer hardware requires. Most posts are just trying to dismiss it as broken hardware, but the issue doesn't happen on Windows. The issue seems to be connected to both the touchpad and touchscreen (or at least a controller that handles both).

The other problem is that the knock doesn't appear to be documented, so they probably have to reverse-engineer the knock from what happens in Windows. :unamused: