Slow r/w speeds on my nvme SSDs

kiddogdad · 20 November 2021 15:33

Hi all. I am running a custom build with two nvme SSDs. I recently used btrfs device add to combine them into a RAID 0, and it worked well. It was the first time I tried that and I was excited! So I decided to test my disk r/w speeds, and I got numbers that seem really low. For context, I'm using two 1-TB versions of these, which claim a Read/Write of 2,100/1,700 MB/s.

Here is the output of my two tests. I copied the commands from this tutorial.

fio --name TEST --eta-newline=5s --filename=temp.file --rw=read --size=2g --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting

TEST: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=1402MiB/s][r=1402 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=66948: Sat Nov 20 09:22:07 2021
read: IOPS=1354, BW=1355MiB/s (1421MB/s)(10.0GiB/7558msec)
slat (usec): min=336, max=1038, avg=424.96, stdev=46.54
clat (usec): min=1392, max=79319, avg=23079.51, stdev=8501.88
lat (usec): min=1814, max=79705, avg=23504.60, stdev=8498.10
clat percentiles (usec):
|  1.00th=[ 3064],  5.00th=[10421], 10.00th=[15270], 20.00th=[21103],
| 30.00th=[22152], 40.00th=[22938], 50.00th=[23462], 60.00th=[24249],
| 70.00th=[24773], 80.00th=[25297], 90.00th=[26084], 95.00th=[27395],
| 99.00th=[70779], 99.50th=[73925], 99.90th=[77071], 99.95th=[78119],
| 99.99th=[79168]
bw (  MiB/s): min= 1308, max= 1438, per=99.66%, avg=1350.27, stdev=47.69, samples=15
iops        : min= 1308, max= 1438, avg=1350.27, stdev=47.69, samples=15
lat (msec)   : 2=0.21%, 4=1.30%, 10=3.12%, 20=13.54%, 50=80.00%
lat (msec)   : 100=1.84%
cpu          : usr=0.25%, sys=55.31%, ctx=8312, majf=0, minf=537
IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=0.8%, 32=98.5%, >=64=0.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
READ: bw=1355MiB/s (1421MB/s), 1355MiB/s-1355MiB/s (1421MB/s-1421MB/s), io=10.0GiB (10.7GB), run=7558-7558msec

fio --name TEST --eta-newline=5s --filename=temp.file --rw=write --size=2g --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting

TEST: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [W(1)][15.6%][w=140MiB/s][w=140 IOPS][eta 00m:38s]
Jobs: 1 (f=1): [W(1)][24.5%][w=142MiB/s][w=142 IOPS][eta 00m:40s]
Jobs: 1 (f=1): [W(1)][33.3%][w=132MiB/s][w=132 IOPS][eta 00m:38s]
Jobs: 1 (f=1): [W(1)][41.7%][w=147MiB/s][w=147 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [W(1)][51.7%][w=197MiB/s][w=197 IOPS][eta 00m:29s]
Jobs: 1 (f=1): [W(1)][62.7%][w=218MiB/s][w=218 IOPS][eta 00m:22s]
Jobs: 1 (f=1): [W(1)][72.9%][w=133MiB/s][w=133 IOPS][eta 00m:16s]
Jobs: 1 (f=1): [W(1)][81.7%][w=120MiB/s][w=120 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [W(1)][91.7%][w=130MiB/s][w=130 IOPS][eta 00m:05s]
Jobs: 1 (f=1): [W(1)][100.0%][w=140MiB/s][w=140 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=67804: Sat Nov 20 09:23:14 2021
write: IOPS=164, BW=164MiB/s (172MB/s)(9894MiB/60227msec); 0 zone resets
slat (usec): min=64, max=231699, avg=186.54, stdev=2328.12
clat (msec): min=4, max=1307, avg=194.47, stdev=95.24
lat (msec): min=5, max=1308, avg=194.65, stdev=95.24
clat percentiles (msec):
|  1.00th=[   30],  5.00th=[   30], 10.00th=[  104], 20.00th=[  140],
| 30.00th=[  155], 40.00th=[  201], 50.00th=[  211], 60.00th=[  224],
| 70.00th=[  230], 80.00th=[  232], 90.00th=[  241], 95.00th=[  259],
| 99.00th=[  527], 99.50th=[  600], 99.90th=[ 1284], 99.95th=[ 1284],
| 99.99th=[ 1301]
bw (  KiB/s): min=124928, max=1015808, per=100.00%, avg=168328.53, stdev=88069.92, samples=120
iops        : min=  122, max=  992, avg=164.38, stdev=86.01, samples=120
lat (msec)   : 10=0.05%, 20=0.02%, 50=6.80%, 100=2.62%, 250=82.39%
lat (msec)   : 500=7.11%, 750=0.54%, 1000=0.14%, 2000=0.33%
cpu          : usr=0.32%, sys=2.42%, ctx=9913, majf=0, minf=12
IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=0.8%, 32=98.4%, >=64=0.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,9894,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
WRITE: bw=164MiB/s (172MB/s), 164MiB/s-164MiB/s (172MB/s-172MB/s), io=9894MiB (10.4GB), run=60227-60227msec

Here's my inxi.

inxi -Faz

System:    Kernel: 5.15.2-zen1-1-zen x86_64 bits: 64 compiler: gcc v: 11.1.0
parameters: BOOT_IMAGE=/@/boot/vmlinuz-linux-zen root=UUID=98598844-044f-41ac-a5a9-7b45c555414d rw
rootflags=subvol=@ quiet splash rd.udev.log_priority=3 vt.global_cursor_default=0
systemd.unified_cgroup_hierarchy=1 resume=UUID=23cc812f-a1d1-453e-85f0-01f210e1e021 loglevel=3
Desktop: KDE Plasma 5.23.3 tk: Qt 5.15.2 info: latte-dock wm: kwin_x11 vt: 1 dm: SDDM Distro: Garuda Linux
base: Arch Linux
Machine:   Type: Desktop System: ASUS product: N/A v: N/A serial: <filter>
Mobo: ASUSTeK model: TUF GAMING X570-PRO (WI-FI) v: Rev X.0x serial: <filter> UEFI: American Megatrends
v: 4021 date: 08/10/2021
CPU:       Info: 6-Core model: AMD Ryzen 5 5600X bits: 64 type: MT MCP arch: Zen 3 family: 19 (25) model-id: 21 (33)
stepping: 0 microcode: A201016 cache: L2: 3 MiB
flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm bogomips: 100803
Speed: 4199 MHz min/max: 2200/4200 MHz boost: enabled Core speeds (MHz): 1: 4199 2: 4232 3: 4200 4: 4202
5: 4160 6: 4200 7: 4197 8: 4200 9: 4194 10: 4200 11: 4198 12: 4199
Vulnerabilities: Type: itlb_multihit status: Not affected
Type: l1tf status: Not affected
Type: mds status: Not affected
Type: meltdown status: Not affected
Type: spec_store_bypass mitigation: Speculative Store Bypass disabled via prctl
Type: spectre_v1 mitigation: usercopy/swapgs barriers and __user pointer sanitization
Type: spectre_v2 mitigation: Full AMD retpoline, IBPB: conditional, IBRS_FW, STIBP: always-on, RSB filling
Type: srbds status: Not affected
Type: tsx_async_abort status: Not affected
Graphics:  Device-1: NVIDIA vendor: Micro-Star MSI driver: nvidia v: 495.44 alternate: nouveau,nvidia_drm
bus-ID: 0a:00.0 chip-ID: 10de:2487 class-ID: 0300
Device-2: Linux Foundation EEM Gadget type: USB driver: snd-usb-audio,uvcvideo bus-ID: 5-1:2
chip-ID: 1d6b:0102 class-ID: 0102
Display: x11 server: X.Org 1.21.1.1 compositor: kwin_x11 driver: loaded: nvidia unloaded: modesetting
alternate: fbdev,nouveau,nv,vesa display-ID: :0 screens: 1
Screen-1: 0 s-res: 3440x1440 s-dpi: 109 s-size: 802x333mm (31.6x13.1") s-diag: 868mm (34.2")
Monitor-1: DP-0 res: 3440x1440 dpi: 109 size: 800x330mm (31.5x13.0") diag: 865mm (34.1")
OpenGL: renderer: NVIDIA GeForce RTX 3060/PCIe/SSE2 v: 4.6.0 NVIDIA 495.44 direct render: Yes
Audio:     Device-1: NVIDIA GA104 High Definition Audio vendor: Micro-Star MSI driver: snd_hda_intel v: kernel
bus-ID: 0a:00.1 chip-ID: 10de:228b class-ID: 0403
Device-2: Advanced Micro Devices [AMD] Starship/Matisse HD Audio vendor: ASUSTeK driver: snd_hda_intel
v: kernel bus-ID: 0c:00.4 chip-ID: 1022:1487 class-ID: 0403
Device-3: Linux Foundation EEM Gadget type: USB driver: snd-usb-audio,uvcvideo bus-ID: 5-1:2
chip-ID: 1d6b:0102 class-ID: 0102
Sound Server-1: ALSA v: k5.15.2-zen1-1-zen running: yes
Sound Server-2: JACK v: 1.9.19 running: no
Sound Server-3: PulseAudio v: 15.0 running: no
Sound Server-4: PipeWire v: 0.3.40 running: yes
Network:   Device-1: Intel Wi-Fi 6 AX200 driver: iwlwifi v: kernel bus-ID: 05:00.0 chip-ID: 8086:2723 class-ID: 0280
IF: wlp5s0 state: down mac: <filter>
Device-2: Intel Ethernet I225-V vendor: ASUSTeK driver: igc v: kernel port: N/A bus-ID: 06:00.0
chip-ID: 8086:15f3 class-ID: 0200
IF: enp6s0 state: up speed: 1000 Mbps duplex: full mac: <filter>
Bluetooth: Device-1: Intel AX200 Bluetooth type: USB driver: btusb v: 0.8 bus-ID: 1-4:5 chip-ID: 8087:0029
class-ID: e001
Report: bt-adapter ID: hci0 rfk-id: 2 state: up address: <filter>
Drives:    Local Storage: total: 1.86 TiB used: 810.92 GiB (42.5%)
SMART Message: Unable to run smartctl. Root privileges required.
ID-1: /dev/nvme0n1 maj-min: 259:0 model: T-FORCE TM8FP8001T size: 953.87 GiB block-size: physical: 512 B
logical: 512 B speed: 31.6 Gb/s lanes: 4 type: SSD serial: <filter> rev: V9002s77 temp: 39.9 C scheme: GPT
ID-2: /dev/nvme1n1 maj-min: 259:1 model: T-FORCE TM8FP8001T size: 953.87 GiB block-size: physical: 512 B
logical: 512 B speed: 31.6 Gb/s lanes: 4 type: SSD serial: <filter> rev: V9002s77 temp: 39.9 C scheme: GPT
Partition: ID-1: / raw-size: 936.48 GiB size: 1.85 TiB (201.86%) used: 810.92 GiB (42.9%) fs: btrfs dev: /dev/nvme1n1p2
maj-min: 259:4
ID-2: /boot/efi raw-size: 300 MiB size: 299.4 MiB (99.80%) used: 576 KiB (0.2%) fs: vfat dev: /dev/nvme1n1p1
maj-min: 259:3
ID-3: /home raw-size: 936.48 GiB size: 1.85 TiB (201.86%) used: 810.92 GiB (42.9%) fs: btrfs
dev: /dev/nvme1n1p2 maj-min: 259:4
ID-4: /var/log raw-size: 936.48 GiB size: 1.85 TiB (201.86%) used: 810.92 GiB (42.9%) fs: btrfs
dev: /dev/nvme1n1p2 maj-min: 259:4
ID-5: /var/tmp raw-size: 936.48 GiB size: 1.85 TiB (201.86%) used: 810.92 GiB (42.9%) fs: btrfs
dev: /dev/nvme1n1p2 maj-min: 259:4
Swap:      Kernel: swappiness: 133 (default 60) cache-pressure: 100 (default)
ID-1: swap-1 type: partition size: 17.09 GiB used: 0 KiB (0.0%) priority: -2 dev: /dev/nvme1n1p3
maj-min: 259:5
ID-2: swap-2 type: zram size: 15.6 GiB used: 2.2 MiB (0.0%) priority: 100 dev: /dev/zram0
Sensors:   Message: No sensor data found. Is lm-sensors configured?
Info:      Processes: 337 Uptime: 31m wakeups: 0 Memory: 15.6 GiB used: 4.75 GiB (30.5%) Init: systemd v: 249
tool: systemctl Compilers: gcc: 11.1.0 Packages: pacman: 1855 lib: 504 Shell: fish v: 3.3.1 default: Bash
v: 5.1.8 running-in: konsole inxi: 3.3.08

Thanks in advance for any assistance, y'all.

mandog · 21 November 2021 09:42

I would read the comments on the Amazon page on this seems the quoted speeds could be exaggerated according to some users

tbg · 21 November 2021 17:41

You should check your logs for any I/O related errors. If you find any you might want to change I/O scheduler. You can do this yourself manually or switch to a kernel using a different scheduler. I have seen switching to an older kernel such as the LTS resolve some people's I/O issues.

Good luck.

anon26187667 · 22 November 2021 01:42

Not sure you'd want to be using RAID0 on that board with NVMe drives and a dGPU as the second drive will be connected through the chipset adding latency and sharing bandwidth with ... just about everything else on the board. As RAID0 is only as fast as the slowest drive, you're limiting yourself to that second drive's speed (which will be hampered further when using network and/or USB).

kiddogdad · 22 November 2021 02:04

Deleted and reposted because I somehow lost the quote of Chrispynut, which actually isn’t that important, but here we are.

Sorry, I’m not sure that I follow this. I understand the idea that RAID0 is only as fast as the slowest drive, but where I get lost is why one of my drives would be slower than the other since they’re both the same. Is the issue that the NVMe slots differ in how they connect, and one of those connections is slower?

anon26187667 · 22 November 2021 02:07

Yes, one is connected direct to CPU, the other is connected to Chipset which is itself connected to the CPU.
Block diagram: AMD X570 Chipset Blockdiagram Surfaces - Specs - PCIe 4.0 All The Way
Confirm within ASUS specs https://www.asus.com/uk/Motherboards-Components/Motherboards/TUF-Gaming/TUF-GAMING-X570-PRO-WI-FI/techspec/

kiddogdad · 22 November 2021 02:25

Interesting! Would you recommend that I keep them as separate drives in the system and just mount the secondary drive through fstab? That's what I had, but I decided to add it because I wanted to take advantage of btrfs.

anon26187667 · 22 November 2021 02:32

It's up to you. Can still use Btrfs on the drives whether RAID or separate.
RAID sounds great, but unless you're performing workloads that'll really make use of it, it's almost always going to create more trouble than it's worth. Given you got the cheapest of cheap NVMe drives, I doubt you're doing those kinds of workloads.

system · 6 December 2021 02:32

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.