在VMware ESXi上对三星PCIe 4.0 NVMe SSD 980 PRO硬盘进行直通DirectPath和虚拟化Virtualization的读写测试
存储虚拟化后，读性能损失约21.5%，写性能损失约18.4%
Read and Write Tests of Samsung PCIe 4.0 NVMe SSD 980 PRO Hard Drive with DirectPath and Virtualization on VMware ESXi
| 物理服务器 | NF5280M5 |
|---|---|
| 硬盘 | PCIEx4转NMVe |
| 虚拟化软件 | ESXi 7.03 |
| 虚拟机配置 | 8核心 16G内存 120G系统盘 |
| 操作系统 | CentOS7.9 |
| 数据盘 | 三星 PCIe 4.0 NVMe SSD 980 PRO |
| 测试软件 | fio |
1、软件安装
yum -y install epel-release yum -y install fio
2、挂载数据目录
sudo vgcreate lvmgroup /dev/vdb sudo lvcreate -l 100%FREE -n data lvmgroup sudo mkfs.xfs /dev/lvmgroup/data sudo mkdir /data sudo echo "/dev/lvmgroup/data /data xfs defaults 0 0" >>/etc/fstab sudo mount -av
3、编写 fio 测试配置文件：创建一个用于 fio 测试的配置文件。您可以使用文本编辑器创建一个新文件，例如 fio_test_config.fio
[global] ioengine=libaio direct=1 thread=1 rw=randrw bs=4k numjobs=4 size=32G runtime=300 group_reporting [randwrite] [global]：这部分指定了全局配置，适用于所有作业。 ioengine=libaio：指定I/O引擎为libaio，表示使用异步I/O。 direct=1：启用直接I/O，绕过操作系统的文件系统缓存。 thread=1：每个作业使用一个线程。 rw=randrw：读写模式为随机读写。 bs=4k：块大小为4KB。 numjobs=4：作业数量为4，表示并发的作业数。 size=32G：每个作业的测试文件大小为32GB。 runtime=300：测试运行时间为300秒。 group_reporting：指定所有线程的统计信息应汇总在一起报告。
直通DirectPath
[root@10-53-220-58 ~]# cd /data/
[root@10-53-220-58 data]# ls
[root@10-53-220-58 data]# sudo dd if=/dev/zero of=/data/fio_test_file bs=1G count=32
32+0 records in
32+0 records out
34359738368 bytes (34 GB) copied, 36.9548 s, 930 MB/s
[root@10-53-220-58 data]# vi fio_test_config.fio
[root@10-53-220-58 data]# sudo fio fio_test_config.fio
randwrite: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 4 threads
randwrite: Laying out IO file (1 file / 32768MiB)
randwrite: Laying out IO file (1 file / 32768MiB)
randwrite: Laying out IO file (1 file / 32768MiB)
randwrite: Laying out IO file (1 file / 32768MiB)
Jobs: 4 (f=4): [m(4)][100.0%][r=102MiB/s,w=103MiB/s][r=26.2k,w=26.3k IOPS][eta 00m:00s]
randwrite: (groupid=0, jobs=4): err= 0: pid=9048: Fri Apr 12 13:55:32 2024
read: IOPS=26.0k, BW=102MiB/s (107MB/s)(29.8GiB/300001msec)
slat (nsec): min=3381, max=69136, avg=4802.53, stdev=1677.41
clat (usec): min=21, max=19215, avg=113.61, stdev=218.66
lat (usec): min=34, max=19220, avg=118.47, stdev=218.66
clat percentiles (usec):
| 1.00th=[ 48], 5.00th=[ 51], 10.00th=[ 53], 20.00th=[ 55],
| 30.00th=[ 57], 40.00th=[ 59], 50.00th=[ 65], 60.00th=[ 74],
| 70.00th=[ 77], 80.00th=[ 85], 90.00th=[ 93], 95.00th=[ 343],
| 99.00th=[ 1270], 99.50th=[ 1369], 99.90th=[ 1516], 99.95th=[ 1565],
| 99.99th=[ 1958]
bw ( KiB/s): min=20680, max=36128, per=24.99%, avg=26004.68, stdev=2194.12, samples=2397
iops : min= 5170, max= 9032, avg=6501.10, stdev=548.54, samples=2397
write: IOPS=25.0k, BW=102MiB/s (106MB/s)(29.7GiB/300001msec)
slat (usec): min=4, max=135, avg= 6.16, stdev= 1.90
clat (nsec): min=1292, max=1175.4k, avg=27396.71, stdev=3962.20
lat (usec): min=19, max=1181, avg=33.62, stdev= 4.09
clat percentiles (nsec):
| 1.00th=[18304], 5.00th=[21888], 10.00th=[23680], 20.00th=[24960],
| 30.00th=[26496], 40.00th=[26752], 50.00th=[27264], 60.00th=[27520],
| 70.00th=[28288], 80.00th=[29312], 90.00th=[31616], 95.00th=[34048],
| 99.00th=[39168], 99.50th=[41216], 99.90th=[43776], 99.95th=[45312],
| 99.99th=[50944]
bw ( KiB/s): min=20280, max=36008, per=24.99%, avg=25980.38, stdev=2247.11, samples=2397
iops : min= 5070, max= 9002, avg=6495.02, stdev=561.78, samples=2397
lat (usec) : 2=0.04%, 4=0.01%, 10=0.01%, 20=0.90%, 50=50.88%
lat (usec) : 100=44.43%, 250=1.06%, 500=0.46%, 750=0.46%, 1000=0.53%
lat (msec) : 2=1.22%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=4.49%, sys=11.07%, ctx=15587579, majf=0, minf=10
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=7804560,7797520,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=102MiB/s (107MB/s), 102MiB/s-102MiB/s (107MB/s-107MB/s), io=29.8GiB (31.0GB), run=300001-300001msec
WRITE: bw=102MiB/s (106MB/s), 102MiB/s-102MiB/s (106MB/s-106MB/s), io=29.7GiB (31.9GB), run=300001-300001msec
Disk stats (read/write):
dm-2: ios=7801557/7794611, merge=0/0, ticks=862107/186550, in_queue=1048584, util=100.00%, aggrios=7804560/7797562, aggrmerge=0/1, aggrticks=857542/181607, aggrin_queue=1039149, aggrutil=100.00%
nvme0n1: ios=7804560/7797562, merge=0/1, ticks=857542/181607, in_queue=1039149, util=100.00%
www.zhangfangzhou.cn
虚拟化Virtualization
[root@10-53-220-44 data]# sudo dd if=/dev/zero of=/data/fio_test_file bs=1G count=32
32+0 records in
32+0 records out
34359738368 bytes (34 GB) copied, 71.8294 s, 478 MB/s
[root@10-53-220-44 data]# vi fio_test_config.fio
[root@10-53-220-44 data]# sudo fio fio_test_config.fio
randwrite: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 4 threads
randwrite: Laying out IO file (1 file / 32768MiB)
randwrite: Laying out IO file (1 file / 32768MiB)
randwrite: Laying out IO file (1 file / 32768MiB)
randwrite: Laying out IO file (1 file / 32768MiB)
Jobs: 4 (f=4): [m(4)][100.0%][r=78.2MiB/s,w=77.8MiB/s][r=20.0k,w=19.9k IOPS][eta 00m:00s]
randwrite: (groupid=0, jobs=4): err= 0: pid=9016: Fri Apr 12 14:43:21 2024
read: IOPS=20.4k, BW=79.7MiB/s (83.6MB/s)(23.4GiB/300001msec)
slat (usec): min=4, max=114, avg= 6.78, stdev= 1.98
clat (usec): min=42, max=38534, avg=134.39, stdev=190.16
lat (usec): min=53, max=38545, avg=141.24, stdev=190.14
clat percentiles (usec):
| 1.00th=[ 79], 5.00th=[ 87], 10.00th=[ 89], 20.00th=[ 92],
| 30.00th=[ 93], 40.00th=[ 96], 50.00th=[ 98], 60.00th=[ 101],
| 70.00th=[ 104], 80.00th=[ 106], 90.00th=[ 113], 95.00th=[ 161],
| 99.00th=[ 1205], 99.50th=[ 1319], 99.90th=[ 1467], 99.95th=[ 1516],
| 99.99th=[ 2057]
bw ( KiB/s): min= 73, max=24856, per=16.35%, avg=13347.32, stdev=9723.44, samples=2397
iops : min= 18, max= 6214, avg=3336.67, stdev=2431.02, samples=2397
write: IOPS=20.4k, BW=79.6MiB/s (83.5MB/s)(23.3GiB/300001msec)
slat (usec): min=5, max=115, avg= 8.06, stdev= 1.98
clat (nsec): min=1403, max=38544k, avg=44914.21, stdev=37185.93
lat (usec): min=32, max=38560, avg=53.04, stdev=37.23
clat percentiles (usec):
| 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 40], 20.00th=[ 42],
| 30.00th=[ 43], 40.00th=[ 44], 50.00th=[ 45], 60.00th=[ 46],
| 70.00th=[ 47], 80.00th=[ 48], 90.00th=[ 50], 95.00th=[ 52],
| 99.00th=[ 57], 99.50th=[ 59], 99.90th=[ 69], 99.95th=[ 87],
| 99.99th=[ 135]
bw ( KiB/s): min= 71, max=25352, per=16.36%, avg=13341.18, stdev=9726.48, samples=2397
iops : min= 17, max= 6338, avg=3335.13, stdev=2431.78, samples=2397
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=45.37%, 100=33.22%
lat (usec) : 250=19.16%, 500=0.43%, 750=0.43%, 1000=0.46%
lat (msec) : 2=0.91%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=3.53%, sys=11.43%, ctx=12237590, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=6121601,6116013,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=79.7MiB/s (83.6MB/s), 79.7MiB/s-79.7MiB/s (83.6MB/s-83.6MB/s), io=23.4GiB (25.1GB), run=300001-300001msec
WRITE: bw=79.6MiB/s (83.5MB/s), 79.6MiB/s-79.6MiB/s (83.5MB/s-83.5MB/s), io=23.3GiB (25.1GB), run=300001-300001msec
Disk stats (read/write):
dm-2: ios=6120646/6115090, merge=0/0, ticks=806337/255779, in_queue=1062184, util=100.00%, aggrios=6121601/6116035, aggrmerge=0/1, aggrticks=801719/251319, aggrin_queue=1051124, aggrutil=100.00%
sdb: ios=6121601/6116035, merge=0/1, ticks=801719/251319, in_queue=1051124, util=100.00%
www.zhangfangzhou.cn
4、结果
直通 (DirectPath)
read: IOPS=26.0k, BW=102MiB/s (107MB/s)(29.8GiB/300001msec)
write: IOPS=25.0k, BW=102MiB/s (106MB/s)(29.7GiB/300001msec)
虚拟化Virtualization
read: IOPS=20.4k, BW=79.7MiB/s (83.6MB/s)(23.4GiB/300001msec)
write: IOPS=20.4k, BW=79.6MiB/s (83.5MB/s)(23.3GiB/300001msec)
| 读取 (Read) | 写入 (Write) | 带宽 (Bandwidth) | |
|---|---|---|---|
| 直通 (DirectPath) | 26.0k | 25.0k | 102MiB/s (107MB/s) |
| 虚拟化 (Virtualization) | 20.4k | 20.4k | 79.7MiB/s (83.6MB/s) |
存储虚拟化后，读性能损失约21.5%，写性能损失约18.4%
"进程每打开一个文件（linux下一切皆文件，包括socket），都会消耗一定的内存资源。如果有不怀好心的人启动一个进程来无限的创建和打开新的文件，会让服务器崩溃。所以linux系统出于安全角度的考虑，在多个位置都限制了可打开的文件描述符的数量，包括系统级、用户级、进程级。这三个限制的含义和修改方式如下："
"TCP分配发送缓存区的大小受参数net.ipv4.tcp_wmem配置影响。"
