云平台与分布式存储的性能测试实践
在云平台运维和分布式存储部署过程中，性能评估是至关重要的一环。本文将介绍我在实际项目中使用的性能测试方法，涵盖虚拟机各项性能指标以及Ceph对象存储的基准测试。
性能测试的关键指标
性能测试涵盖多个维度，就我参与的项目而言，主要包括两类场景：云平台虚拟机能性能评估和Ceph分布式存储性能验证。
虚拟机性能测试指标：
- 网络吞吐能力：跨物理主机部署的两台虚拟机之间的网络带宽上限
- 存储I/O性能：虚拟磁盘的IOPS和吞吐量表现
- 处理器计算能力：虚拟机可获得的CPU计算资源
- 内存访问带宽：虚拟机内存读写的数据传输速率
Ceph存储性能指标：
- 吞吐量：系统每秒能够处理的I/O请求数量
- 延迟：单个请求的平均响应时长
- 并发承载力：系统稳定运行时的最大并发连接数
虚拟机性能测试详解
网络带宽测试
使用iperf3工具可以测量网络链路的最大带宽。测试时需要准备两台位于不同物理宿主机上的虚拟机，分别作为服务端和客户端。
服务端配置：
# iperf3 -s -p 5001
-----------------------------------------------------------
Server listening on 5001
-----------------------------------------------------------
客户端配置：
# iperf3 -c 192.168.0.20 -p 5001 -t 60
-----------------------------------------------------------
Connecting to host 192.168.0.20, port 5001
[ 4] local 192.168.0.15 port 48792 connected to 192.168.0.20 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-60.00 sec 11.2 GBytes 1.60 Gbits/sec
-----------------------------------------------------------
完整的网络性能评估应包含三组对比数据：
| 测试场景 | 内网带宽 |
|---|---|
| 虚拟机间（跨宿主机） | |
| 虚拟机与宿主机间 | |
| 宿主机之间 |
通过对比可以计算虚拟化层的网络性能损耗：
网络性能损耗 = (宿主机间带宽 - 虚拟机间带宽) / 宿主机间带宽 × 100%磁盘I/O性能测试
fio是专业的磁盘I/O基准测试工具，支持多达19种I/O引擎模式。
为虚拟机添加第二块虚拟磁盘（vdc），执行fio测试：
# fio -name="storage_perf_test" -ioengine=libaio -direct=1 -thread \
-filename=/dev/vdc -time_based -runtime=60 -bs=4k -iodepth=32 -rw=randread
storage_perf_test: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-3.30
Starting 4 threads
Jobs: 4 (f=4): [r(4)][100.0% done] [256.8MB/0KB/0KB /s] [65.7K/0/0 iops] [eta 00m:00s]
storage_perf_test: (groupid=0, jobs=4): err= 0: pid=8291: Thu Jun 2 10:15:22 2022
read : io=15483MB, bw=264512KB/s, iops=66128, runt= 60001msec
slat (usec): min=3, max=892, avg= 8.42, stdev= 5.17
clat (usec): min=412, max=185420, avg=1923.51, stdev=1531.28
lat (usec): min=718, max=185425, avg=1932.18, stdev=1531.15
clat percentiles (usec):
| 1.00th=[ 1712], 5.00th=[ 1784], 10.00th=[ 1824], 20.00th=[ 1856],
| 30.00th=[ 1888], 40.00th=[ 1920], 50.00th=[ 1952], 60.00th=[ 1984],
| 70.00th=[ 2016], 80.00th=[ 2048], 90.00th=[ 2128], 95.00th=[ 2176],
| 99.00th=[ 2912], 99.50th=[ 3008], 99.90th=[ 3488], 99.95th=[ 3712],
| 99.99th=[ 7520]
bw (KB /s): min=58240, max=71168, per=25.00%, avg=66128.00, stdev=3251.42
lat (usec) : 500=0.01%, 750=0.02%, 1000=0.01%
lat (msec) : 2=68.42%, 4=31.56%, 10=0.01%, 250=0.01%
cpu : usr=12.35%, sys=28.41%, ctx=192845, majf=0, minf=68
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.8%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, >=64=0.0%
issued : total=r=3969021/w=0/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: io=15483MB, aggrb=264512KB/s, minb=264512KB/s, maxb=264512KB/s, mint=60001msec, maxt=60001msec
Disk stats (read/write):
vdc: ios=3969021/0, merge=0/0, ticks=7638451/0, in_queue=7632158, util=98.12%
关键参数说明：
-ioengine：I/O引擎类型，libaio为Linux原生异步I/O-direct：设为1时绕过系统缓存，直接与磁盘交互-filename：指定测试的块设备路径-runtime：测试持续时间（秒）-bs：单次I/O操作的块大小-iodepth：I/O队列深度，影响并发能力-rw：读写模式（read/write/randread/randwrite）
重点关注指标：bw（带宽）和iops（每秒I/O次数）。通常随机读性能优于顺序读，顺序写性能优于随机写。
CPU性能测试
采用UnixBench进行综合性能评估。该工具是开源的Unix系统基准测试套件，能够全面评估主机系统的计算能力。
UnixBench的测试结果反映的是系统综合性能，与CPU、内存、存储和操作系统都有关系，但CPU影响最为显著。测试时需要分别在虚拟机和宿主机上运行，通过对比评估虚拟化性能开销。
测试项目概览：
- Dhrystone 2：整数运算性能测试，主要考察字符串处理能力
- Whetstone：浮点运算性能测试，包含数学函数和数组操作
- Execl Throughput：进程创建和执行能力
- File Copy：文件读写复制性能
- Pipe Throughput：进程间管道通信能力
- Context Switching：进程上下文切换性能
- Shell Scripts：shell脚本并发执行能力
- System Call Overhead：系统调用开销
虚拟机测试配置： 2核CPU / 4GB内存 / 40GB磁盘
# ./Run -c 2
------------------------------------------------------------------------
Benchmark Run: 四 6月 2 2022 09:20:15 - 09:48:32
2 CPUs in system; running 2 parallel copies of tests
Dhrystone 2 using register variables 48235491.6 lps (10.0 s, 7 samples)
Double-Precision Whetstone 5834.7 MWIPS (9.9 s, 7 samples)
Execl Throughput 5412.3 lps (29.9 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks 1287654.2 KBps (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks 342187.5 KBps (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks 3298471.8 KBps (30.0 s, 2 samples)
Pipe Throughput 2318472.5 lps (10.0 s, 7 samples)
Pipe-based Context Switching 428563.7 lps (10.0 s, 7 samples)
Process Creation 18472.6 lps (30.0 s, 2 samples)
Shell Scripts (1 concurrent) 3205.4 lpm (60.0 s, 2 samples)
Shell Scripts (8 concurrent) 812.6 lpm (60.2 s, 2 samples)
System Call Overhead 2589371.4 lps (10.0 s, 7 samples)
System Benchmarks Index Values BASELINE RESULT INDEX
Dhrystone 2 using register variables 116700.0 48235491.6 4133.2
Double-Precision Whetstone 55.0 5834.7 1060.8
Execl Throughput 43.0 5412.3 1258.7
File Copy 1024 bufsize 2000 maxblocks 3960.0 1287654.2 3251.7
File Copy 256 bufsize 500 maxblocks 1655.0 342187.5 2067.6
File Copy 4096 bufsize 8000 maxblocks 5800.0 3298471.8 5687.0
Pipe Throughput 12440.0 2318472.5 1863.7
Pipe-based Context Switching 4000.0 428563.7 1071.4
Process Creation 126.0 18472.6 1466.2
Shell Scripts (1 concurrent) 42.4 3205.4 755.9
Shell Scripts (8 concurrent) 6.0 812.6 1354.3
System Call Overhead 15000.0 2589371.4 1726.2
========
System Benchmarks Index Score 1789.5
宿主机测试结果对比：
# ./Run -c 2
... ...
------------------------------------------------------------------------
Benchmark Run: 四 6月 2 2022 16:45:20 - 17:13:41
12 CPUs in system; running 2 parallel copies of tests
Dhrystone 2 using register variables 51872635.8 lps (10.0 s, 7 samples)
Double-Precision Whetstone 6415.2 MWIPS (9.9 s, 7 samples)
Execl Throughput 5871.5 lps (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks 958472.6 KBps (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks 254813.9 KBps (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks 2816452.7 KBps (30.0 s, 2 samples)
Pipe Throughput 2738195.8 lps (10.0 s, 7 samples)
Pipe-based Context Switching 265482.3 lps (10.0 s, 7 samples)
Process Creation 17358.4 lps (30.0 s, 2 samples)
Shell Scripts (1 concurrent) 11854.7 lpm (60.0 s, 2 samples)
Shell Scripts (8 concurrent) 4298.2 lpm (60.0 s, 2 samples)
System Call Overhead 2948271.6 lps (10.0 s, 7 samples)
System Benchmarks Index Values BASELINE RESULT INDEX
Dhrystone 2 using register variables 116700.0 51872635.8 4444.9
Double-Precision Whetstone 55.0 6415.2 1166.4
Execl Throughput 43.0 5871.5 1365.5
File Copy 1024 bufsize 2000 maxblocks 3960.0 958472.6 2420.4
File Copy 256 bufsize 500 maxblocks 1655.0 254813.9 1539.6
File Copy 4096 bufsize 8000 maxblocks 5800.0 2816452.7 4855.9
Pipe Throughput 12440.0 2738195.8 2201.1
Pipe-based Context Switching 4000.0 265482.3 663.7
Process Creation 126.0 17358.4 1377.6
Shell Scripts (1 concurrent) 42.4 11854.7 2795.9
Shell Scripts (8 concurrent) 6.0 4298.2 7163.7
System Call Overhead 15000.0 2948271.6 1965.5
========
System Benchmarks Index Score 2187.3
性能损耗计算：(2187.3 - 1789.5) / 2187.3 × 100% = 18.19%
内存性能测试
使用mbw工具测试内存拷贝带宽，评估用户空间程序的内存访问性能。
虚拟机内存测试：
# mbw 1024
Long uses 8 bytes. Allocating 2*134217728 elements = 2147483648 bytes of memory.
Using 262144 bytes as blocks for memcpy block copy test.
Getting down to business... Doing 10 runs per test.
0 Method: MEMCPY Elapsed: 0.42581 MiB: 1024.00000 Copy: 2405.182 MiB/s
1 Method: MEMCPY Elapsed: 0.42634 MiB: 1024.00000 Copy: 2401.467 MiB/s
2 Method: MEMCPY Elapsed: 0.42592 MiB: 1024.00000 Copy: 2404.715 MiB/s
3 Method: MEMCPY Elapsed: 0.42658 MiB: 1024.00000 Copy: 2399.124 MiB/s
4 Method: MEMCPY Elapsed: 0.42547 MiB: 1024.00000 Copy: 2407.382 MiB/s
5 Method: MEMCPY Elapsed: 0.42621 MiB: 1024.00000 Copy: 2402.847 MiB/s
6 Method: MEMCPY Elapsed: 0.42568 MiB: 1024.00000 Copy: 2406.524 MiB/s
7 Method: MEMCPY Elapsed: 0.42645 MiB: 1024.00000 Copy: 2400.328 MiB/s
8 Method: MEMCPY Elapsed: 0.42539 MiB: 1024.00000 Copy: 2408.125 MiB/s
9 Method: MEMCPY Elapsed: 0.42612 MiB: 1024.00000 Copy: 2403.516 MiB/s
AVG Method: MEMCPY Elapsed: 0.42594 MiB: 1024.00000 Copy: 2403.921 MiB/s
0 Method: DUMB Elapsed: 0.23892 MiB: 1024.00000 Copy: 4286.845 MiB/s
1 Method: DUMB Elapsed: 0.23914 MiB: 1024.00000 Copy: 4281.427 MiB/s
2 Method: DUMB Elapsed: 0.23876 MiB: 1024.00000 Copy: 4288.124 MiB/s
3 Method: DUMB Elapsed: 0.23928 MiB: 1024.00000 Copy: 4278.315 MiB/s
4 Method: DUMB Elapsed: 0.23853 MiB: 1024.00000 Copy: 4292.476 MiB/s
5 Method: DUMB Elapsed: 0.23885 MiB: 1024.00000 Copy: 4287.218 MiB/s
6 Method: DUMB Elapsed: 0.23901 MiB: 1024.00000 Copy: 4284.512 MiB/s
7 Method: DUMB Elapsed: 0.23935 MiB: 1024.00000 Copy: 4277.084 MiB/s
8 Method: DUMB Elapsed: 0.23869 MiB: 1024.00000 Copy: 4289.347 MiB/s
9 Method: DUMB Elapsed: 0.23848 MiB: 1024.00000 Copy: 4293.815 MiB/s
AVG Method: DUMB Elapsed: 23891 MiB: 1024.00000 Copy: 4287.106 MiB/s
0 Method: MCBLOCK Elapsed: 0.15782 MiB: 1024.00000 Copy: 6489.724 MiB/s
1 Method: MCBLOCK Elapsed: 0.15824 MiB: 1024.00000 Copy: 6471.258 MiB/s
2 Method: MCBLOCK Elapsed: 0.15796 MiB: 1024.00000 Copy: 6482.415 MiB/s
3 Method: MCBLOCK Elapsed: 0.15841 MiB: 1024.00000 Copy: 6464.927 MiB/s
4 Method: MCBLOCK Elapsed: 0.15768 MiB: 1024.00000 Copy: 6492.176 MiB/s
5 Method: MCBLOCK Elapsed: 0.15813 MiB: 1024.00000 Copy: 6475.384 MiB/s
6 Method: MCBLOCK Elapsed: 0.15789 MiB: 1024.00000 Copy: 6483.271 MiB/s
7 Method: MCBLOCK Elapsed: 0.15835 MiB: 1024.00000 Copy: 6467.512 MiB/s
8 Method: MCBLOCK Elapsed: 0.15772 MiB: 1024.00000 Copy: 6491.283 MiB/s
9 Method: MCBLOCK Elapsed: 0.15828 MiB: 1024.00000 Copy: 6468.927 MiB/s
AVG Method: MCBLOCK Elapsed: 0.15805 MiB: 1024.00000 Copy: 6478.738 MiB/s
宿主机内存测试结果：
# mbw 1024
... ...
AVG Method: MEMCPY Elapsed: 0.38742 MiB: 1024.00000 Copy: 2642.851 MiB/s
AVG Method: DUMB Elapsed: 0.21563 MiB: 1024.00000 Copy: 4748.926 MiB/s
AVG Method: MCBLOCK Elapsed: 0.14125 MiB: 1024.00000 Copy: 7248.372 MiB/s
性能对比汇总：
| 测试方法 | 虚拟机 (MiB/s) | 宿主机 (MiB/s) |
|---|---|---|
| MEMCPY | 2403.921 | 2642.851 |
| DUMB | 4287.106 | 4748.926 |
| MCBLOCK | 6478.738 | 7248.372 |
内存性能损耗分析：
MEMCPY损耗 = (2642.851 - 2403.921) / 2642.851 × 100% = 9.04%
DUMB损耗 = (4748.926 - 4287.106) / 4748.926 × 100% = 9.73%
MCBLOCK损耗= (7248.372 - 6478.738) / 7248.372 × 100% = 10.62%Ceph对象存储性能测试
COSBench测试工具
COSBench是Intel开源的分布式对象存储基准测试工具，支持多种云存储接口。
组件架构：
- Driver节点：负责生成测试负载，向目标存储发起I/O操作并收集性能数据。访问地址：http://[driver-host]:18088/driver/index.html
- Controller节点：协调各Driver节点执行测试，汇总运行状态和测试结果。访问地址：http://[controller-host]:19088/controller/index.html
配置文件结构：
配置文件位于conf/目录，使用XML格式。以Ceph S3接口为例，配置文件模板为conf/s3-config-sample.xml：
<?xml version="1.0" encoding="UTF-8" ?>
<workload name="ceph-s3-benchmark" description="Ceph S3 performance test">
<storage type="s3" config="accesskey=TESTKEY;secretkey=TESTSECRET;endpoint=http://ceph-cluster:7480" />
<workflow>
<workstage name="init_bucket">
<work type="init" workers="2" config="cprefix=benchdata;containers=r(1,4)" />
</workstage>
<workstage name="prepare_objects">
<work type="prepare" workers="4" config="cprefix=benchdata;containers=r(1,4);objects=r(1,20);sizes=c(128)KB" />
</workstage>
<workstage name="main_test">
<work name="mixed_workload" workers="16" runtime="120">
<operation type="read" ratio="70" config="cprefix=benchdata;containers=u(1,4);objects=u(1,20)" />
<operation type="write" ratio="30" config="cprefix=benchdata;containers=u(1,4);objects=u(21,40);sizes=c(128)KB" />
</work>
</workstage>
<workstage name="cleanup">
<work type="cleanup" workers="2" config="cprefix=benchdata;containers=r(1,4);objects=r(1,40)" />
</workstage>
<workstage name="dispose">
<work type="dispose" workers="2" config="cprefix=benchdata;containers=r(1,4)" />
</workstage>
</workflow>
</workload>配置项说明：
- storage.type：存储类型，Ceph使用s3
- storage.config：认证信息（accesskey、secretkey）和集群endpoint
- workstage：测试阶段
- init：创建测试用的bucket
- prepare：预创建测试对象
- main：主测试阶段，支持read/write/delete/filewrite操作
- cleanup：清理测试对象
- dispose：清理测试bucket
执行方式： 登录Controller节点Web界面，上传配置文件即可启动测试。也可通过界面配置生成器创建自定义测试场景。
并发能力测试
对于最大并发数的测试，常用工具包括：
- Apache JMeter：功能全面的负载测试工具，支持分布式部署和远程启动，配备图形化界面
- http_load：轻量级HTTP基准测试工具
- webbench：简单的Web压力测试工具
JMeter适合复杂场景的深度测试，支持多种协议，可组成集群进行大规模压力测试。具体使用方法可参考官方文档。
