暂无文章
TL;DR: Skip to the Conclusion for a summary.
I have always been passionate about virtualization and have consistently used it.
The first solution I installed on my infrastructures (and those of clients) was Xen on NetBSD, with great success. I then used Xen on Linux and, since 2012, OpenNebula, followed by Proxmox in 2013. Proxmox has always given me great satisfaction, and even today I consider it a valuable platform that I install gladly. I have also used other hypervisors like XCP-ng but less frequently, and in recent years, I have started to make extensive use of bhyve.
About two and a half years ago, we began a progressive process of migrating our servers (and those of our clients) from Linux to FreeBSD, using jails (when possible) or VMs on bhyve. In some cases, migrating setups from Proxmox to FreeBSD resulted in performance improvements, even with the same hardware. In some instances, I migrated VMs without notifying clients, and they contacted me a few days later to inquire if we had new hardware because they noticed better performance.
After years, I decided to conduct a test to determine if this was just a perception or if there was a technical basis behind it. Of course, this test has no scientific validity, and the results were obtained on specific hardware and at a specific time, so on different hardware, workload, and situations, the results could be entirely opposite. However, I tried to have as scientific and objective an approach as possible since I am comparing two solutions that I care about and use daily.
I often see comparative tests done on VMs from various providers. In my opinion, this comparison makes no sense because a VM from any provider shares its hardware with many other VMs, so the results will vary depending on the load of the "neighbors" and will never be reliable.
For this test, I decided to take a physical server with the following characteristics:
The hardware is not recent, but still very widespread. On more recent hardware, the results might differ, but the test will be based on this configuration.
I installed Proxmox 8.2.2 starting from the Debian template of the provider and manually installed it following the instructions. I created a partition for Proxmox and left one partition free on each of the two NVME drives to create (at different times) the ZFS pool (in mirror) and the LVM on top of the Linux software raid.
After all the tests, I installed FreeBSD 14.1-RELEASE on ZFS on the same host, using bsdinstall from an mfsbsd image since the provider does not directly support installing FreeBSD from its panel or rescue mode.
In both installations, I always trimmed the NVME drives before starting the tests, and in the case of ZFS, I set (both on Proxmox and FreeBSD) compression to zstd and atime to off. No other changes were made compared to the standard installation.
On FreeBSD, the VM was created and managed with vm-bhyve (devel).
On Proxmox, I tested the physical host on ZFS and ext4 and the VM on ZFS and LVM as LVM is the standard and most common setup in Proxmox.
On FreeBSD, I tested the host on ZFS and the VM with both virtio and nvme drivers, on zvol, and as an image file within a ZFS dataset.
I used sysbench installed from the official Debian repository (on Proxmox and VM) and from the FreeBSD package on the respective host.
The VMs, both on Proxmox and FreeBSD, have nearly identical characteristics and default configuration (apart from the nvme drivers set on bhyve, and for that reason, I also tested virtio).
For those who want to reproduce my tests, here are the detailed configurations of the VMs used in bhyve:
FreeBSD bhyve VM Configuration with NVMe Driver:
loader="uefi"
cpu=4
memory=4096M
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"
FreeBSD bhyve VM Configuration with virtio Driver:
loader="uefi"
cpu=4
memory=4096M
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0.img"
Proxmox VM Configuration:
| Component | Details |
|---|---|
| Memory | 4.00 GiB [balloon=0] |
| Processors | 4 (1 sockets, 4 cores) [x86-64-v2-AES] |
| BIOS | Default (SeaBIOS) |
| Display | Default |
| Machine | Default (i440fx) |
| SCSI Controller | VirtIO SCSI single |
| CD/DVD Drive (ide2) | local:iso/debian-12.5.0-amd64-netinst.iso,media=cdrom,size=629M |
| Hard Disk (scsi0) | zfspool:vm-100-disk-0,cache=writeback,discard=on,iothread=1,size=50G,ssd=1 |
| Network Device (net0) | virtio=BC:24:11:22:3D:F0,bridge=vmbr0 |
In all the configurations, I used Debian 12 as the VM operating system, with the file system on ext4.
I chose Debian 12 as it is a stable, widespread, and modern Linux distribution. I did not test a FreeBSD VM because, in my setups, I tend not to virtualize FreeBSD on FreeBSD but to use nested jails.
All tests were performed multiple times, and I took the median results. CPU and RAM were tested only on the first VM (on Proxmox (ZFS) and FreeBSD (ZFS and nvme)) as they are not dependent on the underlying storage. Storage performance, on the other hand, was tested on all configurations.
On both VMs:
sysbench --test=cpu --cpu-max-prime=20000 run
sysbench --test=memory run
| Configuration | Events per Second | Total Time (s) | Latency (avg) (ms) |
|---|---|---|---|
| Proxmox | 498.08 | 10.0010 | 2.01 |
| FreeBSD | 473.65 | 10.0019 | 2.11 |
| Configuration | Total Operations | Operations per Second | Total MiB Transferred | MiB/sec | Latency (avg) (ms) |
|---|---|---|---|---|---|
| Proxmox | 64777227 | 6476757.59 | 63259.01 | 6324.96 | 0.00 |
| FreeBSD | 68621063 | 6861139.06 | 67012.76 | 6700.33 | 0.00 |
| Test | Metric | Proxmox (KVM) | FreeBSD (bhyve) | Difference (%) |
|---|---|---|---|---|
| CPU | Events/s | 498.08 | 473.65 | -4.91 |
| Time (s) | 10.0010 | 10.0019 | +0.009 | |
| Latency | 2.01 | 2.11 | +4.98 | |
| RAM | Ops | 64777227 | 68621063 | +5.94 |
| Ops/s | 6476757.59 | 6861139.06 | +5.94 | |
| MiB | 63259.01 | 67012.76 | +5.93 | |
| MiB/s | 6324.96 | 6700.33 | +5.93 | |
| Latency | 0.00 | 0.00 | 0.00 |
The average latency is slightly higher on FreeBSD (+4.98%).
RAM Performance:
In summary, while Proxmox provides more consistent CPU performance, FreeBSD demonstrates superior memory performance. The choice between Proxmox and FreeBSD may depend on the specific workload requirements and the importance of consistent performance versus higher throughput.
The test has been conducted using sysbench, with this command line:
sysbench --test=fileio --file-total-size=30G prepare
sysbench --test=fileio --file-total-size=30G --file-test-mode=rndrw --max-time=300 --max-requests=0 run
| Metric | VM on Proxmox (ZFS) | VM on Proxmox (LVM) | VM on FreeBSD (ZFS, NVMe) | VM on FreeBSD (ZFS, Virtio) | VM on FreeBSD (zvol) | Host FreeBSD (ZFS) | Host Proxmox (ZFS) | Host Proxmox (ext4) |
|---|---|---|---|---|---|---|---|---|
| File creation speed (MiB/s) | 407.82 | 461.52 | 1467.83 | 1398.81 | 1333.64 | 1625.67 | 968.64 | 633.13 |
| Reads per second | 650.09 | 504.80 | 11183.44 | 806.93 | 11834.53 | 1234.62 | 920.95 | 498.37 |
| Writes per second | 433.40 | 336.54 | 7455.62 | 537.95 | 7889.69 | 823.08 | 613.96 | 332.25 |
| fsyncs per second | 1387.08 | 1076.97 | 23858.08 | 1721.79 | 25247.36 | 2634.01 | 1964.96 | 1063.19 |
| Read throughput (MiB/s) | 10.16 | 7.89 | 174.74 | 12.61 | 184.91 | 19.29 | 14.39 | 7.79 |
| Write throughput (MiB/s) | 6.77 | 5.26 | 116.49 | 8.41 | 123.28 | 12.86 | 9.59 | 5.19 |
| Total events | 741163 | 575588 | 12749157 | 919952 | 13491459 | 1407592 | 1049894 | 568277 |
| Average latency (ms) | 0.40 | 0.52 | 0.02 | 0.33 | 0.02 | 0.21 | 0.29 | 0.53 |
| 95th percentile latency (ms) | 2.30 | 3.25 | 0.06 | 1.58 | 0.05 | 1.32 | 1.79 | 2.71 |
| Max latency (ms) | 22.65 | 32.30 | 35.49 | 13.60 | 77.53 | 9.03 | 9.47 | 17.39 |
| Total test time (s) | 300.0475 | 300.1147 | 300.0020 | 300.0226 | 300.0012 | 300.0416 | 300.0159 | 300.1381 |
| Metric | VM on Proxmox (LVM) | VM on FreeBSD (ZFS, NVMe) | VM on FreeBSD (ZFS, Virtio) | VM on FreeBSD (zvol) | Host FreeBSD (ZFS) | Host Proxmox (ZFS) | Host Proxmox (ext4) |
|---|---|---|---|---|---|---|---|
| File creation speed (MiB/s) | +13.18% | +259.77% | +242.99% | +227.02% | +298.62% | +137.52% | +55.25% |
| Reads per second | -22.34% | +1619.98% | +24.13% | +1720.45% | +89.92% | +41.67% | -23.34% |
| Writes per second | -22.35% | +1620.26% | +24.12% | +1720.42% | +89.91% | +41.66% | -23.34% |
| fsyncs per second | -22.36% | +1620.02% | +24.13% | +1720.18% | +89.90% | +41.66% | -23.35% |
| Read throughput (MiB/s) | -22.34% | +1619.88% | +24.11% | +1719.98% | +89.86% | +41.63% | -23.33% |
| Write throughput (MiB/s) | -22.30% | +1620.68% | +24.22% | +1720.97% | +89.96% | +41.65% | -23.33% |
| Total events | -22.34% | +1620.16% | +24.12% | +1720.31% | +89.92% | +41.65% | -23.31% |
| Average latency (ms) | +30.00% | -95.00% | -17.50% | -95.00% | -47.50% | -27.50% | +32.50% |
| 95th percentile latency (ms) | +41.30% | -97.39% | -31.30% | -97.83% | -42.61% | -22.17% | +17.83% |
| Max latency (ms) | +42.60% | +56.69% | -39.96% | +242.30% | -60.13% | -58.19% | -23.22% |
| Total test time (s) | +0.02% | -0.02% | -0.01% | -0.02% | -0.02% | -0.01% | +0.01% |
| Metric | VM on Proxmox (ZFS) | VM on FreeBSD (ZFS, NVMe) | VM on FreeBSD (ZFS, Virtio) | VM on FreeBSD (zvol) | Host FreeBSD (ZFS) | Host Proxmox (ZFS) | Host Proxmox (ext4) |
|---|---|---|---|---|---|---|---|
| File creation speed (MiB/s) | -11.64% | +218.04% | +203.09% | +188.97% | +252.24% | +109.88% | +37.18% |
| Reads per second | +28.78% | +2115.42% | +59.85% | +2244.40% | +144.58% | +82.44% | -1.27% |
| Writes per second | +28.78% | +2115.37% | +59.85% | +2244.35% | +144.57% | +82.43% | -1.27% |
| fsyncs per second | +28.79% | +2115.30% | +59.87% | +2244.30% | +144.58% | +82.45% | -1.28% |
| Read throughput (MiB/s) | +28.77% | +2114.70% | +59.82% | +2243.60% | +144.49% | +82.38% | -1.27% |
| Write throughput (MiB/s) | +28.71% | +2114.64% | +59.89% | +2243.73% | +144.49% | +82.32% | -1.33% |
| Total events | +28.77% | +2114.98% | +59.83% | +2243.94% | +144.55% | +82.40% | -1.27% |
| Average latency (ms) | -23.08% | -96.15% | -36.54% | -96.15% | -59.62% | -44.23% | +1.92% |
| 95th percentile latency (ms) | -29.23% | -98.15% | -51.38% | -98.46% | -59.38% | -44.92% | -16.62% |
| Max latency (ms) | -29.88% | +9.88% | -57.89% | +140.03% | -72.04% | -70.68% | -46.16% |
| Total test time (s) | -0.02% | -0.04% | -0.03% | -0.04% | -0.02% | -0.03% | +0.01% |
The performance data collected from various configurations of Proxmox and FreeBSD provides a comprehensive view of the I/O capabilities and highlights some significant differences. Here is an analysis of the key findings:
An important observation from my tests is that VMs with the bhyve NVMe driver show significantly higher performance compared to the same VMs with the virtio driver or compared to the physical host system. This difference initially led me to hypothesize that the bhyve NVMe driver might not correctly respect fsync operations, returning a positive result before the underlying file system has confirmed the final write. However, this was just a theory based on benchmark results and is not supported by concrete data. Furthermore, some developers have reviewed the code and found no evidence to suggest this behavior, and I personally have never encountered any potential issues that would indicate such a problem.
Specifically, I observed that:
File Creation Speed:
Read and Write Operations per Second:
Host FreeBSD (ZFS) demonstrates the highest read and write operations per second with 1234.62 reads/s and 823.08 writes/s.
fsync Operations per Second:
Host FreeBSD (ZFS) achieves the highest fsync operations per second at 2634.01 fsyncs/s, which is +34.02% compared to Host Proxmox (ZFS) and +147.73% compared to Host Proxmox (ext4).
Throughput:
Host FreeBSD (ZFS) again leads in throughput with 19.29 MiB/s read and 12.86 MiB/s write.
Latency:
Host FreeBSD (ZFS) shows the lowest average latency at 0.21 ms and 95th percentile latency at 1.32 ms.
File Creation Speed:
Read and Write Operations per Second:
VM on FreeBSD (ZFS, NVMe) shows exceptional performance with 11183.44 reads/s and 7455.62 writes/s.
fsync Operations per Second:
VM on FreeBSD (ZFS, NVMe) achieves 23858.08 fsyncs/s, and VM on FreeBSD (zvol) achieves 25247.36 fsyncs/s, both significantly higher than any other configuration.
Throughput:
VM on FreeBSD (ZFS, NVMe) achieves the highest throughput with 174.74 MiB/s read and 116.49 MiB/s write.
Latency:
VM on FreeBSD (ZFS, NVMe) shows very low average latency at 0.02 ms and 95th percentile latency at 0.06 ms.
File Creation Speed:
Read and Write Operations per Second:
VM on FreeBSD (ZFS, NVMe) and VM on FreeBSD (zvol) both outperform VM on Proxmox (ZFS) and VM on Proxmox (LVM) configurations significantly.
fsync Operations per Second:
VM on FreeBSD (ZFS, NVMe) and VM on FreeBSD (zvol) have significantly higher fsync operations compared to VM on Proxmox (ZFS) and VM on Proxmox (LVM).
Throughput:
VM on FreeBSD (ZFS, NVMe) and VM on FreeBSD (zvol) have the highest throughput, followed by VM on Proxmox (ZFS) and then VM on Proxmox (LVM).
Latency:
VM on FreeBSD (ZFS, NVMe) and VM on FreeBSD (zvol) show the lowest latencies among the VMs, indicating faster response times.
Cache Settings and Performance Influence
Cache settings can significantly influence the performance of virtualization systems. In my setup, I did not modify the cache settings for the NVMe and virtio drivers, keeping the default settings. It is possible that the observed performance differences are also due to how different operating systems manage the caches of NVMe devices. I encourage other system administrators to explore the cache settings of their systems to see if changes in this area can influence benchmark results.
Regarding RAM and CPU, the performance of the VMs is comparable. There are slight differences in favor of Proxmox for CPU and FreeBSD for RAM, but in my opinion, these differences are so negligible that they wouldn't sway the decision towards one solution or the other.
The I/O performance data clearly indicates that VM on FreeBSD with NVMe and ZFS outperforms all other configurations by a significant margin. This is evident in the file creation speed, read/write operations per second, fsync operations per second, throughput, and latency metrics.
When comparing physical hosts, Host FreeBSD (ZFS) demonstrates excellent performance, particularly in comparison to Host Proxmox (ZFS) and Host Proxmox (ext4).
When comparing VMs, VM on FreeBSD (ZFS, NVMe) and VM on FreeBSD (zvol) configurations stand out as the top performers.
The VM using virtio on FreeBSD also shows strong performance, albeit not as high as the NVMe configuration. It significantly outperforms Proxmox configurations in terms of file creation speed, read/write operations per second, and throughput, while maintaining competitive latencies.
The virtio driver provides a stable and reliable option, making it a suitable choice for environments where the NVMe driver cannot be used. This makes FreeBSD with virtio a balanced option for virtualization, offering both high performance and reliability.
By examining these performance metrics, users can make informed decisions about their virtualization and storage configurations to optimize their systems for specific workloads and performance requirements.
In light of these tests and experiments, I can therefore affirm that my sensations (and those of many users) of greater "snappiness" of the VMs on FreeBSD can be confirmed. Certainly, Proxmox is a stable solution, rich in features, battle-tested, and has many other valid points, but FreeBSD, especially with the nvme driver, demonstrates very high performance and a very low overhead in installation and operation.
I will continue to use both solutions with great satisfaction, but I will be even more encouraged to implement virtualization servers based on FreeBSD and bhyve.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。