Longhorn V2 Data Engine

The Longhorn V2 Data Engine harnesses the power of the Storage Performance Development Kit (SPDK) to significantly reduce I/O latency while boosting IOPS and throughput. The result is a high-performance storage solution that is capable of meeting diverse workload demands.

The Longhorn V2 Data Engine is an Experimental feature in SUSE Virtualization v1.4.0 and should not be utilized in a production environment.

Prerequisites

Every node with an active Longhorn V2 Data Engine requires the following dedicated resources:

  • 1 CPU core for use by the Longhorn instance-manager pod

  • 2 GiB RAM (allocated as 1024 × 2 GiB huge pages)

  • At least one local NVMe disk for data storage

Limitations

The Longhorn V2 Data Engine currently does not support the following operations:

  • Backing image creation and usage

  • Storage network

  • Volume cloning

  • Volume encryption

  • Volume expansion

  • SUSE Virtualization cannot create and restore snapshots of V2 volumes because the functionality relies on volume cloning.

  • SSDs and other non-NVMe disks are managed using the SPDK AIO bdev driver, which does not support the unmap operation. If you are using non-NVMe disks, avoid trimming the filesystem because this results in I/O errors and paused virtual machines. For example, when creating an ext4 filesystem on a Linux virtual machine, use mkfs.ext4 -E nodiscard /dev/vdb (assuming /dev/vdb is your device path). On Windows virtual machines, you can disable trimming for NTFS by running the command fsutil behavior set disabledeletenotify NTFS 1.

Usage

The Longhorn V2 Data Engine is only available for newly created volumes and images. Existing volumes, virtual machine images and virtual machine root volumes will continue to use the V1 Data Engine.

  1. On the SUSE Virtualization UI, go to Advanced → Settings.

  2. Set longhorn-v2-data-engine-enabled to true.

    SUSE Virtualization automatically loads the kernel modules required by the Longhorn V2 Data Engine, and attempts to allocate 1024 × 2 MiB-sized huge pages (for example, 2 GiB of RAM) on all nodes.

    Changing this setting automatically restarts RKE2 on all nodes but does not affect running virtual machine workloads.

    If you encounter error messages that include the phrase "not enough hugepages-2Mi capacity", allow some time for the error to be resolved. If the error persists, reboot the affected nodes.

    To disable the Longhorn V2 Data Engine on specific nodes (for example, nodes with less processing and memory resources), go to the Hosts screen and add the following label to the target nodes:

    • label: node.longhorn.io/disable-v2-data-engine

    • value: true

  3. Go to the Hosts screen, and then add extra disks to each node as described in Multi-disk Management.

    Set the Provisioner of each extra disk to Longhorn V2 (CSI).

    SUSE Virtualization sets the Longhorn disk driver to auto so that NVMe disks use the SPDK NVMe bdev driver, which provides the best performance and also supports advanced operations such as trim (also known as discard).

    SSDs and other non-NVMe disks are managed using the SPDK AIO bdev driver, which requires a disk size that is an even multiple of 4096 bytes. Non-NVMe disks that do not meet this size requirement cannot be added. Additionally, the SPDK AIO bdev driver does not support the unmap operation. If you are using non-NVMe disks, avoid trimming the filesystem because this results in I/O errors and paused virtual machines.

  4. Go to Advanced → Storage Classes, and then add a new StorageClass as described in Creating a StorageClass.

    Set the Provisioner to Longhorn V2 (CSI).

  5. Use the new StorageClass when creating the following:

    • Volumes (either on the Volumes screen or during virtual machine creation)

    • Images (on the Images screen)

      Volumes and images created using the new StorageClass are backed by the Longhorn V2 Data Engine.

Upgrading from SUSE Virtualization v1.4.x

SUSE Virtualization v1.4.x, which uses SUSE Storage v1.7.x, is unable to live-migrate virtual machines with V2 volumes attached. Moreover, the V2 Data Engine cannot be used for virtual machine images and boot volumes. These limitations do not exist in SUSE Virtualization v1.5.0 and later versions, which use SUSE Storage v1.8.1 and later versions. However, this applies only to volumes and images that are created after SUSE Virtualization is upgraded.

In V2 StorageClasses created using SUSE Virtualization v1.4.x, the migratable option is set to false. Like all other StorageClass properties, this cannot be changed once set. Similarly, V2 volumes created using SUSE Virtualization v1.4.x remain non-migratable after the upgrade. If you used the V2 Data Engine in SUSE Virtualization v1.4.x and later upgrade to SUSE Virtualization v1.5, you must create a new V2 StorageClass. The migratable option is set to true by default, so volumes and images created using this new V2 StorageClass can be live-migrated.

  • If you are using the SPDK AIO bdev driver (specifically, if disks were added using /dev/sd* device paths), V2 volumes created before the upgrade become unusable and cannot be recovered after the upgrade. For more information, see issue #10461.

  • If you are using the SPDK NVMe bdev driver (specifically, if disks were added using /dev/nvme* device paths), V2 volumes created before the upgrade will still function. However, these volumes will continue to use the SUSE Storage v1.7.x engine and remain non-migratable. You can export the data and create new migratable volumes, if necessary.

  • All virtual machines with attached V2 volumes must be stopped before you start the upgrade. Active V2 volumes will cause the upgrade process to stall during the "upgrading system services" phase. The apply-manifests pod logs will show repeated messages similar to the following:

    instance-manager (aio)(v2) (image=longhornio/longhorn-instance-manager:v1.8.1) state is not running on node harvester-node-0, will retry...

    Stopping all Virtual Machines that are using V2 volumes will allow the upgrade to proceed.

If you are using the SPDK NVMe bdev driver (specifically, if disks were added using /dev/nvme* device paths) and non-migratable V2 volumes are attached to existing virtual machines, you can transition to live-migratable volumes by performing the following steps:

  1. Stop the virtual machines.

  2. Export each attached V2 volume to an image that uses the new V2 StorageClass (with the migratable option is set to true).

  3. Once the volumes are exported to images, edit the virtual machine and perform the following actions on the Volumes tab:

    • Remove the existing V2 volumes.

    • Add the images that were created from the exported volumes.

  4. Start the virtual machines.

    This step may take some time, depending on the amount of data that must be copied.

  5. Delete the original volumes and exported images.