DPDK Bangalore has ended
Back To Schedule
Friday, March 9 • 3:40pm - 4:05pm
Integrating DPDK with Storage Application

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

The Storage Performance Development Kit (SPDK) provides a set of tools and libraries for writing high performance, scalable, user-mode storage applications. New applications requiring fast access to storage can be built on top of SPDK, however they need to adhere to principles which are fundamental part of SPDK/DPDK framework. To name few of these, there is exactly one thread running on a CPU core, which never blocks and constantly polls for new events and executes corresponding handlers in a loop. Blocking locks in poller’s loop are not acceptable since those could delay execution of other handlers on reactor. Memory is allocated from pinned huge pages, which make DMA transfers to and from device possible avoiding copy of data between buffers etc. Those are fundamental design changes compared to how applications were built 5, 10 or more years ago (legacy applications). Legacy applications have usually many threads, which are using classic synchronization primitives (mutexes, readers/writers locks, etc.), which don’t scale with number of CPUs, and rely on kernel to synchronize and schedule the threads with well known ill effects. Trying to redesign legacy applications is often not feasible especially if they are more complex and have matured for many years. On the other hand, trying to reimplement them from scratch, preserving their stability and quality, can take years. Instead what we suggest is a compromise solution when legacy application runs with minimal changes while to some degree it can leverage performance that SPDK has to offer for doing IOs. Legacy application runs in a separate process from SPDK and VirtIO with vhost-user protocol is used for passing data between them. VirtIO is well proven technology which allows moving data between two processes using shared memory without a need to copy them, without heavy-weight synchronization and without involving kernel. Vhost-user client-server protocol allows to establish VirtIO data channel between two processes using unix domain socket. SPDK framework already comes with vhost-user server implementation. Missing part is vhost-user client implementation coming in form of a simple library which could be easily embedded to legacy application. A library for allocating IO buffers from huge pages is necessary too, although DPDK’s rte_eal library seems sufficient for a proof of concept. Legacy application could completely bypass the kernel for doing disk IO assuming SPDK runs a disk driver in userland and although the application cannot unleash the full potential of SPDK due to its legacy design, it is expected to perform faster. The most important question, which we are trying to find an answer for, is how much speed can be gained using this compromise solution compared to traditional way of reading/writing data blocks through block device file. While comparing IOPS from fio benchmark tool is surely interesting, we also focus and compare performance of real-world storage application - one of the most advanced local file system of present days - ZFS. Perhaps it is less known that it is possible to run ZFS file system in userspace, which was initially introduced only for testing it. It is exciting to observe how much the performance of the file system as a whole can be further improved by using SPDK as a storage backend.
In this talk, Mayank and Vishnu, who are active contributors of an open source project OpenEBS, worked on ZFS to make it SPDK enable, will share their learnings about:
- Integration of DPDK/SPDK libraries with project
- Uses of mempool library for memory allocation of frequently used objects
- Uses of ring library for message passing between threads
- Experiences with developing vhost-user client library


Vishnu Itta

Director Of Engineering, MayaData
Developer who always eager to learn, loves math, algorithms and programming. Have good experience in storage protocols, ZFS, FreeBSD internals, Linux, device drivers. Enjoys playing Table Tennis and doing travel.

Mayank Patel

Software engineer, MayaData Inc
Eager to hack into latest technologies. Loves coding. Enjoys traveling.

Friday March 9, 2018 3:40pm - 4:05pm PST
Leela Palace

Attendees (5)