File vfio.h¶

FileList > docs > sw > include > opae > vfio.h

APIs to manage a PCIe device via vfio-pci. More...

#include <stdio.h>
#include <stdint.h>
#include <pthread.h>
#include <linux/vfio.h>
#include <opae/mem_alloc.h>
#include <opae/hash_map.h>

Classes¶

Type	Name
struct	opae_vfio OPAE VFIO device abstraction.
struct	opae_vfio_buffer System DMA buffer.
struct	opae_vfio_device VFIO device.
struct	opae_vfio_device_irq Interrupt info.
struct	opae_vfio_device_region MMIO region info.
struct	opae_vfio_group VFIO group.
struct	opae_vfio_iova_range IO Virtual Address Range.
struct	opae_vfio_sparse_info MMIO sparse-mappable region info.

Public Types¶

Type	Name
enum	opae_vfio_buffer_flags Flags for opae_vfio_buffer_allocate_ex() .

Public Functions¶

Type	Name
int	opae_vfio_buffer_allocate (struct opae_vfio * v, size_t * size, uint8_t ** buf, uint64_t * iova) Allocate and map system buffer.
int	opae_vfio_buffer_allocate_ex (struct opae_vfio * v, size_t * size, uint8_t ** buf, uint64_t * iova, int flags) Allocate and map system buffer (extended w/ flags)
int	opae_vfio_buffer_free (struct opae_vfio * v, uint8_t * buf) Unmap and free a system buffer.
struct opae_vfio_buffer *	opae_vfio_buffer_info (struct opae_vfio * v, uint8_t * vaddr) Extract the internal data structure pointer for the given vaddr.
void	opae_vfio_close (struct opae_vfio * v) Release and close a VFIO device.
int	opae_vfio_irq_disable (struct opae_vfio * v, uint32_t index, uint32_t subindex) Disable an IRQ.
int	opae_vfio_irq_enable (struct opae_vfio * v, uint32_t index, uint32_t subindex, int event_fd) Enable an IRQ.
int	opae_vfio_irq_mask (struct opae_vfio * v, uint32_t index, uint32_t subindex) Mask an IRQ.
int	opae_vfio_irq_unmask (struct opae_vfio * v, uint32_t index, uint32_t subindex) Unmask an IRQ.
int	opae_vfio_open (struct opae_vfio * v, const char * pciaddr) Open and populate a VFIO device.
int	opae_vfio_region_get (struct opae_vfio * v, uint32_t index, uint8_t ** ptr, size_t * size) Query device MMIO region.
int	opae_vfio_secure_open (struct opae_vfio * v, const char * pciaddr, const char * token) Open and populate a VFIO device.

Detailed Description¶

Presents a simple interface for interacting with a PCIe device that is bound to the vfio-pci driver. See https://kernel.org/doc/Documentation/vfio.txt for a description of vfio-pci.

Provides APIs for opening/closing the device, querying info about the MMIO regions of the device, and allocating/mapping and freeing/unmapping DMA buffers.

Public Types Documentation¶

enum opae_vfio_buffer_flags¶

enum opae_vfio_buffer_flags {
    OPAE_VFIO_BUF_PREALLOCATED = 1
};

Public Functions Documentation¶

function opae_vfio_buffer_allocate¶

Allocate and map system buffer.

int opae_vfio_buffer_allocate (
    struct opae_vfio * v,
    size_t * size,
    uint8_t ** buf,
    uint64_t * iova
)

Allocate, map, and retrieve info for a system buffer capable of DMA. Saves an entry in the v->cont_buffers list. If the buffer is not explicitly freed by opae_vfio_buffer_free, it will be freed during opae_vfio_close.

mmap is used for the allocation. If the size is greater than 2MB, then the allocation request is fulfilled by a 1GB huge page. Else, if the size is greater than 4096, then the request is fulfilled by a 2MB huge page. Else, the request is fulfilled by the non-huge page pool.

Note:

Allocations from the huge page pool require that huge pages be configured on the system. Huge pages may be configured on the kernel boot command prompt. Example default_hugepagesz=1G hugepagesz=1G hugepages=2 hugepagesz=2M hugepages=8

Note:

Huge pages may also be configured at runtime. Example sudo sh -c 'echo 8 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages' sudo sh -c 'echo 2 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages'

Note:

Be sure that the IOMMU is also enabled using the follow kernel boot command: intel_iommu=on

Parameters:

v The open OPAE VFIO device.
size A pointer to the requested size. The size may be rounded to the next page size prior to return from the function.
buf Optional pointer to receive the virtual address for the buffer. Pass NULL to ignore.
iova Optional pointer to receive the IOVA address for the buffer. Pass NULL to ignore.

Returns:

Non-zero on error. Zero on success.

Example opae_vfio v;

size_t sz; uint8_t *buf_2m_virt = NULL; uint8_t *buf_1g_virt = NULL; uint64_t buf_2m_iova = 0; uint64_t buf_1g_iova = 0;

if (opae_vfio_open(&v, "0000:00:00.0")) { // handle error } else { sz = 2 * 1024 * 1024; if (opae_vfio_buffer_allocate(&v, &sz, &buf_2m_virt, &buf_2m_iova)) { // handle allocation error }

sz = 1024 * 1024 * 1024; if (opae_vfio_buffer_allocate(&v, &sz, &buf_1g_virt, &buf_1g_iova)) { // handle allocation error } }

function opae_vfio_buffer_allocate_ex¶

Allocate and map system buffer (extended w/ flags)

int opae_vfio_buffer_allocate_ex (
    struct opae_vfio * v,
    size_t * size,
    uint8_t ** buf,
    uint64_t * iova,
    int flags
)

Allocate, map, and retrieve info for a system buffer capable of DMA. Saves an entry in the v->cont_buffers list. If the buffer is not explicitly freed by opae_vfio_buffer_free, it will be freed during opae_vfio_close, unless OPAE_VFIO_BUF_PREALLOCATED is used in which case the buffer is not freed by this library.

When not using OPAE_VFIO_BUF_PREALLOCATED, mmap is used for the allocation. If the size is greater than 2MB, then the allocation request is fulfilled by a 1GB huge page. Else, if the size is greater than 4096, then the request is fulfilled by a 2MB huge page. Else, the request is fulfilled by the non-huge page pool.

Parameters:

v The open OPAE VFIO device.
size A pointer to the requested size. The size may be rounded to the next page size prior to return from the function.
buf Optional pointer to receive the virtual address for the buffer/input buffer pointer when using OPAE_VFIO_BUF_PREALLOCATED. Pass NULL to ignore.
iova Optional pointer to receive the IOVA address for the buffer. Pass NULL to ignore.

Returns:

Non-zero on error. Zero on success.

Example opae_vfio v;

size_t sz = MY_BUF_SIZE; uint8_t *prealloc_virt = NULL; uint64_t iova = 0;

prealloc_virt = allocate_my_buffer(sz);

if (opae_vfio_open(&v, "0000:00:00.0")) { // handle error } else { if (opae_vfio_buffer_allocate_ex(&v, &sz, &prealloc_virt, &iova, OPAE_VFIO_BUF_PREALLOCATED)) { // handle allocation error } }

function opae_vfio_buffer_free¶

Unmap and free a system buffer.

int opae_vfio_buffer_free (
    struct opae_vfio * v,
    uint8_t * buf
)

The buffer corresponding to buf must have been created by a previous call to opae_vfio_buffer_allocate.

Parameters:

v The open OPAE VFIO device.
buf The virtual address corresponding to the buffer to be freed.

Returns:

Non-zero on error. Zero on success.

Example size_t sz; uint8_t *buf_2m_virt = NULL; uint64_t buf_2m_iova = 0;

sz = 2 * 1024 * 1024; if (opae_vfio_buffer_allocate(&v, &sz, &buf_2m_virt, &buf_2m_iova)) { // handle allocation error } else { // use the buffer

if (opae_vfio_buffer_free(&v, buf_2m_virt)) { // handle free error } }

function opae_vfio_buffer_info¶

Extract the internal data structure pointer for the given vaddr.

struct opae_vfio_buffer * opae_vfio_buffer_info (
    struct opae_vfio * v,
    uint8_t * vaddr
)

The virtual address vaddr must correspond to a buffer previously allocated by opae_vfio_buffer_allocate() or opae_vfio_buffer_allocate_ex().

Parameters:

v The open OPAE VFIO device.
vaddr The user virtual address of the desired buffer info struct.

Returns:

NULL on lookup error.

function opae_vfio_close¶

Release and close a VFIO device.

void opae_vfio_close (
    struct opae_vfio * v
)

The given device pointer must have been previously initialized by opae_vfio_open. Releases all data structures, including any DMA buffer allocations that have not be explicitly freed by opae_vfio_buffer_free.

Parameters:

v Storage for the device info. May be stack-resident.

Example opae_vfio v;

if (opae_vfio_open(&v, "0000:00:00.0")) { // handle error } else { // interact with the device ... // free the device opae_vfio_close(&v); }

Example $ sudo opaevfio -r 0000:00:00.0

function opae_vfio_irq_disable¶

Disable an IRQ.

int opae_vfio_irq_disable (
    struct opae_vfio * v,
    uint32_t index,
    uint32_t subindex
)

Parameters:

v The open OPAE VFIO device.
index The IRQ category. For MSI-X, use VFIO_PCI_MSIX_IRQ_INDEX.
subindex The IRQ to disable.

Returns:

Non-zero on error. Zero on success.

function opae_vfio_irq_enable¶

Enable an IRQ.

int opae_vfio_irq_enable (
    struct opae_vfio * v,
    uint32_t index,
    uint32_t subindex,
    int event_fd
)

Parameters:

v The open OPAE VFIO device.
index The IRQ category. For MSI-X, use VFIO_PCI_MSIX_IRQ_INDEX.
subindex The IRQ to enable.
event_fd The file descriptor, created by eventfd(). Interrupts will result in this event_fd being signaled.

Returns:

Non-zero on error. Zero on success.

function opae_vfio_irq_mask¶

Mask an IRQ.

int opae_vfio_irq_mask (
    struct opae_vfio * v,
    uint32_t index,
    uint32_t subindex
)

Parameters:

v The open OPAE VFIO device.
index The IRQ category. For MSI-X, use VFIO_PCI_MSIX_IRQ_INDEX.
subindex The IRQ to mask.

Returns:

Non-zero on error. Zero on success.

function opae_vfio_irq_unmask¶

Unmask an IRQ.

int opae_vfio_irq_unmask (
    struct opae_vfio * v,
    uint32_t index,
    uint32_t subindex
)

Parameters:

v The open OPAE VFIO device.
index The IRQ category. For MSI-X, use VFIO_PCI_MSIX_IRQ_INDEX.
subindex The IRQ to unmask.

Returns:

Non-zero on error. Zero on success.

function opae_vfio_open¶

Open and populate a VFIO device.

int opae_vfio_open (
    struct opae_vfio * v,
    const char * pciaddr
)

Opens the PCIe device corresponding to the address given in pciaddr. The device must be bound to the vfio-pci driver prior to opening it. The data structures corresponding to IOVA space, MMIO regions, and DMA buffers are initialized.

Parameters:

v Storage for the device info. May be stack-resident.
pciaddr The PCIe address of the requested device.

Returns:

Non-zero on error. Zero on success.

Example $ sudo opaevfio -i 0000:00:00.0 -u user -g group

Example opae_vfio v;

if (opae_vfio_open(&v, "0000:00:00.0")) { // handle error }

function opae_vfio_region_get¶

Query device MMIO region.

int opae_vfio_region_get (
    struct opae_vfio * v,
    uint32_t index,
    uint8_t ** ptr,
    size_t * size
)

Retrieves info describing the MMIO region with the given region index. The device structure v must have been previously opened by a call to opae_vfio_open.

Parameters:

v The open OPAE VFIO device.
index The zero-based index of the desired MMIO region.
ptr Optional pointer to receive the virtual address for the region. Pass NULL to ignore.
size Optional pointer to receive the size of the MMIO region. Pass NULL to ignore.

Returns:

Non-zero on error (including index out-of-range). Zero on success.

Example opae_vfio v;

uint8_t *fme_virt = NULL; uint8_t *port_virt = NULL; size_t fme_size = 0; size_t port_size = 0;

if (opae_vfio_open(&v, "0000:00:00.0")) { // handle error } else { opae_vfio_region_get(&v, 0, &fme_virt, &fme_size); opae_vfio_region_get(&v, 2, &port_virt, &port_size); }

function opae_vfio_secure_open¶

Open and populate a VFIO device.

int opae_vfio_secure_open (
    struct opae_vfio * v,
    const char * pciaddr,
    const char * token
)

Opens the PCIe device corresponding to the address given in pciaddr, using the VF token (GUID) given in token. The device must be bound to the vfio-pci driver prior to opening it. The data structures corresponding to IOVA space, MMIO regions, and DMA buffers are initialized.

Parameters:

v Storage for the device info. May be stack-resident.
pciaddr The PCIe address of the requested device.
token The GUID representing the VF token.

Returns:

Non-zero on error. Zero on success.

Example $ sudo opaevfio -i 0000:00:00.0 -u user -g group

Example opae_vfio v;

if (opae_vfio_secure_open(&v, "0000:00:00.0", "00f5ad6b-2edd-422e-9d1e-34124c686fec")) { // handle error }

The documentation for this class was generated from the following file docs/sw/include/opae/vfio.h