opae.io

SYNOPSIS

opae.io ls [-v,--viddid VIDDID] [-s,--sub-viddid SUB_VIDDID] [--all] [--system-class]
opae.io init [-d,--device PCI_ADDR] [USER[:GROUP]]
opae.io release [-d,--device PCI_ADDR]
opae.io [-d,--device PCI_ADDR] [-r,--region REGION] walk [--offset [OFFSET]] [-u,--show-uuid] [-D,--dump] [-c,--count COUNT] [-y,--delay DELAY] [-s,--safe]
opae.io [-d,--device PCI_ADDR] [-r,--region REGION] dump [--offset [OFFSET]] [-o,--output OUTPUT] [-f,--format {bin,hex}] [-c,--count COUNT]
opae.io [-d,--device PCI_ADDR] [-r,--region REGION] peek OFFSET
opae.io [-d,--device PCI_ADDR] [-r,--region REGION] poke OFFSET VALUE
opae.io [-d,--device PCI_ADDR] [-r,--region REGION] script SCRIPT ARG1 ARG2 ... ARGN
opae.io [-d,--device PCI_ADDR] [-r,--region REGION]

DESCRIPTION

opae.io is an interactive Python environment packaged on top of libopaevfio.so, which provides user space access to PCIe devices via the vfio-pci driver. The main feature of opae.io is its built-in Python command interpreter, along with some Python bindings that provide a means to access Configuration and Status Registers (CSRs) that reside on the PCIe device.

Note: opae.io peek/poke will call fpgaEnumerate() internally and so a reset will be asserted on the AFU region. You can avoid initializing these registers by passing opae.io a python script directly.

opae.io has two operating modes: command line mode and interactive mode.

COMMAND LINE MODE

To view the accelerator devices that are present on the system, opae.io provides the ls command option.

opae.io ls [-v,--viddid VIDDID] [-s,--sub-viddid SUB_VIDDID] [--all] [--system-class]

Each accelerator device is listed along with the PCIe address, the PCIe vendor/device ID, a brief description of the device, and the driver to which the device is currently bound.

Device filtering is available by providing a Vendor ID:Device ID pair, eg -v 8086:bcce. Further filtering can be done by providing a sub- Vendor ID:sub-Device ID pair, eg -s 8086:1771. The --all option provides a list of all of the PCIe devices in the system, which an be quite verbose. The --system-class option prints the PCIe database class of the accelerator device, rather than the product name.

opae.io provides an option to initialize a PCIe device for use with the vfio-pci driver. In order for the device CSRs to be accessed from user space, the device must first be bound to the vfio-pci driver. This is the job of the init command option.

opae.io init [-d,--device PCI_ADDR] [USER[:GROUP]]

The init command unbinds the specified device from its current driver and binds it to vfio-pci. This creates a new vfio group under /dev/vfio. This group path is then used by the libopaevfio.so library to interact with the device.

To release the PCIe device from vfio-pci and return it to use with its previous driver, the release command option is used.

opae.io release [-d,--device PCI_ADDR]

The release command option reverses the actions of the last init command, releasing the device from vfio-pci and binding it to the driver which was bound at the time the init command was issued.

The walk command option traverses and displays the Device Feature List of the given region.

opae.io walk [--offset [OFFSET]] [-u,--show-uuid] [-D,--dump] [-c,--count COUNT] [-y,--delay DELAY] [-s,--safe]

The various fields of each Device Feature Header are displayed. The --show-uuid option additionally displays the GUID for each feature. OFFSET can be used to specify the beginning of the DFL in the MMIO region. --dump displays the raw DFH contents in hex format. COUNT limits the number of DFH entries traversed. DELAY causes a pause between each printout. --safe examines each DFH offset for proper alignment.

The dump command provides a means to dump the MMIO space in ASCII hex or binary format.

opae.io dump [--offset [OFFSET]] [-o,--output OUTPUT] [-f,--format {bin,hex}] [-c,--count COUNT]

OFFSET specifies the starting MMIO offset. OUTPUT gives the name of a file to capture the dump output, where sys.stdout is used by default. --format allows changing the output format. COUNT specifies the number of qwords to dump.

The peek command option reads and displays a CSR value.

opae.io peek OFFSET

The poke command option writes a given value to a CSR.

opae.io poke OFFSET VALUE

opae.io can also execute Python scripts from the command line. These Python scripts may contain calls to the device built-in functions that are available during an interactive session. Refer to the description of interactive mode for details.

opae.io script myscript.py a b c

In order to enter the interactive mode of opae.io, simply invoke it and optionally pass the desired device address and MMIO region options.

opae.io [-d,--device PCI_ADDR] [-r,--region REGION]

INTERACTIVE MODE

Upon entering interactive mode, opae.io begins a Python interpreter session and displays the command prompt shown below:

0000:3f:00.0[0]>>

The first portion of the prompt shows the address of the active PCIe device, here 0000:3f:00.0. The part in square brackets shows the active MMIO region, here [0].

The interpreter waits for a valid Python command, then attempts to execute the given command in the usual way. The only differences between the traditional Python command intepreter and opae.io are that opae.io provides 1) the notion of an active PCIe device and MMIO region and 2) several built-in functions and objects that allow manipulating the active device and MMIO region.

BUILT-IN FUNCTIONS

The opae.io built-in functions assume an active device and MMIO region. Attempting to use the built-in functions without first opening a device and region will result in errors.

peek(OFFSET)

The peek built-in function reads and displays a CSR value from the active device and region, at the offset supplied by its argument.

0000:3f:00.0[0]>> peek(0x28)
0xdeadbeef

poke(OFFSET, VALUE)

The poke built-in function writes the given VALUE to the current MMIO region at the given OFFSET.

0000:3f:00.0[0]>> poke(0x28, 0xdeadbeef)

read_csr(OFFSET)

The read_csr built-in function returns the value of the CSR at the active MMIO region and the given OFFSET.

0000:3f:00.0[0]>> print('0x{:0x}'.format(read_csr(0x28)))
0xdeadbeef

write_csr(OFFSET, VALUE)

The write_csr built-in function writes the given VALUE to the current MMIO region at the given OFFSET.

0000:3f:00.0[0]>> write_csr(0x28, 0xdeadbeef)

device(PCI_ADDR)

The device built-in function allows changing the active PCIe device.

0000:3f:00.0[0]>> device('0000:2b:00.0')
0000:2b:00.0>>

region(REGION)

The region built-in function allows changing the active MMIO region.

0000:2b:00.0>> region(0)
0000:2b:00.0[0]>>

allocate_buffer(SIZE)

The allocate_buffer built-in function creates and returns a DMA buffer object. The underlying buffer will be SIZE bytes in length.

0000:2b:00.0[0]>> b1 = allocate_buffer(4096)
0000:2b:00.0[0]>> print(b1.size, '0x{:0x}'.format(b1.address), b1.io_address)
4096 0x7f9361c66000 0

version()

The version built-in function returns a tuple containing the four components used to identify the opae.io version:

0000:2b:00.0[0]>> print(version())
('opae.io', 0, 2, 0)

BUILT-IN OBJECTS

opae.io interactive mode provides two global objects corresponding to the current device and that device's current MMIO region. These objects are referred to by global variables the_device and the_region, respectively.

The device class:

the_device.descriptor() : method that returns the integer file descriptor of the VFIO container.

0000:2b:00.0[0]>> print(the_device.descriptor())
5

the_device.repr() : method that is invoked when a device object is printed.

0000:2b:00.0[0]>> print(the_device)
0000:2b:00.0

the_device.allocate(SIZE) : method that allocates and returns a system_buffer object. The buffer will be mapped into the DMA space of the_device.

0000:2b:00.0[0]>> b1 = the_device.allocate(4096)

the_device.pci_address() : read-only property that returns the PCIe address of the_device.

0000:2b:00.0[0]>> print(the_device.pci_address)
0000:2b:00.0

the_device.num_regions : read-only property that returns the number of MMIO regions in the_device.

0000:2b:00.0[0]>> print(the_device.num_regions)
2

the_device.regions : read-only property that returns a list of the active MMIO regions of the_device:

0000:2b:00.0[0]>> print(the_device.regions)
[0, 2]

The region class:

the_region.write32(OFFSET, VALUE) : method that writes a 32-bit VALUE to the CSR at OFFSET.

the_region.read32(OFFSET) : method that returns a 32-bit CSR at the given OFFSET.

0000:2b:00.0[0]>> the_region.write32(0x28, 0xdeadbeef)
0000:2b:00.0[0]>> print('0x{:0x}'.format(the_region.read32(0x28)))
0xdeadbeef

the_region.write64(OFFSET, VALUE): method that writes a 64-bit VALUE to the CSR at OFFSET.

the_region.read64(OFFSET): method that returns a 64-bit CSR at the given OFFSET.

0000:2b:00.0[0]>> the_region.write64(0x28, 0xbaddecaf)
0000:2b:00.0[0]>> print('0x{:0x}'.format(the_region.read64(0x28)))
0xbaddecaf

the_region.index(): method that returns the MMIO index of the_region.

0000:2b:00.0[0]>> print(the_region.index())
0

the_region.repr(): method that is invoked when a region object is printed.

0000:2b:00.0[0]>> print(the_region)
0

the_region.len(): method that is invoked to determine the MMIO region size.

0000:2b:00.0[0]>> print(len(the_region))
524288

The allocate_buffer() built-in function and the device.allocate() method return objects of type system_buffer.

The system_buffer class is as follows:

buf.size: read-only property that gives the buffer size.

0000:2b:00.0[0]>> print(b1.size)
4096

buf.address: read-only property that gives the buffer's user mode virtual address.

0000:2b:00.0[0]>> print('0x{:0x}'.format(b1.address))
0x7f2c15d8200

buf.io_address: read-only property that gives the buffer's IO address.

0000:2b:00.0[0]>> print('0x{:0x}'.format(b1.io_address))
0x0

buf.__getitem__ and buf.__setitem__: indexing get/set of 64-bit data item.

0000:2b:00.0[0]>> b1[0] = 0xdecafbad
0000:2b:00.0[0]>> print('0x{:0x}'.format(b1[0]))
0xdecafbad

buf.read8(OFFSET)
buf.read16(OFFSET)
buf.read32(OFFSET)
buf.read64(OFFSET) : methods that read the given size data item from the given buffer OFFSET.

buf.fill8(VALUE)
buf.fill16(VALUE)
buf.fill32(VALUE)
buf.fill64(VALUE) : methods that fill the buffer with the given VALUE, using the given size.

b1.compare(b2): method that compares buffers. The method returns the index of the first byte that miscompares, or the length of b1.