AFU Developer Guide: OFS for Agilex® 7 FPGA PCIe Attach FPGAs¶

Last updated: May 13, 2024

1. Introduction¶

This document is a design guide for the creation of an Accelerator Functional Unit (AFU) using Open FPGA Stack (OFS) for Agilex® 7 FPGAs PCIe Attach. The AFU concept consists of separating out the FPGA design development process into two parts, the construction of the foundational FPGA Interface Manager (FIM), and the development of the Acceleration Function Unit (AFU), as shown in the diagram below.

This diagram shows the separation of FPGA board interface development from the internal FPGA workload creation. This separation starts with the FPGA Interface Manager (FIM) which consists of the external interfaces and board management functions. The FIM is the base system layer and is typically provided by board vendors. The FIM interface is specific to a particular physical platform. The AFU makes use of the external interfaces with user defined logic to perform a specific application. By separating out the lengthy and complicated process of developing and integrating external interfaces for an FPGA into a board allows the AFU developer to focus on the needs of their workload. OFS for Agilex® 7 FPGAs PCIe Attach provides the following tools for rapid AFU development:

Scripts for both compilation and simulation setup
Optional Platform Interface Manager (PIM) which is a set of SystemVerilog shims and scripts for flexible FIM to AFU interfacing
Acceleration Simulation Environment (ASE) which is a hardware/software co-simulation environment scripts for compilation and Acceleration
Integration with Open Programmable Acceleration Engine (OPAE) SDK for rapid software development for your AFU application

Please notice in the above block diagram that the AFU region consists of static and partial reconfiguration (PR) regions where the PR region can be dynamically reconfigured while the remaining FPGA design continues to function. Creating AFU logic for the static region is described in Shell Developer Guide for the associated platform. This guide covers logic in the AFU Main region.

1.1. Document Organization¶

This document is organized as follows:

Description of design flow
Interfaces and functionality provided in the Agilex® 7 FPGAs PCIe Attach FIM
Setup of the AFU Development environment
Synthesize the AFU example
Testing the AFU example on the Intel® FPGA SmartNIC N6001-PL card
Hardware/Software co-simulation using ASE
Debugging an AFU with Remote Signal Tap

This guide provides theory followed by tutorial steps to solidify your AFU development knowledge.

NOTE:

This guide uses the Intel® FPGA SmartNIC N6001-PL as the platform for all tutorial steps. Additionally, this guide and the tutorial steps can be used with other platforms, such as the Agilex® 7 FPGA F-Series Development Kit (2x F-Tile).

Some of the document links in this guide are specific to the Intel® FPGA SmartNIC N6001-PL. If using a different platform, please use the associated documentation for your platform as there could be differences in building the FIM and downloading FIM images.

If you have worked with previous Altera® Programmable Acceleration products, you will find out that OFS for Agilex® 7 FPGAs PCIe Attach is similar. However, there are differences and you are advised to carefully read and follow the tutorial steps to fully understand the design tools and flow.

1.2. Prerequisite¶

This guide assumes you have the following FPGA logic design-related knowledge and skills:

FPGA compilation flows including the Quartus® Prime Pro Edition design flow
Static Timing closure, including familiarity with the Timing Analyzer tool in Quartus® Prime Pro Edition software, applying timing constraints, Synopsys* Design Constraints (.sdc) language and Tcl scripting, and design methods to close on timing critical paths.
RTL and coding practices to create synthesizable logic.
Understanding of AXI and Avalon memory mapped and streaming interfaces.
Simulation of complex RTL using industry standard simulators (Synopsys® VCS® or Siemens® QuestaSim®).
Signal Tap Logic Analyzer tool in the Quartus® Prime Pro Edition software.

You are strongly encouraged to review the Shell Developer Guide for the associated platform.

1.3. Acceleration Functional Unit (AFU) Development Flow¶

The AFU development flow is shown below:

1.3.1. Understanding Platform Capabilities¶

The block diagram of the N6001 Board is shown below:

The N6001 FIM provided with this release is shown below:

This release FIM provides the following features:

Host interface
- PCIe Gen4 x 16
- 5 - PF, 4 - VF, AXI-S TLP packets
- MSI-X interrupts
- Logic to demonstrate simple PCIe loopback
Network interface
- 2 - QSFP28/56 cages
- 2 x 4 x 25 GbE with exerciser logic demonstrating traffic generation/monitoring
External Memory - DDR4 - 2400
- HPS - 1GB organized as 256 Mb x 32 with 256 Mb x 8 ECC
- Channel 0, 1 - 4 GB organized as 1 Gb x 32
- Channel 2, 3 - 4 GB organized as 1 Gb x 32 with 1 Gb x 8 ECC (ECC is not implemented in this release)
- Memory exerciser logic demonstrating external memory operation
Board Management
- SPI interface
- FPGA configuration
- Example logic showing DFH operation
Remote Signal Tap logic
Partial reconfiguration control logic
ARM HPS subsystem with embedded Linux
- HPS Copy engine

1.3.2. High Level Data Flow¶

The OFS high level data flow is shown below:

1.3.3. Considerations for PIM Usage¶

When creating an AFU, a designer needs to decide what type of interfaces the platform (FIM) should provide to the AFU. The FIM can provide the native interfaces (i.e. PCIe TLP commands) or standard memory mapped interfaces (i.e. AXI-MM or AVMM) by using the PIM. The PIM is an abstraction layer consisting of a collection of SystemVerilog interfaces and shims to enable partial AFU portability across hardware despite variations in hardware topology and native interfaces. The PIM adds a level of logic between the AFU and the FIM converting the native interfaces from the FIM to match the interfaces provided by the AFU.

The following resources are available to assist in creating an AFU:

PIM Core Concepts provides details on using the PIM and its capabilities.

PIM Based AFU Developer Guide provides details on interfacing your AFU to the FIM using the PIM.

The examples-afu repo provides several AFU examples:

Example	Description	PIM-based	Hybrid	Native
clocks	Example AFU using user configurable clocks.	X
copy_engine	Example AFU moving data between host channel and a data engine.	X
dma	Example AFU moving data between host channel and local memory with a DMA.	X
hello_world	Example AFU sending "Hello World!" to host channel.	X	X	X
local_memory	Example AFU reading and writing local memory.	X	X

These examples can be run with the current OFS FIM package. There are three AFU types of examples provided (PIM based, hybrid and native). Each example provides the following:

RTL, which includes the following interfaces:
- Host Channel:
  - Host memory, providing a DMA interface.
  - MMIO, providing a CSR interface.
- Local Memory
Host software example interfacing to the CSR interface and host memory interface, using the OPAE C API.
Accelerator Description File .json file
Source file list

1.3.4. AFU Interfaces Included with Intel® FPGA SmartNIC N6001-PL¶

The figure below shows the interfaces available to the AFU in this architecture. It also shows the design hierarchy with module names from the fim (top.sv) to the PR region AFU (afu_main.sv). One of the main differences from the Stratix 10 PAC OFS architecture to this one is the presence of the static port gasket region (port_gasket.sv) that has components to facilitate the AFU and also consists of the PR region (afu_main.sv) via the PR slot. The Port Gasket contains all the PR specific modules and logic, e.g., PR slot reset/freeze control, user clock, remote STP etc. Architecturally, a Port Gasket can have multiple PR slots where user workload can be programmed into. However, only one PR slot is supported for OFS Release for Agilex®. Everything in the Port Gasket until the PR slot should be provided by the FIM developer. The task of the AFU developer is to add their desired application in the afu_main.sv module by stripping out unwanted logic and instantiating the target accelerator. As shown in the figure below, here are the interfaces connected to the AFU (highlighted in green) via the PCIe Attach FIM:

AXI Streaming (AXI-S) interface to the Host via PCIe Gen4x16
AXI Memory Mapped Channels (4) to the DDR4 EMIF interface
AXI Streaming (AXI-S) interface to the HSSI 25 Gb Ethernet

2. Set Up AFU Development Environment¶

This section covers the setup of the AFU development environment.

2.1. AFU development environment overview¶

A typical development and hardware test environment consists of a development server or workstation with FPGA development tools installed and a separate server with the target OFS compatible FPGA PCIe card installed. The typical usage and flow of data between these two servers is shown below:

Note: both development and hardware testing can be performed on the same server if desired.

This guide uses Intel® FPGA SmartNIC N6001-PL as the target OFS compatible FPGA PCIe card for demonstration steps. The Intel® FPGA SmartNIC N6001-PL must be fully installed following the Board Installation Guide: OFS for Agilex® 7 PCIe Attach Development Kits. If using a different OFS FPGA PCIe card, contact your supplier for instructions on how to install and operate user developed AFUs.

The following is a summary of the steps to set up for AFU development:

Install Quartus Prime Pro Version 23.4 for Linux with Agilex device support and required Quartus patches.
Make sure support tools are installed and meet version requirements.
Install OPAE SDK.
Download the Basic Building Blocks repository.
Build or download the relocatable AFU PR-able build tree based on your Agilex® FPGA PCIe Attach FIM.
Download FIM to the Agilex® FPGA PCIe Attach platform.

Building AFUs with OFS for Agilex® requires the build machine to have at least 64 GB of RAM.

2.2. Installation of Quartus and required patches¶

Intel Quartus Prime Pro Version 23.4 is verified to work with the latest OFS release ofs-2024.1-1. However, you have the option to port and verify the release on newer versions of Intel Quartus Prime Pro software.

Use RedHat® Enterprise Linux® (RHEL) 8.6 for compatibility with your development flow and also testing your FIM design in your platform.

Prior to installing Quartus:

Ensure you have at least 64 GB of free space for Quartus Prime Pro installation and your development work.
- Intel® recommends that your system be configured to provide virtual memory equal in size or larger than the recommended physical RAM size that is required to process your design.
- The disk space may be significantly more based on the device families included in the install. Prior to installation, the disk space should be enough to hold both zipped tar files and uncompressed installation files. After successful installation, delete the downloaded zipped files and uncompressed zip files to release the disk space.

Perform the following steps to satisfy the required dependencies.

$ sudo dnf install -y gcc gcc-c++ make cmake libuuid-devel rpm-build autoconf automake bison boost boost-devel libxml2 libxml2-devel make ncurses grub2 bc csh flex glibc-locale-source libnsl ncurses-compat-libs

Apply the following configurations.

$ sudo localedef -f UTF-8 -i en_US en_US.UTF-8 
$ sudo ln -s /usr/lib64/libncurses.so.6 /usr/lib64/libncurses.so.5 
$ sudo ln -s /usr/bin/python3 /usr/bin/python

Create the default installation path: /intelFPGA_pro/, where is the default path of the Linux workstation, or as set by the system administrator and is your Quartus version number.

The installation path must satisfy the following requirements:
- Contain only alphanumeric characters
- No special characters or symbols, such as !$%@^&*<>,
- Only English characters
- No spaces
Download your required Quartus Prime Pro Linux version here.
Install required Quartus patches. The Quartus patch .run files can be found in the Assets tab on the OFS Release GitHub page. The patches for this release are 0.17 patch (PCIe).
After running the Quartus Prime Pro installer, set the PATH environment variable to make utilities quartus, jtagconfig, and quartus_pgm discoverable. Edit your bashrc file ~/.bashrc to add the following line:
```
export PATH=<Quartus install directory>/quartus/bin:$PATH
export PATH=<Quartus install directory>/qsys/bin:$PATH
```
For example, if the Quartus install directory is /home/intelFPGA_pro/23.4 then the new line is:
```
export PATH=/home/intelFPGA_pro/23.4/quartus/bin:$PATH
export PATH=/home/intelFPGA_pro/23.4/qsys/bin:$PATH
```

Verify, Quartus is discoverable by opening a new shell:

$ which quartus
/home/intelFPGA_pro/23.4/quartus/bin/quartus

2.3. Installation of Support Tools¶

Make sure support tools are installed and meet version requirements.

The OFS provided Quartus build scripts require the following tools. Verify these are installed in your development environment.

Item	Version
Python	3.6.8
GCC	8.5.0
cmake	3.15
git	1.8.3.1
perl	5.8.8

2.4. Installation of OPAE SDK¶

Working with the Intel® FPGA SmartNIC N6001-PL card requires opae-2.12.0-5. Follow the instructions in the Follow the instructions in the Software Installation Guide: OFS for PCIe Attach FPGAs to build and install the required OPAE SDK for the Intel® FPGA SmartNIC N6001-PL. Make sure to check out the cloned repository to tag 2.12.0-5 and branch release/2.12.0.

$ git checkout tags/2.12.0-5 -b release/2.12.0

Note: The tutorial steps provided in the next sections assume the OPAE SDK is installed in default system locations, under the directory, /usr. In most system configurations, this will allow the OS and tools to automatically locate the OPAE binaries, scripts, libraries and include files required for the compilation and simulation of the FIM and AFUs.

2.5. Download the Basic Building Blocks repositories¶

The ofs-platform-afu-bbb repository contains the PIM files as well as example PIM-based AFUs that can be used for testing and demonstration purposes. This guide will use the host_chan_mmio AFU example in the ofs-platform-afu-bbb repository and the hello_world sample accompanying the examples-afu repository to demonstrate how to synthesize, load, simulate, and test a PIM-based AFU using the Intel® FPGA SmartNIC N6001-PL card with the PCIe Attach FIM.

Execute the next commands to clone the BBB repository.

  # Create top level directory for AFU development
$ mkdir OFS_BUILD_ROOT
$ cd OFS_BUILD_ROOT
$ export OFS_BUILD_ROOT=$PWD

  # Clone the ofs-platform-afu-bbb repository.
$ cd $OFS_BUILD_ROOT
$ git clone https://github.com/OFS/ofs-platform-afu-bbb.git
$ cd ofs-platform-afu-bbb
$ git checkout tags/ofs-2024.1-1
$ export OFS_PLATFORM_AFU_BBB=$PWD

  # Verify retrieval
$ ls
LICENSE  plat_if_develop  plat_if_release  plat_if_tests  README.md

The documentation in the ofs-platform-afu-bbb repository further addresses - The PIM concept. - The structure of the PIM-based AFU examples. - How to generate a release and configure the PIM. - How to connect an AFU to an FIM.

2.6. Build or download the relocatable PR build tree¶

A relocatable PR build tree is needed to build the AFU partial reconfiguration area for the intended FIM. The tree is relocatable and may be copied to a new location. It does not depend on files in the original FIM build.

You can use the Intel® FPGA SmartNIC N6001-PL release package and download the PR build tree and FIM images, to develop your AFU. These are located at OFS-N6001 release

Or you can build your own FIM and generate the PR build tree during the process.

To download and untar the pr_build_template:

$ cd $OFS_BUILD_ROOT
$ wget https://github.com/OFS/ofs-agx7-pcie-attach/releases/download/ofs-2024.1-1/n6001-images_ofs-2024-1-1.tar.gz
$ tar -zxvf n6001-images_ofs-2024-1-1.tar.gz
$ cd n6001-images_ofs-2024-1-1/
$ mkdir pr_build_template
$ tar -zxvf pr_build_template.tar.gz -C ./pr_build_template
$ cd pr_build_template
$ export OPAE_PLATFORM_ROOT=$PWD

To build your own FIM and generate the PR build tree for the Intel® FPGA SmartNIC N6001-PL platform, refer to the Shell Developer Guide: OFS for Agilex® 7 PCIe Attach (P-tile, E-tile) FPGAs and follow the Out-of-Tree PR FIM build flow. If you are using a different platform, refer to the Shell Developer Guide for your platform and follow the Out-of-Tree PR FIM build flow.

2.7. Download FIM to FPGA¶

The AFU requires that the FIM from which the AFU is derived be loaded onto the FPGA.

If you are using the Intel® FPGA SmartNIC N6001-PL release package downloaded in the previous section:

$ cd $OFS_BUILD_ROOT/n6001-images_ofs-2024-1-1/

If you are generating your own FIM, use the unsigned FPGA binary images from your FIM build.

Downlaod the FIM to the Intel® FPGA SmartNIC N6001-PL platform. If you are running on a Virtual Machine, refer to the KVM User Guide: Open FPGA Stack for passing the devices to the VM.

$ sudo fpgasupdate ofs_top_page1_unsigned_user1.bin <N6001 SKU2 PCIe b:d.f>
$ sudo fpgasupdate ofs_top_page2_unsigned_user2.bin <N6001 SKU2 PCIe b:d.f>
$ sudo rsu fpga --page=user1 <N6001 SKU2 PCIe b:d.f>

If you are using a different platform, refer to the documentation for your platform to download the FIM images onto your Agilex® PCIe Attach Platform.

2.8. Set up required Environment Variables¶

Set the required environment variables as shown below. These environment variables must be set prior to simulation or compilation tasks. You can create a simple script to set these variables and save time going forward.

# If not already done, export OFS_BUILD_ROOT to the top level directory for AFU development
$ export OFS_BUILD_ROOT=<path to ofs build directory>

# If not already done, export OPAE_PLATFORM_ROOT to the PR build tree directory
$ export OPAE_PLATFORM_ROOT=<path to pr build tree>

# If not already done, export OFS_PLATFORM_AFU_BBB to the clone of ofs-platform-afu-bbb repository which contains PIM files and AFU examples.
$ export OFS_PLATFORM_AFU_BBB=<path to ofs-platform-afu-bbb>

# Quartus Tools
# Note, QUARTUS_HOME is your Quartus installation directory, e.g. $QUARTUS_HOME/bin contains Quartus executable.
$ export QUARTUS_HOME=<user_path>/intelFPGA_pro/23.4/quartus
$ export QUARTUS_ROOTDIR=$QUARTUS_HOME
$ export QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR
$ export QUARTUS_ROOTDIR_OVERRIDE=$QUARTUS_ROOTDIR
$ export IMPORT_IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip
$ export IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip
$ export QSYS_ROOTDIR=$QUARTUS_ROOTDIR/../qsys
$ export PATH=$QUARTUS_HOME/bin:$QSYS_ROOTDIR/bin:$QUARTUS_HOME/../sopc_builder/bin/:$PATH

# OPAE SDK release
$ export OPAE_SDK_REPO_BRANCH=release/2.12.0

# OPAE and MPF libraries must either be on the default linker search paths or on both LIBRARY_PATH and LD_LIBRARY_PATH.  
$ export OPAE_LOC=/usr
$ export LIBRARY_PATH=$OPAE_LOC/lib:$LIBRARY_PATH
$ export LD_LIBRARY_PATH=$OPAE_LOC/lib64:$LD_LIBRARY_PATH

3. Compiling an AFU¶

In this section, you will use the relocatable PR build tree created in the previous steps from the FIM to compile an example PIM-based AFU. This section will be developed around the host_chan_mmio and hello_world AFU examples to showcase the synthesis of a PIM-based AFU.

The build steps presented below demonstrate the ease in building and running an actual AFU on the board. To successfully execute the instructions in this section, you must have set up your development environment and have a relocateable PR Build tree as instructed in section 2 of this document.

3.1. Creating the AFU Synthesis Environment¶

The PIM flow provides the script afu_synth_setup to create the synthesis environment to build the AFU examples. See how to use it below.

usage: afu_synth_setup [-h] -s SOURCES [-p PLATFORM] [-l LIB] [-f] dst

Generate a Quartus build environment for an AFU. A build environment is
instantiated from a release and then configured for the specified AFU. AFU
source files are specified in a text file that is parsed by rtl_src_config,
which is part of the OPAE base environment.

positional arguments:
  dst                   Target directory path (directory must not exist).

optional arguments:
  -h, --help            show this help message and exit
  -s SOURCES, --sources SOURCES
                        AFU source specification file that will be passed to
                        rtl_src_config. See "rtl_src_config --help" for the
                        file's syntax. rtl_src_config translates the source
                        list into either Quartus or RTL simulator syntax.
  -p PLATFORM, --platform PLATFORM
                        FPGA platform name.
  -l LIB, --lib LIB     FPGA platform release hw/lib directory. If not
                        specified, the environment variables OPAE_FPGA_HW_LIB
                        and then BBS_LIB_PATH are checked.
  -f, --force           Overwrite target directory if it exists.

3.2. Building and Running host_chan_mmio example AFU¶

The $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio is a simple example demonstrating both hardware and software access to an AFU. The host_chan_mmio example AFU consists of the following files. The hw directory contains the RTL to implement the hardware functionality using Avalon and AXI interfaces. However, this guide will use the AXI version of the host_chan_mmio AFU to go through the compilation steps. The sw directory of the AFU contains the source code of the host application that communicates with the actual AFU hardware.

host_chan_mmio
├── hw
│   └── rtl
│       ├── avalon
│       │   ├── afu_avalon512.sv
│       │   ├── afu_avalon.sv
│       │   ├── ofs_plat_afu_avalon512.sv
│       │   ├── ofs_plat_afu_avalon_from_ccip.sv
│       │   └── ofs_plat_afu_avalon.sv
│       ├── axi
│       │   ├── afu_axi512.sv
│       │   ├── afu_axi.sv
│       │   ├── ofs_plat_afu_axi512.sv
│       │   ├── ofs_plat_afu_axi_from_ccip.sv
│       │   └── ofs_plat_afu_axi.sv
│       ├── host_chan_mmio.json
│       ├── test_mmio_avalon0_from_ccip.txt
│       ├── test_mmio_avalon1.txt
│       ├── test_mmio_avalon2_512rw.txt
│       ├── test_mmio_axi0_from_ccip.txt
│       ├── test_mmio_axi1.txt
│       └── test_mmio_axi2_512rw.txt
└── sw
    ├── main.c
    ├── Makefile

3.2.1. Build the host_chan_mmio example AFU¶

Execute afu_synth_setup as follows to create the synthesis environment for a host_chan_mmio AFU that fits the PCIe Attach FIM previously constructed.

$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio
$ afu_synth_setup -s ./hw/rtl/test_mmio_axi1.txt afu_dev

Now, move into the synthesis environment afu_dev directory just created. From there, execute the afu_synth command. The successful completion of the command will produce the host_chan_mmio.gbs file under the synthesis environment directory, $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_dev.

$ cd afu_dev
$ $OPAE_PLATFORM_ROOT/bin/afu_synth
Compiling ofs_top ofs_pr_afu
Generating host_chan_mmio.gbs
==================================
...
...
===========================================================================
 PR AFU compilation complete
 AFU gbs file is 'host_chan_mmio.gbs'
 Design meets timing
===========================================================================

The previous output indicates the successful compilation of the AFU and the compliance with the timing requirements. Analyze the reports generated in case the design does not meet timing. The timing reports are stored in the directory, $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_dev/build/syn/board/n6001/syn_top/output_files/timing_report.

Once the compilation finishes successfully, load the new host_chan_mmio.gbs bitstream file into the partial reconfiguration region of the target Intel® FPGA SmartNIC N6001-PL board. Keep in mind, that the loaded image is dynamic - this image is not stored in flash and if the card is power cycled, then the PR region is re-loaded with the default AFU.

3.2.2. Download the host_chan_mmio example AFU¶

To test the AFU in actual hardware, load the host_chan_mmio.gbs to the Intel® FPGA SmartNIC N6001-PL card. For this step to be successful, the PCIe Attach FIM must have already been loaded to the Intel® FPGA SmartNIC N6001-PL card following the steps described in Section 2 of this document. If you are running on a Virtual Machine, refer to the KVM User Guide: Open FPGA Stack for passing the devices to the VM.

Verify Board and PCIe b.d.f. For the following example, the N6001 SKU2 PCIe b:d.f is B1:00.0, however this may be different in your system.

$ fpgainfo fme
Intel Acceleration Development Platform N6001
Board Management Controller NIOS FW version: 3.15.0
Board Management Controller Build version: 3.15.0
//****** FME ******//
Interface                        : DFL
Object Id                        : 0xEF00000
PCIe s:b:d.f                     : 0000:B1:00.0
Vendor Id                        : 0x8086
Device Id                        : 0xBCCE
SubVendor Id                     : 0x8086
SubDevice Id                     : 0x1771
...

Download AFU.

$ cd $OFS_PLATFORM_AFU_BBB/plat_if_tests/host_chan_mmio/afu_dev
$ sudo fpgasupdate host_chan_mmio.gbs B1:00.0
[sudo] password for <<Your username>>: 
[2022-04-15 20:22:18.85] [WARNING ] Update starting. Please do not interrupt.
[2022-04-15 20:22:19.75] [INFO    ] 
Partial Reconfiguration OK
[2022-04-15 20:22:19.75] [INFO    ] Total time: 0:00:00.90

3.2.3. Set up host to interface with example AFU¶

Set up host to interface with the newly loaded AFU.

List the PFs available, the default N6001 FIM has 5 PFs.

$ lspci -s B1:00
B1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)
B1:00.1 Processing accelerators: Intel Corporation Device bcce (rev 01)
B1:00.2 Processing accelerators: Intel Corporation Device bcce (rev 01)
B1:00.3 Processing accelerators: Intel Corporation Device bcce (rev 01)
B1:00.4 Processing accelerators: Intel Corporation Device bcce (rev 01)

Create the Virtual Functions (VFs) provided by the FIM, the default N6001 FIM has 3 VFs. If your FIM uses only PFs, skip this step.

$ sudo pci_device B1:00.0 vf 3

# Verify the VFs have been added (device id: bccf)
$ lspci -s B1:00
B1:00.0 Processing accelerators: Intel Corporation Device bcce (rev 01)
B1:00.1 Processing accelerators: Intel Corporation Device bcce (rev 01)
B1:00.2 Processing accelerators: Intel Corporation Device bcce (rev 01)
B1:00.3 Processing accelerators: Intel Corporation Device bcce (rev 01)
B1:00.4 Processing accelerators: Intel Corporation Device bcce (rev 01)
B1:00.5 Processing accelerators: Intel Corporation Device bccf
B1:00.6 Processing accelerators: Intel Corporation Device bccf
B1:00.7 Processing accelerators: Intel Corporation Device bccf

Bind PFs and VFs to VFIO driver (except PF0/B1:00.0, which is the FME PF).

# Enter your username.
$ sudo opae.io init -d 0000:b1:00.1 <<Your username>>
[sudo] password for <<Your username>>:
Unbinding (0x8086,0xbcce) at 0000:b1:00.1 from dfl-pci
Binding (0x8086,0xbcce) at 0000:b1:00.1 to vfio-pci
iommu group for (0x8086,0xbcce) at 0000:b1:00.1 is 183
Assigning /dev/vfio/183 to ceg
Changing permissions for /dev/vfio/183 to rw-rw----

$ sudo opae.io init -d 0000:b1:00.2 <<Your username>>
[sudo] password for <<Your username>>:
Unbinding (0x8086,0xbcce) at 0000:b1:00.2 from dfl-pci
Binding (0x8086,0xbcce) at 0000:b1:00.2 to vfio-pci
iommu group for (0x8086,0xbcce) at 0000:b1:00.2 is 184
Assigning /dev/vfio/184 to ceg
Changing permissions for /dev/vfio/184 to rw-rw----

$ sudo opae.io init -d 0000:b1:00.3 <<Your username>>
[sudo] password for <<Your username>>:
Unbinding (0x1af4,0x1000) at 0000:b1:00.3 from virtio-pci
Binding (0x1af4,0x1000) at 0000:b1:00.3 to vfio-pci
iommu group for (0x1af4,0x1000) at 0000:b1:00.3 is 185
Assigning /dev/vfio/185 to ceg
Changing permissions for /dev/vfio/185 to rw-rw----

$ sudo opae.io init -d 0000:b1:00.4 <<Your username>>
[sudo] password for <<Your username>>: 
Unbinding (0x8086,0xbcce) at 0000:b1:00.4 from dfl-pci
Binding (0x8086,0xbcce) at 0000:b1:00.4 to vfio-pci
iommu group for (0x8086,0xbcce) at 0000:b1:00.4 is 186
Assigning /dev/vfio/186 to ceg
Changing permissions for /dev/vfio/186 to rw-rw----

$ sudo opae.io init -d 0000:b1:00.5 <<Your username>>
[sudo] password for <<Your username>>:
Binding (0x8086,0xbccf) at 0000:b1:00.5 to vfio-pci
iommu group for (0x8086,0xbccf) at 0000:b1:00.5 is 315
Assigning /dev/vfio/315 to ceg
Changing permissions for /dev/vfio/315 to rw-rw----

$ sudo opae.io init -d 0000:b1:00.6 <<Your username>>
[sudo] password for <<Your username>>:
Binding (0x8086,0xbccf) at 0000:b1:00.6 to vfio-pci
iommu group for (0x8086,0xbccf) at 0000:b1:00.6 is 316
Assigning /dev/vfio/316 to ceg
Changing permissions for /dev/vfio/316 to rw-rw----

$ sudo opae.io init -d 0000:b1:00.7 <<Your username>>
[sudo] password for <<Your username>>:
Binding (0x8086,0xbccf) at 0000:b1:00.7 to vfio-pci
iommu group for (0x8086,0xbccf) at 0000:b1:00.7 is 317
Assigning /dev/vfio/317 to ceg
Changing permissions for /dev/vfio/317 to rw-rw----

Verify the new

$//****** PORTObject Id PCIe s:b:d.f Vendor Id Device Id SubVendor Id SubDevice Id Socket Id //****** PORTObject Id PCIe s:b:d.f Vendor Id Device Id SubVendor Id SubDevice Id Socket Id Accelerator GUID //****** PORTObject Id PCIe s:b:d.f Vendor Id Device Id SubVendor Id SubDevice Id Socket Id Accelerator GUID //****** PORTObject Id PCIe s:b:d.f Vendor Id Device Id SubVendor Id SubDevice Id Socket Id Accelerator GUID //****** PORTObject Id PCIe s:b:d.f Vendor Id Device Id SubVendor Id SubDevice Id Socket Id Accelerator GUID //****** PORTObject Id PCIe s:b:d.f Vendor Id Device Id SubVendor Id SubDevice Id Socket Id Accelerator GUID //****** PORTObject Id PCIe s:b:d.f Vendor Id Device Id SubVendor Id SubDevice Id Socket Id Accelerator GUID //****** PORTObject Id PCIe s:b:d.f Vendor Id Device Id SubVendor Id SubDevice Id Socket Id Accelerator GUID

AFU Developer Guide: OFS for Agilex® 7 FPGA PCIe Attach FPGAs¶

1. Introduction¶

1.1. Document Organization¶

1.2. Prerequisite¶

1.3. Acceleration Functional Unit (AFU) Development Flow¶

1.3.1. Understanding Platform Capabilities¶

1.3.2. High Level Data Flow¶

1.3.3. Considerations for PIM Usage¶

1.3.4. AFU Interfaces Included with Intel® FPGA SmartNIC N6001-PL¶

2. Set Up AFU Development Environment¶

2.1. AFU development environment overview¶

2.2. Installation of Quartus and required patches¶

2.3. Installation of Support Tools¶

2.4. Installation of OPAE SDK¶

2.5. Download the Basic Building Blocks repositories¶

2.6. Build or download the relocatable PR build tree¶

2.7. Download FIM to FPGA¶

2.8. Set up required Environment Variables¶

3. Compiling an AFU¶

3.1. Creating the AFU Synthesis Environment¶

3.2. Building and Running host_chan_mmio example AFU¶

3.2.1. Build the host_chan_mmio example AFU¶

3.2.2. Download the host_chan_mmio example AFU¶

3.2.3. Set up host to interface with example AFU¶

3.2.4. Run the host_chan_mmio example AFU¶

3.3. Building and running the hello_world example AFU¶

3.3.1. Build the hello_world example AFU¶

3.3.2. Download the hello_world example AFU¶

3.3.3. Set up host to interface with example AFU¶

3.3.4. Run the hello_world example AFU¶

3.4. Modify the AFU user clocks frequency¶

4. Simulating an AFU using ASE¶

4.1. Set Up Steps to Run ASE¶

4.1.1. Install OPAE SDK¶

4.1.2 Install ASE Tools¶

4.1.3. Setup Required ASE Environment Variables¶

4.2. Simulating the host_chan_mmio AFU¶

4.2.1 Set Up and Run the HW Simulation Process¶

4.2.2 Set Up and Run the SW Process¶

4.3 Simulating the hello_world AFU¶

5. Adding Remote Signal Tap Logic Analyzer to debug the AFU¶

5.1. Adding RSTP to the host_chan_mmio AFU¶

6. How to modify the PF/VF MUX configuration¶

Notices & Disclaimers¶