OPAE Installation Guide¶

How to download the OPAE SDK¶

OPAE SDK releases are available on GitHub. Source code for the OPAE DFL device driver for Linux is also available on GitHub.

Install the Fedora¶

Download the Fedora (x86_64 version) installation file in fedora, and install the Fedora in yourserver. You can choose Fedora Workstation or Fedora server.

Build the kernel and DFL drivers¶

For building the OPAE kernel and kernel driver, the kernel development environment is required. So before you build the kernel, you must install the required packages. Run the following commands:

$ sudo dnf install gcc gcc-c++ make kernel-headers kernel-devel elfutils-libelf-devel ncurses-devel openssl-devel bison flex

Download the OPAE upstream kernel tree from github, for example download from fpga-ofs-dev-5.15-lts branch.

$ git clone https://github.com/OPAE/linux-dfl.git -b fpga-ofs-dev-5.15-lts

Configure the kernel.

$ cd linux-dfl
$ cp /boot/config-`uname -r` .config
$ cat configs/dfl-config >> .config
$ echo 'CONFIG_LOCALVERSION="-dfl"' >> .config
$ echo 'CONFIG_LOCALVERSION_AUTO=y' >> .config
$ sed -i -r 's/CONFIG_SYSTEM_TRUSTED_KEYS=.*/CONFIG_SYSTEM_TRUSTED_KEYS=""/' .config
$ sed -i '/^CONFIG_DEBUG_INFO_BTF/ s/./#&/' .config
$ echo 'CONFIG_DEBUG_ATOMIC_SLEEP=y' >> .config
$ make olddefconfig

Compile and install the new kernel.

$ make -j $(nproc)
$ sudo make modules_install -j $(nproc)
$ sudo make install

Build linux DFL Kernel instructions please also refer to: https://github.com/OPAE/linux-dfl/wiki/Build-the-linux-dfl-kernel

When install finished, reboot your system. When the system login again, verify the kernel version is correct. For example:

[figo@localhost linux-dfl]$ uname -a
Linux localhost.localdomain 5.15.lts-dfl-g73e16386cda0 #6 SMP Mon Jun 13 21:21:31 -04 2022 x86_64 x86_64 x86_64

And also you can check the OPAE dfl drivers have auto-loaded.

ifpga_sec_mgr fpga_region fpga_bridge fpga_mgr [figo@localhost dfl_eth_group dfl_fme_region dfl_emif dfl_n3000_nios dfl_fme_br dfl_fme_mgr dfl_fme dfl_afu dfl_pci dfl fpga_region fpga_bridge fpga_mgr

odelineno-5-1">[figo@localhost linux-dfl]$ lsmod | grep fpga 20480 1 intel_m10_bmc_secure 20480 3 dfl_fme_region,dfl_fme,dfl 24576 4 dfl_fme_region,fpga_region,dfl_fme,dfl_fme_br 16384 4 dfl_fme_region,fpga_region,dfl_fme_mgr,dfl_fme linux-dfl]$ lsmod | grep dfl 36864 0 20480 0 16384 0 20480 0 16384 0 20480 1 49152 0 36864 0 20480 0 40960 7 dfl_pci,dfl_fme,dfl_fme_br,dfl_eth_group,dfl_n3000_nios,dfl_afu,dfl_emif 20480 3 dfl_fme_region,dfl_fme,dfl 24576 4 dfl_fme_region,fpga_region,dfl_fme,dfl_fme_br 16384 4 dfl_fme_region,fpga_region,dfl_fme_mgr,dfl_fme

Build the OPAE-SDK¶

Before you build the OPAE SDK, you must install the required packages. Run the following commands:

Rocky Linux 8.5¶

# dnf install -y 'dnf-command(config-manager)'
# dnf config-manager --set-enabled powertools
# dnf install -y epel-release
# dnf check-update
# dnf upgrade -y
# dnf install -y python3 python3-pip python3-devel python3-jsonschema python3-pyyaml python3-pybind11 git gcc gcc-c++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel cli11-devel spdlog-devel libedit-devel systemd-devel rpm-build rpmdevtools pybind11-devel yaml-cpp-devel libudev-devel linuxptp
# python3 -m pip install jsonschema virtualenv pyyaml

Fedora¶

# dnf check-update
# dnf upgrade -y
# dnf install -y python3 python3-pip python3-devel python3-jsonschema python3-pyyaml python3-pybind11 git gcc g++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel libedit-devel rpm-build rpmdevtools pybind11-devel yaml-cpp-devel libudev-devel cli11-devel spdlog-devel linuxptp
# pip3 install jsonschema virtualenv pyyaml

Ubuntu 20.04¶

# apt-get update
# apt-get upgrade -y
# apt-get install -y python3 python3-pip python3-dev git gcc g++ make cmake uuid-dev libjson-c-dev libhwloc-dev libtbb-dev libedit-dev libudev-dev linuxptp pandoc devscripts debhelper doxygen
# pip3 install jsonschema virtualenv pyyaml pybind11

RHEL 8.2¶

Register and enable Red Hat subscription to install any packages on the system.

# subscription-manager register --proxy=PROXY --username=USER --password=PASSWORD --auto-attach

Set the RHEL version and install packages. Set proxy name and port number.

# subscription-manager release --set=8.2 --proxy proxy-name.com:port number
# subscription-manager repos --enable codeready-builder-for-rhel-8-x86_64-rpms
# dnf upgrade -y
# dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
# dnf install -y python3 python3-pip python3-devel gdb vim git gcc gcc-c++ make cmake libuuid-devel rpm-build systemd-devel  nmap
# dnf install -y python3-jsonschema json-c-devel tbb-devel rpmdevtools libcap-devel 
# dnf check-update || true
# dnf install -y spdlog-devel cli11-devel python3-pyyaml python3-pybind11 hwloc-devel libedit-devel
# python3 -m pip install --user jsonschema virtualenv pudb pyyaml

Install the latest version of cmake on top of the outdated cmake package from the package manager.

# cd cmake-3.25.1/
# ./bootstrap --prefix=/usr
# make
# make install
# which cmake
/usr/bin/cmake

Create opae-sdk packages¶

Download the OPAE-SDK source code from github. For example, download from Master branch.

$ git clone https://github.com/OPAE/opae-sdk.git

Compile and build the OPAE-SDK RPMs (Fedora, Rocky, RHEL 8.2).

$ cd opae-sdk/packaging/opae/rpm
$ ./create fedora

Note that if you find that your distribution has changed package names such that there is a conflict when building RPMs, you can install all of the build dependencies so that the SDK compiles and then build the RPMs in unrestricted mode:

$ cd opae-sdk/packaging/opae/rpm
$ ./create unrestricted

After a successful compile, there are 3 rpm packages generated (Fedora, Rocky, RHEL8.2). For example:

opae-2.1.0-1.fc34.x86_64.rpm
opae-devel-2.1.0-1.fc34.x86_64.rpm
opae-extra-tools-2.1.0-1.fc34.x86_64.rpm

Compile and build the OPAE-SDK deb packages (Ubuntu 22.04).

$ cd opae-sdk/packaging/opae/deb
$ ./create

After a successful compile, there are 3 deb packages generated (Ubuntu 22.04). For example:

opae_2.1.1-1_amd64.deb  
opae-devel_2.1.1-1_amd64.deb  
opae-extra-tools_2.1.1-1_amd64.deb

OPAE SDK installation with rpm/deb packages¶

The rpm packages generated in the previous step can be installed using these commands:

$ sudo dnf install ./*.rpm

The deb packages generated in the previous step can be installed using these commands:

$ sudo dpkg -i  ./*.deb

When you installed the rpms, you can run fpgainfo command to check the FPGA FME infomation. For example:

[figo@localhost install_guide]$ fpgainfo fme
Board Management Controller, MAX10 NIOS FW version: D.2.1.24
Board Management Controller, MAX10 Build version: D.2.0.7
//****** FME ******//
Object Id                        : 0xEF00000
PCIe s:b:d.f                     : 0000:08:00.0
Device Id                        : 0x0B30
Socket Id                        : 0x00
Ports Num                        : 01
Bitstream Id                     : 0x2300011001030F
Bitstream Version                : 0.2.3
Pr Interface Id                  : f3c99413-5081-4aad-bced-07eb84a6d0bb
Boot Page                        : user

To uninstall the OPAE rpms, you can use this commands

$ dnf list installed | grep opae
$ sudo dnf remove opae*.x86_64

To uninstall the OPAE deb, you can use this commands

$ dpkg -l  | grep opae
$ dpkg -r opae-extra-tools:amd64
$ dpkg -r opae-devel:amd64
$ dpkg -r opae

FPGA Device Access Permissions¶

Access to FPGA accelerators and devices is controlled using file access permissions on the Intel® FPGA device files, /dev/dfl-fme.* and /dev/dfl-port.*, as well as to the files reachable through /sys/class/fpga_region/.

In order to allow regular (non-root) users to access accelerators, you need to grant them read and write permissions on /dev/dfl-port.* (with * denoting the respective socket, i.e. 0 or 1). E.g.:

$ sudo chmod a+rw /dev/dfl-port.0

Memlock limit¶

Depending on the requirements of your application, you may also want to increase the maximum amount of memory a user process is allowed to lock. The exact way to do this depends on your Linux distribution.

You can check the current memlock limit using

$ ulimit -l

A way to permanently remove the limit for locked memory for a regular user is to add the following lines to your /etc/security/limits.conf:

user1    hard   memlock           unlimited
user1    soft   memlock           unlimited

This removes the limit on locked memory for user user1. To remove it for all users, you can replace user1 with *:

*    hard   memlock           unlimited
*    soft   memlock           unlimited

Note that settings in the /etc/security/limits.conf file don't apply to services. To increase the locked memory limit for a service you need to modify the application's systemd service file and add the line:

[Service]
LimitMEMLOCK=infinity

Hugepage Settings¶

Users need to configure system hugepages to reserve 2MB-hugepages or 1GB-hugepages. For example, the 'hello_fpga' sample requires several 2MB-hugepages. And the fpgadiag tool requires several 1GB-hugepages.

The command below reserves 20 2M-hugepages:

$ sudo sh -c 'echo 20 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages'

The command below reserves 4 1GB-hugepages:

$ sudo sh -c 'echo 4 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages'

For x86_64 architecture processors, user can use following command to find out avaiable hugepage sizes:

$ grep pse /proc/cpuinfo | uniq
flags : ... pse ...

If this commands returns a non-empty string, 2MB pages are supported.

$ grep pse /proc/cpuinfo | uniq
flags : ... pdpe1gb ...

If this commands returns a non-empty string, 1GB pages are supported.