OPAE Installation Guide
How to download the OPAE SDK
OPAE SDK releases are available on GitHub. Source code for the OPAE DFL device driver for Linux is also available on GitHub.
Install the Fedora
Download the Fedora (x86_64 version) installation file in fedora, and install the Fedora in yourserver. You can choose Fedora Workstation or Fedora server.
Build the kernel and DFL drivers
For building the OPAE kernel and kernel driver, the kernel development environment is required. So before you build the kernel, you must install the required packages. Run the following commands:
$ sudo dnf install gcc gcc-c++ make kernel-headers kernel-devel elfutils-libelf-devel ncurses-devel openssl-devel bison flex
Download the OPAE upstream kernel tree from github, for example download from fpga-ofs-dev-5.15-lts branch.
Configure the kernel.
$ cd linux-dfl
$ cp /boot/config-`uname -r` .config
$ cat configs/dfl-config >> .config
$ echo 'CONFIG_LOCALVERSION="-dfl"' >> .config
$ echo 'CONFIG_LOCALVERSION_AUTO=y' >> .config
$ sed -i -r 's/CONFIG_SYSTEM_TRUSTED_KEYS=.*/CONFIG_SYSTEM_TRUSTED_KEYS=""/' .config
$ sed -i '/^CONFIG_DEBUG_INFO_BTF/ s/./#&/' .config
$ echo 'CONFIG_DEBUG_ATOMIC_SLEEP=y' >> .config
$ make olddefconfig
Compile and install the new kernel.
Build linux DFL Kernel instructions please also refer to: https://github.com/OPAE/linux-dfl/wiki/Build-the-linux-dfl-kernel
When install finished, reboot your system. When the system login again, verify the kernel version is correct. For example:
[figo@localhost linux-dfl]$ uname -a
Linux localhost.localdomain 5.15.lts-dfl-g73e16386cda0 #6 SMP Mon Jun 13 21:21:31 -04 2022 x86_64 x86_64 x86_64
And also you can check the OPAE dfl drivers have auto-loaded.
[figo@localhost linux-dfl]$ lsmod | grep fpga
ifpga_sec_mgr 20480 1 intel_m10_bmc_secure
fpga_region 20480 3 dfl_fme_region,dfl_fme,dfl
fpga_bridge 24576 4 dfl_fme_region,fpga_region,dfl_fme,dfl_fme_br
fpga_mgr 16384 4 dfl_fme_region,fpga_region,dfl_fme_mgr,dfl_fme
[figo@localhost linux-dfl]$ lsmod | grep dfl
dfl_eth_group 36864 0
dfl_fme_region 20480 0
dfl_emif 16384 0
dfl_n3000_nios 20480 0
dfl_fme_br 16384 0
dfl_fme_mgr 20480 1
dfl_fme 49152 0
dfl_afu 36864 0
dfl_pci 20480 0
dfl 40960 7 dfl_pci,dfl_fme,dfl_fme_br,dfl_eth_group,dfl_n3000_nios,dfl_afu,dfl_emif
fpga_region 20480 3 dfl_fme_region,dfl_fme,dfl
fpga_bridge 24576 4 dfl_fme_region,fpga_region,dfl_fme,dfl_fme_br
fpga_mgr 16384 4 dfl_fme_region,fpga_region,dfl_fme_mgr,dfl_fme
Build the OPAE-SDK
Before you build the OPAE SDK, you must install the required packages. Run the following commands:
Rocky Linux 8.5
# dnf install -y 'dnf-command(config-manager)'
# dnf config-manager --set-enabled powertools
# dnf install -y epel-release
# dnf check-update
# dnf upgrade -y
# dnf install -y python3 python3-pip python3-devel python3-jsonschema python3-pyyaml python3-pybind11 git gcc gcc-c++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel cli11-devel spdlog-devel libedit-devel systemd-devel rpm-build rpmdevtools pybind11-devel yaml-cpp-devel libudev-devel linuxptp
# python3 -m pip install jsonschema virtualenv pyyaml
Fedora
# dnf check-update
# dnf upgrade -y
# dnf install -y python3 python3-pip python3-devel python3-jsonschema python3-pyyaml python3-pybind11 git gcc g++ make cmake libuuid-devel json-c-devel hwloc-devel tbb-devel libedit-devel rpm-build rpmdevtools pybind11-devel yaml-cpp-devel libudev-devel cli11-devel spdlog-devel linuxptp
# pip3 install jsonschema virtualenv pyyaml
Ubuntu 20.04
# apt-get update
# apt-get upgrade -y
# apt-get install -y python3 python3-pip python3-dev git gcc g++ make cmake uuid-dev libjson-c-dev libhwloc-dev libtbb-dev libedit-dev libudev-dev linuxptp pandoc devscripts debhelper doxygen
# pip3 install jsonschema virtualenv pyyaml pybind11
RHEL 8.2
Register and enable Red Hat subscription to install any packages on the system.
Set the RHEL version and install packages. Set proxy name and port number.
# subscription-manager release --set=8.2 --proxy proxy-name.com:port number
# subscription-manager repos --enable codeready-builder-for-rhel-8-x86_64-rpms
# dnf upgrade -y
# dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
# dnf install -y python3 python3-pip python3-devel gdb vim git gcc gcc-c++ make cmake libuuid-devel rpm-build systemd-devel nmap
# dnf install -y python3-jsonschema json-c-devel tbb-devel rpmdevtools libcap-devel
# dnf check-update || true
# dnf install -y spdlog-devel cli11-devel python3-pyyaml python3-pybind11 hwloc-devel libedit-devel
# python3 -m pip install --user jsonschema virtualenv pudb pyyaml
Install the latest version of cmake on top of the outdated cmake package from the package manager.
Create opae-sdk packages
Download the OPAE-SDK source code from github. For example, download from Master branch.
Compile and build the OPAE-SDK RPMs (Fedora, Rocky, RHEL 8.2).
Note that if you find that your distribution has changed package names such that there is a conflict when building RPMs, you can install all of the build dependencies so that the SDK compiles and then build the RPMs in unrestricted mode:
After a successful compile, there are 3 rpm packages generated (Fedora, Rocky, RHEL8.2). For example:
opae-2.1.0-1.fc34.x86_64.rpm
opae-devel-2.1.0-1.fc34.x86_64.rpm
opae-extra-tools-2.1.0-1.fc34.x86_64.rpm
Compile and build the OPAE-SDK deb packages (Ubuntu 22.04).
After a successful compile, there are 3 deb packages generated (Ubuntu 22.04). For example:
OPAE SDK installation with rpm/deb packages
The rpm packages generated in the previous step can be installed using these commands:
The deb packages generated in the previous step can be installed using these commands:
When you installed the rpms, you can run fpgainfo command to check the FPGA FME infomation. For example:
[figo@localhost install_guide]$ fpgainfo fme
Board Management Controller, MAX10 NIOS FW version: D.2.1.24
Board Management Controller, MAX10 Build version: D.2.0.7
//****** FME ******//
Object Id : 0xEF00000
PCIe s:b:d.f : 0000:08:00.0
Device Id : 0x0B30
Socket Id : 0x00
Ports Num : 01
Bitstream Id : 0x2300011001030F
Bitstream Version : 0.2.3
Pr Interface Id : f3c99413-5081-4aad-bced-07eb84a6d0bb
Boot Page : user
To uninstall the OPAE rpms, you can use this commands
To uninstall the OPAE deb, you can use this commands
FPGA Device Access Permissions
Access to FPGA accelerators and devices is controlled using file access permissions on the
Intel® FPGA device files, /dev/dfl-fme.* and /dev/dfl-port.*, as well as to the files reachable through /sys/class/fpga_region/.
In order to allow regular (non-root) users to access accelerators, you need to grant them read and write permissions on /dev/dfl-port.* (with * denoting the respective socket, i.e. 0 or 1). E.g.:
Memlock limit
Depending on the requirements of your application, you may also want to increase the maximum amount of memory a user process is allowed to lock. The exact way to do this depends on your Linux distribution.
You can check the current memlock limit using
A way to permanently remove the limit for locked memory for a regular user is to add the following lines to your /etc/security/limits.conf:
This removes the limit on locked memory for user user1. To remove it for
all users, you can replace user1 with *:
Note that settings in the /etc/security/limits.conf file don't apply to services. To increase the locked memory limit for a service you need to modify the application's systemd service file and add the line:
Hugepage Settings
Users need to configure system hugepages to reserve 2MB-hugepages or 1GB-hugepages. For example, the 'hello_fpga' sample requires several 2MB-hugepages. And the fpgadiag tool requires several 1GB-hugepages.
The command below reserves 20 2M-hugepages:
The command below reserves 4 1GB-hugepages:
For x86_64 architecture processors, user can use following command to find out avaiable hugepage sizes:
If this commands returns a non-empty string, 2MB pages are supported.
If this commands returns a non-empty string, 1GB pages are supported.