Shell Developer Guide: Open FPGA Stack for Stratix 10® FPGA

1. Introduction

1.1. About This Document

This document serves as a design guide for FPGA developers, system architects and hardware developers using OFS as a starting point for the creating the FPGA Interface Manager (FIM) for a custom FPGA acceleration board or Platform with Altera FPGAs.

This document uses the Intel® FPGA PAC D5005 as an example platform to illustrate key points and demonstrate how to extend the capabilities provided in OFS (Open FPGA Stack) to custom platforms. The demonstration steps serves as a tutorial for the development of your OFS knowledge.

This document covers OFS architecture lightly. For more details on the OFS architecture, please see Shell Technical Reference Manual: OFS for Stratix® 10 PCIe Attach FPGAs.

You are encouraged to read Workload Developer Guide: OFS for Stratix® 10 PCIe Attach FPGAs to fully understand how AFU Developers will use your newly developed FIM.

Term	Description
AER	Advanced Error Reporting, The PCIe AER driver is the extended PCI Express error reporting capability providing more robust error reporting.
AFU	Accelerator Functional Unit, Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region
BBB	Basic Building Block, Features within an AFU or part of an FPGA interface that can be reused across designs. These building blocks do not have stringent interface requirements like the FIM's AFU and host interface requires. All BBBs must have a (globally unique identifier) GUID.
BKC	Best Known Configuration, The exact hardware configuration Intel has optimized and validated the solution against.
BMC	Board Management Controller, Acts as the Root of Trust (RoT) on the Intel FPGA PAC platform. Supports features such as power sequence management and board monitoring through on-board sensors.
CSR	Command/status registers (CSR) and software interface, OFS uses a defined set of CSR's to expose the functionality of the FPGA to the host software.
DFL	Device Feature List, A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.
FIM	FPGA Interface Manager, Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring.
FME	FPGA Management Engine, Provides a way to manage the platform and enable acceleration functions on the platform.
HEM	Host Exerciser Module, Host exercisers are used to exercise and characterize the various host-FPGA interactions, including Memory Mapped Input/Output (MMIO), data transfer from host to FPGA, PR, host to FPGA memory, etc.
Intel FPGA PAC D5005	Intel FPGA Programmable Acceleration Card D5005, A high performance PCI Express (PCIe)-based FPGA acceleration card for data centers. This card is the target platform for the initial OFS release.
Intel VT-d	Intel Virtualization Technology for Directed I/O, Extension of the VT-x and VT-I processor virtualization technologies which adds new support for I/O device virtualization.
IOCTL	Input/Output Control, System calls used to manipulate underlying device parameters of special files.
JTAG	Joint Test Action Group, Refers to the IEEE 1149.1 JTAG standard; Another FPGA configuration methodology.
MMIO	Memory Mapped Input/Output, Users may map and access both control registers and system memory buffers with accelerators.
OFS	Open FPGA Stack, A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs.
OPAE SDK	Open Programmable Acceleration Engine Software Development Kit, A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE.
PAC	Programmable Acceleration Card: FPGA based Accelerator card
PIM	Platform Interface Manager, An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols.
PR	Partial Reconfiguration, The ability to dynamically reconfigure a portion of an FPGA while the remaining FPGA design continues to function. In the context of Intel FPGA PAC, a PR bitstream refers to an Intel FPGA PAC AFU. Refer to Partial Reconfiguration support page.
RSU	Remote System Update, A Remote System Update operation sends an instruction to the Intel FPGA PAC D5005 device that triggers a power cycle of the card only, forcing reconfiguration.
SR-IOV	Single-Root Input-Output Virtualization, Allows the isolation of PCI Express resources for manageability and performance.
TB	Testbench, Testbench or Verification Environment is used to check the functional correctness of the Design Under Test (DUT) by generating and driving a predefined input sequence to a design, capturing the design output and comparing with-respect-to expected output.
UVM	Universal Verification Methodology, A modular, reusable, and scalable testbench structure via an API framework.
VFIO	Virtual Function Input/Output, An IOMMU/device agnostic framework for exposing direct device access to userspace.

1.2. Introduction

Open FPGA Stack (OFS) addresses the scalability for FPGA acceleration boards and workloads by providing a powerful and systematic methodology for the rapid development of FPGA-based Acceleration systems. This methodology addresses the key challenges of hardware, software and workload developers by providing a complete FPGA project consisting of RTL and simulation code, build scripts and software. The FPGA project released in OFS can be rapidly customized to meet new market requirements by adding new features, custom IPs and interface subsystems IPs.

A high-level overview of the OFS Stratix® 10 FPGA hardware architecture on the Stratix® 10 FPGA reference platform, Intel® FPGA PAC D5005 is shown in the below figure. The provided FPGA architecture is divided into two main components

- The outer area in white, the FPGA Interface manager (or FIM) - The inner area in green, the Acceleration Function Unit or AFU Region.

The outer area, the FIM, provides the core infrastructure and interfaces within the FPGA. The AFU region is where a user’s custom logic would reside for their specific workload.

* FPGA external interfaces and IP cores (e.g. Ethernet, DDR-4, PCIe, etc) * PLLs/resets * FPGA - Board management infrastructure * Interface to Acceleration Function Unit (AFU)

The AFU region has both static and dynamic partial reconfiguration regions enabling a lot of customization.

* Uses the FIM interfaces to perform useful work inside the FPGA * Contains logic supporting partial reconfiguration * Remote Signal Tap core for remote debugging of workload

Outside of the FPGA is the Board Management Controller which provides board management, root of trust, board monitoring, and remote system updates.

The overall architecture is built to be very composable and modular in blocks that can be modified while leaving the rest of the infrastructure intact so you may only need to modify a few of these blocks.

1.2. Release Capabilities

This release of OFS FIM supports the following key features:

• 1 - Host channel interface via PCIe Gen 3 x 16 SRIOV (1PF, 3 VF, AXI-S TLP packets)
• DDR4 SDRAM External memory interface (AXI-M)
• 1 - 10G Ethernet interfaces (1x10G)
• MSI-X Interrupts (PF, VF)
• 1 - AFU
• Exercisers demonstrating PCIe, external memory and Ethernet interfaces
• Port, FME CSR
• Remote Signal Tap

OFS is extensible to meet the needs of a broad set of customer applications, however not all use cases are easily served. The general uses cases listed below are examples where the OFS base design can be easily re-used to build a custom FIM: 1. Use OFS reference design as-is - Porting the code to another platform that is identical to the OFS reference platform only changing target FPGA device and pinout - Change I/O assignments without changing design 2. Update the configuration of peripheral IP in OFS reference design, not affecting FIM architecture - External memory settings - HSSI analog settings 3. Remove/update peripheral feature in OFS reference design, not affecting FIM architecture - External memory speed/width change - Change 10G Ethernet to 25 or 100G Ethernet IP - Change number of VFs supported 4. Add new features as an extension to OFS reference design, not affecting FIM architecture - Add/remove external memory interface to the design - Add/remove user clocks for AFU - Add/remove IP to the design with connection to AFU

More advanced use cases requiring changes or additions to the host PCIe channel are not easily supported with this release of the OFS FIM.

Reuse of the provided host management FPGA logic and software is the fastest and most simple approach to FIM customization.

1.3. Prerequisites

1.3.1. Base Knowledge and Skills Prerequisites

OFS is an advanced application of FPGA technology. This guide assumes you have the following FPGA logic design-related knowledge and skills:

• FPGA compilation flows using Quartus^® Prime Pro Edition design flow.
• Static Timing closure, including familiarity with the Timing Analyzer tool in Quartus^® Prime Pro Edition, applying timing constraints, Synopsys* Design Constraints (.sdc) language and Tcl scripting, and design methods to close on timing critical paths.
• RTL and coding practices for FPGA implementation.
• RTL simulation tools.
• Quartus^® Prime Pro Edition Signal Tap Logic Analyzer tool software.

1.3.2. Development Environment

To run the tutorial steps in this guide requires this development environment:

Item	Version
Quartus Prime Pro	Quartus Prime Pro 23.4 (with license patch)
Target D5005 Sever Operating System	RHEL 8.6
OPAE SDK	2.12.0-5
Linux DFL	ofs-2024.1-6.1-2
Python	3.6.8
cmake	3.15
GCC	8.5.0
perl	5.8.8

The following server and Intel PAC card are required to run the examples in this guide:

1. Qualified Intel Xeon ^® server see Qualified Servers.
2. Intel® FPGA PAC D5005 with root entry hash erased (Please contact Altera for root entry hash erase instructions). The standard Intel® FPGA PAC D5005 card is programmed to only allow the FIM binary files signed by Altera to be loaded. The root entry hash erase process will allow newly created, unsigned FIM binary files to be loaded.
3. Intel® FPGA PAC D5005 installed in the qualified server following instructions in Board Installation Guide: OFS for Acceleration Development Platforms.

The steps included in this guide have been verified in the Dell R740 and HPE ProLiant DL380 Gen10 servers.

2. High Level Description

The FIM targets operation in the Intel® FPGA PAC D5005 card. The block diagram of the D5005 is shown below:

The key D5005 FPGA interfaces are:

• Host interface - PCIe Gen3 x 16
• Network interface
  • 2 - QSFP28 cages
  • Current FIM supports 1 x 10 GbE, other interfaces can be created
• External Memory
  • 2 or 4 channels of DDR4-2400 to RDIMM modules
  • RDIMM modules = 8GB organized as 1 Gb X 72
• Board Management
  • SPI interface
  • FPGA configuration

2.1. FPGA Interface Manager Overview

The FPGA Interface Manager architecture is shown in the below diagram:

The FIM consists of the following components - PCIe Subsystem - Memory Subsystem - HSSI Subsystem - Platform Management Component Intercommunications (PMCI) - Board Peripheral Fabric (BPF) - AFU Peripheral Fabric (APF) - Port Gasket - AXI-S PF/VF Demux/Mux - Host Exerciser Modules - HE-MEM, HE-LB, HE-HSSI - FPGA Management Engine (FME)

2.2. FIM FPGA Resource Usage

The FIM uses a small portion of the available FPGA resources. The table below shows resource usage for a base FIM built with 2 channels of external memory, a small AFU instantiated that has host CSR read/write, external memory test and Ethernet test functionality.

Entity	ALMs Used	% ALMS Used	M20Ks	% M20Ks used	DSP Blocks	Pins	IOPLLs
OFS_top	125009.4	13.0%	661	5.4%	0	630	15
afu_top	70522.7	7.0%	228	2.4%	0	0	1
auto_fab_0	1305.7	0.0%	9	0.1%	0	0	0
bpf_rsv_5_slv	0.6	0.0%	0	0.0%	0	0	0
bpf_rsv_6_slv	0.6	0.0%	0	0.0%	0	0	0
bpf_rsv_7_slv	0.4	0.0%	0	0.0%	0	0	0
bpf	241.9	0.0%	0	0.0%	0	0	0
emif_top_inst	10508.6	1.0%	0	0.0%	0	0	12
eth_ac_wrapper	6024.8	0.5%	9	0.1%	0	0	0
fme_top	615.5	0.2%	7	0.1%	0	0	0
pcie_wrapper	35424.7	3.5%	348	2.9%	0	0	1
pmci_top	318.5	0.1%	0	0.0%	0	0	0
rst_ctrl	40.2	0.0%	0	0.0%	0	0	0
sys_pll	0.5	0.0%	0	0.0%	0	0	1
Total ALMS	933,120
Total M20Ks	11,721

Summary	FPGA Resource Utilization
Logic utilization (in ALMs)	124,092 / 933,120 ( 13 % )
Total dedicated logic registers	282822
Total pins	630 / 912 ( 69 % )
Total block memory bits	3,425,120 / 240,046,080 ( 1 % )
Total RAM Blocks	661 / 11,721 ( 6 % )
Total DSP Blocks	0 / 5,760 ( 0 % )
Total eSRAMs	0 / 75 ( 0 % )
Total HSSI P-Tiles	17 / 48 ( 35 % )
Total HSSI E-Tile Channels	17 / 48 ( 35 % )
Total HSSI HPS	0 / 1 ( 0 % )
Total HSSI EHIPs	0 / 2 ( 0 % )
Total PLLs	36 / 104 ( 35 % )

2.3. OFS Directory Structure

The OFS Git OFS repository ofs-d5005 directory structure is shown below:

├── eval_script
|   ├── ofs_d5005_eval.sh
|   └── README_ofs_d5005_eval.txt
├── ipss
│   ├── hssi
|   ├── mem
|   ├── pcie
|   ├── pmci
|   ├── spi
|   └── README.md
├── license
│   └── quartus-0.0-0.01Intel OFS-linux.run
├── ofs-common
|   ├── scripts
|   ├── src
|   ├── verification
|   ├── LICENSE.txt
|   └── README.md
├── sim
|   ├── bfm
|   ├── rp_bfm
│   ├── scripts
|   ├── unit_test 
│   └── readme.txt
├── src
│   ├── afu_top
│   ├── includes
│   ├── pd_qsys
│   ├── README.md
│   └── top
├── syn
│   ├── scripts
│   ├── setup
│   ├── syn_top
│   ├── readme.txt
│   └── README
├── LICENSE.txt
└── README.md

The contents of each directory are described below:

Eval Script - Contains scripts for evaluation of OFS for D5005 including compiling FIM/AFU from source, unit level test. Also includes resources to report and setup D5005 development environment

ipss - Contains the code and supporting files that define or set up the IP subsystems (HSSI, PCIe, memory, PMCI, SPI, etc...) contained in the D5005 FPGA Interface Manager (FIM).

license - License file for the Low Latency 10Gbps Ethernet MAC (6AF7 0119) IP core.

ofs-common - This directory contains resources that may be used across the board-specific repositories. This directory is referenced via a link within each of the FPGA-specific repositories.

sim - Contains the testbenches and supporting code for all the unit test simulations. - Bus Functional Model code is contained here. - Scripts are included for automating a myriad of tasks. - All of the individual unit tests and their supporting code is also located here.

src - SystemVerilog source and script files
- Contains all of the structural and behavioral code for the FIM. - Scripts for generating the AXI buses for module interconnect. - Top-level RTL for synthesis is located in this directory. - Accelerated Functional Unit (AFU) infrastructure code is contained in this directory.

syn - This directory contains all of the scripts, settings, and setup files for running synthesis on the FIM.

3. Description of Sub-Systems

3.1. Host Control and Data Flow

The host control and data flow are shown in the diagram below:

The control and data flow is composed of the following:

• Host Interface Adapter (PCIe)
• Low Performance Peripherals
  • Slow speed peripherals (I2C, Smbus, etc)
  • Management peripherals (FME)

• High Performance Peripherals
  • Memory peripherals
  • Acceleration Function peripherals
  • HPS Peripheral

• Fabrics
  • Peripheral Fabric (multi drop)
  • AFU Streaming fabric (point to point)

Peripherals are connected to one another using AXI:

• Via the peripheral fabric (AXI4-Lite, multi drop)
• Via the AFU streaming fabric (AXI-S, point to point)

Peripherals are presented to software as:

• OFS managed peripherals that implement DFH CSR structure.
• Native driver managed peripherals (i.e. Exposed via an independent PF, VF)

The peripherals connected to the peripheral fabric are primarily OPAE managed resources, whereas the peripherals connected to the AFU are “primarily” managed by native OS drivers. The word “primarily” is used since the AFU is not mandated to expose all its peripherals to OPAE. It can be connected to the peripheral fabric, but can choose to expose only a subset of its capability to OPAE.

OFS uses a defined set of CSRs to expose the functionality of the FPGA to the host software. These registers are described in Open FPGA Stack Reference Manual - MMIO Regions section.

If you make changes to the FIM that affect the software operation, then OFS provides a mechanism to communicate that information to the proper software driver. The Device Feature Header (DFH) structure provides a mechanism to maintain compatibility with OPAE software. Please see FPGA Device Feature List (DFL) Framework Overview for an excellent description of DFL operation from the driver perspective.

When you are planning your address space for your FIM updates, please be aware that the OFS FIM targeting Intel® FPGA PAC D5005, 256KB of MMIO region is allocated for external FME features and 128kB of MMIO region is allocated for external port features. Each external feature must implement a feature DFH, and the DFH needs to be placed at 4KB boundary. The last feature in the external feature list must have the EOL bit in its DFH set to 1 to mark the end of external feature list. Since the FPGA address space is limited, consider using an indirect addressing scheme to conserve address space.

4. FIM Development Flow

OFS provides a framework of FPGA synthesizable code, simulation environment, and synthesis/simulation scripts. FIM designers can take the provided code and scripts and modify existing code or add new code to meet their specific product requirements.

FIM development for a new acceleration card consists of the following steps:

1. Installation of OFS and familiarization with scripts and source code
2. Development of high-level block diagram with your specific functionality
   1. Determination of requirements and key performance metrics
   2. Selection of IP cores
   3. Selection of FPGA device
   4. Software memory map
3. Selection and implementation of FIM Physical interfaces including:
   1. External clock sources and creation of internal PLL clocks
   2. General I/O
   3. Transceivers
   4. External memories
   5. FPGA programming methodology
4. Device physical implementation
   1. FPGA device pin assignment
   2. Inclusion of logic lock regions
   3. Creation of timing constraints
   4. Create Quartus FIM test project and validate:
      1. Placement
      2. Timing constraints
      3. Build script process
      4. Review test FIM FPGA resource usage
5. Select FIM to AFU interfaces and development of PIM
6. FIM design implementation
   1. RTL coding
   2. IP instantiation
   3. Development of test AFU to validate FIM
   4. Unit and device level simulation
   5. Timing constraints and build scripts
   6. Timing closure and build validation
7. Creation of FIM documentation to support AFU development and synthesis
8. Software Device Feature discovery
9. Hardware/software integration, validation and debugging
10. High volume production preparation

The FIM developer works closely with the hardware design of the target board, software development and system validation.

Understanding how the AFU developer utilizes the FIM is important for FIM development success. Please read Workload Developer Guide: OFS for Stratix® 10 PCIe Attach FPGAs for a detailed description of AFU development.

4.1. Installation of OFS

In this section you set up a development machine for compiling the OFS FIM. These steps are separate from the setup for a deployment machine where the FPGA acceleration card is installed. Typically, FPGA development and deployment work is performed on separate machines, however, both development and deployment can be performed on the same server if desired. Please see Software Installation Guide: OFS for PCIe Attach FPGAs for instructions on installing software for deployment of your FPGA FIM, AFU and software application on a server.

Building the OFS FIM requires the development machine to have at least 64 GB of RAM.

The following is a summary of the steps to set up for FIM development:

1. Install Quartus Prime Pro 23.4 Linux and setup environment
2. Clone the github ofs-d5005 repository
3. Test installation by building the provided FIM

Quartus Prime Pro version 23.4 is the currently verified version of Quartus used for building the FIM and AFU images for this release. Porting to newer versions of Quartus may be performed by developers. Download Quartus Prime Pro Linux version 23.4 from Quartus® Prime Pro Edition Linux.

Intel Quartus Prime Pro Version 23.4 is verified to work with the latest OFS release ofs-2024.1-1. However, you have the option to port and verify the release on newer versions of Intel Quartus Prime Pro software.

Use RedHat® Enterprise Linux® (RHEL) 8.6 for compatibility with your development flow and also testing your FIM design in your platform.

Prior to installing Quartus:

1. Ensure you have at least 64 GB of free space for Quartus Prime Pro installation and your development work.

   • Intel® recommends that your system be configured to provide virtual memory equal in size or larger than the recommended physical RAM size that is required to process your design.
   • The disk space may be significantly more based on the device families included in the install. Prior to installation, the disk space should be enough to hold both zipped tar files and uncompressed installation files. After successful installation, delete the downloaded zipped files and uncompressed zip files to release the disk space.

2. Perform the following steps to satisfy the required dependencies.

   $ sudo dnf install -y gcc gcc-c++ make cmake libuuid-devel rpm-build autoconf automake bison boost boost-devel libxml2 libxml2-devel make ncurses grub2 bc csh flex glibc-locale-source libnsl ncurses-compat-libs 
   

   Apply the following configurations.

   $ sudo localedef -f UTF-8 -i en_US en_US.UTF-8 
   $ sudo ln -s /usr/lib64/libncurses.so.6 /usr/lib64/libncurses.so.5 
   $ sudo ln -s /usr/bin/python3 /usr/bin/python
   

3. Create the default installation path: /intelFPGA_pro/, where is the default path of the Linux workstation, or as set by the system administrator and is your Quartus version number.

   The installation path must satisfy the following requirements:

   • Contain only alphanumeric characters
   • No special characters or symbols, such as !$%@^&*<>,
   • Only English characters
   • No spaces

4. Download your required Quartus Prime Pro Linux version here.

5. Install required Quartus patches. The Quartus patch .run files can be found in the Assets tab on the OFS Release GitHub page. The patches for this release are None.

6. After running the Quartus Prime Pro installer, set the PATH environment variable to make utilities quartus, jtagconfig, and quartus_pgm discoverable. Edit your bashrc file ~/.bashrc to add the following line:

   export PATH=<Quartus install directory>/quartus/bin:$PATH
   export PATH=<Quartus install directory>/qsys/bin:$PATH
   

   For example, if the Quartus install directory is /home/intelFPGA_pro/23.4 then the new line is:

   export PATH=/home/intelFPGA_pro/23.4/quartus/bin:$PATH
   export PATH=/home/intelFPGA_pro/23.4/qsys/bin:$PATH
   

7. Verify, Quartus is discoverable by opening a new shell:

   $ which quartus
   /home/intelFPGA_pro/23.4/quartus/bin/quartus
   

After running the Quartus Prime Pro installer, set the PATH environment variable to make utilities quartus, jtagconfig, and quartus_pgm discoverable. Edit your bashrc file ~/.bashrc to add the following line:

export PATH=$PATH:<Quartus install directory>/quartus/bin

For example, if the Quartus install directory is /home/intelFPGA_pro/23.4 then the new line is:

export PATH=$PATH:/home/intelFPGA_pro/23.4/quartus/bin

Verify, Quartus is discoverable by opening a new shell:

which quartus
## Output
/home/intelFPGA_pro/23.4/quartus/bin/quartus

Note, for some Linux distributions such as RHEL 8.6, Quartus requires installation of the following libraries:

sudo dnf install libnsl
sudo dnf install ncurses-compat-libs
sudo ln -s /usr/bin/python3 /usr/bin/python

You will need to obtain a license for Quartus Prime Pro version 23.4 to compile the design. Additionally, OFS for Stratix® 10 FPGA requires a license for the Low Latency 10Gbps Ethernet MAC (6AF7 0119) IP core. This license is required to generate a programming file using the provided OFS source code. The Low Latency 10Gbps Ethernet MAC (6AF7 0119) IP core license patch installer is provided in the ofs-d5005 git repository in the /license directory. After cloning the OFS release in step 4 below, you can install this IP license.

1. Install git.

sudo dnf install git

1. Retrieve OFS repositories:

​ The OFS FIM source code is included in the GitHub repository. Create a new directory to use as a clean starting point to store the retrieved files.

1. Navigate to location for storage of OFS source, create the top-level source directory and clone OFS repositories.

mkdir OFS_fim_build_root
cd OFS_fim_build_root
export OFS_BUILD_ROOT=$PWD
git clone --recurse-submodules  https://github.com/OFS/ofs-d5005.git
cd ofs-d5005
git checkout tags/ofs-2024.1-1

Verify proper tag is selected:

git describe --tags
ofs-2024.1-1

2. Install the Low Latency 10Gbps Ethernet MAC (6AF7 0119) IP license by running provided license installer.

cd license
chmod +x quartus-0.0-0.01Intel OFS-linux.run
sudo ./quartus-0.0-0.01Intel OFS-linux.run

1. Verify patch installed

   quartus_sh --version
   ##Output
   Quartus Prime Shell
   Version 23.4 Pro Edition
   

4.2. Compiling OFS FIM

OFS provides a build script with the following FPGA image creation options:

• Flat compile which combines the FIM and AFU into one FPGA image that is loaded into the entire FPGA device

• A PR compile which creates a FPGA image consisting of the FIM that is loaded into the static region of the FPGA and a default AFU that is loaded into dynamic region. Additional AFUs maybe loaded into the dynamic region using partial reconfiguration.

The build scripts included with OFS are verified to run in a bash shell. Other shells have not been tested. Each build script step will take several hours to completes, Please note, building directly in Quartus GUI is not supported - you must build with the provided scripts.

The following sections describe how to set up the environment and build the provided FIM and AFU. Follow these steps as a tutorial to learn the build flow. You will use this environment and build scripts for the creation of your specialized FIM.

4.2.1. Setting Up Required Environment Variables

Set required environment variables as shown below. These environment variables must be set prior to simulation or compilation tasks so creating a simple script to set these variables saves time.

cd $OFS_BUILD_ROOT/ofs-d5005
export OFS_ROOTDIR=$PWD

##   Note, OFS_ROOTDIR is the directory where you cloned the repo, e.g. /home/MyProject/ofs-d5005 *

export WORKDIR=$OFS_ROOTDIR
export VERDIR=$OFS_ROOTDIR/verification
export QUARTUS_HOME=$QUARTUS_ROOTDIR

##   Note, QUARTUS_ROOTDIR is your Quartus installation directory, e.g. $QUARTUS_ROOTDIR/bin contains Quartus executuable*

export QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR
export IMPORT_IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip
export IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip
export OPAE_SDK_REPO_BRANCH=release/2.12.0

4.2.2. Compiling

The usage of the compile build script is shown below:

ofs-common/scripts/common/syn/build_top.sh [-p] target_configuration work_dir 
Usage: ofs-common/scripts/common/syn/build_top.sh [-k] [-p] <build target> [<work dir name>]

  Build a FIM instance specified by <build target>. The target names an FPGA architecture, board and configuration.

  The FIM is built in <work dir name>. If not specified, the target is ${OFS_ROOTDIR}/work.

  The -k option preserves and rebuilds within an existing work tree instead of overwriting it.

  When -p is set, if the FIM is able then a partial reconfiguration template tree is generated at the end of the FIM build. The PR template tree is located in the top of the work directory but is relocatable
  and uses only relative paths. See ofs-common/scripts/common/syn/generate_pr_release.sh for details.

  The -e option runs only Quartus analysis and elaboration.

      * target_configuration - Specifies the project  
         For example: d5005

      * work_dir - Work Directory for this build in the form a directory name. It is created in the <local repo directory>/ofs-d5005/<work_dir> 
          - NOTE: The directory name must start with "work".  If the work directory exists, then the script stops and asks if you want to overwrite the directory.
            - e.g.
                - ofs-common/scripts/common/syn/build_top.sh d5005 work_d5005

                work directory as a name will be created in <local repo directory>/ofs-d5005/work_d5005


                The obmission of <work_dir> results in a default work directory (<local repo  directory>/ofs-d5005/work)

        - compile reports and artifacts (.rpt, .sof, etc) are stored in <work_dir>/syn/syn_top/output_files

        - There is a log file created in ofs-d5005 directory.  
        - [-p]  Optional switch for creation of a relocatable PR build tree supporting the creation of a PR-able AFU workload.   
        The "-p" switch invokes generate_pr_release.sh at the end of the FIM build and writes the PR build tree to the top of the work directory.  More information on this option is provided below. 

In the next example, you will build the provided example design using a flat, non-PR build flow.

Build the provided base example design:

```bash cd $OFS_BUILD_ROOT/ofs-d5005

ofs-common/scripts/common/syn/build_top.sh d5005 work_d5005 ```

```bash ... build takes ~5 hours to complete

Compile work directory: <$OFS_BUILD_ROOT>/work_d5005/syn/syn_top Compile artifact directory: <$OFS_BUILD_ROOT>/work_d5005/syn/syn_top/output_files

---

---

*** OFS_PROJECT: d5005 *** Q_PROJECT: d5005 *** Q_REVISION: d5005 *** SEED: 03 *** Build Complete *** Timing Passed!

---

---

The build script copies the ipss, sim, src and syn directories to the specified work directory and then these copied files are used in the Quartus compilation process.  Do not edit the files in the work directory, these files are copies of source files.

Some of the key files are described below:

<work_dir>/syn/syn_top == 
```bash
├── syn_top                    // D5005 Quartus build area with Quartus files used this build
│  ├── d5005.ipregen.rpt       // IP regeneration report states the output of IP upgrade
│  ├── d5005.qpf               // Quartus Project File (qpf) mentions about Quartus version and project revision
│  ├── d5005.qsf               // Quartus Settings File (qsf) lists current project settings and entity level assignments
│  ├── d5005.stp               // Signal Tap file included in the d5005.qsf. This file can be modified as required if you need to add an Signal Tap instance
│  ├── fme_id.mif              // the fme id hex value is stored in a mif file format
│  ├── OFS_pr_afu.json        // PR JSON file
│  ├── OFS_pr_afu.qsf                // PR AFU qsf file
│  ├── OFS_pr_afu_sources.tcl        // AFU source file list
│  ├── ip_upgrade_port_diff_reports   // IP upgrade report files for reference

/syn/syn_top/output_files == Directory with build reports and FPGA programming files.

The programming files consist of the Quartus generated d5005.sof and d5005.pof. The D5005 board hardware provides a 2 Gb flash device to store the FPGA programming files and a MAX10 BMC that reads this flash and programs the D5005 Stratix® 10 FPGA FPGA. The syn/build_top.sh script runs script file syn/syn_top/build_flash/build_flash.s which takes the Quartus generated d5005.sof and creates binary files in the proper format to be loaded into the 2 Gb flash device. You can also run build_flash.sh by yourself if needed. The build_flash script runs PACSign (if installed) to create an unsigned FPGA programming file that can be stored in the D5005 FPGA flash. Please note, if the D5005 has the root entry hash key loaded, then PACsign must be run with d5005_page1.bin as the input with the proper key to create an authenticated FPGA binary file. Please see Security User Guide: Open FPGA Stack for details on the security aspects of Open FPGA Stack.

The following table provides further detail on the generated bin files.

File	Description
d5005.sof	This is the Quartus generated programming file created by Quartus synthesis and place and route. This file can be used to programming the FPGA using a JTAG programmer. This file is used as the source file for the binary files used to program the FPGA flash.
d5005.bin	This is an intermediate raw binary image of the FPGA
d5005_page1.bin	This is the binary file created from input file, d5005.sof. This file is used as the input file to the PACSign utility to generate d5005_page1_unsigned.bin binary image file.
d5005_page1_unsigned.bin	This is the unsigned PACSign output which can be programmed into the FPGA flash of an unsigned D5005 usign the OPAE SDK utility fpgasupdate
mfg_d5005_reversed.bin	A special programming file for a third party programming device used in board manufacturing. This file is typically not used.

build/output_files/timing_report == Directory containing clocks report, failing paths and passing margin reports

4.2.3. Relocatable PR Directory Tree

If you are developing a FIM to be used by another team developing the AFU workload, scripts are provided that create a relocatable PR directory tree. ODM and board developers will make use of this capability to enable a broad set of AFUs to be loaded on a board using PR. The relocatable PR directory contains the Quartus *.qdb file that goes the FIM.

The creation of the relocatable PR directory tree requires a clone of the Basic Building Blocks (BBB) repository. The OFS_PLATFORM_AFU_BBB environment variable must point to the repository, for example.

cd $OFS_BUILD_ROOT
git clone https://github.com/OPAE/ofs-platform-afu-bbb
cd ofs-platform-afu-bbb
export OFS_PLATFORM_AFU_BBB=$PWD
cd $OFS_ROOTDIR

You can create this relocatable PR directory tree by either:

• Build FIM and AFU using /syn/build_top.sh followed by running./ofs-common/scripts/common/syn/generate_pr_release.sh
• Build FIM and AFU using /syn/build_top.sh with optional -p switch included

The generate_pr_release.sh has the following command structure:

./ofs-common/scripts/common/syn/generate_pr_release.sh -t <path to generated release tree> *Board Build Target* <work dir from build_top.sh>

Where:

-t <path to generated release tree> = location for your relocatable PR directory tree
*Board Build Target* is the name of the board target/FIM e.g. d5005
<work dir from build_top.sh> 

Here is an example of running the generate_pr_release.sh script:

ofs-common/scripts/common/syn/generate_pr_release.sh -t work_d5005/build_tree d5005  work_d5005

**********************************
********* ENV SETUP **************

FIM Project:
  OFS_PROJECT = d5005
  OFS_FIM     = .
  OFS_BOARD   = .
  Q_PROJECT   = d5005
  Q_REVISION  = d5005
  Fitter SEED = 03
FME id
  BITSTREAM_ID = 04010002c7cab852
  BITSTREAM_MD = 0000000002204283

...
...

The resulting relocatable build tree has the following structure:

.
├── bin
│   ├── afu_synth
│   ├── build_env_config
│   ├── run.sh -> afu_synth
│   └── update_pim
├── hw
│   ├── blue_bits
│   │   ├── d5005_page1_unsigned.bin
│   │   └── d5005.sof -> ../lib/build/syn/syn_top/output_files/d5005.sof
│   └── lib
│       ├── build
│       ├── fme-ifc-id.txt
│       ├── fme-platform-class.txt
│       └── platform

This build tree can be moved to a different location and used for AFU development of a PR capable AFU to be used with this board.

4.2.4. Unit Level Simulation

Unit level simulation of key components is provided. These simulations provide verification of the following areas:

• HSSI
• PCIe
• External Memory
• FIM management

These simulations use the Synopsys VCS simulator. Each simulation contains a readme file explaining how to run the simulation. Refer to UVM Simulation User Guide: OFS for Stratix® 10 PCIe Attach for details of simulation examples. Your simulation shell requires Python, Quartus, and VCS to run. To run a simulation of the dfh_walker that simulates host access to the internal DFH registers, perform the following steps:

Before running unit simulation, you must set environment variables as described below:

cd $OFS_BUILD_ROOT/ofs-d5005
export OFS_ROOTDIR=$PWD

##   *Note, OFS_ROOTDIR is the directory where you cloned the repo, e.g. /home/MyProject/ofs-d5005 *

export WORKDIR=$OFS_ROOTDIR
export VERDIR=$OFS_ROOTDIR/verification
export QUARTUS_HOME=$QUARTUS_ROOTDIR

##   *Note, QUARTUS_ROOTDIR is your Quartus installation directory, e.g. $QUARTUS_ROOTDIR/bin contains Quartus executuable*

export QUARTUS_INSTALL_DIR=$QUARTUS_ROOTDIR
export IMPORT_IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip
export IP_ROOTDIR=$QUARTUS_ROOTDIR/../ip
export OPAE_SDK_REPO_BRANCH=release/2.12.0

To compile all IPs:

To Generate Simulation Files & compile all IPs, run the following command:

cd $OFS_ROOTDIR/ofs-common/scripts/common/sim
sh gen_sim_files.sh d5005

The RTL file list for unit_test is located here: $OFS_ROOTDIR/sim/scripts/rtl_comb.f

The IPs are generated here:

$OFS_ROOTDIR/sim/scripts/qip_gen

The IP simulation filelist is generated here:

$OFS_ROOTDIR/sim/scripts/ip_flist.f

Once the IPs are generated, they can be used for any unit test.

To run the simulation, run the following command:

cd $OFS_ROOTDIR/sim/unit_test/<Unit Test Name>/scripts
sh run_sim.sh VCS=1

Simulation files are located in the sim/unit_test//sim directory.

To view simulation waveform:

cd $OFS_ROOTDIR/sim/unit_test/<test_name>/script/sim/unit_test/<test_name>/scripts/sim_vcs
dve -full64 -vpd vcdplus.vpd &

4.2.4.1. DFH Walking Unit Simulation Output

********************************************
 Running TEST(0) : test_fme_dfh_walking
********************************************
READ64: address=0x00000000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x4000000010000000

FME_DFH
   Address   (0x0)
   DFH value (0x4000000010000000)
READ64: address=0x00001000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x3000000020000001

THERM_MNGM_DFH
   Address   (0x1000)
   DFH value (0x3000000020000001)
READ64: address=0x00003000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x3000000010000007

GLBL_PERF_DFH
   Address   (0x3000)
   DFH value (0x3000000010000007)
READ64: address=0x00004000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x30000000c0001004

GLBL_ERROR_DFH
   Address   (0x4000)
   DFH value (0x30000000c0001004)
READ64: address=0x00010000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x300000010000000e

SPI_DFH
   Address   (0x10000)
   DFH value (0x300000010000000e)
READ64: address=0x00020000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x3000000100000020

PCIE_DFH
   Address   (0x20000)
   DFH value (0x3000000100000020)
READ64: address=0x00030000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x300000010000100f

HSSI_DFH
   Address   (0x30000)
   DFH value (0x300000010000100f)
READ64: address=0x00040000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x3000000500000009

EMIF_DFH
   Address   (0x40000)
   DFH value (0x3000000500000009)
READ64: address=0x00090000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x3000000010001005

FME_PR_DFH
   Address   (0x90000)
   DFH value (0x3000000010001005)
READ64: address=0x00091000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x4000000010001001

PORT_DFH
   Address   (0x91000)
   DFH value (0x4000000010001001)
READ64: address=0x00092000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x3000000010000014

USER_CLOCK_DFH
   Address   (0x92000)
   DFH value (0x3000000010000014)
READ64: address=0x00093000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x30000000d0002013

PORT_STP_DFH
   Address   (0x93000)
   DFH value (0x30000000d0002013)
READ64: address=0x000a0000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x3000010000002010

AFU_INTF_DFH
   Address   (0xa0000)
   DFH value (0x3000010000002010)
MMIO error count matches: x

Test status: OK

********************
  Test summary
********************
   test_fme_dfh_walking (id=0) - pass
Test passed!
Assertion count: 0

The simulation transcript is displayed while the simulation runs. The transcript is saved to the file transcript.out for review after the simulation completes. The simulation waveform database is saved as vcdplus.vpd for post simulation review. You are encouraged to run the additional simulation examples to learn about each key area of the OFS shell.

4.3. Compiling the OFS FIM using Eval Script

The Evaluation Script provides resources to setup and report D5005 development environment. You can use the evaluation script to compile and simulate the FIM. Refer to README_ofs_d5005_eval.txt for details of using the evaluation script.

4.4. Debugging

For debugging issues within the FIM, Signal Tap can be used to gain internal visibility into your design. This section describes the process of adding a Signal Tap instance to your FIM design

4.4.1. Signal Tap Prerequisites

To use Signal Tap with OFS, you will need the following:

• Understanding of Signal Tap fundamentals - please review Quartus Prime Pro Edition User Guide: Debug Tools.
  section 2. Design Debugging with the Signal Tap Logic Analyzer.

• The Intel® FPGA PAC D5005 has a built in FPGA Download Cable II allowing JTAG access to the S10 FPGA. You can access the D5005 built in FPGA Download Cable II by connecting your server to the Micro USB connector as shown below:

• If you are using a custom board without a built-in FPGA Download Cable then an external FPGA Download Cable II (see Download Cables for more information) can be used for Signal Tap access. The custom board must have JTAG access to the target FPGA for the FPGA Download Cable II.

4.4.2. Adding Signal Tap

The following steps guide you through the process of adding a Signal Tap instance to your design. The added Signal Tap instance provides hardware to capture the desired internal signals and connect the stored trace information via JTAG. Please be aware, the added Signal Tap hardware will consume FPGA resources and may require additional floorplanning steps to accommodate these resources. Some areas of the FIM use logic lock regions and these regions may need to be re-sized. These steps assume the use of the Intel® FPGA PAC D5005.

1. Perform a full compile using the script build_top.sh.
2. Once the compile completes open the Quartus GUI using the FIM project. The Quartus project is named d5005 and is located in the work directory syn/syn_top/d5005.qpf. Once the project is loaded, go to Tools > Signal Tap Logic Analyzer to bring up the Signal Tap GUI.

1. Accept the "Default" selection and click "Create".

1. This brings up Signal Tap Logic Analyzer window as shown below:

1. Set up clock for STP instance. In this example, the EMIF CSR module is being instrumented. If unfamiliar with code, it is helpful to use the Quartus Project Navigator to find the specific block of interest and open the design instance for review. For example, see snip below using Project Navigator to open emif_csr block:

1. After reviewing code, assign clock for sampling Signal Tap instrumented signals of interest. Note, the clock selected and the signals you want to view should be the same for best trace fidelity. Different clocks can be used however there maybe issues with trace inaccuracy due sampling time differences. In the middle right of the Signal Tap window under Signal Configuration, Clock: select "…" as shown below:

1. After reviewing code, assign clock for sampling Signal Tap instrumented signals of interest. In the middle right of the Signal Tap window under Signal Configuration, Clock: select "…" as shown below: This brings up the Node Finder tool. Input "emif_csr" into Named and select "Search". This brings up all nodes from the pre-synthesis view. Expand, "mem" and "emif_csr" and scroll through this list to become familiar with nodes, and then select csr_if.clk and click ">" to select this clock as shown below and click "OK":

1. Update the sample depth and other Signal Tap settings as needed for your debugging criteria.

1. In the Signal Tap GUI add nodes to be instrumented by double clicking on "Double-click to add nodes".

1. This brings up the Node Finder. Add signals to be traced by the Signal Tap instance. Click "Insert" to add the signals.
2. To provide a unique name for your Signal Tap instance, select "auto signaltap_0", right click and select rename instance and provide a descriptive name for your instance.
3. Save the newly created Signal Tap file and click "Yes" to add the new Signal Tap file to the project.
4. Compile the project with the Signal Tap file added to the project.
5. Once the compile successfully completes with proper timing, you can load the generated d5005.sof using the FPGA Downloader cable.

4.4.3. Signal Tap trace acquisition

To acquire signals using SignalTap, first load the Signal Tap instrumented SOF file into your target board, open the STP file in the Signal Tap GUI and start the signal acquisition.

Avoid system hang during programming the sof file, mask AER regsiter using below steps

Find Root complex - End Point mapping using the below command

lspci -vt

+-[0000:3a]-+-00.0-[3b-3c]----00.0  Intel Corporation Device bcce
 |           +-05.0  Intel Corporation Sky Lake-E VT-d
 |           +-05.2  Intel Corporation Sky Lake-E RAS Configuration Registers
 |           +-05.4  Intel Corporation Sky Lake-E IOxAPIC Configuration Registers
 |           +-08.0  Intel Corporation Sky Lake-E Integrated Memory Controller
 |           +-09.0  Intel Corporation Sky Lake-E Integrated Memory Controller
 |           +-0a.0  Intel Corporation Sky Lake-E Integrated Memory Controller
 |           +-0a.1  Intel Corporation Sky Lake-E Integrated Memory Controller
 |           +-0a.2  Intel Corporation Sky Lake-E Integrated Memory Controller
 |           +-0a.3  Intel Corporation Sky Lake-E Integrated Memory Controller
 |           +-0a.4  Intel Corporation Sky Lake-E Integrated Memory Controller
 |           +-0a.5  Intel Corporation Sky Lake-E LM Channel 1

Use the bus information from the lspci logs to mask the AER (Advanced Error Reporting) register

sudo su

setpci -s 0000:3b:00.0 ECAP_AER+0x08.L=0xFFFFFFFF 
setpci -s 0000:3b:00.0 ECAP_AER+0x14.L=0xFFFFFFFF
setpci -s 0000:3a:00.0 ECAP_AER+0x08.L=0xFFFFFFFF
setpci -s 0000:3a:00.0 ECAP_AER+0x14.L=0xFFFFFFFF
echo "1" > /sys/bus/pci/devices/0000:3b:00.0/remove

exit

1. The SOF file is located in the work directory work_d5005/syn/syn_top/output_files/d5005.sof. If the target FPGA is on a different server, then transfer d5005.sof and STP files to the server with the target FPGA. Load the SOF using the Intel® FPGA PAC D5005 built-in FPGA Download Cable II.

sudo su
echo "1" > /sys/bus/pci/rescan

1. Make sure D5005 is present by checking expected bitstream ID using command:

sudo fpgainfo fme
Intel FPGA Programmable Acceleration Card D5005
Board Management Controller, MAX10 NIOS FW version: 2.0.13 
Board Management Controller, MAX10 Build version: 2.0.8 
//****** FME ******//
Object Id                        : 0xF000000
PCIe s:b:d.f                     : 0000:3B:00.0
Vendor Id                        : 0x8086
Device Id                        : 0xBCCE
SubVendor Id                     : 0x8086
SubDevice Id                     : 0x138D
Socket Id                        : 0x00
Ports Num                        : 01
Bitstream Id                     : 0x40100022C164DB1
Bitstream Version                : 4.0.1
Pr Interface Id                  : 210a4631-18bb-57d1-879f-2c3d59b26e37
Boot Page                        : user

1. Once the SOF file is loaded, start the Quartus Signal Tap GUI.

quartus_stpw

The Signal Tap GUI comes up.

1. In the Signal Tap GUI, Hardware: selection box select the cable "Stratix10 Darby Creek [ JTAG cable number ]" as shown below:

1. In File open your STP file. Your STP file settings will load. If not already set, you can create the trigger conditions, then start analysis with F5.

5. FIM Modification Example

An example of FIM modification is provided in this section. This example can be used in your specific application as a starting point. This example shows the basic flow and listing of files that are to be changed.

5.1. Hello FIM example

If you intend to add a new module to the FIM area, then you will need to inform the host software of the new module. The FIM exposes its functionalities to host software through a set of CSR registers that are mapped to an MMIO region (Memory Mapped IO). This set of CSR registers and their operation is described in FIM MMIO Regions.

See FPGA Device Feature List (DFL) Framework Overview for a description of the software process to read and process the linked list of Device Feature Header (DFH) CSRs within a FPGA.

This example adds a simple DFH register set to the FIM. You can use this example as the basis for adding a new feature to your FIM.

The steps to add this simple DFH register are described below.

1. Review current design documentation: OFS Tech Ref MMIO Regions
2. Understand FME and Port regions, DFH walking, DFH register structure
3. Run unit level simulations and review output: i. sim/unit_test/dfh_walker
4. Note DFH link list order, see DFH Walker Unit Level Simulation Output
5. Make code changes to top level FIM file to instantiate new DFH register
6. The DFH registers follow a link list. This example inserts the hello_fim DFH register after the EMIF DFH register, so the emif_csr.sv parameters are updated to insert the hello_fim DFH register as next register in the link list.
7. Create the new hello_fim SystemVerilog files.
8. Update and run the dfh_walker unit simulation files
9. Update synthesis files to include the new hello_fim source files
10. Build and test the new FIM

The following sections describe changes to add the hello_fim DFH example to the provided FPGA design.

5.1.1. src/top/iofs_top.sv

1. Edit top level design module: src/top/iofs_top.sv
   1. Instantiate new hello_fim module in OFS_top.sv at line 294

//*******************************
// FME
//*******************************

  fme_top 
  fme_top(
          .clk               (clk_1x                    ),
          .rst_n             (rst_n_d_1x       ),
          .pwr_good_n        (ninit_done                ),
          .i_pcie_error      ('0                        ),

          .axi_lite_m_if     (bpf_fme_mst_if            ),
          .axi_lite_s_if     (bpf_fme_slv_if            )
         );


`ifdef INCLUDE_HELLO_FIM

        hello_fim_top 
           hello_fim_top (
        .clk   (clk_1x),

       .rst_n                      (rst_n_d_1x),
       .csr_lite_if             (bpf_rsv_5_slv_if)


        );
`endif

//*******************************
// AFU
//*******************************

You will connect the Hello_FIM DFH register to the existing BPF reserved link 5. The provided OFS reference design includes 3 reserved BPF interfaces available for custom usage such as new OPAE controlled modules. The address map of BPF is shown below:

Address	Size (Byte)	Feature	Master
0x00000 – 0x0FFFF	64K	FME (FME, Error, etc)	Yes
0x10000 – 0x1FFFF	64K	PMCI Proxy (SPI Controller)	Yes
0x20000 – 0x2FFFF	64K	PCIe CSR
0x30000 – 0x3FFFF	64K	HSSI CSR
0x40000 – 0x4FFFF	64K	EMIF CSR
0x50000 – 0x5FFFF	64K	Reserved
0x60000 – 0x6FFFF	64K	Reserved
0x70000 – 0x7FFFF	64K	Reserved

The BPF reserved link 5 is connected to a dummy connection to prevent this link from being optimized out the design during synthesis. You will add a compiler `define that will cause this dummy connection to be removed when the variable INCLUDE_HELLO_FIM is defined by editing line 575 if iofs_top.sv as shown below:

    // Reserved address response
   `ifndef INCLUDE_HELLO_FIM
    bpf_dummy_slv
    bpf_rsv_5_slv (
        .clk            (clk_1x),
        .dummy_slv_if   (bpf_rsv_5_slv_if)
    );
    `endif

5.1.2. ipss/mem/emif_csr.sv

The Hello_FIM DFH is inserted in the DFH link list after the EMIF CSR DFH and before the FME_PR DFH. The file ipss/d5005/emif/emif_csr.sv contains a parameter defining the next address for the next DFH in in the link list chain. You will change the next address offset to be 0x10000 so the reveserved BPF AXI lite link connected to the Hello_FIM DFH register is next in the DFH link list.

module emif_csr #(
  parameter NUM_LOCAL_MEM_BANKS = 1,
  parameter END_OF_LIST         = 1'b0,
  `ifndef INCLUDE_HELLO_FIM
  parameter NEXT_DFH_OFFSET     = 24'h05_0000
  `else
  parameter NEXT_DFH_OFFSET     = 24'h01_0000//New for Hello_FIM, next offset now at 0x50000
  `endif
)

5.1.3. src/hello_fim/hello_fim_top.sv

Create hello_fim_top.sv, and store it in src/hello_fim directory. The main purpose of this RTL is to convert AXI4-Lite interface to a simple interface to interface with the registers in hello_fim_com.sv.

For the purposes of this example, the hello_fim module instantiation sets the DFH feature ID (FEAT_ID) to 0xfff which is not currently defined. Using an undefined feature ID will result in no driver being used. Normally, a defined feature ID will be used to associate a specific driver with the FPGA module. Refer to the Device Feature List Feature IDs for a list of DFL feature types and IDs. If you are adding a new module to your design, make sure the Type/ID pair does not conflict with existing Type/ID pairs. You may reserve Type/ID pairs by submitting a pull request at the link above.

// ***************************************************************************
//                               INTEL CONFIDENTIAL
//
//        Copyright (C) 2021 Intel Corporation All Rights Reserved.
//
// The source code contained or described herein and all  documents related to
// the  source  code  ("Material")  are  owned  by  Intel  Corporation  or its
// suppliers  or  licensors.    Title  to  the  Material  remains  with  Intel
// Corporation or  its suppliers  and licensors.  The Material  contains trade
// secrets  and  proprietary  and  confidential  information  of  Intel or its
// suppliers and licensors.  The Material is protected  by worldwide copyright
// and trade secret laws and treaty provisions. No part of the Material may be
// used,   copied,   reproduced,   modified,   published,   uploaded,  posted,
// transmitted,  distributed,  or  disclosed  in any way without Intel's prior
// express written permission.
//
// No license under any patent,  copyright, trade secret or other intellectual
// property  right  is  granted  to  or  conferred  upon  you by disclosure or
// delivery  of  the  Materials, either expressly, by implication, inducement,
// estoppel or otherwise.  Any license under such intellectual property rights
// must be express and approved by Intel in writing.
//
// You will not, and will not allow any third party to modify, adapt, enhance, 
// disassemble, decompile, reverse engineer, change or create derivative works 
// from the Software except and only to the extent as specifically required by 
// mandatory applicable laws or any applicable third party license terms 
// accompanying the Software.
//
// -----------------------------------------------------------------------------
// Engineer     : 
// Create Date  : Nov 2021
// Module Name  : hello_fim_top.sv
// Project      : OFS
// -----------------------------------------------------------------------------
//
// Description: 
// This is a simple module that implements DFH registers and 
// AVMM address decoding logic.


module hello_fim_top  #(
   parameter ADDR_WIDTH  = 12, 
   parameter DATA_WIDTH = 64, 
   parameter bit [11:0] FEAT_ID = 12'hfff,
   parameter bit [3:0]  FEAT_VER = 4'h0,
   parameter bit [23:0] NEXT_DFH_OFFSET = 24'h04_0000,
   parameter bit END_OF_LIST = 1'b0
)(
   input  logic    clk,
   input  logic    rst_n,
// -----------------------------------------------------------
//  AXI4LITE Interface
// -----------------------------------------------------------
   ofs_fim_axi_lite_if.slave   csr_lite_if
);

import ofs_fim_cfg_pkg::*;
import ofs_csr_pkg::*;

//-------------------------------------
// Signals
//-------------------------------------
   logic [ADDR_WIDTH-1:0]              csr_waddr;
   logic [DATA_WIDTH-1:0]              csr_wdata;
   logic [DATA_WIDTH/8-1:0]            csr_wstrb;
   logic                               csr_write;
   logic                               csr_slv_wready;
   csr_access_type_t                   csr_write_type;

   logic [ADDR_WIDTH-1:0]              csr_raddr;
   logic                               csr_read;
   logic                               csr_read_32b;
   logic [DATA_WIDTH-1:0]              csr_readdata;
   logic                               csr_readdata_valid;
   logic [ADDR_WIDTH-1:0]              csr_addr;

   logic [63:0]                        com_csr_writedata;
   logic                               com_csr_read;
   logic                               com_csr_write;
   logic [63:0]                        com_csr_readdata;
   logic                               com_csr_readdatavalid;
   logic [5:0]                         com_csr_address;

// AXI-M CSR interfaces
ofs_fim_axi_mmio_if #(
   .AWID_WIDTH   (ofs_fim_cfg_pkg::MMIO_TID_WIDTH),
   .AWADDR_WIDTH (ADDR_WIDTH),
   .WDATA_WIDTH  (ofs_fim_cfg_pkg::MMIO_DATA_WIDTH),
   .ARID_WIDTH   (ofs_fim_cfg_pkg::MMIO_TID_WIDTH),
   .ARADDR_WIDTH (ADDR_WIDTH),
   .RDATA_WIDTH  (ofs_fim_cfg_pkg::MMIO_DATA_WIDTH)
) csr_if();

// AXI4-lite to AXI-M adapter
axi_lite2mmio axi_lite2mmio (
   .clk       (clk),
   .rst_n     (rst_n),
   .lite_if   (csr_lite_if),
   .mmio_if   (csr_if)
);

//---------------------------------
// Map AXI write/read request to CSR write/read,
// and send the write/read response back
//---------------------------------
ofs_fim_axi_csr_slave #(
   .ADDR_WIDTH (ADDR_WIDTH),
   .USE_SLV_READY (1'b1)

   ) csr_slave (
   .csr_if             (csr_if),

   .csr_write          (csr_write),
   .csr_waddr          (csr_waddr),
   .csr_write_type     (csr_write_type),
   .csr_wdata          (csr_wdata),
   .csr_wstrb          (csr_wstrb),
   .csr_slv_wready     (csr_slv_wready),
   .csr_read           (csr_read),
   .csr_raddr          (csr_raddr),
   .csr_read_32b       (csr_read_32b),
   .csr_readdata       (csr_readdata),
   .csr_readdata_valid (csr_readdata_valid)
);

// Address mapping
assign csr_addr             = csr_write ? csr_waddr : csr_raddr;
assign com_csr_address      = csr_addr[5:0];  // byte address
assign csr_slv_wready       = 1'b1 ;
// Write data mapping
assign com_csr_writedata    = csr_wdata;

// Read-Write mapping
always_comb
begin
   com_csr_read             = 1'b0;
   com_csr_write            = 1'b0;
   casez (csr_addr[11:6])
      6'h00 : begin // Common CSR
         com_csr_read       = csr_read;
         com_csr_write      = csr_write;
      end   
      default: begin
         com_csr_read       = 1'b0;
         com_csr_write      = 1'b0;
      end
   endcase
end

// Read data mapping
   always_comb begin
      if (com_csr_readdatavalid) begin
         csr_readdata       = com_csr_readdata;
         csr_readdata_valid = 1'b1;
      end
      else begin
         csr_readdata       = '0;
         csr_readdata_valid = 1'b0;
      end
   end

   hello_fim_com  #(
   .FEAT_ID          (FEAT_ID),
   .FEAT_VER         (FEAT_VER),
   .NEXT_DFH_OFFSET  (NEXT_DFH_OFFSET),
   .END_OF_LIST      (END_OF_LIST)
) hello_fim_com_inst (
   .clk                   (clk                     ),
   .rst_n                 (rst_n                  ),
   .writedata             (com_csr_writedata       ),
   .read                  (com_csr_read            ),
   .write                 (com_csr_write           ),
   .byteenable            (4'hF                    ),
   .readdata              (com_csr_readdata        ),
   .readdatavalid         (com_csr_readdatavalid   ),
   .address               (com_csr_address         )
   );


endmodule

5.1.4. src/hello_fim/hello_fim_com.sv

Create hello_fim_com.sv, and store it in src/hello_fim directory. This is the simple RTL to implement the Hello FIM registers. You may use this set of registers as the basis for your custom implementation.

// ***************************************************************************
//                               INTEL CONFIDENTIAL
//
//        Copyright (C) 2021 Intel Corporation All Rights Reserved.
//
// The source code contained or described herein and all  documents related to
// the  source  code  ("Material")  are  owned  by  Intel  Corporation  or its
// suppliers  or  licensors.    Title  to  the  Material  remains  with  Intel
// Corporation or  its suppliers  and licensors.  The Material  contains trade
// secrets  and  proprietary  and  confidential  information  of  Intel or its
// suppliers and licensors.  The Material is protected  by worldwide copyright
// and trade secret laws and treaty provisions. No part of the Material may be
// used,   copied,   reproduced,   modified,   published,   uploaded,  posted,
// transmitted,  distributed,  or  disclosed  in any way without Intel's prior
// express written permission.
//
// No license under any patent,  copyright, trade secret or other intellectual
// property  right  is  granted  to  or  conferred  upon  you by disclosure or
// delivery  of  the  Materials, either expressly, by implication, inducement,
// estoppel or otherwise.  Any license under such intellectual property rights
// must be express and approved by Intel in writing.
//
// You will not, and will not allow any third party to modify, adapt, enhance, 
// disassemble, decompile, reverse engineer, change or create derivative works 
// from the Software except and only to the extent as specifically required by 
// mandatory applicable laws or any applicable third party license terms 
// accompanying the Software.
//
// -----------------------------------------------------------------------------
// Engineer     : 
// Create Date  : Nov 2021
// Module Name  : hello_fim_com.sv
// Project      : IOFS
// -----------------------------------------------------------------------------
//
// Description: 
// This is a simple module that implements DFH registers and 
// AVMM address decoding logic.


module hello_fim_com #(
    parameter bit [11:0] FEAT_ID = 12'hfff,
    parameter bit [3:0]  FEAT_VER = 4'h0,
    parameter bit [23:0] NEXT_DFH_OFFSET = 24'h1000,
    parameter bit END_OF_LIST = 1'b0
 )(
 input clk,
 input rst_n,
 input [63:0] writedata,
 input read,
input write,
input [3:0] byteenable,
output reg [63:0] readdata,
output reg readdatavalid,
input [5:0] address
);


reg [63:0] rdata_comb;
reg [63:0] scratch_reg;

always @(posedge clk)  
   if (!rst_n) readdata[63:0] <= 64'h0; else readdata[63:0] <= rdata_comb[63:0];

always @(posedge clk)
   if (!rst_n) readdatavalid <= 1'b0; else readdatavalid <= read;

wire wr = write;
wire re = read;
wire [5:0] addr = address[5:0];
wire [63:0] din  = writedata [63:0];
wire wr_scratch_reg = wr & (addr[5:0]  == 6'h30)? byteenable[0]:1'b0;

// 64 bit scratch register
always @( posedge clk)
   if (!rst_n)  begin
      scratch_reg <= 64'h0;
   end
   else begin
   if (wr_scratch_reg) begin 
      scratch_reg <=  din;  
   end
end

always @ (*)
begin
rdata_comb = 64'h0000000000000000;
   if(re) begin
      case (addr)  
    6'h00 : begin
        rdata_comb [11:0]   = FEAT_ID ;  // dfh_feature_id  is reserved or a constant value, a read access gives the reset value
        rdata_comb [15:12]  = FEAT_VER ;  // dfh_feature_rev    is reserved or a constant value, a read access gives the reset value
        rdata_comb [39:16]  = NEXT_DFH_OFFSET ;  // dfh_dfh_ofst is reserved or a constant value, a read access gives the reset value
        rdata_comb [40]     = END_OF_LIST ;        //dfh_end_of_list
        rdata_comb [59:40]  = 20'h00000 ;  // dfh_rsvd1     is reserved or a constant value, a read access gives the reset value
        rdata_comb [63:60]  = 4'h3 ;  // dfh_feat_type  is reserved or a constant value, a read access gives the reset value
    end
    6'h30 : begin
        rdata_comb [63:0]   = scratch_reg; 
    end
        6'h38 : begin
                rdata_comb [63:0]       = 64'h6626_0701_5000_0034;
        end
    default : begin
        rdata_comb = 64'h0000000000000000;
    end
      endcase
   end
end

endmodule

5.1.5. Unit Level Simulations

To run a unit level simulation test for the updated RTL files, make modifications to your cloned /my_ofs_project/ofs-d5005/sim/d5005/unit_test/dfh_walker files. The following simulation files are updated to test the new hello_fim.

1. Edit sim/unit_test/dfh_walker/testbench/test_csr_defs.sv

   1. Update enum line 38

       typedef enum {
      FME_DFH_IDX,
      THERM_MNGM_DFH_IDX,
      GLBL_PERF_DFH_IDX,
      GLBL_ERROR_DFH_IDX,
      SPI_DFH_IDX,
      PCIE_DFH_IDX,
      HSSI_DFH_IDX,
      EMIF_DFH_IDX,
      HELLO_FIM_DFH_IDX,//New for HELLO_FIM
      FME_PR_DFH_IDX,
      PORT_DFH_IDX,
      USER_CLOCK_DFH_IDX,
      PORT_STP_DFH_IDX,
      AFU_INTF_DFH_IDX,
      MAX_FME_DFH_IDX
   } t_fme_dfh_idx;

1. Edit function dfh_name line 78

function automatic dfh_name[MAX_FME_DFH_IDX-1:0] get_fme_dfh_names();
      dfh_name[MAX_FME_DFH_IDX-1:0] fme_dfh_names;

      fme_dfh_names[FME_DFH_IDX]         = "FME_DFH";
      fme_dfh_names[THERM_MNGM_DFH_IDX]  = "THERM_MNGM_DFH";
      fme_dfh_names[GLBL_PERF_DFH_IDX]   = "GLBL_PERF_DFH";
      fme_dfh_names[GLBL_ERROR_DFH_IDX]  = "GLBL_ERROR_DFH";
      fme_dfh_names[SPI_DFH_IDX]         = "SPI_DFH";
      fme_dfh_names[PCIE_DFH_IDX]        = "PCIE_DFH";
      fme_dfh_names[HSSI_DFH_IDX]        = "HSSI_DFH";
      fme_dfh_names[EMIF_DFH_IDX]        = "EMIF_DFH";
      fme_dfh_names[HELLO_FIM_DFH_IDX]        = "HELLO_FIM_DFH";//New for HELLO_FIM
      fme_dfh_names[FME_PR_DFH_IDX]      = "FME_PR_DFH";
      fme_dfh_names[PORT_DFH_IDX]        = "PORT_DFH";
      fme_dfh_names[USER_CLOCK_DFH_IDX]  = "USER_CLOCK_DFH";
      fme_dfh_names[PORT_STP_DFH_IDX]    = "PORT_STP_DFH";
      fme_dfh_names[AFU_INTF_DFH_IDX]    = "AFU_INTF_DFH";

      return fme_dfh_names;
   endfunction

1. Update get_fme_dfh_values

  function automatic [MAX_FME_DFH_IDX-1:0][63:0] get_fme_dfh_values();
      logic[MAX_FME_DFH_IDX-1:0][63:0] fme_dfh_values;

      fme_dfh_values[FME_DFH_IDX]        = 64'h4000_0000_1000_0000;
      fme_dfh_values[THERM_MNGM_DFH_IDX] = 64'h3_00000_002000_0001;
      fme_dfh_values[GLBL_PERF_DFH_IDX]  = 64'h3_00000_001000_0007;
      fme_dfh_values[GLBL_ERROR_DFH_IDX] = 64'h3_00000_00C000_1004;  
      fme_dfh_values[SPI_DFH_IDX]        = 64'h3_00000_010000_000e;  
      fme_dfh_values[PCIE_DFH_IDX]       = 64'h3_00000_010000_0020;  
      fme_dfh_values[HSSI_DFH_IDX]       = 64'h3_00000_010000_100f;  
      fme_dfh_values[EMIF_DFH_IDX]       = 64'h3_00000_010000_0009; //Update to link to Hello_FIM 
      fme_dfh_values[HELLO_FIM_DFH_IDX]  = 64'h3_00000_040000_0FFF;  //New for Hello_FIM
      fme_dfh_values[FME_PR_DFH_IDX]     = 64'h3_00000_001000_1005;  
      fme_dfh_values[PORT_DFH_IDX]       = 64'h4000_0000_1000_1001;
      fme_dfh_values[USER_CLOCK_DFH_IDX] = 64'h3_00000_001000_0014;
      fme_dfh_values[PORT_STP_DFH_IDX]   = 64'h3_00000_00D000_2013;
      fme_dfh_values[AFU_INTF_DFH_IDX]   = 64'h3_00001_000000_2010; 

      return fme_dfh_values;
   endfunction

1. Update verification/scripts/Makefile_VCS.mk to set macro for INCLUDE_HELLO_FIM starting at line 56 to add +define+INCLUDE_HELLO_FIM

   VLOG_OPT += +define+SIM_MODE +define+VCS_S10 +define+RP_MAX_TAGS=64 +define+INCLUDE_DDR4 +define+INCLUDE_SPI_BRIDGE +define+INCLUDE_USER_CLOCK +define+INCLUDE_HSSI +define+SIM_USE_PCIE_DUMMY_CSR +define+INCLUDE_HELLO_FIM
   

2. Update sim/scripts/rtl_comb.f to add the path to your new hello_fim_top and hello_top_com SystemVerilog files. The update is shown below as the new line - 329 below:

$WORKDIR/src/hello_fim/hello_fim_com.sv
$WORKDIR/src/hello_fim/hello_fim_top.sv

After making these changes, run the unit level simulation using sim/unit_test/dfh_walker test. Before running, ensure your shell has the environment variables set properly as defined in Setting Up Required Environment Variables.

cd verification/scripts
gmake -f Makefile_VCS.mk cmplib
gmake -f Makefile_VCS.mk build run [DUMP=1]

Expected output:

 ********************************************
 Running TEST(0) : test_fme_dfh_walking
********************************************
READ64: address=0x00000000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x4000000010000000

FME_DFH
   Address   (0x0)
   DFH value (0x4000000010000000)
READ64: address=0x00001000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x3000000020000001

THERM_MNGM_DFH
   Address   (0x1000)
   DFH value (0x3000000020000001)
READ64: address=0x00003000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x3000000010000007

GLBL_PERF_DFH
   Address   (0x3000)
   DFH value (0x3000000010000007)
READ64: address=0x00004000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x30000000c0001004

GLBL_ERROR_DFH
   Address   (0x4000)
   DFH value (0x30000000c0001004)
READ64: address=0x00010000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x300000010000000e

SPI_DFH
   Address   (0x10000)
   DFH value (0x300000010000000e)
READ64: address=0x00020000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x3000000100000020

PCIE_DFH
   Address   (0x20000)
   DFH value (0x3000000100000020)
READ64: address=0x00030000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x300000010000100f

HSSI_DFH
   Address   (0x30000)
   DFH value (0x300000010000100f)
READ64: address=0x00040000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x3000000100000009

EMIF_DFH
   Address   (0x40000)
   DFH value (0x3000000100000009)
READ64: address=0x00050000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x3000000400000fff

HELLO_FIM_DFH
   Address   (0x50000)
   DFH value (0x3000000400000fff)
READ64: address=0x00090000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x3000000010001005

FME_PR_DFH
   Address   (0x90000)
   DFH value (0x3000000010001005)
READ64: address=0x00091000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x4000000010001001

PORT_DFH
   Address   (0x91000)
   DFH value (0x4000000010001001)
READ64: address=0x00092000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x3000000010000014

USER_CLOCK_DFH
   Address   (0x92000)
   DFH value (0x3000000010000014)
READ64: address=0x00093000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x30000000d0002013

PORT_STP_DFH
   Address   (0x93000)
   DFH value (0x30000000d0002013)
READ64: address=0x000a0000 bar=0 vf_active=0 pfn=0 vfn=0

   READDATA: 0x3000010000002010

AFU_INTF_DFH
   Address   (0xa0000)
   DFH value (0x3000010000002010)
MMIO error count matches: x

Test status: OK

********************
  Test summary
********************
   test_fme_dfh_walking (id=0) - pass
Test passed!
Assertion count: 0

5.1.6. syn/syn_top/d5005.qsf

1. Edit syn/syn_top/d5005.qsf

   1. Add new macro "INCLUDE_HELLO_FIM" line 107

              set_global_assignment -name VERILOG_MACRO "INCLUDE_HELLO_FIM"
      

   2. Add new line 211 to source TCL script with new hello_fim files

              set_global_assignment -name SOURCE_TCL_SCRIPT_FILE ../../../syn/setup/hello_fim_design_files.tcl
      

   5.1.7. syn/setup/hello_fim_design_files.tcl

Create "hello_fim_design_files.tcl" file and store in the syn/setup directory. This tcl file is called from d5005.qsf.

# Copyright 2021 Intel Corporation.
#
# THIS SOFTWARE MAY CONTAIN PREPRODUCTION CODE AND IS PROVIDED BY THE
# COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED
# WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
# BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
# WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
# OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
# EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
# Hello FIM Files
#--------------------
set_global_assignment -name SYSTEMVERILOG_FILE src/hello_fim/hello_fim_com.sv
set_global_assignment -name SYSTEMVERILOG_FILE src/hello_fim/hello_fim_top.sv

5.1.8. Build hello_fim example

With the preceding changes complete, build the new hello_fim example using the following steps:

cd $OFS_ROOTDIR
ofs-common/scripts/common/syn/build_top.sh d5005 work_d5005_hello_fim

Verify the design successfully compiled and timing closure is achieved by checking work_d5005_hello_fim/syn/syn_top/output_files/timing_report/clocks.sta.fail.summary - this file should be empty. If there are timing failures, then this file will list the failing clock domain(s).

5.1.9. Test the hello_fim on a D5005

Load the built FPGA binary file using an unsigned image. The FPGA image will be in work_d5005_hello_fim/syn/syn_top/output_files/d5005_page1_unsigned.bin

Provide the file d5005_page1_unsigned.bin on the server with the Intel® FPGA PAC D5005.

sudo fpgasupdate d5005_page1_unsigned.bin <D5005 PCIe B:D.F>
sudo rsu bmcimg <D5005 PCIe B:D.F>

Verify FPGA image is loaded.

sudo fpgainfo fme
## Output
Intel FPGA Programmable Acceleration Card D5005
Board Management Controller, MAX10 NIOS FW version: 2.0.13 
Board Management Controller, MAX10 Build version: 2.0.8 
//****** FME ******//
Object Id                        : 0xF000000
PCIe s:b:d.f                     : 0000:3B:00.0
Vendor Id                        : 0x8086
Device Id                        : 0xBCCE
SubVendor Id                     : 0x8086
SubDevice Id                     : 0x138D
Socket Id                        : 0x00
Ports Num                        : 01
Bitstream Id                     : 0x40100022C164DB1
Bitstream Version                : 4.0.1
Pr Interface Id                  : 7d91e0d0-4dcd-58c3-a93d-b9295e6e29b0
Boot Page                        : user

Use the OPAE SDK tool opae.io to check default driver binding using your card under test PCIe B:D.F. The steps below will use 0000:12:00.0 as the card under test PCIe B:D.F.

 sudo opae.io init -d 0000:12:00.0 $USER
 ##Output
 [0000:12:00.0] (0x8086, 0xbcce) Intel D5005 ADP (Driver: dfl-pci)

The dfl-pci driver is used by OPAE SDK fpgainfo commands. The next steps will bind the card under test to the vfio driver to enable access to the registers.

 sudo opae.io init -d 0000:12:00.0 $USER
 ##Output
 opae.io 0.2.3
Unbinding (0x8086,0xbcce) at 0000:12:00.0 from dfl-pci
Binding (0x8086,0xbcce) at 0000:12:00.0 to vfio-pci
iommu group for (0x8086,0xbcce) at 0000:12:00.0 is 35
Assigning /dev/vfio/35 to $USER

Confirm the vfio driver is bound to the card under test.

opae.io ls
## Output
opae.io 0.2.3
[0000:12:00.0] (0x8086, 0xbcce) Intel D5005 ADP (Driver: vfio-pci)

Run the following command to walk DFH link list. The new hello_fim register is located at offset 0x50000.

opae.io walk -d 0000:12:00.0
## Output
opae.io 0.2.3
offset: 0x0000, value: 0x4000000010000000
    dfh: id = 0x0, rev = 0x0, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x4
offset: 0x1000, value: 0x3000000020000001
    dfh: id = 0x1, rev = 0x0, next = 0x2000, eol = 0x0, reserved = 0x0, feature_type = 0x3
offset: 0x3000, value: 0x3000000010000007
    dfh: id = 0x7, rev = 0x0, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3
offset: 0x4000, value: 0x30000000c0001004
    dfh: id = 0x4, rev = 0x1, next = 0xc000, eol = 0x0, reserved = 0x0, feature_type = 0x3
offset: 0x10000, value: 0x300000010000000e
    dfh: id = 0xe, rev = 0x0, next = 0x10000, eol = 0x0, reserved = 0x0, feature_type = 0x3
offset: 0x20000, value: 0x3000000100000020
    dfh: id = 0x20, rev = 0x0, next = 0x10000, eol = 0x0, reserved = 0x0, feature_type = 0x3
offset: 0x30000, value: 0x300000010000100f
    dfh: id = 0xf, rev = 0x1, next = 0x10000, eol = 0x0, reserved = 0x0, feature_type = 0x3
offset: 0x40000, value: 0x3000000100000009
    dfh: id = 0x9, rev = 0x0, next = 0x10000, eol = 0x0, reserved = 0x0, feature_type = 0x3
offset: 0x50000, value: 0x3000000400000fff
    dfh: id = 0xfff, rev = 0x0, next = 0x40000, eol = 0x0, reserved = 0x0, feature_type = 0x3
offset: 0x90000, value: 0x3000000010001005
    dfh: id = 0x5, rev = 0x1, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3
offset: 0x91000, value: 0x4000000010001001
    dfh: id = 0x1, rev = 0x1, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x4
offset: 0x92000, value: 0x3000000010000014
    dfh: id = 0x14, rev = 0x0, next = 0x1000, eol = 0x0, reserved = 0x0, feature_type = 0x3
offset: 0x93000, value: 0x30000000d0002013
    dfh: id = 0x13, rev = 0x2, next = 0xd000, eol = 0x0, reserved = 0x0, feature_type = 0x3
offset: 0xa0000, value: 0x3000010000002010
    dfh: id = 0x10, rev = 0x2, next = 0x0, eol = 0x1, reserved = 0x0, feature_type = 0x3

Read the default values from the hello_fim registers:

$ opae.io -d 0000:12:00.0 -r 0 peek 0x50000
opae.io 0.2.3
0x3000000400000fff
$ opae.io -d 0000:12:00.0 -r 0 peek 0x50030
opae.io 0.2.3
0x0
$ opae.io -d 0000:12:00.0 -r 0 peek 0x50038
opae.io 0.2.3
0x6626070150000034

Write the scratchpad register at 0x50030

$ opae.io -d 0000:12:00.0 -r 0 poke 0x50038 0x123456789abcdef
opae.io 0.2.3
$ opae.io -d 0000:12:00.0 -r 0 peek 0x50038
opae.io 0.2.3
0x6626070150000034
$ opae.io -d 0000:12:00.0 -r 0 poke 0x50030 0x123456789abcdef
opae.io 0.2.3
$ opae.io -d 0000:12:00.0 -r 0 peek 0x50030
opae.io 0.2.3
0x123456789abcdef
$ opae.io -d 0000:12:00.0 -r 0 poke 0x50030 0xfedcba9876543210
opae.io 0.2.3
$ opae.io -d 0000:12:00.0 -r 0 peek 0x50030
opae.io 0.2.3
0xfedcba9876543210
$ opae.io -d 0000:12:00.0 -r 0 poke 0x50030 0x55550000aaaaffff
opae.io 0.2.3
$ opae.io -d 0000:12:00.0 -r 0 peek 0x50030
opae.io 0.2.3
0x55550000aaaaffff

Release the card under test from the vfio driver to re-bind to the dfl-pci driver:

sudo opae.io release -d 0000:12:00.0
## Output
opae.io 0.2.3
Releasing (0x8086,0xbcce) at 0000:12:00.0 from vfio-pci
Rebinding (0x8086,0xbcce) at 0000:12:00.0 to dfl-pci
$ sudo opae.io ls
opae.io 0.2.3
[0000:12:00.0] (0x8086, 0xbcce) Intel D5005 ADP (Driver: dfl-pci)

5.2. Memory Subsystem Modification

OFS enables modifications on the different subsystems that encompass the FIM. To customize the Memory Subsystem follow these instructions.

1. Set up the environment variables as described in section 4.2.1. Setting Up Required Environment Variables

2. Modify the NUM_MEM_CH parameter in src/afu_top/mux/top_cfg_pkg.sv Change NUM_MEM_CH from 4 to 2 as shown in below code

//=========================================================================================================================
//                         OFS Configuration Parameters                                                                 
//=========================================================================================================================
     parameter NUM_MEM_CH     = 2                                                 ,// Number of Memory/DDR Channel         
               NUM_HOST       = 1                                                 ,// Number of Host/Upstream Ports        
               NUM_PORT       = 4                                                 ,// Number of Functions/Downstream Ports 
               DATA_WIDTH     = 512                                               ,// Data Width of Interface              
               TOTAL_BAR_SIZE = 20                                                ,// Total Space for APF/BPF BARs (2^N) 
           //------------+-------------+-------------+-----------------+           //--------------------------------------
           // VF Active  |     PF #    |     VF #    |  Mux Port Map   |           //  PF/VF Mapping Parameters            
           //------------+-------------+-------------+-----------------+           //--------------------------------------
             CFG_VA = 0  , CFG_PF = 0  , CFG_VF =  0 ,  CFG_PID = 3    ,           //  Configuration Register Block        
             HLB_VA = 1  , HLB_PF = 0  , HLB_VF =  0 ,  HLB_PID = 0    ,           //  HE Loopback Engine                  
             PRG_VA = 1  , PRG_PF = 0  , PRG_VF =  1 ,  PRG_PID = 1    ,           //  Partial Reconfiguration Gasket      
             HSI_VA = 1  , HSI_PF = 0  , HSI_VF =  2 ,  HSI_PID = 2    ;           //  HSSI interface 

Compile a new FIM that incorporates the newly configured Memory Subsystem.

cd $OFS_BUILD_ROOT/ofs-d5005
ofs-common/scripts/common/syn/build_top.sh d5005 work_d5005_mem_2channel

***********************************
***
***        OFS_PROJECT: d5005
***        Q_PROJECT:  d5005
***        Q_REVISION: d5005
***        SEED: 03
***        Build Complete
***        Timing Passed!
***
***********************************

Program d5005_page1_unsigned.bin file using below command

sudo fpgasupdate d5005_page1_unsigned.bin 3b:00.0

Run rsu command

sudo rsu bmcimg 3b:00.0

Check if binary was loaded correctly

fpgainfo fme
## Output
Intel FPGA Programmable Acceleration Card D5005
Board Management Controller, MAX10 NIOS FW version: 2.0.8
Board Management Controller, MAX10 Build version: 2.0.8
//****** FME ******//
Object Id                        : 0xF000000
PCIe s:b:d.f                     : 0000:3B:00.0
Vendor Id                        : 0x8086
Device Id                        : 0xBCCE
SubVendor Id                     : 0x8086
SubDevice Id                     : 0x138D
Socket Id                        : 0x00
Ports Num                        : 01
Bitstream Id                     : 288511860124977321
Bitstream Version                : 4.0.1
Pr Interface Id                  : a195b6f7-cf23-5a2b-8ef9-1161e184ec4e
Boot Page                        : user

Run Host Excersiser to check Memory Subsystem performance

sudo host_exerciser mem
## Output
    starting test run, count of 1
API version: 1
AFU clock: 250 MHz
Allocate SRC Buffer
Allocate DST Buffer
Allocate DSM Buffer
    Host Exerciser Performance Counter:
    Host Exerciser numReads: 1024
    Host Exerciser numWrites: 1025
    Host Exerciser numPendReads: 0
    Host Exerciser numPendWrites: 0
    Host Exerciser numPendEmifReads: 0
    Host Exerciser numPendEmifWrites: 0
    Number of clocks: 5365
    Total number of Reads sent: 1024
    Total number of Writes sent: 1024
    Bandwidth: 3.054 GB/s
    Test mem(1): PASS

Verify Memory controller placement in syn/syn_top/output_files/d5005.fit.place.rpt file. Open fitter place stage report in any text editor of your choice, find keyword emif in the file. You should see emif[0] & emif[1] for Memory channel 0 & 1 respectively.

|emif[0].ddr4_pr_freeze_sync|                ; 0.4 (0.0)            ; 0.5 (0.0)                        ; 0.1 (0.0)                            ;
|resync_chains[0].synchronizer_nocut|        ; 0.4 (0.4)            ; 0.5 (0.5)                        ; 0.1 (0.1)                            ;
|emif[0].ddr4_softreset_sync|                ; 0.5 (0.0)            ; 0.7 (0.0)                        ; 0.2 (0.0)                            ;
|resync_chains[0].synchronizer_nocut|        ; 0.5 (0.5)            ; 0.7 (0.7)                        ; 0.2 (0.2)                            ;
|emif[0].pr_frz_afu_avmm_if|                 ; 647.5 (647.5)        ; 917.3 (917.3)                    ; 272.8 (272.8)                        ;
|emif[1].ddr4_pr_freeze_sync|                ; 0.4 (0.0)            ; 0.8 (0.0)                        ; 0.4 (0.0)                            ;
|resync_chains[0].synchronizer_nocut|        ; 0.4 (0.4)            ; 0.8 (0.8)                        ; 0.4 (0.4)                            ;
|emif[1].ddr4_softreset_sync|                ; 0.4 (0.0)            ; 1.0 (0.0)                        ; 0.6 (0.0)                            ;
|resync_chains[0].synchronizer_nocut|        ; 0.4 (0.4)            ; 1.0 (1.0)                        ; 0.6 (0.6)                            ;
|emif[1].pr_frz_afu_avmm_if|                 ; 641.1 (641.1)        ; 914.0 (914.0)                    ; 272.9 (272.9)                        ;
|p[0].pr_frz_fn2mx_a_port|                   ; 435.4 (0.0)          ; 476.2 (0.0)                      ; 40.8 (0.0)                           ;
|r.axis_pl_stage[0].axis_reg_inst|           ; 435.4 (435.4)        ; 476.2 (476.2)                    ; 40.8 (40.8)                          ;
|p[0].pr_frz_fn2mx_b_port|                   ; 434.6 (0.0)          ; 494.3 (0.0)                      ; 59.6 (0.0)                           ;

6. Conclusion

Using the OFS reference design and OPAE SDK enables the rapid creation of market leading FPGA based Acceleration systems. OFS facilitates customization of the FIM area for your custom board or platforms.

Notices & Disclaimers

Intel<sup>®</sup> technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. <sup>©</sup> Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.

OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group™.