oneAPI Accelerator Support Package(ASP) Reference Manual: Open FPGA Stack¶

1.0 Overview¶

1.1 About this Document¶

This document serves as a reference manual for platform designers wanting to enable oneAPI support on their Open FPGA Stack(OFS) platforms. The document describes essential hardware and software components required for enabling this design flow using OFS.

Please use table below as a quick reference for the contents in this document.

Table 1-1: Document Overview

Concept	Description	Sections
Overview, Prerequisites	Covers the flow of the document, important terms and prerequisite knowledge for this manual	* 1.0 Overview * 1.1 About this Document * 1.2 Terminology * 1.3 Prerequisites
Introduction to the oneAPI FPGA Acceleration Flow using Open FPGA Stack	Covers the introduction to accelerating oneAPI applications on an FPGA platform design using Open FPGA Stack. This gives an overview of the complete stack consisting of the oneAPI application and explains the high level tool . it also covers an introduction to what the oneAPI Accelerator Support Package	1.4 Introduction to oneAPI on Open FPGA Stack(OFS) * 1.5 Introduction to oneAPI Accelerator Support Package(ASP)
oneAPI Accelerator Support Package Design Fundamentals	This covers the information on design elements required to ensure the FPGA hardware design is functional with the oneAPI base toolkit. It covers both the components required in hardware design as well as in the software layer that oneAPI runtime requires	* 2.0 XML Files in oneAPI ASP * 2.1 board_spec.xml File * 2.2 board_env.xml File * 3.0 oneAPI ASP Hardware * 3.1 Host to External Memory Interface(EMIF) * 3.2 Host to Kernel Interface * 3.3 Kernel to External Memory Interface * 4.0 oneAPI ASP Software * 4.1 Memory Mapped Device(MMD) Layer * 4.2 Board Utilities
oneAPI ASP Implementation on OFS Reference Platforms	Provides detailed explanation of the oneAPI ASP design for OFS reference platforms. These sections cover information about the harware as well as software design files in the `oneapi-asp` repository	* 5.0 oneapi-asp Implementation Details * 5.1 oneapi-asp Directory Structure * 5.2 oneapi-asp Build Flow * 5.3 oneapi-asp Hardware Implementation * 5.4 oneapi-asp Memory Mapped Device(MMD) Layer Implementation * 5.5 oneapi-asp Utilities Implementation
Appendix	This currently has information about additional features provided in the `oneapi-asp` repository. Appendix A covers debug capabilities in the MMD. Appendix B has information about a new functionality added in the `oneapi-asp` tag ofs-2024.2-1 called `oneAPI ASP Editor`, enabling a GUI based approach to design oneAPI ASPs	* Appendix A: Memory Mapped Device(MMD) Layer Debug Variables * Appendix B: oneAPI Accelerator Support Package(ASP) Editor

Note: Table 1-1 in oneAPI Accelerator Support Package (ASP): Getting Started User Guide lists OFS reference platforms.

For more information about developing application kernels for FPGA using oneAPI Base Toolkit (Base Kit) refer to Intel® FPGA Add-on for oneAPI Base Toolkit webpage.

1.2 Terminology¶

This table defines some of the common terms used when discussing OFS.

Table 1-2: Terminology

Term	Abbreviation	Description
Open FPGA Stack	OFS	A modular collection of hardware platform components, open source software, and broad ecosystem support that provides a standard and scalable model for AFU and software developers to optimize and reuse their designs.
Accelerator Functional Unit	AFU	Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance. Note: An AFU region is the part of the design where an AFU may reside. This AFU may or may not be a partial reconfiguration region.
FPGA Interface Manager	FIM	Provides platform management, functionality, clocks, resets and standard interfaces to host and AFUs. The FIM resides in the static region of the FPGA and contains the FPGA Management Engine (FME) and I/O ring.
High Level Design	HLD	For the purpose of this guide, this term refers to designing with High Level Design tools like oneAPI Base Toolkit (Base Kit).
oneAPI Accelerator Support Package	oneAPI ASP	A collection of hardware and software components that enable oneAPI kernel to communicate with oneAPI runtime and other OFS hardware, software components. oneAPI ASP hardware components and oneAPI kernel form the AFU region of a oneAPI system in OFS.
Intel® FPGA Basic Building Blocks	BBB	Basic Building Blocks (BBB) for Altera® FPGAs is a suite of application building blocks and shims like Memory Properties Factory (MPF).
BBB Memory Properties Factory	BBB MPF	Intel® FPGA BBB MPF block provides features like virtual to physical address (VTP), ordering read responses, read/write hazard detection, and masked (partial) writes. `oneapi-asp` uses MPF VTP feature.
Open Programmable Acceleration Engine Software Development Kit	OPAE SDK	A collection of libraries and tools to facilitate the development of software applications and accelerators using OPAE.
Platform Interface Manager	PIM	An interface manager that comprises two components: a configurable platform specific interface for board developers and a collection of shims that AFU developers can use to handle clock crossing, response sorting, buffering and different protocols.
Device Feature List	DFL	A concept inherited from OFS. The DFL drivers provide support for FPGA devices that are designed to support the Device Feature List. The DFL, which is implemented in RTL, consists of a self-describing data structure in PCI BAR space that allows the DFL driver to automatically load the drivers required for a given FPGA configuration.
Best Known Configuration	BKC	The exact hardware configuration that the solution has been optimized and validated against.
SYCL	-	SYCL (pronounced "sickle") is a royalty-free, cross-platform abstraction layer that enables code for heterogeneous and offload processors to be written using modern ISO C++ (at least C++ 17). It provides several features that make it well-suited for programming heterogeneous systems, allowing the same code to be used for CPUs, GPUs, FPGAs or any other hardware accelerator. SYCL was developed by the Khronos Group, a non-profit organization that develops open standards (including OpenCL™) for graphics, compute, vision, and multimedia. SYCL is being used by a growing number of developers in a variety of industries, including automotive, aerospace, and consumer electronics.
Data Parallel C++	DPC++	DPC++ is Intel’s implementation of the SYCL standard. It supports additional attributes and language extensions which ensure DCP++ (SYCL) is efficiently implanted on Intel hardware.
Installable Client Driver	ICD	Intel® FPGA Runtime for OpenCL™ Software Technology supports the OpenCL ICD extension from the Khronos Group™. The OpenCL ICD extension allows you to have multiple OpenCL implementations on your system. With the OpenCL ICD Loader Library, you may choose from a list of installed platforms and execute OpenCL API calls that are specific to your OpenCL implementation of choice.
FPGA Client Driver	FCD	Intel® FPGA Runtime for OpenCL™ Software Technology supports FPGA Client Driver(FCD) extension. FCD allows the runtime to automatically find and load the oneAPI ASP libraries at host run time

Note: oneapi-asp was referred to as ofs-hld-shim in OFS (for Agilex® 7 FPGA & Stratix® 10 FPGA) and OpenCL AFU Shim (ofs-opencl-afu-shim) in OFS early access(EA) release (for Stratix® 10 FPGA with Intel® FPGA PAC D5005 as reference platform).

1.3 Prerequisites¶

The content in this manual requires readers to be familiar with:

Hardware and software components of Open FPGA Stack, especially the following:
- FPGA Interface Manager(FIM)
  - Stratix® 10 FPGA:
    - Shell Technical Reference Manual: OFS for Stratix® 10 PCIe Attach FPGAs
    - Shell Developer Guide: OFS for Stratix® 10 PCIe Attach FPGAs
  - Agilex® 7 FPGA:
- Accelerator Functional Unit(AFU)
  - Stratix® 10 FPGA: Workload Developer Guide: OFS for Stratix® 10 PCIe Attach FPGAs
  - Agilex® 7 FPGA: Workload Developer Guide: OFS for Agilex® 7 PCIe Attach FPGAs
- Software Reference Manual: Open FPGA Stack (OPAE SDK and Linux-DFL section)
- ofs-platform-afu-bbb

In addition to above, developers must be familiar with the following tools & concepts:

Quartus® Prime Design Software (Quartus® software revisions, Platform Designer, compilation Flows, timing analysis, compilation reports, understanding FPGA resource utilization, programming Altera® FPGAs)
Partial Reconfiguration (PR)
FPGA Peripheral IPs (PCIe, External Memory IP, Ethernet IP)
Avalon® Interface
Scripting (TCL, Python, Shell scripts)
Verilog, SystemVerilog
C++
Familiarity with SYCL
Familiarity with oneAPI compilation process for FPGAs & oneAPI code samples
Familiarity with oneAPI Accelerator Support Package (ASP): Getting Started User Guide

1.4 Introduction to oneAPI on Open FPGA Stack(OFS)¶

The Intel® oneAPI Base Toolkit (Base Kit) is a core set of tools and libraries for developing high-performance, data-centric applications across diverse architectures (CPUs, GPUs and FPGAs). It features an industry-leading C++ compiler that implements SYCL, an evolution of C++ for heterogeneous computing.

Figure 1-1 shows the high-level representation of oneAPI application developers using FPGAs for acceleration. The runtime flow consists of a host code running on a processor and an application kernel code running on an FPGA. Open FPGA Stack enables vendors to enable support for this flow on their platforms.

Figure 1-1: oneAPI Application on OFS Platforms

oneAPI Base Toolkit (Base Kit) consists of a compiler and runtime environment. The compiler converts a SYCL kernel (FPGA application code) into a hardware circuit. This hardware circuit requires additional logic to communicate with the runtime and FPGA board peripherals. This additional logic is provided by oneAPI Accelerator Support Package(oneAPI ASP). oneAPI ASP consists of hardware components that enable this generated hardware circuit to communicate with the host processor as well as software components that enable the runtime to identify and communicate with the kernel.

Figure 1-2 shows the workload design steps and steps in which the oneAPI Base Toolkit (Base Kit) requires oneAPI ASP as input. For more information about workload development and how workload developers target a specific platform during compilation, refer to Intel oneAPI Programming Guide and Intel® oneAPI DPC++/C++ Compiler Handbook for Intel® FPGAs. The next section introduces oneAPI ASP.

Figure 1-2: High Level Design Flow for FPGAs with oneAPI Base Toolkit (Base Kit)

1.5 Introduction to oneAPI Accelerator Support Package(ASP)¶

As mentioned in previous section, oneAPI ASP is a collection of hardware and software components that interface with the hardware circuit generated by the oneAPI compiler. The hardware circuit generated by the oneAPI compiler from a oneAPI kernel is referred to as the kernel system. While the kernel system consists of logic controlled by the workload developer's specifications, the kernel system interfaces are generated by the oneAPI compiler based on specifications provided by the oneAPI ASP designer. These specifications are input to the compiler using XML files (discussed in section 2.0).

Note: All the interfaces generated by the oneAPI compiler are Avalon® Interfaces.

Figure 1-3: Kernel System Interfaces

Figure 1-3 shows a high-level representation of an OFS hardware design and interfaces to/from kernel_system. The numbered arrows depict the following:

Path 1 represents host-to-External Memory Interface (EMIF)
Path 2 represents the host to kernel interface
Path 3 represents kernel to EMIF
Path 4 represents kernel to Unified Shared Memory (USM) Interface
Path 5 represents kernel to HSSI interface

oneAPI ASP hardware components can be divided into 3 categories:

RTL components: constituting various interface logic, for example, host to External Memory Interface (EMIF), kernel to EMIF interface, host to kernel interface, kernel to host memory interface as well as additional components to handle kernel control signals and perform Direct Memory Access (DMA)
XML files: for describing hardware interfaces and compilation environment to oneAPI Base Toolkit (Base Kit)
Scripts: to control compile flow

In addition to the hardware components, a software layer is required for handling I/O operations between oneAPI runtime and the board. The oneAPI ASP software layer can be divided into 2 categories:

Memory Mapped Device (MMD) Layer: required by the host & runtime to communicate with the oneAPI kernel & other oneAPI ASP hardware registers
oneAPI ASP utilities: required to setup and diagnose the board

The MMD uses API provided by OPAE SDK to communicate with the device. The FPGA driver is provided by the linux-DFL kernel driver.

Figure 1-4 shows how the above oneAPI ASP components tie into Open FPGA Stack.

Figure 1-4: Open FPGA Stack (OFS) components

2.0 XML Files in oneAPI ASP¶

The kernel system interfaces generated by the oneAPI compiler are based on specifications provided by oneAPI ASP developer. An XML file, called board_spec.xml is used to pass the specifications to the oneAPI compiler. oneAPI ASP developers must create this XML file for their boards.

In addition to board_spec.xml, the oneAPI Base Toolkit (Base Kit) relies on another XML file called board_env.xml to get information about the board environment. The board_env.xml file helps the runtime setup board installation.

The next section explains the contents of board_spec.xml. Section 2.2 covers contents of board_env.xml file.

2.1 `board_spec.xml` File¶

A typical board_spec.xml structure is shown in Fig 2-1. In addition to kernel system interfaces, the board_spec.xml file is also used to specify other compilation details like Quartus® Prime Pro Edition Software version used in platform design, compilation scripts to help control Quartus® software compile flows, FPGA resource utilization details.

Elements of board_spec.xml file are summarized in table 2-1. Each element has additional attributes and parameters. Details are covered in respective sections for each element.

Figure 2-1: board_spec.xml File Structure

Table 2-1: Elements of board_spec.xml

Element	Use of Element	Attributes
board	Used to specify a name for the board and the version of Quartus® Prime Pro Edition Software used to develop the platform design. This board name is used to identify the board design to be compiled and must match the name of the directory in which board_spec.xml resides.	version, name
compile	Used to describe different compile flows	project, revision, qsys_file, generic_kernel, generate_cmd, synthesize_cmd, auto_migrate
device	Used to specify the FPGA device model file for the FPGA part on the board.	device_model, used_resources
global_mem	Different attributes in this element are used to provide details about the external memory used as the global memory for the FPGA oneAPI kernel/application.	name, max_bandwidth, interleaved_bytes, config_addr, default, interface
host	Used to specify the offset at which the kernel resides.	kernel_config
interfaces	Used to specify control signals to oneAPI kernels	interface, kernel_clk_reset
channels	Used to describe interface to stream data directly between kernels and I/O	interface

The compiler expects a separate board_spec.xml file for every board variant a platform supports. Board variants are different hardware design implementations for the same platform, a oneAPI ASP can have multiple board variants. A oneAPI kernel developer can select the board variant suitable for their application at compile time.

A board_spec.xml file must be located at the top most level of each board variant's hardware directory (the hardware directory is specified by board_env.xml, please refer to section 2.2 for details on hardware element). For example, a separate board_spec.xml file for each board variant for OFS reference platforms is located in oneapi-asp/Platform-Name/hardware/Board-Variant/ directory, where Platform-Name is n6001, fseries-dk, iseries-dk for OFS targeting Agilex® 7 FPGA and d5005 for OFS targeting Stratix® 10 FPGA.

2.1.1 board Element¶

The board element of the board_spec.xml file provides the Quartus® Prime Pro Edition Software version and the name of the board.

Table 2-2: Attributes for the board Element

Attribute	Description
version	The version of the board. The board version should match the version of the Quartus® Prime Pro Edition Software you use to develop the platform design. The oneAPI compiler uses this value to perform environment checks for supported version during application compile
name	The name of the accelerator board, which must match the name of the directory in which the board_spec.xml file resides. The name must contain a combination of only letters, numbers, underscores (_), hyphens (-), or periods (.) (for example, ofs_n6000).

Example below shows the board element populated for a board designed with Quartus® Prime Pro Edition Software version 22.3 and board variant named "Agilex_brd1".

Note: A board variant name is different from a platform directory name. Please see Note in section 2.2 for more information on board variants.

Figure 2-2: board Element

2.1.2 compile Element¶

Depending on the application requirements, the design may have different compilation flows and different design settings for each flow (for example, there can be a flat flow without partial reconfiguration support or a flow with partitions in the design to enable partial reconfiguration). Designers can control the flow and its settings using scripts. To allow selection of compile flow during application compile & to describe control of Quartus® software compilation as well as registration, automigration, the compile element of the board_spec.xml file and its associated attributes and parameters are used.

Table 2-3: Attributes for compile Element

Attribute	Description
name	Name of the compilation flow. This name can be used to differentiate between flows at oneAPI kernel compilation time. oneAPI compiler allows selecting a compile flow using `-Xsbsp-flow` option.
project	Name of the Quartus® software project file (.qpf) that the Quartus® Prime Pro Edition Software intends to compile.
revision	Name of the revision within the Quartus® software project that the Quartus® Prime Pro Edition Software compiles to generate the final bitstream.
qsys_file	Name of the Platform Designer file into which the oneAPI kernel is embedded. You have the option to assign a value of "none" to qsys_file if you do not require the Quartus® Prime Pro Edition Software to create a top-level .qsys file for your design. In this scenario, oneAPI compiler adds a .qip file into the Quartus® software project. In this case, the custom oneAPI ASP must manually instantiate the generated HDL entity (generated entity is in the kernel_system.v file).
generic_kernel	Set this value to 1 if you want the offline compiler to generate a common Verilog interface for all compilations. This setting is necessary in situations where you must set up design partitions around the kernel, such as in the Configuration via Protocol (CvP) flow.
generate_cmd	Command required to prepare for full compilation, such as to generate the Verilog files for the Platform Designer system into which the oneAPI kernel is embedded.
synthesize_cmd	Command required to generate the fpga.bin file from the Custom Platform. Usually, this command instructs the Quartus® Prime Pro Edition Software to perform a full compilation.
auto_migrate	platform_type—Choose this value based on the value referenced in the Reference Platform from which you derive your Custom Platform. Valid values are a10_ref, s10_ref, and none. include fixes—Comma-separated list of named fixes that you want to apply to the Custom Platform. *exclude fixes—Comma-separated list of named fixes that you do not want to apply to the Custom Platform.

Example below shows a populated compile element for a sample Quartus® software Project called ofs.qpf, the Quartus® software revision to be compiled is called asp (asp.qsf). In this example, the compiler generates the kernel system (entity is called kernel_system) and this entity is instantiated manually in the Quartus® software project (e.g. in a file called kernel_wrapper.v), hence qsys_file is set to "none". The synthesize_cmd points to a script "compile.tcl" located in the same directory as the board_spec.xml, compile script performs all necessary system generation and compile steps for generation of final bitstream. The project directory snippet below is for demonstration only. The compile flow is named "demo_flow".

There can be multiple compile elements for the different compilation flows that a platform designer wishes to enable in their platform (e.g. different revisions with different Quartus® software settings or a PR revision).

Figure 2-3: compile Element

2.1.3 device Element¶

A device model(DM) file is an XML file that has the total resources on the device (i.e. ALMs, FFs, DSPs, RAMs). This is required for any FPGA part used in a oneAPI design. Most device model files are provided as part of the oneAPI Base Toolkit (Base Kit) installation ($INTELFPGAOCLSDKROOT/share/models/dm, where INTELFPGAOCLSDKROOT is set by the setvars.sh environment setup script provided by oneAPI toolkit). If the device model file for your part number is not included in $INTELFPGAOCLSDKROOT/share/models/dm, it must be created and placed in the same folder as board_spec.xml. A new device model file can be created using existing files as reference.

The device model file name must be specified in the device_model attribute of device element. The used_resources attribute is used to specify the resources being utilized by the oneAPI ASP and peripheral IPs. The utilization by non-kernel logic is calculated during platform design. The compiler utilizes the total resources from device model file and utilized resources in used_resources section to estimate the available resources for application kernel.

Table 2-4: Attributes for device Element

Attribute

Description

device_model

The file name of the device model file that describes the available FPGA resources on the accelerator board.

used_resources

Reports the number of adaptive logic modules (ALMs), flip-flops, digital signal processor (DSP) blocks and RAM blocks that the board design consumes in the absence of any kernel. If you create a defined partition around all the board logic, you can obtain the used resources data from the Partition Statistics section of the Fitter report. Extract the information from the following parameters:
* alms num — The number of logic ALMs used, excluding the number of ALMs with only their registers used. The value should correspond to [a]+[b]+[d] from part [A] of the Fitter Partition Statistics.
* ffs num — The number of flip flops.
* dsps num — The number of DSP blocks.
* rams num — The number of RAM blocks.

Example below shows the device element added for a Agilex® 7 FPGA based platform with device model file named "agfb014r24a2e2vr0_dm.xml". The number of used_resources are for demonstration purposes and are not to be used by oneAPI ASP developers.

Figure 2-4: device Element

2.1.4 interface Attribute¶

Note: This is different from the interfaces element discussed in upcoming sections. In the board_spec.xml file, each global memory, channel or kernel connection is comprised of individual interfaces. For the global_mem, channels, and interfaces XML elements, an interface attribute must be included to specify the corresponding parameters for each connection.

Table 2-5: Parameters for interface attribute

Parameter	Description	Applicable Interface
name	* For global_mem: instance name of the Platform Designer component. * For channels: instance name of the Platform Designer component that has the channel interface. * For interfaces: name of the entity in which the kernel interface resides (for example, board).	All
port	port parameter can be defined either inline in `interface` attribute or as a separate element in `interface` attribute. See section 2.1.4.1 for more information.	All
type	* For global_mem: set to agent. * For channels: - Set to streamsource for a stream source that provides data to the kernel. - Set to streamsink for a stream sink interface that consumes data from the kernel. * For interfaces: set to either host, irq, or streamsource.	All
width	* For global_mem: width of the memory interface in bits. * For channels: number of bits in the channel interface. * For interfaces: width of the kernel interface in bits.	All
waitrequest_allowance	* For global_mem: [Optional] Amount of Avalon®-MM waitrequest allowance supported on the agent interface (that is, kernel-facing interface) of the clock-crossing bridge that spans between the memory and the kernel clock domains. * For kernel_cra: [Optional] Amount of Avalon®-MM waitrequest allowance that the kernel_cra agent interface must support. This parameter defaults to 0 if you do not specify it in the board_spec.xml file. A value of 0 indicates that this waitrequest allowance feature is disabled.	All
maxburst	Maximum burst size for the agent interface. Attention: The value of width ÷ 8 x maxburst must be less than the value of interleaved_bytes.	global_mem
address	Starting address of the memory interface that corresponds to the host interface-side address. For example, address 0 should correspond to the bank1 memory host from the `Memory Bank Divider`. In addition, any non-zero starting address must abut the end address of the previous memory.	global_mem
size	Size of the memory interface in bytes. * For global_mem: The sizes of all memory interfaces should be equal. * For interfaces: interface can have variable sizes.	global_mem interfaces > Note: Support for size parameter for interface attribute is available in oneAPI Base Toolkit (Base Kit) version 2024.0 and beyond.
latency_type	If the memory interface has variable latency, set this parameter to average to signify that the specified latency is considered the average case. If the complete kernel-to-memory path has a guaranteed fixed latency, set this parameter to fixed.	global_mem
chan_id	A string used to identify the channel interface. The string may have up to 128 characters.	channels
clock	For the streamsource kernel interface type, the parameter specifies the name of the clock that the snoop stream uses. Usually, this clock is the kernel clock.	interfaces

Example for how the interface attribute is used in global_mem and interfaces elements is covered in section for these elements respectively.

2.1.4.1 port Parameter¶

As mentioned in Table 2-5, port parameter can be defined either inline in interface attribute or as a separate element in interface attribute. The definition method to use depends on the direction of the port.

If the direction of the port is either read or write, it must be a separate element in interface attribute.
If the direction is readwrite, port must be inline with the port name in interface attribute. No direction specification is required.

Table below shows the port element attributes.

Table 2-6: port parameter

Parameter	Description
name	* For global_mem: name of the Avalon®-MM interface in the Platform Designer component that corresponds to the interface attribute. * For channels: name of the streaming interface in the Platform Designer component. * For interfaces: name of the interface to the `Kernel Interface` Platform Designer component. For example, kernel_cra is the Avalon®-MM interface, and kernel_irq is an interrupt.
direction	Direction of the port. Valid values are: * "r" : Indicates read * "w" : Indicates `write`

Snippet below shows the inline and separate element definitions of port parameter.

Note: Direction specification for port is available in oneAPI Base Toolkit (Base Kit) versions 2024.0 and beyond. For versions prior to oneAPI Base Toolkit (Base Kit) version 2024.0, only the default inline definition of port parameter is supported.

Examples for global_mem and interfaces elements in sections below use the inline definition of port.

2.1.5 global_mem Element¶

The global_mem element of the board_spec.xml file is used to provide information on the memory interfaces that connect to the kernel.

Note: For each global memory that the kernel accesses, you must include one interface element that describes its characteristics. The different attributes for global_mem element are discussed in table below.

Table 2-7: Attributes for global_mem Element

Attribute	Description
name	The name FPGA application/kernel developer should use to identify the memory type. Each name must be unique and must comprise of less than 32 characters.
max_bandwidth	The maximum bandwidth, in megabytes per second (MB/s), of all global memory interfaces combined in their current configuration. The oneAPI compiler uses max_bandwidth to choose an architecture suitable for the application and the board. Compute this bandwidth value from datasheets of memories on your board. Example max_bandwidth calculation for a 64-bit DDR4 interface running at 1200 MHz: max_bandwidth = 1200 MHz x 2 x 64 bits ÷ 8-bits = 19200 MB/s The max_bandwidth value will change based on global memory configuration, for example, if the memory configuration comprises of 4 banks of DDR4 configured as a single homogenous memory, the max_bandwidth will be 19200 x 4 (i.e. number of memory interfaces from kernel). Please see section 2.1.5.1 for more information on global memory configurations. Designers have the option to use block RAM instead of or in conjunction with external memory as global memory. The formula for calculating max_bandwidth for block RAM is max_bandwidth = block RAM speed x (block RAM interface size ÷ 8 bits). Example max_bandwidth calculation for a 512-bit block RAM running at 100 MHz: max_bandwidth = 100 MHz x 512 bits ÷ 8 bits = 6400 MB/s
interleaved_bytes	Include the interleaved_bytes attribute in the board_spec.xml file when you instantiate multiple interfaces(i.e. memory banks) for a given global memory system. This attribute controls the size of data that the offline compiler distributes across the interfaces. The offline compiler currently can interleave data across banks no finer than the size of one full burst. This attribute specifies this size in bytes and following are the recommended values: For two or fewer global memory banks: maxburst x width_bytes For four or more global memory banks: maxburst x width_bytes x 4 The interleaved_bytes value must be the same for the host interface and the kernels. Therefore, the configuration of the `Memory Bank Divider` must match the exported kernel agent interfaces in this respect (refer to section 3.1.1 for information about `Memory Bank Divider`) For block RAM, interleaved_bytes equals the width of the interface in bytes.
config_addr	The address of the ACL Mem Organization Control Platform Designer component (mem_org_mode) that the host software uses to configure memory. You may omit this attribute if your board has homogeneous memory; the software uses the default address (0x18) for this component. If your board has heterogeneous memory, there is a mem_org_mode component in the board system for each memory type. Enter the config_addr attribute and set it to the value of the base address of the mem_org_mode component(s).
default	Include this optional attribute and assign a value of 1 to set the global memory as the default memory interface. The default memory must start at address 0x0. If you do not implement this attribute, the first memory type defined in the board_spec.xml file becomes the default memory interface.
interface	See the `interface` section above for the parameters you must specify for each interface.
allocation_type	A list that specifies which USM allocator is used to allocate from the global_mem element. Values allowed in this list are `host`, `shared`, and `device`. The following conditions apply: If there are multiple global_mem elements with the same allocation_type attribute, the first allocation_type attribute in the board_spec.xml is assumed to be the one used by the specified allocator. If there is a single global_mem element with multiple allocation_type attributes, this indicates that allocations of the specified types use this global_mem interface. [Legacy support] If you have not specified the allocation_type attribute, it is assumed that all global memory interfaces have the `device` allocation_type.

Example below shows a global_mem element configuration for a kernel system connected to four 4GB DDR4 memory banks. The DDR4 interface is 64 bit operating at 1200MHz. Note that the name of the platform designer system name is board.qsys. As mentioned in description for interleaved_bytes in table above, the Memory Bank Divider configuration ensures that the host interface matches the interleaved_bytes setting (i.e. 512 bits x 64 burst size = 4096 bytes). For information on waitrequest_allowance, refer to section 2.1.4 on interface attribute.

Note: More details on the Memory Bank Divider and the Clock Crossing Bridges is covered in section 3.0

Figure 2-6: Memory Connection Example Block Diagram and Corresponding global_mem Element in board_spec.xml

2.1.5.1 Global Memory Configurations¶

A board can have a single memory bank, multiple memory banks of the same type (e.g. 4 banks of DDR4) or different banks of different types.

The partitioning of memory for oneAPI kernel developers is explained in the Intel® oneAPI DPC++/C++ Compiler Handbook for Intel® FPGAs. The global memory configuration required by an application kernel must match the configuration in board_spec.xml as the compiler uses this information to generate a suitable architecture for the application. The different memory configurations are

A single global memory region (possible with same type of memory banks)
Different global memories (heterogeneous memory)

2.1.5.1.1 Contiguous Global Memory¶

For boards with multiple memory banks of the same type, designers can configure these as a single contiguous global memory region. This is done by specifying each memory interface within a single global_mem element. Figure 2-6 showed 4 DDR4 memory banks configured as a single global memory region.

With this configuration, FPGA application developers have the option to use contiguous memory region in an interleaved or a non-interleaved fashion. Even with contiguous memory regions, kernel developers can partition data buffers across the banks/memory channels. Please refer to Global Memory Access Optimization section in Intel® oneAPI DPC++/C++ Compiler Handbook for Intel® FPGAs for more details on these partitioning techniques.

2.1.5.1.2 Heterogeneous Memory¶

For boards with different memory technologies, designers must specify each type of memory that the kernel needs to access as a separate global memory.

Figure 2-7 shows heterogeneous configurations and the global_mem element structure for two different types of memories (QDR, DDR4). The global_mem element in example below also demonstrates use of the default attribute. It is set to "1" for the DDR4 memory banks, indicating to the oneAPI compiler that the default global memory for the kernel is DDR4.

Figure 2-7: Heterogeneous Memory Example Block Diagram and Corresponding global_mem Element in board_spec.xml

2.1.5.1.3 Unified Shared Memory¶

For applications that require USM support, the board_spec.xml must specify host and device memories in a heterogeneous manner. The allocation_type must be host for global memory region on the host processor. The allocation_type must be set to device for global memory on the FPGA board. Example below extends the board_spec.xml snippet in figure 2-6 to add a global_mem element for the kernel system to host processor memory interface.

Figure 2-8: global_mem Element Example for Unified Shared Memory(USM)

2.1.6 host Element¶

The host element of the board_spec.xml file provides information on the interface from the host to the kernel. Figure below shows an example of host element.

Figure 2-9: host Element Example

Table 2-8: Attributes for the host Element

Attribute	Description
kernel_config	This attribute informs the oneAPI compiler at what offset the kernel resides, from the perspective of the `kernel_cra` host on the kernel_interface module. * start: the starting address of the kernel. Normally, this attribute has a value of 0 because the kernel_cra host should not host anything except kernels. * size: keep this parameter at the default value of 0x0100000.

2.1.7 interfaces Element¶

The interfaces element of the board_spec.xml file describes the kernel interfaces that connect to application kernels and control kernel behavior. For this element, include one of each interface of types host, irq and streamsource. Refer to the interface section for the parameters you must specify for each interface. In addition to the host, irq, and streamsource interfaces, if your design includes a separate Platform Designer subsystem containing the board logic, the kernel clock and reset interfaces exported from it are also part of the interfaces element. Specify these interfaces with the kernel_clk_reset attribute and its corresponding parameters.

Figure below shows example of interfaces element.

Figure 2-10: interfaces Element Example

Table 2-9: Parameters for the kernel_clk_reset Attribute

Attribute	Description
clk	The Platform Designer name for the kernel clock interface
clk2x	The Platform Designer name for the 2xkernel clock interface
reset	The Platform Designer connection for the kernel reset

Note: Name the kernel clock and reset interfaces in the Platform Designer connection format (that is, .). For example: board.kernel_clk

2.1.8 channels Element¶

The channels element provides channels for streaming data directly between kernel and I/O. Each channel (implemented using Avalon-ST specification) must be connected to the kernel via the interface attribute. The channel interface only supports data, and valid and ready Avalon-ST signals. The I/O channel defaults to 8-bit symbols and big-endian ordering at the interface level.

Figure below shows an example of channels element for a single channel with a width of 64 bits. The chan_id attribute identified helps identify the port in the generated kernel_system. Refer to section 2.1.4 for more information about the interface attribute parameters. Additional interface attributes can be added for additional channels.

Figure 2-11: channels Element Example

For more information about kernel development using channels, refer to I/O Pipes section in Intel® oneAPI DPC++/C++ Compiler Handbook for Intel® FPGAs.

2.2 `board_env.xml` File¶

The board_env.xml file is used by the oneAPI toolkit to set up the board installation that enables the compiler to target a specific accelerator platform. The board_env.xml file must be located in the top most level of the oneAPI ASP for each platform. For example, the board_env.xml for oneAPI ASP for OFS reference platforms is located in the oneapi-asp/Platform-Name folder, where Platform-Name is n6001, fseries-dk, iseries-dk for OFS targeting Agilex® 7 FPGA and d5005 for OFS targeting Stratix® 10 FPGA.

A sample board_env.xml file is shown below. Table 2-10 explains the elements of this file.

Figure 2-12: board_env.xml File Structure

Table 2-10: Specifications of XML Elements and Attributes in the board_env.xml File

Element	Attribute Description
board_env	* version: The oneAPI compiler version used to create oneAPI ASP * name: The runtime uses this as the name of the FPGA Client Driver(FCD) file name
hardware	* dir: Name of the subdirectory, within the oneAPI ASP directory, that contains the board variant directories for a platform * default: The default board variant that the compiler targets when a platform has multiple board variants and user does not specify an explicit argument using -Xstarget option
platform	name: Name of the operating system. A separate `platform` element must be specified for each supported OS for the oneAPI ASP platform
mmdlib	A string that specifies the path to the MMD library of your oneAPI ASP. To load multiple libraries, specify them in an ordered, comma-separated list. The host application will load the libraries in the order that they appear in the list > Note: You can use `%b` to reference your oneAPI ASP directory and provide path relative to oneAPI ASP directory, for example, if MMD library is located inside `linux64/lib` folder in oneAPI ASP, the path would be `%b/linux64/lib/libintel_opae_mmd.so`
linkflags	A string that specifies the linker flags necessary for linking with the MMD layer available with the board > Note: You can use %b to reference your oneAPI ASP directory and provide path relative to oneAPI ASP directory, for example, if MMD library is located inside `linux64/lib` folder in oneAPI ASP, the path would be `%b/linux64/lib`.
linklibs	A string that specifies the libraries the oneAPI runtime must link against to use the MMD layer available with the board
utilbindir	Directory in which the runtime expects to locate board utility executables (i.e. install, uninstall, program, diagnose, flash) > Note: You can use %b to reference your oneAPI ASP directory and provide path relative to oneAPI ASP directory, for example, if the utilities are located in `linux64/libexec` folder in oneAPI ASP, the path would be `%b/linux64/libexec`

3.0 oneAPI ASP Hardware¶

The oneAPI compiler generates the kernel system interfaces based on specifications provided by the oneAPI ASP developer in the board_spec.xml file. The kernel system interfaces with the rest of the oneAPI ASP RTL as shown in figure 1-3.

Figure 1-3 shows 5 different paths, summarized below:

Host to EMIF: Consisting of RTL to handle data transfer between host and on-board memory (e.g. DDR4)
Host to Kernel: Consisting of RTL to handle control signals & interrupts between host and kernel
Kernel to EMIF: Consisting of RTL to handle data transfer between kernel and on-board memory
Kernel to Host memory: Required to support Unified Shared Memory. This requires some additional RTL to handle data transfer between kernel and host memory
Kernel to HSSI: Consisting of RTL to handle data streaming between kernel and I/O

Please note that the kernel system generated by oneAPI compiler has Avalon® interfaces. OFS FIM has AXI interfaces. Additional logic blocks from Platform Interface Manager are used to handle protocol conversions. Please refer to section 5.3.1 for more details on PIM. The next few sections cover some of the important IP components required to enable kernel communications with host and board peripherals. More design implementation details are covered in section 5.0.

3.1 Host to External Memory Interface(EMIF)¶

The host to EMIF datapath consists of a PCIe Subsytem(SS), EMIF Subsystem located in the FIM and a Direct Memory Access(DMA) engine in the oneAPI ASP.

PCIe Subsystem(SS) has the PCIe IP and additional logic to handle PCIe packet format and routing. FIM handles routing signals received from host to the user design located in a region referred to as Accelerator Functional Unit(AFU) (the Kernel system resides in the AFU).

Note: For more information about the PCIe SS, please refer to Intel FPGA IP Subsystem for PCI Express IP User Guide

The External Memory Interface Subsystem (EMIF SS) consists of EMIF IP and additional logic for handling transfers between AFU and on-board memories.

Note: For more information about the EMIF SS, please refer to Memory Subsystem Intel FPGA IP User Guide

Large buffers of data are usually transferred between host and on-board memory in oneAPI applications. This necessitates a Direct Memory Access(DMA) Engine between host and on-board memory. In oneAPI ASP designs for OFS reference platform, this DMA engine is placed in the AFU region.

As described in section 2.1.5.1, there are different configurations for memories on board. In addition to above, figure 1-3 also shows an additional IP in the host to memory datapath, called Memory Bank Divider. This IP is used for handling one of the most commonly used configurations, i.e. configuring multiple memory banks of same type as a contiguous memory region. In this case, the kernel has a contiguous view of the memory and data can be interleaved across the different memory channels. The host must also have the same view of the memory in order to ensure read and write transactions from correct addresses.

3.1.1 Memory Bank Divider¶

The Memory Bank Divider is a oneAPI ASP IP component that takes an incoming request from the host interface on the Avalon®-MM agent port and routes it to the appropriate bank host port. This component must reside on the path between the host and the global memory interfaces. In addition, it must reside outside of the path between the kernel and the global memory interfaces.

The following image shows the IP interfaces. Table 3-2 provides more information on the interface signals.

Figure 3-1: Memory Bank Divider IP

This IP is currently made available as part of the oneapi-asp located in oneapi-asp/common/hardware/common/build/ip folder (memory_bank_divider_hw.tcl). The IP can be instantiated in a top level system by passing values to the parameters as shown in snippet below. Table 3-1 provides more information on the parameters. The oneAPI ASP for OFS reference platforms instantiates this IP in a similar way. Refer to section 5.0 for more information on implementation details for oneAPI ASP for OFS reference platforms.

    add_instance memory_bank_divider memory_bank_divider <version>
    set_instance_parameter_value memory_bank_divider {NUM_BANKS} {number-of-memory-banks}
    set_instance_parameter_value memory_bank_divider {SEPARATE_RW_PORTS} {true/false-value}
    set_instance_parameter_value memory_bank_divider {PIPELINE_OUTPUTS} {true/false-value}
    set_instance_parameter_value memory_bank_divider {SPLIT_ON_BURSTBOUNDARY} {true/false-value}
    set_instance_parameter_value memory_bank_divider {DATA_WIDTH} {data-width}
    set_instance_parameter_value memory_bank_divider {ADDRESS_WIDTH} {total-addressable-width}
    set_instance_parameter_value memory_bank_divider {BURST_SIZE} {burst-size}
    set_instance_parameter_value memory_bank_divider {MAX_PENDING_READS} {max-pending-reads}
    set_instance_parameter_value memory_bank_divider {ASYNC_RESET} {value}
    set_instance_parameter_value memory_bank_divider {SYNCHRONIZE_RESET} {value}

Table 3-1: Parameter Settings for the Memory Bank Divider Component

Parameter Name in IP TCL	Parameter	Description
NUM_BANKS	Number of banks	Number of memory banks for each of the global memory types included in your board system. A single memory bank divider has the following allowed values: 1,2,4,8 for this parameter.
SEPARATE_RW_PORTS	Separate read/write ports	Enable this parameter so that each bank has one port for read operation and one for write operation.
PIPELINE_OUTPUTS	Add pipeline stage to output	Enable this parameter to allow for potential timing improvements.
DATA_WIDTH	Data Width	Width of the data bus to the memory in bits.
ADDRESS_WIDTH	Address Width (total addressable)	Total number of address bits necessary to address all global memory.
BURST_SIZE	Burst size (maximum)	Set to a value equal to interleaved_bytes/(width/8), where interleaved_bytes and width are defined in the interface attribute of the global_mem element in the `board_spec.xml` file.
MAX_PENDING_READS	Maximum Pending Reads	Maximum number of pending read transfers the component can process without asserting a waitrequest signal. Recommended value is 64 if ASP has two global memory banks or fewer and 128 if ASP has four or more global memory banks. CAUTION: A high Maximum Pending Reads value causes Platform Designer to insert a deep response FIFO buffer, between the component's host and agent, that consumes a lot of device resources. It also increases the achievable bandwidth between host and memory interfaces.
SPLIT_ON_BURSTBOUNDARY	Split read/write bursts on burst word boundary	Enable splitting of read and write bursts on burst word boundary. Enable this parameter if the Number of banks parameter value is greater than 1, and the burst reads and writes that the host controller sends to the agent port crosses burst word boundary.

Table 3-2: Signals and Ports for the Memory Bank Divider Component

Signal or Port	Description
clk	The bank divider logic uses this clock input. If the IP of your host and memory interfaces have different clocks, ensure that clk clock rate is not slower than the slowest of the two IP clocks.
reset	The reset input that connects to the board power-on reset.
s	The agent port that connects to the host interface controller.
kernel_clk	The kernel_clk drives this clock input
kernel_reset	The kernel_reset output from the `Kernel Interface` IP drives this reset input.
acl_asp_snoop	Export this Avalon® Streaming (Avalon®-ST) source. In the board_spec.xml file, under interfaces, describe only the snoop interface for the default memory (acl_internal_snoop). If you have a heterogeneous memory design, perform these tasks only for the `Memory Bank Divider` component associated with the default memory. Important: The memory system you build in HW tcl (e.g. `ddr_board_hw.tcl` in oneapi-asp for OFS reference platforms. Refer to section 5 for more details on `ddr_board_hw.tcl`) alters the width of acl_asp_snoop. You must update the width of the streamsource interface within the channels element in the `board_spec.xml` file to match the width of acl_asp_snoop. In the `board_spec.xml` file, update the width of the snoop interface (acl_internal_snoop) specified with the streamsource kernel interface within the interfaces element. Updating the width ensures that the global_mem interface entries in `board_spec.xml` match the characteristics of the bank`N` Avalon®-MM hosts from corresponding `Memory Bank Divider` component for the default memory.
acl_asp_memorg_host	This conduit connects to the acl_asp_memorg_host interface of the `Kernel Interface` IP. > Note: Signal present if Number of banks > 1.
bank1, bank2, ..., bank8	The number of memory hosts available in the `Memory Bank Divider` depends on the number of memory banks that were included when the unit was instantiated. Connect each bank with each memory interface in the same order as the starting address for the corresponding kernel memory interface specified in the board_spec.xml file. For example, global_mem interface that begins at address 0 must correspond to the memory host in bank1 from the `Memory Bank Divider`.

3.2 Host to Kernel Interface¶

The host exchanges control signals with kernel with the help of an additional oneAPI ASP IP . The control signals coming from the host are on a different clock domain (PCIe clock) while the kernel runs on different clock frequency . The Kernel Interface IP handles the clock domain crossing for these control signals as well as handles communication with kernel CSR, interrupts and generates the reset for oneAPI kernel. All oneAPI ASP designs must instantiate Kernel Interface IPs to ensure the kernel functions correctly.

3.2.1 Kernel Interface¶

The Kernel Interface is an oneAPI ASP component that allows the host interface to access and control the oneAPI kernel.

The following image shows the IP interfaces. Table 3-4 provides more information on the interface signals.

Figure 3-2: Kernel Interface IP

This IP is currently made available as part of the oneapi-asp located in oneapi-asp/common/hardware/common/build/ip folder (kernel_interface_hw.tcl). The IP can be instantiated in a top level system by passing values to the parameters as shown in snippet below. Table 3-3 provides more information on the parameter. The oneAPI ASP for OFS reference platforms instantiates this IP in a similar way. Refer to section 5.0 for more information on implementation details for oneAPI ASP for OFS reference platforms.

    add_instance kernel_interface kernel_interface <version>
    set_instance_parameter_value kernel_interface {NUM_GLOBAL_MEMS} {value}

Table 3-3: Parameter Settings for the Kernel Interface Component

Parameter Name in IP TCL	Parameter	Description
NUM_GLOBAL_MEMS	Number of global memory systems	Number of global memory types in your board design.

Table 3-4: Signals and Ports for the Kernel Interface Component

Signal or Port	Description
clk	The clock input used for the host control interface. The clock rate of clk can be slow.
reset	This reset input resets the control interface. It also triggers the kernel_reset signal, which resets all kernel logic.
ctrl	Use this agent port to connect to the host interface. This interface is a low-speed interface with which you set kernel arguments and start the kernel's execution.
kernel_clk	kernel clock drives this clock input.
kernel_cra	This Avalon®-MM host interface communicates directly with the kernels generated by the oneAPI compiler. Export the Avalon®-MM interface to the Kernel Interface and name it in the board_spec.xml file.
sw_reset_in	When necessary, the host interface resets the kernel via the ctrl interface. If the board design requires a kernel reset, it can do so via this reset input. Otherwise, connect the interface to a global power-on reset.
kernel_reset	Use this reset output to reset the kernel and any other hardware that communicates with the kernel. Warning: This reset occurs between the MMD open and close calls. Therefore, it must not reset anything necessary for the operation of your MMD.
sw_reset_export	This reset output is the same as kernel_reset, but it is synchronized to the clk interface. Use this output to reset logic that is not in the kernel_clk clock domain but should be reset whenever the kernel resets.
acl_asp_memorg_host	The memory interfaces use these signals. Based on the number of global memory systems you specify in the `Kernel Interface` component parameter editor, the Quartus® Prime Pro Edition Software creates the corresponding number of copies of this signal, each with a different hexadecimal suffix. Connect each signal to the `Memory Bank Divider` component associated with each global memory system (for example, DDR). Then, list the hexadecimal suffix in the config_addr attribute of the global_mem element in the board_spec.xml file.
kernel_irq_from_kernel	An interrupt input from the kernel. This signal is exported and named in the board_spec.xml file.
kernel_irq_to_host	An interrupt output from the kernel. This signal connects to the host interface.

3.3 Kernel to External Memory Interface¶

The kernel system masters the interface from kernel to external memory. oneAPI compiler generates kernel system memory interface logic (e.g. Load-Store Unit) according to the global memory configuration and interface specifications in board_spec.xml file. The kernel system operates at kernel clock, hence, oneAPI ASP developers must handle clock domain crossing from kernel to EMIF clock domain.

For implementation details for all datapaths discussed above, please refer to section 5.3.

4.0 oneAPI ASP Software¶

The software components of oneAPI ASP consist of the Memory Mapped Device(MMD) layer and the board utility routine required by runtime.

Section 4.1 introduces MMD layer and section 4.2 explains board utilities.

4.1 Memory Mapped Device(MMD) Layer¶

The oneAPI ASP Memory Mapped Device (MMD) layer sits in between the oneAPI runtime and OPAE SDK and provides a set of API for device communication and control. The runtime calls into the MMD API for various operations like opening a handle to the device, allocating memory etc.

Note: For more information about the FPGA runtime, please refer to FPGA Runtime documentation here.

A header file, called aocl_mmd.h, has the list of MMD API calls that must be implemented by oneAPI ASPs. From the perspective of the caller, below is typical MMD API lifecycle:

Open device to provide handle for further operations
Set interrupt and status handlers
Program device with kernel bitstream
Allocate memory if required
Perform Read, Write operations (DMA or MMIO)
Free memory if allocation done in step 4
Close device. No further operations permitted until subsequent open device call

Table below summarizes all APIs listed in aocl_mmd.h.

Table 4-1: Summary of MMD API from aocl_mmd.h

API	Purpose
aocl_mmd_get_offline_info	Obtain offline information about the board. This function is offline because it is device-independent and does not require a handle from the `aocl_mmd_open()` call
aocl_mmd_get_info	Obtain information about the board specified in the `requested_info_id` argument (refer to section 4.1.2 for more information)
aocl_mmd_open	Open and initialize the specified device
aocl_mmd_close	Close an opened device via its handle
aocl_mmd_set_interrupt_handler	Set the interrupt handler for the opened device
aocl_mmd_set_device_interrupt_handler	Sets the device interrupt handler for opened device, interrupt handler is called to notify runtime of any exceptions
aocl_mmd_set_status_handler	Set the operation status handler for the opened device
aocl_mmd_yield	The `aocl_mmd_yield` function is called when the host interface is idle. The host interface might be idle because it is waiting for the device to process certain events
aocl_mmd_read	Read operation on a single interface
aocl_mmd_write	Write operation on a single interface
aocl_mmd_copy	Copy operation on a single interface
aocl_mmd_hostchannel_create	Creates a channel interface
aocl_mmd_hostchannel_destroy	Destroys channel interface
aocl_mmd_hostchannel_get_buffer	Provides host with pointer used to read/write from channel interface
aocl_mmd_hostchannel_ack_buffer	Acknowledges read/write from channel
aocl_mmd_program	Reprogram operation for the specified device
aocl_mmd_host_alloc	Provide memory that is allocated on the host. Host allocations are accessible by the host and one or more devices
aocl_mmd_free	Free memory that has been allocated by MMD
aocl_mmd_device_alloc	Allocate memory that is owned by the device
aocl_mmd_shared_alloc	Allocate shared memory between the host and the FPGA
aocl_mmd_shared_migrate	Handle migration of non-concurrent shared allocations any time the accessor of the allocation changes

Sections below cover more details for each API (expected arguments, return values). Section 5.4 discusses more about the implementation of the MMD layer APIs in oneAPI ASPs for OFS reference platforms.

4.1.1 `aocl_mmd_get_offline_info`¶

The aocl_mmd_get_offline_info function obtains offline information about the board specified in the requested_info_id argument. This function is offline because it is device-independent and does not require a handle from the aocl_mmd_open() call.

Syntax


    int aocl_mmd_get_offline_info (
        aocl_mmd_offline_info_t requested_info_id,
        size_t param_value_size,
        void* param_value,
        size_t* param_size_ret )

Function Arguments

requested_info_id: An enum value of type aocl_mmd_offline_info_t that indicates the offline device information returning to the caller.

Table 4-2: Possible Enum Values for the requested_info_id Argument

Name	Description	Type
AOCL_MMD_VERSION	Version of MMD layer	char*
AOCL_MMD_NUM_BOARDS	Number of candidate boards	int
AOCL_MMD_BOARD_NAMES	Names of available boards > Note: Separate each board name by a semicolon (;) delimiter.	char*
AOCL_MMD_VENDOR_NAME	Name of board vendor	char*
AOCL_MMD_VENDOR_ID	An integer board vendor ID	int
AOCL_MMD_USES_YIELD	A value of 0 instructs the runtime to suspend user's processes. The runtime resumes these processes after it receives an event update (for example, an interrupt) from the MMD layer. A value of 1 instructs the runtime to continuously call the `aocl_mmd_yield` function while it waits for events to complete. CAUTION: Setting `AOCL_MMD_USES_YIELD` to 1 might cause high CPU utilization if the `aocl_mmd_yield` function does not suspend the current thread.	int

param_value_size: Size of the param_value field in bytes. This size_t value should match the size of the expected return type that the enum definition indicates. For example, if AOCL_MMD_NUM_BOARDS returns a value of type int, set the param_value_size to sizeof (int). You should see the same number of bytes returned in the param_size_ret argument.
param_value: A void* pointer to the variable that receives the returned information.
param_size_ret: A pointer argument of type size_t* that receives the number of bytes of returned data.

Return Value

A negative return value indicates an error.

4.1.2 `aocl_mmd_get_info`¶

The aocl_mmd_get_info function obtains information about the board specified in the requested_info_id argument.

Syntax


    int aocl_mmd_get_info (
        int handle,
        aocl_mmd_info_t requested_info_id,
        size_t param_value_size,
        void* param_value,
        size_t* param_size_ret )

Function Arguments

handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
requested_info_id: An enum value of type aocl_mmd_info_t that indicates the device information returning to the caller.

Table 4-3: Possible Enum Values for the requested_info_id Argument

Name	Description	Type
AOCL_MMD_NUM_KERNEL_INTERFACES	Number of kernel interfaces	int
AOCL_MMD_KERNEL_INTERFACES	Kernel interfaces	int*
AOCL_MMD_PLL_INTERFACES	Kernel clock handles	int*
AOCL_MMD_MEMORY_INTERFACE	Global memory handle	int
AOCL_MMD_TERMPERATURE	Temperature measurement	float
AOCL_MMD_PCIE_INFO	PCIe® information	char*
AOCL_MMD_BOARD_NAME	Board name	char*
AOCL_MMD_BOARD_UNIQUE_ID	Unique board ID	char*
AOCL_MMD_CONCURRENT_READS	Number of parallel reads A value of 1 indicates serial reads.	int
AOCL_MMD_CONCURRENT_WRITES	Number of parallel writes A value of 1 indicates serial writes.	int
AOCL_MMD_CONCURRENT_READS_OR_WRITES	Total number of concurrent read and write operations	int
AOCL_MMD_MIN_HOST_MEMORY_ALIGNMENT	Minimum alignment that the oneAPI ASP supports for host allocations	size_t
AOCL_MMD_HOST_MEM_CAPABILITIES	Capabilities of `aocl_mmd_host_alloc()` function	unsigned int
AOCL_MMD_SHARED_MEM_CAPABILITIES	Capabilities of `aocl_mmd_shared_alloc()` function	unsigned int
AOCL_MMD_DEVICE_MEM_CAPABILITIES	Capabilities of `aocl_mmd_device_alloc()` function	unsigned int
AOCL_MMD_HOST_MEM_CONCURRENT_GRANULARITY	Granularity of concurrent host accesses	size_t
AOCL_MMD_SHARED_MEM_CONCURRENT_GRANULARITY	Granularity of concurrent shared accesses	size_t
AOCL_MMD_DEVICE_MEM_CONCURRENT_GRANULARITY	Granularity of concurrent device accesses	size_t

param_value_size: Size of the param_value field in bytes. This size_t value should match the size of the expected return type that the enum definition indicates. For example, if AOCL_MMD_TEMPERATURE returns a value of type float, set the param_value_size to sizeof (float). You should see the same number of bytes returned in the param_size_ret argument.
param_value: A void* pointer to the variable that receives the returned information.
param_size_ret: A pointer argument of type size_t* that receives the number of bytes of returned data.

Capability Values

Table 4-4: Capability Values for aocl_mmd_get_info Function

Value	Description
AOCL_MMD_MEM_CAPABILITY_SUPPORTED	If you do not set this value, allocation function is not supported even if other capabilities are set.
AOCL_MMD_MEM_CAPABILITY_ATOMIC	Supports atomic access to the memory by either the host or the device.
AOCL_MMD_MEM_CAPABILITY_CONCURRENT	Supports concurrent access to the memory either by the host or the device if the accesses are not on the same block. Block granularity is defined by AOCL_MMD_*_MEM_CONCURRENT_GRANULARITY. Blocks are aligned to this granularity.
AOCL_MMD_MEM_CAPABILITY_P2P	Memory can be accessed by multiple devices at the same time.

Return Value

A negative return value indicates an error.

4.1.3 `aocl_mmd_open`¶

The aocl_mmd_open function opens and initializes the specified device.

Syntax


    int aocl_mmd_open (const char *name)

Function Arguments

name: The function opens the board with a name that matches this const char* string. The name typically matches the one specified by the AOCL_MMD_BOARD_NAMES offline information.

The runtime first queries the AOCL_MMD_BOARD_NAMES offline information to identify the boards that it might be able to open. Then it attempts to open all possible devices by calling aocl_mmd_open and using each of the board names as argument.

Note: The name must be a C-style NULL-terminated ASCII string.

Return Value

If aocl_mmd_open() executes successfully, the return value is a positive integer that acts as a handle to the board.

If aocl_mmd_open() fails to execute, a negative return value indicates an error. In the event of an error, the runtime proceeds to open other known devices. Therefore, it is imperative that the MMD layer does not exit the application if an open call fails.

4.1.4 `aocl_mmd_close`¶

The aocl_mmd_close function closes an opened device via its handle.

Syntax


    int aocl_mmd_close (int handle)

Function Arguments

handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.

Return Value

If the aocl_mmd_close() executes successfully, the return value is 0.

If aocl_mmd_close() fails to execute, a negative return value indicates an error.

4.1.5 `aocl_mmd_set_interrupt_handler`¶

The aocl_mmd_set_interrupt_handler function sets the interrupt handler for the opened device. When the device internals identify an asynchronous kernel event (for example, a kernel completion), the interrupt handler is called to notify the runtime of the event.

Note: Ignore the interrupts from the kernel until this handler is set.

Syntax


    int aocl_mmd_set_interrupt_handler (
        int handle, 
        aocl_mmd_interrupt_handler_fn fn,
        void* user_data )

Function Arguments

handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
fn: The callback function to invoke when a kernel interrupt occurs. The fn argument is of type aocl_mmd_interrupt_handler_fn, which is defined as follows:

    typedef void (*aocl_mmd_interrupt_handler_fn)( int handle, void* user_data );

user_data: The void* type user-provided data that passes to fn when it is called.

Return Value

If the function executes successfully, the return value is 0.

If the function fails to execute, a negative return value indicates an error.

4.1.6 `aocl_mmd_set_device_interrupt_handler`¶

The aocl_mmd_set_device_interrupt_handler function sets the device interrupt handler for the opened device. When the device internals identify an asynchronous exception event (for example, a bit correction event), the device interrupt handler is called to notify the runtime of the event.

Note: Ignore the interrupts from the device until this handler is set.

Syntax


    int aocl_mmd_set_device_interrupt_handler (
        int handle, 
        aocl_mmd_device_interrupt_handler_fn fn,
        void* user_data )

Function Arguments

handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
fn: The callback function to invoke when a kernel interrupt occurs. The fn argument is of type aocl_mmd_device_interrupt_handler_fn, which is defined as follows:

 
    typedef void (*aocl_mmd_device_interrupt_handler_fn)( int handle, aocl_mmd_interrupt_info* data_in, void* user_data );

aocl_mmd_interrupt_info is defined as:


    typedef struct {
        unsigned long long int exception_type;
        void *user_private_info;
        size_t user_cb;
    } aocl_mmd_interrupt_info;

Where:

exception_type acts as a bitfield that contains exactly one bit, corresponding to an exception number.
user_private_info and user_cb represent pointers to binary data that the OpenCL implementation return. These pointers log additional information that is helpful for debugging the error.

user_data: The void* type user-provided data that passes to fn when it is called.

Return Value

If the function executes successfully, the return value is 0.

If the function fails to execute, a negative return value indicates an error.

4.1.7 `aocl_mmd_set_status_handler`¶

The aocl_mmd_set_status_handler function sets the operation status handler for the opened device. The operation status handler is called under the following circumstances:

When the operation completes successfully and status is 0.
When the operation completes with errors and status is a negative value.

Syntax


    int aocl_mmd_set_status_handler (
        int handle,
        aocl_mmd_status_handler_fn fn,
        void* user_data )

Function Arguments

handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
fn: The callback function to invoke when a status update occurs. The fn argument is of type aocl_mmd_status_handler_fn, which is defined as follows:

    typedef void (*aocl_mmd_status_handler_fn)( int handle, void* user_data, aocl_mmd_op_t op, int status );

user_data: The void* type user-provided data that passes to fn when it is called.

Return Value

If the function executes successfully, the return value is 0.

If the function fails to execute, a negative return value indicates an error.

4.1.8 `aocl_mmd_yield`¶

The aocl_mmd_yield function is called when the host interface is idle. The host interface might be idle because it is waiting for the device to process certain events.

Syntax


    int aocl_mmd_yield (int handle)

Function Arguments

handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.

Return Value

A nonzero return value indicates that the yield function performed work necessary for proper device functioning such as processing direct memory access (DMA) transactions.

A return value of 0 indicates that the yield function did not perform work necessary for proper device functioning.

Note: The yield function might be called continuously if it reports that it has necessary work to perform.

4.1.9 `aocl_mmd_read`¶

The aocl_mmd_read function is the read operation on a single interface.

Syntax


    int aocl_mmd_read (
        int handle,
        aocl_mmd_op_t op,
        size_t len,
        void* dst,
        int mmd_interface, size_t offset )

Function Arguments

handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
op: The operation object of type aocl_mmd_op_t used to track the progress of the operation. If op is NULL, the call must block, and return only after the operation completes.

Note: aocl_mmd_op_t is defined as follows:

    typedef void* aocl_mmd_op_t;

len: The size of the data, in bytes, that the function transfers. Declare len with type size_t.
dst: The host buffer, of type void*, to which data is written.
mmd_interface: the handle to the interface being accessed. For example, to access global memory this handle will be value obtained from aocl_mmd_get_info call with AOCL_MMD_MEMORY_INTERFACE as requested_info_id argument.
offset: The size_t byte offset within the interface at which the data transfer begins.

Return Value

If the read operation is successful, the return value is 0.

If the read operation fails, a negative return value indicates an error.

4.1.10 `aocl_mmd_write`¶

The aocl_mmd_write function is the write operation on a single interface.

Syntax


    int aocl_mmd_write (
        int handle,
        aocl_mmd_op_t op,
        size_t len,
        const void* src,
        int mmd_interface, size_t offset )

Function Arguments

handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
op: The operation object of type aocl_mmd_op_t used to track the progress of the operation. If op is NULL, the call must block, and return only after the operation completes.

Note: aocl_mmd_op_t is defined as follows:

    typedef void* aocl_mmd_op_t;

len: The size of the data, in bytes, that the function transfers. Declare len with type size_t.
src: The host buffer, of type const void*, from which data is read.
mmd_interface: the handle to the interface being accessed. For example, to access global memory this handle will be value obtained from aocl_mmd_get_info call with AOCL_MMD_MEMORY_INTERFACE as requested_info_id argument.
offset: The size_t byte offset within the interface at which the data transfer begins.

Return Value

If the write operation is successful, the return value is 0.

If the write operation fails, a negative return value indicates an error.

4.1.11 `aocl_mmd_copy`¶

The aocl_mmd_copy function is the copy operation on a single interface.

Syntax


    int aocl_mmd_copy (
        int handle,
        aocl_mmd_op_t op,
        size_t len,
        int mmd_interface, size_t src_offset, size_t dst_offset )

Function Arguments

handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
op: The operation object of type aocl_mmd_op_t used to track the progress of the operation. If op is NULL, the call must block, and return only after the operation completes.

Note: aocl_mmd_op_t is defined as follows:

    typedef void* aocl_mmd_op_t;

len: The size of the data, in bytes, that the function transfers. Declare len with type size_t.
mmd_interface: the handle to the interface being accessed. For example, to access global memory this handle will be value obtained from aocl_mmd_get_info call with AOCL_MMD_MEMORY_INTERFACE as requested_info_id argument.
src_offset: The size_t byte offset within the source interface at which the data transfer begins.
dst_offset: The size_t byte offset within the destination interface at which the data transfer begins

Return Value

If the copy operation is successful, the return value is 0.

If the copy operation fails, a negative return value indicates an error.

4.1.12 `aocl_mmd_hostchannel_create`¶

The aocl_mmd_hostchannel_create function creates a channel interface.

Syntax


    int aocl_mmd_hostchannel_create (
        int handle,
        char *channel_name,
        size_t queue_depth,
        int direction )

Function Arguments

handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
channel_name: Name of the channel to be initialized. The channel name is same as that used in the board_spec.xml file.
queue_depth: The size of pinned internal buffer in bytes. Pointer to the internal buffer is provided when the user calls the aocl_mmd_hostchannel_get_buffer() function.
direction: The direction of the channel.

Return Value

If the function executes successfully, the return value is positive and is handle to the channel.

If the function fails to execute, a negative return value indicates an error.

4.1.13 `aocl_mmd_hostchannel_destroy`¶

The aocl_mmd_hostchannel_destroy function destroys the channel interface.

Syntax


    int aocl_mmd_hostchannel_destroy (
        int handle,
        int channel )

Function Arguments

handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
channel: A positive int value representing handle to the channel to close obtained from the aocl_mmd_hostchannel_create() call.

Return Value

If the function executes successfully, the return value is 0.

If the function fails to execute, a negative return value indicates an error.

4.1.14 `aocl_mmd_hostchannel_get_buffer`¶

The aocl_mmd_hostchannel_get_buffer function provides a host with a pointer to the buffer they can access to write or read from the channel interface, along with the space or data available in the buffer, in bytes.

Syntax


    void *aocl_mmd_hostchannel_get_buffer (
        int handle,
        int channel,
        size_t *buffer_size,
        int *status )

Function Arguments

handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
channel: A positive int value representing handle to the channel to close obtained from the aocl_mmd_hostchannel_create() call.
buffer_size: A pointer to size_t that the function writes available buffer space or size to.
status: A pointer to int that the function writes result of the call to.

Return Value

If the function executes successfully, int pointed to by the status pointer is 0. Returned void* may still be NULL, in which case size_t pointed by the buffer_size is 0.

If the function fails to execute, int pointed by the status pointer is a negative value.

4.1.15 `aocl_mmd_hostchannel_ack_buffer`¶

You can acknowledge write or read from the channel by calling aocl_mmd_hostchannel_ack_buffer.

Syntax


    size_t aocl_mmd_hostchannel_ack_buffer (
        int handle,
        int channel,
        size_t send_size,
        int *status )

Function Arguments

handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
channel: A positive int value representing handle to the channel to close obtained from the aocl_mmd_hostchannel_create() call.
send_size: The size in bytes that the user is acknowledging.
status: A pointer to int that the function writes result of the call to.

Return Value

If the function executes successfully, int pointed to by status pointer is 0. Also, there is no guarantee that the user's send_size is the actual size that gets acknowledged. The returned size_t is the amount of bytes that was actually acknowledged.

If the function fails to execute, int pointed by status pointer is a negative value.

4.1.16 `aocl_mmd_program`¶

The aocl_mmd_program function is the program operation for the specified device. The host must guarantee that no other operations are executing on the device during the program operation.

During aocl_mmd_program execution, the kernels are idle and no read, write, or copy operation can occur.

Disable interrupts and program the FPGA with the data from user_data, which has a size specified by the size argument. The host then calls aocl_mmd_set_status_handler and aocl_mmd_set_interrupt_handler again, which enable the interrupts. If events such as interrupts occur during aocl_mmd_program execution, race conditions or data corruption might occur.

Syntax


 int aocl_mmd_program (
    int handle,
    void * user_data,
    size_t size,
    aocl_mmd_program_mode_t program_mode )

Function Arguments

handle: A positive int value representing the handle to the board obtained from the aocl_mmd_open() call.
user_data: The void* type binary contents of the fpga.bin file that is created during kernel compilation.
size: The size of user_data in bytes. The size argument is of size_t.
program_mode: The bit-field that specifies the mode of device programming.

Table 4-5: Possible Values for the program_mode Argument

`program_mode` Argument Value	Description
AOCL_MMD_PROGRAM_PRESERVE_GLOBAL_MEMORY	This flag specifies that during programming the global memory on the devices are preserved.

Return Value

If aocl_mmd_program executes successfully, the return value is the pointer value that the host uses to access shared memory.

4.1.17 `aocl_mmd_host_alloc`¶

Host allocations provide memory that is allocated on the host. This memory must be deallocated with the aocl_mmd_free function. Host allocations are accessible by the host and one or more devices. The same pointer to a host allocation may be used on the host and all supported devices. They have address equivalence.

Syntax

Once the device has signaled completion through the aocl_mmd_interrupt_handler_fn function, the host can assume it has access to the latest contents of the memory, allocated by the aocl_mmd_host_alloc function call.


    void* aocl_mmd_host_alloc (
        int* handles,
        size_t num_devices,
        size_t size,
        size_t alignment,
        aocl_mmd_mem_properties_t *properties,
        int* error )

Function Arguments

handles: Handles for devices that needs access to this memory.
num_devices: Number of devices in the handles.
size: The size of the memory region.
alignment: The alignment (in bytes) of the allocation.
properties: Specifies additional information about the allocated memory, described by a property type name and its corresponding value. Each property type name is immediately followed by the corresponding desired value. The list is terminated with a zero. For example, [, , , , 0]
error: The error code defined by AOCL_MMD_ERROR*:

AOCL_MMD_ERROR_SUCCESS: No error occurred.
AOCL_MMD_ERROR_INVALID_HANDLE: The device handle provided is invalid.
AOCL_MMD_ERROR_OUT_OF_MEMORY: Ran out of memory.
AOCL_MMD_ERROR_UNSUPPORTED_ALIGNMENT: The device does not support the provided alignment.
AOCL_MMD_ERROR_UNSUPPORTED_PROPERTY: The device does not support the provided property.

Return Value

If the aocl_mmd_host_alloc function executes successfully, the return value is a valid pointer value. Otherwise, the return value is NULL.

4.1.18 `aocl_mmd_free`¶

Releases memory that was allocated by MMD.

Syntax


    int aocl_mmd_free (void* mem)

Function Arguments

mem: The pointer to the memory region. Must be a pointer that is allocated by the MMD.

Return Value

Returns one of the following error code:

AOCL_MMD_ERROR_SUCCESS: No error occurred
AOCL_MMD_ERROR_INVALID_POINTER: Invalid pointer provided

4.1.19 `aocl_mmd_device_alloc`¶

Allocate memory that is owned by the device. This pointer can only be accessed by the kernel. It cannot be accessed by the host. The host is able to manipulate the pointer (for example, increment it) and not just access the underlying data. This memory must be deallocated by the aocl_mmd_free() function.

Syntax


    void * aocl_mmd_device_alloc (
        int handle, 
        size_t size, 
        size_t alignment, 
        aocl_mmd_mem_properties_t *properties, 
        int* error )

Function Arguments

handle: Device that has access to this memory.
size: The size of the memory region.
alignment: The alignment (in bytes) of the memory region.
properties: Specifies additional information about the allocated memory, described by a property type name and its corresponding value. Each property type name is immediately followed by the corresponding desired value. The list is terminated with a zero. For example, [, , , , 0]

Return Value

Returns one of the following error code:

AOCL_MMD_ERROR_SUCCESS: No error occurred
AOCL_MMD_ERROR_INVALID_HANDLE: The device handle provided is invalid.
AOCL_MMD_ERROR_OUT_OF_MEMORY: Ran out of memory.
AOCL_MMD_ERROR_UNSUPPORTED_ALIGNMENT: The device does not support the provided alignment.
AOCL_MMD_ERROR_UNSUPPORTED_PROPERTY: The device does not support the provided property.

4.1.20 `aocl_mmd_shared_alloc`¶

Shared allocations can migrate between the host and one or more associated device. The same pointer to a shared allocation can be used on the host and the supported device. They have address equivalence.

Syntax


    void * aocl_mmd_shared_alloc (
        int handle, 
        size_t size, 
        size_t alignment, 
        aocl_mmd_mem_properties_t* properties, 
        int* error )

Function Arguments

handle: Device that needs access to this memory.
size: The size of the memory region.
alignment: The alignment (in bytes) of the allocation.
properties: Specifies additional information about the allocated memory described by a property type name and its corresponding value. Each property type name is immediately followed by the corresponding desired value. The list is terminated with a zero. For example, [, , , , 0]
error: The error code defined by AOCL_MMD_ERROR*.

AOCL_MMD_ERROR_SUCCESS: No error occurred.
AOCL_MMD_ERROR_INVALID_HANDLE: The device handle provided is invalid.
AOCL_MMD_ERROR_OUT_OF_MEMORY: Ran out of memory.
AOCL_MMD_ERROR_UNSUPPORTED_ALIGNMENT: The device does not support the provided alignment.
AOCL_MMD_ERROR_UNSUPPORTED_PROPERTY: The device does not support the provided property.

Return Value

If the aocl_mmd_shared_alloc function executes successfully, the return value is a valid pointer value. Otherwise, the return value is NULL.

4.1.21 `aocl_mmd_shared_migrate`¶

A call to the aocl_mmd_shared_migrate() function must be made for non-concurrent shared allocations any time the accessor of the allocation changes. For example, the aocl_mmd_shared_migrate() function should be called indicating that the allocation should be migrated to the device before a kernel accessing the allocation is launched on the device. Similarly, the aocl_mmd_shared_migrate() function should be called indicating that the allocation is migrated to the host before the host accesses the memory after kernel completion. For concurrent allocations, this call may be used as a performance hint, but it is not strictly required for functionality.

Syntax


    int aocl_mmd_shared_migrate (
        int handle, 
        void* shared_ptr, 
        size_t size, 
        aocl_mmd_migrate_t destination )

Function Arguments

handle: Device that has access to this memory.
shared_ptr: Pointer allocated by the aocl_mmd_shared_alloc() function.
size: Size (in bytes) of the migration. Must be a multiple of a page boundary that the oneAPI ASP supports.
destination: The destination of the migration.

Return Value

Returns one of the following error code:

AOCL_MMD_ERROR_SUCCESS: No error occurred.
AOCL_MMD_ERROR_INVALID_HANDLE: The device handle provided is invalid.
AOCL_MMD_ERROR_INVALID_MIGRATION_SIZE: The migration size is not supported by the device.

4.2 Board Utilities¶

oneAPI runtime provides a set of options for the aocl utility.

Note: aocl is an utility available in the oneAPI runtime environment, please use aocl help command for more information on this.

Table 4-6 shows the subcommands that aocl utility provides for FPGA platforms.

Table 4-6: aocl Board Utilities

Subcommand	Description	Executable Call
install	Install board into the host system. This installs the FPGA Client Driver (FCD) for your FPGA platform. FCD allows runtime to find and load the FPGA platform libraries at runtime	aocl install `path-to-FPGA-platform-oneapi-asp`
uninstall	Uninstall board from the host system. Removes FCD.	aocl uninstall `path-to-FPGA-platform-oneapi-asp`
initialize	Configure a default FPGA image onto the board. For more information about initialization refer to Intel® oneAPI DPC++/C++ Compiler Handbook for Intel® FPGAs	Two methods are available to initalize the board: 1. aocl initialize `device-name` `board-variant` 2. aocl initialize `device-name` `oneAPI fat binary` > Note: The second option requires oneAPI Base Toolkit (Base Kit) version 2024.0 & above as well as 2023.3 OFS Release for oneAPI Accelerator Support Package and above
program	Configure a new FPGA image onto the board	aocl program `device-name` `aocx file`
diagnose	Runs ICD and FCD diagnostics followed by querying devices in installed platforms. If a `device-name` is specified in the call, it run board vendor's test program for the FPGA platform	* aocl diagnose : This queries the devices in FPGA platform and supplies a list of valid strings assigned to the list of devices * aocl diagnose `device-name` : This runs full diagnostic test on the FPGA platform

The runtime expects the routine for each of this utilities to be defined in the oneAPI ASP. It looks for the routine executables in the location specified by the utilbinder element in the board_env.xml file.

install

oneAPI runtime uses the information in board_env.xml file to create a FCD file. The FCD file name matches name attribute of board_env element and the FCD contents are the platform libraries specified in mmdlib element. Refer to section 2.2 for more information about board_env.xml file. The runtime adds the installed platform to the list of installed packages(file used by runtime to track installed platforms) and then invokes the install routine from oneAPI ASP.

uninstall

oneAPI runtime removes the FCD file and removes the platform from list of installed packages. It then invokes the uninstall routine for oneAPI ASP.

initialize

oneAPI runtime invokes initialize routine provided by oneAPI ASPs for installed platforms.

program

oneAPI runtime loads the programming file on the FPGA by invoking program routine provided by the oneAPI ASPs for the installed platform.

diagnose

oneAPI runtime runs ICD and FCD diagnostics to check the ICD and FCD files installed on the system. It then queries for available boards in the installed platform and lists boards matching every installed platform. If a device-name is specified in the call, runtime invokes the diagnostic routine provided in oneAPI ASP.

For more information about the implementation of these routines in oneAPI ASPs for OFS reference platforms, please refer to section 5.5.

5.0 `oneapi-asp` Implementation Details¶

oneapi-asp in the OFS has four reference platform releases, one is based on Stratix® 10 FPGA and the other three are based on Agilex® 7 FPGA. This chapter aims to explain the architecture and current implementation details of oneapi-asp for these platforms. The oneapi-asp repository is located here.

The next section explains the oneapi-asp directory structure, followed by sections on hardware and MMD layers.

In addition to the information covered in the sections 5.1 to 5.5, there is a new functionality added to the oneapi-asp repository called the oneAPI ASP Editor. The editor enables easy parameterization and generation of the oneAPI ASP using IP Parameter Editor GUI in Quartus Prime.

If you want to customize oneAPI ASP implementation, you can use one of the two approaches:

Follow the design implementation in section 5.1 to 5.5 as a reference to manually make updates to your oneAPI ASP design or
Use the oneAPI ASP Editor to parameterize your ASP design and let the editor handle ASP directory & design files creation, information about the ASP editor is covered in Appendix B.

Using approach (1) allows fine grained control over design, allowing you to tweak all parameters, interfaces, code blocks in the ASP. The ASP editor (approach 2) abrstracts away some of this details, allowing easy parameter and interface settings, hence reducing design time. There are also limits on some parameters in the editor. Please refer to Appendix B for more information about the parameters in the editor.

5.1 `oneapi-asp` Directory Structure¶

As described in section 2.0, oneAPI compiler & runtime use the board_env.xml and board_spec.xml files to get information about the FPGA platform. The compiler expects the board_env.xml file to be at the topmost level in the platform directory. The board_env.xml file describes the location of the hardware files & platform libraries.


    <?xml version="1.0"?>
    <board_env version="Compiler-version" name="ofs_Platform-name/FCD-name">
      <hardware dir="hardware" default="Board-Variant-1"></hardware>
      <platform name="linux64">
        <mmdlib>libopae-c.so,%b/linux64/lib/libMPF.so,%b/linux64/lib/libintel_opae_mmd.so</mmdlib>
        <linkflags>-L%a/linux64/lib -L%b/linux64/lib</linkflags>
        <linklibs>-lintel_opae_mmd -lrt -lMP</linklibs>
        <utilbindir>%b/linux64/libexec</utilbindir>
      </platform>

    </board_env>

Snippet above shows a sample board_env.xml file, the corresponding oneAPI ASP directory structure must match the following format. Table 5-1 provides details on each folder.

In addition to below folders, oneAPI ASP for OFS reference platforms have another folder called commmon (oneapi-asp/common) containing the hardware files common across all reference platforms (oneapi-asp/common/hardware) and source code for the software layer, i.e. MMD & board utilities ( oneapi-asp/common/source). This is because the software layer is compatible with both Stratix® 10 FPGA and Agilex® 7 FPGA PCIe attach reference platform ASPs.

    oneapi-asp/Platform-Name/
    |--hardware/
    |--|--Board-Variant-1/
    |--|--Board-Variant-2/
    |--linux64/
    |--board_env.xml

Note:
In addition to above folders, oneAPI ASPs for OFS reference platforms have additional directories called scripts (in oneapi-asp/common and oneapi-asp/Platform-Name) which contains helper scripts for platform generation and oneapi-asp/common/bringup folder which contains a sample for board bring up. Please refer to the README for each reference platform in the oneASP-asp repository for more information on these additional folders.
* README for oneapi-asp targeting Intel® FPGA PAC D5005 reference platform: README
* README for oneapi-asp targeting Intel® FPGA SmartNIC N6001-PL reference platform: README
* README for oneapi-asp targeting Agilex® 7 FPGA F-Series Development Kit (2x F-Tile):README
* README for oneapi-asp targeting Agilex® 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile): README

The Platform-Name is used for identifying the platform and can be alphanumeric value decided by the platform developer. For example, Platform-Name d5005 is used for oneapi-asp for Stratix® 10 FPGA as the reference platform is Intel® FPGA PAC D5005.

Table 5-1: Directory Contents

Files/Folder (path relative to `oneapi-asp/Platform-Name/`)	Description
`hardware`	Contains hardware files (RTL, platform designer files, SDCs, compilation scripts, floorplan settings) and the `board_spec.xml` files for each board variants. See table 5-2 for more details
`../common/source`	Source code for MMD layer as well as `oneapi-asp` board utilities
`linux64`	Location for FPGA platform libraries and executables for `oneapi-asp` board utilities > Note: The `linux64` folder does not exist in the `Platform-Name` directory, it is added after the `oneapi-asp` build flow is complete
`board_env.xml`	Contains platform installation information. Please refer to section 2.2 for more details on board_env.xml elements

Tables 5-2 to 5-4 give more details on each of these folders for oneAPI ASPs for OFS reference platforms (before oneapi-asp build flow unless noted otherwise, for more information about oneapi-asp build flow refer to section 5.2).

5.1.1 hardware Folder¶

oneapi-asp/
|--common
|--|--hardware/
|--Platform-Name
|--|--hardware/
|--|--|--common/
|--|--|--|--build
|--|--|--board-variant-1/
|--|--|--|--build
|--|--|--|--board_spec.xml
|--|--|--|--part-number_dm.xml (Please see note for this file in table 5-2)

Table 5-2: hardware Folder Contents

Files/Folder (path relative to `oneapi-asp/`)	Description
`common/hardware`	Contains hardware design files common to all reference platforms including Quartus® software Settings Files (.qsf), IP files (.v, _hw.tcl), common RTL code (.v, *.sv) as well as scripts to control compile flow
`Platform-Name/hardware/common/build`	Contains platform specific hardware design files (common to each board variant for a platform) including .sv, .sdc, Quartus® software ini settings file as well as scripts to control compile flow
`Platform-Name/hardware/board-variant-1/build`	Contains board variant specific hardware design files like variant specific Verilog header file (.vh), oneapi_afu.json, .tcl
`Platform-Name/hardware/board-variant-1/board_spec.xml`	Defines compile flow, global memory, kernel interfaces. Please see section 2.1 for more information about `board_spec.xml` file
`Platform-Name/hardware/board-variant-1/part-number_dm.xml`	Device Model file for Altera® FPGA part on the target platform. The name must be of the format `part-number`_dm.xml. This file has the total resources available on the device. > Note: The device model files provided as part of the oneAPI Base Toolkit (Base Kit) installation are located in `$INTELFPGAOCLSDKROOT/share/models/dm`, where INTELFPGAOCLSDKROOT is set by the `setvars.sh` environment setup script provided by oneAPI toolkit. If the device model file for your part number is not included in `$INTELFPGAOCLSDKROOT/share/models/dm`, it must be created and placed in the same folder as `board_spec.xml`. In oneapi-asp for reference platforms, this file only exists for platforms whose target device model files are not provided by oneAPI Base Toolkit (Base Kit) installation.

5.1.2 common/source Folder¶

oneapi-asp/common/source/
|--cmake/
|--|--modules
|--extra/
|--|--intel-fpga-bbb
|--host
|--include
|--util
|--CMakeLists.txt

Table 5-3: oneapi-asp/common/source Folder Contents

Files/Folder	Description
`cmake/modules`	Contains `Find***.cmake` files to find packages required for building MMD library
`extra/intel-fpga-bbb`	oneAPI ASP for OFS reference platforms uses a library provided as part of Intel® FPGA Basic Building Blocks (BBB) repository. The `oneapi-asp` build scripts clone this repository in `extra` directory by default
`host`	Source code for MMD API
`include`	Contains MMD header files
`util`	Contains source code for `oneapi-asp` reference platform routines for diagnose and program utilities
`CMakeLists.txt`	Top-level CMakeLists.txt file for building MMD and other libraries required by `oneapi-asp` as well as the executables for diagnose and program utilities

5.1.3 linux64 Folder¶

oneapi-asp/Platform-Name/linux64/
|--include (see note below)
|--lib
|--libexec

Note: The linux64 folder does not exist in the Platform-Name directory, it is present in the oneapi-asp/common folder as the utilities are common for all reference platforms. The include and lib folders also do not exist in the oneapi-asp repository. These along with the utilities are added to the oneapi-asp/Platform-Name folder after the oneapi-asp build flow is complete.

Table 5-4: linux64 Folder Contents

Files/Folder	Description
`include`	Contains header files from Intel® FPGA BBB required by MMD (see section 5.4 for more information on use of MPF from Intel® FPGA BBB in MMD)
`lib`	Contains MMD and Intel® FPGA BBB libraries
`libexec`	Contains executables/scripts for `oneapi-asp` board utilities

5.2 `oneapi-asp` Build Flow¶

Figure 1-3 shows a high level overview of the hardware design and Figure 1-4 shows oneAPI ASP as part of the Open FPGA Stack. As shown in these images, all oneapi-asp hardware components reside in the AFU region in the PR slot.

Note: The architecture of the FIM with PR slot is explained in the FIM technical reference manuals, please refer to section 1.3 for links to the manuals.

The oneapi-asp repository contains source files for components that reside in the AFU region for each reference platform. oneapi-asp expects a compiled FIM netlist and a corresponding PR tree. The FIM database is copied to the oneapi-asp during ASP build flow (oneapi-asp/Platform-Name/scripts/build-asp.sh) and ASP compile scripts import the FIM database during oneAPI kernel compilation.

Notes:
1. FIM developer guide outlines steps to compile a FIM and generate PR tree, please refer to section 1.3 for links to FIM developer guides
2. The steps to build oneapi-asp using PR tree and build-asp.sh script are covered in the oneAPI Accelerator Support Package (ASP): Getting Started User Guide

The following figure shows the oneapi-asp build process.

Figure 5-1: oneapi-asp Build Flow

Table 5-5 Environment Variables used in Build Flow

Variable Number	Environment Variable	Description
1	OFS_PLATFORM_AFU_BBB	Should point to location where ofs-platform-afu-bbb repository is cloned, if this variable is not set, `build-asp.sh` script clones the repository
2	OPAE_PLATFORM_ROOT	Must point to the PR tree generated during FIM build, this is a required variable and build flow fails without this
3	LIBOPAE_C_ROOT	Should point to the installation location for OPAE libraries (please see oneAPI Accelerator Support Package (ASP): Getting Started User Guide for more information on this variable setting), `build-opae.sh` script is used to clone & build OPAE library if this variable is not set.
4	OPAE_SDK_REPO_BRANCH	If `LIBOPAE_C_ROOT` is not set, it is recommended to set this variable to indicate the OPAE SDK branch to be used for building OPAE libraries. > Note: `build-opae.sh` always clones the master branch if `OPAE_SDK_REPO_BRANCH` environment variable is not set. If you do not have the OPAE SDK installed, please ensure the `OPAE_SDK_REPO_BRANCH` is set to point to the OPAE release tag matching the release notes for the OFS release you are using. Please refer to oneAPI Accelerator Support Package (ASP): Getting Started User Guide for more information on environment variable and build steps

The scripts required for oneapi-asp build that are common to all OFS reference platforms are located in oneapi-asp/common/scripts and build scripts specific to individual platforms are located in oneapi-asp/Platform-Name/scripts folders in the oneapi-asp repository, where Platform-Name is:

n6001 for Intel® FPGA SmartNIC N6001-PL reference platform
d5005 for Intel® FPGA PAC D5005 reference platform
fseries-dk for Agilex® 7 FPGA F-Series Development Kit (2x F-Tile)
iseries-dkfor Agilex® 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile)

The build flow copies the scripts from oneapi-asp/common/scripts into the oneapi-asp/Platform-Name/scripts directory and generates the complete hardware directories for all board variants in oneapi-asp.

Note: For more information about the build scripts, please refer to README in oneapi-asp/common/scripts directory.

5.3 `oneapi-asp` Hardware Implementation¶

This section goes deeper into the current hardware architecture of the oneapi-asp.

Figure 1-3 shows a high level overview of the hardware design. OFS reference platforms have different board variants enabling the different paths shown in Figure 1-3. Table below summarizes the board variants and paths enabled in the oneAPI ASP reference design for each. The Path numbers in the table match the ones in Figure 1-3. Figure 5-2 to 5-5 show the detailed diagram for oneapi-asp components in each of these board variants.

Note: The design files have the option to enable multiple global memories (oneapi-asp tag ofs-2024.2-1 supports upto 4 global memory systems), however, for the purpose of readability, the figures 5-2 to 5-5 show the design with one global memory. For more information about multiple global memories, refer to section 5.3.3.

Table 5-6: OFS Reference Platform Board Variants

#	Device	Board Variant	Host to EMIF with DMA Engine (Path 1)	Host to Kernel Interface (Path 2)	Kernel to EMIF (Path 3)	Kernel to Unified Shared Memory (Path 4)	Kernel to HSSI (Path 5)	Figure #
1	Agilex® 7 FPGA Stratix® 10 FPGA	`oneapi-asp` for Agilex® 7 FPGA : * ofs_n6001 * ofs_fseries-dk * ofs_iseries-dk `oneapi-asp` for Stratix® 10 FPGA : * ofs_d5005	Yes	Yes	Yes	No	No	5-2
2	Agilex® 7 FPGA Stratix® 10 FPGA	`oneapi-asp` for Agilex® 7 FPGA : * ofs_n6001_usm * ofs_fseries-dk_usm * ofs_iseries-dk_usm `oneapi-asp` for Stratix® 10 FPGA : * ofs_d5005_usm	Yes	Yes	Yes	Yes	No	5-3
3	Agilex® 7 FPGA	`oneapi-asp` for Agilex® 7 FPGA : * ofs_n6001_iopies	Yes	Yes	Yes	No	Yes	5-4
4	Agilex® 7 FPGA	`oneapi-asp` for Agilex® 7 FPGA : * ofs_n6001_usm_iopipes	Yes	Yes	Yes	Yes	Yes	5-5

Note: Please see oneapi-asp/Platform-Name/hardware folder for the board variants for each OFS reference platform.

Figure 5-2: oneapi-asp Reference Platform Hardware Design - Board Variant #1

Figure 5-3: oneapi-asp Reference Platform with USM Hardware Design - Board Variant #2

Figure 5-4: oneapi-asp Reference Platform with IO Pipes Hardware Design - Board Variant #3

Figure 5-5: oneapi-asp Reference Platform with IO Pipes and USM Hardware Design - Board Variant #4

The ofs_plat_afu.sv is the top level entity for the oneapi-asp.

Hardware design files that are common for all OFS reference platforms (for the components inside ofs_plat_afu module) are located in oneapi-asp/common/hardware/common/build and hardware design file specific to the individual platforms are located in the oneapi-asp/Platform-Name/hardware/Board-Variant/build directory. The common files are copied to the oneapi-asp/Platform-Name/hardware/Board-Variant folder during oneapi-asp build flow. The FIM database files imported during oneapi-asp build (section 5.2) are located in oneapi-asp/Platform-Name/hardware/Board-Variant/fim_platform directory.

Tables 5-7 give a brief description of the important design files in all board variants, the location for these files shown in table 5-7 is post oneapi-asp build flow.

Table 5-7: Hardware Design Files in oneapi-asp/Platform-Name/hardware/Board-Variant/build

Files/Folder	Description
afu_ip.qsf	Adds platform specific settings & design files, this file is sourced in the oneapi-asp revision being compiled (e.g.`afu_flat.qsf`)
mpf_vtp.qsf	Adds IP components from Intel® FPGA BBB (see section 5.3.2 below for information about use of MPF blocks from Intel® FPGA BBB) repository used in the design
asp_design_files.tcl	Adds design files to the project, this file is source in `afu_ip.qsf`
oneapi_afu.json	Accelerator Description File that describes the metadata associated with a oneAPI ASP. The Open Programmable Accelerator Engine (OPAE) uses this metadata during reconfiguration > Note: For more information about use of a JSON file for storing meadata for an AFU, refer to Accelerator Description File section in PIM Based AFU Developer Guide
ip	Contains the HW TCL and Verilog source code files for IP components used in the hardware design
scripts	Contains scripts to control oneAPI kernel and asp Quartus® software compile flow (see section 5.3.5 for more information on compile flow)
rtl	Contains the SystemVerilog and Verilog design files shown in figures 5-2 to 5-5 above
rtl/ofs_plat_afu.sv	Top level wrapper file for `oneapi-asp`. Contains the Platform Interface Manager (PIM) blocks for protocol translation & connecting FIM ports to `oneapi-asp` ports. For more information about PIM, see the PIM subtopic below.
rtl/afu.sv	Contains the PIM blocks for VTP address translation and instantiates the `kernel_wrapper` entity. For more information about PIM, see the PIM subtopic below.
rtl/kernel_wrapper.v	Contains the `kernel_system` instantiation
rtl/asp_logic.sv	Contains the DMA engine, `board` system instantiation, please see information about `board_hw.tcl`, `ddr_board_hw.tcl` and `ddr_channel_hw.tcl` below
rtl/ofs_asp.vh	Contains the macros to control instantiation of logic on host to memory path, logic required to enable Unified Shared Memory(USM) support and IO pipes support
rtl/ofs_asp_pkg.sv	Contains Verilog parameters used in controlling values of number of channels (memory/IO pipes), address widths, pipeline stages etc. for logic in the `oneapi-asp`
board_hw.tcl	Contains the `Kernel Interface` IP instantiation, register to store board AFU ID and `ddr_board` system instantiation. Please refer to `ddr_board_hw.tcl` and `ddr_channel_hw.tcl` below. > Note: The board AFU ID is used by the MMD layer during board discovery
ddr_board_hw.tcl	Instantiated in the board_hw.tcl, contains IP components in the host to EMIF and kernel to EMIF datapath. `Memory Bank Divider` is instantiated in this file. Please refer to section 3.1.1 for more details on `Memory Bank Divider`
ddr_channel_hw.tcl	Instantiated in ddr_board_hw.tcl, contains bridges required for clock domain crossings between PCIe and EMIF clocks as well as kernel and EMIF clock
ofs_asp.sdc	Contains clock group constraints for all clocks in oneAPI ASP
../fim_platform	Contains Platform Interface Manager(PIM) and FIM database used in the design. See PIM subtopic below

The hardware implementation diagrams show a PF/VF Mux/De-mux module in the AFU region. The PF/VF mux routes packets to AFU component based on pf_num and vf_num information in PCIe TLP header. For more information about the PF/VF mux, please refer to the PF/VF Mapping details in FPGA Interface Manager Technical Reference Manual for your target device (links are available in section 1.3).

The oneapi-asp resides inside the AFU region and, depending on the FIM configuration, connects to the lowest PF or lowest VF number that routes into the PR slot. Table below summazires the PF/VF mapping in oneapi-asp for some of the different FIM configurations.

Table 5-8: PF/VF Mapping in oneapi-asp

Target Device for OFS	FIM Configuration*	PF/VF Mapping in `oneapi-asp`
Stratix® 10 FPGA	Default	PF0-VF1
Agilex® 7 FPGA	Default	PF0-VF0
Agilex® 7 FPGA	Compiled using `<n6001/iseries-dk>_1pf_1vf.ofss`, i.e. PCIe Subsystem configuration is set to PF0 with 1 VF	PF0-VF0
Agilex® 7 FPGA	Compiled using `n6001_2pf.ofss`, i.e. PCIe Subsystem configuration is set to two physical functions PF0 and PF1	PF1

*Note: For more information on different FIM configurations & how to compile these, please refer to FPGA Interface Manager(FIM) Developer Guides for Open FPGA Stack for your target device (links are available in section 1.3).

Sections below provide some more information on some blocks in the hardware design block diagram shown above. Refer to section 3.1 for more information about Memory Bank Divider and to section 3.2 for information about Kernel Interface.

5.3.1 Platform Interface Manager(PIM)¶

In addition to above files/folders, an important part of the design are the ofs_plat_* modules provided by Platform Interface Manager (PIM). oneAPI kernel system and oneapi-asp have Avalon® interfaces. FIM components have AXI interfaces. The oneAPI compiler generated kernel_system with Avalon interfaces. The PIM modules are used for protocol translation.

In addition to this, PIM blocks are also used for Virtual to Physical (VTP) address translation. The Memory Mapped Device(MMD) layer provides virtual addresses which are converted to physical addresses by the VTP blocks to allow Unified Shared Memory(USM) and DMA transfers to/from host memory.

Note: For more information about PIM, please refer to PIM README here.

5.3.2 Direct Memory Access(DMA) Module¶

The Direct Memory Access(DMA) module is located in the host to EMIF datapath in the oneapi-asp and provides a controller to execute transfers from host to DDR on the board and vice versa. The source files for the DMA module used in oneapi-asp for OFS reference platforms are located in oneapi-asp/Platform-Name/hardware/Board-Variant/build/rtl/dma directory (location post oneapi-asp build flow). Figure below shows the DMA module interface.

Figure 5-7: DMA Controller Block Diagram

Figure 5-2 shows the instantiation of this dma_top module as dma_controller_inst. Table 5-9 shows the signal names and descriptions. Section 5-5 covers the complete compilation flow.

Table 5-9: DMA Module Signal Descriptions

Signal or Port	Description
mmio64_if	Control signal from host, connects to DMA Control Status Registers(CSR)
host_mem_rd_avmm_if	Avalon® memory-mapped interface to read from host memory
host_mem_wr_avmm_if	Avalon® memory-mapped interface to write to host memory
local_mem_rd_avmm_if	Avalon® memory-mapped interface to read from on-board DDR4 memory
local_mem_wr_avmm_if	Avalon® memory-mapped interface to write to on-board DDR4 memory
dma_irq_host2fpga	Interrupt to indicate completion of host to FPGA DMA transaction (read from host, write to DDR4)
dma_irq_fpga2host	Interrupt to indicate completion of FPGA DDR4 to host DMA transaction (read from DDR4, write to host) > Note: The `oneapi-asp` for OFS reference platforms uses a host memory write to indicate completion of FPGA to host transaction

The data transfer module manages data movement from source to destination addresses.

To start a data transfer, the data transfer module requires following information, this is set by the dma_dispatcher:

Source address
Destination address
Number of bytes to transfer

oneapi-asp for OFS reference platforms uses virtual addresses in the DMA controller for data transfer. The Memory Properties Factory(MPF) Virtual to Physical(VTP) blocks (i.e.mpf_vtp_* modules) shown in figure 5-2 translate virtual addresses to appropriate host addresses.

Note: Memory Properties Factory(MPF) is a part of Intel® FPGA Basic Building Blocks. Please refer to Intel® FPGA BBB repository for more information.

5.3.3 `board_hw.tcl` File¶

As described in table 5-7, the board_hw.tcl contains the Kernel Interface IP instantiation, register to store board AFU ID and ddr_board system instantiation. The board_hw.tcl component has a few parameters that can be used to control the internal composition. Snippet below shows the portion of the TCL script adding these parameters. These are described in table below.

board_hw.tcl sources another file called parameters.tcl, this file is located in oneapi-asp/Platform-Name/hardware/Board-Variant/build directory and contains different values for the parameters based on the board variant. All the parameters in the board_hw.tcl derive their value from respective values in parameters.tcl(all variables starting with p_<parameter-name>) for each board variant when the oneapi-asp is built.


 +-----------------------------------
 | parameters
 |
source parameters.tcl

add_parameter AFU_ID_H STD_LOGIC_VECTOR $p_AFU_ID_H
set_parameter_property AFU_ID_H DEFAULT_VALUE $p_AFU_ID_H
set_parameter_property AFU_ID_H DISPLAY_NAME "AFU ID H"
set_parameter_property AFU_ID_H AFFECTS_ELABORATION true

add_parameter AFU_ID_L STD_LOGIC_VECTOR $p_AFU_ID_L
set_parameter_property AFU_ID_L DEFAULT_VALUE $p_AFU_ID_L
set_parameter_property AFU_ID_L DISPLAY_NAME "AFU ID L"
set_parameter_property AFU_ID_L AFFECTS_ELABORATION true

add_parameter IOPIPE_SUPPORT BOOLEAN $p_IOPIPE_SUPPORT
set_parameter_property IOPIPE_SUPPORT DEFAULT_VALUE $p_IOPIPE_SUPPORT
set_parameter_property IOPIPE_SUPPORT DISPLAY_NAME "IO Pipe Support"
set_parameter_property IOPIPE_SUPPORT AFFECTS_ELABORATION true

add_parameter NUMBER_OF_DMA_CHANNELS INTEGER $p_NUMBER_OF_DMA_CHANNELS
set_parameter_property NUMBER_OF_DMA_CHANNELS DEFAULT_VALUE $p_NUMBER_OF_DMA_CHANNELS
set_parameter_property NUMBER_OF_DMA_CHANNELS DISPLAY_NAME "Number of DMA Channels"
set_parameter_property NUMBER_OF_DMA_CHANNELS AFFECTS_ELABORATION true

add_parameter SNOOP_PORT_ENABLE BOOLEAN $p_SNOOP_PORT_ENABLE
set_parameter_property SNOOP_PORT_ENABLE DEFAULT_VALUE $p_SNOOP_PORT_ENABLE
set_parameter_property SNOOP_PORT_ENABLE DISPLAY_NAME "Enable Snoop Port"
set_parameter_property SNOOP_PORT_ENABLE AFFECTS_ELABORATION true

add_parameter NUMBER_OF_GLOBAL_MEMORY_SYSTEMS INTEGER $p_NUMBER_OF_GLOBAL_MEMORY_SYSTEMS
set_parameter_property NUMBER_OF_GLOBAL_MEMORY_SYSTEMS DEFAULT_VALUE $p_NUMBER_OF_GLOBAL_MEMORY_SYSTEMS
set_parameter_property NUMBER_OF_GLOBAL_MEMORY_SYSTEMS DISPLAY_NAME "Number of Global Memory Systems"
set_parameter_property NUMBER_OF_GLOBAL_MEMORY_SYSTEMS AFFECTS_ELABORATION true

if { $p_NUMBER_OF_GLOBAL_MEMORY_SYSTEMS > 0 } {
  add_parameter MEM_0_NUMBER_OF_MEMORY_BANKS INTEGER $p_MEM_0_NUMBER_OF_MEMORY_BANKS
  set_parameter_property MEM_0_NUMBER_OF_MEMORY_BANKS DEFAULT_VALUE $p_MEM_0_NUMBER_OF_MEMORY_BANKS
  set_parameter_property MEM_0_NUMBER_OF_MEMORY_BANKS DISPLAY_NAME "Number of Global Memory 0 Banks"
  set_parameter_property MEM_0_NUMBER_OF_MEMORY_BANKS AFFECTS_ELABORATION true

  add_parameter MEM_0_MEMORY_BANK_ADDRESS_WIDTH INTEGER $p_MEM_0_MEMORY_BANK_ADDRESS_WIDTH
  set_parameter_property MEM_0_MEMORY_BANK_ADDRESS_WIDTH DEFAULT_VALUE $p_MEM_0_MEMORY_BANK_ADDRESS_WIDTH
  set_parameter_property MEM_0_MEMORY_BANK_ADDRESS_WIDTH DISPLAY_NAME "Global Memory 0 Memory Bank Address Width"
  set_parameter_property MEM_0_MEMORY_BANK_ADDRESS_WIDTH AFFECTS_ELABORATION true

  add_parameter MEM_0_DATA_WIDTH INTEGER $p_MEM_0_DATA_WIDTH
  set_parameter_property MEM_0_DATA_WIDTH DEFAULT_VALUE $p_MEM_0_DATA_WIDTH
  set_parameter_property MEM_0_DATA_WIDTH DISPLAY_NAME "Global Memory 0 Data Width"
  set_parameter_property MEM_0_DATA_WIDTH AFFECTS_ELABORATION true

  add_parameter MEM_0_MAX_BURST_SIZE INTEGER $p_MEM_0_MAX_BURST_SIZE
  set_parameter_property MEM_0_MAX_BURST_SIZE DEFAULT_VALUE $p_MEM_0_MAX_BURST_SIZE
  set_parameter_property MEM_0_MAX_BURST_SIZE DISPLAY_NAME "Global Memory 0 Maximum Burst Size"
  set_parameter_property MEM_0_MAX_BURST_SIZE AFFECTS_ELABORATION true

  add_parameter MEM_0_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE INTEGER $p_MEM_0_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE
  set_parameter_property MEM_0_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE DEFAULT_VALUE $p_MEM_0_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE
  set_parameter_property MEM_0_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE DISPLAY_NAME "Global Memory 0 Kernel to global memory waitrequest allowance"
  set_parameter_property MEM_0_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE AFFECTS_ELABORATION true

  add_parameter MEM_0_MBD_TO_MEMORY_PIPE_STAGES INTEGER $p_MEM_0_MBD_TO_MEMORY_PIPE_STAGES
  set_parameter_property MEM_0_MBD_TO_MEMORY_PIPE_STAGES DEFAULT_VALUE $p_MEM_0_MBD_TO_MEMORY_PIPE_STAGES
  set_parameter_property MEM_0_MBD_TO_MEMORY_PIPE_STAGES DISPLAY_NAME "Global Memory 0 MBD to Memory Pipeline Stages"
  set_parameter_property MEM_0_MBD_TO_MEMORY_PIPE_STAGES AFFECTS_ELABORATION true
}

if { $p_NUMBER_OF_GLOBAL_MEMORY_SYSTEMS > 1 } {
  add_parameter MEM_1_NUMBER_OF_MEMORY_BANKS INTEGER $p_MEM_1_NUMBER_OF_MEMORY_BANKS
  set_parameter_property MEM_1_NUMBER_OF_MEMORY_BANKS DEFAULT_VALUE $p_MEM_1_NUMBER_OF_MEMORY_BANKS
  set_parameter_property MEM_1_NUMBER_OF_MEMORY_BANKS DISPLAY_NAME "Number of Global Memory 1 Banks"
  set_parameter_property MEM_1_NUMBER_OF_MEMORY_BANKS AFFECTS_ELABORATION true

  add_parameter MEM_1_MEMORY_BANK_ADDRESS_WIDTH INTEGER $p_MEM_1_MEMORY_BANK_ADDRESS_WIDTH
  set_parameter_property MEM_1_MEMORY_BANK_ADDRESS_WIDTH DEFAULT_VALUE $p_MEM_1_MEMORY_BANK_ADDRESS_WIDTH
  set_parameter_property MEM_1_MEMORY_BANK_ADDRESS_WIDTH DISPLAY_NAME "Global Memory 1 Memory Bank Address Width"
  set_parameter_property MEM_1_MEMORY_BANK_ADDRESS_WIDTH AFFECTS_ELABORATION true

  add_parameter MEM_1_DATA_WIDTH INTEGER $p_MEM_1_DATA_WIDTH
  set_parameter_property MEM_1_DATA_WIDTH DEFAULT_VALUE $p_MEM_1_DATA_WIDTH
  set_parameter_property MEM_1_DATA_WIDTH DISPLAY_NAME "Global Memory 1 Data Width"
  set_parameter_property MEM_1_DATA_WIDTH AFFECTS_ELABORATION true

  add_parameter MEM_1_MAX_BURST_SIZE INTEGER $p_MEM_1_MAX_BURST_SIZE
  set_parameter_property MEM_1_MAX_BURST_SIZE DEFAULT_VALUE $p_MEM_1_MAX_BURST_SIZE
  set_parameter_property MEM_1_MAX_BURST_SIZE DISPLAY_NAME "Global Memory 1 Maximum Burst Size"
  set_parameter_property MEM_1_MAX_BURST_SIZE AFFECTS_ELABORATION true

  add_parameter MEM_1_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE INTEGER $p_MEM_1_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE
  set_parameter_property MEM_1_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE DEFAULT_VALUE $p_MEM_1_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE
  set_parameter_property MEM_1_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE DISPLAY_NAME "Global Memory 1 Kernel to global memory waitrequest allowance"
  set_parameter_property MEM_1_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE AFFECTS_ELABORATION true

  add_parameter MEM_1_MBD_TO_MEMORY_PIPE_STAGES INTEGER $p_MEM_1_MBD_TO_MEMORY_PIPE_STAGES
  set_parameter_property MEM_1_MBD_TO_MEMORY_PIPE_STAGES DEFAULT_VALUE $p_MEM_1_MBD_TO_MEMORY_PIPE_STAGES
  set_parameter_property MEM_1_MBD_TO_MEMORY_PIPE_STAGES DISPLAY_NAME "Global Memory 1 MBD to Memory Pipeline Stages"
  set_parameter_property MEM_1_MBD_TO_MEMORY_PIPE_STAGES AFFECTS_ELABORATION true
}

if { $p_NUMBER_OF_GLOBAL_MEMORY_SYSTEMS > 2 } {
  add_parameter MEM_2_NUMBER_OF_MEMORY_BANKS INTEGER $p_MEM_2_NUMBER_OF_MEMORY_BANKS
  set_parameter_property MEM_2_NUMBER_OF_MEMORY_BANKS DEFAULT_VALUE $p_MEM_2_NUMBER_OF_MEMORY_BANKS
  set_parameter_property MEM_2_NUMBER_OF_MEMORY_BANKS DISPLAY_NAME "Number of Global Memory 2 Banks"
  set_parameter_property MEM_2_NUMBER_OF_MEMORY_BANKS AFFECTS_ELABORATION true

  add_parameter MEM_2_MEMORY_BANK_ADDRESS_WIDTH INTEGER $p_MEM_2_MEMORY_BANK_ADDRESS_WIDTH
  set_parameter_property MEM_2_MEMORY_BANK_ADDRESS_WIDTH DEFAULT_VALUE $p_MEM_2_MEMORY_BANK_ADDRESS_WIDTH
  set_parameter_property MEM_2_MEMORY_BANK_ADDRESS_WIDTH DISPLAY_NAME "Global Memory 2 Memory Bank Address Width"
  set_parameter_property MEM_2_MEMORY_BANK_ADDRESS_WIDTH AFFECTS_ELABORATION true

  add_parameter MEM_2_DATA_WIDTH INTEGER $p_MEM_2_DATA_WIDTH
  set_parameter_property MEM_2_DATA_WIDTH DEFAULT_VALUE $p_MEM_2_DATA_WIDTH
  set_parameter_property MEM_2_DATA_WIDTH DISPLAY_NAME "Global Memory 2 Data Width"
  set_parameter_property MEM_2_DATA_WIDTH AFFECTS_ELABORATION true

  add_parameter MEM_2_MAX_BURST_SIZE INTEGER $p_MEM_2_MAX_BURST_SIZE
  set_parameter_property MEM_2_MAX_BURST_SIZE DEFAULT_VALUE $p_MEM_2_MAX_BURST_SIZE
  set_parameter_property MEM_2_MAX_BURST_SIZE DISPLAY_NAME "Global Memory 2 Maximum Burst Size"
  set_parameter_property MEM_2_MAX_BURST_SIZE AFFECTS_ELABORATION true

  add_parameter MEM_2_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE INTEGER $p_MEM_2_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE
  set_parameter_property MEM_2_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE DEFAULT_VALUE $p_MEM_2_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE
  set_parameter_property MEM_2_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE DISPLAY_NAME "Global Memory 2 Kernel to global memory waitrequest allowance"
  set_parameter_property MEM_2_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE AFFECTS_ELABORATION true

  add_parameter MEM_2_MBD_TO_MEMORY_PIPE_STAGES INTEGER $p_MEM_2_MBD_TO_MEMORY_PIPE_STAGES
  set_parameter_property MEM_2_MBD_TO_MEMORY_PIPE_STAGES DEFAULT_VALUE $p_MEM_2_MBD_TO_MEMORY_PIPE_STAGES
  set_parameter_property MEM_2_MBD_TO_MEMORY_PIPE_STAGES DISPLAY_NAME "Global Memory 2 MBD to Memory Pipeline Stages"
  set_parameter_property MEM_2_MBD_TO_MEMORY_PIPE_STAGES AFFECTS_ELABORATION true
}

if { $p_NUMBER_OF_GLOBAL_MEMORY_SYSTEMS > 3 } {
  add_parameter MEM_3_NUMBER_OF_MEMORY_BANKS INTEGER $p_MEM_3_NUMBER_OF_MEMORY_BANKS
  set_parameter_property MEM_3_NUMBER_OF_MEMORY_BANKS DEFAULT_VALUE $p_MEM_3_NUMBER_OF_MEMORY_BANKS
  set_parameter_property MEM_3_NUMBER_OF_MEMORY_BANKS DISPLAY_NAME "Number of Global Memory 3 Banks"
  set_parameter_property MEM_3_NUMBER_OF_MEMORY_BANKS AFFECTS_ELABORATION true

  add_parameter MEM_3_MEMORY_BANK_ADDRESS_WIDTH INTEGER $p_MEM_3_MEMORY_BANK_ADDRESS_WIDTH
  set_parameter_property MEM_3_MEMORY_BANK_ADDRESS_WIDTH DEFAULT_VALUE $p_MEM_3_MEMORY_BANK_ADDRESS_WIDTH
  set_parameter_property MEM_3_MEMORY_BANK_ADDRESS_WIDTH DISPLAY_NAME "Global Memory 3 Memory Bank Address Width"
  set_parameter_property MEM_3_MEMORY_BANK_ADDRESS_WIDTH AFFECTS_ELABORATION true

  add_parameter MEM_3_DATA_WIDTH INTEGER $p_MEM_3_DATA_WIDTH
  set_parameter_property MEM_3_DATA_WIDTH DEFAULT_VALUE $p_MEM_3_DATA_WIDTH
  set_parameter_property MEM_3_DATA_WIDTH DISPLAY_NAME "Global Memory 3 Data Width"
  set_parameter_property MEM_3_DATA_WIDTH AFFECTS_ELABORATION true

  add_parameter MEM_3_MAX_BURST_SIZE INTEGER $p_MEM_3_MAX_BURST_SIZE
  set_parameter_property MEM_3_MAX_BURST_SIZE DEFAULT_VALUE $p_MEM_3_MAX_BURST_SIZE
  set_parameter_property MEM_3_MAX_BURST_SIZE DISPLAY_NAME "Global Memory 3 Maximum Burst Size"
  set_parameter_property MEM_3_MAX_BURST_SIZE AFFECTS_ELABORATION true

  add_parameter MEM_3_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE INTEGER $p_MEM_3_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE
  set_parameter_property MEM_3_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE DEFAULT_VALUE $p_MEM_3_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE
  set_parameter_property MEM_3_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE DISPLAY_NAME "Global Memory 3 Kernel to global memory waitrequest allowance"
  set_parameter_property MEM_3_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE AFFECTS_ELABORATION true

  add_parameter MEM_3_MBD_TO_MEMORY_PIPE_STAGES INTEGER $p_MEM_3_MBD_TO_MEMORY_PIPE_STAGES
  set_parameter_property MEM_3_MBD_TO_MEMORY_PIPE_STAGES DEFAULT_VALUE $p_MEM_3_MBD_TO_MEMORY_PIPE_STAGES
  set_parameter_property MEM_3_MBD_TO_MEMORY_PIPE_STAGES DISPLAY_NAME "Global Memory 3 MBD to Memory Pipeline Stages"
  set_parameter_property MEM_3_MBD_TO_MEMORY_PIPE_STAGES AFFECTS_ELABORATION true
}

Table 5-10 board_hw.tcl Parameters

Parameter Name in IP TCL	Parameter	Description
AFU_ID_H	AFU ID H	Upper 32 bits of the board AFU ID. > Note: The board AFU ID is used by the MMD layer during board discovery
AFU_ID_L	AFU ID L	Lower 32 bits of the board AFU ID.
IOPIPE_SUPPORT	IO Pipe Support	Boolean value used to determine if the `oneapi-asp` has IO pipes support. This must be set to true for `oneapi-asp` board variant that has the `channels` element defined in the `board_spec.xml`. Table 5-6 shows the OFS reference platform board variants that have IO pipes enabled. To ensure correct functionality, the HSSI susbsystem in the FIM must support the settings in the `oneapi-asp`. For more information about `channels` element refer to section 2.1.8
NUMBER_OF_DMA_CHANNELS	Number of DMA Channels	Reserved for future use > Note: oneapi-asp tag ofs-2024.2-1 supports only 1 DMA channel
SNOOP_PORT_ENABLE	Enable Snoop Port	Used to enable the `acl_asp_snoop` port for `Memory Bank Divider`, for more information about `acl_asp_snoop` port, refer to Table 3-2
NUMBER_OF_GLOBAL_MEMORY_SYSTEMS* Please see notes below the table for additional design details	Number of Global Memory Systems	Number of global memories connected to the oneAPI kernel > Note: oneapi-asp tag ofs-2024.2-1 supports upto 4 global memory systems
MEM_`<id>`_NUMBER_OF_MEMORY_BANKS	Number of Global Memory `<id>` Banks	The number of homogenous memory banks in `<id>`th global memory system, where `id` can be 0, 1, 2 or 3 (oneapi-asp tag ofs-2024.2-1 supports upto 4 global memory systems). Used for parameterizing the `ddr_board` system. The `ddr_board` system in `oneapi-asp` for OFS reference platforms has a single `Memory Bank Divider` instantiated for each global memory. The number of memory banks in a single global memory can be 1, 2, 4 or 8 (`Memory Bank Divider` IP requirement). To expand number of banks in a global memory beyond this range, additional `Memory Bank Didvider` IP will need to be instantiated and connected in `ddr_board_hw.tcl`. For more information about `Memory Bank Divider`, please refer to section 3.1.1
MEM_`<id>`_MEMORY_BANK_ADDRESS_WIDTH	Global Memory `<id>` Memory Bank Address Width	This is the total addressable width for a single memory bank/interface in `<id>`th global memory. For example, if global memory `0 (id)` consists of 32 GB on-board DDR4 memory with four 8 GB memory banks, then this memory bank address width parameter must be set to 33 (single 8 GB channel is considered) for `MEM_0_MEMORY_BANK_ADDRESS_WIDTH`. This value is used for parameterizing the `ddr_board` system.
MEM_`<id>`_DATA_WIDTH	Global Memory `<id>` Data Width	This is the width of the data bus for a single single memory bank/interface in the `<id>`th global memory to the oneAPI kernel
MEM_`<id>`_MAX_BURST_SIZE	Global Memory `<id>` Maximum Burst Size	Maximum burst size for the pipeline bridges in the host to memory as well as kernel to memory data path for `<id>`th global memory
MEM_`<id>`_KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE	Global Memory `<id>` Kernel to global memory waitrequest allowance	This is the waitrequest_allowance for the clock crossing bridge in the kernel to memory data path for `<id>`th global memory > Note: For more information about `waitrequest_allowance` refer to section 2.1.4 on interface attribute
MEM_`<id>`_MBD_TO_MEMORY_PIPE_STAGES	Global Memory `<id>` MBD to Memory Pipeline Stages	Value is used to set the number of Avalon-MM pipeline bridges between Memory Bank Divider and the PCIe clock domain to EMIF clock domain clock crossing bridge for `<id>`th global memory

* > Notes (about Number of Global Memory):
1. If the number of global memory systems in more than 1, a pipeline bridge is added in the host to memory data path ensuring the host sees a continuous address space for the total addressable memory. The kernel gets added as a single memory space using pipeline bridge, the kernel connected to each global memory system.
2. If you are using the oneAPI ASP Editor (described in Appendix B), then the RTL design parameterization and board_spec.xml file generaration is handled by the editor. However if you are manually making changes to the ASP implementation, then the ofs_asp_pkg.sv file must be updated to ensure it has all the ASP_GLOBAL_MEM_<id>_* parameters for all global memories in your system. The default file (in oneapi-asp/common/hardware/common/build/rtl if oneapi-asp build flow is not run, else update the file in oneapi-asp/Platform-Name/hardware/Board-Variant/build/rtl) contains the parameters for Global memory 0 only. The board_spec.xml file must also be edited to add global memory interfaces. For more information about board_spec.xml and different global memory configurations, refer to secton 2.1.5.

The internal component instantiations and connections are done in the composition callback (compose procedure) in board_hw.tcl.

For more information about the different component commands and callback used in board_hw.tcl, ddr_board_hw.tcl and ddr_channel.tcl, please refer to the Quartus® Prime Pro Edition User Guide: Platform Designer(Component Interface Tcl Reference section)

5.3.4 User Datagram Protocol(UDP) Engine¶

I/O pipes allow kernel to stream data directly using HSSI. To demonstrate this functionality, reference design in oneapi-asp repository (refer to Figure 5-4 and 5-5) has a UDP protocol engine to allow transmitting UDP/IP packets over HSSI..

Figure below shows a simple block diagram of the UDP engine.

Figure 5-8: UDP Offload Engine

The UDP engine consists of a separate receive (rx) and trasmit (tx) path. The following functionalilty is performed by this reference design engine:

Implements an Address Resolution Protocol (ARP) functionality to respond to be able to send & respond to ARP requests. This is needed for routing between different subnets using a gateway.
Packetizes data from kernel to add the required header information (for MAC, IP & UDP layers)
Extracts data from packets received by removing header information
Handles clock crossing between kernel clock and Ethernet MAC clock domains

The source files for UDP engine used in oneapi-asp for OFS reference platform are located in oneapi-asp/Platform-Name/hardware/Board-Variant/build/rtl/udp_offload_engine directory.

Note: The same engine is used in the board variant with USM shown in Figure 5-5.

5.3.5 Hardware Compile Flow¶

Figure below shows the compile flow overview; the oneAPI compiler generated hardware circuit is compiled by Quartus® software along with design files for oneapi-asp.

Figure 5-9: oneAPI Compile Flow Overview

The oneAPI compiler uses the board_spec.xml to get more information about the oneapi-asp configuration. board_spec.xml file has a compile element to allow control of the Quartus® software compilation flow. The attributes of this element are discussed in section 2.1.2. The oneapi-asp uses tcl scripts to control the Quartus® software compilation flow. Figure 5-10 shows the flow and scripts used. All compilation scripts are located in oneapi-asp/Platform-Name/hardware/Board-Variant/build/scripts folder (location post oneapi-asp build flow).

Figure 5-10: Compilation Scripts in oneapi-asp

Table 5-11 summarizes notes for reference numbers 1-2 marked in figure above.

Table 5-11: Notes for Reference Numbers in Figure 5-10

Reference Number	Note
1	revision_name is `afu_flat` for `oneapi-asps` for OFS reference platforms
2	`gen-asp-quartus-report.tcl` script generates a report (`acl_quartus_report.txt`) containing resource utilization and kernel clock frequency summary. See Note below about the `acl_quartus_report.txt` file

Note: The acl_quartus_report.txt is required by the oneAPI compiler to generate the Quartus (Static) summary in FPGA optimization reports successfully. The oneAPI compiler expects the format to be as follows (order of metrics can be different, but the names of these FPGA metrics should match the format given here):

 
ALUTs: <value>
Registers: <value>
Logic utilization: <value>
I/O pins: <value>
DSP blocks: <value>
Memory bits: <value>
RAM blocks: <value>
Actual clock freq: <value>
Kernel fmax: <value>
1x clock fmax: <value>
2x clock fmax: <value>
Highest non-global fanout: <value>

If acl_quartus_report.txt is missing, the oneAPI compiler will fail to generate the Quartus(Static) summary in FPGA optimization reports successfully. If you are creating a custom platform, please ensure the acl_quartus_report.txt report is generated. The gen-asp-quartus-report.tcl can be used as a reference to implement the generation of this report in your custom platform

5.4 `oneapi-asp` Memory Mapped Device(MMD) Layer Implementation¶

As discussed in section 4.1, the MMD provides a set of API that allow the runtime to control the device and communicate with it.

The source code for MMD layer is located in oneapi-asp/common/source/host folder. aocl_mmd.h is the header file for the implemented API and is located in oneapi-asp/common/source/include folder. Table below summarizes the APIs that have been implemented in oneapi-asp for OFS reference platforms.

Note: For more details about the API, its arguments and enums please refer to the aocl_mmd.h file and to section 4.1.

Table 5-12: MMD API Implemented in oneapi-asp for OFS Reference Platforms

API
aocl_mmd_get_offline_info
aocl_mmd_get_info
aocl_mmd_open
aocl_mmd_close
aocl_mmd_set_interrupt_handler
aocl_mmd_set_status_handler
aocl_mmd_yield
aocl_mmd_read
aocl_mmd_write
aocl_mmd_copy
aocl_mmd_program
aocl_mmd_host_alloc
aocl_mmd_free
aocl_mmd_shared_alloc
aocl_mmd_shared_migrate

The implementation of these APIs is in oneapi-asp/common/source/host/mmd.cpp. The functions used in the implementation are distributed across various source files. Table below provides details on source code files.

Table 5-13: MMD Source Code Files

Files/Folder	Description
mmd.cpp	This file has the implementation for all MMD API calls listed in table 5-12
fpgaconf.h fpgaconf.c	Contains bitstream reconfiguration function declaration(.h) & definition(.c)
kernel_interrupt.h	Contains `KernelInterrupt` class declaration; the class consists of functions to handle kernel interrupts
kernel_interrupt.cpp	Contains `KernelInterrupt` class constructor and function definitions
mmd_device.h	Contains `Device` class declaration, which stores device data and has functions to interact with the device
mmd_device.cpp	Contains `Device` class constructor and function definitions
mmd_dma.h mmd_dma.cpp	Contain DMA functions declaration(.h) & definition(.cpp)
mmd_iopipes.h	Contains the `iopipes` class declaration(.h)
mmd_iopipes.cpp	Contains function definitions, these functions include `iopipes` class constructor as well as fucntions to detect and setup CSR space for IO pipes feature in board variants that support IO pipes
zlib_inflate.h zlib_inflate.c	Function declaration(.h) and definition(.c) for decompressing bitstream data
CMakeLists.txt	CMakeLists.txt file for building MMD source code

The build flow scripts build the MMD library, i.e. libintel_opae_mmd, and place them in oneapi-asp/Platform-Name/linux64/lib folder. The MMD library is specified as part of mmdlib, linklibs element in board_env.xml and used at runtime (refer to section 5-1 for sample board_env.xml file and section 2.2 for more information about board_env.xml elements).

Use of OPAE library in MMD

The MMD layer uses API from OPAE SDK for various device operations. Hence, the MMD layers requires OPAE library to be loaded to execute successfully. The mmdlib element specifies the libopae-c library to be loaded before the MMD library (demonstrated in sample board_env.xml in section 5-1).

Note: Please refer to Software Reference Manual: Open FPGA Stack for more information about OPAE SDK API. The document also has information about linux-dfl driver.

Use of Memory Properties Factory(MPF) library in MMD

In addition to OPAE, the MMD also uses API from Memory Properties Factory(MPF) software for memory operations. The libMPF library is compiled as part of oneapi-asp build flow and placed in oneapi-asp/Platform-Name/linux64/lib. This library is also loaded before the MMD library by specifying before libintel_opae_mmd in mmdlib element in board_env.xml.

Note: Memory Properties Factory(MPF) is a part of Intel® FPGA BBB. Please refer to Intel® FPGA BBB wiki for more information about MPF.

5.5 `oneapi-asp` Utilities Implementation¶

This section covers the implementation of board utilities (refer to section 4.2 for more information on board utilities) in oneapi-asp for OFS reference platforms.

Table below shows the source code/script locations for the utilities.

Table 5-14: oneapi-asp Utilities Source Code Locations

Utility	Source Location
diagnose	`oneapi-asp/common/source/util/diagnostic`
program	`oneapi-asp/common/source/util/reprogram`
install uninstall initialize > Note: These are executable scripts	`oneapi-asp/common/linux64/libexec/` (copied to `oneapi-asp/Platform-Name/linux64/libexec/` during oneapi-asp build flow, for more information about oneapi-asp build flow, refer to section 5.2)

diagnose and program are compiled as part of the oneapi-asp build flow and executables are placed in oneapi-asp/Platform-Name/linux64/libexec/. A CMakeLists.txt file is provided for building the utilities, located in oneapi-asp/common/source/util directory.

The path to all of the above utility executables is used in utilbinder element in board_env.xml (demonstrated in sample board_env.xml in section 5-1). The runtime uses this when the corresponding aocl utility is invoked.

Brief descriptions for the source code files are given in table below.

Table 5-15: Brief Descriptions of oneapi-asp Utility Routines for OFS Reference Platforms

File	Description
setup_permissions.sh	Helper script to configure correct device port permissions, make changes to allow users to lock pages in memory and set the hugepages required for the software stack to function correctly. The helper script is used by `install`, `initialize` routines
install	install routine invokes the `setup_permissions.sh` script after the FPGA Client Driver (FCD) is setup by the runtime
uninstall	uninstall routine reverts the port permission, memory locking and hugepage setting changes performed by `install` routine and is invoked by runtime after the FCD is removed by runtime
initialize	initialize routine performs the following steps: * looks for the initialization binary for the board variant to be initialized * extracts the FPGA hardware configuration file from the oneAPI fat binary using `clang-offload-extract` command provided by oneAPI Base Toolkit (Base Kit) version 2024.0 and beyond * invokes the `setup_permissions.sh` script to set correct device permissions * performs partial reconfiguration of the FPGA device by invoking `program` routine with the initialization bitstream as an argument >Note: For more information about how `initialize` utility extracts FPGA hardware configuration file from oneAPI fat binary, refer to Intel® oneAPI DPC++/C++ Compiler Handbook for Intel® FPGAs
program	`program` routine allocates memory and loads the supplied initialization bitstream in memory followed by a call to reprogramming function provided by `oneapi-asp's` MMD library. The MMD library uses `fpgaReconfigureSlot` API provided by OPAE library to perform device reconfiguration > Note: Please refer to Software Reference Manual: Open FPGA Stack for more information about OPAE SDK API
diagnose	`diagnose` routine scans for the available devices for the installed platform and performs DMA transactions between host & device. It also reports the DMA transfer bandwidth. `diagnose` routine uses functions provided by the MMD library for scanning & opening connection to available devices

Appendix¶

Appendix A: Debug Variables and Commands¶

A.1 Memory Mapped Device(MMD) Layer Debug Variables¶

The Memory Mapped Device(MMD) layer provides debug capability for custom ASP and oneAPI application developers. Snippet below shows how to set the variable and table below documents the environment variables that can be used to get debug information from the MMD layer at runtime.


export MMD_<DMA/ENABLE/PROGRAM>_DEBUG = 1
.<oneapi-binary> # Execute sample with MMD debug environment variable set

Table A-1: Environment Variable to Enable Debug Information in MMD

Environment Variable	Set to Value	Description
MMD_DMA_DEBUG	1	Set this environment variable to see debug information for Direct Memory Access (DMA) transactions
MMD_PROGRAM_DEBUG	1	Set this environment variable to see debug information when the MMD reprograms the PR region
MMD_ENABLE_DEBUG	1	Set this to see debug information for all steps in the MMD layer. Maximum debug information is printed when this environment variable is set<>

A.2 Runtime Debug Variables¶

In addition to the debug variables provided by the MMD layer, there are debug variables provided by the Intel® FPGA Runtime for OpenCL™ Software Technology used in oneAPI base toolkit. For a full list of runtime debug variables, please refer to the Intel® FPGA Runtime for OpenCL™ Software Technology README.

A.3 Extracting oneAPI Binary Informtion using `aocl binedit` Command¶

The aocl binedit utility allows you to extract the following useful information about the compiled binary:

Compilation environment details, such as:
* oneAPI compiler version

* Quartus® Prime software version

* Compiler command used

board_spec.xml from the BSP used for compiling

Kernel fMAX (Quartus-compiled fMAX)

oneAPI ASP and board variant used for compiling

You can use the aocl binedit utility with the following command:

aocl binedit <oneapi-binary> <option:list/get/print/exists> <section_name> <output_file>

Table A-2: Environment Variable to Enable Debug Information in MMD

Example

You can first use the option list to see all the available sections of the oneAPI binary:

aocl binedit <oneapi-binary> list

Command below shows the sections available of a binary generated targeting a oneapi-asp board variant, in this case, the sample compiled is board_test.


    $ aocl binedit board_test.fpga list
    AOCX File: binedit/aocx.0
    Sections in package file:
    .acl.board, 9 bytes
    .acl.board_package, 38 bytes
    .acl.compilation_env, 2528 bytes
    .acl.rand_hash, 40 bytes
    .acl.quartus_input_hash, 163 bytes
    .acl.compileoptions, 0 bytes
    .acl.version, 49 bytes
    .acl.autodiscovery, 2048 bytes
    .acl.board_spec.xml, 2360 bytes
    .acl.kernel_arg_info.xml, 4681 bytes
    .acl.target, 4 bytes
    .acl.rtl_hash, 566 bytes
    .acl.fpga.bin, 11948560 bytes
    .acl.quartus_report, 329 bytes
    .acl.quartus_json, 347 bytes

After knowing the sections available in the oneAPI binary, you can explore the value of each section with the printoption or save the information into a file using the get option.

Command below shows an example of the use of aocl binedit to know the board variant from which the sample was created.

aocl binedit oneapi-binary print .acl.board

Sample output for a board_test.fpga binary that was created with an ofs_n6001 board variant.

$ aocl binedit board_test.fpga print .acl.board
AOCX File: binedit/aocx.0
ofs_n6001

Appendix B: oneAPI Accelerator Support Package(ASP) Editor¶

Note: This section requires readers to be faimilar with the Prerequisites (section 1.3) and the main chapters in this reference manual.

The oneapi-asp tag ofs-2024.2-1 adds a preliminary version of an editor for enabling easy parametrization of oneAPI Accelerator Support Pacakges in OFS based platforms. The features provided by this editor are same as the oneapi-asp for OFS reference platforms described in the main chapters in this manual including support for on-board global memory, Unified Shared Memory (USM), Direct Memory Access(DMA) engine, VTP (Virtual to Physical) Address Translation, I/O Pipes, User Datagram Protocol(UDP ) offload engine. In additon to this the editor handles generation of oneAPI directory structure for your board variant, generation of XML files (board_spec.xml, board_env.xml) and JSON file(oneapi_afu.json) as well as setting parameters for the oneAPI ASP hardware design. The editor is meant to abstract away some of the design steps and reduce design time, it can be used to parameterize oneAPI ASP using IP Parameter Editor GUI in Quartus Prime.

The following sections explain functionality, parameterization and user flow for the oneAPI ASP Editor. The oneAPI Accelerator Support Package (ASP): Getting Started User Guide covers the steps to demonstrate use of this editor for OFS reference platforms.

B.1 oneAPI ASP Editor Overview¶

The following diagram shows GUI view and summarizes the functionality provided by the editor.

Figure B-1: oneAPI ASP Editor GUI

Table below explains the functionality marked in the figure.

Table B-1 oneAPI ASP Editor GUI Functionality

#	Editor Function	Description
1	Parameters Window	The oneAPI ASP interface settings can be set using the parameters window. Users can also derive these settings from the FIM interfaces using presets created from the FIM settings (this requires `OPAE_PLATFORM_ROOT` environment variable to be set, for more information, refer to the user flow in section B.2). Details of each parameter is provided in section B.3
2	Presets Window	Quartus Prime provides an option to create, modify, and save parameter values as a preset file. You can then apply the parameter values in the preset file to the current component that you are parameterizing. The presets window shows the OFS reference platform presets provided in the oneapi-asp. Table B-3 summarizes the presets provided. You can also save your own preset file , e.g. to saving a custom board variant settings. Note: For more information about applying presets, please refer to Quartus® Prime Pro Edition User Guide: Getting Started
3	Generate HDL	The Generate HDL button creates the directory structure for building the oneAPI ASP for selected board variant with the parameters set in the parameter window. It performs the following tasks: * Creates the directory structure and copies required platform files for the selected board variant & device * Generates the Verilog header and package files (`ofs_asp.vh` and `ofs_asp_pkg.sv`) for a board variant based on parameters set in the oneAPI ASP Editor * Generates the `board_env.xml`, `board_spec.xml`, `oneapi_afu.json` > Note: For more information about this XML files, please refer to section 2.0. For more information about `oneapi_afu.json`, refer to table 5-7 in section 5.3 * Generates `parameters.tcl` file used by the `board_hw.tcl` file > Note: For more information about the `board_hw.tcl` file refer to section 5.3.3

The design files for the oneAPI ASP Editor are located in oneapi-asp/oneapi_asp_editor/ directory. The directory structure of the oneapi_asp_editor folder is shown below.

oneapi-asp/
|--common/
|--|--bringup/
|--|--hardware/
|--|--linux64/
|--|--scripts/
|--|--source/
|--Platform-Directories (example : n6001, d5005, fseries-dk, iseries-dk)
|--oneapi_asp_editor
|--|--oneapi_asp_editor_hw.tcl
|--|--oneapi_asp_editor.qpf
|--|--oneapi_asp_editor.qsf
|--|--scripts/
|--|--ip/
|--|--|--presets/
|--|--|--oneapi_asp_editor.ip

Table below provides more information about the files in the oneapi-asp/oneapi_asp_editor directory.

Table B-2: oneapi_asp_editor Directory Contents

File/Folder	Description
oneapi_asp_editor_hw.tcl	The TCL file providing the oneAPI ASP editor functionality
scripts	Helper scripts used by the `oneapi_asp_editor_hw.tcl` to built the `oneapi-asp`
oneapi_asp_editor.qpf oneapi_asp_editor.qsf	Dummy Quartus project files to allow parameterization of the editor using IP Parameter Editor GUI
ip	Contains the `oneapi_asp_editor.ip` file and `presets` folder
ip/oneapi_asp_editor.ip	This is the default .ip file for the oneAPI ASP editor which is used to parameterize and generate `oneapi-asp` design files
ip/presets	This contains the preset files for OFS reference platforms. Table B-3 summarizes the preset files

Table B-3: Presets

Device	OFS Reference Platform	Plaform Name in `oneapi-asp`	Board Variant	Preset File
Agilex® 7 FPGA	Intel® FPGA SmartNIC N6001-PL	n6001	ofs_n6001 ofs_n6001_usm ofs_n6001_iopipes ofs_n6001_usm_iopipes	ofs_n6001.qprs ofs_n6001_usm.qprs ofs_n6001_iopipes.qprs ofs_n6001_usm_iopipes.qprs
Agilex® 7 FPGA	Agilex® 7 FPGA F-Series Development Kit (2x F-Tile)	fseries-dk	ofs_fseries-dk ofs_fseries-dk_usm	ofs_fseries-dk.qprs ofs_fseries-dk_usm.qprs
Agilex® 7 FPGA	Agilex® 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile)	iseries-dk	ofs_iseries-dk ofs_Iseries-dk_usm	ofs_iseries-dk.qprs ofs_iseries-dk_usm.qprs
Stratix® 10 FPGA	Intel® FPGA PAC D5005	d5005	ofs_d5005 ofs_d5005_usm	ofs_d5005.qprs ofs_d5005_usm.qprs

Note: For more information about the board variants, please refer to Table 5-6 in section 5.3.

B.2 oneAPI ASP Editor User Flow¶

The expected user flow with the oneAPI ASP Editor is shown in figure below.

Figure B-2: oneAPI ASP Editor User Flow

The oneAPI ASP Editor can be opened in IP Parameter Editor GUI in Quartus Prime. Building oneAPI ASPs requires FIM compilation to be complete successfully and a PR tree to be generated. The OPAE_PLATFORM_ROOT environment variable must be set to point to the PR tree generated during FIM build.

You can use the oneAPI ASP Editor GUI to set parameters for your oneAPI ASP. Based on your FIM and target platform, you can use one of the options below:

If you are building oneapi-asp for one of the OFS reference platforms, presets are provided in the oneapi-asp/oneapi_asp_editor/ip/presets directory. These presets should show up in the Presets window as shown in figure B-1. Table B-3 summarizes the available presets. Select and apply the preset matching the reference platform you are targeting. Save the settings and click on "Generate HDL".

Notes:
1. For more information about how to apply presets, refer to Quartus® Prime Pro Edition User Guide: Getting Started
2. The oneAPI ASP editor generates only the selected board variant, if you would like to generate all board variants for an OFS reference platform, you will have to apply each preset and click "Generate HDL" for each one sequentially.

If you have a custom FIM, you can either use the ofs_fim.qprs or set all parameters for the oneAPI ASP yourself.

The ofs_fim.qprs file is created from presets generated during FIM compilation flow. This file gets created when the oneAPI ASP editor is opened in the IP Parameter Editor GUI if the OPAE_PLATFORM_ROOT environment variable is set to point to the PR tree generated during FIM build. The ofs_fim.qprs covers some of the interface settings for the oneAPI ASP, there are other parameters required by the ASP that must be set by you when using the ofs_fim.qprs. For more information about which parameters are set by the ofs_fim.qprs, please refer to section B.3.

Another option is to set all the oneAPI ASP Editor parameters yourself, use this option with caution. Please ensure the interface settings in your ASP match your FIM design.

Once the parameters are set, you can save optionally save your own preset file (if cutomizing editor parameters); then save the .ip (File -> Save) and click Generate HDL. For more information on the steps automated by Generate HDL, refer to table B-1.

Note: For more information about saving custom presets, refer to Quartus® Prime Pro Edition User Guide: Getting Started

For more information about each parameter, refer to section B.3. For a step-by-step example of using the oneAPI ASP editor, refer to the oneAPI Accelerator Support Package (ASP): Getting Started User Guide.

B.3 oneAPI ASP Editor Parameters¶

Figure and table below summarizes the oneAPI ASP Editor parameter settings. Figures B-3 to B-8 along with Tables B-5 to B-9 provide details on the parameters under each tab of the editor Parameter window.

The tables B-3 to B-8 have an additional column Derived from FIM settings, the parameters marked yes in this are derived from the preset file generated from the FIM as shown in figure B-2. If you have applied this preset (ofs_fim.qprs) then the parameters marked Yes under Derived from FIM settings column are correctly set to support the interface provided by FIM, you only need to provide settings marked No under this column.

Figure B-3: oneAPI ASP Editor Parameters Summary

Table B-4: oneAPI ASP Editor Parameters Summary

Parameter/Tab	Description
AFU ID	The AFU ID is stored in a register (`board_afu_id` register shown in Figures 5-2 to 5-5) in the oneAPI ASP hardware design. The board AFU ID is used by the MMD layer during board discovery. * 32-bit AFU ID MSB: Upper 32 bits of the board AFU ID * 32-bit AFU ID LSB: Lower 32 bits of the board AFU ID > Note: The AFU ID is not derived from `ofs_fim.qprs`, if you are using the `ofs_fim.qprs` file, this value must be set manually. Ensure the value in MMD (`*_ASP_AFU_ID` in `oneapi-asp/common/source/CMakeLists.txt` matches the value set in this register before building the `oneapi-asp` using `build-asp.sh`)
Global Memory(On-board) Tab	Contains the parameter settings for configuring the global memory for the oneAPI kernel. The settings from this tab are used to generate the global memory interface in the `board_spec.xml` as well as parameterize the global memory interface in the oneAPI ASP hardware design > Note: For more information about `global_mem` element in `board_spec.xml`, please refer to section 2.1.5 in this manual
Unified Shared Memory(USM) Tab	Contains the parameter settings for configuring the Unified Shared Memory interface. The settings from this tab are used to generate the global memory interface with `allocation_type=host, shared` in `board_spec.xml` to support USM. These parameters are also used in the oneAPI ASP hardware design for the USM datapath. > Note: For more information about the USM interface support in `board_spec.xml`, please refer to the section 2.1.5.1.3 in this manual
Direct Memory Access(DMA) Tab	This tab is reserved for future use. Please do not edit the DMA settings. Keep the `Number of DMA Channels` set to 1
Channels Tab	Contains the parameter settings for configuring I/O Pipes to/from oneAPI kernel. The settings from this tab are used to generate the `channels` settings in `board_spec.xml` as well as parameterize the I/O pipes interface in oneAPI ASP hardware design > Note: For more information about `channels`, please refer to section 2.1.8 in this manual
Project Settings Tab	In addition to interface settings, oneAPI ASP design (hardware as well as the `board_spec.xml` requires some more details (e.g. project names, resources) to create oneAPI ASP directory sucessfully. This tab contains some of the additional settings required

Figure B-4: oneAPI ASP Editor Global Memory(On-board) Parameters

Table B-5: oneAPI ASP Editor Global Memory(On-board) Parameters

Note: All parameters in the table below except Number of Global Memory Systems are for different for each global memory in the oneAPI ASP. If the Number of Global Memory Systems is increased, a new tab (titled Global Memory <id>, where <id> starts at 0 for the lowest global memory) gets added for each global memory and you must set the parameters for each additional global memory under its tab.

Parameter/Tab	Editable	Derived from FIM Settings (only if `ofs_fim.qprs` is applied)	Description
Number of Global Memory Systems	- (see Note in description)	No	Set the numbr of global memory systems (for on-board memories) > Note: oneapi-asp tag ofs-2024.2-1 supports upto 4 global memory systems
Name	No	No	The name of global memory (not editable, set to `device`), this value is used as the value for `name` attribute for `global_mem` element in `board_spec.xml` > Note: For more information about `global_mem` element attributes, please refer to table 2-7 in section 2.1.5 in this manual
Maximum theoretical bandwidth	Yes	Yes	The maximum theoretical bandwidth for the global memory, this is used as the value for `max_bandwidth` attribute for `global_mem` element in `board_spec.xml` > Note: For more information about calculating `max_bandwidth`, please refer to table 2-7 in section 2.1.5 in this manual
Number of memory banks	Yes	Yes	Number of memory banks in a single global memory (e.g. if your board has four DDR4 banks in a single global memory, this value must be set to 4). This value is used in the oneAPI kernel interface design in the oneAPI ASP hardware design files as well as to calculate number of `interface` for each global memory in the `board_spec.xml` Permitted Values: 1, 2, 4, 8
Data width	Yes	Yes	Width of the data bus from oneAPI kernel to on-board memory bank (interface width to individual memory bank in a global memory). This is used as the value for `width` of `interface` in `global_mem` element in `board_spec.xml` as well as in oneAPI ASP hardware design parameterization > Note: For more information about `interface`, please refer to in section 2.1.4 in this manual
Address width per bank	Yes	Yes	The address width for each individual memory bank in the global memory. This is used to calculate the `size` and `address` attributes of each `interface` of `global_mem` element in `board_spec.xml` as well as in oneAPI ASP hardware design parameterization
Burst size	Yes	No	The burst size for each individual memory bank interface in the global memory. This is used to calculate the `maxburst` value of `interface` in `global_mem` element in `board_spec.xml` as well as in oneAPI ASP hardware design parameterization
Pipeline stages (kernel to clock crosser)	Yes	No	Number of pipeline stages in kernel to global memory data path (controls pipe depth of pipeline bridge on this data path in `kernel_wrapper.v`)
Waitrequest allowance	No	No	The value is not editable through the Editor and is calculated from number of `Pipeline stages (kernel to clock crosser)`. This is used to set the `waitrequest_allowance` for `interface` in `global_mem` element in `board_spec.xml` as well as in oneAPI ASP hardware design parameterization
Pipeline stages (MBD to global memory)	Yes	No	Number of pipeline stages in the data path from Memory Bank Divider (MBD) to the global memory interface. This value is used in oneAPI ASP hardware design parameterization
AVMM write acknowledge	Yes	No	This is used in controlling the value for `bsp_avmm_write_ack` for `interface` in `global_mem` element in `board_spec.xml` as well as in deciding the addition of `avmm_wr_ack_gen_inst` block in the oneAPI ASP hardware design (refer to Figure 5-2 to 5-6)
Configuration address	No	No	This value is not editable and is calculated automatically by the `oneapi_asp_editor_hw.tcl` for all global memories. The value is used for the `config_addr` attribute of `global_mem` element in `board_spec.xml` > Note: For more information about `config_addr`, refer to table 2-7 in section 2.1.5
Interleaved bytes	No	No	This value is not editable and is set to `4096` in oneapi-asp tag ofs-2024.2-1, this is a known issue, please refer to release notes for more information on the workaround. This value is used for the `interleaved_bytes` attribute of `global_mem` element in `board_spec.xml` > Note: For more information about the calculation of `interleaved_bytes` refer to table 2-7 in section 2.1.5
Latency (for oneAPI compiler)	Yes	No	This value is used for `latency` attribute of `global_mem` element in `board_spec.xml`

Figure B-5: oneAPI ASP Editor Unified Shared Memory Parameters

Table B-6: oneAPI ASP Editor Unified Shared Memory Parameters

Parameter/Tab	Editable	Derived from FIM Settings (only if `ofs_fim.qprs` is applied)	Description
Unified Shared Memory Interface	Yes	No	Used to enable or disable Unified Shared Memory(USM) interface in the oneAPI ASP design. When this box is selected, a `global_mem` element with `allocation_type=host, shared` is added to the `board_spec.xml` and the data path for USM is added to the oneAPI ASP hardware design (Figures 5-3, 5-5 show the implementaion of the USM data path)
Name	No	No	The name of global memory interface for USM (not editable, set to `host`), this value is used as the value for `name` attribute for `global_mem` element for USM interface in `board_spec.xml` > Notes: * For more information about `global_mem` element attributes, please refer to table 2-7 in section 2.1.5 in this manual * For more information about Unified Shared Memory interface, refer to section 2.1.5.1.3
Maximum theoretical bandwidth	Yes	No	Maximum bandwidth for the USM interface. This is used as the value for `max_bandwidth` attribute for `global_mem` element in `board_spec.xml`
Number of interfaces	No	No	This is not editable, set to 1 for USM interface
Data width	Yes	No	The data width for the USM interface, this value is used for the `width` of `interface` in `global_mem` element for USM in `board_spec.xml
Address width	Yes	No	The address width for Unified Shared Memory. This is used to calculate the `size` attribute for `interface` of `global_mem` element for USM in `board_spec.xml`
Burst size	Yes	No	The burst size for USM interface. This is used to calculate the `maxburst` value for `interface` in `global_mem` element for USM in `board_spec.xml` as well as to calculate the burst count setting for this interface in oneAPI ASP hardware design
Pipeline stages	Yes	No	Number of pipeline stages in USM data path (controls pipe depth of pipeline bridge on this data path in `kernel_wrapper.v`)
Waitrequest allowance	No	No	The value is not editable through the Editor and is calculated from number of `Pipeline stages` setting. This is used to set the `waitrequest_allowance` for `interface` in `global_mem` element for USM in `board_spec.xml`
Interleaved bytes	No	No	This value is not editable and is set to `1024`, this value is used for the `interleaved_bytes` attribute of `global_mem` element for USM in `board_spec.xml`
Latency (for oneAPI compiler)	Yes	No	This value is used for `latency` attribute of `global_mem` element for USM in `board_spec.xml`

Figure B-6: oneAPI ASP Editor Direct Memory Access(DMA) Parameters

Table B-7: oneAPI ASP Editor Direct Memory Access(DMA) Parameters

Parameter/Tab	Editable	Derived from FIM Settings (only if `ofs_fim.qprs` is applied)	Description
Number of DMA Channels	- (see Note in description)	No	Reserved for future use - This value is used in the oneAPI ASP hardware design in implementation of the data path from host to DMA engine (see Figure 5-2 to 5-5 for the data path diagram) > Note: oneapi-asp tag ofs-2024.2-1 supports only 1 DMA channel

Figure B-7: oneAPI ASP Editor Channels Parameters

Table B-8: oneAPI ASP Editor Channels Parameters

Parameter/Tab	Editable	Derived from FIM Settings (only if `ofs_fim.qprs` is applied)	Description
Number of I/O Channels	Yes	Yes	Number of I/O pipes to/from oneAPI kernel. This value is used in generating the `channels` interface in `board_spec.xml` as well adding the I/O pipes interface in the oneAPI ASP hardware design (Figures 5-4 to 5-5 show the I/O pipes interface from oneAPI kernel to HSSI Subsystem)
Data Width for I/O Channel `<id>`	Yes	Yes	The width of each individual I/O pipe interface to/from oneAPI kernel. The `<id>` value starts at 0 for the first channel

Figure B-8: oneAPI ASP Editor Project Settings Parameters

Table B-9: oneAPI ASP Editor Project Settings Parameters

Parameter/Tab	Editable	Derived from FIM Settings (only if `ofs_fim.qprs` is applied)	Description
Device Family	Yes	Yes	Set to target device family for your platform, options are: * Stratix® 10 FPGA * Agilex® 7 FPGA
Platform name	Yes	No	Sets the name of the top-level directory that stores oneAPI ASP design files for your target platform and board variant
Board variant name	Yes	No	Sets the name for the board variant directory
Quartus version	Yes	Yes	Sets the Quartus version for the oneAPI ASP
OneAPI compiler version	Yes	No	Sets the oneAPI compiler version, this is used as the value for `version` attribute in `board_env.xml` > Note: For more information about `board_env.xml` attributes, refer to section 2.2 in this manual
Snoop port enable	Yes	No	When this is enabled, it enables the snoop port in the Memory Bank Divider (`acl_asp_snoop_port`) > Note: For more information about Memory Bank Divider, refer to section 3.1.1
Quartus project name	Yes	No	Sets the value of `project` attribute of `compile` element in `board_spec.xml` > Note: For more information about the `compile` element attributes, refer to section 2.1.2 in this manual
Quartus revision name	Yes	No	Sets the value of `revision` attribute of `compile` element in `board_spec.xml` > Note: For more information about the `compile` element attributes, refer to section 2.1.2 in this manual
Device model	Yes	No	Specify the name of the device model file. This value is used to set the file name for `device_model` attribute of `device` element > Note: For more information about `device` element attributes, refer to section 2.1.3
ALMs	Yes	Yes	Adaptive Logic Modules(ALMs) that the board design consumes in the absence of any kernel. This value is used for populating `used_resources` attribute of `device` element > Note: For more information about `device` element attributes, refer to section 2.1.3
FFs	Yes	Yes	Flip-flops(FFs) that the board design consumes in the absence of any kernel. This value is used for populating `used_resources` attribute of `device` element > Note: For more information about `device` element attributes, refer to section 2.1.3
DSPs	Yes	Yes	Digital Signal Processing(DSPs) blocks that the board design consumes in the absence of any kernel. This value is used for populating `used_resources` attribute of `device` element > Note: For more information about `device` element attributes, refer to section 2.1.3
M20Ks	Yes	Yes	RAM (M20Ks) blocks that the board design consumes in the absence of any kernel. This value is used for populating `used_resources` attribute of `device` element > Note: For more information about `device` element attributes, refer to section 2.1.3
Kernel CRA data width	No	No	This is not editable and is set to `64`. This parameter sets the width of Kenel CRA interface and is used as the `width` value of `kernel_cra` port in `interfaces` element in `board_spec.xml` > Notes: * For more information about `kernel_cra`, please refer to table 3-4 in section 3.2.1 * For more information about `interfaces` element, refer to section 2.1.7 * For more information about `interface` attribute, refer to section 2.1.4
Kernel CRA pipeline stages	Yes	No	Number of pipeline stages in host to kernel CRA Avalon® interface (controls pipe depth of pipeline bridge on this control path in `kernel_wrapper.v`) > Note: For more information about `kernel_cra`, please refer to table 3-4 in section 3.2.1
Kernel CRA waitrequest allowance	No	No	The value is not editable through the Editor and is calculated from number of `Kernel CRA pipeline stages` setting. This is used to set the `waitrequest_allowance` of `kernel_cra` port in `interfaces` element in `board_spec.xml`
Range for low clock	No	No	Sets the lower threshold for the user clock frequency (kernel clock) range. This value is used in the creation of `oneapi_afu.json` file. This is not editable and is set to the following values based on target device: * Stratix® 10 FPGA: auto-400 * Agilex® 7 FPGA: auto-600 > Note: For more information about `oneapi_afu.json`, refer to table 5-7 in section 5.3
Range for high clock	No	No	Sets the upper threshold for the user clock frequency (kernel clock) range. This value is used in the creation of `oneapi_afu.json` file. This is not editable and is set to the following values based on target device: * Stratix® 10 FPGA: auto-800 * Agilex® 7 FPGA: auto-1200 > Note: For more information about `oneapi_afu.json`, refer to table 5-7 in section 5.3

Document Revision History¶

Date	Release	Changes
May 26, 2023	2023.1	First release of oneAPI Accelerator Support Package(ASP) Reference Manual: Open FPGA Stack on https://ofs.github.io/
September 15, 2023	2023.2	1. Section 1: * Updated Figure 1-3 to add HSSI path 2. Section 2: * Added `channels` element in figure 2-1 and table 2-1, added information about board variants below this table * Added Section 2.1.8 on `channels` element 3. Section 4: * Added information about MMD API (table 4-1) and new sections 4.1.1 to 4.1.21 4. Section 5: * Moved `oneapi-asp` build flow information into new section 5.2 * Added new table 5-6 with oneAPI ASP board variants information * Added hardware design diagrams and information about new board variants with I/O pipes support (Hardware Design with IO Pipes and Hardware Design with IO Pipes and USM) * Updated hardware design diagrams to show PF/VF Mux/De-mux and added information about PF/VF mapping in section 5.3 * Added new section on UDP engine (section 5.3.4) * Updated figure 5-10 to remove `import_opencl_kernel.tcl` and `add_bbb_to_pr_project.tcl` * Updated table 5-13 to add `mmd_iopipes.h` and `mmd_iopipes.cpp` files
Dec 13, 2023	2023.3-1	1. Section 2: * Update table 2-5 to add information about port parameter * Added new section 2.1.4.1 on port parameter * Update table & figure numbers * Changed "master" and "slave" ports to "host" & "agent" ports in figures 2. Section 4: * Added new executable call supported by initialize utility 3. Section 5: * Updated `initialize` utility information to add `clang-offload-extract`command usage * Updated table with hardware design files information, added information about ofs_asp.sdc * Updated compile flow diagram, added information about new `gen-asp-quartus-report.tcl` script * Updated hardware implementation diagrams for signal name and IP name changes, replaced `mem_if_vtp` block with `host_mem_if_vtp` * Updated `OpenCL Memory Bank Divider` to `Memory Bank Divider` * Updated `OpenCL Kernel Interface` to `Kernel Interface`
Feb 2, 2024	2023.3-2	1. Section 5: * Updated Table 5-8 to add PF/VF mapping for different FIM configurations
Mar 19, 2024	2024.1	1. Section 1: * Added links for FIM developer guides for Agilex® 7 FPGA F-Series Development Kit (2x F-Tile) and Agilex® 7 FPGA I-Series Development Kit (2x R-Tile and 1xF-Tile) * Added link to Intel® oneAPI DPC++/C++ Compiler Handbook for Intel® FPGAs * Updated figure 1-3 to rename `OpenCL Memory Bank Divider` to `Memory Bank Divider` 2. Section 2: * Replaced links to FPGA Optimization Guide for Intel® oneAPI Toolkits with links to Intel® oneAPI DPC++/C++ Compiler Handbook for Intel® FPGAs * Fixed format for table 2-6 * Updated figure 2-6 to rename `OpenCL Memory Bank Divider` to `Memory Bank Divider` * Replaced broken link for USM (Unified Shared Memory) 3. Section 3: * Updated `Memory Bank Divider`section to add information about `memory_bank_divider_hw.tcl`. Replaced figure showing GUI for the IP with parameters in `memory_bank_divider_hw.tcl` and added information about these parameters in table 3-1 * Updated port name `acl_bsp_snoop` to `acl_asp_snoop`and `acl_bsp_memorg_host` to `acl_asp_memorg_host` in table 3-2 * Updated `Kernel Interface` section to add information about `kernel_interface_hw.tcl`. Replaced figure showing GUI for the IP with parameters in `kernel_interface_hw.tcl` and added information about these parameters in table 3-3 4. Section 5: * Replaced `board_env.xml` figure with code snippet * Added links to README for `fseries-dk` & `iseries-dk` oneapi-asp folders in section 5.1 * Updated directory structure in section 5.1 and added information about design files common to oneapi-asp for OFS reference platforms * Update file paths in table 5-2 * Update section for script file name changes from `build-bsp.sh` to `build-asp.sh`, `setup-bsp.py` to `setup-asp.py`, `build-bsp-sw.sh` to `build-asp-sw.sh` * Updated figure 5-2 to add a step in `build-asp.sh` script to copy common files to platform directory * Added Note on `OPAE_SDK_REPO_BRANCH` environment variable in table 5-5 * Added new OFS reference platforms (`fseries-dk` and `iseries-dk`) in table 5-6 * Update section for design file name changes from `board.qsys` to `board_hw.tcl`, `ddr_board.qsys` to `ddr_board_hw.tcl`, `ddr_channel.qsys` to `ddr_channel_hw.tcl`, `bsp_logic.sv` to `asp_logic.sv`, `bsp_design_files.tcl` to `asp_design_files.tcl` * Updated signal names `bsp_mem_if` to `asp_mem_if`, `acl_bsp_memorg_host` to `acl_asp_memorg_host`, `host_mem_if_pa_bsp` to `host_mem_if_pa_asp` * Updated hardware implementation diagrams (figure 5-2 to 5-5) for new file, signal names * Added information about RTL files to table 5-7 * Added `iseries-dk_1pf_1vf.ofss` in table 5-8 * Fixed section 5.3.1 header and added more information about PIM blocks in section 5.3.1 * Added new section 5.3.3 on `board_hw.tcl` * Updated figure 5-10 to add information about `ip-deploy` and `qsys-generate` for `board_hw.tcl` 5. Appendix: * Added appendix section with MMD debug variable information
July 15, 2024	2024.2	1. Section 1: * Added document overview table (Table 1-1) 2. Section 2: * Added a section number 2.1.5.1.3 to `Unified Shared Memory Section` 3. Section 5: * Added information about `oneapi_afu.json` to table 5-7 * Added information about new parameters in `board_hw.tcl` in table 5-10(NUMBER_OF_DMA_CHANNELS, DATA_WIDTH, MAX_BURST_SIZE, KERNEL_GLOBALMEM_WAITREQUEST_ALLOWANCE) and `parameters.tcl` * Added Note about `acl_quartus_report.txt` in section 5.3.5 * Updated table 5-10 for parameters for multiple global memories (NUMBER_OF_GLOBAL_MEMORY_SYSTEMS) 4. Appendix A: * Modified appendix to split into sections explaining debug variables for MMD and runtime * Added new section for `Extracting oneAPI Binary Informtion using`aocl binedit`Command` 5. Appendix B: * Added Appendix B to describe new GUI based flow using oneAPI ASP Editor

Notices & Disclaimers¶

Intel^® technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. ^© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.

OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission of the Khronos Group™.

oneAPI Accelerator Support Package(ASP) Reference Manual: Open FPGA Stack¶

1.0 Overview¶

1.1 About this Document¶

1.2 Terminology¶

1.3 Prerequisites¶

1.4 Introduction to oneAPI on Open FPGA Stack(OFS)¶

1.5 Introduction to oneAPI Accelerator Support Package(ASP)¶

2.0 XML Files in oneAPI ASP¶

2.1 board_spec.xml File¶

2.1.1 board Element¶

2.1.2 compile Element¶

2.1.3 device Element¶

2.1.4 interface Attribute¶

2.1.4.1 port Parameter¶

2.1.5 global_mem Element¶

2.1.5.1 Global Memory Configurations¶

2.1.5.1.1 Contiguous Global Memory¶

2.1.5.1.2 Heterogeneous Memory¶

2.1.5.1.3 Unified Shared Memory¶

2.1.6 host Element¶

2.1.7 interfaces Element¶

2.1.8 channels Element¶

2.2 board_env.xml File¶

3.0 oneAPI ASP Hardware¶

3.1 Host to External Memory Interface(EMIF)¶

3.1.1 Memory Bank Divider¶

3.2 Host to Kernel Interface¶

3.2.1 Kernel Interface¶

3.3 Kernel to External Memory Interface¶

4.0 oneAPI ASP Software¶

4.1 Memory Mapped Device(MMD) Layer¶

4.1.1 aocl_mmd_get_offline_info¶

4.1.2 aocl_mmd_get_info¶

4.1.3 aocl_mmd_open¶

4.1.4 aocl_mmd_close¶

4.1.5 aocl_mmd_set_interrupt_handler¶

4.1.6 aocl_mmd_set_device_interrupt_handler¶

4.1.7 aocl_mmd_set_status_handler¶

4.1.8 aocl_mmd_yield¶

4.1.9 aocl_mmd_read¶

4.1.10 aocl_mmd_write¶

4.1.11 aocl_mmd_copy¶

4.1.12 aocl_mmd_hostchannel_create¶

4.1.13 aocl_mmd_hostchannel_destroy¶

4.1.14 aocl_mmd_hostchannel_get_buffer¶

4.1.15 aocl_mmd_hostchannel_ack_buffer¶

4.1.16 aocl_mmd_program¶

4.1.17 aocl_mmd_host_alloc¶

4.1.18 aocl_mmd_free¶

4.1.19 aocl_mmd_device_alloc¶

4.1.20 aocl_mmd_shared_alloc¶

4.1.21 aocl_mmd_shared_migrate¶

4.2 Board Utilities¶

5.0 oneapi-asp Implementation Details¶

5.1 oneapi-asp Directory Structure¶

5.1.1 hardware Folder¶

5.1.2 common/source Folder¶

5.1.3 linux64 Folder¶

5.2 oneapi-asp Build Flow¶

5.3 oneapi-asp Hardware Implementation¶

5.3.1 Platform Interface Manager(PIM)¶

5.3.2 Direct Memory Access(DMA) Module¶

5.3.3 board_hw.tcl File¶

5.3.4 User Datagram Protocol(UDP) Engine¶

5.3.5 Hardware Compile Flow¶

5.4 oneapi-asp Memory Mapped Device(MMD) Layer Implementation¶

5.5 oneapi-asp Utilities Implementation¶

Appendix¶

Appendix A: Debug Variables and Commands¶

A.1 Memory Mapped Device(MMD) Layer Debug Variables¶

A.2 Runtime Debug Variables¶

A.3 Extracting oneAPI Binary Informtion using aocl binedit Command¶

Appendix B: oneAPI Accelerator Support Package(ASP) Editor¶

B.1 oneAPI ASP Editor Overview¶

B.2 oneAPI ASP Editor User Flow¶

B.3 oneAPI ASP Editor Parameters¶

Document Revision History¶

Notices & Disclaimers¶

2.1 `board_spec.xml` File¶

2.2 `board_env.xml` File¶

4.1.1 `aocl_mmd_get_offline_info`¶

4.1.2 `aocl_mmd_get_info`¶

4.1.3 `aocl_mmd_open`¶

4.1.4 `aocl_mmd_close`¶

4.1.5 `aocl_mmd_set_interrupt_handler`¶

4.1.6 `aocl_mmd_set_device_interrupt_handler`¶

4.1.7 `aocl_mmd_set_status_handler`¶

4.1.8 `aocl_mmd_yield`¶

4.1.9 `aocl_mmd_read`¶

4.1.10 `aocl_mmd_write`¶

4.1.11 `aocl_mmd_copy`¶

4.1.12 `aocl_mmd_hostchannel_create`¶

4.1.13 `aocl_mmd_hostchannel_destroy`¶

4.1.14 `aocl_mmd_hostchannel_get_buffer`¶

4.1.15 `aocl_mmd_hostchannel_ack_buffer`¶

4.1.16 `aocl_mmd_program`¶

4.1.17 `aocl_mmd_host_alloc`¶

4.1.18 `aocl_mmd_free`¶

4.1.19 `aocl_mmd_device_alloc`¶

4.1.20 `aocl_mmd_shared_alloc`¶

4.1.21 `aocl_mmd_shared_migrate`¶

5.0 `oneapi-asp` Implementation Details¶

5.1 `oneapi-asp` Directory Structure¶

5.2 `oneapi-asp` Build Flow¶

5.3 `oneapi-asp` Hardware Implementation¶

5.3.3 `board_hw.tcl` File¶

5.4 `oneapi-asp` Memory Mapped Device(MMD) Layer Implementation¶

5.5 `oneapi-asp` Utilities Implementation¶

A.3 Extracting oneAPI Binary Informtion using `aocl binedit` Command¶