Plugin Developer's Guide¶

Overview¶

Beginning with OPAE C library version 1.2.0, OPAE implements a plugin-centric model. This guide serves as a reference to define the makeup of an OPAE C API plugin and to describe a sequence of steps that one may follow when constructing an OPAE C API plugin.

Plugin Required Functions¶

An OPAE C API plugin is a runtime-loadable shared object library, also known as a module. On Linux systems, the dl family of APIs from libdl are used to interact with shared objects. Refer to "man dlopen" and "man dlsym" for examples of using the libdl API.

An OPAE C API plugin implements one required function. This function is required to have C linkage, so that its name is not mangled.

    int opae_plugin_configure(opae_api_adapter_table *table, const char *config);

During initialization, the OPAE plugin manager component loads each plugin, searching for its opae_plugin_configure function. If none is found, then the plugin manager rejects that plugin. When it is found, opae_plugin_configure is called passing a pointer to a freshly-created opae_api_adapter_table and a buffer consisting of configuration data for the plugin.

The job of the opae_plugin_configure function is to populate the given adapter table with each of the plugin's API entry points and to consume and comprehend the given configuration data in preparation for initialization.

OPAE API Adapter Table¶

The adapter table is a data structure that contains function pointer entry points for each of the OPAE APIs implemented by a plugin. In this way, it adapts the plugin-specific behavior to the more general case of a flat C API. Note that OPAE applications are only required to link with opae-c. In other words, the name of the plugin library should not appear on the linker command line. In this way, plugins are truly decoupled from the OPAE C API, and they are required to adapt to the strict API specification by populating the adapter table only. No other linkage is required nor recommended.

adapter.h contains the definition of the opae_api_adapter_table. An abbreviated version is depicted below, along with supporting type opae_plugin:

    typedef struct _opae_plugin {
        char *path;
        void *dl_handle;
    } opae_plugin;

    typedef struct _opae_api_adapter_table {

        struct _opae_api_adapater_table *next;
        opae_plugin plugin;

        fpga_result (*fpgaOpen)(fpga_token token, fpga_handle *handle,
                                int flags);

        fpga_result (*fpgaClose)(fpga_handle handle);

        ...

        fpga_result (*fpgaEnumerate)(const fpga_properties *filters,
                                     uint32_t num_filters, fpga_token *tokens,
                                     uint32_t max_tokens,
                                     uint32_t *num_matches);

        ...

        // configuration functions
        int (*initialize)(void);
        int (*finalize)(void);

        // first-level query
        bool (*supports_device)(const char *device_type);
        bool (*supports_host)(const char *hostname);

    } opae_api_adapter_table;

Some points worth noting are that the adapter tables are organized in memory by adding them to a linked list data structure. This is the use of the next structure member. (The list management is handled by the plugin manager.) The plugin structure member contains the handle to the shared object instance, as created by dlopen. This handle is used in the plugin's opae_plugin_configure to load plugin entry points. A plugin need only implement the portion of the OPAE C API that a target application needs. Any API entry points that are not supported should be left as NULL pointers (the default) in the adapter table. When an OPAE API that has no associated entry point in the adapter table is called, the result for objects associated with that plugin will be FPGA_NOT_SUPPORTED.

The following code illustrates a portion of the opae_plugin_configure for a theoretical OPAE C API plugin libfoo.so:

    /* foo_plugin.c */

    int opae_plugin_configure(opae_api_adapter_table *table, const char *config)
    {
        adapter->fpgaOpen = dlsym(adapter->plugin.dl_handle, "foo_fpgaOpen");
        adapter->fpgaClose =
                dlsym(adapter->plugin.dl_handle, "foo_fpgaClose");

        ...

        adapter->fpgaEnumerate =
                dlsym(adapter->plugin.dl_handle, "foo_fpgaEnumerate");

        ...

        return 0;
    }

Notice that the implementations of the API entry points for plugin libfoo.so are prefixed with foo_. This is the recommended practice to avoid name collisions and to enhance the debugability of the application. Upon successful configuration, opae_plugin_configure returns 0 to indicate success. A non-zero return value indicates failure and causes the plugin manager to reject the plugin from futher consideration.

Plugin Optional Functions¶

Once the plugin manager loads and configures each plugin, it uses the adapter table to call back into the plugin so that it can be made ready for runtime. This is the job of the opae_plugin_initialize entry point, whose signature is defined as:

    int opae_plugin_initialize(void);

The function takes no parameters, as the configuration data was already given to the plugin by opae_plugin_configure. opae_plugin_initialize returns 0 if no errors were encountered during initialization. A non-zero return code indicates that plugin initialization failed. A plugin makes its opae_plugin_initialize available to the plugin manager by populating the adapter table's initialize entry point as shown:

    /* foo_plugin.c */

    int foo_plugin_initialize(void)
    {
        ...

        return 0; /* success */
    }

    int opae_plugin_configure(opae_api_adapter_table *table, const char *config)
    {
        ... 

        adapter->initialize =
                dlsym(adapter->plugin.dl_handle, "foo_plugin_initialize");

        ...

        return 0;
    }

If a plugin does not implement an opae_plugin_initialize entry point, then the initialize member of the adapter table should be left uninitialized. During plugin initialization, if a plugin has no opae_plugin_initialize entry in its adapter table, the plugin initialization step will be skipped, and the plugin will be considered to have initialized successfully.

Once plugin initialization is complete for all loaded plugins, the system is considered to be running and fully functional.

During teardown, the plugin manager uses the adapter table to call into each plugin's opae_plugin_finalize entry point, whose signature is defined as:

    int opae_plugin_finalize(void);

opae_plugin_finalize returns 0 if no errors were encountered during teardown. A non-zero return code indicates that plugin teardown failed. A plugin makes its opae_plugin_finalize available to the plugin manager by populating the adapter table's finalize entry point as shown:

    /* foo_plugin.c */

    int foo_plugin_finalize(void)
    {
        ...

        return 0; /* success */
    }

    int opae_plugin_configure(opae_api_adapter_table *table, const char *config)
    {
        ... 

        adapter->finalize =
                dlsym(adapter->plugin.dl_handle, "foo_plugin_finalize");

        ...

        return 0;
    }

If a plugin does not implement an opae_plugin_finalize entry point, then the finalize member of the adapter table should be left uninitialized. During plugin cleanup, if a plugin has no opae_plugin_finalize entry point in its adapter table, the plugin finalize step will be skipped, and the plugin will be considered to have finalized successfully.

In addition to initialize and finalize, an OPAE C API plugin has two further optional entry points that relate to device enumeration. During enumeration, when a plugin is being considered for a type of device, the plugin may provide input on that decision by exporting an opae_plugin_supports_device entry point in the adapter table:

    bool opae_plugin_supports_device(const char *device_type);

opae_plugin_supports_device returns true if the given device type is supported and false if it is not. A false return value from opae_plugin_supports_device causes device enumeration to skip the plugin.

Populating the opae_plugin_supports_device is done as:

    /* foo_plugin.c */

    bool foo_plugin_supports_device(const char *device_type)
    {
        if (/* device_type is supported */)
            return true;

        ...

        return false;
    }

    int opae_plugin_configure(opae_api_adapter_table *table, const char *config)
    {
        ... 

        adapter->supports_device =
                dlsym(adapter->plugin.dl_handle, "foo_plugin_supports_device");

        ...

        return 0;
    }

.. note::
    The `opae_plugin_supports_device` mechanism serves as a placeholder only.
    It is not implemented in the current version of the OPAE C API.

Similarly to determining whether a plugin supports a type of device, a plugin may also answer questions about network host support by populating an opae_plugin_supports_host entry point in the adapter table:

    bool opae_plugin_supports_host(const char *hostname);

opae_plugin_supports_host returns true if the given hostname is supported and false if it is not. A false return value from opae_plugin_supports_host causes device enumeration to skip the plugin.

Populating the opae_plugin_supports_host is done as:

    /* foo_plugin.c */

    bool foo_plugin_supports_host(const char *hostname)
    {
        if (/* hostname is supported */)
            return true;

        ...

        return false;
    }

    int opae_plugin_configure(opae_api_adapter_table *table, const char *config)
    {
        ... 

        adapter->supports_host =
                dlsym(adapter->plugin.dl_handle, "foo_plugin_supports_host");

        ...

        return 0;
    }

.. note::
    The `opae_plugin_supports_host` mechanism serves as a placeholder only.
    It is not implemented in the current version of the OPAE C API.

Plugin Construction¶

The steps required to implement an OPAE C API plugin, libfoo.so, are:

Create foo_plugin.c: implements opae_plugin_configure, opae_plugin_initialize, opae_plugin_finalize, opae_plugin_supports_device, and opae_plugin_supports_host as described in the previous sections.
Create foo_plugin.h: implements function prototypes for each of the plugin-specific OPAE C APIs.

    /* foo_plugin.h */

    fpga_result foo_fpgaOpen(fpga_token token, fpga_handle *handle,
                             int flags);

    fpga_result foo_fpgaClose(fpga_handle handle);

    ...

    fpga_result foo_fpgaEnumerate(const fpga_properties *filters,
                                  uint32_t num_filters, fpga_token *tokens,
                                  uint32_t max_tokens,
                                  uint32_t *num_matches);
    ...

Create foo_types.h: implements plugin-specific types for opaque data structures.

    /* foo_types.h */

    struct _foo_token {
        ...
    };

    struct _foo_handle {
        ...
    };

    struct _foo_event_handle {
        ...
    };

    struct _foo_object {
        ...
    };

Create foo_enum.c: implements foo_fpgaEnumerate, foo_fpgaCloneToken, and foo_fpgaDestroyToken.
Create foo_open.c: implements foo_fpgaOpen.
Create foo_close.c: implements foo_fpgaClose.
Create foo_props.c: implements foo_fpgaGetProperties, foo_fpgaGetPropertiesFromHandle, foo_fpgaUpdateProperties
Create foo_mmio.c: implements foo_fpgaMapMMIO, foo_fpgaUnmapMMIO foo_fpgaWriteMMIO64, foo_fpgaReadMMIO64, foo_fpgaWriteMMIO32, foo_fpgaReadMMIO32.
Create foo_buff.c: implements foo_fpgaPrepareBuffer, foo_fpgaReleaseBuffer, foo_fpgaGetIOAddress.
Create foo_error.c: implements foo_fpgaReadError, foo_fpgaClearError, foo_fpgaClearAllErrors, foo_fpgaGetErrorInfo.
Create foo_event.c: implements foo_fpgaCreateEventHandle, foo_fpgaDestroyEventHandle, foo_fpgaGetOSObjectFromEventHandle, foo_fpgaRegisterEvent, foo_fpgaUnregisterEvent.
Create foo_reconf.c: implements foo_fpgaReconfigureSlot.
Create foo_obj.c: implements foo_fpgaTokenGetObject, foo_fpgaHandleGetObject, foo_fpgaObjectGetObject, foo_fpgaDestroyObject, foo_fpgaObjectGetSize, foo_fpgaObjectRead, foo_fpgaObjectRead64, foo_fpgaObjectWrite64.
Create foo_clk.c: implements foo_fpgaSetUserClock, foo_fpgaGetUserClock.