Plugins case study: Pluggy

Defining hooks

Out host defines two hooks:

import pluggy

hookspec = pluggy.HookspecMarker("htmlize")

@hookspec(firstresult=True)
def htmlize_role_handler(role_name):
    """Return a function accepting role contents.

    The function will be called with a single argument - the role contents, and
    should return what the role gets replaced with.
    """
    pass

@hookspec
def htmlize_contents(post, db):
    """Return a function accepting full document contents.

    The function will be called with a single argument - the document contents
    (after paragraph splitting and role processing), and should return the
    transformed contents.
    """
    pass

A hook is created by calling HookspecMarker with the project's name. This project name has to match between the host and its plugins. Pluggy is permissive about what hooks accept as parameters and what they return; for maximal flexibility and to stay true to the original htmlize example, our hooks return functions.

To accompany this HookspecMarker, the host also defines a HookimplMarker with the same name:

hookimpl = pluggy.HookimplMarker("htmlize")

This is used by plugins to attach to hooks when they're loaded.

Loading plugins in the host

The host's main function loads plugins at startup as follows:

pm = pluggy.PluginManager("htmlize")
pm.add_hookspecs(hookspecs)
pm.load_setuptools_entrypoints("htmlize")

hookspecs is our Python module containing the hooks shown above. load_setuptools_entrypoints is Pluggy's helper for loading plugins that were pip-installed into the same environment and registered as setuptools entry points. It's a way to signal - in one's setup.py or pyproject.toml file - some metadata that projects can review at runtime. In our project, the plugins register themselves with this section in the pyproject.toml file:

[project.entry-points.htmlize]
tt = "tt"

This says "for entry point htmlize, define a new entry named tt". Pluggy's load_setuptools_entrypoints then uses importlib.metadata to access this information.

Note that Pluggy doesn't require using this mechanism. Hosts can implement any plugin discovery method they want, and add plugins directly to their PluginManager with the register method. But this is the mechanism used for pytest and many other projects; it makes it very easy to automatically discover and register plugins that are installed with pip and equivalent tools.

Invoking plugins

Once PluginManager loads the plugins, invoking them is straightforward; here's how htmlize invokes the contents hooks [1]:

# Build full contents back again, and ask plugins to act on
# contents.
contents = ''.join(parts)
for handler in plugin_manager.hook.htmlize_contents(post=post, db=db):
    contents = handler(contents)
return contents

Generally, hook invocations return a list of all the hooks attached to by different plugins (a single host application can have multiple plugins installed and attaching to the same hook). When the host invokes the hook as shown above, the default order is LIFO, but plugins can affect this with hook options like tryfirst and trylast.

Implementing hooks in plugins

Here's our entire narcissist plugin that's attaching to the contents hook:

import htmlize

@htmlize.hookimpl
def htmlize_contents(post, db):
    repl = f'<b>I ({post.author})</b>'

    def hook(contents):
        return re.sub(r'\bI\b', repl, contents)

    return hook

Some notes:

• It expects htmlize to be installed; as discussed previously, we rely on Pluggy's default install-based approach where both the host and plugins are installed into the same Python environment and can thus find each other. However, Pluggy supports any custom discovery method.
• It uses the hookimpl exported value shown earlier.
• It returns a function that acts on contents; this is the htmlize-specific contract (ABI, if you will) we've discussed before.

Discovery

Generally, Pluggy leaves discovery logic to the user's discretion. Its PluginManager has a register method for adding plugins, and these can be discovered in any way the application chooses.

That said, Pluggy comes with one discovery mechanism built in - through the entry points process of Python packaging, as shown above. This is hugely convenient for a large number of applications, as long as both the application and its plugins are installed via standard Python packaging tools (which is a very reasonable assumption in the Python ecosystem).

Registration

In the entry point process, plugins register themselves by adding a [project.entry-points.<HOST-ID>] section in their pyproject.toml file.

Otherwise - as in the previous section - users are free to devise their own registration schemes.

Hooks

This one is easy, since it's called hooks in Pluggy parlance as well! Pluggy's implementation of hooks is rather elegant, with function decorators available for plugins to set. We've seen an example of this above with @htmlize.hookimpl decorating htmlize_contents.

Exposing an application API to plugins

Since Pluggy is designed for Python hosts and Python plugins, this one is fairly straightforward. The plugins typically assume the host project is already installed in the Python environment and its modules can be imported.

In our example, hookimpl is imported from htmlize by the plugin to accomplish this. It also shows how host data is passed to the plugin - the post and db parameters. These are APIs exposed by the host for the plugins' use.