PF4J and Clojure

Today we’ll take a look at an interesting Java library, called PF4J, which describes itself as a Plugin Framework for Java. The main purpose of this library is to provide a way of detecting, initializing, and using plugins to extend your Java application with new features without the need to modify the code. The official repository has examples of how to use this library with Java, and Kotlin, but there is no Clojure example.

Recently I had to find a way how to use this library with Clojure, and it has turned a bit hard to achieve, because this library is very Javaish, if you know what I mean :). Classes, interfaces, and Java annotations are not the most common thing to see in Clojure, as we usually have no classes, interfaces, or annotations, but functions, data abstractions, and metadata. I’m not very experienced in Java bytecode inspection, and Clojure class generation, and I was unable to find any information on how to integrate PF4J and Clojure, so this post will explain what I’ve found so far. You know, so I would be able to refresh this if I ever again will need to use this library, or will find myself in a similar situation where I need to work with annotations or class generation. And hopefully, this information will be useful for others as well.

So here we will write an example plugin, in the same way how Java example plugins are written in the pf4j demo application. But first, let’s look at how the Java plugin is structured.

Anatomy of a plugin

The complete plugin consists of two main components - the plugin itself, and the extension point. We will take the .jar approach for storing and distributing plugins, as it is the easiest one to reach, and we can directly compare the bytecode of the Java plugin, and Clojure plugin. So let’s first create an extension point in Java.

Create project Greeting in your Java IDE of choice. I’m creating a maven project, so the first thing we’ll need is to add pf4j dependency to our pom.xml:

<!-- snip -->
<dependencies>
  <dependency>
    <groupId>org.pf4j</groupId>
    <artifactId>pf4j</artifactId>
    <version>3.6.0</version>
  </dependency>
</dependencies>
<!-- snap -->

Now we can create extension point class org.example.Greeting, and add this code into it:

package org.example;

import org.pf4j.ExtensionPoint;

public interface Greeting extends ExtensionPoint {
    String getGreeting();
}

This is all we need for our small extension point, and we can install it in our local repo with mvn install. AFAIU, extension point’s main purpose is to help PF4J detect particular plugins, and not all plugins.

Now we need the plugin itself. This plugin depends on both pf4j and this extension point we’ve just created and installed. Create a new project JavaWelcomePlugin, and add there pf4j dependency as explained above, and the org.example.Greeting dependency:

<!-- snip -->
<dependency>
  <groupId>org.example</groupId>
  <artifactId>Greeting</artifactId>
  <version>1.0-SNAPSHOT</version>
</dependency>
<!-- snap -->

Again, create a new class GreetingsPlugin:

package org.example;

import org.pf4j.Extension;
import org.pf4j.Plugin;
import org.pf4j.PluginWrapper;
import org.example.Greeting;

public class GreetingsPlugin extends Plugin {
    public GreetingsPlugin(PluginWrapper wrapper) {
        super(wrapper);
    }

    @Override
    public void start() { System.out.println("Java plugin start"); }

    @Override
    public void stop() { System.out.println("Java plugin stop"); }

    @Extension
    public static class Welcome implements Greeting {
        public String getGreeting() {
            return "Hi from Java!";
        }
    }
}

This plugin has start and stop methods that pf4j will call during initialization, and deinitialization of the plugin, and the main part here is the nested Welcome class. This class implements our Greeting extension point, and is annotated with @Extension, which helps pf4j discover this class during the build process.

One more thing we need to add to our pom.xml are new manifest entries. Pf4j expects each plugin to have at least two entries in MANIFEST.mf file in the .jar archive: Plugin-Version, and Plugin-ID. In our case, we will also need to add Plugin-Class entry, as we’ve created a wrapper class GreetingsPlugin. Add this to the pom.xml:

<!-- snip -->
<build>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-jar-plugin</artifactId>
      <configuration>
        <archive>
          <manifestEntries>
            <Plugin-ID>java-welcome-plugin</Plugin-ID>
            <Plugin-Class>org.example.GreetingsPlugin</Plugin-Class>
            <Plugin-Version>1.0</Plugin-Version>
          </manifestEntries>
        </archive>
      </configuration>
    </plugin>
  </plugins>
</build>
<!-- snap -->

It’s a bit too verbose (well, it is XML after all), but here we’re just adding these three entries to our manifest file during the build process. Now let’s package this plugin with the mvn package command.

This makes the complete example plugin in Java - much like the one in the demo application from PF4J examples. Both the interface and the plugin are written, and we can plug this code into any application, that looks for our Greeting interface. Now let’s test this plugin in a small Clojure program.

Writing Clojure program that uses plugins

Let’s create a small project based on the app template, that uses pf4j library, and loads this plugin we’ve just written in Java. I’m using Leiningen for project automation, but you may use deps or other tools if you prefer. But we will later need to edit project.clj and do some tweaks other than adding dependencies, so if you’re using deps you’re on your own here. For me, the command is:

$ lein new app pf4j-clj-app

A new project is created and we need to add org.pf4j/pf4j and org.example.Greeting dependencies to the :dependencies section in project.clj:

:dependencies [[org.clojure/clojure "1.10.1"]
               [org.pf4j/pf4j "3.6.0"]
               [org.example/Greeting "1.0-SNAPSHOT"]]

Rest of the project.clj can be left as is. Now, let’s write our -main function in the core.clj:

(ns pf4j-clj-app.core
  (:import [org.pf4j JarPluginManager]
           [org.example Greeting])
  (:gen-class))

(defn -main
  [& _]
  (let [plugin-manager (JarPluginManager.)]
    (doto plugin-manager
      (.loadPlugins)
      (.startPlugins))
    (let [greets (.getExtensions plugin-manager Greeting)]
      (println "Found" (count greets) "extensions for extension point" (.getName Greeting))
      (doseq [greet greets]
        (println (.getGreeting greet))))
    (doto plugin-manager
      (.stopPlugins)
      (.unloadPlugins))))

I’ll keep this brief - we’re importing our Greeting extension point, and the pf4j library, creating plugin-manager, loading and starting plugins. Then we print how many plugins we’ve found for the Greeting extension point, and execute the .getGreeting method for each plugin. And after that, we’re stopping and unloading plugins to keep things clean.

By default pf4j searches for jar plugins in the plugin directory at the root of the project. Let’s create this directory, copy our Java plugin there, and start our application with lein run:

$ mkdir plugins
$ cp path/to/JavaWelcomePlugin/target/JavaWelcomePlugin-1.0-SNAPSHOT.jar ./plugins/
$ lein run
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Java plugin start
Found 1 extensions for extension point 'org.example.Greeting'
Hi from Java!
Java plugin stop

It works! We’ve seen the message from the start method, the getGreeting call result, and then the stop method message. Now let’s do everything in Clojure!

Clojure ExtensionPoint implementation

To make an extension point in Clojure, let’s create a new library project with lein new clj-greeting, and add needed changes to our project.clj:

(defproject org.example/Greeting "1.0-SNAPSHOT"
  :description "pf4j extension poin example in Clojure"
  :license {:name "EPL-2.0 OR GPL-2.0-or-later WITH Classpath-exception-2.0"
            :url "https://www.eclipse.org/legal/epl-2.0/"}
  :dependencies [[org.clojure/clojure "1.10.1"]
                 [org.pf4j/pf4j "3.6.0"]]
  :aot :all)

I’ve mainly changed the project name from clj-greeting to org.example/Greeting so we could hot-swap our Java extension point to the Clojure one with simple lein install. Similarly, the version was changed to 1.0-SNAPSHOT to match the version of our Java extension point. The pf4j dependency was added to the :dependencies key. Lastly, :aot :all entry was added to make sure we AOT compile every file.

Now we can create our interface. To do so, first we need to create needed directories with src/org/example directories, and move core.clj to src/org/example/greeting.clj:

$ mkdir -p src/org/example
$ mv src/clj_greeting/core.clj src/org/example/greeting.clj
$ rmdir src/clj_greeting

Next, we add this code to greeting.clj:

(ns org.example.greeting
  (:import org.pf4j.ExtensionPoint))

(gen-interface
 :name org.example.Greeting
 :extends [org.pf4j.ExtensionPoint]
 :methods [[getGreeting [] String]])

Here, we’re creating an interface with gen-inteface and naming it to org.example.Greeting. This interface extends the org.pf4j.ExtensionPoint class, and exposes one method getGreeting which takes zero arguments, and returns a String.

Let’s package this as a jar archive with lein jar command, and see its contents with zipinfo. If you don’t have zipinfo you can use any other way to view archives. Here are the contents:

lein jar && zipinfo -1 target/Greeting-1.0-SNAPSHOT.jar
Compiling clj-greetings.core
Created /home/alistopadov/Git/clj-greetings/target/Greeting-1.0-SNAPSHOT.jar
META-INF/MANIFEST.MF
META-INF/maven/org.example/Greeting/pom.xml
META-INF/leiningen/org.example/Greeting/project.clj
META-INF/leiningen/org.example/Greeting/README.md
META-INF/leiningen/org.example/Greeting/LICENSE
META-INF/
META-INF/maven/
META-INF/maven/org.example/
META-INF/maven/org.example/Greeting/
META-INF/maven/org.example/Greeting/pom.properties
org/
org/example/
org/example/Greeting.class
clj_greetings/
clj_greetings/core__init.class
clj_greetings/core$loading__6721__auto____171.class
clj_greetings/core$fn__173.class
clj_greetings/core.clj

We’re mainly interested in the org/example/Greeting.class entry. Let’s decompile it. This can be either done with Intellij Idea or by using fernflower directly. The decompiled bytecode looks like this:

package org.example;

import org.pf4j.ExtensionPoint;

public interface Greeting extends ExtensionPoint {
   String getGreeting();
}

And this looks exactly like the Java example we’ve written before. Great, let’s test it!

We do this by installing this extension point in place of our Java one, by using the lein install command (you can remove the old extension from the .m2 directory for extra safety). Then we need to repackage our Java plugin and copy it again to our Clojure application. When we run it, we should see the same output:

Java plugin start
Found 1 extension for extension point org. example.Greeting
Hi from Java!
Java plugin stop

So, our extension point is now implemented in Clojure. Now we can write the plugin itself!

PF4J plugin in Clojure

To write a plugin let’s create another project with lein new clj-welcome-plugin. Again, we need to alter project.clj a bit:

(defproject clj-welcome-plugin "0.1.0-SNAPSHOT"
  :description "pf4j plugin example in Clojure"
  :license {:name "EPL-2.0 OR GPL-2.0-or-later WITH Classpath-exception-2.0"
            :url "https://www.eclipse.org/legal/epl-2.0/"}
  :dependencies [[org.clojure/clojure "1.10.1"]
                 [org.pf4j/pf4j "3.6.0"]
                 [org.example/Greeting "1.0-SNAPSHOT"]]
  :manifest {"Plugin-Id" "clojure-welcome-plugin"
             "Plugin-Version" "0.1.0"
             "Plugin-Class" "clj_welcome_plugin.GreetingsPlugin"}
  :aot :all)

Again, we’re adding :aot :all entry and the :manifest key with a map of keys and values of manifest entries we need to add. Thankfully, Leiningen makes it really easy to deal with this, compared to pom.xml we’ve had to modify previously.

Now, we need to create a wrapper class, and the @Extension class, and here’s a problem. To my knowledge, there’s no way to write nested classes in Clojure. And annotations are also a bit finicky. So let’s approach this step by step.

The first thing we’ve done in Java code was import all needed libraries, and create a class that extended Plugin class:

package org.example;

import org.pf4j.Extension;
import org.pf4j.Plugin;
import org.pf4j.PluginWrapper;
import org.example.Greeting;

public class GreetingsPlugin extends Plugin {
    public GreetingsPlugin(PluginWrapper wrapper) {
        super(wrapper);
    }
    /* ... */
}

Let’s do the same in Clojure. We start by renaming the core.clj to greetings_plugin.clj, and importing all needed libraries, also saying that our :gen-class has no main and :extends org.pf4j.Plugin:

(ns clj-welcome-plugin.greetings-plugin
  (:import [org.pf4j Plugin PluginWrapper Extension]
           [org.example Greeting])
  (:gen-class
   :name clj_welcome_plugin.GreetingsPlugin
   :main false
   :extends org.pf4j.Plugin))

So far so good. Now we need a constructor, but in our case, Clojure has already generated one, which we can see by compiling our code into the jar file, and looking through the bytecode:

package clj_welcome_plugin;

/* Clojure imports */
import org.pf4j.Plugin;
import org.pf4j.PluginWrapper;

public class GreetingsPlugin extends Plugin {
   /* some initialization Clojure stuff */

   public GreetingsPlugin(PluginWrapper var1) {
      super(var1);
   }

   /* rest of the Clojure stuff */
}

A custom constructor can be provided by using the :init key in :gen-class, but I’ll leave it out to the reader.

Now we can add start and stop methods:

(defn -start [_]
  (println "Clojure plugin start"))

(defn -stop [_]
  (println "Clojure plugin stop"))

And this basically completes our wrapper class, so how do we add a nested class? Well, actually, from a bytecode standpoint this is not a nested class, and the outer class simply acts as a namespace. We can actually see that in the contents of the jar file which contains compiled code of our Java plugin:

org/example/GreetingsPlugin.class
org/example/GreetingsPlugin$Welcome.class

So the inner public static Welcome class is actually a separate .class file, and we can replicate this in Clojure pretty easily. Let’s add another call to gen-class in the same file:

(gen-class
 :name ^{org.pf4j.Extension {}} clj_welcome_plugin.GreetingsPlugin$Welcome
 :prefix "impl-"
 :implements [org.example.Greeting])

(defn impl-getGreeting [_] "Hi from Clojure!")

Or we can create another file, and use :gen-class there, but to match Java plugin source code layout, I’m using the single file.

The most important thing here is the :name key. You can see that we’re providing a metadata entry ^{org.pf4j.Extension {}} which is a way to add Java annotation to a class. And we’re naming our class as clj_welcome_plugin.GreetingsPlugin$Welcome. Lastly, the :implements key is used to tell, that our class implements our extension point. And because we’re using the same file, we change :prefix to "impl-", and the getGreeting method is defined as impl-getGreeting.

Let’s compile our code with lein jar again, and see if we’ve managed to replicate the .class layout in the jar archive:

$ lein jar && zipinfo -1 target/clj-welcome-plugin-0.1.0-SNAPSHOT.jar
...
META-INF/MANIFEST.MF
META-INF/leiningen/...
META-INF/maven/...
clj_welcome_plugin/GreetingsPlugin.class
clj_welcome_plugin/GreetingsPlugin$Welcome.class
...

Great! Let’s see if our GreetingsPlugin$Welcome.class actually decompiles to correct class:

package clj_welcome_plugin;

/* Clojure imports */
import org.example.Greeting;
import org.pf4j.Extension;

@Extension
public class GreetingsPlugin$Welcome implements Greeting {
   /* Clojure stuff */
   public String getGreeting() {
      Var var10000 = getGreeting__var;
      Object var1 = getGreeting__var.isBound() ? var10000.get() : null;
      if (var1 != null) {
         return (String)((IFn)var1).invoke(this);
      } else {
         throw new UnsupportedOperationException("getGreeting (clj-welcome-plugin.greetings-plugin/impl-getGreeting not defined?)");
      }
   }
}

And it is! We can see the @Extension annotation, and public class GreetingsPlugin$Welcome that implements Greeting extension point. Later, after some Clojure internal code, we can see our getGreeting method definition.

However, if we try to use this plugin, it will not work just yet:

$ cp path/to/clj-welcome-plugin/target/clj-welcome-plugin-0.1.0-SNAPSHOT.jar ./plugins/
$ lein run
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Java plugin start
Found 1 extensions for extension point 'org.example.Greeting'
Hi from Java!
Java plugin stop

Still, only the Java plugin runs. This took me some time to figure out why it doesn’t work - the .class files layout of the resulting .jar archives are the same. Decompiled classes, except for the Clojure stuff, look the same as Java classes.

The catch is, that when we’re building our Java plugin with mvn package, the extra file extensions.idx is created in the META-INF directory of the archive. Let’s look at this file:

# Generated by PF4J
org.example.GreetingsPlugin$Welcome

Aha! This file contains the necessary info to detect the @Extension annotated class in the jar archive and is generated by processing Java annotation during compilation. Unfortunately, the Clojure compiler doesn’t try to process our metadata as Java annotation, and hence no file is created. But we can add this file by hand, by creating META-INF/extensions.idx in the resources directory with the following contents:

clj_welcome_plugin.GreetingsPlugin$Welcome

Here, we’re indicating that our @Extension class is clj_welcome_plugin.GreetingsPlugin$Welcome. Let’s re-package our plugin with lein jar, and re-run our app:

$ cp path/to/clj-welcome-plugin/target/clj-welcome-plugin-0.1.0-SNAPSHOT.jar ./plugins/
$ lein run
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Clojure plugin start
Java plugin start
Found 2 extensions for extension point 'org.example.Greeting'
Hi from Clojure!
Hi from Java!
Java plugin stop
Clojure plugin stop

It works! Now we can make a more complex app, and extend it with plugins written both in Java and Clojure. Of course, more complex plugins will be bigger and have more code, but the general idea will be the same.

Some ending notes

You don’t have to put both the wrapper class and the Extension class in the same file. I mainly did this because it is in line with the Java code, so sources could be compared mostly directly. But a separate file, that sets the class name in a similar way is a more clean approach and should be preferred.

There’s also no need to rename most of the classes, except for the @Extension one, as we have to explicitly make it into a nested class via $ notation. Again, I’ve mainly done this to keep more or less the same code structure with the Java examples, for easier direct comparison.

Lastly, some automation may be needed to generate extensions.idx file based on source code. Unfortunately, I’ve not found any info on how to make Java annotation processing work with Clojure metadata annotations. So I’ve done this manually, but I’ve also written this script, that analyzes Clojure source code as EDN, and searches for given extension point class, printing class name from gen-class form if present:

#!/usr/bin/env bb

(require '[clojure.edn :as edn])
(require '[clojure.string :as str])

(defn analyze-gen-class
  "Tests if `form` is a `gen-class` and gets the `:name` or uses the
  `default` if `:name` is not set, but `:implements` matches the
  `query`.  Replaces all dashes with underscores in case the `default`
  name came from the `ns` form."
  ([form query]
   (analyze-gen-class form query nil))
  ([form query default]
   (when (and (list? form)
              (#{:gen-class 'gen-class} (first form))
              (some #{:implements} form))
     (let [gencl (->> form
                      (drop 1)
                      (partition 2)
                      (into {} (map vec)))]
       (when-let [name (and (= (:implements gencl) query)
                            (get gencl :name default))]
         (str/replace name #"-" "_"))))))

(defn analyze-ns
  "The `ns` form already contains class name as a namespace name,
  but we check inner `:gen-class` for a `:name` override."
  [form query]
  (let [name (second form)]
    (some #(analyze-gen-class % query name) form)))

(defn analyze-form
  "Check whether current `form` is a `gen-class` or `:gen-class` expression.
  If it is, check if gen-class has `:implements` key that is equal to
  `query` and gets the value of the `:name` key.  If `form` is an `ns`
  form finds `:gen-class` in it and checks it as above.  Recursively
  analyzes nested lists.  Terminates at first successful analysis."
  [query form]
  (when (list? form)
    (cond (= 'ns (first form)) (analyze-ns form query)
          (some list? form) (some #(analyze-form query %) form)
          :else (analyze-gen-class form query))))

(let [[query] *command-line-args*]
  (doseq [form *input*]
    (when-let [name (analyze-form
                     (edn/read-string query)
                     form)]
      (println name))))

This is a Babashka script, which analyzes S-expressions, and can be used together with another tool for parsing EDN - Jet (huge thanks to Michiel Borkent for awesome Clojure tooling!):

$ jet -c < src/clj_welcome_plugin/greetings_plugin.clj | ~/script.clj "[org.example.Greeting]"
clj_welcome_plugin.GreetingsPlugin$Welcome

Or, similarly for every .clj file in the project:

$ find -name '*.clj' -exec sh -c 'jet -c < "{}" | ~/script.clj "[org.example.Greeting]"' \;
clj_welcome_plugin.GreetingsPlugin$Welcome

For this project, the script finds the extension class name correctly from the source code we’ve given to it, but it is more a proof of concept, and by far is not a well-tested solution.

One particular point to improve is the fact that we have to pass "[org.example.Greeting]" to the script, to help it find gen-class entries that implement the interface we’re interested in. A much better solution would be to parse metadata, but unfortunately, I was not able to make Jet keep metadata in the resulting EDN, and I’m not sure how to properly read Clojure source code files otherwise. Although, it’s not really hard to check if all classes are present, as I highly doubt that there will be more than 5 extension classes presented in a single plugin, so this solution is good enough for now.

Other than that, the process of using the PF4J library is pretty straightforward, and we can fully utilize this library from Clojure projects with some hand tweaking. Thanks for reading!