Go couldn't load code dynamically prior to Go 1.8. I'm a big proponent of plugin-based systems, which in many cases require loading plugins dynamically. I even considered at some point writing a plugin package based on C integration.
I'm super excited that the Go designers added this capability to the language. In this tutorial, you'll learn why plugins are so important, what platforms are currently supported, and how to create, build, load and use plugins in your programs.
The Rationale for Go Plugins
Go plugins can be used for many purposes. They let you break down your system into a generic engine that's easy to reason about and test, and a lot of plugins adhere to a strict interface with well-defined responsibilities. Plugins can be developed independently from the main program that uses them.
The program can use different combinations of plugins and even multiple versions of the same plugin at the same time. The crisp boundaries between the main program and plugins promote the best practice of loose coupling and separation of concerns.
The "plugin" Package
The new "plugin" package introduced in Go 1.8 has a very narrow scope and interface. It provides the Open() function to load a shared library, which returns a Plugin object. The Plugin object has a Lookup() function that returns a Symbol (empty interface{}) hat can be type asserted to a function or variable exposed by the plugin. That's it.
Platform Support
The plugin package is supported only on Linux at this time. But there are ways, as you'll see, to play with plugins on any operating system.
Preparing a Docker-Based Environment
If you're developing on a Linux box then you just need to install Go 1.8 and you're good to go. But, if you're on Windows or macOS, you need a Linux VM or Docker container. To use it, you must first install Docker.
Once you have Docker installed, open a console Window and type: docker run -it -v ~/go:/go golang:1.8-wheezy bash
This command maps my local $GOPATH at ~/go to /go inside the container. That lets me edit the code using my favorite tools on the host and have it available inside the container for building and running in the Linux environment.
For more information on Docker, check out my "Docker From the Ground Up" series here on Envato Tuts+:
- Docker From the Ground Up: Understanding Images
- Docker From the Ground Up: Building Images
- Docker From the Ground Up: Working With Containers, Part 1
- Docker From the Ground Up: Working With Containers, Part 2
Creating a Go Plugin
A Go plugin looks like a regular package, and you can use it as a regular package too. It becomes a plugin only when you build it as a plugin. Here are a couple of plugins that implement a Sort() function that sorts a slice of integers.
QuickSort Plugin
The first plugin implements a naive QuickSort algorithm. The implementation works on slices with unique elements or with duplicates. The return value is a pointer to a slice of integers. This is useful for sort functions that sort their elements in place because it allows returning without copying.
In this case, I actually create multiple interim slices, so the effect is mostly wasted. I sacrifice performance for readability here since the goal is to demonstrate plugins and not implement a super efficient algorithm. The logic goes as follows:
- If there are zero items or one item, return the original slice (already sorted).
- Pick a random element as a peg.
- Add all the elements that are less than the peg to the below slice.
- Add all the elements that are greater than the peg to the above slice.
- Add all the elements that are equal to the peg to the middle slice.
At this point, the middle slice is sorted because all its elements are equal (if there were duplicates of the peg, there will be multiple elements here). Now comes the recursive part. It sorts the below and above slices by calling Sort() again. When those calls return, all the slices will be sorted. Then, simply appending them results in a full sort of the original slice of items.
package main
import "math/rand"
func Sort(items []int) *[]int {
if len(items) < 2 {
return &items
}
peg := items[rand.Intn(len(items))]
below := make([]int, 0, len(items))
above := make([]int, 0, len(items))
middle := make([]int, 0, len(items))
for _, item := range items {
switch {
case item < peg:
below = append(below, item)
case item == peg:
middle = append(middle, item)
case item > peg:
above = append(above, item)
}
}
below = *Sort(below)
above = *Sort(above)
sorted := append(append(below, middle...), above...)
return &sorted
}
BubbleSort Plugin
The second plugin implements the BubbleSort algorithm in a naive way. BubbleSort is often considered slow, but for a small number of elements and with some minor optimization it often beats more sophisticated algorithms like QuickSort.
It is actually common to use a hybrid sort algorithm that starts with QuickSort, and when the recursion gets to small enough arrays the algorithm switches to BubbleSort. The bubble sort plugin implements a Sort() function with the same signature as the quick sort algorithm. The logic goes as follows:
- If there are zero items or one item, return the original slice (already sorted).
- Iterate over all the elements.
- In each iteration, iterate over the rest of the items.
- Swap the current item with any item that is greater.
- At the end of each iteration, the current item will be in its proper place.
package main
func Sort(items []int) *[]int {
if len(items) < 2 {
return &items
}
tmp := 0
for i := 0; i < len(items); i++ {
for j := 0; j < len(items)-1; j++ {
if items[j] > items[j+1] {
tmp = items[j]
items[j] = items[j+1]
items[j+1] = tmp
}
}
}
return &items
}
Building the Plugin
Now, we have two plugins we need to build to create a shareable library that can be loaded dynamically by our main program. The command to build is: go build -buildmode=plugin
Since we have multiple plugins, I placed each one in a separate directory under a shared "plugins" directory. Here is the directory layout of the plugins directory. In each plugin subdirectory, there is the source file "<algorithm>_plugin.go" and a little shell script "build.sh" to build the plugin. The final .so files go into the parent "plugins" directory:
$ tree plugins plugins ├── bubble_sort │ ├── bubble_sort_plugin.go │ └── build.sh ├── bubble_sort_plugin.so ├── quick_sort │ ├── build.sh │ └── quick_sort_plugin.go └── quick_sort_plugin.so
The reason the *.so files go into the plugins directory is that they can be discovered easily by the main program, as you'll see later. The actual build command in each "build.sh" script specifies that the output file should go into the parent directory. For example, for the bubble sort plugin it is:
go build -buildmode=plugin -o ../bubble_sort_plugin.so
Loading the Plugin
Loading the plugin requires knowledge of where to locate the target plugins (the *.so shared libraries). This can be done in various ways:
- passing command-line arguments
- setting an environment variable
- using a well-known directory
- using a configuration file
Another concern is if the main program knows the plugin names or if it discovers dynamically all the plugins in a certain directory. In the following example, the program expects that there will be a sub-directory called "plugins" under the current working directory, and it loads all the plugins it finds.
The call to the filepath.Glob("plugins/*.so") function returns all the files with the ".so" extension in the plugins sub-directory, and plugin.Open(filename) loads the plugin. If anything goes wrong, the program panics.
package main
import (
"fmt"
"plugin"
"path/filepath"
)
func main() {
all_plugins, err := filepath.Glob("plugins/*.so")
if err != nil {
panic(err)
}
for _, filename := range (all_plugins) {
fmt.Println(filename)
p, err := plugin.Open(filename)
if err != nil {
panic(err)
}
}
}
Using the Plugin in a Program
Locating and loading the plugin is only half the battle. The plugin object provides the Lookup() method that given a symbol name returns an interface. You need to type assert that interface into a concrete object (e.g. a function like Sort()). There is no way to discover what symbols are available. You just have to know their names and their type, so you can type assert properly.
When the symbol is a function, you can invoke it like any other function after a successful type assert. The following example program demonstrates all these concepts. It dynamically loads all the available plugins without knowing which plugins are there except that they are in the "plugins" sub-directory. It follows by looking up the "Sort" symbol in each plugin and type asserting it into a function with the signature func([]int) *[]int. Then, for each plugin, it invokes the sort function with a slice of integers and prints the result.
package main
import (
"fmt"
"plugin"
"path/filepath"
)
func main() {
numbers := []int{5, 2, 7, 6, 1, 3, 4, 8}
// The plugins (the *.so files) must be in a 'plugins' sub-directory
all_plugins, err := filepath.Glob("plugins/*.so")
if err != nil {
panic(err)
}
for _, filename := range (all_plugins) {
p, err := plugin.Open(filename)
if err != nil {
panic(err)
}
symbol, err := p.Lookup("Sort")
if err != nil {
panic(err)
}
sortFunc, ok = symbol.(func([]int) *[]int)
if !ok {
panic("Plugin has no 'Sort([]int) []int' function")
}
sorted := sortFunc(numbers)
fmt.Println(filename, sorted)
}
}
Output:
plugins/bubble_sort_plugin.so &[1 2 3 4 5 6 7 8]
plugins/quick_sort_plugin.so &[1 2 3 4 5 6 7 8]
Conclusion
The "plugin" package provides a great foundation for writing sophisticated Go programs that can dynamically load plugins as necessary. The programming interface is very simple and requires detailed knowledge of the using program on the plugin interface.
It is possible to build a more advanced and user-friendly plugin framework on top of the "plugin" package. Hopefully, it will be ported to all the platforms soon. If you deploy your systems on Linux, consider using plugins to make your programs more flexible and extensible.
No comments:
Post a Comment