Conduit v0.3.0 was recently released and brought lots of useful features that make the user as well as the developer experience nicer and simpler. One of these features is connector middleware in the connector SDK. In this blog post we will explain what middleware is, why we added it, how it solves our problems and how to use it yourself.
The problem we faced
Before we dive into middleware, let’s first give you some context around Conduit and the problem we faced.
Conduit is a data integration tool that uses connectors to fetch data from and write data to third-party systems. A connector is a plugin that runs in its own process and follows the prescribed connector protocol. We use protocol buffers and gRPC to define the interface used in the connector protocol. On one hand, this gives us the flexibility to write connectors in any programming language, but on the other hand, it requires the connector developer to deal with the complexity of gRPC streams and write a lot of boilerplate code themselves. Because we want to make the developer experience better and standardize the behavior of connectors as much as possible we provide a connector SDK for connectors written in Go. The SDK hides the complexity, implements common boilerplate code, provides utilities for implementing a connector, and allows the developer to focus on writing the connector functionality without worrying about the protocol.
After implementing more than 25 connectors it became clear that there was still room for improvement in terms of reducing duplicated code found in multiple connectors. We saw repeated code in some connectors that needed the same functionality, like rate limiting or batching. The problem we faced is that these features are not applicable for all connectors so we can’t bake them into the SDK and enable them for all connectors. Furthermore, even if connectors require the same functionality, they may expect different default values to configure the functionality (e.g. default batch size).
To solve this problem, we came up with the following requirements:
- We want to be able to add features that are needed across all connectors (e.g. batching).
- These added features need to be configurable by the end-user.
- Connector developers should be in control of adding or opting out of a feature in their connector (no hidden logic).
- There should be a default set of features, so we can add more in the future and easily roll them out to all connectors.
- Connector developers should be able to choose the defaults for these features in their connector.
Fulfilling these requirements will bring many benefits - it would further standardize the behavior of connectors, cut down on code duplication, and in the long run, it will help us reduce the number of bugs and making it easier to maintain our connectors.
Middleware
As soon as we had a clear list of requirements a lightbulb went off in our heads - we need to introduce a middleware!
What is middleware, you ask? Different people understand different things under the term. Some may think of OS middleware that expands the functionality of an operating system, others might think of middleware as services in the context of distributed applications. Regardless of the specific middleware you think of, one thing is true for all: as the name suggests, it’s a piece of software that sits in the middle of two components and provides additional functionality. You can imagine middleware like augmented reality glasses - they allow the wearer to see and interact with their environment as before while providing additional information on top.
In this post we use the term middleware to describe a piece of code that functions like a wrapper around an object and forwards calls to the underlying object while manipulating the parameters and/or return values. It’s common that the underlying object implements a certain interface so that the middleware does not have to be aware of what specific object it is wrapping. The middleware in turn also implements the same interface, so that a wrapped object can still be used through that interface.
Perhaps the most common use of the middleware pattern in Go are HTTP handlers. It’s common practice to wrap http.Handler objects with middleware that adds functionality like logging or authentication. Here’s an example of HTTP middleware in Go:
package main
import (
"log"
"net/http"
)
func loggingMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
log.Print("received request")
next.ServeHTTP(w, r)
log.Print("send response")
})
}
func hello(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("hello"))
}
func main() {
handler := http.HandlerFunc(hello)
err := http.ListenAndServe(":8080", loggingMiddleware(handler))
log.Fatal(err)
}
Notice that loggingMiddleware is unaware of what http.Handler it is wrapping. Since the middleware itself is an http.Handler it can even wrap another middleware (chaining middleware is also common practice). The base functionality of the HTTP handler is still the same, the middleware forwards the call while executing some operations before and after.
How Connector Middleware solves our problem
The middleware pattern checks all the boxes of our requirements list. Let’s go through them one by one.
We want to be able to add features that are needed across all connectors.
This is exactly what middleware does; it adds additional functionality without changing the basic functionality. It can be applied to any object that implements a certain interface, in our case the interfaces are Source and Destination.
These added features need to be configurable by the end-user.
The Source and Destination interfaces are in control of defining how the connector can be configured. The middleware can wrap the function Parameters to adjust the specifications and tell the UI to display additional parameters. When the user creates the connector, the configuration is passed to the function Config, which can again be wrapped by the middleware to parse the injected parameters.
Connector developers should be in control of adding or opting out of a feature in their connector.
We were already using a constructor function in our connectors which is the perfect place for adding middleware. The constructor is implemented by the connector developer so they can choose to add any middleware they want. Note that we encourage developers to add at least the default middleware unless they have a good reason not to do so.
There should be a default set of features, so we can add more in the future and easily roll them out to all connectors.
The SDK provides functions that return the default connector middleware (DefaultSourceMiddleware and DefaultDestinationMiddleware). Developers are encouraged to add the default middleware to their connectors unless they have a good reason not to do so. All connectors that will use the default middleware will automatically benefit from new middleware that gets added in future SDK releases. This will ensure that we can further standardize the behavior of our connectors and easily roll out common features.
Connector developers should be able to choose the defaults for these features in their connector.
We solved this by implementing middleware as structs with public fields that contain the default values for the parameters it introduces. The connector developer can choose the default values when adding middleware to their connector.
Example usage
Here we will show how easy it is to apply middleware on connectors. We will focus on the Destination, although the same principles apply when implementing a Source.
We start with a simple destination struct and a constructor function.
type Destination struct {
sdk.UnimplementedDestination
}
func NewDestination() sdk.Destination {
// return an instance of Destination
return &Destination{}
}
To add the middleware to the destination the SDK provides a utility function called DestinationWithMiddleware. The SDK also provides the function DefaultDestinationMiddleware which returns a set of default middleware and should be used in most connectors. In future SDK releases we may add more middleware to the set, this way most connectors will benefit from new middleware simply by updating the SDK version.
type Destination struct {
sdk.UnimplementedDestination
}
func NewDestination() sdk.Destination {
// return an instance of Destination wrapped in the default middleware
destination := &Destination{}
middleware := sdk.DefaultDestinationMiddleware()
return sdk.DestinationWithMiddleware(destination, middleware...)
}
If there is a good reason not to use the default middleware (e.g. choose different defaults or remove a middleware) the developer can freely choose which middleware to apply. For example, this is how we would apply only the batching middleware and set a default batch size of 100.
type Destination struct {
sdk.UnimplementedDestination
}
func NewDestination() sdk.Destination {
// return an instance of Destination wrapped in custom middleware
destination := &Destination{}
middleware := []sdk.DestinationMiddleware{
sdk.DestinationWithBatch{ DefaultBatchSize: 100 },
}
return sdk.DestinationWithMiddleware(destination, middleware...)
}
Conclusion
With the introduction of a connector middleware we intend to make the connector developer experience even nicer and simpler. Connector developers can utilize middleware provided by the SDK to enrich the functionality of their connectors without reinventing the wheel. Even Conduit users will benefit from the middleware, as the functionality provided by a middleware will work the same way across all connectors.
If this got you interested in Conduit don’t hesitate to join our Discord and say hello! We invite you to give Conduit a try and let us know what you like and don’t like. Our mission is to make Conduit the go-to tool for data integration and your feedback can help us reach that goal!