Feature 936 Standalone connection pool

Feature 936 Standalone connection pool

• Status: Design
• Author: Jianxiang Ran
• Co-author: Rocky Jin
• Discussion: https://github.com/lf-edge/ekuiper/issues/936

Requirement

Motivation

As data process middleware, eKuiper must exchange data with other entities. In eKuiper, we have Source and Sink to do the data input and output work. Both Source and Sink need think about data transport and authentication with other entities, for example eKuiper needs get raw data from mqtt broker, after processing, it needs send out the processed data to a HTTP Server. In this case, Source needs do the mqtt connection and authentication work, as for the Sink, it needs do the http connection and authentication work.

However, in some cases both Source and Sink are the same type, which means eKuiper can only set up a conneciton which Source and Sink can reuse. Here are two examples:

• mqtt source and mqtt sink across all rules may want to use the same mqtt connection
• There is no way for EdgeX sink to connect to secure redis. This could be a way to reuse the connection with source

Survey

At existing implement, Source and Sink set up and hold the client instances by themselvs. We can introduce a client-manager entity to maintain different client instances. Since the client instance are thread-safe, so can be used by different Sources and Sinks at the same time.

TODO add a picture here

Design

As multiple input/output middleware, eKuiper not only supports different protocols, like MQTT, EdgeX, but also supports connection to different endpointURLs under same connection type. For example, both Source and Sink use MQTT protocol but connect to different endpoints. The key to support this is user can create custom configuration under connection protocol, when create the Sink/Source, user can choose the configuration they write before. by this way, eKuiper knows how to set up the connection:

• Set up which kind client, MQTT or EdgeX
• Connect to which endpointURL

In order to share the connection, we can use client-manager entity to manage the connections. This manager is a key-value storage for different client instances: connection type and configuration key's combination as the key, client instances as values. The connection pool must have these capabilities:

• builder function to build different client types according to the connection type and configure key
• get the unique connection by the connection type and configure key
• when all Sink and Source have no reference to a specific connection, should close it

According to the user configuration, the Sink/Soure should decide whether use the share connection. When use share connection, need get connection from client-manager; otherwise, just init the connection by itself. client-manager should provide the following APIs:

        // called anytime when need get the client
	GetConnection(connectionType string, configKey strgin) (interface{}, error)

	// called when this source/sink exit 
	Close(connectionType string, configKey strgin) error

Example

Management

configure model

recommand configuration file layout

   etc/sources   
   etc/sinks