Load data from Kafka topics based on their (nested) schema into a relational model of Hana Cloud
Source code available here: github Docker image here: dockerhub
On any computer install the Docker Daemon - if it is not already - and download this docker image with
docker pull rtdi/hanacloudloader
Then start the image via docker run. For a quick test this command is sufficient
docker run -d -p 80:8080 --rm --name hanacloudloader rtdi/hanacloudloader
to expose a webserver at port 80 on the host running the container. Make sure to open the web page via the http prefix, as https needs more configuration. For example http://localhost:80/ might do the trick of the container is hosted on the same computer.
The default login for this startup method is: rtdi / rtdi!io
The probably better start command is to mount two host directories into the container. In this example the host's /data/files contains all files to be loaded into Kafka and the /data/config is an (initially) empty directory where all settings made when configuring the connector will be stored permanently.
docker run -d -p 80:8080 --rm -v /data/files:/data/ -v /data/config:/usr/local/tomcat/conf/security \
--name hanacloudloader rtdi/hanacloudloader
For proper start commands, especially https and security related, see the ConnectorRootApp project, this application is based on.
The first step is to connect the application to a Kafka server, in this example Confluent Cloud.
A Connection holds all information about the Hana cloud database connection. The database user should be a new user which has the permission to create new tables, which will be derived from the message schema.
Clicking on the Add icon allows opens the setting dialog
The endpoint for a SAP Hana Cloud instance is found in the SCP Cockpit, like here:
This is used in the Endpoint setting of the Connection.
One topic can contain multiple schemas and each schema might have nested data. The consumer creates for each schema the required tables. For example for the schema SalesOrder with the nested schema of name Items two tables will be created, one called SalesOrder, the other SalesOrder_Items.
This application is provided as dual license. For all users with less than 100'000 messages created per month, the application can be used free of charge and the code falls under a Gnu Public License. Users with more than 100'000 messages are asked to get a commercial license to support further development of this solution. The commercial license is on a monthly pay-per-use basis.
Every ten minutes the application does send the message statistics via a http call to a central server where the data is stored for information along with the public IP address (usually the IP address of the router). It is just a count which service was invoked how often, no information about endpoints, users, data or URL parameters. This information is collected to get an idea about the adoption. To disable that, set the environment variable HANAAPPCONTAINERSTATISTICS=FALSE.