k8s

Deploying with Kubernetes (k8s)

This directory contains a Kubernetes configuration file capable of provisioning a Data Prepper cluster.

Use kubectl to apply these configurations files with kubectl apply -f data-prepper-k8s.yaml

This will provision a scalable Data Prepper cluster capable of receiving trace data, however it is up to the user to decide how the service will be accessed by clients. A few examples include:

1. Adding a load balancer with an ingress controller (for server-side load balancing)
2. Exposing Kubernetes DNS so that clients can retrieve the addresses of Data Prepper pods (for client-side load balancing)
3. Using something like ExternalDNS to synchronize an external DNS (for client-side load balancing)

---

File Contents

kind: ConfigMap

Contains a ConfigMap which provides pods the 2 configuration files required to launch a Data Prepper instance: pipelines.yaml and data-prepper-config.yaml.

pipelines.yaml

This file contains the standard Trace Analytics usecase pipeline configuration. A few adjustments are needed by the user:

1. Replace the 2 stdout sinks with Elasticsearch sinks to actually send data to Elasticsearch.
2. Optionally enable TLS by providing the necessary key files for the otel_trace_source and peer_forwarder plugins.

data-prepper-config.yaml

This file disables TLS for the Data Prepper service APIs (e.g. /metrics), so please adjust this if you wish to enable encrpytion in transit.

kind: Service

Contains a headless Service which enables Data Prepper pods to discover others pods in the cluster peers via periodic DNS lookups. No changes are required.

kind: Deployment

Contains a Deployment which will create a single Data Prepper pod by default. No changes are required, however you can adjust the initial replica count if desired.