OTel Demo App on Nomad

Please note that at the time of this writing, all Metrics and Traces are being sent to Lightstep. Learn more about how to configure the OTel Collector on Nomad to send OTel data to Lightstep here.

For details on how to convert Kubernetes manifests to Nomad Jobspecs, check out my blog post here.

Gotchas

If you are using HashiQube, make sure that you allocate enough memory to Docker. I usually allocate 5 CPUs and 12GB RAM
Unlike Docker Compose, you cannot specify service dependencies in Nomad; however, the jobs are set up so that they will keep trying to restart if there's a service that they depend on that's not up.
Sometimes if a service keeps restarting (especially every minute or so), it's because it doesn't have enough memory allocated to it. This can also happen because it's waiting for a dependent service to start.

Deployment Steps

This assumes that you have HashiCorp Nomad, Consul, and Vault running somewhere. For a quick and easy local dev setup of the aforementioned tools, I highly recommend using HashiQube.

Add endpoints to /etc/hosts

**Only if you're doing local dev **

# For HashiQube
127.0.0.1   traefik.localhost
127.0.0.1   frontend.localhost
127.0.0.1   otel-demo.localhost
127.0.0.1   grafana.localhost
127.0.0.1   jaeger-ui.localhost
127.0.0.1   prometheus.localhost

Deploy Demo App services

First, set memory over-subscription per this article, to deal with any memory funny business from services. This is a one-time, cluster-wide setting.

nomad operator scheduler set-config -memory-oversubscription true

Now, deploy the services.

nomad job run -detach otel-demo-app/jobspec/traefik.nomad
nomad job run -detach otel-demo-app/jobspec/redis.nomad
nomad job run -detach otel-demo-app/jobspec/ffspostgres.nomad
nomad job run -detach otel-demo-app/jobspec/otel-collector.nomad
nomad job run -detach otel-demo-app/jobspec/adservice.nomad
nomad job run -detach otel-demo-app/jobspec/cartservice.nomad
nomad job run -detach otel-demo-app/jobspec/currencyservice.nomad
nomad job run -detach otel-demo-app/jobspec/emailservice.nomad
nomad job run -detach otel-demo-app/jobspec/featureflagservice.nomad
nomad job run -detach otel-demo-app/jobspec/paymentservice.nomad
nomad job run -detach otel-demo-app/jobspec/productcatalogservice.nomad
nomad job run -detach otel-demo-app/jobspec/quoteservice.nomad
nomad job run -detach otel-demo-app/jobspec/shippingservice.nomad
nomad job run -detach otel-demo-app/jobspec/checkoutservice.nomad
nomad job run -detach otel-demo-app/jobspec/recommendationservice.nomad
nomad job run -detach otel-demo-app/jobspec/frontend.nomad
nomad job run -detach otel-demo-app/jobspec/loadgenerator.nomad
nomad job run -detach otel-demo-app/jobspec/frontendproxy.nomad
nomad job run -detach otel-demo-app/jobspec/grafana.nomad
nomad job run -detach otel-demo-app/jobspec/jaeger.nomad
nomad job run -detach otel-demo-app/jobspec/prometheus.nomad

Webstore http://otel-demo.localhost/
Grafana http://otel-demo.localhost/grafana/ or http://grafana.localhost
Feature Flags UI http://otel-demo.localhost/feature/
Load Generator UI http://otel-demo.localhost/loadgen/
Jaeger UI http://otel-demo.localhost/jaeger/ui/ or http://jaeger-ui.localhost
Prometheus UI http://prometheus.localhost

Send Traces to Lightstep

By default, the OTel Demo App’s OpenTelemetry Collector is configured to send Traces and Metrics to Jaeger, and Prometheus, respectively. For this demo, I also configured the Collector to send Traces and Metrics to Lightstep.

If you’d like to send Traces and Metrics to Lightstep, you’ll need to do the following:

Get a Lightstep Access Token. (Make sure that you sign up for a Lightstep account first, if you don’t already have one.)
Configure Vault by following the instructions here.
Add your Lightstep Access Token to Vault by running the command:

vault kv put kv/otel/o11y/lightstep ls_token="<LS_TOKEN>"

Where <LS_TOKEN> is your Lightstep Access Token

The OTel Collector job pulls this value from Vault, into the Collector’s config YAML, so that we can also send Traces and Metrics to Lightstep:

otlp/ls:
  endpoint: ingest.lightstep.com:443
  headers: 
    "lightstep-access-token": "{{ with secret "kv/data/otel/o11y/lightstep" }}{{ .Data.data.ls_token }}{{ end }}"

Run the version of the OTel Collector jobspec that contains the Lightstep configurations by replacing nomad job run -detach otel-demo-app/jobspec/otel-collector.nomad with nomad job run -detach otel-demo-app/jobspec/otel-collector-with-LS.nomad

Nuke deployments

nomad job stop -purge traefik
nomad job stop -purge redis
nomad job stop -purge ffspostgres
nomad job stop -purge otel-collector
nomad job stop -purge adservice
nomad job stop -purge cartservice
nomad job stop -purge currencyservice
nomad job stop -purge emailservice
nomad job stop -purge featureflagservice
nomad job stop -purge paymentservice
nomad job stop -purge productcatalogservice
nomad job stop -purge quoteservice
nomad job stop -purge shippingservice
nomad job stop -purge checkoutservice
nomad job stop -purge recommendationservice
nomad job stop -purge frontend
nomad job stop -purge frontendproxy
nomad job stop -purge loadgenerator
nomad job stop -purge grafana
nomad job stop -purge jaeger
nomad job stop -purge prometheus

Running Tracetest for OTel Demo App

Deploy

    nomad operator scheduler set-config -memory-oversubscription true

    nomad job run -detach tracetest/jobspec/traefik.nomad
    nomad job run -detach tracetest/jobspec/postgres.nomad
    nomad job run -detach tracetest/jobspec/tracetest.nomad
    nomad job run -detach tracetest/jobspec/otel-collector.nomad
    nomad job run -detach tracetest/jobspec/go-server.nomad
    nomad job run -detach otel-demo-app/jobspec/redis.nomad
    nomad job run -detach otel-demo-app/jobspec/ffspostgres.nomad

    nomad job run -detach otel-demo-app/jobspec/adservice.nomad
    nomad job run -detach otel-demo-app/jobspec/cartservice.nomad
    nomad job run -detach otel-demo-app/jobspec/currencyservice.nomad
    nomad job run -detach otel-demo-app/jobspec/emailservice.nomad
    nomad job run -detach otel-demo-app/jobspec/featureflagservice.nomad
    nomad job run -detach otel-demo-app/jobspec/paymentservice.nomad
    nomad job run -detach otel-demo-app/jobspec/productcatalogservice.nomad
    nomad job run -detach otel-demo-app/jobspec/quoteservice.nomad
    nomad job run -detach otel-demo-app/jobspec/shippingservice.nomad
    nomad job run -detach otel-demo-app/jobspec/checkoutservice.nomad
    nomad job run -detach otel-demo-app/jobspec/recommendationservice.nomad
    nomad job run -detach otel-demo-app/jobspec/frontend.nomad
    nomad job run -detach otel-demo-app/jobspec/loadgenerator.nomad
    nomad job run -detach otel-demo-app/jobspec/frontendproxy.nomad
    nomad job run -detach otel-demo-app/jobspec/grafana.nomad
    nomad job run -detach otel-demo-app/jobspec/jaeger.nomad
    nomad job run -detach otel-demo-app/jobspec/prometheus.nomad

Run tests

Cart Service test

tracetest test run --definition tracetest/tests/cart-service-test.yml

Sample output:

✔ CartService gRPC Test (http://tracetest.localhost/test/avtest123/run/3/test)

Recommendation Service test

tracetest test run --definition tracetest/tests/recommendation-service-test.yml

Sample output:

✔ RecommendationService test (http://tracetest.localhost/test/avtest456/run/4/test)

Note that in order to get these to run, I had to set static ports for both recommendationservice.nomad and cartservice.nomad.

Nukify

nomad job stop -purge traefik
nomad job stop -purge postgres
nomad job stop -purge tracetest
nomad job stop -purge otel-collector
nomad job stop -purge go-server
nomad job stop -purge redis
nomad job stop -purge ffspostgres

nomad job stop -purge adservice
nomad job stop -purge cartservice
nomad job stop -purge currencyservice
nomad job stop -purge emailservice
nomad job stop -purge featureflagservice
nomad job stop -purge paymentservice
nomad job stop -purge productcatalogservice
nomad job stop -purge quoteservice
nomad job stop -purge shippingservice
nomad job stop -purge checkoutservice
nomad job stop -purge recommendationservice
nomad job stop -purge frontend
nomad job stop -purge loadgenerator
nomad job stop -purge frontendproxy
nomad job stop -purge grafana
nomad job stop -purge jaeger
nomad job stop -purge prometheus

Notes

Communication Between Services

In order for services to communicate between each other, you need to use Consul templating. For example:

template {
  data = <<EOF
{{ range service "redis-service" }}
REDIS_ADDR = "{{ .Address }}:{{ .Port }}"
{{ end }}

{{ range service "otelcol-grpc" }}
OTEL_EXPORTER_OTLP_ENDPOINT = "http://{{ .Address }}:{{ .Port }}"
{{ end }}

EOF
  destination = "local/env"
  env         = true
}

This pulls the IP and port of a service based on its Consul name, and sets it ato an environment variable.

See reference here.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

OTel Demo App on Nomad

Gotchas

Deployment Steps

Send Traces to Lightstep

Nuke deployments

Running Tracetest for OTel Demo App

Notes

Communication Between Services

Files

README.md

Latest commit

History

README.md

File metadata and controls

OTel Demo App on Nomad

Gotchas

Deployment Steps

Send Traces to Lightstep

Nuke deployments

Running Tracetest for OTel Demo App

Notes

Communication Between Services