Dev Cookbook

Tastylicious Quantum Leap recipes.

Dev env

We develop on MacOS but the various scripts we use to set up dev & test environment should (in principle!) work on most Linuxes too—well, they definitely work on Ubuntu Xenial which is the default Travis build environment where we build and test Quantum Leap. If you're into IDEs, try PyCharm for a smooth coding experience—you should be able to just define a project out of the repo and start coding right away. We use Docker/Docker Compose to build and test Quantum Leap images locally so you should install them—on the Mac you can try Docker Desktop which ships with both. To actually write Python code, you'll need Python 3.6 and pipenv

$ pip install pipenv

or, on MacOS, if you have Home Brew you can

$ brew install pipenv

From time to time, you may run into dependency hell and pipenv could start acting up, like

pkg_resources.DistributionNotFound:
The 'pipenv==2018.11.26' distribution was not found
and is required by the application

You'll have to reinstall it

$ brew reinstall pipenv

See this and this about it. (I wish Nix was more popular among Pythonistas.) After you've installed the dev tools, you're ready to hack away

$ git clone https://github.com/smartsdk/ngsi-timeseries-api.git
$ cd ngsi-timeseries-api
$ pipenv install
$ source setup_dev_env.sh
$ charm .

pipenv install installs all Python deps from our Pipfile in the root dir. You'll have to run this when starting from scratch or when a new lib gets added to the Pipfile. Also you should start your fave editor in a shell after sourcing our setup_dev_env.sh script which exports a whole bunch of vars you'll need to run and test Quantum Leap.

Testing

For the impatient, here's how to run our test suite if you're starting from scratch

$ cd where/you/cloned/ngsi-timeseries-api
$ pipenv install
$ source setup_dev_env.sh
$ sh run_tests.sh

Let's have a look at the details now. Test files go in the tests directories of the module you're testing, e.g. src/reporter/tests/. Our test framework is pytest. Each test file name should start with a test_ and so should test function names within the file. To run test cases, you'll have to prep your shell env, which you do by running the setup_dev_env.sh script in the repo root dir:

$ cd ngsi-timeseries-api/
$ source setup_dev_env.sh

Then you can run every test case known to man with:

$ sh run_tests.sh

This is painstakingly slow though (about 20 mins on my Mac) since most tests are integration or end to end tests (rather than unit!) which run in a containerised environment through docker compose. So you'd typically only do that before a commit to double check the new commit doesn't wreak havoc. To speed up development a notch, you could try running just the test cases in your module, e.g. those in src/reporter/tests/. Each test dir comes with a run_tests.sh script that's called by the main run_tests.sh in the root dir. These scripts build the QL container image, bring up the containerised env as spec'd by docker-compose.yml (in the same test dir) and finally start a pytest session to run the module's test cases. Long story short: to run e.g. reporter tests

$ cd src/reporter/tests/
$ sh run_tests.sh

If you have any unit tests, you can run them separately using your IDE or pytest directly from the shell. This way you won't have to twiddle your thumbs waiting on integration and e2e tests to cross the finishing line...

Debugging

Sometimes you really need a debugger to figure out what the heck Quantum Leap is up to. Here's one way to debug a Quantum Leap server process connected to the various back-ends—DB, cache, etc. First off, as usual, you'll need to prep your shell and start your IDE from there

$ cd ngsi-timeseries-api/
$ source setup_dev_env.sh
$ charm .

Now edit

src/reporter/tests/docker-compose.yml

to comment out the whole quantumleap service block. Then

$ cd src/reporter/tests
$ docker-compose up -d

This brings up a Docker network with all the back-end services Quantum Leap will connect to. (QL finds the services through the env vars that setup_dev_env.sh exports.) Wait a few secs to make sure all the services are up and running, then start debugging

src/app.py

Hammer away, e.g. curl a request or start another debug session with one of the reporter tests. When done, don't forget to clean up after yourself

$ docker-compose down -v