diff --git a/versioned_docs/version-1.1.0/.version b/versioned_docs/version-1.1.0/.version new file mode 100644 index 0000000..1cc5f65 --- /dev/null +++ b/versioned_docs/version-1.1.0/.version @@ -0,0 +1 @@ +1.1.0 \ No newline at end of file diff --git a/versioned_docs/version-1.1.0/contribute/build.md b/versioned_docs/version-1.1.0/contribute/build.md new file mode 100644 index 0000000..923bbf0 --- /dev/null +++ b/versioned_docs/version-1.1.0/contribute/build.md @@ -0,0 +1,217 @@ +--- +title: Build +tags: [Contribute, Build] +description: Build %%te%% from the source code +--- + +This page details how to build %%te%% from the source code. + +## Requirements + +You can use any OS to build from source (below has been tested on Ubuntu, but we also use Debian, macOS, and FreeBSD successfully). + +Our recommended setup and required tools are: + +* Ubuntu 20.04 or Debian 12 (Bookworm) +* git +* Rust toolchain + +Following packages are required: + +* build-essentials +* curl +* gcc + +A list of our test platforms can be found [here](../references/supported-platforms.md). + +## Get the code + +%%te%% code is in git repository on github to acquire the code use following command: + +* via SSH: + +```sh +git clone git@github.com:thin-edge/thin-edge.io.git +``` + +* or via HTTPS: + +```sh +git clone https://github.com/thin-edge/thin-edge.io.git +``` + +## Installing toolchain + +### Rust toolchain + +To install Rust follow [Official installation guide](https://www.rust-lang.org/tools/install). +To get started you need Cargo's bin directory (`$HOME/.cargo/bin`) in your `PATH` environment variable. + +```sh +export PATH=$PATH:$HOME/.cargo/bin +``` + +And then you can run `rustc` to view current version: + +```sh +rustc --version +``` + +```text title="Output" +rustc 1.65.0 (897e37553 2022-11-02) +``` + +:::note +Above command will add rust to path only for existing session, +after you restart the session you will have to add it again, +to add rust to the path permanently it will depend on your shell but for Bash, +you simply need to add the line from above, `export PATH=$PATH:$HOME/.cargo/bin` to your `~/.bashrc`. + +For other shells, you'll want to find the appropriate place to set a configuration at start time, +eg. zsh uses `~/.zshrc`. Check your shell's documentation to find what file it uses. +::: + +%%te%% operates the `MSRV` (Minimum Supported Rust Version) and uses stable toolchain. + +Current MSRV is `1.65`. + +### Cross compilation toolchain (optional) + +%%te%% can be compiled for target architecture on non-target device, this is called cross compilation. +Currently we support `Raspberry Pi 3B` for `armv7` architecture with Rust's cross compilation toolchain called [cargo cross](https://github.com/rust-embedded/cross). + +To install [cargo cross](https://github.com/rust-embedded/cross): + +```sh +cargo install cross +``` + +### Linux packaging (optional) + +We use [nfpm](https://github.com/goreleaser/nfpm) to build our linux packages (deb, rpm and apk). + +Follow the [nfpm install instructions](https://nfpm.goreleaser.com/install/) to install the dependency. The linux packages will automatically be built when running `just release`. + + +```sh +just release +``` + +The virtual packages (e.g. `tedge-full` and `tedge-minimal`) can be built using the following command: + +```sh +just release-linux-virtual +``` + +## Compiling + +To build %%te%% we are using `cargo`. + +As we are using `cargo workspace` for all our crates. All compiled files are put in `./target/` directory with target's name eg: `./target/debug` or `./target/release` for native builds and for cross compiled targets `./target//debug` or `./target//release` dependent on the target of the build. + +### Compiling dev + +To compile dev profile (with debug symbols) we use following command: + +```sh +cargo build +``` + +Build artifacts can be found in `./target/debug` and will include executables: + +```sh +ls -l ./target/debug/tedge* +``` + +```text title="Output" +-rwxrwxr-x 2 user user 11111 Jan 1 00:00 tedge +-rwxrwxr-x 2 user user 11111 Jan 1 00:00 tedge-mapper +``` + +Binaries can be run eg: `./target/debug/tedge`. +Alternatively, you can use `cargo` to build and run executable in a single command: + +```sh +cargo run --bin tedge +``` + +### Compiling release + +To compile release profile we use following command: + +```sh +cargo build --release +``` + +Build artifacts can be found in `./target/release` and will include executables: + +```sh +ls -l ./target/release/tedge* +``` + +```text title="Output" +-rwxrwxr-x 2 user user 11111 Jan 1 00:00 tedge +-rwxrwxr-x 2 user user 11111 Jan 1 00:00 tedge-mapper +``` + +Binaries can be run eg: `./target/release/tedge`. + +## Building deb package + +Currently %%te%% contains 2 binaries, `tedge` (cli) and `tedge-mapper` which are packaged as separate debian packages. To create following commands are to be issued: + +```sh +cargo deb -p tedge +cargo deb -p tedge-mapper +``` + +All resulting packages are going to be in: `./target/debian/` directory: + +```sh +ls -l ./target/debian +``` + +```text title="Output" +total 2948 +-rw-rw-r-- 1 user user 11111 Jan 1 00:00 tedge_0.9.0_amd64.deb +-rw-rw-r-- 1 user user 11111 Jan 1 00:00 tedge-mapper_0.9.0_amd64.deb +``` + +## Cross compiling + +To create binaries which can run on different platform than one you are currently on you can use [cargo cross](https://github.com/rust-embedded/cross): + +```sh +cross build --target armv7-unknown-linux-gnueabihf +``` + +Build artifacts can be found in `./target/armv7-unknown-linux-gnueabihf/debug` and will include executables: + +```sh +ls -l ./target/armv7-unknown-linux-gnueabihf/debug/tedge* +``` + +```text title="Output" +-rwxrwxr-x 2 user user 11111 Jan 1 00:00 tedge +-rwxrwxr-x 2 user user 11111 Jan 1 00:00 tedge-mapper +``` + +To cross compile release version of the binaries just add `--release` to the above command like so: + +```sh +cross build --target armv7-unknown-linux-gnueabihf --release +``` + +## Running tests + +When contributing to %%te%% we ask you to write tests for the code you have written. The tests will be run by build pipeline when you create pull request, but you can easily run all the tests when you are developing with following command: + +```sh +cargo test +``` + +This will run all tests from the repository and sometime may take long time, `cargo` allows you to run specific test or set of tests for binary: + +```sh +cargo test --bin tedge +``` diff --git a/versioned_docs/version-1.1.0/contribute/design/index.md b/versioned_docs/version-1.1.0/contribute/design/index.md new file mode 100644 index 0000000..4932aa5 --- /dev/null +++ b/versioned_docs/version-1.1.0/contribute/design/index.md @@ -0,0 +1,10 @@ +--- +title: Design +tags: [Documentation, next] +--- + +import DocCardList from '@theme/DocCardList'; + +The following pages describe important design elements of %%te%%. + + diff --git a/versioned_docs/version-1.1.0/contribute/design/mqtt-topic-design.md b/versioned_docs/version-1.1.0/contribute/design/mqtt-topic-design.md new file mode 100644 index 0000000..9b3ca4a --- /dev/null +++ b/versioned_docs/version-1.1.0/contribute/design/mqtt-topic-design.md @@ -0,0 +1,172 @@ +--- +title: MQTT Topic Design +tags: [Documentation] +sidebar_position: 2 +--- + +A detailed look into the MQTT topic design giving additional context which the reference page does not provide. It is assumed that you are familiar with the [MQTT Topic Schema](../../references/mqtt-api.md) page, if not, then please read it first before continuing. + +## Main concepts + +### Identifier - fixed 4 segment group + +The identifier is used to group command and telemetry data to a given entity: a device, a service or any source or target of MQTT messages. + +Having a fixed **identifier** group in the topic structure makes it easier for other clients to subscribe to the messages being published on the MQTT broker, as well as making parsing the topics simple (e.g. splitting the string by slash (`/`) and taking the first 5 items (including the root segment)). + +A client can subscribe to all measurements for the main device by subscribing to a single topic. + +```sh te2mqtt formats=v1 +tedge mqtt sub 'te/device/main///m/+' +``` + +Alternatively, if a client wants to subscribe to all measurements regardless of the **identifier** then it can also can be done using a single subscription, as demonstrated by the following example: + +```sh te2mqtt formats=v1 +tedge mqtt sub 'te/+/+/+/+/m/+' +``` + +If a variable number of segments were used for the **identifier**, it would result in each client having to subscribe to multiple MQTT topics, which makes it both cumbersome and error prone. + +```sh title="Invalid example" +mosquitto_sub \ + -t 'te/+/m/+' \ + -t 'te/+/+/m/+' \ + -t 'te/+/+/+/m/+' \ + -t 'te/+/+/+/+/m/+' +``` + +### Support auto registration via inference + +Automatic registration was a commonly used feature in the legacy MQTT topic (e.g. topics under `tedge/#`), so in order to keep supporting such as feature, a simple convention is applied to the 4-segment **identifier** to provide all of the information required for auto registration to work. For instance the **identifier** should contains whether it represents the main device, a child device, or a service hosted either on a main or child device. + +It must be noted that even though the `tedge/#` does support auto registration, there are limitations which the new `te/#` topic schema remove. + +The main limitation of the `tedge/#` auto registration was the assumption that publishing telemetry data was always associated to either the main or child device. For example, if measurements for a service were published to the `tedge/measurements/` topic before the service had published to the `tedge/health/[/]` topic, then the service would be represented as a child device instead of a service. + +#### Solution + +As mentioned in the MQTT referenced guide, the **identifier** group of the topic schema is used to define the target device/service, and it is illustrated by the following diagram. + +

+ +```mermaid +graph LR + device --/--- device_id["<device_id>"] --/--- service --/--- service_id["<service_id>"] +``` + +

+ +The above topic schema uses fixed positions to indicate the entity type, e.g. to represent if the entity is a device or a service on a device. + +:::note +A service represents an entity that has dependence on a device. For example a service is running on device, and therefore can not exist on its own. The lifecycle of the service is linked to the device it is running on, so when the device is removed, then the service is also removed. +::: + +The `` can be either set to `main`, which represents the %%te%% device, or to a id/name used to represent a child device. Telemetry data targeted to the main or child devices, require the `service` and `` segments to be set to an empty string. + +The following table shows some examples showing the relationship between the topic and the entity/component. + +|Topic identifier (with root prefix)|Entity/Component| +|--|--| +|`te/device/main///`|Main device| +|`te/device/main/service/nodered`|Service called "nodered" running on the main device| +|`te/device/fooBar///`|Child device called "fooBar". The device is an immediate child of the main device| +|`te/device/fooBar/service/nodered`|Service called "nodered" running on a child device called "fooBar"| + + + +### Support decoupling topic and device hierarchy + +Another key design choice was to support decoupling the topic hierarchy to the device/service hierarchy. Decoupling the topic structure and hierarchy allows you to change the local MQTT topic structure independently from the digital twin representation in the cloud, as the two environments (cloud and edge) can have very different requirements when it comes to data filtering and visualization. + +Decoupling via the registration interface which allows different entities to declare their own entity type (e.g. device, child-device or service) and an optional parent entity. Check out the [Using custom identifier schemas](#using-custom-identifier-schemas) section for an example showing the decoupling of topic and device hierarchy. + +## Using custom identifier schemas + +:::note Preface +Custom **identifier** schemas are only intended to be used for complex scenarios where the typical (default) MQTT topic schema is not suitable. Having too much freedom in the topic structure can cause confusion amongst users, so it was decided to use the typical use-case to a single simplistic view on the topic structure which is suitable for the majority of use-cases. +::: + +This section highlights the flexible nature of the MQTT topic schema which allows you to provide your own semantic meaning to each of the **identifier** segments within the topic schema. + +Let's start off by taking a closer look at the typical topic structure. The typical use-case defines the **identifier** (including root prefix) as follows: + +```mermaid +graph LR + te --/--- segment1["device"] --/--- segment2["<device_id>"] --/--- segment3["service"] --/--- segment4["<service_id>"] +``` + +But what was not mentioned is that the two static segments, "device" and "service" are actually only used to provide the auto registration functionality, as the cloud mappers need to know how to map the topic to the specific digital twin entity (e.g. as a device, child device or service). The typical topic schema allows the mapper to infer this relationship purely form the topic (which is ok for the majority of use-cases). However, when you take away the auto registration functionality (e.g. disable it), it allows you to apply your own semantics to each of the 4-segment **identifier** providing more powerful modelling mechanisms. + +The **identifier** topic schema can be rewritten in its most generic form. + +```mermaid +graph LR + te --/--- segment1["<segment1>"] --/--- segment2["<segment2>"] --/--- segment3["<segment3>"] --/--- segment4["<segment4>"] +``` + +With the generic form, the structure of the topics is still the same as before, 4 segments, however you can apply your own meaning to each of the segment by registering the entities (devices/services) at each of the specific segments. The mapper then matches the **identifier** used when publishing telemetry data or receiving commands, to the registered entity. + +This allows you to also encode specific information about the entity in the MQTT topic enabling powerful (local) filtering possibilities for other MQTT clients also running on the edge. + +Below is a simple example showing how the **identifier** can be modelled to represent the location of equipment in a factory. The custom **identifier** schema is utilized as follows: + +```mermaid +graph LR + te --/--- segment1["<factory>"] --/--- segment2["<building>"] --/--- segment3["<area>"] --/--- segment4["<equipment>"] +``` + +The equipment, which is a conveyor belt called "belt01", is located in factory "factory01", in the "hallA" building, in the "packaging" area. The conveyor belt can be registered as a child-device by publishing the following message. + +```sh te2mqtt formats=v1 +tedge mqtt pub -r 'te/factory01/hallA/packaging/belt001' '{ + "@type": "child-device", + "name": "belt001", + "type": "ConveyorBelt", + "factory": "factory01", + "building": "hallA", + "area": "packaging", + "equipment": "belt001" +}' +``` + +:::info +The registration message can include additional static information about the entity to give more information about the entity to other clients who subscribe to the registration messages. +::: + +The registration message will associate the telemetry data or commands published or received from the `te/factory01/hallA/packaging/belt001` topic prefix with the `belt001` child-device from the registration message. + +For example, an event can be published to the `belt01` equipment using the following message. + +```sh te2mqtt formats=v1 +tedge mqtt pub 'te/factory01/hallA/packaging/belt001/e/running_status' '{ + "text": "Belt started" +}' +``` + +Since the topic schema encodes additional location information about the equipment, other MQTT clients can subscribe to telemetry data coming from all equipment located in the same area using a single MQTT subscription. + +```sh te2mqtt formats=v1 +tedge mqtt sub 'te/factory01/hallA/packaging/+/e/+' +``` + +:::tip +When applying your own semantics to the **identifier**, you can leave any segment blank if you don't want to apply a meaning to it. + +For example, if it does not make sense to have the factory and building in the **identifier**, then they can be removed and the equipment can be registered using: + +```sh te2mqtt formats=v1 +tedge mqtt pub -r 'te/packaging/belt001' '{ + "@type": "child-device" +}' +``` + +Publishing an event requires just leaving the last two segments of the **identifier** blank (whilst keeping the slashes `/` in place). + +```sh te2mqtt formats=v1 +tedge mqtt pub 'te/packaging/belt001///e/running_status' '{ + "text": "Belt started" +}' +``` +::: diff --git a/versioned_docs/version-1.1.0/contribute/index.md b/versioned_docs/version-1.1.0/contribute/index.md new file mode 100644 index 0000000..10600e5 --- /dev/null +++ b/versioned_docs/version-1.1.0/contribute/index.md @@ -0,0 +1,12 @@ +--- +title: Contribute to thin-edge.io +tags: [Contribute] +sidebar_position: 5 +--- + +import DocCardList from '@theme/DocCardList'; + +This section is for who wants to contribute to %%te%%, +submitting issues, writing documentation, implementing features. + + \ No newline at end of file diff --git a/versioned_docs/version-1.1.0/contribute/package-hosting.md b/versioned_docs/version-1.1.0/contribute/package-hosting.md new file mode 100644 index 0000000..dea43af --- /dev/null +++ b/versioned_docs/version-1.1.0/contribute/package-hosting.md @@ -0,0 +1,135 @@ +--- +title: Package Hosting +tags: [Contribute, Packaging] +description: Package hosting of %%te%% components +--- + +In additional to the %%te%% install script, the packages are also publicly hosted APT repository. + +## Official releases + +The following APT repositories contain the official releases of %%te%%. The packages will have a nice `x.y.z` version number, and all of the packages go through our full suite of automated and manual testing. + +The packages are the same ones which are uploaded to the [GitHub Releases](https://github.com/thin-edge/thin-edge.io/releases) page. + +### tedge-release (default) + +This is the default repository that most users should be using. + +**Setup script** +```sh +curl -1sLf \ + 'https://dl.cloudsmith.io/public/thinedge/tedge-release/setup.deb.sh' \ + | sudo -E bash +``` + +**Supported Architectures** +* `amd64` +* `arm64` +* `armhf` (armv7) + +### tedge-release-armv6 + +If you are using a Raspberry Pi with a `armv6l` CPU Architecture, then you will need to use this repository. + +```sh +curl -1sLf \ + 'https://dl.cloudsmith.io/public/thinedge/tedge-release-armv6/setup.deb.sh' \ + | sudo -E bash +``` + +**Supported Architectures** +* `armhf` (armv6) + +## Pre releases + +The latest built packages from the `main` branch of the project. The packages go through the same automated testing process as the official releases, however they are not tagged in git, so the version numbers will look like `0.8.1-171-ga72e5432` (see the [Version syntax](#version-syntax) for description about the version). + +These repositories allow you to test new features as they get merged to `main`, rather than waiting for the official release. However it is still advised to only use these repositories for development and testing purposes as the official versions go through additional testing. + +### tedge-main + +```sh +curl -1sLf \ + 'https://dl.cloudsmith.io/public/thinedge/tedge-main/setup.deb.sh' \ + | sudo -E bash +``` + +**Supported Architectures** +* `amd64` +* `arm64` +* `armhf` (armv7) + +### tedge-main-armv6 + +If you are using a Raspberry Pi with a `armv6l` CPU Architecture, then you will need to use this repository. + +```sh +curl -1sLf \ + 'https://dl.cloudsmith.io/public/thinedge/tedge-main-armv6/setup.deb.sh' \ + | sudo -E bash +``` + +**Supported Architectures** +* `armhf` (armv6) + + + +## Installing and upgrading + +1. Assuming you have already configured the appropriate APT repository, you will need to make sure it is up to date. + + ```sh + sudo apt-get update + ``` + +2. Install/update each of the packages + + ```sh + sudo apt-get install -y \ + tedge \ + tedge-mapper \ + tedge-agent \ + tedge-apt-plugin \ + c8y-firmware-plugin \ + c8y-remote-access-plugin + ``` + + The latest version will be automatically selected by `apt-get`. + +## Removing the APT repositories + +All of the %%te%% apt repositories can be removed by the following command. + +```sh +sudo rm -f /etc/apt/sources.list.d/thinedge-tedge-*.list +sudo apt-get clean +sudo rm -rf /var/lib/apt/lists/* +sudo apt-get update +``` + +## Version syntax {#version-syntax} + +The version is automatically generated from the source code management tool, git. The version is based on the commit used to build the packages and its distance from the last tag (e.g. the last official released version). + +```sh +{base_version}-{distance}g{git_sha} + +# Example +0.8.1-171-ga72e5432 +``` + +|Part|Description| +|----|-----------| +|`base_version`|Last official release| +|`distance`|Number of commits on the `main` branch since last official release| +|`git_sha`|Git commit sha which the package was built from. This makes it easier to trace the version back to the exact commit| + +# How is this made possible? + +Package repository hosting is graciously provided by [Cloudsmith](https://cloudsmith.com). +Cloudsmith is the only fully hosted, cloud-native, universal package management solution, that +enables your organization to create, store and share packages in any format, to any place, with total +confidence. + +[![Hosted By: Cloudsmith](https://img.shields.io/badge/OSS%20hosting%20by-cloudsmith-blue?logo=cloudsmith&style=flat-square)](https://cloudsmith.com) diff --git a/versioned_docs/version-1.1.0/contribute/setting-up-docker.md b/versioned_docs/version-1.1.0/contribute/setting-up-docker.md new file mode 100644 index 0000000..b4abee5 --- /dev/null +++ b/versioned_docs/version-1.1.0/contribute/setting-up-docker.md @@ -0,0 +1,113 @@ +--- +title: Setting up Docker +tags: [Contribute] +--- + +This guide is to assist with the installation of docker (OSS Variant) and some of the docker cli tools. It is not meant to replace other online articles but more of a cheat sheet to getting your setup up and running as fast as possible. If you encounter issues with the setup, please search online for a fix. + +## Installation + +### MacOS (amd64/arm64) + +There are different ways to install the docker engine on MacOS, however this guide will focus on using [colima](https://github.com/abiosoft/colima) which uses `limactl` under the hood to create a virtual machine and supporting functionality such as automatic port forwarding. + +1. Install homebrew. You can also install homebrew without `sudo` rights using this [guide](https://docs.brew.sh/Installation#untar-anywhere-unsupported) + + :::tip + Make sure you read the homebrew instructions which are printed out on the console as it will guide you to the additional setup steps. + ::: + +2. Install [colima](https://github.com/abiosoft/colima) and the docker cli + + ``` + brew install colima + ``` + + :::info + The installation will take a while as it will have to compile various dependencies. + ::: + +3. Install the docker cli tools + + ``` + brew install docker docker-compose docker-credential-helper + ``` + + Then configure `docker-compose` as a docker plugin so that you can use `docker compose` instead of the legacy `docker-compose` script. + + ``` + mkdir -p ~/.docker/cli-plugins + ln -sfn $(brew --prefix)/opt/docker-compose/bin/docker-compose ~/.docker/cli-plugins/docker-compose + ``` + +4. Configure and start `colima` + + ``` + colima start --cpu 4 --memory 8 + ``` + + :::info + * The extra arguments are only needed the first time you start it. You can change the settings to based on what you are need/have. + * You can adjust the number of CPUs and amount of RAM used by colima to suite your machine + * You will have to start `colima` each time your machine is restarted (as it is not configured to autostart, though there are ways on MacOS to do this) + ::: + +5. Check that everything is working by starting the `hello-world` container + + ``` + docker run -it --rm hello-world + ``` + + +**References** + +* https://smallsharpsoftwaretools.com/tutorials/use-colima-to-run-docker-containers-on-macos/ + + +### Windows (WSL2) + +If you are using Windows, it is recommended to use WSL2 to run your a Ubuntu distribution where `docker-ce` is installed within it. Please do not bother with Docker Desktop or any other "Desktop" related product (e.g. `Rancher Desktop`). It will save you a lot of hassle by using the native `docker-ce` version inside the WSL2 distribution itself. + +Once you have a Ubuntu distribution running under WSL2, follow the instructions under the [Linux installation](#Linux) section. + +### Linux + +Checkout the [online documentation](https://docs.docker.com/engine/install/) how to install the docker engine (`docker-ce`) on your linux distribution. Don't forget to run the [Linux postinstall instructions](https://docs.docker.com/engine/install/linux-postinstall/) to enable docker to be controlled by non-root linux user. + +The following steps should be executed: + +* Install [docker-ce](https://docs.docker.com/engine/install/) +* Install [docker compose plugin](https://docs.docker.com/compose/install/) +* Run docker-ce [post installation instructions](https://docs.docker.com/engine/install/linux-postinstall/) + +After the install verify that everything is working correctly by running the following commands + +1. Check docker + + ```sh + docker run -it --rm hello-world + ``` + +2. Verify that the `docker compose` plugin was configured correctly + + ```sh + docker compose --help + ``` + +## Troubleshooting + +Some typical problems can be found here. Remember, if you don't find your error then please search online using your preferred search engine. + +### Invalid + +Check the docker config file `~/.docker/config.json`. The credential manager setting (`credsStore`) should be set to `osxkeychain`. + +**Example: ~/.docker/config.json** + +```json +{ + "auths": {}, + "credsStore": "osxkeychain", + "currentContext": "colima" +} +``` diff --git a/versioned_docs/version-1.1.0/contribute/vscode-dev-containers.md b/versioned_docs/version-1.1.0/contribute/vscode-dev-containers.md new file mode 100644 index 0000000..7b57428 --- /dev/null +++ b/versioned_docs/version-1.1.0/contribute/vscode-dev-containers.md @@ -0,0 +1,112 @@ +--- +title: VSCode Dev Containers +tags: [Contribute] +--- + +VSCode Dev Containers are great for developing as they contain all of the project's dependencies and in the end you have a ready-to-use dockerFile which can also be used in your CI/CD pipeline. + +It provides a normalized experience to all VSCode users across MacOS, Windows and Linux. All you need to get going is a the open source docker engine (not Docker Desktop!) and docker-compose. + +More information about Dev Containers can be found from the [official documentation](https://code.visualstudio.com/docs/devcontainers/containers). + +## Pre-requisites + +Before VSCode Dev container can be used, you need to install a few things. + +1. Install `docker` and `docker-compose` if you do not already have these + + Checkout the [INSTALLING_DOCKER](./setting-up-docker.md) instructions to help guide you + +1. Install VSCode and make sure it is up to date + +2. Install the [Remote Development extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) + + ``` + ms-vscode-remote.vscode-remote-extensionpack + ``` + +## Getting started with Dev Containers + +Once the pre-requisites have been installed, you are ready to start using the dev container. You only need to run this instructions once. If you have already setup your dev container and just want to know how to re-open it, please read the next section. + +There are two main methods when setting up the dev container, we recommend using the `Method 1`. `Method 1` uses the `Clone in Named Container Volume` strategy as it is more performant on most Operating Systems. For more information about it please refer to the following [Dev Container documentation](https://code.visualstudio.com/remote/advancedcontainers/improve-performance). The only downside to this method is that the code will be stored in a docker volume. This means you can not directly navigate to the project from the host system's file explorer, however you can still `download` files from it from VSCode file explorer. + +But in both methods it is recommended to fork the project first, then clone from your fork. You can fork from the [GitHub](https://github.com/thin-edge/thin-edge.io) website, and then copy the git url from your fork. + +### Method 1: Cloning in named container volume (Recommended) + +1. In VSCode, open the command pallet (`Cmd + Shift + P` (MacOS) or `Ctrl + Shift + P` (Windows)) + +2. Enter `Clone Repository In Named Container Volume` + + ``` + Dev Containers: Clone Repository In Named Container Volume + ``` + + Enter your forks git url, for example the https style url will look something like this: + + ``` + https://github.com//thin-edge.io.git + ``` + + Then follow the remaining prompts (usually you can just accept the default values). + +3. Now you can sit back and wait for the dev container to be built. VSCode will set everything up for you (including installing the project specific extensions automatically). Don't worry it just takes a while to build the first time around. Reopening the container again later on is quick + +### Method 2: Cloning to the host filesystem + +1. Open a terminal + +2. Clone the forked %%te%% project + + **Example** + + ```sh + git clone https://github.com//thin-edge.io.git + ``` + +3. Change directory into the project folder + + ```sh + cd thin-edge.io + ``` + +4. Open the project folder inside VSCode + + Easiest way is to use the `code` helper which is hopefully installed already by VSCode. If it isn't then you can install it from VSCode via the command pallet under `Shell Command: Install 'code' command in PATH`. + + ```sh + code . + ``` + + Alternatively you can open VSCode, then select `File > Open Folder...` from the menu, and select where the project folder was cloned to. + +5. VSCode should ask you if you want to re-open the project in the Dev Container (in the bottom right-hand corner). + +6. Now you can sit back and wait for the dev container to be built. VSCode will set everything up for you (including installing the project specific extensions automatically). Don't worry it just takes a while to build the first time around. Reopening the container again later on is quick + +## Re-opening an already setup dev container + +1. Open VSCode + +2. Click on the `Remote Explorer` Icon on the left hand navigation menu (the icon looks like a computer monitor) + +3. From the `Dev Containers` section, right-click on the %%te%% dev container, and select `Open Folder in Container` + + +## Rebuilding the dev container + +Rebuilding the dev container is sometimes required/useful. Some common reason to rebuild it are: + +* Some dependencies were added/changed in the `.devcontainer` folder and you would like to use the changes +* You want a fresh dev container as you have modified some of the container's OS system and you suspect you broke something + +But don't worry, rebuilding is easy, you just need to follow these short instructions: + +1. In VSCode, open the command pallet (`Cmd + Shift + P` (MacOS) or `Ctrl + Shift + P` (Windows)) + +2. Enter the following command + + ``` + Dev Containers: Rebuild Container + ``` diff --git a/versioned_docs/version-1.1.0/contribute/writing-documentation/index.md b/versioned_docs/version-1.1.0/contribute/writing-documentation/index.md new file mode 100644 index 0000000..ca54a5b --- /dev/null +++ b/versioned_docs/version-1.1.0/contribute/writing-documentation/index.md @@ -0,0 +1,11 @@ +--- +title: Writing Documentation +tags: [Documentation] +--- + +import DocCardList from '@theme/DocCardList'; + +The following pages provide guidance to writing useful, +and consistent documentation by providing examples of custom components and common dos and don'ts. + + diff --git a/versioned_docs/version-1.1.0/contribute/writing-documentation/markdown-components.md b/versioned_docs/version-1.1.0/contribute/writing-documentation/markdown-components.md new file mode 100644 index 0000000..8cd91eb --- /dev/null +++ b/versioned_docs/version-1.1.0/contribute/writing-documentation/markdown-components.md @@ -0,0 +1,357 @@ +--- +title: Markdown Components +tags: [Documentation] +sidebar_position: 1 +--- + +# Markdown components + +Docusaurus provides a rich set of features to write really nice documentation. Please read their [online docs](https://docusaurus.io/docs/markdown-features) to check out the markdown features. + +This page details the custom markdown extensions available when writing the documentation. + +## Command block + +If you what to generate the contents of a code block at build time, then you can use the special `command` directive to indicate that the code-blocks contents should be set from the output of a command. + +|Property|Type|Description| +|--------|----|-----------| +|`command`|string|Command to be executed during build time. Command must return an exit code 0.| + +If the code block title is not provided, then the plugin will set it to `Output`. + + +### Example: Simple command + +For example, the following snippet will execute `ls -l` and store the command's output in the code block. + +````markdown title="Markdown" +```text command="ls -l" +``` +```` + +**Output** + +```text command="ls -l" +``` + +### Example: Simple command with custom title + +Execute a command and store the output. Use a custom title for the code block. + +````markdown title="Markdown" +```text command="ls -l" title="My custom title" +``` +```` + +**Output** + +```text command="ls -l" title="My custom title" +``` + +### Example: Simple command with default contents + +If you would like to make the code block still readable by other markdown renders (e.g. GitHub), then you can provide a sensible default value inside the code block. + +Execute a command and store the output. Use a custom title for the code block. + +````markdown title="Markdown" +```text command="ls -l" title="My custom title" +total 848 +-rw-r--r--@ 1 tedge staff 209 Jun 22 11:13 Dockerfile +-rw-r--r-- 1 tedge staff 2718 Jul 4 15:31 README.md +-rw-r--r--@ 1 tedge staff 89 Jun 22 11:13 babel.config.js +drwxr-xr-x@ 21 tedge staff 672 Jul 4 15:13 build +lrwxr-xr-x@ 1 tedge staff 24 Jun 20 18:41 docs -> ../thin-edge.io/docs/src +-rw-r--r-- 1 tedge staff 6200 Jul 4 15:19 docusaurus.config.js +-rw-r--r-- 1 tedge staff 3296 Jul 3 17:36 justfile +drwxr-xr-x@ 751 tedge staff 24032 Jul 4 09:10 node_modules +-rw-r--r-- 1 tedge staff 1363 Jul 4 09:18 package.json +-rw-r--r--@ 1 tedge staff 781 Jun 22 11:13 sidebars.js +drwxr-xr-x@ 6 tedge staff 192 Jul 1 11:08 src +drwxr-xr-x@ 4 tedge staff 128 Jun 22 11:13 static +drwxr-xr-x@ 3 tedge staff 96 Jul 4 15:22 versioned_docs +drwxr-xr-x@ 3 tedge staff 96 Jul 3 18:55 versioned_sidebars +-rw-r--r--@ 1 tedge staff 10 Jul 4 15:22 versions.json +-rw-r--r-- 1 tedge staff 394026 Jul 4 09:18 yarn.lock +``` +```` + +**Output** + +```text command="ls -l" +total 848 +-rw-r--r--@ 1 tedge staff 209 Jun 22 11:13 Dockerfile +-rw-r--r-- 1 tedge staff 2718 Jul 4 15:31 README.md +-rw-r--r--@ 1 tedge staff 89 Jun 22 11:13 babel.config.js +drwxr-xr-x@ 21 tedge staff 672 Jul 4 15:13 build +lrwxr-xr-x@ 1 tedge staff 24 Jun 20 18:41 docs -> ../thin-edge.io/docs/src +-rw-r--r-- 1 tedge staff 6200 Jul 4 15:19 docusaurus.config.js +-rw-r--r-- 1 tedge staff 3296 Jul 3 17:36 justfile +drwxr-xr-x@ 751 tedge staff 24032 Jul 4 09:10 node_modules +-rw-r--r-- 1 tedge staff 1363 Jul 4 09:18 package.json +-rw-r--r--@ 1 tedge staff 781 Jun 22 11:13 sidebars.js +drwxr-xr-x@ 6 tedge staff 192 Jul 1 11:08 src +drwxr-xr-x@ 4 tedge staff 128 Jun 22 11:13 static +drwxr-xr-x@ 3 tedge staff 96 Jul 4 15:22 versioned_docs +drwxr-xr-x@ 3 tedge staff 96 Jul 3 18:55 versioned_sidebars +-rw-r--r--@ 1 tedge staff 10 Jul 4 15:22 versions.json +-rw-r--r-- 1 tedge staff 394026 Jul 4 09:18 yarn.lock +``` + +## Generic code block tabs + +Generic code block tabs can be created by adding the `tab` directive to consecutive code blocks. This allows you to display the related code blocks in a tab format which not only saves spaces, but also give the user more control over what they want to focus on. + +The directive is provided by the [mrazauskas/docusaurus-remark-plugin-tab-blocks](https://github.com/mrazauskas/docusaurus-remark-plugin-tab-blocks) plugin. + +### Example: Simple tabs + +The following uses code blocks and uses the syntax highlighter option as the tab names. This only makes sense if you don't want to use syntax highlighting in your commands. + +````markdown title="Markdown" +```systemd tab +systemctl status tedge +``` + +```openrc tab +service tedge status +``` +```` + +**Output** + +```systemd tab +systemctl status tedge +``` + +```openrc tab +rc-service tedge status +``` + +### Example: Tabs with custom labels + +Custom labels can be added to each table by specifying the `"label"` option on the `tab` directive. + +Below shows two sets of code blocks, the first being instructions how to connect the mapper to 3 different clouds, and the other set shows how to disconnect the mapper for the same clouds. By default any tabs that use the same label-name will be automatically synced. + +````markdown title="Markdown" +**Connect mapper** + +```sh tab={"label":"c8y"} +tedge connect c8y +``` + +```sh tab={"label":"az"} +tedge connect az +``` + +```sh tab={"label":"aws"} +tedge connect aws +``` + +**Disconnect mapper** + +```sh tab={"label":"c8y"} +tedge disconnect c8y +``` + +```sh tab={"label":"az"} +tedge disconnect az +``` + +```sh tab={"label":"aws"} +tedge disconnect aws +``` +```` + +**Output** + +**Connect mapper** + +```sh tab={"label":"c8y"} +tedge connect c8y +``` + +```sh tab={"label":"az"} +tedge connect az +``` + +```sh tab={"label":"aws"} +tedge connect aws +``` + +**Disconnect mapper** + +```sh tab={"label":"c8y"} +tedge disconnect c8y +``` + +```sh tab={"label":"az"} +tedge disconnect az +``` + +```sh tab={"label":"aws"} +tedge disconnect aws +``` + +## MQTT Code block + +One of the main interfaces for %%te%% is MQTT, therefore a special plugin has been created to display how to publish/subscribe to messages on an MQTT broker. + +The code block only expects the writer to provide the `tedge mqtt ...` command, and the plugin will take care of the translation to the various formats. The decision was made to use the `tedge mqtt` command as the base format so that the code blocks would also still be useful when viewing the raw markdown documents. + +The MQTT code block is activated by using the `te2mqtt` directive. The directive will translate the `tedge mqtt` command into the different variants and display them as a grouped tab. The reader is then able to select the tab which is relevant to their use-case and this setting will sticky (stored in their browser's local storage). + +### Example: Publish MQTT message + +The following shows how to document how to publish a single MQTT message. + +**Markdown** + +````markdown title="Markdown" +```sh te2mqtt formats=v1 +tedge mqtt pub te/device/main///m/ '{"temperature": 21.3}' +``` +```` + +**Output** + +```sh te2mqtt formats=v1 +tedge mqtt pub te/device/main///m/ '{"temperature": 21.3}' +``` + +### Example: Use legacy MQTT api + +The tedge api code blocks also have the option to set which formats should be displayed, for example the legacy or new v1 api. + +**Markdown** + +````markdown title="Markdown" +```sh te2mqtt formats=legacy,v1 +tedge mqtt pub tedge/alarms/major/temp_hi_hi/child01 '{"text": "Temperature Hi Hi"}' +``` +```` + +**Output** + +```sh te2mqtt formats=legacy,v1 +tedge mqtt pub tedge/alarms/major/temp_hi_hi/child01 '{"text": "Temperature Hi Hi"}' +``` + +### Example: Use new MQTT api + +The MQTT code block can also be written using the new MQTT api and the command will be automatically translated to the legacy (if there is an equivalent). + +**Markdown** + +````markdown title="Markdown" +```sh te2mqtt formats=legacy,v1 +tedge mqtt pub te/device/child01///a/temp_hi_hi '{"text": "Temperature Hi Hi", "severity": "warning"}' +``` +```` + +**Output** + +```sh te2mqtt formats=legacy,v1 +tedge mqtt pub te/device/child01///a/temp_hi_hi '{"text": "Temperature Hi Hi", "severity": "warning"}' +``` + +### Example: MQTT code blocks with different grouping options + +The tedge api code blocks can also be grouped in different ways. + +#### Grouped + +**Markdown** + +````markdown title="Markdown" +```sh te2mqtt formats=legacy,v1 groupTabs=true +tedge mqtt pub te/device/child01///a/temp_hi_hi '{"text": "Temperature Hi Hi", "severity": "warning"}' +``` +```` + +**Output** + +```sh te2mqtt formats=legacy,v1 groupTabs=true +tedge mqtt pub te/device/child01///a/temp_hi_hi '{"text": "Temperature Hi Hi", "severity": "warning"}' +``` + +#### Ungrouped + +**Markdown** + +````markdown title="Markdown" +```sh te2mqtt formats=legacy,v1 groupTabs=false +tedge mqtt pub te/device/child01///a/temp_hi_hi '{"text": "Temperature Hi Hi", "severity": "warning"}' +``` +```` + +**Output** + +```sh te2mqtt formats=legacy,v1 groupTabs=false +tedge mqtt pub te/device/child01///a/temp_hi_hi '{"text": "Temperature Hi Hi", "severity": "warning"}' +``` + +### Example: Subscribe to MQTT messages + +````markdown title="Markdown" +```sh te2mqtt formats=v1 +tedge mqtt sub 'te/#' +``` +```` + +**Output** + +```sh te2mqtt formats=v1 +tedge mqtt sub 'te/#' +``` + +### Example: Subscribe to MQTT messages and redirect the output + +Any additional arguments will be automatically preserved when translating the commands. Though the `mqtt` tab will not display this information it is just meant to represent the MQTT message and does not know anything about the terminal/console. + +````markdown title="Markdown" +```sh te2mqtt formats=v1 +tedge mqtt sub 'te/#' > logged.tedge.messages.txt +``` +```` + +**Output** + +```sh te2mqtt formats=v1 +tedge mqtt sub 'te/#' > logged.tedge.messages.txt +``` + +## Proposal Banner + +A proposal banner is used to indicate that a page details a proposal only and is not yet implemented. The banner allows the proposal to be made public to encourage early feedback from the community as the page will be included in the public documentation. + +:::note +If a proposal is not ready for public feedback, then it should be marked as a draft by setting the `draft: true` in the page's [front matter](https://docusaurus.io/docs/markdown-features#front-matter) section. Draft pages are not included in the public documentation, though they are included when building the docs locally in development mode. +::: + +The proposal banner can be included in a page by adding the following markdown to the top of the page (after the [front matter](https://docusaurus.io/docs/markdown-features#front-matter) section). + +````markdown title="Markdown" +import ProposalBanner from '@site/src/components/ProposalBanner' + + +```` + +**Output** + +import ProposalBanner from '@site/src/components/ProposalBanner' + + + +In addition to the ``, a proposal page should include the construction icon (🚧) before the page's title so that the page's entry in the sidebar menu clearly shows that the page is a proposal or a work in progress. The example below shows how the page's title is set via the front matter at the top of the markdown document. + + +````markdown title="Markdown" +--- +title: 🚧 MQTT API +tags: [Reference, MQTT] +sidebar_position: 6 +--- +```` diff --git a/versioned_docs/version-1.1.0/contribute/writing-documentation/markdown-guidelines.md b/versioned_docs/version-1.1.0/contribute/writing-documentation/markdown-guidelines.md new file mode 100644 index 0000000..84a33f2 --- /dev/null +++ b/versioned_docs/version-1.1.0/contribute/writing-documentation/markdown-guidelines.md @@ -0,0 +1,182 @@ +--- +title: Markdown Guidelines +tags: [Documentation] +sidebar_position: 2 +--- + +A guideline to writing markdown in a consistent manner. + +## Headers + +### Avoid in-line code blocks in headers + +Headers should not include in-line code blocks. + +**βœ… Good** + +````md +## thin-edge.io info +```` + +**❌ Bad** + +````md +## `thin-edge.io` info +```` + +### Don't end a subsection with a colon + +Headers (either headers or bold text acting as a header) does not need punctuation. + +**βœ… Good** + +````md +**Topic** +```` + +**❌ Bad** + +````md +**Topic:** +```` + + +## Code blocks + +### Shell/console syntax highlighting + +Use the `sh` syntax highlight option for any console/terminal related code blocks. + +**βœ… Good** + +````md +```sh +tedge --help +``` +```` + +**❌ Bad** + +````md +```console +tedge --help +``` +```` + +````md +```shell +tedge --help +``` +```` + +### Console output + +Console output should use the `text` syntax highlighter and should have a title block indicating that it is output. + +You can use other syntax highlighting if the output is of another format (e.g. json). + +**βœ… Good** + +````md +```sh title="Output" +tedge --help +``` +```` + +**❌ Bad** + +````md +```sh +tedge --help +``` +```` + +**βœ… Good** + +````md +```json title="Output" +{ + "text": "example" +} +``` +```` + +**❌ Bad: Incorrect syntax highlighting for the output** + +````md +```sh +{ + "text": "example" +} +``` +```` + +### Avoid in-line code blocks for long commands + +In-line code blocks are not easy for users to copy the text from. Use a full code-block instead as these blocks are rendered with a "copy" button built-in. + +**βœ… Good** + +````md +**Topic** + +```text +te/device/{child}///cmd/config_snapshot/1 +``` +```` + +**❌ Bad** + +````md +**Topic** + + `te/device/{child}///cmd/config_snapshot/1` +```` + +### Systemd/journald service commands + +Don't use the `.service` suffix in calls to `systemctl` and `journalctl`. + +**βœ… Good** + +````md +```sh +sudo systemctl restart tedge-mapper-c8y +``` +```` + +**❌ Bad** + +````md +```sh +sudo systemctl restart tedge-mapper-c8y.service +``` +```` + +### Set file path in title + +If the contents of a code block relate to the contents of a file, then the file path should be included in the title. + +**βœ… Good** + +````md +```toml title="file: /etc/tedge/tedge.toml" +[c8y] +url = "mytenant.cumulocity.com" + +[mqtt] +bind_address = "127.0.0.1" +``` +```` + +**❌ Bad** + +````md +```toml +[c8y] +url = "mytenant.cumulocity.com" + +[mqtt] +bind_address = "127.0.0.1" +``` +```` diff --git a/versioned_docs/version-1.1.0/extend/firmware-management/building-image/index.md b/versioned_docs/version-1.1.0/extend/firmware-management/building-image/index.md new file mode 100644 index 0000000..3904a7c --- /dev/null +++ b/versioned_docs/version-1.1.0/extend/firmware-management/building-image/index.md @@ -0,0 +1,11 @@ +--- +title: Building an Image +tags: [Extend] +sidebar_position: 1 +--- + +import DocCardList from '@theme/DocCardList'; + +Instructions on how to build different Operation System images with %%te%% setup to perform Over-the-Air (OTA) updates of the operating system. + + \ No newline at end of file diff --git a/versioned_docs/version-1.1.0/extend/firmware-management/building-image/rugpi/index.md b/versioned_docs/version-1.1.0/extend/firmware-management/building-image/rugpi/index.md new file mode 100644 index 0000000..2221fcc --- /dev/null +++ b/versioned_docs/version-1.1.0/extend/firmware-management/building-image/rugpi/index.md @@ -0,0 +1,13 @@ +--- +title: Rugpi +tags: [Extend] +sidebar_position: 1 +--- + +import DocCardList from '@theme/DocCardList'; + +Guides to building custom %%te%% based images using [Rugpi](https://github.com/silitics/rugpi). The images produced support Over-the-Air (OTA) Operating System updates. + +Currently Rugpi only supports Raspberry Pi devices, however this maybe expanded upon in the future. + + diff --git a/versioned_docs/version-1.1.0/extend/firmware-management/building-image/rugpi/tutorial.md b/versioned_docs/version-1.1.0/extend/firmware-management/building-image/rugpi/tutorial.md new file mode 100644 index 0000000..d967ebc --- /dev/null +++ b/versioned_docs/version-1.1.0/extend/firmware-management/building-image/rugpi/tutorial.md @@ -0,0 +1,230 @@ +--- +title: Building an image +tags: [Extend, Build] +sidebar_position: 3 +--- + +This guide walks you through building a custom image using [Rugpi](https://github.com/silitics/rugpi). Rugpi is an open source tool to build images with support for robust Over-the-Air (OTA) updates. Currently it only supports Raspberry Pi devices, however in the future more device types may be considered. + +Rugpi is very beginner friendly compared to The Yocto Project, and has much shorter build times since it works by extending the official Raspberry Pi images with custom recipes which either install or configure components in the image, rather than rebuilding everything from scratch. Building an image with custom recipes only takes about 10 mins compared with the > 4 hours using The Yocto Project. + +The instructions on this page can be used to build images for the following devices: + +* Raspberry Pi 1 (32 bit) +* Raspberry Pi 2 (64 bit) +* Raspberry Pi 3 (64 bit) +* Raspberry Pi 4 / 400 (64 bit) +* Raspberry Pi Compute Module 4 (64 bit) +* Raspberry Pi 5 (64 bit) +* Raspberry Pi Zero W (32 bit) +* Raspberry Pi Zero 2W (64 bit) + +Please check out the [official Rugpi documentation](https://oss.silitics.com/rugpi/) for more details on how to use and customize it. + +## Setting up your build system + +Building images using Rugpi is considerably faster and easier when compared to The Yocto Project, so it makes it a great tool for beginners. + +The following tools are required to build an image: + +* Docker Engine + +:::tip +Images can be built using Rugpi using a CI Workflow. An example for a Github Workflow is included in the template project. +::: + +## Building your image + +The [tedge-rugpi-images](https://github.com/thin-edge/tedge-rugpi-image) project includes out of the box configurations to perform robust Over-the-Air Operating System updates. + +The [tedge-rugpi-core](https://github.com/thin-edge/tedge-rugpi-core) project contains the %%te%% specific recipes which are used to build the image. + +Feel free to clone the project if you want to make your own customizations, however please always refer back to the project if you run into any problems (as it may have changed in the meantime). + +### Pre-requisites + +To run the project tasks, you will need to install a command line tool called [just](https://just.systems/man/en/chapter_5.html). + +You can install just using on of the following commands: + +```sh tab={"label":"homebrew"} +brew install just +``` + +```sh tab={"label":"cargo"} +cargo install just +``` + +```sh tab={"label":"script"} +curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | sudo bash -s -- --to /usr/bin +``` + +For other installation possibilities check out the [just documentation](https://just.systems/man/en/chapter_4.html). + +### Cloning the project {#clone-project} + +1. Clone the project and change directory to the locally checked out folder + + ```sh + git clone https://github.com/thin-edge/tedge-rugpi-image.git + cd tedge-rugpi-image + ``` + +2. Create a custom `.env` file which will be used to store secrets + + ```sh + cp env.template .env + ``` + + The `.env` file will not be committed to the repo + +3. Add your public ssh key to the `.env` file + + Adding public SSH keys to the project will allow you to access after the image is flashed to the device (which is critical for the onboarding process). The SSH keys are provided in the form of environment variables where the variable names start with `SSH_KEY_` and are added to the authorized keys for the root user, e.g. `/root/.ssh/authorized_keys`. + + For example: + + ``` + SSH_KEYS_bootstrap="ssh-rsa xxxxxxx" + ``` + + :::tip + This step is critical as it will enable you to connect via SSH to your device to perform tasks such as onboarding! If you don't set your ssh public key in the authorized keys, you then need to connect your device to a monitor/display and keyboard in order to perform the onboarding. + ::: + +4. Optional: Add Wifi ssid/password + + If your device does not have an ethernet adapter, or if you want to connect the device to a Wifi network for onboarding, then you will have to add the Wifi credentials to the `.env` file. + + Below shows the environment variables that should be added to the `.env` file. + + ```sh + SECRETS_WIFI_SSID=example + SECRETS_WIFI_PASSWORD=yoursecurepassword + ``` + + :::tip + The Wifi credentials only need to be included in the image that is flashed to the SD card. Subsequent images don't need to include the Wifi credentials, as the network connection configuration files are persisted across images. + + If an image has Wifi credentials baked in, then you should not make this image public, as it would expose your credentials! + ::: + + +### Building an image + +Building an image will produce a `.xz` file which can be flashed to an SD card (or device). Afterwards your device will be able to apply the same type of image via OTA updates. + +Images can be built in MacOS or Linux environments (including WSL 2), however if you have problems building the image, then you can build the images using the Github workflow after forking the project. + + +1. Build an image for your device (machine) + + ```sh tab={"label":"Pi\tZero"} + just build-pizero + ``` + + ```sh tab={"label":"Pi\t1"} + just build-pi1 + ``` + + ```sh tab={"label":"Pi\t2"} + just build-pi2 + ``` + ```sh tab={"label":"Pi\t3"} + just build-pi3 + ``` + ```sh tab={"label":"Pi\tZero2W"} + just build-pizero2w + ``` + + ```sh tab={"label":"Pi\t4\t(With\tFirmware)"} + just build-pi4-include-firmware + ``` + + ```sh tab={"label":"Pi\t4\t(Without\tFirmware)"} + just build-pi4 + ``` + + ```sh tab={"label":"Pi\t5"} + just build-pi5 + ``` + + :::note + See the [tips](#raspberry-pi-4-image-selection) for helping you select which Raspberry Pi 4 image is suitable for you (e.g. with or without the EEPROM firmware update) + ::: + +2. Flash the base image using the instructions on the [Flashing an image](../../flashing-an-image.md) page + + +:::info +The above tasks are actually just user-friendly helpers to make it easier to select the right image for your device, so some build-xxx tasks actually build the same output image (e.g. `build-pi4` and `build-pi5` produce the same image). + +If you want more control over the process you can use the command, and select one of the images as defined in the [rugpi-bakery.toml](https://github.com/thin-edge/tedge-rugpi-image/blob/main/rugpi-bakery.toml) + +For more information about Rugpi repositories, layers and overall concepts please read the [official Rugpi documentation](https://oss.silitics.com/rugpi/docs/guide/system-customization/). +::: + + +## Tips + +This section contains general tips which can be helpful whilst either getting things setup, or what to do when you encounter an error. + +### Building on MacOS Apple Silicon + +If you receive the following error during the build process, then it indicates that your current docker setup needs to be adjusted: + +```sh +fallocate: fallocate failed: Operation not supported +``` + +On MacOS, there are a few solutions which provide the docker engine for us within MacOS, some known solutions are: + +* Docker Desktop +* Rancher Desktop +* colima + +Generally all of the above solutions require creating some kind of Virtual Machine (vm), however it is important that the virtual machine uses `virtiofs` (not `sshfs`) for managing the shared disks. Most of the above solutions should work out-of-the-box, however check below for any solution-specific instructions. + +#### colima + +Earlier [colima](https://github.com/abiosoft/colima) versions use `qemu` as the default vm-type, however this type uses `sshfs` to manage the VMs shared disk. Newer versions will use `vz` and `virtiofs` by default, however you will have to delete your existing colima instance, and recreate it opting into the `vz` vm-type. + +For example, you can remove any existing instance, and create an instance which uses `vz` instead of `qemu`: + +```sh +colima delete +colima start --vm-type=vz +``` + +After starting colima, you can verify the "mountType" (disk type) by checking the `colima status`: + +```sh +colima status +``` + +```text title="Output" +INFO[0000] colima is running using macOS Virtualization.Framework +INFO[0000] arch: aarch64 +INFO[0000] runtime: docker +INFO[0000] mountType: virtiofs +INFO[0000] socket: unix:///Users/johnsmith/.colima/default/docker.sock +``` + +### Raspberry Pi 4 image selection + +Raspberry Pi 4 devices need to have their (EEPROM) firmware updated before the OTA updates can be issued. This is because the initial Raspberry Pi 4's were released without the [tryboot feature](https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#fail-safe-os-updates-tryboot). The tryboot feature is used by Rugpi to provide the reliable partition switching between the A/B partitions. Raspberry Pi 5's have support for tryboot out of the box, so they do not require a EEPROM upgrade. + +You can build an image which includes the required EEPROM firmware to enable the tryboot feature, however this image can only be used to deploy to Raspberry Pi 4 devices (not Raspberry Pi 5!) + +```sh +just build-pi4-include-firmware +``` + +After the above image has been flashed to the device once, you can switch back to the image without the EEPROM firmware so that the same image can be used for both Raspberry Pi 4 and 5. + +```sh +just build-pi4 + +# or +just build-pi5 +``` diff --git a/versioned_docs/version-1.1.0/extend/firmware-management/building-image/yocto/index.md b/versioned_docs/version-1.1.0/extend/firmware-management/building-image/yocto/index.md new file mode 100644 index 0000000..18ec79a --- /dev/null +++ b/versioned_docs/version-1.1.0/extend/firmware-management/building-image/yocto/index.md @@ -0,0 +1,11 @@ +--- +title: Yocto +tags: [Extend] +sidebar_position: 1 +--- + +import DocCardList from '@theme/DocCardList'; + +Guides to building custom %%te%% based images using [The Yocto Project](https://www.yoctoproject.org/). The images produced support Over-the-Air (OTA) Operating System updates using proven technologies such as [RAUC](https://rauc.readthedocs.io/en/latest/) or [mender](https://github.com/mendersoftware/mender). + + diff --git a/versioned_docs/version-1.1.0/extend/firmware-management/building-image/yocto/tutorial.md b/versioned_docs/version-1.1.0/extend/firmware-management/building-image/yocto/tutorial.md new file mode 100644 index 0000000..d02ea56 --- /dev/null +++ b/versioned_docs/version-1.1.0/extend/firmware-management/building-image/yocto/tutorial.md @@ -0,0 +1,170 @@ +--- +title: Building an image +tags: [Extend, Build] +sidebar_position: 3 +--- + +:::caution +Yocto is the defacto embedded linux distribution toolkit, however it is famous for its steep learning curve. Due to this it is **critical** that you follow the instructions, as deviating from them will mostly lead to problems! +::: + +This guide walks you through a building a custom image using Yocto. The images produced by Yocto are always for dedicated devices as this allows the size of the images to significantly reduced as only the components which are related to the device are included. For example the images produced in this tutorial are only about 80MB in size. In comparison the "lite" image from Raspberry Pi is about 550MB! + +Whilst the methodology described on this page can be extended to support other devices, these instructions focus in the following devices: + +* Raspberry Pi 3 (64 bit) +* Raspberry Pi 4 (64 bit) + +## Setting up your build system + +Before being able to build your own image, you need to prepare an environment to run Yocto. Follow the instructions to get your build system setup. + +1. Create a Virtual Machine running Ubuntu 20.04 (LTS) + + The virtual machine should be created + * Use a ubuntu 20.04 image (You can choose if you want a UI or not) + * Disk with at least 100 GB, though more is better and generally it hard changing the size afterwards + * Network enabled + +2. Install essential packages + + ```sh tab={"label":"Ubuntu-20.04"} + sudo apt install file gawk wget git diffstat unzip texinfo gcc build-essential chrpath socat cpio python3 python3-pip python3-pexpect xz-utils debianutils iputils-ping python3-git python3-jinja2 libegl1-mesa libsdl1.2-dev xterm python3-subunit mesa-common-dev zstd liblz4-tool + ``` + +3. Install [git-lfs](https://git-lfs.com/) (this dependency will be removed in the next %%te%% release after 1.0.1) + + ```sh tab={"label":"Ubuntu-20.04"} + curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash + sudo apt-get install git-lfs + ``` + +:::note +The large disk size might seem like a lot, but Yocto builds everything from scratch, and this takes time and disk space! +::: + + +## Building your image + +The [meta-tedge-project](https://github.com/thin-edge/meta-tedge-project) is a meta project which uses [kas](https://kas.readthedocs.io/en/latest/) to package multiple Yocto (bitbake) projects into a single repository where the user can select which project they would like to build. It supports reusing configuration files to make it easier to configure different image definitions to control what should be included in your image. + +Feel free to clone the project if you want to make your own customizations, however please always refer back to the project if you run into any problems (as it may have changed in the meantime). + +The project uses different Yocto layers to build an image which contains: + +|Item|Description| +|--|--| +|Yocto Linux Distribution|Base operating system| +|%%te%%|%%te%% and all of its components| +|%%te%% workflow|A workflow definition to perform Operation System A/B updates which interacts with the underlying technology, e.g. [RAUC](https://rauc.readthedocs.io/en/latest/) or [mender](https://github.com/mendersoftware/mender)| +|dependencies|Dependencies of the underlying firmware update mechanism, e.g. [RAUC](https://rauc.readthedocs.io/en/latest/) or [mender](https://github.com/mendersoftware/mender)| + +### Cloning the project + +1. Clone the project and change directory to the locally checked out folder + + ```sh + git clone https://github.com/thin-edge/meta-tedge-project.git + cd meta-tedge-project + ``` + +2. Install [just](https://just.systems/man/en/chapter_5.html) which is used to run different tasks provided by the project + + ```sh + curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | sudo bash -s -- --to /usr/bin + ``` + + Check out the [justfile website](https://just.systems/man/en/chapter_5.html) for more installation options. + +3. Install [kas](https://kas.readthedocs.io/en/latest/) which is used to managed bitbake projects + + ```sh + pip3 install kas + ``` + + :::note + [kas](https://kas.readthedocs.io/en/latest/) is used to managed multiple Yocto bitbake projects. + ::: + +### Configure shared folders + +Using shared folders will help reduce the overall build times when building for different targets as the downloaded files can be reused. + +1. Open a console in the project's root directory (e.g. inside the `tedge-meta-project` directory) + +2. Create the following file with the given contents + + ```sh title="file: .env" + SSTATE_DIR=/data/yocto/sstate-cache + DL_DIR=/data/yocto/downloads + ``` + +3. Create the shared folders which are referenced in the above `.env` file + + ```sh + sudo mkdir -p /data/yocto/sstate-cache + sudo chown -R "$(whoami):$(whoami)" /data/yocto/sstate-cache + + sudo mkdir -p /data/yocto/downloads + sudo chown -R "$(whoami):$(whoami)" /data/yocto/downloads + ``` + + +### Building an image + +The following steps will build an image with %%te%% and [RAUC](https://rauc.readthedocs.io/en/latest/) installed to perform firmware (OS) updates. + +1. Build an image for your device (machine) + + ```sh tab={"label":"RaspberryPi-3"} + KAS_MACHINE=raspberrypi3-64 just build-project ./projects/tedge-rauc.yaml + ``` + + ```sh tab={"label":"RaspberryPi-4"} + KAS_MACHINE=raspberrypi4-64 just build-project ./projects/tedge-rauc.yaml + ``` + + The `KAS_MACHINE` is used to select the target device. + + :::note + Yocto build take a long time (> 4 hours) to build so be patient. A build time of 12 hours is also not unheard of! + ::: + +2. Inspect the built image + + ```sh + ls -l tmp + ``` + +3. Flash the base image using the instructions on the [Flashing an image](../../flashing-an-image.md) page + +:::tip +Check out the project files under the `./projects` directory for other available project examples. If you don't find a project that fits your needs, then just create your own project definition. +::: + +:::note +The **tedge-rauc** project includes a package manager (apt) so that additional packages can be installed without having to do a full image update. If you don't want the image to include a package manager, you can customize the project yourself and remove the related configuration file. +::: + +## Tips + +This section contains general tips which can be helpful whilst either getting things setup, or what to do when you encounter an error. + +### Building on MacOS Apple Silicon + +If you are trying to build on a MacOS Apple Silicon computer, then you will have to create a Virtual Machine (e.g. [UTM](https://mac.getutm.app/), [lima](https://github.com/lima-vm/lima) or equivalents). + +* Ubuntu 20.04 (headless variant is enough) +* Don't use CPU emulation, use the native `aarch64` option, otherwise your builds will be even slower! + +### Unexpected/cryptic build errors + +Whilst building device images it is common that at some people you will run into a build error. These errors can be very cryptic and hard to find the root cause. Luckily, more often than not, these errors are caused due to some corrupted state within the temporary build output folder. + +So before creating a ticket, make sure you clean the folder using the following command: + +```sh +just clean +``` + +Then rebuild the project that you originally tried to build. diff --git a/versioned_docs/version-1.1.0/extend/firmware-management/device-onboarding.md b/versioned_docs/version-1.1.0/extend/firmware-management/device-onboarding.md new file mode 100644 index 0000000..246f663 --- /dev/null +++ b/versioned_docs/version-1.1.0/extend/firmware-management/device-onboarding.md @@ -0,0 +1,121 @@ +--- +title: Onboarding a Device +tags: [Extend, Build] +sidebar_position: 5 +description: Connecting devices to the cloud for the first time +--- + +Onboarding a new devices involves configuring %%te%% to connect to a cloud for the first time. This generally involves the following steps: + +1. Configure the cloud url to connect to +2. Generate the device certificate (if it does not already exist) +3. Upload the device certificate to the cloud +4. Connect the associated tedge-mapper + +## Onboarding a device to Cumulocity + +The onboarding described in this section involved onboarding a device by executing remote commands on the device via SSH. + +It is a two step process which is described as follows: + +1. Detect the device (by using DNS Service Discovery) +2. Onboard it via ssh commands + + +### Pre-requisites + +On your machine you will need on of the following: + +1. One of the following CLI tools to scan for devices + + * avahi-browse + * dns-sd + + :::tip MacOS Users + **dns-sd** is installed by default. + ::: + +2. A valid ssh key which matches the authorized key loaded in the built image + + The ssh key on your local machine should be loaded. You can check if your key is loaded into the `ssh-agent` on your host by running: + + ```sh + ssh-add -l + ``` + + ```sh title="Output" + 2048 SHA256:abcdef/E1bA+DbVEsw3RoqKEMK9kU /home/powersuser/.ssh/id_rsa (RSA) + ``` + + :::info + If you are using an image built using **Rugpi**, the authorized ssh key/s should have been included in the image. If you can't ssh into your device please review the [build instructions](../building-image/rugpi/tutorial/#clone-project) again to make sure the correct ssh key/s was added during build time. + ::: + +3. [go-c8y-cli](https://goc8ycli.netlify.app/docs/installation/shell-installation/) + +4. go-c8y-cli extension [c8y-tedge](https://github.com/thin-edge/c8y-tedge) + + +#### Setting up go-c8y-cli + +1. Install [go-c8y-cli](https://goc8ycli.netlify.app/docs/installation/shell-installation/) as per the instructions + +2. Install the [](https://github.com/thin-edge/c8y-tedge) + + ```sh + c8y extension install thin-edge/c8y-tedge + ``` + +3. Create a Cumulocity session file (if one does not already for the Cumulocity instance you wish to connect to) + + ```sh + c8y sessions create + ``` + +### Procedure + +1. Insert the SD Card and power on the device, making sure the ethernet cable is also connected + +2. Activate your go-c8y-cli session where you would like to onboard your device to + + ```sh + set-session + ``` + +3. Scan for the device using DNS Service Discovery (DNS-SD) + + ```sh + c8y tedge scan + ``` + + ```sh title="Output" + rpi1-b827eb21100e + rpi2-b827ebed6e5a + rpi3-b827ebe1f7d6 + rpi4-dcb630486720 + rpi5-d83add9a145a + rpizero-b827ebdddb46 + rpizero2-d83add030bfd + ``` + + :::info + The device may take a few minutes to start and run the initialization tasks, so be patient. + + If the device is still not connected after 10 minutes, then double check that the ethernet is connected correctly, and that you are in a network which assigns an automatic IP address via DHCP. + ::: + +4. Bootstrap the image using the detected device in the previous scan + + ``` + c8y tedge bootstrap root@rpi5-d83add9a145a.local + ``` + +5. If you have Wi-Fi on your device, then you can also connect to it so that you can remove the ethernet cable + + The `ncmli` command can be issued via ssh to configure and connect the device to a given Wi-Fi network: + + ```sh + ssh root@mydevice.local nmcli device wifi connect "" password "" + ``` + + Where `` is the Access Point Name (e.g. SSID) of the network, and the `` is the password required to connect to the access point. diff --git a/versioned_docs/version-1.1.0/extend/firmware-management/flashing-an-image.md b/versioned_docs/version-1.1.0/extend/firmware-management/flashing-an-image.md new file mode 100644 index 0000000..1c381e3 --- /dev/null +++ b/versioned_docs/version-1.1.0/extend/firmware-management/flashing-an-image.md @@ -0,0 +1,50 @@ +--- +title: Flashing an image +tags: [Extend, Build] +sidebar_position: 4 +--- + +## Flashing an image using Raspberry Pi Imager + +The [Raspberry Pi Imager](https://www.raspberrypi.com/software/) software can also be used to flash either official Raspberry Pi Operating System version or custom built images to an SD Card. You can use this for both Raspberry PI and non-Raspberry Pi devices. + +### Pre-requisites + +To following these instructions you will need to have the following software installed: + +* [Raspberry Pi Imager](https://www.raspberrypi.com/software/) +* A custom built image + +### Procedure + +1. Open up the [Raspberry Pi Imager](https://www.raspberrypi.com/software/) + + ![Open Raspberry Pi Imager](./images/raspberrypi-imager-open.png) + +2. Select the **Use custom** option + + ![Use custom](./images/raspberrypi-imager-select-image.png) + +3. Browser to the `*.sdimg.bz2` image that you wish to use + + ![Select Image](./images/raspberrypi-imager-selected-file.png) + + :::tip + You may need to change the file filter to use the "All Files" option if you intended image file is not selectable in the select file dialog. + ::: + +3. Choose the storage where the image will be written to + + ![Select Image](./images/raspberrypi-imager-select-storage.png) + +4. Select **Write** and if prompted don't include any customization settings + + ![Select Image](./images/raspberrypi-imager-writing.png) + + :::caution + You can not use any Raspberry Pi Imager customization settings, as the Yocto image does not know what do with the settings, plus everything you need should be included in the image (even initial Wifi credentials). But if you use an ethernet connection, then you don't need to worry about Wifi credentials until you have bootstrapped the device. + ::: + +5. Wait for the image to be completed + + ![Select Image](./images/raspberrypi-imager-write-completed.png) diff --git a/versioned_docs/version-1.1.0/extend/firmware-management/images/raspberrypi-imager-open.png b/versioned_docs/version-1.1.0/extend/firmware-management/images/raspberrypi-imager-open.png new file mode 100644 index 0000000..2fd8a4e Binary files /dev/null and b/versioned_docs/version-1.1.0/extend/firmware-management/images/raspberrypi-imager-open.png differ diff --git a/versioned_docs/version-1.1.0/extend/firmware-management/images/raspberrypi-imager-select-image.png b/versioned_docs/version-1.1.0/extend/firmware-management/images/raspberrypi-imager-select-image.png new file mode 100644 index 0000000..ec4636e Binary files /dev/null and b/versioned_docs/version-1.1.0/extend/firmware-management/images/raspberrypi-imager-select-image.png differ diff --git a/versioned_docs/version-1.1.0/extend/firmware-management/images/raspberrypi-imager-select-storage.png b/versioned_docs/version-1.1.0/extend/firmware-management/images/raspberrypi-imager-select-storage.png new file mode 100644 index 0000000..39624f1 Binary files /dev/null and b/versioned_docs/version-1.1.0/extend/firmware-management/images/raspberrypi-imager-select-storage.png differ diff --git a/versioned_docs/version-1.1.0/extend/firmware-management/images/raspberrypi-imager-selected-file.png b/versioned_docs/version-1.1.0/extend/firmware-management/images/raspberrypi-imager-selected-file.png new file mode 100644 index 0000000..7d68cc3 Binary files /dev/null and b/versioned_docs/version-1.1.0/extend/firmware-management/images/raspberrypi-imager-selected-file.png differ diff --git a/versioned_docs/version-1.1.0/extend/firmware-management/images/raspberrypi-imager-write-completed.png b/versioned_docs/version-1.1.0/extend/firmware-management/images/raspberrypi-imager-write-completed.png new file mode 100644 index 0000000..775dc86 Binary files /dev/null and b/versioned_docs/version-1.1.0/extend/firmware-management/images/raspberrypi-imager-write-completed.png differ diff --git a/versioned_docs/version-1.1.0/extend/firmware-management/images/raspberrypi-imager-writing.png b/versioned_docs/version-1.1.0/extend/firmware-management/images/raspberrypi-imager-writing.png new file mode 100644 index 0000000..361bbb3 Binary files /dev/null and b/versioned_docs/version-1.1.0/extend/firmware-management/images/raspberrypi-imager-writing.png differ diff --git a/versioned_docs/version-1.1.0/extend/firmware-management/index.md b/versioned_docs/version-1.1.0/extend/firmware-management/index.md new file mode 100644 index 0000000..14ddc5f --- /dev/null +++ b/versioned_docs/version-1.1.0/extend/firmware-management/index.md @@ -0,0 +1,11 @@ +--- +title: Firmware Management +tags: [Extend] +sidebar_position: 1 +--- + +import DocCardList from '@theme/DocCardList'; + +Instructions on how to build different Operation System images with %%te%% setup to perform Over-the-Air (OTA) updates of the operating system. + + \ No newline at end of file diff --git a/versioned_docs/version-1.1.0/extend/index.md b/versioned_docs/version-1.1.0/extend/index.md new file mode 100644 index 0000000..8404c9c --- /dev/null +++ b/versioned_docs/version-1.1.0/extend/index.md @@ -0,0 +1,12 @@ +--- +title: Extend thin-edge.io +tags: [Extend] +sidebar_position: 4 +--- + +import DocCardList from '@theme/DocCardList'; + +This section is for the agent developers who want to adapt and extend %%te%% +to meet the requirements of their domain application, target devices and cloud. + + \ No newline at end of file diff --git a/versioned_docs/version-1.1.0/extend/software-management.md b/versioned_docs/version-1.1.0/extend/software-management.md new file mode 100644 index 0000000..cbc560c --- /dev/null +++ b/versioned_docs/version-1.1.0/extend/software-management.md @@ -0,0 +1,421 @@ +--- +title: Software Management +tags: [Extend, Software Management] +sidebar_position: 1 +description: How to create a custom software management plugin +--- + +%%te%% Software Management natively supports APT (Debian) packages. +However, there are many package management systems in the world, +and you may want to have a plugin that is suitable for your device. +For such a demand, we provide the [**Package Manager Plugin API**](./../references/software-management-plugin-api.md) +to write a custom Software Management plugin in your preferred programming language. + +In this tutorial, we will look into the **Package Manager Plugin API**, +and learn how to write your own plugin with a docker plugin shell script example. + +## Create a plugin + +Create a file called `docker` in the directory `/etc/tedge/sm-plugins/`. +A plugin must be an executable file located in that directory. + +```sh title="file: /etc/tedge/sm-plugins/docker" +#!/bin/sh + +COMMAND="$1" +IMAGE_NAME="$2" + +case "$COMMAND" in + list) + docker image list --format '{{.Repository}}\t{{.Tag}}' || exit 2 + ;; + install) + docker pull $IMAGE_NAME || exit 2 + ;; + remove) + docker rmi $IMAGE_NAME || exit 2 + ;; + prepare) + ;; + finalize) + ;; + update-list) + exit 1 + ;; +esac +exit 0 +``` + +:::info +The plugin filename will be used as a plugin type to report the software list to a cloud. +If you name it `docker.sh`, you will see `docker.sh` as a plugin type in cloud. +::: + +If you execute `./docker list`, you will see this kind of output. + +```csv +alpine 3.14 +eclipse-mosquitto 2.0-openssl +... +``` + +The Software Management Agent runs executable plugins with a special argument, like `list`. +Let's call the pre-defined argument such as `list`, `install`, and `remove` a **command** here. +As you can see from this example, a plugin should be an executable file +that accepts the commands and outputs to stdout and stderr. +Hence, you can implement a plugin in your preferred language. + +Here is the table of the commands that you can use in a plugin. + +|Command|Input arguments|Expected output|Description| +|---|---|---|---| +|list| - | lines with tab separated values |Returns the list of software modules that have been installed with this plugin.| +|prepare| - | - |Executes the provided actions before a sequence of install and remove commands.| +|finalize| - | - |Executes the provided actions after a sequence of install and remove commands.| +|install| `NAME [--module-version VERSION] [--file FILE]` | - |Executes the action of installation.| +|remove| `NAME [--module-version VERSION]` | - |Executes the action of uninstallation.| +|update-list| `COMMAND NAME [--module-version VERSION] [--file FILE]` | - |Executes the list of `install` and `remove` commands.| + +The order of the commands invoked by the Software Management Agent is: + +```mermaid +graph LR + prepare --> update-list --> install/remove --> finalize +``` + +:::info +There is no guarantee of the order between `install` and `remove`. +If you need a specific order, use the `update-list` command instead. +::: + +In the following sections, we will dive into each command and other rules deeply. + +## Input, Output, and Errors + +Before we dive into each command, we should clarify the basic rules of plugins. + +### Input + +The command themselves and further required arguments must be given as command-line arguments. +The only exception is `update-list`, which requires **stdin** input. + +### Output + +The **stdout** and **stderr** of the process running a plugin command are captured by the Software Management Agent. + +### Exit status + +The exit status of plugins are interpreted by sm-agent as follows: + +|Exit Code|Summary|Description| +|---------|-------|-----------| +|0|Success|The command executed successfully without errors| +|1|Invalid usage|The command arguments cannot be interpreted, and the command has not been launched| +|2|Failure|The command failed and there is no point to retry| +|3|Retry|The command failed but might be successful later (for instance, when the network will be back)| + +## List + +The `list` command is responsible to return the list of the installed software modules. + +Rules: + +- This command takes no arguments. +- The list is returned using [CSV with tabulations as separators](https://en.wikipedia.org/wiki/Tab-separated_values), + including: + - **name**: the name of the software module, e.g. `mosquitto`. + This name is the name that has been used to install it and that needs to be used to remove it. + - **version**: the version currently installed. This is a string that can only be interpreted in the context of the plugin. + + :::info + If the version is not present for a module, then list can return only the module name without trailing tabulation. + Given that your plugin is named `docker`, then the Software Management Agent calls + ::: + +```sh +sudo /etc/tedge/sm-plugins/docker list +``` + +to report the list of software modules installed. + +:::caution +The Software Management Agent executes the plugin commands using `sudo` using the `tedge` user. +::: + +`docker` should output in the CSV with tabulations as separators like + +```csv +alpine 3.14 +eclipse-mosquitto 2.0-openssl +rust 1.51-alpine +``` + +with exit code `0` (successful). + +In most cases, the output of the `list` command is multi-lines. +The line separator should be `\n`. + +A plugin must return a CSV line per software module, using a tabulation `\t` as separator. +If there is no version field then only the module name will be returned. +In the _docker_ file example, the following command outputs CSV structures with tabulations as separator. + +```sh +docker image list --format '{{.Repository}}\t{{.Tag}}' +``` + +## Prepare + +The `prepare` command is invoked by the sm-agent before a sequence of install and remove commands. + +Rules: + +- It takes no argument and no output is expected. +- If the `prepare` command fails, + then the whole Software Management operation is cancelled. + +For many plugins, this command has nothing specific to do, and can simply return with a `0` exit status. + +In some plugin types, this `prepare` command can help you. +For example, assume that you want to implement a plugin for APT, +and want to run `apt-get update` always before calling the `install` command. +In this example, the `prepare` command is the right place to invoke `apt-get update`. + + +## Finalize + +The `finalize` command closes a sequence of install and removes commands started by a prepare command. + +Rules: + +- It takes no argument and no output is expected. +- If the `finalize` command fails, then the whole Software Management operation is reported as failed, + even if all the atomic actions have been successfully completed. + +Similar to the `prepare` plugin, you must define the command even if you want nothing in the `finalize` command. + +The command can be used in several situations. For example, +- remove any unnecessary software module after a sequence of actions. +- commit or roll back the sequence of actions. +- restart any processes using the modules, + e.g. restart the analytics engines if the modules have changed. + + +## Install + +The `install` command installs a software module, possibly of some expected version. +A plugin must be executable in the below format. + +```sh +myplugin install NAME [--module-version VERSION] [--file FILE] +``` + +This command takes 1 mandatory argument and has 2 optional flags. +- **NAME**: the name of the software module to be installed, e.g. `mosquitto`. (Mandatory) +- **VERSION**: the version to be installed. e.g. `1.5.7-1+deb10u1`. + The version can be blank, so it's recommended to define the behaviour if a version is not provided. + For example, always installs the "latest" version if a version is not provided. (Optional) +- **FILE**: the path to the software to be installed. (Optional) + +The installation phase may fail due to the following reasons. +An error must be reported if: +- The module name is unknown. +- There is no version for the module that matches the constraint provided by the `--module-version` option. +- The file content provided by `--file` option: + - is not in the expected format, + - doesn't correspond to the software module name, + - has a version that doesn't match the constraint provided by the `--module-version` option (if any). +- The module cannot be downloaded. +- The module cannot be installed. + +At the API level, there is no command to distinguish install or upgrade. + +Back to the first _docker_ example, it doesn't address the case with version. +Let's expand the example file as below. + +```sh title="file: /etc/tedge/sm-plugins/docker" +#!/bin/sh + +COMMAND="$1" +IMAGE_NAME="$2" +VERSION_FLAG="$3" +IMAGE_TAG="$4" + +case "$COMMAND" in + list) + docker image list --format '{{.Repository}}\t{{.Tag}}' || exit 2 + ;; + install) + if [ $# -eq 2 ]; then + docker pull $IMAGE_NAME || exit 2 + elif [ $# -eq 4 ] && [ $VERSION_FLAG = "--module-version" ]; then + docker pull $IMAGE_NAME:$IMAGE_TAG || exit 2 + else + echo "Invalid arguments" + exit 1 + fi + ;; + remove) + if [ $# -eq 2 ]; then + docker rmi $IMAGE_NAME || exit 2 + elif [ $# -eq 4 ] && [ $VERSION_FLAG = "--module-version" ]; then + docker rmi $IMAGE_NAME:$IMAGE_TAG || exit 2 + else + echo "Invalid arguments" + exit 1 + fi + ;; + prepare) + ;; + finalize) + ;; + update-list) + exit 1 + ;; +esac +exit 0 +``` + +Pay attention to the exit statuses. +In case of invalid arguments, the plugin returns `1`. +If a command is executed but fails, the plugin returns `2`. +Each exit status is defined [here](#exit-status). + +If the given NAME is `mosquitto`, and the given VERSION is `1.5.7-1+deb10u1`, +the Software Management Agent calls + +```sh +sudo /etc/tedge/sm-plugins/docker install mosquitto --module-version 1.5.7-1+deb10u1 +``` + +Then, the plugin executes + +```sh +docker pull mosquitto:1.5.7-1+deb10u1 +``` + +## Remove + +The `remove` command uninstalls a software module, +and possibly its dependencies if no other modules are dependent on those. +A plugin must be executable in the below format. + +```sh +myplugin remove NAME [--module-version VERSION] +``` + +This command takes 1 mandatory argument and 1 optional argument with a flag. + +- **NAME**: the name of the software module to be removed, e.g. `mosquitto`. (Mandatory) +- **VERSION**: the version to be installed. e.g. `1.5.7-1+deb10u1`. + The version can be blank, so it's recommended to define the behaviour if a version is not provided. + For example, uninstall a software module regardless of its version if a version is not provided. (Optional) + +The uninstallation phase can be failed due to several reasons. An error must be reported if: +- The module name is unknown. +- The module cannot be uninstalled. + +Back to the first _docker_ plugin example, +if the NAME is `mosquitto`, and the VERSION is `1.5.7-1+deb10u1`, +the Software Management Agent calls + +```sh +sudo /etc/tedge/sm-plugins/docker remove mosquitto --module-version 1.5.7-1+deb10u1 +``` + +Then, the plugin executes + +```sh +docker rmi mosquitto:1.5.7-1+deb10u1 +``` + +## Update-list + +The `update-list` command accepts a list of software modules and associated operations as `install` or `remove`. +This basically achieves the same purpose as original commands install and remove, +but gets passed all software modules to be processed in one command. +This can be needed when an order of processing software modules is relevant. + +In other words, you can choose a combination of the `install` or `remove` commands or this `update-list` command up to your requirement. +If you don't want to use `update-list`, the plugin must return `1` like the first _docker_ plugin example. + +```sh +case "$COMMAND" in + ... + update-list) + exit 1 + ;; +esac +``` + +Let's expand the first _docker_ plugin example to use `update-list`. +First, learn what is the input of `update-list`. + +The Software Management Agent calls a plugin as below. Note that each argument is tab separated: + +```sh +sudo /etc/tedge/sm-plugins/docker update-list < +
+ + + +
+ + +|Package Manager|Format|Distributions| +|---------------|------|-------------| +|apt|deb|Debian, Ubuntu and other debian-based operating systems| +|yum/dnf/microdnf|rpm|RHEL, RockyLinux, AlmaLinux, Fedora| +|zypper|rpm|openSUSE| +|apk|apk|Alpine Linux| +|-|tarball (*.tar.gz)|All other Linux distributions, e.g. Yocto| + + +## Alternative installation methods + +In cases were you would not like to run the automatic install script, you can choose one to run the steps manually. This allows you more control over the process which can be useful if you are experiencing problems with the auto detection used in the install script. + +### Manual repository setup and installation + +The software repositories used by the package managers can be configured using the setup scripts. These scripts are normally executed by the *install.sh* script in the installation section, however they can also be manually executed if you want more fine-grain control over the process. + +:::tip +If you are having problems setting any of the repositories, check out the [Cloudsmith](https://cloudsmith.io/~thinedge/repos/tedge-release/setup/#formats-deb) website where they have **Set Me Up** instructions in additional formats, e.g. manual configuration rather than via the `setup.*.sh` script. +::: + +**Pre-requisites** + +The instructions require you to have the following tools installed. + +* curl +* bash + +#### Setup + +**Running with sudo** + +You will need to have `sudo` also installed if you want to run these instructions. + +```sh tab={"label":"Debian/Ubuntu"} +curl -1sLf 'https://dl.cloudsmith.io/public/thinedge/tedge-release/setup.deb.sh' | sudo bash +``` + +```sh tab={"label":"RHEL/Fedora/RockyLinux"} +curl -1sLf 'https://dl.cloudsmith.io/public/thinedge/tedge-release/setup.rpm.sh' | sudo bash +``` + +```sh tab={"label":"Alpine"} +curl -1sLf 'https://dl.cloudsmith.io/public/thinedge/tedge-release/setup.alpine.sh' | sudo bash +``` + +**Running as root** + +These commands must be run as the root user. + +```sh tab={"label":"Debian/Ubuntu"} +curl -1sLf 'https://dl.cloudsmith.io/public/thinedge/tedge-release/setup.deb.sh' | bash +``` + +```sh tab={"label":"RHEL/Fedora/RockyLinux"} +curl -1sLf 'https://dl.cloudsmith.io/public/thinedge/tedge-release/setup.rpm.sh' | bash +``` + +```sh tab={"label":"Alpine"} +curl -1sLf 'https://dl.cloudsmith.io/public/thinedge/tedge-release/setup.alpine.sh' | bash +``` + + +#### Installing and updating using a package manager + +Once you have the repository setup, you can install the **tedge-full** virtual package which will automatically pull in all of the %%te%% packages. This makes it easier to install and update in the future, as you only have to type in one package name, `tedge-full`. + +```sh tab={"label":"Debian/Ubuntu"} +sudo apt-get update +sudo apt-get install tedge-full +``` + +```sh tab={"label":"RHEL/Fedora/RockyLinux"} +sudo dnf install --best tedge-full --refresh +``` + +```sh tab={"label":"Alpine"} +sudo apk update +sudo apk add --no-cache tedge-full +``` + +### Install via tarball + +You can force the install.sh script to install via the tarball instead of via a package manager. The install script will also take care of the required post installation steps. + +To install the %%te%% via the tarball run the following command: + + +```sh tab={"label":"curl"} +curl -fsSL https://thin-edge.io/install.sh | sh -s -- --package-manager tarball +``` + +```sh tab={"label":"wget"} +wget -O - https://thin-edge.io/install.sh | sh -s -- --package-manager tarball +``` + +## Package repository hosting + +[_![Hosted By: Cloudsmith](https://img.shields.io/badge/OSS%20hosting%20by-cloudsmith-blue?logo=cloudsmith&style=for-the-badge)_](https://cloudsmith.com) + +Package repository hosting is graciously provided by [Cloudsmith](https://cloudsmith.com). +Cloudsmith is the only fully hosted, cloud-native, universal package management solution, that +enables your organization to create, store and share packages in any format, to any place, with total +confidence. + +The packages can be viewed directly from the [Cloudsmith.io](https://cloudsmith.io/~thinedge/repos/) website. + + + + + + + + + + + + + + + + + + +
LinuxRepository
+ Debian logo + + Latest version of 'tedge-full' @ Cloudsmith +
+ RedHat logo + + Latest version of 'tedge-full' @ Cloudsmith +
+ Alpine logo + + Latest version of 'tedge-full' @ Cloudsmith +
diff --git a/versioned_docs/version-1.1.0/legacy/child-device-configuration-management.md b/versioned_docs/version-1.1.0/legacy/child-device-configuration-management.md new file mode 100644 index 0000000..2b8f755 --- /dev/null +++ b/versioned_docs/version-1.1.0/legacy/child-device-configuration-management.md @@ -0,0 +1,223 @@ +--- +title: Child Device Configuration Management +tags: [Cumulocity, Configuration, Legacy] +sidebar_position: 6 +description: Legacy configuration management for child devices using Cumulocity IoT +--- + +# Enable configuration management on child devices + +Configuration management can be enabled for child devices using the same `c8y-configuration-plugin`, +used for configuration management on %%te%% devices. +But, an additional piece of software must be developed by the child device owner, +to coordinate configuration management operations on it with the `c8y-configuration-plugin` running on %%te%%. +This software is referred to as "child device agent" in the rest of this document. + +The child device agent needs to handle the following responsibilities: + +* Declare the supported configuration list to %%te%% +* Handle configuration snapshot requests from %%te%% +* Handle configuration update requests from %%te%% + +The *supported configuration list* is the list of configuration files on the child device +that needs to be managed from the cloud. +Configuration management by %%te%% is enabled *only* for the files provided in this list. +These declared configuration files can be fetched from %%te%% with config snapshot requests +and can be updated with config update requests. + +Handling the above mentioned responsibilities involve multiple interactions with %%te%% +over MQTT to receive and respond to configuration management requests, +and HTTP to upload/download files while handling those requests. + +For example, during the bootstrapping/startup of the child device, +the agent needs to upload the supported configuration list of the child device to %%te%% +by uploading a file using the HTTP `file-transfer` API of %%te%%, +followed by an MQTT message informing %%te%% that the upload completed. +Similarly, handling of a configuration snapshot or update request involves sending +MQTT messages before and after the configuration file is uploaded/downloaded via HTTP to/from %%te%%. + +Since child device agents typically run on an external device and not on the %%te%% device itself, +the MQTT and HTTP APIs of %%te%% need to be accessed over the network using its IP address, +which is configured using the tedge configuration settings `mqtt.external.bind.address` or `mqtt.bind.address`. +The MQTT APIs are exposed via port 1883 and the HTTP APIs are exposed via port 8000. +In rare cases, where the child device agent is installed alongside %%te%% on the same device, +these APIs can be accessed via a local IP or even `127.0.0.1`. + +The following sections explain the child device agent responsibilities in detail. + +## Declare supported configuration list to thin-edge + +The supported configuration list should be sent to %%te%% during the startup/bootstrap phase of the child device agent. +This bootstrapping is a 3 step process: + +1. Prepare a `c8y-configuration-plugin.toml` file with the supported configuration list +1. Upload this file to %%te%% via HTTP +1. Notify %%te%% about the upload via MQTT + +The child device agent needs to capture the list of configuration files that needs be managed from the cloud +in a `c8y-configuration-plugin.toml` file in the same format as specified in the [configuration management documentation](../operate/c8y/configuration-management.md) as follows: + +```toml title="file: c8y-configuration-plugin.toml" +files = [ + {path = '/path/to/some/config', type = 'config1'}, + {path = '/path/to/another/config', type = 'config2'}, +] +``` + +* `path` is the full path to the configuration file on the child device file system. +* `type` is a unique alias for each file entry which will be used to represent that file in Cumulocity UI + +The child device agent needs to upload this file to %%te%% [File Transfer Service](../references/file-transfer-service.md) with an HTTP PUT request +to the URL: `http://{fts-address}:8000/tedge/file-transfer/{child-id}/c8y-configuration-plugin` + +* `{fts-address}` is the address of the %%te%% device on which the [File Transfer Service](../references/file-transfer-service.md) is running +* `{child-id}` is the child-device-id + +Once the upload is complete, the agent should notify %%te%% about the upload by sending the following MQTT message: + +```sh te2mqtt formats=v1 +tedge mqtt pub 'tedge/{child-d}/commands/res/config_snapshot' '{"type": "c8y-configuration-plugin", "path": "/child/local/fs/path"}' +``` + +## Handle config snapshot requests from thin-edge + +Handling config snapshot requests from %%te%% is a 4-step process: + +1. Subscribe to, and receive config snapshot requests via MQTT +1. Send an "executing" operation status update to acknowledge the receipt of the request via MQTT +1. Upload the requested config file to the URL received in the request via HTTP +1. Send a "successful" operation status update via MQTT + +These steps are explained in detail below: + +The child device agent must subscribe to the `tedge/{child-d}/commands/req/config_snapshot` MQTT topic +to receive the config snapshot requests from %%te%%. +These requests arrive in the following JSON format: + +```json +{ + "type": "{config-type}", + "path": "/child/local/fs/path", + "url": "http://{fts-address}:8000/tedge/file-transfer/{child-d}/config_snapshot/{config-type}" +} +``` + +The `type` and `path` fields are the same values that the child device sent to %%te%% in its `c8y-configuration-plugin.toml` file. +The `url` value is what the child device agent must use to upload the requested config file. + +On receipt of the request, the agent must send an "executing" MQTT status message as follows: + +```sh te2mqtt formats=v1 +tedge mqtt pub tedge/{child-d}/commands/res/config_snapshot '{ + "status": "executing", + "type": "{config-type}", + "path": "/child/local/fs/path" +}' +``` + +After sending this status message, the agent must upload the requested configuration file content to +the `url` received in the request with an HTTP PUT request. + +Once the upload is complete, send a "successful" MQTT status message as follows: + +**Topic** + +```text +tedge/{child-d}/commands/res/config_snapshot +``` + +**Payload** + +```json +{ + "status": "successful", + "type": "{config-type}", + "path": "/child/local/fs/path" +} +``` + +If there are any failures while reading or uploading the requested config file, +a "failed" status update must be sent instead, to the same topic as follows: + +```json +{ + "status": "failed", + "type": "{config-type}", + "path": "/child/local/fs/path" +} +``` + +## Handle config update requests from thin-edge + +Handling config update requests from %%te%% is a 5-step process: + +1. Subscribe to, and receive config update requests via MQTT +1. Send an "executing" operation status update to acknowledge the receipt of the request via MQTT +1. Download the config file update from the URL received in the request via HTTP +1. Apply the config file update on the child device +1. Send a "successful" operation status update via MQTT + +The child device agent must subscribe to the `tedge/{child-d}/commands/req/config_update` MQTT topic +to receive the config update requests from %%te%%. +These requests arrive in the following JSON format: + +```json +{ + "type": "{config-type}", + "path": "/child/local/fs/path", + "url": "http://{fts-address}:8000/tedge/file-transfer/{child-d}/config_update/{config-type}" +} +``` + +The child device agent must download the config file update for the given `type` from %%te%% using the `url`. + +On receipt of the request, the agent must send an "executing" MQTT status message as follows: + +**Topic** + +```text +tedge/{child-d}/commands/res/config_update +``` + +**Payload** + +```json +{ + "status": "executing", + "type": "{config-type}", + "path": "/child/local/fs/path" +} +``` + +After sending this status message, the agent must download the configuration file update +from the `url` received in the request with an HTTP GET request. +The agent can then apply the downloaded configuration file update on the device. + +Once the update is applied, send a "successful" MQTT status message as follows: + +**Topic** + +```text +tedge/{child-d}/commands/res/config_update +``` + +**Payload** + +```json +{ + "status": "successful", + "type": "{config-type}", + "path": "/child/local/fs/path" +} +``` + +If there are any failures while downloading and applying the update, +a "failed" status update must be sent instead, to the same topic as follows: + +```json +{ + "status": "failed", + "type": "{config-type}", + "path": "/child/local/fs/path" +} +``` diff --git a/versioned_docs/version-1.1.0/legacy/child-device-firmware-management.md b/versioned_docs/version-1.1.0/legacy/child-device-firmware-management.md new file mode 100644 index 0000000..0dd68e6 --- /dev/null +++ b/versioned_docs/version-1.1.0/legacy/child-device-firmware-management.md @@ -0,0 +1,439 @@ +--- +title: Child Device Firmware Management +tags: [Cumulocity, Firmware, Legacy] +sidebar_position: 9 +description: Legacy child device firmware management using Cumulocity IoT +--- + +# Child-Device Firmware Management using Cumulocity (legacy API) + +%%te%% provides a legacy operation plugin to +[manage device firmware using Cumulocity](https://cumulocity.com/guides/users-guide/device-management/#firmware-repo) +on child devices. + +:::caution +- This operation plugin only supports firmware update on child devices and not on the main tedge device. +- This is a legacy API. For new developments, the recommended approach is to implement a [custom workflow](../references/agent/operation-workflow.md). +::: + +Firmware management can be enabled for child devices by installing the `c8y-firmware-plugin` on the main device, along the Cumulocity mapper. +This legacy %%te%% plugin coordinates the firmware update operation handling with Cumulocity, +by establishing secure communication with the cloud, +managing firmware file downloads, which are typically large files, even over flaky networks, +caching the downloaded files for re-use across multiple child devices etc. + +The `c8y-firmware-plugin`acts as the proxy between Cumulocity and the child device +facilitating the routing of firmware update requests as well as the transfer of firmware binary files from cloud to the device. + +For that to work, an additional piece of software must be provided by the child device owner as well. +The main responsibility of this piece of software is to handle the specificities of firmware updates on the child device +but also to coordinate the firmware installation on the device with the `c8y-firmware-plugin` running on %%te%%. +This software, which is not provided out-of-the-box, is referred to as the child-device connector in the rest of this document. + +This document describes: +- how to install, configure and use the `c8y-firmware-plugin` +- how to implement a child-device connector following the protocol of the `c8y-firmware-plugin` + +## c8y-firmware-plugin + +### Installation + +The `c8y-firmware-plugin` has to be installed on the main device and run as a daemon. + +The plugin will be installed at `/usr/bin/c8y-firmware-plugin` by the debian package. +The systemd service definition files for the plugin are also installed at `/lib/systemd/system/c8y-firmware-plugin.service`. +On systemd supported OSes, it can be run as a daemon service as follows: + +```sh +sudo systemctl enable c8y-firmware-plugin +sudo systemctl start c8y-firmware-plugin +``` + +No operations files are created under `/etc/tedge/operations/c8y/` +as this plugin doesn't support firmware updates for the main device. + +Child devices must be register the firmware update capability as part of their bootstrap process, +as [explained below](#registration). + +### Configuration + +Support for this plugin is disabled by default and must be explicitly enabled on the c8y mapper. + +```sh +sudo tedge config set c8y.enable.firmware_update true +sudo systemctl restart tedge-mapper-c8y.service +``` + +The plugin supports a single tedge configuration named `firmware.child.update.timeout`, +that defines the amount of time the plugin wait for a child device to finish a firmware update once the request is delivered. +The default timeout value (in seconds) is `3600` and can be updated with: + +```sh +sudo tedge config set firmware.child.update.timeout +``` + +### Usage + +```sh +c8y-firmware-plugin --help +``` + +```run command="c8y-firmware-plugin --help" lang="text" title="Output" +Thin-edge device firmware management for Cumulocity + +USAGE: + c8y-firmware-plugin [OPTIONS] + +OPTIONS: + --config-dir + [default: /etc/tedge] + + --debug + Turn-on the debug log level. + + If off only reports ERROR, WARN, and INFO If on also reports DEBUG + + -h, --help + Print help information + + -i, --init + Create required directories + + -V, --version + Print version information + +`c8y-firmware-plugin` subscribes to `c8y/s/ds` listening for firmware operation requests (message +`515`). +Notifying the Cumulocity tenant of their progress (messages `501`, `502` and `503`). +During a successful operation, `c8y-firmware-plugin` updates the installed firmware info in +Cumulocity tenant with SmartREST message `115`. + +The %%te%% `CONFIG_DIR` is used to find where: + * to store temporary files on download: `tedge config get tmp.path`, + * to log operation errors and progress: `tedge config get log.path`, + * to connect the MQTT bus: `tedge config get mqtt.bind.port`, + * to timeout pending operations: `tedge config get firmware.child.update.timeout +``` + +### Logging + +The following details are logged by the `c8y-firmware-plugin`: +* All the `c8y_Firmware` requests received from Cumulocity +* All the mapped `firmware_update` requests sent to each child device +* The `firmware_update` responses received from the child devices +* All errors are reported with the operation context + +### Cleanup + +To save bandwidth,the `c8y-firmware-plugin` downloads a single firmware file only once and keeps it cached for reuse +across multiple child devices, as firmware updates could be applied to a fleet of devices together. +The cached files are stored under the tedge data directory `/var/tedge/cache`, by default. +Since %%te%% does not know how many devices it will be reused for and for how long, it can not clean them up on its own. +So, the user must manually delete the cached firmware files once the update is complete on all child devices. + +## Child-Device Connector + +Handling firmware update requests on a child device is a 7-step process: + +1. Connect to the main device over MQTT and HTTP +1. Publish the capability to handle firmware updates. +1. Subscribe to, and receive firmware update requests via MQTT +1. Send an "executing" operation status update to acknowledge the receipt of the request via MQTT +1. Download the firmware file update from the URL received in the request via HTTP +1. Apply the firmware file update on the child device +1. Send a "successful" operation status update via MQTT + +### HTTP and MQTT Connection + +The child-device connector interacts with %%te%% over its MQTT and HTTP APIs. + +In cases where the child device connector is installed alongside %%te%% on the same device, +these APIs can be accessed via a local IP or even `127.0.0.1`. +The MQTT APIs are exposed via port 1883 and the HTTP APIs are exposed via port 8000, by default. +When the child device connector is running directly on the external child device, +the MQTT and HTTP APIs of %%te%% need to be accessed over the network using its IP address and ports, +which are configured using the tedge config settings `mqtt.client.host` or `mqtt.client.port` for MQTT +and `http.address` and `http.port` for HTTP. + +### Capability Registration {#registration} + +The child-device connector must register the capability to process software update requests for that child-device. + +This registration is done over MQTT, +by sending a retained message to the topic `te/device/${$CHILD_DEVICE_ID}///cmd/firmware_update` +where `$CHILD_DEVICE_ID` should be replaced with the child's identity. + +```sh te2mqtt formats=v1 +tedge mqtt pub --retain te/device/child-1///cmd/firmware_update '{}' +``` + +The Cumulocity mapper will detect the registration of these child device operation capabilities +and report them as supported operations for those child devices. + +### Request Subscription + +The child device connector must subscribe to the `tedge/{CHILD_ID}/commands/req/firmware_update` MQTT topic +to receive the firmware update requests from %%te%%. + +**Example** + +```sh +mosquitto_sub -h {TEDGE_DEVICE_IP} -t "tedge/{CHILD_ID}/commands/req/firmware_update" +``` + +These requests arrive in the following JSON format: + +**Payload** + +```json +{ + "id": "{OP_ID}", + "attempt": 1, + "name": "{FIRMWARE_NAME}", + "version": "{FIRMWARE_VERSION}", + "url":"http://{TEDGE_HTTP_ADDRESS}:{TEDGE_HTTP_PORT}/tedge/file-transfer/tedge-child/firmware_update/{FILE_ID}", + "sha256":"{FIRMWARE_FILE_SHA256}" +} +``` + +**Example** + +```json +{ + "id": "tLrLthPXksKbqRqIsrDmy", + "attempt": 1, + "name": "OpenWRT", + "version": "22.03", + "url":"http://127.0.0.1:8000/tedge/file-transfer/tedge-child/firmware_update/93d50a297a8c235", + "sha256":"c036cbb7553a909f8b8877d4461924307f27ecb66cff928eeeafd569c3887e29" +} +``` + +The fields in the request are describe below. + +|Property|Description| +|--------|-----------| +|id| A unique id for the request. All responses must be sent back with the same id.| +|attempt| This `attempt` count can be used to differentiate between fresh requests and re-sent requests, as this `attempt` count will be higher than `1` if the same request is resent from %%te%% on rare occasions (e.g: %%te%% gets restarted while the child device is processing a firmware request).| +|name| Name of the firmware package| +|version| Version of the firmware package| +|sha256| The SHA-256 checksum of the firmware binary served in the `url` which can be used to verify the integrity of the downloaded package post-download.| + +### Execution Notification + +On receipt of the request, the connector may optionally send an "executing" MQTT status message, +to acknowledge the receipt of the request, as follows: + +**Topic** + +```text +tedge/{CHILD_ID}/commands/res/firmware_update +``` + +**Payload** + +```json +{ + "id": "{OP_ID}", + "status": "executing" +} +``` + +where the `OP_ID` must match the `id` received in the `firmware_update` request. + +**Example** + +```sh +mosquitto_pub -h {TEDGE_DEVICE_IP} -t "tedge/{CHILD_ID}/commands/res/firmware_update" -m '{"id": "{OP_ID}", "status": "executing"}' +``` + +### Package Download + +After sending this status message, the connector must download the firmware file +from the `url` received in the request with an HTTP GET request. +Validating the integrity of the downloaded package by matching its SHA-256 hash value +against the `sha256` checksum received in the request is highly recommended. + +### Firmware Update + +The connector can then apply the downloaded firmware file update on the device. +This step is very device specific and might require use of other device specific protocols as well. + +### Success Notification + +Once the update is successfully applied, send a "successful" MQTT status message as follows: + +**Topic** + +```text +tedge/{CHILD_ID}/commands/res/firmware_update +``` + +**Payload** + +```json +{ + "id": "{OP_ID}", + "status": "successful" +} +``` + +**Example** + +```sh +mosquitto_pub -h {TEDGE_DEVICE_IP} -t "tedge/{CHILD_ID}/commands/res/firmware_update" -m '{ "id": "{OP_ID}", "status": "successful" }' +``` + +If there are any failures while downloading or applying the update, +a "failed" status update (with an optional `reason`) must be sent instead, to the same topic as follows: + +**Payload** + +```json +{ + "id": "{OP_ID}", + "status": "failed", + "reason": "Failure reason" +} +``` + +**Example** + +```sh +mosquitto_pub -h {TEDGE_DEVICE_IP} -t "tedge/{CHILD_ID}/commands/res/firmware_update" -m '{ "id": "{OP_ID}", "status": "failed", "reason": "SHA-256 checksum validation failed" }' +``` + +## Reference Implementation + +The reference implementation of a [child device connector](https://github.com/thin-edge/thin-edge.io_examples/tree/main/child-device-agent) +written in Python to demonstrate the contract described in this document. +This connector supports both configuration and firmware management operations. +So, just focus on the firmware management aspects. + +## Firmware Update Protocol + +The plugin manages the download and delivery of firmware files for child-devices connected to the %%te%% device, +acting as a proxy between the cloud and the child-devices. +The firmware updates are downloaded from the cloud on the main device then made available to the child-devices over HTTP. +The child devices are notified of incoming firmware update requests via MQTT. +The child-device connector has to subscribe to these MQTT messages, download the firmware files via HTTP, +and notify the firmware plugin of the firmware update progress via MQTT. + +* The responsibilities of the `c8y-firmware-plugin` are: + * to download the firmware files pushed from the cloud, caching it to be shared with child devices + * to handle network failures during the download even on flaky networks + * to publish the downloaded firmware files over a local HTTP server and make them available to the child-devices, + * to notify the child-devices when firmware updates are available, + * to receive forward the firmware update status updates from the child devices to the cloud +* By contrast, the `c8y-firmware-plugin` is not responsible for: + * checking the integrity of the downloaded file which is a third-party binary + * installing the firmware files on the child-devices. +* The child-device connector is required to listen for firmware update related MQTT notifications from the plugin + and behave accordingly along the protocol defined by this plugin. + * Being specific to each type of child device based on its device specific protocol for applying a firmware update. + * This software can be installed on the child device. + * This software can also be installed on the main device, + when the target device cannot be altered or connected to the main device over MQTT and HTTP. +* The child-device connector may be installed directly on the child device or alongside %%te%% as well, + as long as it can access the HTTP and MQTT APIs of %%te%% and interact with the child device directly. + +The following diagram captures the required interactions between all relevant parties: + +```mermaid +sequenceDiagram + participant C8Y Cloud + participant C8Y Firmware Plugin + participant Tedge Agent + participant Child Device Connector + + + C8Y Cloud->>C8Y Firmware Plugin: MQTT: c8y_Firmware request with `child-id` and `c8y.url` for the updated firmware file + C8Y Firmware Plugin->>C8Y Cloud: Download the firmware file from `c8y.url` to firmware cache + C8Y Firmware Plugin->>Tedge Agent: Symlink the cached firmware file to file-transfer repository and generate a `tedge.url` for it + C8Y Firmware Plugin->>Child Device Connector: MQTT: firmware_update request to `child-id` with `tedge.url` + + Child Device Connector ->> C8Y Firmware Plugin: MQTT: firmware_update response with operation status: "executing" + C8Y Firmware Plugin ->> C8Y Cloud: MQTT: Update c8y_Firmware operation status to "EXECUTING" + Child Device Connector->>Tedge Agent: HTTP: Download the firmware file from `tedge.url` + Child Device Connector->>Child Device Connector: Apply downloaded firmware file + Child Device Connector ->> C8Y Firmware Plugin: MQTT: firmware_update response with operation status: "successful" + + C8Y Firmware Plugin ->> C8Y Cloud: MQTT: Update c8y_Firmware operation status to "SUCCESSFUL" + C8Y Firmware Plugin ->> C8Y Firmware Plugin: Remove the symlink from file-transfer repository but keep the cached firmware copy for reuse +``` + +The following keywords are used in the following section for brevity: + +* `TEDGE_DATA_PATH`: The path set by tedge config `data.path`. Default: `/var/tedge` +* `TEDGE_TMP_PATH`: The path set by tedge config `tmp.path`. Default: `/tmp` +* `FIRMWARE_CACHE_PATH`: `$TEDGE_DATA_PATH/cache` +* `FIRMWARE_OP_PATH`: `$TEDGE_DATA_PATH/firmware` +* `FILE_TRANSFER_REPO`: `$TEDGE_DATA_PATH/file-transfer` +* `TEDGE_HTTP_ADDRESS`: The combination of tedge configs `http.address`:`http.port` +* `OP_ID`: An operation ID +* `FILE_ID`: A firmware file id derived from the SHA-256 digest of the firmware url + +1. The plugin, on reception of a `c8y_Firmware` request from Cumulocity for a child device named `$CHILD_DEVICE_ID` + in the SmartREST format `515,$CHILD_DEVICE_ID,$FIRMWARE_NAME,$FIRMWARE_VERSION,$FIRMWARE_URL` + 1. Validate if the same firmware update operation is already in progress + by iterating over all the operation files in the `$FIRMWARE_OP_PATH` directory. + The operation files contains the last `firmware_update` request's JSON payload along with the `device` ID. + If an operation file with the `child_id` id, `name`, `version` and `url` fields matching + the incoming `$FIRMWARE_NAME`,`$FIRMWARE_VERSION` and `$FIRMWARE_URL` is found, + the same request is re-sent to the child device by just incrementing the `attempt` count value. + The operation file content is also overwritten the with updated `attempt` count. + 1. If a pending operation match is not found, do a look up if the firmware file for the given url already exists + in its firmware cache at `$FIRMWARE_CACHE_PATH`. + The file name for the lookup is derived from the SHA-256 digest of the firmware url. + 1. If a cached copy is not found in the firmware cache, the plugin downloads the firmware file from the `url` + to `$FIRMWARE_CACHE_PATH` with the name derived from the SHA-256 digest of the firmware url. + If a cached firmware copy is found, downloading is skipped. + 1. Create an operation file at `$FIRMWARE_OP_PATH/$OP_ID` + with a JSON record containing the following fields: + * `operation_id`: A unique id generated by the plugin + * `child_id`: The child device ID received in the cloud request + * `name`: Name of the firmware received in the cloud request + * `version`: Version of the firmware received in the cloud request + * `server_url`: The firmware URL received in the cloud request + * `tedge_url`: The file-transfer service entry URL for the downloaded firmware file (`http://$TEDGE_HTTP_ADDRESS/tedge/file-transfer/$CHILD_DEVICE_ID/firmware_update/$FILE_ID`) + * `sha256`: The SHA-256 checksum of the firmware file served via the `tedge_url` + * `attempt`: The count that indicates if this request is being resent or not, with an initial value of `1` + 1. After creating the operation file, do a look up if the firmware file for the given url already exists +1. The cached firmware file is published via the file-transfer repository of `tedge-agent` + by creating a symlink to the cached firmware file is created in the file-transfer repository at + `$FILE_TRANSFER_REPO/$CHILD_DEVICE_ID/firmware_update/$FILE_ID` making this file available via + the HTTP endpoint: `http://$TEDGE_HTTP_ADDRESS/tedge/file-transfer/$CHILD_DEVICE_ID/firmware_update/$FILE_ID`. +1. Once the updated firmware file is published via the HTTP file transfer service, + the plugin send the `firmware_update` request to the child device connector by publishing an MQTT message: + * Topic: `tedge/$CHILD_DEVICE_ID/commands/req/firmware_update` + * The payload is a JSON record with the following fields + * `id`: A unique id generated by the plugin + * `name`: Name of the firmware received in the cloud request + * `version`: Version of the firmware received in the cloud request + * `url`: The file-transfer service entry URL(`http://$TEDGE_HTTP_ADDRESS/tedge/file-transfer/$CHILD_DEVICE_ID/firmware_update/$FILE_ID`) + * `sha256`: The SHA-256 checksum of the firmware file served via the `url` + * `attempt`: The count that indicates if this request is being resent or not, starting from `1` for the original request +1. On reception of the firmware update request on the topic `tedge/$CHILD_DEVICE_ID/commands/req/firmware_update`, + the child device connector is expected to do the following: + 1. Send an acknowledgement of the receipt of the request by sending an executing status message via MQTT: + * Topic: `tedge/$CHILD_DEVICE_ID/commands/res/firmware_update` + * Payload must be a JSON record with the following fields + * `id`: The `id` of the request + * `status`: "executing" + 1. `GET`s the firmware file from the `url` specified by the notification message. + 1. Validate the integrity of the downloaded binary by matching its SHA-256 hash value + against the `sha256` checksum value received in the request. + 1. Apply the downloaded firmware file update on the device using whatever device specific protocol. +1. After applying the update, send the final operation status update to %%te%% via MQTT: + 1. Topic: `tedge/$CHILD_DEVICE_ID/commands/res/firmware_update` + 1. The payload must be a JSON record with the following fields: + * `id`: The `id` of the request received + * `status`: `successful` or `failed` based on the result of updating the firmware + * `reason`: The reason for the failure, applicable only for `failed` status. +1. On reception of an operation status message, the plugin maps it to SmartREST and forwards it to the cloud. + * When a `successful` or `failed` status message is finally received, + then the plugin cleans up the corresponding operation file at `$FIRMWARE_OP_PATH/$OP_ID` and + the firmware file entry in the file transfer repository at `$FILE_TRANSFER_REPO/$CHILD_DEVICE_ID/firmware_update/$FILE_ID`. + * If a notification message is received while none is expected, + i.e with an operation `id` that doesn't exist at `$TEDGE_DATA_PATH/firmware/`, + then this notification message is deemed stale and ignored. diff --git a/versioned_docs/version-1.1.0/legacy/index.md b/versioned_docs/version-1.1.0/legacy/index.md new file mode 100644 index 0000000..06e1c33 --- /dev/null +++ b/versioned_docs/version-1.1.0/legacy/index.md @@ -0,0 +1,19 @@ +--- +title: Legacy +tags: [Legacy] +sidebar_position: 9 +--- + +import DocCardList from '@theme/DocCardList'; + +%%te%% 1.0 introduced a set of breaking changes that might affect plugins and extensions implemented on previous versions. + +In most cases, a compatibility layer has been introduced to smooth the transition. +For instance, any measurement published by an extension on the former topic `tedge/measurement` +is republished by the **tedge-agent** to the topic `te/device/main///m/` +which is dedicated to untyped measurements for the main device. + +However, the compatibility layers don't address all the breaking changes and, in any case, they will be deprecated medium-term. +Here are the developer guides to port a legacy extension to the new %%te%% API. + + \ No newline at end of file diff --git a/versioned_docs/version-1.1.0/legacy/mqtt-topics.md b/versioned_docs/version-1.1.0/legacy/mqtt-topics.md new file mode 100644 index 0000000..8b8a286 --- /dev/null +++ b/versioned_docs/version-1.1.0/legacy/mqtt-topics.md @@ -0,0 +1,244 @@ +--- +title: MQTT topics +tags: [MQTT, Legacy] +sidebar_position: 1 +description: Legacy %%te%% MQTT topics and how to map to the new topics +--- + +The most visible breaking change introduced by %%te%% 1.0 is the [new topic structure](../references/mqtt-api.md), +which has been made more consistent and extensible with a better support for child devices and services. + +Porting an extension that publishes telemetry data on the legacy `tedge` topic, should not pose any difficulty: +- *measurements* and *events* can be mapped directly to the new scheme by just changing the topics +- for *alarms*, the severity in the topic in the old scheme must be mapped to the payload in the new scheme +- handling of child devices is now consistent for *measurements*, *events* and *alarms*. + +## Backward compatibility + +The **tedge-agent** running on the main device implements a compatibility layer +and republishes on the new topics any message received on the legacy topics. + +Thanks to this compatibility mechanism a legacy extension works out of the box with %%te%% 1.0. + +However, this mechanism will be deprecated medium-term, and we encourage you to port any legacy extension to the new API. + +## Telemetry: Main device + +Here is the mapping between the legacy and new topics for the main device: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TypeTopicPayload Changes
Measurements +

Legacy

+ +```sh +tedge/measurements +``` + +

New

+ +```sh +te/device/main///m/ +``` + + 		+ No Change. If "<type>" is not provided, a default value of "ThinEdgeMeasurement" will be used. +
Events +

Legacy

+ +```sh +tedge/events/ +``` + +

New

+ +```sh +te/device/main///e/ +``` + + 		+ No Change +
Alarms +

Legacy

+ +```sh +tedge/alarms// +``` + +

New

+ +```sh +te/device/main///a/ +``` + + 		+ +The alarm severity should be set in the payload. + +```json5 +{ + "severity": "" + // ... +} +``` + +
Health status +

Legacy

+ +```sh +tedge/health/ +``` + +

New

+ +```sh +te//status/health +``` + + 		+ type property removed from the payload. +
+ + +## Telemetry: Child device + +Here is the mapping between the legacy and new topics for a child device: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TypeTopicPayload Changes
Measurements +

Legacy

+ +```sh +tedge/measurements/ +``` + +

New

+ +```sh +te/device////m/ +``` + + 		+ No Change. If the "<type>" is not provided, a default value of "ThinEdgeMeasurement" will be used. +
Events +

Legacy

+ +```sh +tedge/events// +``` + +

New

+ +```sh +te/device////e/ +``` + + 		+ No Change +
Alarms +

Legacy

+ +```sh +tedge/alarms/// +``` + +

New

+ +```sh +te/device////a/ +``` + + 		+ +The alarm severity should be set in the payload. + +```json5 +{ + "severity": "" + // ... +} +``` + +
Health status +

Legacy

+ +```sh +tedge/health// +``` + +

New

+ +```sh +te//status/health +``` + + 		+ type property removed from the payload. +
\ No newline at end of file diff --git a/versioned_docs/version-1.1.0/operate/c8y/apama.md b/versioned_docs/version-1.1.0/operate/c8y/apama.md new file mode 100644 index 0000000..389e975 --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/c8y/apama.md @@ -0,0 +1,97 @@ +--- +title: Apama +tags: [Operate, Cumulocity, Software Management] +description: Using the Apama plugin with %%te%% +--- + +The Apama plugin, part of the Apama %%te%% support package, can be used to install Apama projects using the Cumulocity software management feature. + +## Prerequisites + +The Apama %%te%% support package must be installed before you can install Apama projects on %%te%%. +The recommended way of installing Apama support on Debian based distributions is with the apt package manager. +Further details about this can be found in the %%te%% documentation at [Manage the software on your devices from Cumulocity cloud](../../start/software-management.md). + +#### Set up the Debian repository from which the Apama packages will be installed + +1. In the Cumulocity IoT tenant, open the **Device Management** app, go to the **Management** menu option and select the **Software repository**. +2. Click **Add software** at the right of the top menu bar. +3. In the **ADD SOFTWARE** dialog enter the following details: + - **Software**: apama-repo + - **Description**: apama-repo (or something else if you choose) + - **Device Filter Type**: (leave empty) + - **Software Type**: apt + - **Version**: 2022 + - **SOFTWARE FILE**: check the *Provide a file path* option and enter the URL: https://downloads.apamacommunity.com/debian/apama-repo_2022_all.deb + +4. Click the **Add Software** button. +5. Now select the **Devices** menu option and then select **All devices**. +6. In the list of devices, select the %%te%% device installed previously. +7. In the sub-menu for the device, select the **Software** option. +8. Click the **Install software** button in the bottom left; the apama-repo should be listed. +9. Click the drop-down arrow on the right and check the `2022` radio button, then click **Install**. +10. Finally, click the **Apply changes** button in the lower right of the panel. + +#### Add the Apama thin-edge support package to the tenant software repository and deploy it to thin-edge +1. Return to the **Device Management** app and go to the **Management** menu option and select the **Software repository**. +2. Click **Add software** at the right of the top menu bar. +3. In the **ADD SOFTWARE** dialog enter the following details: + - **Software**: apama-thin-edge-support + - **Description**: apama-thin-edge-support (or something else if you choose) + - **Device Filter Type**: (leave empty) + - **Software Type**: apt + - **Version**: latest + - **SOFTWARE FILE**: select the **Provide a file path** option and give an **empty space** (' '). +4. Click the **Add Software** button. +5. Now return to the **Devices** menu option and then select **All devices**. +6. In the list of devices, select the %%te%% device installed previously. +7. In the sub-menu for the device, select the **Software** option. +8. Click the **Install software** button in the bottom left; the apama-thin-edge-support should be listed. +9. Click the drop-down arrow on the right and check the `latest` radio button, then click **Install**. +10. Finally, click the **Apply changes** button in the lower right of the panel. + +After installation of the support for Apama onto the %%te%% device, the correlator will attempt to start, but will fail initially because there is not yet an Apama project installed. Instructions are given below for installing an Apama project on the device. + +## Install Apama projects from Cumulocity + +Before an Apama project can be installed on the device using the software management feature in Cumulocity, the project files need to be added to the Cumulocity software repository. + +There is a naming convention that you need to follow while creating software entries for Apama artifacts in the software repository. + +For Apama projects: + +1. The uploaded binary must be a `zip` file that contains the `project` directory. If a directory named `project` is not found by the plugin at the root level in the zip, it is considered invalid. + +

+ Add new apama project in Software Repository +

+ +Once the software modules have been added to the software repository, these can be installed on the device just like any other software from the `Software` tab of the device in the Cumulocity device UI. + +### Testing Apama Plugin + +Here is an Apama project that you can use to test this plugin. + +[project zip](https://github.com/thin-edge/thin-edge.io/raw/main/tests/PySys/plugin_apama/Input/quickstart.zip) + +Add this project as a software package in the Cumulocity software repository by following the instructions in the previous section. +Once added, this Apama project can be installed on any target device. +You can test if the project was successfully installed by running the following Apama command: + +```sh +/opt/softwareag/Apama/bin/apama_env engine_inspect -m +``` + +```text title="Example Output" +Monitors +======== +Name Num Sub Monitors +---- ---------------- +TedgeDemoMonitor 1 +``` + +You can find more information on this test Apama project [here](https://github.com/thin-edge/thin-edge.io_examples/tree/main/StreamingAnalytics#testing-a-project). diff --git a/versioned_docs/version-1.1.0/operate/c8y/configuration-management.md b/versioned_docs/version-1.1.0/operate/c8y/configuration-management.md new file mode 100644 index 0000000..12b46cd --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/c8y/configuration-management.md @@ -0,0 +1,65 @@ +--- +title: Configuration Management +tags: [Operate, Cumulocity, Configuration] +description: Managing configuration on devices +--- + +With %%te%%, you can manage config files on a device by using the [Cumulocity configuration management feature](https://cumulocity.com/guides/users-guide/device-management/#managing-configurations) as a part of Device Management. + +If you are new to the Cumulocity **Configuration Management** feature, +we recommend you to read [the Cumulocity user guide](https://cumulocity.com/guides/users-guide/device-management/#managing-configurations) along with this how-to guide. + +The configuration management functionality is provided by the **tedge-agent** service which is installed by default. + +## Getting started + +Before starting anything, make sure [your device is connected to Cumulocity](../../start/connect-c8y.md). + +**Step 1** +Open the file `/etc/tedge/plugins/tedge-configuration-plugin.toml` and add entries for the configuration files that you'd like to manage from Cumulocity cloud in the following format: + +```toml title="file: /etc/tedge/plugins/tedge-configuration-plugin.toml" +files = [ + { path = '/etc/tedge/tedge.toml', type = 'tedge.toml'}, + { path = '/etc/tedge/mosquitto-conf/c8y-bridge.conf', type = 'c8y-bridge.conf' }, + { path = '/etc/tedge/mosquitto-conf/tedge-mosquitto.conf', type = 'tedge-mosquitto.conf' }, + { path = '/etc/mosquitto/mosquitto.conf', type = 'mosquitto.conf' }, + { path = '/etc/tedge/c8y/example.txt', type = 'example', user = 'tedge', group = 'tedge', mode = 0o444 } +] +``` + +* `path` is the full path to the configuration file. +* `type` is a unique alias for each file entry which will be used to represent that file in Cumulocity UI. +* `user`, `group` and `mode` are UNIX file permission settings to be used to create a configuration file. If not provided, the files will be created with `root` user. If the file exists already, its ownership will be retained. + +For more details on this configuration file format, refer to the [reference guide](../../references/agent/tedge-configuration-management.md#configuration). + +:::note +You can also configure the `tedge-configuration-plugin.toml` from the cloud later. +::: + +**Step 2** +Navigate to your Cumulocity Device Management and the desired device. Open its **Configuration** tab. +You can find `tedge-configuration-plugin` and more are listed as supported configuration types, as declared in the plugin configuration file in step 1. + +![Cumulocity Configuration Management Upload](../../images/c8y-config-plugin-upload.png) + +This is the configuration file of `tedge-configuration-plugin`, where you can add file entries that you want to manage with Cumulocity. + +## Update tedge-configuration-plugin configuration from Cumulocity + +To update any configuration file, create a local copy of that config file and then upload that file to the [Cumulocity configuration repository](https://cumulocity.com/guides/users-guide/device-management/#to-add-a-configuration-snapshot) with the appropriate configuration type. + +The `tedge-configuration-plugin.toml` file can also be updated from the cloud in a similar manner to add/remove further configuration file entries. The updated TOML file has to be uploaded with the configuration type: **tedge-configuration-plugin**. + +Then, go back to the **Configuration** tab of your desired device in Cumulocity. + +![Cumulocity Configuration Management Download](../../images/c8y-config-plugin-download.png) + +Click on the config file entry from the **DEVICE SUPPORTED CONFIGURATIONS** files list. +You can choose the file that you uploaded from the **AVAILABLE SUPPORTED CONFIGURATIONS** section, and then apply that file to your device by clicking on the **Send configuration to device** button. + +After the operation created gets marked SUCCESSFUL, reload the page. +Then you can find new supported configuration types as you defined. + +To get to know more about the `tedge-configuration-plugin`, refer to [Specifications of Device Configuration Management using Cumulocity](../../references/agent/tedge-configuration-management.md). diff --git a/versioned_docs/version-1.1.0/operate/c8y/connect.md b/versioned_docs/version-1.1.0/operate/c8y/connect.md new file mode 100644 index 0000000..14ddcad --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/c8y/connect.md @@ -0,0 +1,317 @@ +--- +title: Connect +tags: [Operate, Cloud] +description: Connecting %%te%% to Cumulocity IoT +--- + +To create northbound connection a local bridge shall be established and this can be achieved with `tedge` cli and following commands: + +:::note +`tedge connect` requires root privileges, so you need to run it using `sudo` or run the command as root. +::: + +## Setting the cloud end-point + +Configure required parameters for %%te%% with [`tedge config set`](../../references/cli/tedge-config.md): + +```sh +sudo tedge config set c8y.url example.cumulocity.com +``` + +:::info +If you are unsure which parameters required by the command, simply run the command and it will tell you which parameters are missing. + +For example, if we issue [`tedge connect c8y`](../../references/cli/tedge-connect.md) without any configuration following advice will be given: + +```sh +sudo tedge connect c8y +``` + +```sh title="Output" +... +Error: failed to execute `tedge connect`. + +Caused by: + Required configuration item is not provided 'c8y.url', run 'tedge config set c8y.url ' to add it to config. +``` + +This message explains which configuration parameter is missing and how to add it to configuration, +in this case we are told to run `tedge config set c8y.url `. +::: + +## Making the cloud trust the device + +The next step is to have the device certificate trusted by Cumulocity. This is done by uploading the certificate of the signee. +You can upload the root certificate using the [Cumulocity UI](https://cumulocity.com/guides/users-guide/device-management/#trusted-certificates) +or with [`tedge cert upload`](../../references/cli/tedge-cert.md) command as described below. + +:::note +The `tedge cert upload` command requires the credentials of a Cumulocity user +having the permissions to upload trusted certificates on the Cumulocity tenant of the device. + +The user name is provided as `--user ` parameter, +and the command will prompt you for this user's password. +These credentials are used only for this upload and will in no case be stored on the device. +::: + +```sh +sudo tedge cert upload c8y --user "${C8Y_USER}" +``` + +```sh title="Example" +sudo tedge cert upload c8y --user "john.smith@example.com" +``` + +## Creating an MQTT bridge between the device and the cloud + +The connection from the device to the cloud is established using a so-called MQTT bridge: +a permanent secured bidirectional MQTT connection that forward messages back and forth +between the two end-points. + +To create the bridge use the [`tedge connect`](../../references/cli/tedge-connect.md) command. + +```sh +sudo tedge connect c8y +``` + +```text title="Output" +Checking if systemd is available. + +Checking if configuration for requested bridge already exists. + +Validating the bridge certificates. + +Creating the device in Cumulocity cloud. + +Saving configuration for requested bridge. + +Restarting mosquitto service. + +Awaiting mosquitto to start. This may take up to 5 seconds. + +Enabling mosquitto service on reboots. + +Successfully created bridge connection! + +Sending packets to check connection. This may take up to 2 seconds. + +Connection check is successful. + +Checking if tedge-mapper is installed. + +Starting tedge-mapper-c8y service. + +Persisting tedge-mapper-c8y on reboot. + +tedge-mapper-c8y service successfully started and enabled! + +Enabling software management. + +Checking if tedge-agent is installed. + +Starting tedge-agent service. + +Persisting tedge-agent on reboot. + +tedge-agent service successfully started and enabled! +``` + +## Errors + +### Connection already established + +If connection has already been established following error may appear: + +```sh +sudo tedge connect c8y +``` + +```text title="Output" +Checking if systemd is available. + +Checking if configuration for requested bridge already exists. + +Error: failed to create bridge to connect Cumulocity cloud. + +Caused by: + Connection is already established. To remove existing connection use 'tedge disconnect c8y' and try again. +``` + +To remove existing connection and create new one follow the advice and issue [`tedge disconnect c8y`](../../references/cli/tedge-disconnect.md): + +```sh +sudo tedge disconnect c8y +``` + +```text title="Output" +Removing Cumulocity bridge. + +Applying changes to mosquitto. + +Cumulocity Bridge successfully disconnected! + +Stopping tedge-mapper-c8y service. + +Disabling tedge-mapper-c8y service. + +tedge-mapper-c8y service successfully stopped and disabled! + +Stopping tedge-agent service. + +Disabling tedge-agent service. + +tedge-agent service successfully stopped and disabled! +``` + +:::note +`tedge disconnect c8y` also stops and disables the **tedge-mapper** service if it is installed on the device. +::: + +And now you can issue [`tedge connect c8y`](../../references/cli/tedge-connect.md) to create new bridge. + +### Connection check warning + +Sample output of tedge connect when this warning occurs: + +```sh +sudo tedge connect c8y +``` + +```text title="Output" +Checking if systemd is available. + +Checking if configuration for requested bridge already exists. + +Validating the bridge certificates. + +Creating the device in Cumulocity cloud. + +Saving configuration for requested bridge. + +Restarting mosquitto service. + +Awaiting mosquitto to start. This may take up to 5 seconds. + +Enabling mosquitto service on reboots. + +Successfully created bridge connection! + +Sending packets to check connection. This may take up to 2 seconds. + +ERROR: Local MQTT publish has timed out. +Warning: Bridge has been configured, but Cumulocity connection check failed. + +Checking if tedge-mapper is installed. + +Starting tedge-mapper-c8y service. + +Persisting tedge-mapper-c8y on reboot. + +tedge-mapper-c8y service successfully started and enabled! + +Enabling software management. + +Checking if tedge-agent is installed. + +Starting tedge-agent service. + +Persisting tedge-agent on reboot. + +tedge-agent service successfully started and enabled! +``` + +This warning may be caused by some of the following reasons: + +- No access to Internet connection + +Local bridge has been configured and is running but the connection check has failed due to no access to the northbound endpoint. + +- Cumulocity tenant not available + +Tenant couldn't be reached and therefore connection check has failed. + +- Check bridge + +Bridge configuration is correct but the connection couldn't be established to unknown reason. + +To retry start with [`tedge disconnect c8y`](../../references/cli/tedge-disconnect.md) removing this bridge: + +```sh +sudo tedge disconnect c8y +``` + +When this is done, issue [`tedge connect c8y`](../../references/cli/tedge-connect.md) again. + +### File permissions + +Connecting without sudo will result in the following error: + +```sh +tedge connect c8y +``` + +```text title="Output" +Checking if systemd is available. + +Checking if configuration for requested bridge already exists. + +Validating the bridge certificates. + +Saving configuration for requested bridge. + +Error: failed to create bridge to connect Cumulocity cloud. + +Caused by: + 0: File Error. Check permissions for /etc/tedge/mosquitto-conf/tedge-mosquitto.conf. + 1: failed to persist temporary file: Permission denied (os error 13) + 2: Permission denied (os error 13) +``` + +tedge connect cannot access directory to create the bridge configuration (`/etc/tedge/mosquitto-conf`), check permissions for the directory and adjust it to allow the tedge connect to access it. + +Example of incorrect permissions: + +```sh +ls -l /etc/tedge +``` + +```text title="Output" +dr--r--r-- 2 tedge tedge 4096 Mar 30 15:40 mosquitto-conf +``` + +You should give it the permission 755. + +```sh +ls -l /etc/tedge +``` + +```text title="Output" +drwxr-xr-x 2 tedge tedge 4096 Mar 30 15:40 mosquitto-conf +``` + +### Mosquitto and systemd check fails + +Sample output: + +```sh +sudo tedge connect c8y +``` + +```text title="Output" +Checking if systemd is available. + +Checking if configuration for requested bridge already exists. + +Validating the bridge certificates. + +Saving configuration for requested bridge. + +Restarting mosquitto service. + +Error: failed to create bridge to connect Cumulocity cloud. + +Caused by: + Service mosquitto not found. Install mosquitto to use this command. +``` + +mosquitto server has not been installed on the system and it is required to run this command, refer to the [install guide](../../install/index.md) to install mosquitto and try again. diff --git a/versioned_docs/version-1.1.0/operate/c8y/custom-fragments.md b/versioned_docs/version-1.1.0/operate/c8y/custom-fragments.md new file mode 100644 index 0000000..dec0564 --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/c8y/custom-fragments.md @@ -0,0 +1,108 @@ +--- +title: Custom Fragments +tags: [Operate, Cumulocity] +description: Publishing custom fragments/properties to Cumulocity IoT +--- + +%%te%% supports update custom fragments (also known as properties) on the device's digital twin representation in Cumulocity IoT. + +## Default fragments + +By default, the device will send the following information to Cumulocity IoT. The information makes it easy to identify devices which are using %%te%% in your fleet. + +```json +{ + "type": "thin-edge.io", + "c8y_Agent": { + "name": "thin-edge.io", + "url": "https://thin-edge.io", + "version": "1.0.0" + } +} +``` + +The default value for the `type` property can be changed using the `tedge config` command as follows: + +```sh +sudo tedge config set device.type VALUE +``` + +For example, you can set the `type` property to `edge_gateway` using: + +```sh +sudo tedge config set device.type edge_gateway +``` + +## Custom fragments + +Additional fragments can be added to the device by either publishing to a give MQTT topic, or via a file based method. Each section describes what data and when to use it. + +### MQTT Dynamic Fragments {#dynamic-fragments} + +%%te%% offers an MQTT topic which can be used to publish data to custom fragments for a device, child devices or services. + +* Values which change over time +* Update values without having to restart any services + +The following shows an example of publishing the name and version of the Operating System to the `os_Version` fragment for the main device. + +```sh te2mqtt +tedge mqtt pub te/device/main///twin/os_Version '{ + "name": "Poky (Yocto Project Reference Distro)", + "version": "4.0.15 (kirkstone)" +}' +``` + +The example above will result in the following fragment being added to the device's digital twin (e.g. *Managed Object*) in Cumulocity IoT. + +```json5 +{ + // ... other fragments are left out for simplicity + "os_Version": { + "name": "Poky (Yocto Project Reference Distro)", + "version": "4.0.15 (kirkstone)" + } +} +``` + +### File-based Static Fragments {#static-fragments} + +The file based approach is intended for static information, e.g. build date, or a custom image type assigned to the device. The values are only published on startup of the **tedge-mapper-c8y** service. + +If you wish to add more fragments to Cumulocity, you can do so by populating `/etc/tedge/device/inventory.json`. + +An example `inventory.json` looks something like this: + +```json title="file: /etc/tedge/device/inventory.json" +{ + "c8y_RequiredAvailability": { + "responseInterval": 5 + }, + "c8y_Hardware": { + "model": "BCM2708", + "revision": "000e", + "serialNumber": "00000000e2f5ad4d" + } +} +``` + +To see the changes you need to restart the tedge-mapper. +If you're using systemctl you can do: + +```sh +sudo systemctl restart tedge-mapper-c8y +``` + +In the Cumulocity UI this will looks something like this: + +

+ Cumulocity custom fragments +

+ + +For information on which fragments Cumulocity supports please see the +[Cumulocity API docs](https://cumulocity.com/guides/reference/device-management-library/). diff --git a/versioned_docs/version-1.1.0/operate/c8y/health-monitoring.md b/versioned_docs/version-1.1.0/operate/c8y/health-monitoring.md new file mode 100644 index 0000000..44f9017 --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/c8y/health-monitoring.md @@ -0,0 +1,93 @@ +--- +title: Health Monitoring +tags: [Operate, Cumulocity, Monitoring] +description: Monitoring the health of services on devices +--- + +The health of a %%te%% service or any other `service` that is running on the %%te%% device +or on the `child` device can be monitored from the **Cumulocity IoT** by sending the `health-status` message to **Cumulocity IoT**. + +## Publish health status + +A health status message can be published for any service on a `status/health` channel. The health message should at least contain the `status` of the service. + +:::note +The `status` here can be `up or down` or any other string. For example, `unknown`. +::: + +For example, to update the health status of `device/my-device/service/my-test-service` service, one has to send the +following message: + +```sh te2mqtt formats=v1 +tedge mqtt pub te/device/my-device/service/my-test-service/status/health '{"status":"up"}' -q 2 -r +``` + +:::note +The health status message has to be sent as a *retain* message. +::: + +When an empty health status message is sent, e.g. `{}` or `''`, the `status` will be replaced with `unknown`. + +## Conversion of the health status message to Cumulocity IoT service monitor message + +The `tedge-mapper-c8y` will translate any health status message that is received on `te/+/+/+/+/status/health` topic to +Cumulocity [Service status update](https://cumulocity.com/guides/reference/smartrest-two/#104) SmartREST message and +send it to the `Cumulocity` cloud. If a service was not previously registered and it fulfills the requirements for +auto-registration, it will be auto-registered as described [in the Auto Registration +section](https://thin-edge.github.io/thin-edge.io/next/references/mqtt-api/#auto-registration). + +For example, assuming a service `device/child1/service/service1`, running on a device `device/child1//`, which is a +child device of %%te%% device `device/main//` with an ID of `TE_DEVICE`, the resulting topic mapping looks like +this: + +
+ +**%%te%% health status message** + +```sh te2mqtt formats=v1 +tedge mqtt pub te/device/child1/service/service1/status/health '{"status":"up"}' -q 2 -r +``` + +
+ +
+ +**Cumulocity IoT (output)** + +```text title="Topic" +c8y/s/us/:device:child/:device:child:service:service1 +``` + +```text title="Payload" +104,up +``` + +
+ +## Configuring the default service type + +The `default service type` can be configured using the `tedge` cli. + +The example below shows how one can set the default service type to `systemd`. + +```sh +sudo tedge config set service.type systemd +``` + +:::note +When the `type` property was not included in the service registration message, then the configured default value +will be used by the mapper while auto-registering the service. +::: + +To clear the configured default service type one can use the command below. +This will set the `service.type` to `service`. + +```sh +sudo tedge config unset service.type +``` + +# References + +More info about the service monitoring can be found in the below link + +[Service monitoring Cumulocity IoT](https://cumulocity.com/guides/reference/smartrest-two/#service-creation-102) diff --git a/versioned_docs/version-1.1.0/operate/c8y/images/custom-smartrest-template-list.png b/versioned_docs/version-1.1.0/operate/c8y/images/custom-smartrest-template-list.png new file mode 100644 index 0000000..02b0a2a Binary files /dev/null and b/versioned_docs/version-1.1.0/operate/c8y/images/custom-smartrest-template-list.png differ diff --git a/versioned_docs/version-1.1.0/operate/c8y/images/custom-smartrest-template-response.png b/versioned_docs/version-1.1.0/operate/c8y/images/custom-smartrest-template-response.png new file mode 100644 index 0000000..0887c74 Binary files /dev/null and b/versioned_docs/version-1.1.0/operate/c8y/images/custom-smartrest-template-response.png differ diff --git a/versioned_docs/version-1.1.0/operate/c8y/images/smartrest-custom-operation-control.png b/versioned_docs/version-1.1.0/operate/c8y/images/smartrest-custom-operation-control.png new file mode 100644 index 0000000..b011b20 Binary files /dev/null and b/versioned_docs/version-1.1.0/operate/c8y/images/smartrest-custom-operation-control.png differ diff --git a/versioned_docs/version-1.1.0/operate/c8y/images/smartrest-template-add-operation.png b/versioned_docs/version-1.1.0/operate/c8y/images/smartrest-template-add-operation.png new file mode 100644 index 0000000..906acf2 Binary files /dev/null and b/versioned_docs/version-1.1.0/operate/c8y/images/smartrest-template-add-operation.png differ diff --git a/versioned_docs/version-1.1.0/operate/c8y/images/smartrest-template-create-new.png b/versioned_docs/version-1.1.0/operate/c8y/images/smartrest-template-create-new.png new file mode 100644 index 0000000..025a7a8 Binary files /dev/null and b/versioned_docs/version-1.1.0/operate/c8y/images/smartrest-template-create-new.png differ diff --git a/versioned_docs/version-1.1.0/operate/c8y/images/smartrest-template-import-after-selection.png b/versioned_docs/version-1.1.0/operate/c8y/images/smartrest-template-import-after-selection.png new file mode 100644 index 0000000..59d2884 Binary files /dev/null and b/versioned_docs/version-1.1.0/operate/c8y/images/smartrest-template-import-after-selection.png differ diff --git a/versioned_docs/version-1.1.0/operate/c8y/images/smartrest-template-import-file-dialog.png b/versioned_docs/version-1.1.0/operate/c8y/images/smartrest-template-import-file-dialog.png new file mode 100644 index 0000000..7d54072 Binary files /dev/null and b/versioned_docs/version-1.1.0/operate/c8y/images/smartrest-template-import-file-dialog.png differ diff --git a/versioned_docs/version-1.1.0/operate/c8y/images/smartrest-template-import.png b/versioned_docs/version-1.1.0/operate/c8y/images/smartrest-template-import.png new file mode 100644 index 0000000..d374d85 Binary files /dev/null and b/versioned_docs/version-1.1.0/operate/c8y/images/smartrest-template-import.png differ diff --git a/versioned_docs/version-1.1.0/operate/c8y/images/smartrest-template-list-empty.png b/versioned_docs/version-1.1.0/operate/c8y/images/smartrest-template-list-empty.png new file mode 100644 index 0000000..c17e184 Binary files /dev/null and b/versioned_docs/version-1.1.0/operate/c8y/images/smartrest-template-list-empty.png differ diff --git a/versioned_docs/version-1.1.0/operate/c8y/images/software-install-1-select.png b/versioned_docs/version-1.1.0/operate/c8y/images/software-install-1-select.png new file mode 100644 index 0000000..25a2a69 Binary files /dev/null and b/versioned_docs/version-1.1.0/operate/c8y/images/software-install-1-select.png differ diff --git a/versioned_docs/version-1.1.0/operate/c8y/images/software-install-2-review.png b/versioned_docs/version-1.1.0/operate/c8y/images/software-install-2-review.png new file mode 100644 index 0000000..923bc61 Binary files /dev/null and b/versioned_docs/version-1.1.0/operate/c8y/images/software-install-2-review.png differ diff --git a/versioned_docs/version-1.1.0/operate/c8y/images/software-install-3-apply.png b/versioned_docs/version-1.1.0/operate/c8y/images/software-install-3-apply.png new file mode 100644 index 0000000..63eb476 Binary files /dev/null and b/versioned_docs/version-1.1.0/operate/c8y/images/software-install-3-apply.png differ diff --git a/versioned_docs/version-1.1.0/operate/c8y/images/software-install-4-done.png b/versioned_docs/version-1.1.0/operate/c8y/images/software-install-4-done.png new file mode 100644 index 0000000..cdb7c22 Binary files /dev/null and b/versioned_docs/version-1.1.0/operate/c8y/images/software-install-4-done.png differ diff --git a/versioned_docs/version-1.1.0/operate/c8y/images/software-install-5-use-result.png b/versioned_docs/version-1.1.0/operate/c8y/images/software-install-5-use-result.png new file mode 100644 index 0000000..5a8aa4b Binary files /dev/null and b/versioned_docs/version-1.1.0/operate/c8y/images/software-install-5-use-result.png differ diff --git a/versioned_docs/version-1.1.0/operate/c8y/images/software-remove-1-select.png b/versioned_docs/version-1.1.0/operate/c8y/images/software-remove-1-select.png new file mode 100644 index 0000000..2df6040 Binary files /dev/null and b/versioned_docs/version-1.1.0/operate/c8y/images/software-remove-1-select.png differ diff --git a/versioned_docs/version-1.1.0/operate/c8y/images/software-remove-2-review.png b/versioned_docs/version-1.1.0/operate/c8y/images/software-remove-2-review.png new file mode 100644 index 0000000..93eb8c8 Binary files /dev/null and b/versioned_docs/version-1.1.0/operate/c8y/images/software-remove-2-review.png differ diff --git a/versioned_docs/version-1.1.0/operate/c8y/images/software-remove-3-apply.png b/versioned_docs/version-1.1.0/operate/c8y/images/software-remove-3-apply.png new file mode 100644 index 0000000..be2b372 Binary files /dev/null and b/versioned_docs/version-1.1.0/operate/c8y/images/software-remove-3-apply.png differ diff --git a/versioned_docs/version-1.1.0/operate/c8y/images/software-remove-4-done.png b/versioned_docs/version-1.1.0/operate/c8y/images/software-remove-4-done.png new file mode 100644 index 0000000..ad25b50 Binary files /dev/null and b/versioned_docs/version-1.1.0/operate/c8y/images/software-remove-4-done.png differ diff --git a/versioned_docs/version-1.1.0/operate/c8y/images/software-repo-1-list.png b/versioned_docs/version-1.1.0/operate/c8y/images/software-repo-1-list.png new file mode 100644 index 0000000..be14223 Binary files /dev/null and b/versioned_docs/version-1.1.0/operate/c8y/images/software-repo-1-list.png differ diff --git a/versioned_docs/version-1.1.0/operate/c8y/images/software-repo-2-add.png b/versioned_docs/version-1.1.0/operate/c8y/images/software-repo-2-add.png new file mode 100644 index 0000000..5b1bf56 Binary files /dev/null and b/versioned_docs/version-1.1.0/operate/c8y/images/software-repo-2-add.png differ diff --git a/versioned_docs/version-1.1.0/operate/c8y/images/software-repo-hosted-binary.png b/versioned_docs/version-1.1.0/operate/c8y/images/software-repo-hosted-binary.png new file mode 100644 index 0000000..39472e5 Binary files /dev/null and b/versioned_docs/version-1.1.0/operate/c8y/images/software-repo-hosted-binary.png differ diff --git a/versioned_docs/version-1.1.0/operate/c8y/images/software-self-update-1-select.png b/versioned_docs/version-1.1.0/operate/c8y/images/software-self-update-1-select.png new file mode 100644 index 0000000..5653947 Binary files /dev/null and b/versioned_docs/version-1.1.0/operate/c8y/images/software-self-update-1-select.png differ diff --git a/versioned_docs/version-1.1.0/operate/c8y/images/software-self-update-2-review.png b/versioned_docs/version-1.1.0/operate/c8y/images/software-self-update-2-review.png new file mode 100644 index 0000000..2e89f6b Binary files /dev/null and b/versioned_docs/version-1.1.0/operate/c8y/images/software-self-update-2-review.png differ diff --git a/versioned_docs/version-1.1.0/operate/c8y/index.md b/versioned_docs/version-1.1.0/operate/c8y/index.md new file mode 100644 index 0000000..241e7a0 --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/c8y/index.md @@ -0,0 +1,12 @@ +--- +title: Device Management with Cumulocity +tags: [Operate, Cloud] +sidebar_position: 9 +description: Device management with %%te%% and Cumulocity IoT +--- + +import DocCardList from '@theme/DocCardList'; + +This section is for the cloud operators who manage a fleet of %%te%% devices connected to Cumulocity. + + diff --git a/versioned_docs/version-1.1.0/operate/c8y/log-management.md b/versioned_docs/version-1.1.0/operate/c8y/log-management.md new file mode 100644 index 0000000..2161fa1 --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/c8y/log-management.md @@ -0,0 +1,34 @@ +--- +title: Log Management +tags: [Operate, Cumulocity, Log Files] +description: Managing log files on devices +--- + +The **tedge-agent** service provides out of the box support for requesting log files from a device directly from your Cumulocity UI. + +If you go to Cumulocity, you should see that you are able to see the logs tab. +However, no log type is yet available. +To add a new log type, you need to edit the `tedge-log-plugin.toml` in `/etc/tedge/plugins/tedge-log-plugin.toml`. + +In this toml file you specify the log type and log path of the logs wished to +be retrieved from Cumulocity UI. +For example, if you wish to request %%te%% software logs and mosquitto logs +an example toml file would be: + +```toml title="file: /etc/tedge/plugins/tedge-log-plugin.toml" +files = [ + { type = "software-management", path = "/var/log/tedge/agent/software-*" }, + { type = "mosquitto", path = "/var/log/mosquitto/mosquitto.log" } +] +``` + +Note that `path` need not be a complete path. It can be a full path to a log +file or a [glob pattern](https://en.wikipedia.org/wiki/Glob_(programming)). +For example the "software-management" type is a glob pattern that groups +any file inside "/var/log/tedge/agent/" that starts with "software-". + +The `type` key in the toml is the name of the log with you will see in the +Cumulocity UI: + +![Log request dropdown](../../images/tedge-log-plugin_log-types.png) + diff --git a/versioned_docs/version-1.1.0/operate/c8y/remote-access.md b/versioned_docs/version-1.1.0/operate/c8y/remote-access.md new file mode 100644 index 0000000..3d71ebe --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/c8y/remote-access.md @@ -0,0 +1,56 @@ +--- +title: Remote Access +tags: [Operate, Cumulocity, Remote Access] +description: Accessing devices remote using tcp based protocols (e.g. ssh, vnc etc.) +--- + +# Remote Access + +To access a device remotely that runs %%te%%, a plugin of the operation plugin concept is used. The tedge-mapper is checking for cloud remote access operation and is triggering the particular plugin. You can use the remote access tab in device management to access the device via SSH or VNC. + +Background information on the remote access feature provided by Cumulocity IoT can be found in their [official documentation](https://cumulocity.com/guides/cloud-remote-access/using-cloud-remote-access/). + +## Requirements + +- Working %%te%% installation + +- The **Cloud Remote Access Feature** is assigned to your Tenant. If not ask your Administrator to get it assigned to your Tenant. Please note that the Version must be at least 1007.2.0+ + +- The *Cloud Remote Access Role* must be assigned to the user who wants to use that Feature: Administration → Role → <any Role> → check "Remote Access". Assign the role to the user used for the next steps. + +- A VNC or SSH server running on the device you wish to connect to. + + +## Usage + +Make sure %%te%% is connected to Cumulocity. + +You device within Cumulocity should look similar to this (the "Remote access" tab should be visible in the menu on the left): + +

+ Cumulocity remote access device management +

+ +You can configure now within the Remote access tab to which e.g. VNC or SSH server you want to jump to. Please keep in mind that the Host is from the %%te%% point of view. + +

+ Cumulocity remote access endpoint +

+ +If you click on connect after the proper configuration an websocket window opens and %%te%% triggers the c8y-remote-access-connect plugin to reach that websocket. + +

+ Cumulocity remote access websocket +

diff --git a/versioned_docs/version-1.1.0/operate/c8y/restart-device.md b/versioned_docs/version-1.1.0/operate/c8y/restart-device.md new file mode 100644 index 0000000..37d9344 --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/c8y/restart-device.md @@ -0,0 +1,21 @@ +--- +title: Restart Device +tags: [Operate, Operation, Cumulocity] +description: Restarting a device from the cloud +--- + +If your device is running %%te%%, you can restart it from the cloud. + +### Cumulocity IoT + +1. Go to the *Device Management* application in Cumulocity IoT + +2. Find your device and open up its homepage + +3. Open the *Control* menu item + + ![Control button](../../images/control-button-red-highlight.png) + +4. In the top right corner, you will find the *More* button, click it and select, *Restart device* + + ![Restart device button](../../images/restart-button-red-highlight.png) diff --git a/versioned_docs/version-1.1.0/operate/c8y/smartrest-templates.md b/versioned_docs/version-1.1.0/operate/c8y/smartrest-templates.md new file mode 100644 index 0000000..19baff8 --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/c8y/smartrest-templates.md @@ -0,0 +1,411 @@ +--- +title: SmartREST Templates +tags: [Operate, Cumulocity] +description: SmartREST 2.0 template registration and usage +--- + +[Custom SmartREST 2.0 Templates](https://cumulocity.com/guides/reference/smartrest-two) can be used to extend the functionality of a device to support more operations than what the [static SmartREST templates](https://cumulocity.com/guides/reference/smartrest-two/#mqtt-static-templates) offer. + +%%te%% supports subscription to custom templates as documented [here](https://cumulocity.com/guides/users-guide/device-management/#smartrest-templates). + +For every template that the device uses, it must publish all data to `s/uc/` topic and subscribe to `s/dc/` to receive data from the cloud, based on that template. +When these templates are configured with %%te%%, subscriptions to all these relevant topics on Cumulocity cloud will be done by %%te%% internally. +Local processes on the device can access these templates on the local MQTT broker by simply publishing to `c8y/s/uc/` and subscribing to `c8y/s/dc/` topics (note the `c8y/` prefix in topics). + +A template named `$TEMPLATE_NAME` requires the following subscriptions to be added when connecting to Cumulocity: + +```text +s/dc/$TEMPLATE_NAME +s/uc/$TEMPLATE_NAME +``` + +**This is not done automatically and the custom templates have to be declared using the `tedge` command.** + +## Checking existing templates + +```sh +tedge config get c8y.smartrest.templates +``` + +## Add new template to thin-edge configuration + +To add new template to %%te%% the `tedge config` cli tool can be used as following: + +```sh +sudo tedge config set c8y.smartrest.templates template-1,template-2 +``` + +After adding or removing a template, you will need to run the following command before %%te%% will use the new settings: + +```sh +sudo tedge reconnect c8y +``` + +:::note +To add/append a new template to a device that's already configured with some, all the existing templates should also be declared along with the new one in the `tedge config set` command. +For example, if `template-1` is already configured on the device, as following: + +```sh +tedge config get c8y.smartrest.templates +["template-1"] +``` + +To add new template to the set it is required to include current template, so the command would look like this: + +```sh +tedge config set c8y.smartrest.templates template-1,template-2 +``` + +Now when we get the configuration the both templates will be there: + +```sh +tedge config get c8y.smartrest.templates +["template-1", "template-2"] +``` +::: + +## Removing templates from configuration + +To remove all the templates, the `unset` subcommand can used as follows: + +```sh +sudo tedge config unset c8y.smartrest.templates +``` + +To remove one of existing templates you can overwrite the existing `c8y.smartrest.templates` with the new set which doesn't contain the unwanted template. + +```sh +tedge config get c8y.smartrest.templates +``` + +```toml title="Output" +["template-1", "template-2"] +``` + +```sh +sudo tedge config set c8y.smartrest.templates template-1 +``` + +You can verify that the template has changed by reading the value again. + +```sh +tedge config get c8y.smartrest.templates +``` + +```toml title="Output" +["template-1"] +``` + +After adding or removing a template, you will need to run the following command before %%te%% will use the new settings: + +```sh +sudo tedge reconnect c8y +``` + +## Example: Creating a custom operation + +The following example shows how to create a new SmartREST template with a single custom operation which will be activated when an operation is created with the `set_wifi` fragment. The operation includes 3 parameters where the wifi `name`, `ssid` and `type` are included in the message which is sent to the device via MQTT. + +In this example, the operation handler on the %%te%% side only prints out the received message on the console, but it can be extended to execute any command that is required for your task. + +The operation response defined in the SmartREST template will convert the Cumulocity IoT Operation from json to an MQTT message in a Comma Separated Variables (CSV) format. The MQTT message is then received by %%te%% and a script is called passing the message as the message to it. The script is used to perform the desired actions using the parameters provided in the message. + +The snippets below are used to illustrate the message translation of the Cumulocity IoT operation to the message received by the device. + +**Cumulocity IoT Operation** + +```json +{ + "description": "Configure wifi", + "deviceId": "664853968", + "set_wifi": { + "name": "Factory Wifi", + "ssid": "factory-onboarding-wifi", + "type": "WPA3-Personal" + } +} +``` + +**SmartREST (CSV) format received by %%te%%** + +The above Cumulocity IoT operation is transformed into CSV (using the formatting/rules defined in the SmartREST template) and sent to the device via MQTT. The example below shows the format of the message as received by the device: + +```csv title="topic: c8y/s/dc/custom_devmgmt" +dm101,,,, +``` + +The following procedure details the step-by-step instructions on how to implement the above example. + +### Step 1: Creating a SmartREST template + +1. Open the Cumulocity IoT *Device Management* Application in your web browser + +2. Navigate to *Device Types → SmartREST Templates* + + ![smartrest-template-list-empty](./images/smartrest-template-list-empty.png) + +3. Click *Add SmartREST template* + + ![smartrest-template-create-new](./images/smartrest-template-create-new.png) + +4. Add a new *Response* message, and enter the details as detailed in the screenshot. + + ![smartrest-template-add-operation](./images/smartrest-template-add-operation.png) + + The properties used in the example can be also described as follows: + + |Property|Value|Description| + |----|---|---| + |Response ID|dm101|Unique id of the message| + |Name|set_wifi|Human readable name of the message| + |Base pattern|set_wifi|The common prefix added to each pattern (see Pattern.x rows)| + |Condition|set_wifi|The fragment which will "activate" the SmartREST message translation| + |Patterns.0|name|Wifi Connection Name (Custom parameter to be included in the body)| + |Patterns.1|ssid|Wifi SSID (Custom parameter to be included in the body)| + |Patterns.2|type|Wifi Type (Custom parameter to be included in the body)| + +5. Click *Save* + + ![custom-smartrest-template-list](./images/custom-smartrest-template-list.png) + +#### Alternative: Import SmartREST template from file + +Alternatively, you can import a SmartREST template from an existing file. This approach is less error prone as you don't need to enter any of the values manually, but it obviously requires you to have already exported an existing SmartREST template. + +1. Save the following SmartREST template to a local file (on your machine) called **custom_devmgmt.json** + + ```json title="file: custom_devmgmt.json" + { + "name": "custom_devmgmt", + "type": "c8y_SmartRest2Template", + "com_cumulocity_model_smartrest_csv_CsvSmartRestTemplate": { + "requestTemplates": [], + "responseTemplates": [ + { + "msgId": "dm101", + "condition": "set_wifi", + "base": "set_wifi", + "name": "set_wifi", + "pattern": [ + "name", + "ssid", + "type" + ] + } + ] + }, + "__externalId": "custom_devmgmt" + } + ``` + +2. Open the Cumulocity IoT *Device Management* Application + +3. Navigate to *Device Types → SmartREST templates* + + ![smartrest-template-list-empty](./images/smartrest-template-list-empty.png) + +4. Click *Import template* + + ![custom-smartrest-template-response](./images/smartrest-template-import.png) + +5. Select the *Load from file* and select the json file you saved in the first step and click *Open* + + ![smartrest-template-import-file-dialog](./images/smartrest-template-import-file-dialog.png) + +6. Double check the information and select *Import* + + ![smartrest-template-import-after-selection](./images/smartrest-template-import-after-selection.png) + + If the import was successful, then you should see the SmartREST template, **custom_devmgmt**, in the list: + + ![custom-smartrest-template-list](./images/custom-smartrest-template-list.png) + + +### Step 2: Configure thin-edge.io + +On the device, perform the following steps: + +1. Set the custom SmartREST template to be used by %%te%% + + ```sh + tedge config set c8y.smartrest.templates "custom_devmgmt" + ``` + + The `custom_devmgmt` is the id of the SmartREST template that was created in the previous step. + +2. Reconnect to Cumulocity IoT (assuming you have already connected to Cumulocity IoT once) + + ```sh + tedge reconnect c8y + ``` + + :::info + This step ensures that the new SmartREST template id is also added in the list of MQTT topics to subscribe to. Without this step, %%te%% will not be able to receive the custom operation. + ::: + +:::tip +You don't need to re-run this step when you add a new message definition to an existing template id. +::: + +### Step 3: Creating the operation handler + +On your %%te%% device, run the following steps: + +1. Create the following custom operation handler file + + ```toml title="file: /etc/tedge/operations/c8y/set_wifi" + [exec] + command = "/usr/bin/set_wifi" + topic = "c8y/s/dc/custom_devmgmt" + on_message = "dm101" + ``` + + The operation definition tells %%te%% what to do when receiving a specific message (with the message id) on the given topic. Specifically, the definition will execute the `/usr/bin/set_wifi` script when a `dm101` message is received on the `c8y/s/dc/custom_devmgmt` topic. + +2. Create the script which is called when receiving the `dm101` message + + ```sh title="file: /usr/bin/set_wifi" + #!/bin/sh + set -e + + # Constants + OK=0 + + # Input arguments + MESSAGE="$1" + NAME=$(echo "$COMMAND" | cut -d, -f 3) + SSID=$(echo "$COMMAND" | cut -d, -f 4) + TYPE=$(echo "$COMMAND" | cut -d, -f 5) + + echo "Processing message: $MESSAGE" + echo "NAME: $NAME" + echo "SSID: $SSID" + echo "TYPE: $TYPE" + exit "$OK" + ``` + + :::note + You can change the path where this file is located, but it **MUST** match the path given in the custom operation definition file under the `exec.command` property. + ::: + +3. Make the script executable + + ```sh + sudo chmod 755 /usr/bin/set_wifi + ``` + +### Step 4: Sending a custom operation + +1. On your local machine, create a custom operation instance in Cumulocity IoT + + Below shows an example of creating the operation using the [go-c8y-cli)](https://goc8ycli.netlify.app/) cli tool. It assumes you have already activated your go-c8y-cli session which points to your intended Cumulocity IoT tenant. + + ```sh + c8y operations create \ + --device 12345 \ + --template "{set_wifi:{name:'Factory Wifi',ssid:'factory-onboarding-wifi',type:'WPA3-Personal'}}" \ + --description "Configure wifi" + ``` + + Where `12345` should be replaced with the Cumulocity IoT device id of your device. + + If you're not familiar with go-c8y-cli, then you can send the Cumulocity REST API request using other tools such as Postman, curl etc., though you will have use the appropriate Authorization Header as defined by the official [Cumulocity API Guide](https://cumulocity.com/api/core/). + + **Cumulocity IoT REST Request** + + The custom operation can be created via the Cumulocity IoT REST API using the following details: + + ``` + POST /devicecontrol/operations + ``` + + **Body** + + ```json + { + "description": "Configure wifi", + "deviceId": "664853968", + "set_wifi": { + "name": "Factory Wifi", + "ssid": "factory-onboarding-wifi", + "type": "WPA3-Personal" + } + } + ``` + +2. In the Cumulocity IoT *Device Management* application, check the *Control* page where you should see the "Configure Wifi" operation. + + ![smartrest-custom-operation-control.png](./images/smartrest-custom-operation-control.png) + +3. On the %%te%% device, open a console, and check the log file which when processing the custom operation + + The following command uses a simple bash one-liner to print the contents of the most recent file created matching the `set_wifi*` pattern under the `/var/log/tedge/agent/` folder. + + ```sh + cat "$(ls -t /var/log/tedge/agent/set_wifi* | head -1)" + ``` + + ```text title="Output" + ----- $ /usr/bin/set-wifi "dm101,rpi5-d83add9f145a,Factory Wifi,factory-onboarding-wifi,WPA3-Personal" + exit status: 0 + + stdout < + Add software +

+ + :::tip + * If you are using the `apt` software type and hosting the package in an external repository, then just use an a single space in the *Provide a file path* option. + ::: + +3. Select *Add software* to create the item + +### To install software on a device {#install} + +1. Select the software you want to install on the device + + ![Select software](images/software-install-1-select.png) + +2. Review the software changes + + ![Review software](images/software-install-2-review.png) + + :::info + You can combine installing and removing software in the same operation + ::: + +3. Select *Apply Changes* and wait for the operation to complete + + ![Apply software changes](images/software-install-3-apply.png) + +4. Use the installed package + + ![Software changes finished](images/software-install-5-use-result.png) + + +### To delete software from a device {#delete} + +1. Select the software you want to remove from the device + + ![Select software](images/software-remove-1-select.png) + +2. Review the software changes + + ![Review software](images/software-remove-2-review.png) + + :::note + You can combine installing and removing software in the same operation + ::: + +3. Select *Apply Changes* and wait for the operation to complete + + ![Apply software changes](images/software-remove-3-apply.png) + + ![Software changes finished](images/software-remove-4-done.png) + +### Updating to the latest %%te%% version {#self-update} + +Updating %%te%% from Cumulocity IoT is the same process as installing any other software. You can update all of the %%te%% packages by installing a single package called `tedge-full`. + +Below shows instructions on how to update %%te%% to the latest version: + +1. Create a software package using the instructions in the [add a new software package section](#add-to-repo) and use the following package properties + + |Property|Value| + |----|-----| + |Name|tedge-full| + |Version|latest| + |Software type|apt| + |Software File|Select *Provide a file path*, and use a single space character| + + :::tip + The **Software type** value depends on which Software Management Plugin you're using. + + On Debian systems the software type will be `apt`, but you can also install other Software management Plugin's to support installing other types of packages such as Alpine Linux (`apk`) or RPM packages (`rpm`). + ::: + +2. Select the `tedge-full` software you want to install on the device + + ![Select software](images/software-self-update-1-select.png) + +3. Review the software changes and select *Apply changes* + + ![Select software](images/software-self-update-2-review.png) + +4. Wait for the operation to complete + +:::warning +If you are upgrading %%te%% from any version prior to `1.0.0` (including `1.0.0-rc.1`), then you will have to restart the device after the update before the new version of the `tedge-agent` will be activated. Until the device is restarted (or the `tedge-agent` service is restarted), the previous version of the `tedge-agent` will still be running. +::: + +## Configuration {#configuration} + +### tedge-apt-plugin: Filter packages by name and maintainer + +By default the `tedge-apt-plugin` lists all of the installed Debian (*.deb) packages. On typical Debian installations, the list of packages could easily be more than 500 packages. In order to focus on the Debian packages which are important for your device, the `tedge-apt-plugin` supports filtering by either name or maintainer. + +The package list filter criteria can be configured via the %%te%% configuration (e.g. `tedge.toml`). Create the `apt` table in `tedge.toml` and fill it with the `name` and `maintainer` keys. The value of each filter key should be a **valid regex pattern**. Your filters should look like this: + +```toml +[apt] +name = "(tedge|c8y|python|wget|vim|curl|apt|mosquitto|ssh|sudo).*" +maintainer = ".*(thin-edge.io|other).*" +``` + +You can also test the filters by running the `tedge-apt-plugin` locally on the command line. Running the commands locally help you tune the filters to be sure they work as expected. Below shows some examples of using various filters: + +```sh +# Match by name +tedge-apt-plugin list --name '^lib.*' --maintainer '' + +# Match by maintainer +tedge-apt-plugin list --name '' --maintainer '.*thin-edge.io.*' + +# Match by name or maintainer +tedge-apt-plugin list --name 'python3.*' --maintainer '.*thin-edge.io.*' +``` + +Once you have the filter criteria that you're happy with, you can make the setting permanent by updating the %%te%% configuration. + +```sh +sudo tedge config set apt.name '(tedge|c8y|python|wget|vim|curl|apt|mosquitto|ssh|sudo).*' +sudo tedge config set apt.maintainer '.*(thin-edge.io|other).*' +``` + +## FAQ + +The following contains frequently asked questions regarding the software management feature. + +### Hosting linux packages in Cumulocity IoT + +The recommended way to install linux packages such as deb, apk and rpm is to install packages directly from a configure package repository such as public repositories for your Operating System, or using a package service such as Cloudsmith, JFrog, Gemfury, Packagecloud etc. Using a package repository supports the full dependency resolution that users are familiar with when install packages manually, e.g. using `apt-get install mypackage`, and the correct architecture will be automatically selected for your device (e.g arm64/aarch or amd64/x86_64 etc.). + +If you choose to host the software packages in Cumulocity IoT or via some other blob store, then the automatic dependency installation and architecture will not function so you have to manage all of this yourself. It might be manageable if you have simple packages with no dependencies and have an architecture agnostic package (e.g. the architecture is set to `all` or `noarch`). + +When hosting the linux packages within Cumulocity IoT then ensure that: + +* the package does not have any dependencies +* the name and version used in the Cumulocity IoT *Software repository* matches the exact information contained in the package's meta data + + For example, below shows the `apt-cache show jq` information for the `jq` package where the **Package** and **Version** fields should match the **name** and **version** properties in Cumulocity IoT. + + ``` + Package: jq + Version: 1.6-2.1 + Architecture: arm64 + ``` + + If the package name and version does not match, then the package might fail to be installed due to a validation error. +* the *Device type filter* property is set in the Cumulocity IoT Software repository if the packages is only valid for a specific CPU architecture. The *Device type filter* is used to filter which packages are available for different devices types to ensure you can only install compatible packages on a given device. + +

+ Hosted binary +

diff --git a/versioned_docs/version-1.1.0/operate/c8y/supported-operations.md b/versioned_docs/version-1.1.0/operate/c8y/supported-operations.md new file mode 100644 index 0000000..459225f --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/c8y/supported-operations.md @@ -0,0 +1,249 @@ +--- +title: Supported Operations +tags: [Operate, Cumulocity, Operation] +description: Declaring and using custom operations +--- + +## Concepts + +### Device operations + +IoT devices often do more than just send data to the cloud. They also do things like: + +* receive triggers from the operator +* reboot on demand +* install or remove software + +These operations are supported by [Cumulocity IoT](https://cumulocity.com/guides/reference/device-management-library) and other cloud providers. +When such an operation is triggered from the cloud, the cloud mapper (e.g: `tedge-mapper-c8y`) processes that request. + +The Cumulocity mapper treats the following operations as inbuilt operations and converts those into their equivalent tedge commands: + +| Operation | Cumulocity Operation Type | Tedge Command | +| ----------|---------------------------|---------------| +| Device restart | `c8y_Restart` | `te//cmd/restart` | +| Software update | `c8y_SoftwareUpdate` | `te//cmd/software_update` | +| Configuration retrieval | `c8y_UploadConfigFile` | `te//cmd/config_snapshot` | +| Configuration update | `c8y_DownloadConfigFile` | `te//cmd/config_update` | +| Log retrieval | `c8y_LogfileRequest` | `te//cmd/log_upload` | +| Firmware update | `c8y_Firmware` | `te//cmd/firmware_update` | + +Another process like the `tedge-agent` or an external plugin may process these mapped tedge commands. +The `tedge-agent` currently supports all the above mentioned inbuilt operations out-of-the-box. + +For all other operation types, the mapper can execute a custom operation plugin if one is configured. + +The `Supported Operations API` of the Cumulocity mapper can be used to add support for these custom operations, +or when the user wants to handle any of the inbuilt operations differently than how the `tedge-agent` handles it. + +### Supported Operations API + +The Supported Operations API utilises the file system to add or remove operations. +An operation file placed in `/etc/tedge/operations/c8y` indicates that +an operation with that name is supported by the tedge device on Cumulocity. +For e.g, an empty file named `c8y_Restart` in this directory represents that +the tedge device supports Cumulocity device restart operation. + +The aggregated list of all the operation files in this directory represents the [Cumulocity supported operations list](https://cumulocity.com/guides/reference/device-management-library/#announcing-capabilities) of that device. +Whenever a new operation file is added to + +Similarly, an operation file at `/etc/tedge/operations/c8y/` indicates that +the child device with the given external id `` supports that operation. + +## How to use Supported Operations + +### Listing supported operations + +The Cumulocity supported operations list of the tedge device can be retrieved by listing all the files in the `/etc/tedge/operations/c8y` directory. + +```sh +ls -l /etc/tedge/operations/c8y +``` + +```text title="Output" +-rw-r--r-- 1 tedge tedge 0 Jan 01 00:00 c8y_Restart +``` + +Similarly, one can list all the currently supported operations for a child device as follows: + +```sh +ls -l /etc/tedge/operations/c8y/ +``` + +```text title="Output" +-rw-r--r-- 1 tedge tedge 0 Oct 26 11:24 c8y_LogfileRequest +``` + +### Adding new operations + +To add new operation we need to create new file in the `/etc/tedge/operations/c8y` directory. +For e.g, to enable device restart operations from Cumulocity, a device must declare `c8y_Restart` as a supported operation. +This can be done by publishing the following operation capability MQTT message: + +```sh +tedge mqtt pub -r 'te/device/main///cmd/restart' '{}' +``` + +The mapper, in response to this tedge capability message, creates a `c8y_Restart` operation file at `/etc/tedge/operations/c8y`. + +:::note +The `tedge-agent` sends these capability messages automatically for all the inbuilt operations, when it starts up. +::: + +Operation files can also be created manually at this directory: + +```sh +sudo -u tedge touch /etc/tedge/operations/c8y/c8y_Restart +``` + +:::note +We are using `sudo -u` to create the file because we want to make sure that the file is owned by `tedge` user. +::: + +Whenever a new operation file is added to this directory, the supported operations list for that device is updated +by aggregating all the operation file names in that directory and this updated list is sent to the cloud +using the [SmartRest 114 message](https://cumulocity.com/guides/reference/smartrest-two/#114). + +:::warning +Updating the supported operations list by manually adding files to the operations directory is currently deprecated +and will be removed in an upcoming release. +It is advised to use the MQTT capability message mechanism for the same. +::: + +Similarly, a child device can declare that it supports restart operation by publishing the following operation capability message: + +```sh +tedge mqtt pub -r 'te/device////cmd/restart' '{}' +``` + +Operation files can also be placed manually in the child device operations directory at `/etc/tedge/operations/c8y/`. +But, unlike the main device, the supported operations list of child devices are not aggregated and published whenever this directory is updated, +but only when such a capability message is received from a child device. + +### Removing supported operations + +To remove a supported operation for a %%te%% device, the corresponding operation file must be removed from the `/etc/tedge/operations/c8y` directory. eg: + +```sh +sudo rm /etc/tedge/operations/c8y/c8y_Restart +``` + +Now the operation will be automatically removed from the list and the list will be sent to the cloud. + +:::warning +Dynamic removal of an operation from the supported operation list is not supported for child devices. +::: + +## Working with custom operations + +The `Supported Operations API` can also be used to add support for custom operations beyond the inbuilt ones. +For e.g: Cumulocity supports the `c8y_Command` that enables execution of shell commands on the device from the cloud. +The same operation files can be used to define how the mapper should handle these operations when triggered from the cloud. + +### Adding new custom operation + +An operation file must be created with the name of the operation: + +```sh +sudo -u tedge touch /etc/tedge/operations/c8y/c8y_Command +``` + +...with the following content: + +```toml title="file: /etc/tedge/operations/c8y/c8y_Command" +[exec] + topic = "c8y/s/ds" + on_message = "511" + command = "/etc/tedge/operations/command" + timeout = 10 +``` + +In this example, the mapper is configured to pick up the `c8y_Command` operation message received on `c8y/s/ds` topic +with the SmartRest template message prefix `511`. +When such a message is received, the operation plugin located at `/etc/tedge/operations/command` would be executed. +The operation is configured to `timeout` after 10 seconds, to avoid it from running for too long/forever. + +:::note +The operation file needs to be readable by %%te%% user - `tedge` - and should have permissions `644`. +The filename **MUST** only use alphanumeric and underscore characters, e.g. `A-Z`, `a-z`, `0-9` and `_`. +You cannot use a dash "-", or any other characters in the filename, otherwise the custom operation definition will be ignored. +::: + +:::note +The `timeout` that is configured will be in seconds. +If a custom operation is not configured with a `timeout` value, then it will use default `timeout`,.i.e. 3600 seconds. +If the operation does not complete within that specified `timeout` period, then the operation will be stopped/killed, and marked as failed in the cloud. +::: + +Here is a sample operation plugin that can handle the `c8y_Command`: + +```sh title="file: /etc/tedge/operations/command" +#!/bin/bash +# Parse the smart rest message, ignore the first two field, and everything afterwards is the command +COMMAND="${1#*,*,}" + +# Check if command is wrapped with quotes, if so then remove them +if [[ "$COMMAND" == \"*\" ]]; then + COMMAND="${COMMAND:1:-1}" +fi + +# Execute command +bash -c "$COMMAND" +EXIT_CODE=$? +if [ $EXIT_CODE -ne 0 ]; then + echo "Command returned a non-zero exit code. code=$EXIT_CODE" >&2 +fi + +exit "$EXIT_CODE" +``` + +This script is invoked with the received SmartREST message (`511,DeviceSerial,`). +This simple script parses the third field of the received SmartREST message and executes that command. +If it exits with the status code `0`, a successful message with the stdout content will be reported to Cumulocity. +If it exits with a non-zero code, a failure message with the stderr content will be sent out. + +:::note +The command will be executed with `tedge` permission level. +So, most of the system level commands will not work. +::: + +The operation files for inbuilt operations can also be defined in this format to override +the inbuilt handling of those operations by the mapper, which just converts those to their equivalent tedge commands. +For e.g: if the tedge device wants to handle the `c8y_Restart` operation differently than how the `tedge-agent` handles it, +the `c8y_Restart` operation file for the tedge device can be defined in a similar manner. + +This same custom operation file mechanism can be used for the child devices as well, +to either add support for additional operations or override the inbuilt ones, +as long as that operation can be fully handled by executing a configured `command` from the tedge device itself. +For e.g: if a child device is incapable of receiving and processing the tedge `restart` command via MQTT, +but can be restarted directly from the tedge device via some remote restart commands, +the `c8y_Restart` operation file for the child device can be defined to invoke those remote restart commands. + +### List of currently supported operations parameters + +* `topic` - The topic on which the operation will be executed. +* `on_message` - The SmartRest template on which the operation will be executed. +* `command` - The command to execute. +* `result_format` - The expected command output format: `"text"` or `"csv"`, `"text"` being the default. + +:::info +The `command` parameter accepts command arguments when provided as a one string, e.g. + +```toml title="file: /etc/tedge/operations/c8y/c8y_Command" +[exec] + topic = "c8y/s/ds" + on_message = "511" + command = "python /etc/tedge/operations/command.py" + timeout = 10 +``` + +Arguments will be parsed correctly as long as following features are not included in input: operators, variable assignments, tilde expansion, parameter expansion, command substitution, arithmetic expansion and pathname expansion. + +In case those unsupported shell features are present, the syntax that introduce them is interpreted literally. + +Be aware that SmartREST payload is always added as the last argument. The command presented above will actually lead to following code execution + +```bash +python /etc/tedge/operations/command.py $SMART_REST_PAYLOAD +``` +::: \ No newline at end of file diff --git a/versioned_docs/version-1.1.0/operate/configuration/index.md b/versioned_docs/version-1.1.0/operate/configuration/index.md new file mode 100644 index 0000000..421f354 --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/configuration/index.md @@ -0,0 +1,177 @@ +--- +title: Configuration +tags: [Operate, Configuration] +sidebar_position: 2 +description: How to configure %%te%% +--- + +The settings of all the %%te%% components are grouped in the `/etc/tedge/tedge.toml` file, using [TOML](https://toml.io/). +These configuration settings are organized in a hierarchy that reflects the component hierarchy. +For instance, all the settings related to Cumulocity IoT share a `c8y` prefix, such as `c8y.url` for the Cumulocity URL. + +This file can be edited directly and can even be extended to include plugin-specific settings. +However, it's recommended to use the [`tedge config`](../../references/cli/tedge-config.md) command +to edit the settings as it provides guidance for expected settings and checks for invalid entries. + +## Common Commands + +The following is a list of common commands which can be used to get/set/list %%te%% configuration. + +### List configuration with descriptions + +Display a complete list of available settings with their purpose. + +```sh +tedge config list --doc +``` + +### List configuration that have been set or have defaults + +List the settings for which a specific value has been set. + +```sh +tedge config list +``` + +### Get a single configuration value + +Display the value for the `c8y.url` setting, if one has been set. + +```sh +tedge config get c8y.url +``` + +### Set configuration value + +Update/set the value for the `c8y.url` setting. + +``` +tedge config set c8y.url mytenant.cumulocity.com` +``` + +### Reset a configuration value to use default value + +Unset any user-specific value for the `c8y.url` setting, using then the default value. + +```sh +tedge config unset c8y.url +``` + +## Examples + +### Change path used for temporary files + +The following shows how to change the temp directory used by %%te%% and its components. + +1. Create a new directory which will be used by %%te%% + + ```sh + # create a directory (with/without sudo) + mkdir ~/tedge_tmp_dir + + # give ownership to tedge user and group + sudo chown tedge:tedge ~/tedge_tmp_dir + ``` + +2. Update the tedge configuration to point to the newly created directory + + ```sh title="Example" + sudo tedge config set tmp.path ~/tedge_tmp_dir + ``` + + :::info + The directory must be available to `tedge` user and `tedge` group. + ::: + +3. Restart the `tedge` daemons after any of these paths are updated, for it to take effect. + + ```sh + sudo systemctl restart tedge-agent + ``` + +To revert any of these paths back to their default locations, `unset` that config as follows: + +```sh +sudo tedge config unset tmp.path +``` + +Then restart the relevant services. + +```sh +sudo systemctl restart tedge-agent +``` + +## Customizing Settings + +### Configuration Path + +`/etc/tedge/tedge.toml` is the default location for %%te%% configuration. + +This can be changed by passing an explicit `--config-dir` to all the %%te%% command invocations. + +For instance, the following uses `/tmp/tedge.toml` to set the `c8y.url` and launch the Cumulocity mapper. + +```sh +tedge --config-dir /tmp config set c8y.url mytenant.cumulocity.com +tedge-mapper --config-dir /tmp c8y +``` + +### Environment variables + +To aid in configuring %%te%% in containerised environments, %%te%% supports passing in the configuration via environment variables. For instance, to configure the Cumulocity URL and MQTT bind address, you can run: + +```sh +env TEDGE_C8Y_URL=mytenant.cumulocity.com TEDGE_MQTT_BIND_ADDRESS=127.0.0.1 tedge connect c8y +``` + +The environment variables won't be stored anywhere, so you will need to set the relevant values when running the mapper and agent: + +```sh +env TEDGE_C8Y_URL=mytenant.cumulocity.com tedge-mapper c8y +env TEDGE_C8Y_URL=mytenant.cumulocity.com tedge-agent +``` + +The names for these environment variables are prefixed with `TEDGE_` to avoid conflicts with other applications, and any `.`s in the variable names are replaced with `_`s. Some example mappings are shown below: + +| Setting | Environment variable | +| ------------------- | ------------------------- | +| `c8y.url` | `TEDGE_C8Y_URL` | +| `device.key_path` | `TEDGE_DEVICE_KEY_PATH` | +| `device.cert_path` | `TEDGE_DEVICE_CERT_PATH` | +| `mqtt.bind.address` | `TEDGE_MQTT_BIND_ADDRESS` | + +You can also use `tedge config` to inspect the value that is set, which may prove useful if you are using a mix of toml configuration and environment variables. If you had tedge.toml file set as shown [above](#tedgetoml), you could run: + +```sh +tedge config get c8y.url +``` + +```text title="Output" +mytenant.cumulocity.com +``` + +Now we can run the same command but set an environment variable to override the value stored in the `tedge.toml` file. + +```sh +env TEDGE_C8Y_URL=example.com tedge config get +``` + +```text title="Output" +example.com +``` + +### User-specific Configurations + +The `/etc/tedge/tedge.toml` file can include extra settings used by user-specific plugins. + +When the %%te%% commands (`tedge`, `tedge-agent`, `tedge-mapper`) detect a configuration setting they don't recognise, +they will emit a warning log message: + +```sh +env TEDGE_C8Y_UNKNOWN_CONFIGURATION=test tedge config get c8y.url +``` + +```log title="Output" +2023-03-22 WARN tedge_config: Unknown configuration field "c8y.unknown_configuration" from environment variable TEDGE_C8Y_UNKNOWN_CONFIGURATION +mytenant.cumulocity.com +``` diff --git a/versioned_docs/version-1.1.0/operate/configuration/mapper-configuration.md b/versioned_docs/version-1.1.0/operate/configuration/mapper-configuration.md new file mode 100644 index 0000000..5e7b856 --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/configuration/mapper-configuration.md @@ -0,0 +1,62 @@ +--- +title: Mapper Configuration +tags: [Operate, Configuration, Cloud, MQTT] +description: How to control which MQTT topics the mappers subscribe to +--- + +The cloud-specific mappers subscribe to the reserved MQTT topics and convert incoming MQTT messages to cloud-specific messages. +In an advanced use case, such as using more than one cloud mappers for the same device, +you may want to customize the external tedge MQTT topics that each cloud mapper subscribes to. + +The `tedge config` command and the keys `c8y.topics`, `az.topics`, and `aws.topics` are usable for this use-case. + +| Cloud | tedge config key | Environmental variable | systemctl service | +|----------------|------------------|------------------------|-------------------| +| Cumulocity IoT | c8y.topics | TEDGE_C8Y_TOPICS | tedge-mapper-c8y | +| Azure IoT | az.topics | TEDGE_AZ_TOPICS | tedge-mapper-az | +| AWS IoT | aws.topics | TEDGE_AWS_TOPICS | tedge-mapper-aws | + +:::note +This guide uses `c8y.topics`, `TEDGE_C8Y_TOPICS`, and `tedge-mapper-c8y` as an example. +For other cloud mappers, use the keys in the table. +::: + +## Check the subscribed MQTT topics + +First, check which MQTT topics are subscribed by a cloud mapper. Run: + +```sh +tedge config get c8y.topics +``` + +```sh title="Output" +["te/+/+/+/+", "te/+/+/+/+/twin/+", "te/+/+/+/+/m/+", "te/+/+/+/+/e/+", "te/+/+/+/+/a/+", "te/+/+/+/+/status/health"] +``` + +## Set the desired new MQTT topics + +If you want to change the subscribed MQTT topics, use `tedge config set`. +For example, if you want the Cumulocity IoT mapper to subscribe only to `te/+/+/+/+/m/+` and `te/+/+/+/+/a/+` topic, +the command to run should be as below. + +```sh +sudo tedge config set c8y.topics te/+/+/+/+/m/+,te/+/+/+/+/a/+ +``` + +Alternatively, the same setting can be controlled via environment variables. +The environment variable settings will override any values set by the tedge config command. + +```sh +export TEDGE_C8Y_TOPICS=te/+/+/+/+/m/+,te/+/+/+/+/a/+ +``` + +:::note +If an invalid MQTT topic is given, the mapper will ignore it. +::: + +The service must be restarted for the setting to take effect. +The following command shows how to restart the Cumulocity IoT mapper on a device using systemd as the init system. + +```sh +sudo systemctl restart tedge-mapper-c8y +``` diff --git a/versioned_docs/version-1.1.0/operate/configuration/mosquitto-configuration.md b/versioned_docs/version-1.1.0/operate/configuration/mosquitto-configuration.md new file mode 100644 index 0000000..a82dd4e --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/configuration/mosquitto-configuration.md @@ -0,0 +1,98 @@ +--- +title: Mosquitto Configuration +tags: [Operate, Configuration, MQTT] +description: Mosquitto specific configuration guide +--- + +## Configuring mosquitto bind address and port {#mosquitto-bind-address} + +Configuring a mosquitto port and bind address in %%te%% is a three-step process. + +:::note +The mqtt.bind.port and the mqtt.bind.address can be set/unset independently. +::: + +### Step 1: Disconnect thin-edge.io edge device + +The %%te%% device has to be disconnected from the cloud using the `tedge` command + +```sh +tedge disconnect c8y + +#or +tedge disconnect az + +#or +tedge disconnect aws +``` + +### Step 2: Set the new mqtt port and bind address + +Use the `tedge` command to set the mqtt.bind.port and mqtt.bind.address with a desired port and bind address as below. + +```sh +sudo tedge config set mqtt.bind.port 1024 +``` + +```sh +sudo tedge config set mqtt.bind.address 127.0.0.1 +``` + +:::note +The bind_address is the address of one of the device network interface. +For example, this can be get as `ifconfig | grep inet` or set it to `0.0.0.0` +::: + +This will make sure that all the mqtt clients use the newer port and the bind address that +has been set once the device is connected to the cloud as in step 3. + +### Step 3: Connect the device to cloud + +Use the `tedge` command to connect to the desired cloud as below. + +```sh +tedge connect c8y + +#or +tedge connect az + +#or +tedge connect aws +``` + +This will configure all the services (mosquitto, tedge-mapper-c8y.service, tedge-mapper-az.service, +tedge-mapper-aws.service, tedge-agent.service) to use the newly set port and the bind address. + +## Common Errors + +The below example shows that we cannot set a string value for the port number. + +```sh +tedge config set mqtt.bind.port '"1234"' +``` + +```text title="Output" +Error: failed to set the configuration key: mqtt.bind.port with value: "1234". + +Caused by: + Conversion from String failed +``` + +## Updating the mqtt port and bind address (host) in collectd and for collectd-mapper + +Update the `collectd.conf` with the new port and host in ``. + +Then, restart the collectd service. + +```sh +sudo systemctl restart collectd +``` + +After changing the mqtt port and host, then connect to the cloud using `tedge connect c8y/az`. +Then (Steps 1-3) the collectd-mapper has to be restarted to use the newly set port and bind address (host). + +Restart the tedge-mapper-collectd service. + +```sh +sudo systemctl restart tedge-mapper-collectd +``` diff --git a/versioned_docs/version-1.1.0/operate/configuration/path-configuration.md b/versioned_docs/version-1.1.0/operate/configuration/path-configuration.md new file mode 100644 index 0000000..ef8cff4 --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/configuration/path-configuration.md @@ -0,0 +1,23 @@ +--- +title: Path Configuration +tags: [Operate, Configuration, Unix] +description: Customize %%te%% file/folder paths +--- + +## %%te%% directories + +The `tedge config set` command can be used to change various file system paths used by the %%te%% components. +The following table captures the paths that can be changed along with their default locations. + +| Config Key | Description | Default Value | +|------------|-------------|---------------| +| tmp.path | Directory where temporary files are created/stored. E.g: while downloading files | `/tmp` | +| logs.path | Directory where log files are created | `/var/log/tedge` | +| run.path | Directory where runtime information are stored | `/run` | +| data.path | Directory where data files are stored. E.g: Cached binary files, operation metadata etc | `/var/tedge` | + + +The following daemons also need to be re-started after `data.path` is updated: + +* `tedge-agent` +* `c8y-firmware-plugin` diff --git a/versioned_docs/version-1.1.0/operate/index.md b/versioned_docs/version-1.1.0/operate/index.md new file mode 100644 index 0000000..bc36803 --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/index.md @@ -0,0 +1,13 @@ +--- +title: Operate Devices +tags: [Operate] +sidebar_position: 3 +description: Working with devices that are using %%te%% +--- + +import DocCardList from '@theme/DocCardList'; + +This section is for the device operators who operate a device running %%te%% +and the cloud operators who manage a fleet of %%te%% devices. + + diff --git a/versioned_docs/version-1.1.0/operate/installation/index.md b/versioned_docs/version-1.1.0/operate/installation/index.md new file mode 100644 index 0000000..0fe864a --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/installation/index.md @@ -0,0 +1,11 @@ +--- +title: Installation +tags: [Installation] +sidebar_position: 1 +--- + +import DocCardList from '@theme/DocCardList'; + +How to install %%te%% on a specific hardware and operating systems + + diff --git a/versioned_docs/version-1.1.0/operate/installation/init-systems.md b/versioned_docs/version-1.1.0/operate/installation/init-systems.md new file mode 100644 index 0000000..f7fb980 --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/installation/init-systems.md @@ -0,0 +1,206 @@ +--- +title: Init Systems +tags: [Installation, Unix, Init, Services] +sidebar_position: 1 +description: Extended support for other init systems +--- + +# Init systems + +%%te%% supports Systemd out of the box, however not all Linux distributions use Systemd. To use %%te%% on a Linux distribution without Systemd requires a few extra steps. + +Support for different init systems (service managers) is provided by a community repository, [tedge-services](https://github.com/thin-edge/tedge-services). The following service definitions are currently supported (though check the community repository if you don't see your preferred init system in the list). + +* OpenRC +* runit +* s6-overlay +* SysVinit +* supervisord + +You are also free to use any service manager to run %%te%% how you want. Check out the [init system reference](../../references/init-system-configuration.md) guide to see how to create a configuration to interact with your preferred init system. + +:::tip +Contributions are welcome in the [tedge-services](https://github.com/thin-edge/tedge-services) repository to improve any of the services, or to add support for additional init systems. +::: + +## Install + +You can install the service definitions using a convenient script which will auto detect the init system for you. + +```sh tab={"label":"curl"} +curl -fsSL https://thin-edge.io/install-services.sh | sh -s +``` + +```sh tab={"label":"wget"} +wget -O - https://thin-edge.io/install-services.sh | sh -s +``` + +However, if you know which init system you are using on your device, and would like to manually specify it, then you can use the following command: + +```sh tab={"label":"curl"} +curl -fsSL https://thin-edge.io/install-services.sh | sh -s -- supervisord +``` + +```sh tab={"label":"wget"} +wget -O - https://thin-edge.io/install-services.sh | sh -s -- supervisord +``` + +## Alternative installation methods + +In cases were you would not like to run the automatic install script, you can choose one to run the steps manually. This allows you more control over the process which can be useful if you are experiencing problems with the auto detection used in the install script. + +### Manual repository setup and installation + +The software repositories used by the package managers can be configured using the setup scripts. These scripts are normally executed by the *install-services.sh* script in the installation section, however they can also be manually executed if you want more fine-grain control over the process. + +:::tip +If you are having problems setting any of the repositories, check out the [Cloudsmith](https://cloudsmith.io/~thinedge/repos/community/setup/#formats-deb) website where they have **Set Me Up** instructions in additional formats, e.g. manual configuration rather than via the `setup.*.sh` script. +::: + +**Pre-requisites** + +The instructions require you to have the following tools installed. + +* bash +* curl + +#### Setup + +**Running with sudo** + +You will need to have `sudo` also installed if you want to run these instructions. + +```sh tab={"label":"Debian/Ubuntu"} +curl -1sLf 'https://dl.cloudsmith.io/public/thinedge/community/setup.deb.sh' | sudo bash +``` + +```sh tab={"label":"RHEL/Fedora/RockyLinux"} +curl -1sLf 'https://dl.cloudsmith.io/public/thinedge/community/setup.rpm.sh' | sudo bash +``` + +```sh tab={"label":"Alpine"} +curl -1sLf 'https://dl.cloudsmith.io/public/thinedge/community/setup.alpine.sh' | sudo bash +``` + +**Running as root** + +These commands must be run as the root user. + +```sh tab={"label":"Debian/Ubuntu"} +curl -1sLf 'https://dl.cloudsmith.io/public/thinedge/community/setup.deb.sh' | bash +``` + +```sh tab={"label":"RHEL/Fedora/RockyLinux"} +curl -1sLf 'https://dl.cloudsmith.io/public/thinedge/community/setup.rpm.sh' | bash +``` + +```sh tab={"label":"Alpine"} +curl -1sLf 'https://dl.cloudsmith.io/public/thinedge/community/setup.alpine.sh' | bash +``` + +### Installing and updating using a package manager + +Once you have the repository setup, you can install the service definitions for your preferred init system. + +#### OpenRC + +```sh tab={"label":"Debian/Ubuntu"} +sudo apt-get install tedge-openrc +``` + +```sh tab={"label":"RHEL/Fedora/RockyLinux"} +sudo dnf install tedge-openrc +``` + +```sh tab={"label":"Alpine"} +sudo apk add tedge-openrc +``` + +#### runit + +```sh tab={"label":"Debian/Ubuntu"} +sudo apt-get install tedge-runit +``` + +```sh tab={"label":"RHEL/Fedora/RockyLinux"} +sudo dnf install tedge-runit +``` + +```sh tab={"label":"Alpine"} +sudo apk add tedge-runit +``` + +#### s6-overlay + +```sh tab={"label":"Debian/Ubuntu"} +sudo apt-get install tedge-s6overlay +``` + +```sh tab={"label":"RHEL/Fedora/RockyLinux"} +sudo dnf install tedge-s6overlay +``` + +```sh tab={"label":"Alpine"} +sudo apk add tedge-s6overlay +``` + +#### SysVinit + +```sh tab={"label":"Debian/Ubuntu"} +sudo apt-get install tedge-sysvinit +``` + +```sh tab={"label":"RHEL/Fedora/RockyLinux"} +sudo dnf install tedge-sysvinit +``` + +```sh tab={"label":"Alpine"} +sudo apk add tedge-sysvinit +``` + +#### supervisord + +```sh tab={"label":"Debian/Ubuntu"} +sudo apt-get install tedge-supervisord +``` + +```sh tab={"label":"RHEL/Fedora/RockyLinux"} +sudo dnf install tedge-supervisord +``` + +```sh tab={"label":"Alpine"} +sudo apk add tedge-supervisord +``` + +After installing the supervisord definitions, you will have to make sure the supervisord configuration pulls in the services definitions. Below shows an example `supervisord.conf` file which imports all %%te%% services definitions which were installed. + +```ini title="file: /etc/supervisord.conf" +# ... other supervisord settings + +[include] +files = /etc/supervisor/conf.d/*.conf +``` + +### Install via tarball + +You can force the install-services.sh script to install via the tarball instead of via a package manager. + +To install the service definitions via the tarball run the following command: + +```sh tab={"label":"curl"} +curl -fsSL https://thin-edge.io/install-services.sh | sh -s -- --package-manager tarball +``` + +```sh tab={"label":"wget"} +wget -O - https://thin-edge.io/install-services.sh | sh -s -- --package-manager tarball +``` + +Or if you also want to manually specify the init system to install, then you can the following command: + +```sh tab={"label":"curl"} +curl -fsSL https://thin-edge.io/install-services.sh | sh -s -- supervisord --package-manager tarball +``` + +```sh tab={"label":"wget"} +wget -O - https://thin-edge.io/install-services.sh | sh -s -- supervisord --package-manager tarball +``` diff --git a/versioned_docs/version-1.1.0/operate/monitoring/index.md b/versioned_docs/version-1.1.0/operate/monitoring/index.md new file mode 100644 index 0000000..e3299a7 --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/monitoring/index.md @@ -0,0 +1,11 @@ +--- +title: Monitoring +tags: [Operate, Monitoring] +sidebar_position: 1 +--- + +import DocCardList from '@theme/DocCardList'; + +How to monitor a device running thin-edge + + diff --git a/versioned_docs/version-1.1.0/operate/monitoring/systemd-watchdog.md b/versioned_docs/version-1.1.0/operate/monitoring/systemd-watchdog.md new file mode 100644 index 0000000..3bc638a --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/monitoring/systemd-watchdog.md @@ -0,0 +1,104 @@ +--- +title: Systemd Watchdog +tags: [Operate, Monitoring] +sidebar_position: 1 +unlisted: true +description: Enabling systemd watchdog for %%te%% services +--- + +## Introduction + +The systemd watchdog feature enables systemd to detect when a service is unhealthy or unresponsive and +attempt to fix it by restarting that service. +To detect if a service is healthy or not, systemd relies on periodic health notifications from that service at regular intervals. +If the service fails to send that notification within a time threshold, +then systemd will assume that service to be unhealthy and restart it. + +This document describes how the systemd watchdog mechanism can be enabled for %%te%% services. + +## Enabling the systemd watchdog feature for a tedge service + +Enabling systemd watchdog for a %%te%% service (tedge-agent, tedge-mapper-c8y/az/collectd) is a two-step process. + +### Step 1: Enable the watchdog feature in the systemd service file + +For example, to enable the watchdog feature for `tedge-mapper-c8y` service, +update the systemd service file as shown below: + +:::note +The systemd service file for tedge services are usually present in `/lib/systemd/system` directory, +like `/lib/systemd/system/tedge-mapper-c8y.service`. +::: + +Add `tedge-watchdog.service` as an `After` service dependency under `[Unit]` section. +Add the watchdog interval as `WatchdogSec=30` under `[Service]` section. +Update the restart condition as `Restart=always` under `[Service]` section. + +Here is the updated service file for `tedge-mapper-c8y` service: + +```sh +[Unit] +Description=tedge-mapper-c8y converts Thin Edge JSON measurements to Cumulocity JSON format. +After=syslog.target network.target mosquitto.service tedge-watchdog.service + +[Service] +User=tedge-mapper +ExecStart=/usr/bin/tedge-mapper c8y +Restart=always +RestartPreventExitStatus=255 +WatchdogSec=30 +``` + +### Step 2: Start the tedge-watchdog service + +The `tedge-watchdog` service is responsible for periodically checking the health of +all tedge services for which the watchdog feature is enabled, +and send systemd watchdog notifications on their behalf to systemd. + +Start and enable the `tedge-watchdog` service as follows: + +```sh +sudo systemctl enable tedge-watchdog +sudo systemctl start tedge-watchdog +``` + +Once started, the `tedge-watchdog` service will keep checking the health of the monitored tedge services by periodically +sending health check messages to them within their configured `WatchdogSec` interval. + +The health check request for service is published to `te/device/main/service//cmd/health/check` topic and +the health status response from that service is expected on `te/device/main/service//status/health` topic. + +Once the health status response is received from a particular service, the `tedge-watchdog` service will send the +[systemd notification](https://www.freedesktop.org/software/systemd/man/sd_notify.html#) to systemd on behalf of that +monitored service. + +:::note +If none of the %%te%% services are enabled with the watchdog feature, then the `tedge-watchdog` service will stop with an `inactive` state. +To monitor any of the %%te%% services, one has to update the corresponding `systemd` service file with `WatchdogSec` +and then restart the `tedge-watchdog` service. +::: + +## Debugging + +One can observe the message exchange between the `service` and the `watchdog` +by subscribing to the following topics: + + +```sh te2mqtt formats=v1 +tedge mqtt sub 'te/+/+/+/+/status/health' +``` + +```sh te2mqtt formats=v1 +tedge mqtt sub 'te/+/+/+/+/cmd/health/check' +``` + +Check out the [Monitor tedge health](../troubleshooting/monitoring-service-health.md) for more details about the health endpoint. + +:::note +If the watchdog service does not send the notification to the systemd within `WatchdogSec` interval for a service, +then systemd restarts that service by killing the old process and spawning a new one to replace it. +::: + +:::info +[Here](https://www.medo64.com/2019/01/systemd-watchdog-for-any-service/) is an example about using `systemd watchdog` feature. +::: diff --git a/versioned_docs/version-1.1.0/operate/plugins/github_plugins_list.png b/versioned_docs/version-1.1.0/operate/plugins/github_plugins_list.png new file mode 100644 index 0000000..9cb96ed Binary files /dev/null and b/versioned_docs/version-1.1.0/operate/plugins/github_plugins_list.png differ diff --git a/versioned_docs/version-1.1.0/operate/plugins/index.md b/versioned_docs/version-1.1.0/operate/plugins/index.md new file mode 100644 index 0000000..3904526 --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/plugins/index.md @@ -0,0 +1,32 @@ +--- +title: Community Plugins +tags: [Documentation] +sidebar_position: 10 +--- + +import { PluginCardsList } from '@site/src/components/PluginCardsList'; + +The following pages contain a list of available community plugins. You can use these plugins directly or as a reference for creating your own plugins. + +## Plugins List + +The list can be filtered by keywords and/or text to find plugins which are useful to you. + + + +### Submitting a new plugin + +The list of plugins included in the documentation is maintained in the [documentation repository](https://github.com/thin-edge/tedge-docs?tab=readme-ov-file#community-plugin-documentation). + +New plugins can be added by following these [instructions](https://github.com/thin-edge/tedge-docs?tab=readme-ov-file#community-plugin-documentation). + +## Github hosted plugins + +In addition, a list of [Github plugins](https://github.com/topics/thin-edge) are visible directly from Github. If your plugin is hosted on Github as a public repository, then you can make your plugin discoverable by adding the `thin-edge` tag to the repository, afterwards the plugin will appear in the list. + +
+ ![Github %%te%% plugins](github_plugins_list.png) +
+ Github plugins +
+
diff --git a/versioned_docs/version-1.1.0/operate/registration/deregister.md b/versioned_docs/version-1.1.0/operate/registration/deregister.md new file mode 100644 index 0000000..a8f0f6d --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/registration/deregister.md @@ -0,0 +1,93 @@ +--- +title: Deregister entities +tags: [Child-Device, Registration] +sidebar_position: 1 +description: Deregister child devices and services from %%te%% +--- + +# Deregister entities + +All the entities (devices and services) registered with %%te%% are stored with the MQTT broker as retained messages, +with their metadata spread across multiple topics. +For example, for a child device `child01`, the registration message is stored on the `te/device/child01//` topic, +its twin data is stored across several `te/device/child01///twin/` topics, +its command metadata stored across several `te/device/child01///cmd/` topics and so on. + +On top of that, when a device has services associated with it, or has nested child devices, +they all have their own respective registration topics and other metadata topics. +So, deregistering a device involves deregistering itself, its metadata and +the complete entity hierarchy that is associated with it. + +Even though %%te%% doesn't provide a direct API for this yet, this can be easily be done using third-party tools +like `mosquitto_sub` that supports clearing multiple retained messages together using the `--remove-retained` option. + +## Deregister a child device and its services + +When the topic ids of the devices and services follow the default topic scheme, +which maintains the service topic ids (`te/device//service/`) +directly under the device topic id (`te/device///`) in the hierarchy, +a child device along with all of its services can be deleted as follows: + +```sh +mosquitto_sub --remove-retained -W 3 -t "te/device/child01/+/+/#" +``` + +* The topic filter `te/device/child01/+/+` covers the registration messages of the device and all of its services, + and the trailing `#` covers their their metadata messages as well. +* The `--remove-retained` clears all the retained messages matching that topic filter. + Running the same command without the `remove-retained` option shows the list of messages + that would be cleared with this command, which can be used for dry runs before the entities are really removed. +* The `-W 3` option is used to stop the command after a timeout period of 3 seconds, without which the command will not exit. + Adjust this timeout value based on the number of entities in your deployment. + +:::note +The entities in the cloud must also be removed separately as deregistrations on the device +are not propagated to the cloud. +The Cumulocity mapper must also be restarted if the same entity is to be recreated after deregistration. +::: + +A single service of a device (main or child), can be deregistered as follows: + +```sh +mosquitto_sub --remove-retained -W 3 -t "te/device/main/service/service01/#" +``` + +All the services associated to a device can be deregistered together, as follows: + +```sh +mosquitto_sub --remove-retained -W 3 -t "te/device/child01/service/+/#" +``` + +:::note +De-registering parent and child entities together with a single command only works +when the parent and children are hierarchically linked via the topics, which can be queried with a single topic filter, +as in the case of services while following the default topic scheme. +While using custom topic schemes, that do not maintain the same topic prefix for a device and its services, +querying and deregistering them may not be a on-liner as in the examples listed above. +::: + +## Deregister all child devices and services + +All the child devices and services can be deregistered using a wildcard topic filter that covers all the entities +combined with a topic exclusion filter for the main device, as follows: + +```sh +mosquitto_sub -v -t "te/device/+/#" -T "te/device/main/#" +``` + +## Unsupported cases + +While using the default topic scheme, the following use-cases are not supported as it's not possible to define +wildcard topic filters for the same, as the topic scheme does not capture the parent-child relationship of devices: + +* Deregister all child devices of a given device +* Deregister the entire nested child device hierarchy of a device + +## Custom topic schemes + +Most of the commands listed above are applicable only when the default topic scheme is used. +The same rules, especially the ones involving multiple entities, do not apply to custom topic schemes, +as different entities would be linked differently based on the topic scheme that's used. +The filtering capabilities would also very from scheme to scheme. +As long as you can define topic filters to select individual entities or a set of entities, +the same can be used for deregistering as well. diff --git a/versioned_docs/version-1.1.0/operate/registration/index.md b/versioned_docs/version-1.1.0/operate/registration/index.md new file mode 100644 index 0000000..d9c3cfd --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/registration/index.md @@ -0,0 +1,11 @@ +--- +title: Registration +tags: [Registration, Deregistration] +sidebar_position: 11 +--- + +import DocCardList from '@theme/DocCardList'; + +Managing registration and deregistration of child devices and services on %%te%%. + + diff --git a/versioned_docs/version-1.1.0/operate/security/certificate-signing-request.md b/versioned_docs/version-1.1.0/operate/security/certificate-signing-request.md new file mode 100644 index 0000000..970298f --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/security/certificate-signing-request.md @@ -0,0 +1,100 @@ +--- +title: Certificate signing request +tags: [Operate, Security, Cloud] +description: Generate certificate signing request for %%te%% +--- + +If you want to use a device certificate which is signed by a Certificate Authority (CA), you can generate the Certificate Signing Request (CSR), which is later used by CA to generate a device certificate. This process requires additional tooling as %%te%% provides you only with the CSR. + +## Create a certificate signing request + +To create a CSR you can use [`tedge cert create-csr`](../../references/cli/tedge-cert.md) %%te%% command: + +```sh +sudo tedge cert create-csr --device-id alpha +``` + +or + +```sh +sudo tedge cert create-csr +``` +if a device certificate already exists and you want to reuse device name. + + +```text title="Output" +Certificate Signing Request was successfully created. +``` + +:::note +`tedge cert` requires `sudo` privilege as creating a private key owned by the MQTT broker. This command provides no output on success. +::: + +Now you should have a CSR in the `/etc/tedge/device-certs/` directory: + +```sh +ls -l /etc/tedge/device-certs/ +``` + +```text title="Output" +total 8 +-r--r--r-- 1 mosquitto mosquitto 664 May 31 09:26 tedge.csr +-r-------- 1 mosquitto mosquitto 246 May 31 09:26 tedge-private-key.pem +``` + +[`sudo tedge cert create-csr`](../../references/cli/tedge-cert.md) creates the certificate signing request in a default location (`/etc/tedge/device-certs/`). To use a custom location, refer to [`tedge config`](../../references/cli/tedge-config.md) or provide absolute path as a command argument: + +```sh +sudo tedge cert create-csr --device-id alpha --output-path /custom/path/mycsr.csr +``` + +:::note +`tedge cert create-csr` will reuse the private key if already created, e.g by the `tedge cert create` command. +::: + +To check the content of CSR, you can use external tools, like `openssl`. + +```sh +openssl req -in /etc/tedge/device-certs/tedge.csr -noout -text +``` + +```text title="Output" +Certificate Request: + Data: + Version: 1 (0x0) + Subject: CN = alpha, O = Thin Edge, OU = Test Device + Subject Public Key Info: + Public Key Algorithm: id-ecPublicKey + Public-Key: (256 bit) + pub: + 04:95:e6:48:48:9b:8e:03:a0:fd:07:41:e6:e7:25: + 21:3b:ed:c7:8d:13:2f:69:a7:94:17:43:7c:da:ca: + 33:fb:bb:93:fe:eb:c1:50:65:c2:47:70:87:5e:ab: + a3:d5:ec:9b:5c:65:7a:ba:7d:92:20:a1:80:9b:d6: + 79:71:be:15:56 + ASN1 OID: prime256v1 + NIST CURVE: P-256 + Attributes: + (none) + Requested Extensions: + Signature Algorithm: ecdsa-with-SHA256 + Signature Value: + 30:46:02:21:00:81:28:11:28:9b:92:cb:b8:d9:d2:1c:3c:8d: + 00:1f:4e:44:ae:ba:61:7f:ca:17:75:d9:d4:11:04:fa:11:e8: + a2:02:21:00:f2:f4:11:77:5c:32:c8:d5:86:66:29:d5:ae:27: + 3f:64:31:be:f8:4a:89:29:bf:e0:01:b4:f2:63:1f:f0:f0:fb +``` + +## Errors + +### Certificate Signing Request creation fails due to invalid device id + +If non-supported characters are used for the device id then the cert create-csr will fail with below error: + +```text +Error: failed to Generate the Certificate Signing Request. + +Caused by: + 0: DeviceID Error + 1: The string '"+"' contains characters which cannot be used in a name [use only A-Z, a-z, 0-9, ' = ( ) , - . ? % * _ ! @] +``` diff --git a/versioned_docs/version-1.1.0/operate/security/cloud-authentication.md b/versioned_docs/version-1.1.0/operate/security/cloud-authentication.md new file mode 100644 index 0000000..176918f --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/security/cloud-authentication.md @@ -0,0 +1,92 @@ +--- +title: Cloud Authentication +tags: [Operate, Security] +description: Configuring certificates for your cloud connection +--- + +When %%te%% connects a cloud, the cloud endpoint is authenticated using X.509 certificates. +For that to work, the signing certificate of the cloud certificate must be trusted by the device. +Usually, these certificates are stored in `/etc/ssl/certs` and nothing specific has to done on the device. + +A specific configuration will be required only for cloud endpoints which CA is not trusted by the device OS default setting. + +## Configuration + +Several `tedge config` settings are used by %%te%% to locate the signing certificate of the cloud endpoint. + +- `c8y.root_cert_path` The path where Cumulocity IoT root certificate(s) are stored (MQTT) +- `c8y.proxy.ca_path` The path where Cumulocity IoT root certificate(s) are stored (HTTP) +- `aws.root_cert_path` The path where AWS IoT root certificate(s) are stored (MQTT). +- `az.root_cert_path` The path where Azure IoT root certificate(s) are stored (MQTT). + +All these paths can point to a directory where the trusted certificates are stored +as well as directly to file containing the certificate of the authority signing the cloud endpoint certificate. +Per default, all these paths are set to the system default: `/etc/ssl/certs`. + +## Adding a Root Certificate + +If the server you are trying to connect %%te%% to is presenting a certificate with a root that is not currently trusted, +then you can add the server's root certificate to the list of trusted root certificates. +For the most part the store will be filled with certificates from your TLS/SSL provider, +but if this is not the case you may need to update your local certificate store. + +:::note +Updating the local certificate store is notably required to connect Cumulocity IoT Edge, +as this distribution of Cumulocity uses self-signed certificates to authenticate itself. +::: + +Below are instructions on how to add new CA certificate and update the certificate store. + +:::note +Provided instructions are for supported OSes and may not apply to the flavour you are running, +if you need help with other OS please consult appropriate documentation. +::: + +### Debian/Ubuntu/RaspberryPi OS + +If you do not have the `ca-certificates` package installed on your system, install it with your package manager. + +```sh +sudo apt install ca-certificates +``` + +To add a self-signed certificate to the trusted certificate repository on %%te%% system: + +Create a `/usr/local/share/ca-certificates/` directory if it does not exist on your computer: + +```sh +sudo mkdir /usr/local/share/ca-certificates/ +``` + +The directory should be owned by `root:root` and have `755` permissions set for it. The certificates files should be `644`. + +Copy your root certificate (in `PEM` format with `.crt` extension) to the created directory: + +```sh +sudo cp /usr/local/share/ca-certificates/ +``` + +Install the certificates: + +```sh +sudo update-ca-certificates +``` + +```text title="Output" +Updating certificates in /etc/ssl/certs... +1 added, 0 removed; done. +Running hooks in /etc/ca-certificates/update.d... +done. +``` + +Check the certificate was correctly installed: + +```sh +ls /etc/ssl/certs | grep +``` + +Additionally, you can check correctness of the installed certificate: + +```sh +cat /etc/ssl/certs/ca-certificates.crt | grep -f +``` diff --git a/versioned_docs/version-1.1.0/operate/security/cumulocity-token.md b/versioned_docs/version-1.1.0/operate/security/cumulocity-token.md new file mode 100644 index 0000000..fd3b4d4 --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/security/cumulocity-token.md @@ -0,0 +1,184 @@ +--- +title: Cumulocity IoT Token +tags: [Operate, Security, Cumulocity, JWT] +description: Requesting a token for manual Cumulocity IoT API requests +--- + +:::tip +%%te%% provides an alternative way to access the Cumulocity IoT REST API without having to request a token. The [Cumulocity IoT Proxy Service](../../references/cumulocity-proxy.md) can be used which handles the authorization as required. + +It is recommended to use the [Cumulocity IoT Proxy Service](../../references/cumulocity-proxy.md) where possible. +::: + +## Overview + +For instances where you cannot use the [Cumulocity IoT Proxy Service](../../references/cumulocity-proxy.md), the following instructions detail how to manually request a token, and then how to use the token to make manual REST API calls to the Cumulocity IoT tenant. + +## Retrieving the token + +Follow the below steps in order to retrieve the token from Cumulocity IoT using MQTT. + +1. Subscribe to the token topic + + ```sh te2mqtt formats=v1 + tedge mqtt sub c8y/s/dat --no-topic + ``` + +2. Publish an empty message on the `c8y/s/uat` topic + + ```sh te2mqtt formats=v1 + tedge mqtt pub c8y/s/uat '' + ``` + +3. After a while the token will be published on the subscribed topic `c8y/s/dat` in the below format + + ```sh + 71,${Base64 encoded JWT} + ``` + + Store the token as required (e.g. assign to a variable), and use in REST API request to the Cumulocity IoT tenant. + + :::note + Typically tokens are only valid for 1 hour (depending on your Cumulocity IoT settings), so you will need to request a new token before it expires otherwise your API Requests will return an Unauthorized (401) error. + + The expiration timestamp, issuer (Cumulocity IoT URL), and tenant are encoded in the token and can be decoded using the standard [JSON Web Token Standard](https://datatracker.ietf.org/doc/html/rfc7519) which there should be a library in most popular programming languages. + ::: + +### Alternative: Retrieving the token using mosquitto_rr + +`mosquitto_rr` provides a simple cli interface to handle the request/response pattern, and can be used to create a simple one-liner to retrieve a new token. + +The `mosquitto_rr` cli command can be installed on your operating system via one of the following packages: + +```sh tab={"label":"Debian/Ubuntu"} +mosquitto-clients +``` + +```sh tab={"label":"RHEL/Fedora/RockyLinux"} +mosquitto +``` + +```sh tab={"label":"openSUSE"} +mosquitto-clients +``` + +```sh tab={"label":"Alpine"} +mosquitto +``` + +The token can be retrieved using the following one-liner: + +```sh +export C8Y_TOKEN=$(mosquitto_rr -t c8y/s/uat -e c8y/s/dat -m '' | cut -d, -f2-) +``` + +Where: +* `-t` represents the request topic, where the `-m ''` message is sent to it +* `-e` represents the response topic which will print the message on the console + + +## Using token in REST API calls to Cumulocity IoT + +The retrieved token can be used to make HTTP calls to the [Cumulocity IoT REST API](https://cumulocity.com/api/core/). + +For simplicity, this example will retrieve the Cumulocity IoT URL via the tedge cli command. If you are using a higher level programming language like python3, then you get the Cumulocity IoT URL by decoding the token, e.g. using [PyJWT](https://pyjwt.readthedocs.io/en/latest/). + +The following code snippet does the following steps: + +1. Get the Cumulocity IoT URL (using `tedge`) +2. Get the token (using `mosquitto_rr`) +3. Send a request (using `curl`) + +```sh +# Get the Cumulocity IoT URL +export C8Y_URL="https://$(tedge config get c8y.url)" + +# Get the token (using mosquitto_rr cli command) +export C8Y_TOKEN=$(mosquitto_rr -t c8y/s/uat -e c8y/s/dat -m '' | cut -d, -f2-) + +# Send a REST API call to Cumulocity IoT +curl -s -H "Authorization: Bearer $C8Y_TOKEN" \ + -H "Accept: application/json" \ + "$C8Y_URL/user/currentUser" +``` + +:::tip +The same functionality (as above) can be achieved by using the [Cumulocity IoT Proxy Service](../../references/cumulocity-proxy.md): + +```sh +curl -s -H "Accept: application/json" \ + "http://localhost:8001/c8y/user/currentUser" +``` +::: + +```text title="Output (pretty printed)" +{ + "shouldResetPassword": false, + "userName": "device_rpi4-d83add90fe56", + "self": "https://t123456.eu-latest.cumulocity.com/user/currentUser", + "effectiveRoles": [ + { + "name": "ROLE_IDENTITY_ADMIN", + "self": "https://t123456.eu-latest.cumulocity.com/user/roles/ROLE_IDENTITY_ADMIN", + "id": "ROLE_IDENTITY_ADMIN" + }, + { + "name": "ROLE_EVENT_READ", + "self": "https://t123456.eu-latest.cumulocity.com/user/roles/ROLE_EVENT_READ", + "id": "ROLE_EVENT_READ" + }, + { + "name": "ROLE_IDENTITY_READ", + "self": "https://t123456.eu-latest.cumulocity.com/user/roles/ROLE_IDENTITY_READ", + "id": "ROLE_IDENTITY_READ" + }, + { + "name": "ROLE_INVENTORY_READ", + "self": "https://t123456.eu-latest.cumulocity.com/user/roles/ROLE_INVENTORY_READ", + "id": "ROLE_INVENTORY_READ" + }, + { + "name": "ROLE_DEVICE_CONTROL_READ", + "self": "https://t123456.eu-latest.cumulocity.com/user/roles/ROLE_DEVICE_CONTROL_READ", + "id": "ROLE_DEVICE_CONTROL_READ" + }, + { + "name": "ROLE_MEASUREMENT_READ", + "self": "https://t123456.eu-latest.cumulocity.com/user/roles/ROLE_MEASUREMENT_READ", + "id": "ROLE_MEASUREMENT_READ" + }, + { + "name": "ROLE_AUDIT_READ", + "self": "https://t123456.eu-latest.cumulocity.com/user/roles/ROLE_AUDIT_READ", + "id": "ROLE_AUDIT_READ" + }, + { + "name": "ROLE_USER_MANAGEMENT_OWN_ADMIN", + "self": "https://t123456.eu-latest.cumulocity.com/user/roles/ROLE_USER_MANAGEMENT_OWN_ADMIN", + "id": "ROLE_USER_MANAGEMENT_OWN_ADMIN" + }, + { + "name": "ROLE_ALARM_READ", + "self": "https://t123456.eu-latest.cumulocity.com/user/roles/ROLE_ALARM_READ", + "id": "ROLE_ALARM_READ" + }, + { + "name": "ROLE_DEVICE", + "self": "https://t123456.eu-latest.cumulocity.com/user/roles/ROLE_DEVICE", + "id": "ROLE_DEVICE" + }, + { + "name": "ROLE_USER_MANAGEMENT_OWN_READ", + "self": "https://t123456.eu-latest.cumulocity.com/user/roles/ROLE_USER_MANAGEMENT_OWN_READ", + "id": "ROLE_USER_MANAGEMENT_OWN_READ" + }, + { + "name": "ROLE_INVENTORY_CREATE", + "self": "https://t123456.eu-latest.cumulocity.com/user/roles/ROLE_INVENTORY_CREATE", + "id": "ROLE_INVENTORY_CREATE" + } + ], + "id": "device_rpi4-d83add90fe56", + "lastPasswordChange": "2024-01-14T19:56:05.978Z" +} +``` diff --git a/versioned_docs/version-1.1.0/operate/security/device-certificate.md b/versioned_docs/version-1.1.0/operate/security/device-certificate.md new file mode 100644 index 0000000..5fdf5d8 --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/security/device-certificate.md @@ -0,0 +1,80 @@ +--- +title: Device Certificate Settings +tags: [Operate, Security, Cloud] +description: Controlling device certificate settings +--- + +## Obtaining a Device Certificate + +Obtaining a Device Certificate is a quite involved process: +- A private key and a certificate signing request has to be created on the device +- The certificate signing request is sent to the Certificate Authority (CA) chosen by the tenant for its fleet of devices +- The CA checks that the device issuing the signing request is actually allowed to connect the tenant +- The CA signs the signing request sending back the resulting certificate to the device +- The tenant adds the signing certificate of the CA to its trusted list of certificate + +There are numerous variations notably on the contract between the tenant and the CA, +the chain of signing certificates and the checks to be performed before approving a signing request. +However, the outcome of the certificate signing process is always the same. +It consists in three files: +- The device private key - generated by the device and which must be kept secret as this the proof of ownership +- The device certificate - signed by the CA and shared by the device on-demand +- The signing certificate - the certificate used by the CA to sign the certificate and by the tenant to check device certificates + +## Installing a Device Certificate + +The device certificate must be installed on the gateway device, +i.e. on the same box as the MQTT broker establishing the MQTT connection to the cloud over a bridge. + +```text title="ls -lh /etc/mosquitto/certs" +-r-------- 1 mosquitto root 1679 oct. 21 2022 demo-device-007.key +-r--r--r-- 1 mosquitto root 2095 oct. 21 2022 demo-device-007.pem +``` + +Note that *only* the MQTT broker should be able to read or write the private key, while the certificate itself is public. + +Then %%te%% must be told where the certificate is stored. +Note that the device certificate file must contain not only the device certificate itself +but also the signing certificate +(so the cloud endpoint can check the chain, starting from the device certificate up to the trusted root). + +```sh +# Create a new cert chain, which contains both the device public cert and the signing (in that order) +cat demo-device-007.pem > demo-device-007.chain.pem +cat signing-cert.pem >> demo-device-007.chain.pem + +# Configure the cert chain file as the main cert to be used by %%te%% +tedge config set device.cert_path /etc/mosquitto/certs/demo-device-007.chain.pem +tedge config set device.key_path /etc/mosquitto/certs/demo-device-007.key +``` + +The `tedge cert show` command can be used to look at the content of the certificate. + +```text title="tedge cert show"" +Device certificate: /etc/mosquitto/certs/demo-device-007.chain.pem +Subject: O=Thin-Edge, OU=t398942, CN=demo-device-007 +Issuer: C=DE, O=Software AG, CN=QA Thin-Edge CA G1 +Valid from: Tue, 09 Nov 2021 14:38:41 +0000 +Valid up to: Sat, 09 Nov 2024 14:38:41 +0000 +Thumbprint: 1E0F9A074E6FE67A43EE948335E42EB729CB3974 +``` + +## Cloud tenant setting + +The last point is to make the cloud tenant trust the device certificate. +For that the certificate of the issuer - .i.e. "QA Thin-Edge CA G1" in the example case, +must be added to the list of trusted signing certificate. + +This process is cloud dependent. +See the respective documentation of the cloud you're trying to connect to for details on how to add a new signing certificate: +- [Cumulocity IoT: Managing trusted certificates](https://cumulocity.com/guides/users-guide/device-management/#managing-trusted-certificates) +- [Understand how Azure IoT Edge uses certificates](https://learn.microsoft.com/en-us/azure/iot-edge/iot-edge-certs) +- [AWS IoT: X.509 client certificates](https://docs.aws.amazon.com/iot/latest/developerguide/x509-client-certs.html) + +:::note +The command `tedge cert upload` is of no help here. + +Indeed, the device does not have a copy of the signing certificate of the CA. +This certificate is given by the CA to the tenant owner or the device operator +in the context of the signing process. +::: diff --git a/versioned_docs/version-1.1.0/operate/security/https-configuration.md b/versioned_docs/version-1.1.0/operate/security/https-configuration.md new file mode 100644 index 0000000..27848c7 --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/security/https-configuration.md @@ -0,0 +1,411 @@ +--- +title: HTTPS Configuration +tags: [Operate, Security, HTTP] +description: Setting up HTTPS for secure local communication +--- + +%%te%% provides two services over HTTP: +- The [File Transfer Service](../../references/file-transfer-service.md) is used by the mappers and the child devices to transfer files locally. +- The [Cumulocity Proxy](../../references/cumulocity-proxy.md) acts as a local proxy to the Cumulocity IoT REST API. + +Three levels of security are supported: + +1. HTTP without any client authentication (default) +2. HTTPS with server authentication +3. HTTPS with server and client authentication + + +## Default HTTP Setting + +### File Transfer Service + +The **tedge-agent** running on the main device acts as a local HTTP server +which is the mappers and the child devices to transfer files locally. +Any local process can PUT and GET files there: + +```sh +echo "Hello thin-edge.io" >/tmp/foo.txt +curl -X PUT -F 'file=@/tmp/foo.txt' http://localhost:8000/tedge/file-transfer/foo.txt +curl http://localhost:8000/tedge/file-transfer/foo.txt +``` + +### Cumulocity Proxy + +When a device is successfully connected to Cumulocity IoT, +**tedge-mapper** acts as a proxy to the Cumulocity IoT REST API. +For instance, the following lists the managed objects related to the device: + +```sh +curl http://localhost:8001/c8y/inventory/managedObjects +``` + +:::note +The connection from the Cumulocity Proxy to Cumulocity IoT is always established over HTTPS, +whatever the settings for local connections. + +The identity of the Cumulocity end-point is authenticated using the +root certificates configured with `tedge config get c8y.proxy.ca_path` and the device itself +is authenticated using JWT tokens retrieved from Cumulocity IoT via MQTT. +::: + +## Open HTTP services on the local network + +Per default, the File Transfer Service and Cumulocity Proxy are only available on `localhost`, +the loopback network interface of the main device. +To open these services to child devices, their *bind* addresses must be set explicitly. + +### File Transfer Service + +On the __main device__, two `tedge config` settings define the binding address and port of the File Transfer Service: + +- `http.bind.address` The bind address of the File Transfer Service HTTP server +- `http.bind.port` The port number of the File Transfer Service HTTP server + +On a __child device__, two `tedge config` settings define how to connect the File Transfer Service: + +- `http.client.host` The address or hostname of the main device where the File Transfer Service HTTP server is running +- `http.client.port` The port number on main device on which the File Transfer Service HTTP server is running + +For instance, assuming a main device named `rpi4-dca632efb150` with `192.168.1.6` as IP address, +one has first to configure and restart *tedge-agent* of the main device: + +```sh title="main device" +sudo tedge config set http.bind.address 192.168.1.6 +sudo systemctl restart tedge-agent +``` + +and then, on the child device, *tedge-agent* needs to be configured to use the File Transfer Service of the main device: + +```sh title="child device" +sudo tedge config set http.client.host rpi4-dca632efb150 +sudo systemctl restart tedge-agent +``` + +Any process running on a child device can also use the File Transfer Service: + +```sh title="child device" +echo "Hello thin-edge.io" >/tmp/foo.txt +curl -X PUT -F 'file=@/tmp/foo.txt' http://rpi4-dca632efb150:8000/tedge/file-transfer/foo.txt +curl http://rpi4-dca632efb150:8000/tedge/file-transfer/foo.txt +``` + +:::note +The firewall of the main device has also to be configured +to accept incoming requests on the port used by the File Transfer Service (`8000` per default) +::: + +### Cumulocity Proxy + +As for the File Transfer Service, the Cumulocity Proxy is configured using four settings, +two to be used by the Cumulocity **tedge-mapper**, and two used by its clients +(mainly the **tedge-agent**, running on the main device or a child devices). + +- `c8y.proxy.bind.address` The IP address local Cumulocity HTTP proxy binds to +- `c8y.proxy.bind.port` The port local Cumulocity HTTP proxy binds to +- `c8y.proxy.client.host` The address of the host on which the local Cumulocity HTTP Proxy is running +- `c8y.proxy.client.port` The port number on the remote host on which the local Cumulocity HTTP Proxy is running + +For instance, assuming a main device named `rpi4-dca632efb150` with `192.168.1.6` as IP address, +one has first to configure and restart *tedge-mapper-c8y*: + +```sh title="device running tedge-mapper-c8y" +sudo tedge config set c8y.proxy.bind.address 192.168.1.6 +sudo systemctl restart tedge-mapper-c8y +``` + +and then, on all the devices, *tedge-agent* needs to be configured to connect the Cumulocity HTTP Proxy: + +```sh title="main and child device" +sudo tedge config set c8y.proxy.client.host rpi4-dca632efb150 +sudo systemctl restart tedge-agent +``` + +Any process running on a child device can also use the Cumulocity HTTP Proxy: + +```sh title="main and child device" +curl http://rpi4-dca632efb150:8001/c8y/inventory/managedObjects +``` + +:::note +The firewall of the box running the Cumumocity mapper has also to be configured +to accept incoming requests on the port used by the Cumulocity HTTP Proxy (`8001` per default) +::: + +## Enable HTTPS + +The next step is to enable HTTPS, the clients authenticating the servers. + +For that, one needs two certificates: +- one for the main **tedge-agent** running the File Transfer Service +- another for the Cumulocity **tedge-mapper** running the Cumulocity HTTP Proxy. + +These two certificates have also to be trusted by the clients, i.e. the child devices. +Hence, the signing certificate will have to be added to the list of trusted root certificates on each child device. + +### Generating Certificates + +%%te%% currently provides no specific tool to generate and deploy certificates over child devices. + +You can use the following script to generate all the required certificates. + +First, one needs a signing key that will be used to sign the certificates of the **tedge-agent** and **tedge-mapper**: + +```sh +DEVICE=$(tedge config get device.id) + +## Signing certificate +openssl req \ + -new \ + -x509 \ + -days 100 \ + -extensions v3_ca \ + -nodes \ + -subj "/O=thin-edge/OU=$DEVICE/CN=tedge-ca" \ + -keyout tedge-local-ca.key \ + -out tedge-local-ca.crt +``` + +This signing certificate has to be trusted by the main device as well as all the child devices: + +```sh title="on the main as well as the child devices" +cp tedge-local-ca.crt /usr/local/share/ca-certificates +sudo update-ca-certificates +``` + +One can then proceed with a certificate for the **tedge-agent** of the main device: + +```sh +## main agent private key +openssl genrsa -out main-agent.key 2048 + +## main agent certificate signing request +openssl req -out main-agent.csr \ + -key main-agent.key \ + -subj "/O=thin-edge/OU=$DEVICE/SN=main-agent/CN=localhost" \ + -new + +## signing the main agent certificate +cat > v3.ext << EOF +authorityKeyIdentifier=keyid +basicConstraints=CA:FALSE +keyUsage = digitalSignature, keyAgreement +extendedKeyUsage = serverAuth, clientAuth +subjectAltName=DNS:$(hostname),DNS:localhost +EOF + +openssl x509 -req \ + -in main-agent.csr \ + -CA tedge-local-ca.crt \ + -CAkey tedge-local-ca.key \ + -extfile v3.ext \ + -CAcreateserial \ + -out main-agent.crt \ + -days 100 +``` + +:::note +Note that the hostname used as `subjectAltName` must be the hostname used by the clients to client to connect **tedge-agent**. +If you want to connect **tedge-agent** using its IP address, say `192.168.1.6`, +then the `subjectAltName` must be set to `subjectAltName=IP:192.168.1.6`. +::: + +And another one for the **tedge-mapper**: + +```sh +## mapper private key +openssl genrsa -out c8y-mapper.key 2048 + +## mapper certificate signing request +openssl req -out c8y-mapper.csr -key c8y-mapper.key \ + -subj "/O=thin-edge/OU=$DEVICE/SN=c8y-mapper/CN=localhost" \ + -new + +## signing the mapper certificate +cat > v3.ext << EOF +authorityKeyIdentifier=keyid +basicConstraints=CA:FALSE +keyUsage = digitalSignature, keyAgreement +extendedKeyUsage = serverAuth, clientAuth +subjectAltName=DNS:$(hostname),DNS:localhost +EOF + +openssl x509 -req \ + -in c8y-mapper.csr \ + -CA tedge-local-ca.crt \ + -CAkey tedge-local-ca.key \ + -extfile v3.ext \ + -CAcreateserial \ + -out c8y-mapper.crt \ + -days 100 +``` + +### File Transfer Service + +Two `tedge config` settings enable HTTPS for the File Transfer Service of the main device. + +- `http.cert_path` The file that will be used as the server certificate for the File Transfer Service. +- `http.key_path` The file that will be used as the server private key for the File Transfer Service. + +Assuming the main device certificate is stored in `/etc/tedge/device-local-certs`, +one simply has to set these settings and to restart the **tedge-agent** to enable HTTPS: + +```sh title="main device" +sudo tedge config set http.cert_path /etc/tedge/device-local-certs/main-agent.crt +sudo tedge config set http.key_path /etc/tedge/device-local-certs/main-agent.key +sudo systemctl restart tedge-agent +``` + +Nothing has to be done on the child devices (except trusting the signing certificate). +And file transfer is now available over HTTPS: + +```sh title="child device" +echo "Hello thin-edge.io" >/tmp/foo.txt +curl -X PUT -F 'file=@/tmp/foo.txt' https://rpi4-dca632efb150:8000/tedge/file-transfer/foo.txt +curl https://rpi4-dca632efb150:8000/tedge/file-transfer/foo.txt +``` + +### Cumulocity Proxy + +Two `tedge config` settings enable HTTPS for the Cumulocity Proxy. + +- `c8y.proxy.cert_path` The file that will be used as the server certificate for the Cumulocity proxy. +- `c8y.proxy.key_path` The file that will be used as the server private key for the Cumulocity proxy. + +Assuming the main device certificate is stored in `/etc/tedge/device-local-certs`, +one simply has to set these settings and to restart the **tedge-mapper** to enable HTTPS: + +```sh title="on the box running c8y mapper" +tedge config set c8y.proxy.cert_path /etc/tedge/device-local-certs/c8y-mapper.crt +tedge config set c8y.proxy.key_path /etc/tedge/device-local-certs/c8y-mapper.key +sudo systemctl restart tedge-mapper-c8y +``` + +Nothing has to be done on the child devices (except trusting the signing certificate). +And the Cumulocity proxy is now available over HTTPS: + +```sh title="child device" +echo "Hello thin-edge.io" >/tmp/foo.txt +curl -X PUT -F 'file=@/tmp/foo.txt' https://rpi4-dca632efb150:8000/tedge/file-transfer/foo.txt +curl https://rpi4-dca632efb150:8000/tedge/file-transfer/foo.txt +``` + +```sh title="main and child device" +curl https://rpi4-dca632efb150:8001/c8y/inventory/managedObjects +``` + +## Enable HTTPS client authentication + +The final step is to enforce certificate-based authentication to the clients connecting +the File Transfer Service and the Cumulocity Proxy. + +### Generating Certificates + +Each child device must be given a certificate. +The simpler is to use the same signing key as for the servers. This is not mandatory though. + +```sh title="on each child device" +## child device private key +openssl genrsa -out tedge-client.key 2048 + +## child device certificate signing request +openssl req -out tedge-client.csr \ + -key tedge-client.key \ + -subj "/O=thin-edge/OU=$DEVICE/SN=child/CN=tedge-client" \ + -new +``` + +The certificate signing request (CSR) has to be signed on the box where the signing key is stored. + +```sh title="on the laptop owning the signing key" +## signing the child device certificate +cat > client-v3.ext << EOF +basicConstraints=CA:FALSE +extendedKeyUsage = clientAuth +EOF + +openssl x509 -req \ + -in tedge-client.csr \ + -CA tedge-local-ca.crt \ + -CAkey tedge-local-ca.key \ + -extfile client-v3.ext \ + -CAcreateserial \ + -out tedge-client.crt \ + -days 100 +``` + +The resulting certificate can then be copied and used on the child device. + +### File Transfer Service + +A single `tedge config` setting enables client authentication (once HTTPS is already enabled). + +- `http.ca_path` Path to a directory containing the PEM encoded CA certificates that are trusted + when checking incoming client certificates for the File Transfer Service. + +Assuming the signing certificate used for the child device has been properly added to `/etc/ssl/certs`, +one simply has to set `http.ca_path` and to restart the **tedge-agent** to enforce client authentication: + +```sh title="main device" +sudo tedge config set http.ca_path /etc/ssl/certs +sudo systemctl restart tedge-agent +``` + +Clients have then to authenticate themselves using a certificate trusted by the main **tedge-agent**: + +```sh title="child device" +echo "Hello thin-edge.io" >/tmp/foo.txt +curl --cert /etc/tedge/device-local-certs/tedge-client.crt \ + --key /etc/tedge/device-local-certs/tedge-client.key \ + -X PUT -F 'file=@/tmp/foo.txt' \ + https://rpi4-dca632efb150:8000/tedge/file-transfer/foo.txt +curl --cert /etc/tedge/device-local-certs/tedge-client.crt \ + --key /etc/tedge/device-local-certs/tedge-client.key \ + https://rpi4-dca632efb150:8000/tedge/file-transfer/foo.txt +``` + +Notably, **tedge-agent** must be updated on each child device, +using the following `tedge config` settings: + +- `http.client.auth.cert_file` Path to the certificate which is used by the agent when connecting to external services +- `http.client.auth.key_file` Path to the private key which is used by the agent when connecting to external services + +```sh title="child device" +sudo tedge config set http.client.auth.cert_file /etc/tedge/device-local-certs/tedge-client.crt +sudo tedge config set http.client.auth.key_file /etc/tedge/device-local-certs/tedge-client.key +sudo systemctl restart tedge-agent +``` + +### Cumulocity Proxy + +A single `tedge config` setting enables client authentication (once HTTPS is already enabled). + +- `c8y.proxy.ca_path` Path to a file containing the PEM encoded CA certificates that are trusted + when checking incoming client certificates for the Cumulocity Proxy. + +Assuming the signing certificate used for the child device has been properly added to `/etc/ssl/certs`, +one simply has to set `c8y.proxy.ca_pathh` and to restart the **tedge-mapper** to enforce client authentication: + +```sh title="on the box running tedge-mapper" +sudo tedge config set c8y.proxy.ca_path /etc/ssl/certs +sudo systemctl restart tedge-mapper-c8y +``` + +Clients have then to authenticate themselves using a certificate trusted by the main **tedge-mapper**: + +```sh title="child device" +curl --cert /etc/tedge/device-local-certs/tedge-client.crt \ + --key /etc/tedge/device-local-certs/tedge-client.key \ + https://rpi4-dca632efb150:8001/c8y/inventory/managedObjects +``` + +Notably, **tedge-agent** must be updated on the main device and all the child devices, +using the following `tedge config` settings: + +- `http.client.auth.cert_file` Path to the certificate which is used by the agent when connecting to external services +- `http.client.auth.key_file` Path to the private key which is used by the agent when connecting to external services + +```sh title="main device as well as child devices" +sudo tedge config set http.client.auth.cert_file /etc/tedge/device-local-certs/tedge-client.crt +sudo tedge config set http.client.auth.key_file /etc/tedge/device-local-certs/tedge-client.key +sudo systemctl restart tedge-agent +``` diff --git a/versioned_docs/version-1.1.0/operate/security/index.md b/versioned_docs/version-1.1.0/operate/security/index.md new file mode 100644 index 0000000..bd97712 --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/security/index.md @@ -0,0 +1,28 @@ +--- +title: Security and Access Control +tags: [Operate, Security] +sidebar_position: 1 +description: Configuring %%te%% for secure communication +--- + +import DocCardList from '@theme/DocCardList'; + +%%te%% uses X.509 certificates as the key mechanism to authenticate peers. +- The MQTT connection between the gateway device and the cloud is established over TLS + and uses certificates to authenticate the device on the cloud, as well as to authenticate the cloud on the device. +- The local MQTT connections, from the miscellaneous services and child devices to the local MQTT broker, + can also be configured to be established over TLS. In the stronger setting, the clients have to authenticate themselves using certificates. +- The local HTTP services (namely the [File Transfer Service](../../references/file-transfer-service.md) and the [Cumulocity Proxy](../../references/cumulocity-proxy.md)) + can be configured to use HTTPS. As for MQTT, certificate-based authentication of the clients can also be enforced. + +A complete setting requires numerous private keys, certificates and trust chains. +Nothing really complex, but this requires rigorous settings. +It is therefore recommended to set things up step by step. +- The only mandatory step is to configure the authentication between the gateway device and the cloud. + - This can be done using a [self-signed device certificate](self-signed-device-certificate.md) or a proper [CA-signed certificate](device-certificate.md). + - Most of the time the cloud certificate will be trusted out-of-the-box, + but a [self-signed cloud certificate](cloud-authentication.md) will need specific care. +- The second step is to enable TLS on the local MQTT and HTTP connections. +- The final step is to enforce certificate-based client authentication on the local MQTT and HTTP connections. + + diff --git a/versioned_docs/version-1.1.0/operate/security/mqtt-tls-configuration.md b/versioned_docs/version-1.1.0/operate/security/mqtt-tls-configuration.md new file mode 100644 index 0000000..207c972 --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/security/mqtt-tls-configuration.md @@ -0,0 +1,232 @@ +--- +title: MQTT TLS Configuration +tags: [Operate, Security, MQTT] +description: Setting up TLS for secure local MQTT communication +--- + +%%te%% supports certificate-based authentication when communicating with +an MQTT broker. Three levels of security are supported: + +1. No authentication (default) +2. Server authentication +3. Server + client authentication + +## MQTT Configuration + +The `tedge config` command provides MQTT specific settings +to open to the child devices the MQTT broker running on the gateway device. + +|Property|Description| +|--------|-----------| +|`mqtt.external.bind.port`|Mqtt broker port, which is used by the external mqtt clients to publish or subscribe. Example: 8883| +|`mqtt.external.bind.address`|IP address / hostname, which the mqtt broker limits incoming connections on. Example: 0.0.0.0| +|`mqtt.external.bind.interface`|Name of network interface, which the mqtt broker limits incoming connections on. Example: wlan0| +|`mqtt.external.ca_path`|Path to a file containing the PEM encoded CA certificates that are trusted when checking incoming client certificates. Example: /etc/ssl/certs| +|`mqtt.external.cert_file`|Path to the certificate file, which is used by external MQTT listener. Example: /etc/tedge/server-certs/tedge-certificate.pem| +|`mqtt.external.key_file`|Path to the private key file, which is used by external MQTT listener. Example: /etc/tedge/server-certs/tedge-private-key.pem| + +:::note +If none of these options is set, then no external listener is set. +If one of these options is set, then default values are inferred by the MQTT server (Mosquitto). +For instance, the port defaults to 1883 for a non-TLS listener, and to 8883 for a TLS listener. +::: + +These settings can be considered in 2 groups, listener configuration and TLS configuration. + +### Configure basic listener + +To configure basic listener you should provide port and/or bind address which will use default interface. +To change the default interface you can use mqtt.external.bind.interface configuration option. + +To set them you can use `tedge config` as follows: + +```sh +sudo tedge config set mqtt.external.bind.port 8883 +``` + +To allow connections from all IP addresses on the interface: + +```sh +sudo tedge config set mqtt.external.bind.address "0.0.0.0" +``` + +### Configure TLS on the listener + +To configure the external listener with TLS additional settings are available: + +* `mqtt.external.ca_path` +* `mqtt.external.cert_file` +* `mqtt.external.key_file` + +To enable MQTT over TLS, a server side certificate must be configured using the 2 following settings: + +```sh +sudo tedge config set mqtt.external.cert_file /etc/tedge/server-certs/tedge-certificate.pem +sudo tedge config set mqtt.external.key_file /etc/tedge/server-certs/tedge-private-key.pem +``` + +To fully enable TLS authentication clients, client side certificate validation can be enabled: + +```sh +sudo tedge config set mqtt.external.ca_path /etc/ssl/certs +``` + +## Server authentication + +Enabling server authentication causes %%te%% MQTT clients to require a +valid certificate from a broker when connecting. The broker certificate is valid +when it is signed by a CA that the clients trust. + +To enable server authentication, perform the following: + +### Step 1: Configure server authenticated listener in Mosquitto broker + +Create a file in `/etc/mosquitto/conf.d/` with the following content and restart +mosquitto service: + +```sh +listener 8883 +certfile PATH_TO_SERVER_CERTIFICATE +keyfile PATH_TO_SERVER_PRIVATE_KEY +``` + +- `listener 8883`: defines a new listener on port 8883 +- `certfile`: points to a certificate that will be used by the broker to + authenticate itself to connecting clients +- `keyfile`: points to a private key of the specified certificate, necessary for + encrypted communication + +Be sure that mosquitto can read both files, especially the private key, which +can only be read by the owner. Set `mosquitto:mosquitto` as an owner of these +files. If you're unsure where to place them, `/etc/mosquitto/ca_certificates` +is the directory intended for them, although you can use other paths if +necessary. + +The certificates used need to be X.509 v3 certificates with a `subjectAltName` +section containing the hostname that the broker is running on. + +### Step 2: Configure thin-edge.io to connect to the new listener + +Execute the following commands: + +```sh +sudo tedge config set mqtt.client.port 8883 +sudo tedge config set mqtt.client.cafile PATH_TO_CA_CERTIFICATE + +# optional +sudo tedge config set mqtt.client.cadir PATH_TO_CA_CERTIFICATE_DIRECTORY +``` + +`mqtt.client.cafile` and `mqtt.client.cadir` options point to trusted CA +certificate(s) used to verify the broker. If either is used, server +authentication is enabled. + +### Step 3: Restart services + +Now you will need to manually restart all the affected services so that they can +pick up the configuration change. + + + +## Server + client authentication + +Additionally, the server can require connecting clients to present a valid +certificate. These client certificates need to be signed by a CA that the server +trusts. CA used to sign a server certificate and CA signing client certificates +do not have to be the same. + +### Step 1: Configure server + client authenticated listener in Mosquitto broker + +Change the content of the conf file defined previously to the following and +restart mosquitto service: + +```conf +listener 8883 +allow_anonymous false +require_certificate true +cafile PATH_TO_SERVER_CERTIFICATE +certfile PATH_TO_SERVER_PRIVATE_KEY +keyfile PATH_TO_CLIENT_CA_CERTIFICATE +``` + +- `allow_anonymous` disables anonymous access to the listener; connecting + clients will need to authenticate themselves +- `require_certificate` requires clients to provide a certificate as a means of + authentication + +### Step 2: Configure thin-edge.io to use a client certificate and private key + +```sh +sudo tedge config set mqtt.client.auth.cert_file PATH_TO_CLIENT_CERTIFICATE +sudo tedge config set mqtt.client.auth.key_file PATH_TO_CLIENT_PRIVATE_KEY +``` + +Both `certfile` and `keyfile` are required to enable client authentication. +Setting only one of them will result in an error about the second one not being +set. + +As with the server private key, set `tedge:tedge` as the owner of the +certificate and the private key, so that the private key can be read by +%%te%% components. + +### Step 3: Restart services + +Now you will need to manually restart all the affected services so that they can +pick up the configuration change. + +## Generating certificates + +You can use the following script to generate all required certificates: + +```sh +openssl req \ + -new \ + -x509 \ + -days 365 \ + -extensions v3_ca \ + -nodes \ + -subj "/C=US/ST=Denial/L=Springfield/O=Dis/CN=ca" \ + -keyout ca.key \ + -out ca.crt + +openssl genrsa -out server.key 2048 + +openssl req -out server.csr -key server.key -new \ + -subj "/C=US/ST=Denial/L=Springfield/O=Dis/CN=$(hostname)" + +cat > v3.ext << EOF +authorityKeyIdentifier=keyid +basicConstraints=CA:FALSE +keyUsage = digitalSignature, keyAgreement +subjectAltName=DNS:$(hostname), DNS:localhost +EOF + +openssl x509 -req \ + -in server.csr \ + -CA ca.crt \ + -CAkey ca.key \ + -extfile v3.ext \ + -CAcreateserial \ + -out server.crt \ + -days 365 + +openssl genrsa -out client.key 2048 + +openssl req -out client.csr \ + -key client.key \ + -subj "/C=US/ST=Denial/L=Springfield/O=Dis/CN=client1" \ + -new + +openssl x509 -req \ + -in client.csr \ + -CA ca.crt \ + -CAkey ca.key \ + -CAcreateserial \ + -out client.crt \ + -days 365 +``` + +## Next steps + +- For more options to customize behaviour of mosquitto broker, see + [mosquitto.conf man page](https://mosquitto.org/man/mosquitto-conf-5.html) diff --git a/versioned_docs/version-1.1.0/operate/security/self-signed-device-certificate.md b/versioned_docs/version-1.1.0/operate/security/self-signed-device-certificate.md new file mode 100644 index 0000000..a1e8e9e --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/security/self-signed-device-certificate.md @@ -0,0 +1,127 @@ +--- +title: Self-signed Device Certificate +tags: [Operate, Security, Cloud] +description: Using self-signed device certificates with %%te%% +--- + +Using a self-signed device certificate is the simplest way to connect a %%te%% device to the cloud. +This is a secure method even if more adapted for testing purposes. +Indeed, the self-signed certificates must be trusted individually by the cloud tenant, +raising managing issues when there are more than a few devices. + +## Create self-signed certificate + +To create a new certificate you can use [`tedge cert create`](../../references/cli/tedge-cert.md) %%te%% command: + +```sh +sudo tedge cert create --device-id alpha +``` + +```text title="Output" +Certificate was successfully created +``` + +:::note +`tedge cert` requires `sudo` privilege. This command provides no output on success. +::: + +[`sudo tedge cert create`](../../references/cli/tedge-cert.md) creates the certificate in a default location (`/etc/tedge/device-certs/`). +To use a custom location, refer to [`tedge config`](../../references/cli/tedge-config.md). + +Now you should have a certificate in the `/etc/tedge/device-certs/` directory. + +```sh +ls -l /etc/tedge/device-certs/ +``` + +```text title="Output" +total 8 +-r--r--r-- 1 mosquitto mosquitto 664 May 31 09:26 tedge-certificate.pem +-r-------- 1 mosquitto mosquitto 246 May 31 09:26 tedge-private-key.pem +``` + +## Make the cloud trust the device self-signed certificate + +For the cloud to trust the device certificate, +the signing certificate must be added to the trusted list of signing certificate of the cloud tenant. + +The certificate created with `tedge cert create` being self-signed, one needs to add the device certificate itself to the trusted list. + +How this is done depends on the cloud. In the specific case of Cumulocity, this can be done using the `tedge` cli. + +One has first to set the Cumulocity end-point: + +```sh +tedge config set c8y.url +``` + +And then upload the signing certificate: + +```sh +tedge cert upload c8y --user +``` + +## Renew self-signed certificate + +To renew the expired certificate you can use [`tedge cert renew`](../../references/cli/tedge-cert.md) %%te%% command: + +```sh +sudo tedge cert renew +``` + +```text title="Output" +Certificate was successfully renewed, for un-interrupted service, the certificate has to be uploaded to the cloud +``` + +:::note +`tedge cert renew` will get the device-id from the existing expired certificate and then renews it. +::: + +## Errors + +### Certificate creation fails due to invalid device id + +If non-supported characters are used for the device id then the cert create will fail with below error: + +```text +Error: failed to create a test certificate for the device +. + +Caused by: + 0: DeviceID Error + 1: The string '"+"' contains characters which cannot be used in a name [use only A-Z, a-z, 0-9, ' = ( ) , - . ? % * _ ! @] +``` + + +### Certificate already exists in the given location + +If the certificate already exists you may see following error: + +```text +Error: failed to create a test certificate for the device alpha. + +Caused by: + A certificate already exists and would be overwritten. + Existing file: "/etc/tedge/device-certs/tedge-certificate.pem" + Run `tedge cert remove` first to generate a new certificate. +``` + +:::note +Removing a certificate can break the bridge and more seriously delete a certificate that was a CA-signed certificate. +::: + +Follow the instruction to remove the existing certificate and issue [`tedge cert remove`](../../references/cli/tedge-cert.md): + +```sh +sudo tedge cert remove +``` + +```text title="Output" +Certificate was successfully removed +``` + +Afterwards, try executing [`tedge cert create`](../../references/cli/tedge-cert.md) again. + +## Next steps + +1. [How to connect?](../c8y/connect.md) +2. [How to use mqtt pub/sub?](../telemetry/mqtt-tools.md) diff --git a/versioned_docs/version-1.1.0/operate/telemetry/data-model.md b/versioned_docs/version-1.1.0/operate/telemetry/data-model.md new file mode 100644 index 0000000..e8cba31 --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/telemetry/data-model.md @@ -0,0 +1,189 @@ +--- +title: Data Model +tags: [Concept, MQTT] +description: Overview of the %%te%% data model and how to interact with it +--- + +The **data model** identifies all data send or received from/to %%te%% and its components, to interact with those. +For all data it defines format and explains behaviour. + +## Use of MQTT + +%%te%% expects the MQTT broker [mosquitto](https://mosquitto.org/) to be available on the device. +%%te%% uses **mosquitto** to consume and provide telemetry data. All telemetry data are reflected with specific MQTT topics and payload in JSON format. + +%%te%% assumes **mosquitto** is configured in a secure manner, to avoid any inappropriate access to %%te%% topics and payload. +Any malicious access to the broker can hazard %%te%% and all connected devices. Mosquitto provides a wide range of authentication and access control options. For more details see _Authentication_ and _ACL_ (Access Control List) in the [mosquitto documentation](https://mosquitto.org/man/mosquitto-conf-5.html). + +### Telemetry Data on MQTT + +All telemetry data (**Measurements**, **Events**, **Alarms**) are reflected with MQTT topics, where each has its specific subtopic (e.g. `te/+/+/+/+/m/+`, `te/+/+/+/+/e/+`, `te/+/+/+/+/a/+` etc.). + + * each provider of a **measurement**, **event** or **alarm** sends the occurring data to %%te%%'s MQTT broker + * a provider can be the domain application, other SW components / 3rd parties + * all processes (e.g. the domain application, other SW components / 3rd parties) on the main-device and all child-devices can consume those telemetry data from the MQTT broker + * the cloud mapper on the **main-device** picks-up _all_ telemetry data from the MQTT broker and transfers those to the cloud + +The communication diagram below illustrates that behaviour. + +![MQTT communication flow](../../understand/images/MQTT-communication.svg) + +### Telemetry Data for Child-Devices + +All telemetry data provided to the MQTT bus are associated by %%te%% and all consumers with the %%te%% **main-device** or some **child-device**. + +MQTT topics for the **main-device**: + +``` +te/device/main///m/ +te/device/main///e/ +te/device/main///a/ +``` + +MQTT topics for a **child-device**, including the **child-device's** specific `child-id`: + +``` +te/device////m/ +te/device////e/ +te/device////a/ +``` + + +## Telemetry Data + +**Telemetry Data** consists of **measurements**, **events** and **alarms**. Each is defined by a set of data-elements, each with specific behaviour. + +### Measurements +**Measurements** carry values from physical **Sensors** or a device's **Domain Application**; +e.g. voltage and current of an electricity meter, or current state of the manufacturing control process + +#### MQTT topics for measurements + +``` +te/device/main///m/ + +te/device////m/ +``` + +#### MQTT payload for measurements + +A measurement can carry a **single value**, or **multiple values** all taken at a single point in time. + +One MQTT message can contain a mixture of more than one single-value and multi-value measurements. + +##### Example for a single-value measurement payload + +```javascript +{ + "temperature": // 'name' of that measurement + 25.3, // 'value' of that measurement + "time": "2020-10-15T05:30:47+00:00", // optional 'timestamp' of that measurement +} +``` +##### Example for a multi-value measurement payload +```javascript +{ + "current": { // 'name' of that measurement + "L1": 9.5, // the 1st 'value' of that measurement, named as "L1" + "L2": 1.3 // the 2nd 'value' of that measurement, named as "L2" + // ...even more values can occur + }, + "time": "2020-10-15T05:30:47+00:00", // optional 'timestamp' of that measurement +} +``` + + +|Reference |Description| +| --------- | --------- | +|`name` |a string that identifies the measurement uniquely in context of the device| +|`value` |the value that was sampled; can be named (especially in context of a multi-value measurement) or unnamed; must be an integer or floating point number| +|`timestamp` |optional time that indicates when values were sampled; when not provided, %%te%% uses the current system time as the time of the sample; when provided must be conform to ISO 8601| + +#### Behaviour of measurements +- %%te%% does not store any historical sampled values for measurements +- there is no initialization value for measurements; i.e. a measurement is not visible on %%te%% before the 1st sample was sent to %%te%% +- a measurement should never be published as MQTT retain message; + That is as a single retained measurement might be consumed and processed more than once by a consuming software + component (e.g. when that software component restarts and subscribes again). + +### Events +**Events** are notifications that something happened on a device's environment or software system; +e.g. a sensor detected something like a door has been closed, or a system notification that e.g. a user has started an ssh session + +#### MQTT topics for events + +``` +te/device/main///e/ + +te/device////e/ +``` + +#### MQTT payload for events +```javascript +{ + // example of an event + "text": "A user just logged in", // 'text' message of that event + "time": "2021-01-01T05:30:45+00:00", // optional 'timestamp' of that event + "someOtherCustomFragment": { // optional 'custom fragments' + "nested": { + "value": "extra info" + } + } +} +``` + +|Reference |Description| +| ------------------ | --------- | +|`event-type` |a string part of the MQTT topic, that identifies the event uniquely in context of the device| +|`text` |carries a human readable event-text; must be UTF-8 encoded| +|`timestamp` |optional time that indicates when the event has occurred; when not provided, %%te%% uses the current system time as the time of the event; when provided must be conform to ISO 8601| +|`custom fragments` |additional fields are handled as custom specific information; if the connected cloud supports custom fragments its mapper transfers those accordingly to the cloud| + +#### Behaviour of events +- %%te%% does not store any historical occurrences for events +- an event should never be published as MQTT retain message; + That is as a single retained event might be consumed and processed more than once by a consuming software + component (e.g. when that software component restarts and subscribes again). + +### Alarms +**Alarms** are notifications about some critical behaviour of the device's environment or software system; +e.g. when a temperature sensor detects a temperature went out of its valid range + +#### MQTT topics for alarms + +``` +te/device/main///a/ + +te/device////a/ +``` + +#### MQTT payload for alarms + +```javascript +{ + // example for an alarm + "text": "Temperature is very high", // 'text' message of that alarm + "time": "2021-01-01T05:30:45+00:00", // optional 'timestamp' of that alarm + "severity": "major", // optional 'severity' of the alarm + "someOtherCustomFragment": { // optional 'custom fragments' + "nested": { + "value": "extra info" + } + } +} +``` + +|Reference |Description| +| ------------------ | --------- | +|`alarm-type` |a string part of the MQTT topic, that identifies the alarm uniquely in context of the device| +|`severity` |a string part of the MQTT payload, that indicates the severity of the alarm; recommended to be `critical`, `major`, `minor` or `warning`| +|`text` |carries a human readable alarm-text; must be UTF-8 encoded| +|`timestamp` |optional time that indicates when the alarm has occurred; when not provided, %%te%% uses the current system time as the time of the alarm; when provided must be conform to ISO 8601| +|`custom fragments` |additional fields are handled as custom specific information; if the connected cloud supports custom fragments its mapper transfers those accordingly to the cloud| + +#### Behaviour of alarms +- %%te%% does not store any historical occurrences for alarms +- **alarms** are stateful; i.e. once raised, an **alarm** is active until it was explicitly cleared by the device's software or the cloud +- all alarms shall be published as MQTT retain message to reflect the alarm's stateful behaviour in the broker; The retain messages is kept in + the MQTT broker as long as the alarm is raised. When a raised alarm is gone again, an empty retain message shall be published to clear + the alarm message in the broker. diff --git a/versioned_docs/version-1.1.0/operate/telemetry/index.md b/versioned_docs/version-1.1.0/operate/telemetry/index.md new file mode 100644 index 0000000..3c6abbb --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/telemetry/index.md @@ -0,0 +1,11 @@ +--- +title: Processing Telemetry Data +tags: [Operate, Telemetry, MQTT] +sidebar_position: 1 +--- + +import DocCardList from '@theme/DocCardList'; + +Processing Telemetry Data + + diff --git a/versioned_docs/version-1.1.0/operate/telemetry/mqtt-tools.md b/versioned_docs/version-1.1.0/operate/telemetry/mqtt-tools.md new file mode 100644 index 0000000..ebfb0d5 --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/telemetry/mqtt-tools.md @@ -0,0 +1,67 @@ +--- +title: MQTT Tools +tags: [Operate, MQTT] +description: Publish and subscribe to MQTT messages on the command line +--- + +%%te%% cli provides a convenient way to debug and aid development process. + +## Publish + +Command [`tedge mqtt pub`](../../references/cli/tedge-mqtt.md) can be used to publish MQTT messages on a topic to the local mosquitto server. + +Example: + +```sh te2mqtt formats=v1 +tedge mqtt pub 'te/device/main///m/env_sensor' '{"temperature": 21.3}' +``` + +Messages can also be published with a different Quality of Service (QoS). + +```sh te2mqtt formats=v1 +tedge mqtt pub 'te/device/main///m/env_sensor' '{"temperature": 21.3}' --qos 2 +``` + +MQTT messages can also be published using the retained option which means that the message will be received by new MQTT clients connecting to the broker after the message was published. + +Below shows an example of publishing a retained MQTT message: + +```sh te2mqtt formats=v1 +tedge mqtt pub --retain --qos 1 te/device/main///a/high_temperature '{ + "text": "Temperature is critical", + "severity": "critical" +}' +``` + +:::note +By default the mqtt message will be published with retain flag set to false. +::: + + +## Subscribe + +Command [`tedge mqtt sub`](../../references/cli/tedge-mqtt.md) can be used to ease debugging of of MQTT communication on local bridge. You can subscribe to topic of your choosing: + +```sh te2mqtt formats=v1 +tedge mqtt sub te/errors +``` + +Or you can subscribe to any topic on the server using wildcard (`#`) topic: + +```sh te2mqtt formats=v1 +tedge mqtt sub '#' +``` + +Now using a different console/shell, publish the following measurement so that the previous subscription will receive it: + +```sh te2mqtt formats=v1 +tedge mqtt pub --retain --qos 1 te/device/main///m/env_sensor '{"temperature": 21.3}' +``` + +All messages from sub command are printed to `stdout` and can be captured to a file if you need to: + +```sh te2mqtt formats=v1 +tedge mqtt sub '#' > filename.mqtt +``` + +Wildcard (`#`) topic is used by [MQTT protocol](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901242) as a wildcard and will listen on all topics diff --git a/versioned_docs/version-1.1.0/operate/troubleshooting/device-monitoring.md b/versioned_docs/version-1.1.0/operate/troubleshooting/device-monitoring.md new file mode 100644 index 0000000..db2a345 --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/troubleshooting/device-monitoring.md @@ -0,0 +1,67 @@ +--- +title: Device Monitoring +tags: [Operate, Monitoring] +sidebar_position: 1 +description: How to troubleshoot device monitoring +--- + +To install and configure monitoring on your device, +see the tutorial [Monitor your device with collectd](../../start/device-monitoring.md). + +## Is collectd running? + +```sh +sudo systemctl status collectd +``` + +If not, launch collected + +```sh +sudo systemctl start collectd +``` + +## Is collectd publishing MQTT messages? + +```sh te2mqtt formats=v1 +tedge mqtt sub 'collectd/#' +``` + +If no metrics are collected, please check the [MQTT configuration](../../start/device-monitoring.md#collectd-configuration) + +:::note +The `collectd.conf` file included with %%te%% is configured for conservative interval times, e.g. 10 mins to 1 hour depending on the metric. This is done so that the metrics don't consume unnecessary IoT resources both on the device and in the cloud. If you want to push the metrics more frequently then you will have to adjust the `Interval` settings either globally or on the individual plugins. Make sure you restart the collectd service after making any changes to the configuration. +::: + +## Is the tedge-mapper-collectd running? + +```sh +sudo systemctl status tedge-mapper-collectd +``` + +If not, launch tedge-mapper-collectd.service as below + +```sh +sudo systemctl start tedge-mapper-collectd +``` + +## Are the collectd metrics published in Thin Edge JSON format? + +```sh te2mqtt formats=v1 +tedge mqtt sub 'te/device/main///m/+' +``` + +## Are the collectd metrics published to Cumulocity IoT? + +```sh te2mqtt formats=v1 +tedge mqtt sub 'c8y/#' +``` + +If not see how to [connect a device to Cumulocity IoT](../../start/connect-c8y.md). + +## Are the collectd metrics published to Azure IoT? + +```sh te2mqtt formats=v1 +tedge mqtt sub 'az/#' +``` + +If not see how to [connect a device to Azure IoT](../../start/connect-azure.md). diff --git a/versioned_docs/version-1.1.0/operate/troubleshooting/index.md b/versioned_docs/version-1.1.0/operate/troubleshooting/index.md new file mode 100644 index 0000000..ad44421 --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/troubleshooting/index.md @@ -0,0 +1,12 @@ +--- +title: Troubleshooting +tags: [Operate] +sidebar_position: 8 +description: Troubleshooting %%te%% on a device +--- + +import DocCardList from '@theme/DocCardList'; + +How to troubleshoot a %%te%% device + + diff --git a/versioned_docs/version-1.1.0/operate/troubleshooting/log-files.md b/versioned_docs/version-1.1.0/operate/troubleshooting/log-files.md new file mode 100644 index 0000000..c19e6a0 --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/troubleshooting/log-files.md @@ -0,0 +1,147 @@ +--- +title: Log Files +tags: [Operate, Log Files] +sidebar_position: 1 +description: How to get log files from %%te%% components +--- + +The logs that are useful for debugging %%te%% break down into logs that are created by %%te%% itself and by third party components. + +## %%te%% logs {#thin-edge-logs} +On a %%te%% device different components like mappers, agent, and plugins run. The log messages of these components can be accessed as below. +The logs here capture INFO, WARNING, and ERROR messages. + +### Cloud mapper logs {#mapper} +The %%te%% cloud mapper component that sends the measurement data to the cloud can be accessed as below. + +#### Cumulocity mapper {#c8y-mapper} +The log messages of the Cumulocity mapper component that sends the measurement data from the %%te%% device to the Cumulocity +cloud can be accessed as below + +```sh +journalctl -u tedge-mapper-c8y +``` + +:::note +Run `tedge-mapper --debug c8y` to log more debug messages +::: + +#### Azure mapper {#az-mapper} +The log messages of the Azure mapper component that sends the measurement data from the %%te%% device to the Azure +cloud can be accessed as below. + +```sh +journalctl -u tedge-mapper-az +``` + +:::note +Run `tedge-mapper --debug az` to log more debug messages +::: + +#### AWS mapper {#aws-mapper} +The log messages of the AWS mapper component that sends the measurement data from the %%te%% device to the AWS +cloud can be accessed as below. + +```sh +journalctl -u tedge-mapper-aws +``` + +:::note +Run `tedge_mapper --debug aws` to log more debug messages +::: + +### Device monitoring logs {#device-logs} +The %%te%% device monitoring component logs can be found as below + +#### Collectd mapper logs {#collectd-mapper} +The log messages of the collectd mapper that sends the monitoring data to the cloud can be accessed as below + +```sh +journalctl -u tedge-mapper-collectd +``` + +:::note +Run `tedge-mapper --debug collectd` to log more debug messages +::: + +### Software Management logs {#software-management} +This section describes how to access the software management component logs + +#### Software update operation log {#software-update} +For every new software operation (list/update), a new log file will be created at `/var/log/tedge/agent`. +For each `plugin command` like prepare, update-list (install, remove), finalize, and list, +the log file captures `exit status, stdout, and stderr` messages. + +#### tedge-agent logs {#tedge-agent-logs} +The agent service logs can be accessed as below + +```sh +journalctl -u tedge-agent +``` + +For example: tedge-agent logs plugin calls finalize and list. + +```log title="Logs" +tedge-agent : TTY=unknown ; PWD=/tmp ; USER=root ; COMMAND=/etc/tedge/sm-plugins/apt finalize +tedge-agent : TTY=unknown ; PWD=/tmp ; USER=root ; COMMAND=/etc/tedge/sm-plugins/apt list +``` + +:::note +Run `tedge-agent --debug` to log more debug messages +::: + +## Thirdparty component logs {#thirdparty} +%%te%% uses the third-party components `Mosquitto` as the mqtt broker and `Collectd` for monitoring purpose. +The logs that are created by these components can be accessed on a %%te%% device as below. + +### Mosquitto logs {#mosquitto} +%%te%% uses `Mosquitto` as the `mqtt broker` for local communication as well as to communicate with the cloud. +The `Mosquitto` logs can be found in `/var/log/mosquitto/mosquitto.log`. +`Mosquitto` captures error, warning, notice, information, subscribe, and unsubscribe messages. + +:::note +Set `log_type debug` or `log_type all` on `/etc/mosquitto/mosquitto.conf`, to capture more debug information. +::: + +### Collectd logs {#collectd} +`Collectd` is used for monitoring the resource status of a %%te%% device. +Collectd logs all the messages at `/var/log/syslog`. +So, the collectd specific logs can be accessed using the `journalctl` as below + +```sh +journalctl -u collectd +``` + +## Configuring log levels in %%te%% {#configure-log-levels} + +The log levels can be configured for %%te%% services using either by command line or setting the required log +level in `system.toml` + +### Setting the log level through cli {#configure-log-levels-cli} + +The log level can be enabled for a %%te%% service as below + +For example for tedge-mapper: + +```sh +sudo -u tedge -- tedge-mapper --debug c8y +``` + +:::note +In a similar way it can be set for all the %%te%% services. +Only `debug` level can be set through cli. Also, it enables `trace` level. +::: + +### Setting log level through system.toml {#configure-log-levels-file} +The log levels can also be configured through the `system.toml` file. +The supported log levels are `info, warn, error, trace, debug`. + +```toml title="file: /etc/tedge/system.toml" +[log] +tedge-mapper = "trace" +tedge-agent = "info" +``` + +:::note +The log level strings are case insensitive +::: diff --git a/versioned_docs/version-1.1.0/operate/troubleshooting/monitoring-service-health.md b/versioned_docs/version-1.1.0/operate/troubleshooting/monitoring-service-health.md new file mode 100644 index 0000000..cee97d4 --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/troubleshooting/monitoring-service-health.md @@ -0,0 +1,85 @@ +--- +title: Monitoring Service Health +tags: [Operate, Monitoring] +sidebar_position: 1 +description: How to monitor health of tedge services +--- + +The health of %%te%% services like `tedge-mapper`, `tedge-agent` etc can be monitored via MQTT. +These services expose MQTT health endpoints which you can query to check if the process is still active or not. + +To get the last known health status of a service you can subscribe to the following topic + +```text +te//status/health +``` + +To refresh the health status of the service, publish an empty message on the topic below. + +```text +te//cmd/health/check +``` + +:::note +If the response is not received then most likely the service is down, or not responding +::: + + +For example, `tedge-mapper-c8y` publishes a message on topic `te/device/main/service/tedge-mapper-c8y/status/health` when it starts: + +```json +{ "pid": 290854, "status": "up", "time": 1714676361.3610663 } +``` + + + +| Property | Description | +|----------|------------------------------------------------------------------------------------------------------------------| +| `pid` | Process ID of the service | +| `status` | Service status. Possible values are `up` or `down` | +| `time` | Timestamp in either Unix or RFC-3339 format. Configurable by the tedge config setting `service.timestamp_format` | + +If the tedge service gets stopped, crashed, or killed, then a `down` message will be published on health status topic +and this will be retained until the service is restarted. + +E.g. the mapper being killed: + +```sh te2mqtt formats=v1 +tedge mqtt sub 'te/+/+/+/+/status/health' +``` + +```log title="Output" +INFO: Connected +[te/device/main/service/mosquitto-c8y-bridge/status/health] 1 +[te/device/main/service/tedge-mapper-c8y/status/health] {"pid":51367,"status":"down"} +[te/device/main/service/tedge-agent/status/health] {"pid":13280,"status":"up","time":1714676361.3610663} +``` +## Supported MQTT health endpoint topics + +The following endpoints are currently supported: + +* `te/device/main/service/tedge-agent/status/health` +* `te/device/main/service/tedge-mapper-c8y/status/health` +* `te/device/main/service/tedge-mapper-az/status/health` +* `te/device/main/service/tedge-mapper-aws/status/health` +* `te/device/main/service/tedge-mapper-collectd/status/health` + +All future tedge services will also follow the same topic naming scheme convention. + +## Mosquitto bridge health endpoints + +The mosquitto bridge clients connecting %%te%% devices to the respective cloud platforms also report their health +status as retained messages to `te/device/main/service//status/health` topics. The health check +messages published by these clients are just numeric values `1` or `0`, indicating active and dead bridge clients +respectively. + +Here are the health endpoints of currently supported clouds, bridged with mosquitto: + +| Cloud | Health topic | +|------------|-------------------------------------------------------------| +| Cumulocity | `te/device/main/service/mosquitto-c8y-bridge/status/health` | +| Azure | `te/device/main/service/mosquitto-az-bridge/status/health` | +| AWS | `te/device/main/service/mosquitto-aws-bridge/status/health` | + +Explicit health check requests via `te//cmd/health/check` topics is not supported by these bridge clients. +Since the health status messages are sent as retained messages, just subscribing to these health topics is sufficient to get the latest status. diff --git a/versioned_docs/version-1.1.0/operate/troubleshooting/testing-cloud-connection.md b/versioned_docs/version-1.1.0/operate/troubleshooting/testing-cloud-connection.md new file mode 100644 index 0000000..e7a78be --- /dev/null +++ b/versioned_docs/version-1.1.0/operate/troubleshooting/testing-cloud-connection.md @@ -0,0 +1,49 @@ +--- +title: Testing Cloud Connection +tags: [Operate, Cloud, Connection] +sidebar_position: 1 +description: How to test the connection to the cloud +--- + +%%te%% provides a way to test the connection from your device to a cloud provider. +You can call this connection check function by + +```sh +sudo tedge connect --test +``` + +It returns exit code 0 if the connection check is successful, otherwise, 1. + +This test is already performed as part of the `tedge connect ` command. + +## What does the test do? + +The connection test sends a message to the cloud and waits for a response. +The subsequent sections explain the cloud-specific behaviour. + +### For Cumulocity IoT + +The test publishes [a SmartREST 2.0 static template message for device creation `100`](https://cumulocity.com/guides/device-sdk/mqtt/#a-nameinventory-templatesinventory-templates-1xxa) to the topic `c8y/s/us`. +If the device-twin is already created in your Cumulocity, +the device is supposed to receive `41,100,Device already existing` on the error topic `c8y/s/e`. + +So, the test subscribes to `c8y/s/e` topic and if it receives the expected message on the topic, the test is marked successful. + +The connection test sends maximum two of SmartREST2.0 `100` requests. +This is because the first `100` request can be considered a successful device creation request if the device-twin does not exist in Cumulocity yet. + +### For Azure IoT Hub + +The test subscribes to the topic `az/twin/res/`. +Then, it publishes an empty string to the topic `az/twin/GET/?$rid=1`. + +If the connection check receives a message containing `200` (status success), the test is marked successful. + +The connection test sends the empty string only once. + +### For AWS IoT + +The test subscribes to the topic `aws/connection-success`. +It publishes an empty string to the topic `aws/test-connection`. + +If the connection check receives an empty message on the `aws/connection-success` topic, then the test is marked successful. diff --git a/versioned_docs/version-1.1.0/overview.md b/versioned_docs/version-1.1.0/overview.md new file mode 100644 index 0000000..dc0f801 --- /dev/null +++ b/versioned_docs/version-1.1.0/overview.md @@ -0,0 +1,70 @@ +--- +title: Overview +slug: / +sidebar_position: 0 +--- + +Welcome to %%te%%'s documentation! + +%%te%% is an open-source development toolbox designed for rapid development of IoT agents. +It is based on a versatile set of ready-to-use software components +that can be easily combined with application-specific extensions +into smart, secure, robust and efficient IoT agents +which integrate cloud services, edge computing and operational technologies. + +A typical agent uses as a foundation the building blocks provided by %%te%% +for telemetry data processing and cloud connectivity as well as for device monitoring, configuration and updates. +In combination with these blocks, the agent designer can provide application-specific extensions, +which cooperate with %%te%% over MQTT and HTTP along a JSON API, +to address any hardware, protocol or use-case specificity. + +As such, %%te%% is good choice for implementing smart-equipment +that collect in-situ real-time data, perform analytics on the edge, forward key data to the cloud, +and need to be secured, configured and updated at scale. + +## How to start {#start} + +The easiest way to get started is either to install the docker based [demo container](https://github.com/thin-edge/tedge-demo-container) +that showcases %%te%% and all its features or with the [beginner-friendly tutorial](start/getting-started.md) +that introduces %%te%% and guides you on how to install it on a Raspberry Pi. +After the installation you can directly connect your device to [Cumulocity IoT](https://www.cumulocity.com/guides/concepts/introduction/), +and then monitor it from the cloud. + +You can also explore the main use-cases using these [tutorials](start/index.md). +You will learn to: + +- [install %%te%% on your specific hardware](install/index.md), +- connect your device to your cloud, whether [Cumulocity IoT](start/connect-c8y.md), + [Azure IoT](start/connect-azure.md) or [AWS IoT](start/connect-aws.md), +- [send telemetry data](start//send-measurements.md), [alarms](start//raise-alarm.md) and [events](start//send-events.md), +- operate, configure, update, monitor your device. + + +## The concepts {#concepts} + +Better understand how %%te%% works by reviewing the core [Concepts](understand/index.md). + +## How to operate a device {#operate} + +%%te%% provides a set of building blocks to operate, configure, update, monitor your devices. + +* Use the [how-to guides](operate/index.md) on a daily basis +* Refer to the [reference guides](references/index.md) for any in-depth details + +## How to extend {#extend} + +One of the core feature of %%te%% is to be extensible. + +- [Write a software-management plugin](extend/software-management.md) +- [Build Operating System images with %%te%% setup to perform Over-the-Air (OTA) updates](extend/firmware-management/index.md) + +## How to contribute {#contribute} + +[%%te%%](https://github.com/thin-edge/thin-edge.io) is an open-source project +released under the [Apache License - Version 2.0](https://github.com/thin-edge/thin-edge.io/blob/main/LICENSE.txt). + +All contributions are greatly appreciated. +It can be by reporting issues, improving this documentation, adding new extensions or contributing to the main code base. + +Please refer to the [contribution guide](https://github.com/thin-edge/thin-edge.io/blob/main/CONTRIBUTING.md) +and the [contributor documentation](contribute/index.md). diff --git a/versioned_docs/version-1.1.0/references/agent/device-management-api.md b/versioned_docs/version-1.1.0/references/agent/device-management-api.md new file mode 100644 index 0000000..6e5e88b --- /dev/null +++ b/versioned_docs/version-1.1.0/references/agent/device-management-api.md @@ -0,0 +1,403 @@ +--- +title: Device Management API +tags: [Reference, Agent, API] +sidebar_position: 2 +description: Details of the Device Management API +--- + +# Device Management API + +The agent and the operation plugins enable device management on %%te%%, +giving the cloud operators a uniform interface to install software packages, +update configuration files, inspect log files, restart devices ... + +Each of these device management operation requires a different set of interactions between the participants, +with specific steps, message payloads and expectations. + +Hence, each operation type comes with its own API, and, possibly, with different implementations: +the `tedge-agent` service or user-specific plugins. + +However, despite their diversity, all these APIs are designed along the same lines with normalized ways to +- associate MQTT topics with devices and commands +- tell which types of command is supported by a device +- create new command requests of a specific type for some target device +- monitor the progression of a specific command request up to completion. + +## Concepts + +### Operations, Capabilities, and Commands + +From a user perspective an *operation* is a predefined sequence of actions +that an operator can trigger on a device to reach some desirable state. +It can be to restart the device or to install some new software. +From an implementation perspective, an operation is an API identified by a well-known name such as `restart` or `software_update`. +This API rules the coordination among the software components that need to interact to advance the operation. + +Not all entities and components of a %%te%% device support all the operations, +and, even if they do, the implementations might be specific. +Installing a software package on top of service makes no sense. +Restarting the device is not the same as restarting one of its services. +Each entity or component has to declare its *capabilities* i.e. the operations made available on this target. + +Strictly speaking, capabilities are not implemented nor declared by the devices and the services themselves. +They are implemented by %%te%% services and plugins. +These are the components which actually implement the operations interacting with the operating system and other software. +For instance, device restart and software updates are implemented by the `tedge-agent`. + +Once an operation has been registered as a capability of some target entity or component, +an operator can trigger operation requests a.k.a *commands*, +for this kind of operation on this target, +say to request a software update, then a restart of the device. + +### MQTT-Driven Workflows + +The core idea is to expose over MQTT the different states a specific operation request might go through; +so independent sub-systems can observe the progress of the request +and participate as per their role, when it is their turn. + +- A specific topic is attached to each command under-execution. + - This topic is specific to the target of the command, the requested operation and the request instance. + - e.g. `te/device/child-xyz///cmd/configuration-update/req-123` +- The messages published over this topic represent the current state of the command. + - Each message indicates at which step of its progression the command is and gives all the required information to proceed. + - e.g. `{ "status": "init", "target": "mosquitto", "url": "https://..." }` +- The state messages are published as retained. + - They capture the latest state of the operation request. + - Till some change occurs, this latest state is dispatched to any participant on reconnect. +- Several participants act in concert to move the command execution forward. + - The participants observe the progress of all the operations they are interested in. + - They watch for the specific states they are responsible in moving forward. + - When a step is performed, successfully or not, the new state is published accordingly by the performer. + +## Topics + +Following [%%te%% MQTT topic conventions](../mqtt-api.md#commands), +each device is assigned a specific topic prefix, +with a metadata sub-topic per command type +and specific sub-topics for the requests. + +### Command metadata topics + +The command metadata topics are used to declare the *capabilities* of a device. + +The ability for an entity *a*/*b*/*c*/*d* to handle a given *operation*, is published as a retained message +on the topic __te__/*a*/*b*/*c*/*d*/__cmd__/*operation*. + +```mermaid +graph LR + te --/--- identifier --/--- cmd + subgraph root + te + end + + subgraph target + identifier["<target identifier>"] + end + + subgraph command["command metadata"] + direction LR + cmd --/--- cmd_type["<command_type>"] + end +``` + +Where the groups are described as follows: + +| Group | Description | +|--------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| root | The [topic root prefix](../mqtt-api.md#group-root) for this installation of %%te%%. | +| target | The [topic identifier](../mqtt-api.md#group-identifier) of the target device, possibly a [custom identifier](/contribute/design/mqtt-topic-design.md#using-custom-identifier-schemas). | +| `cmd` | The [command channel](../mqtt-api.md#group-channel) grouping all of the commands for this target. | +| command_type | The type name of the operation. | + +A service that implements an operation for a device publishes on start, a capability message notifying +that this device can be sent commands of this type. +As an example, the `tedge-agent` which implements the `restart` operation emits on start a capability message for that operation: + +```sh te2mqtt +tedge mqtt pub -r 'te/device/main///cmd/restart' '{}' +``` + +These messages are published with the retained flag set. So, a client process, such a mapper, can discover on start +what are __all the capabilities of all the devices__: + +```sh te2mqtt formats=v1 +tedge mqtt sub 'te/+/+/+/+/cmd/+' +``` + +### Command status topics + +The actual command requests are published on the command status topics. + +Each request is given a unique *command identifier* +and the topic __te__/*a*/*b*/*c*/*d*/__cmd__/*operation*/*command-identifier* +is used to trigger and monitor this request for a given *operation* on a target entity *a*/*b*/*c*/*d*. + +```mermaid +graph LR + te --/--- identifier --/--- cmd + subgraph root + te + end + + subgraph target + identifier["<target identifier>"] + end + + subgraph command["command request"] + direction LR + cmd --/--- cmd_type["<command_type>"] --/--- cmd_id["<cmd_id>"] + end +``` + +| Group | Description | +|--------------|-------------------------------------------| +| command_id | The identifier of the operation instance. | + +:::note +The `command_id` is an arbitrary string however it should be unique. +It is recommended to either use a unique id generator, or add a unix timestamp as a suffix, e.g. date +%s. +This unique id assigned by the requester, who is also responsible for creating the topic +with an initial state and for finally removing it. +::: + +The messages published on these topics represent each the current status of a running command. +So, one can list __all the in-progress commands of any type across all the devices__: + +```sh te2mqtt formats=v1 +tedge mqtt sub 'te/+/+/+/+/cmd/+/+' +``` + +As an example, software update is an operation that requires coordination between a mapper and `tedge-agent`. +On reception of a software update request from the cloud operator, +the `tedge-mapper` creates a fresh new topic for this command, +say `te/device/main///cmd/software_update/c8y-mapper-123` with a unique command id: `c8y-mapper-123`. +On this topic, a first retained messages is published to describe the operator expectations for the software updates. + +```sh te2mqtt +tedge mqtt pub -r 'te/device/main///cmd/software_update/c8y-mapper-123' '{ + "status": "init", + "modules": [ + { + "type": "apt", + "name": "collectd", + "version": "5.7", + "action": "install" + } + ] +}' +``` + +Then, the `tedge-agent` and possibly other software components take charge of the command, +making it advance to some final state, +publishing all the successive states as retained messages on the command topic. + +Eventually, the `tedge-mapper` will have to clean the command topic with an empty retained message: + +```sh te2mqtt +tedge mqtt pub -r 'te/device/main///cmd/software_update/c8y-mapper-123' '' +``` + +## Message payloads + +The message payloads are all specific to each operation type. +However, there are some common rules: + +- JSON is used for all messages. + - The schemas are flexible. + - Extra fragments can be added by participants and ignored by others if not relevant. +- Device capabilities are published as retained messages. + - If a capability has no specific metadata, then the capability is published as an empty JSON object `{}`. + - A capability is removed by publishing a retained empty string `""`. +- Command status messages are published as retained messages on their dedicated topic. + - These messages represent the current state of the command. + - Initialized with all the required information to execute the command, + they are updated during the command execution with outcome data. + - A command request is cleared by publishing a retained empty string `""`. +- The command status messages contain a mandatory `"status"` field. + - This status tells at which step the command execution is along its workflow. + - A new command is created with a `"status"` set to `"init"`. + - When execution starts, the `"status"` is set to `"executing"`. + - When execution is successful, the `"status"` is set to `"successful"`. + - If execution fails for some reason, the `"status"` is set to `"failed"` and the reason given as `"reason"` field. + - Each operation might define other specific `"status"` values for extra steps, actions or checks. + +:::info +Health checks is a notable exception to the rule "command status are published as retained messages". +A health check is sent to the channel `cmd/health/check` of a service (or a device) as a non-retained request `{}`, +triggering a health message response published on the `status/health` channel of the same service +(or respectively the channels of all the service running on that device). +::: + +## Operation workflow + +An operation workflow defines the possible sequences of actions for an operation request +from its initialization up to its success or failure. It specifies the actions to perform +as well as any prerequisite checks, outcome validations and possible rollbacks. +However, a workflow doesn't define how to perform these actions. +These are delegated to %%te%% services, scripts, application-specific services or other devices. +More precisely, an operation workflow defines: +- the *observable states* of an ongoing operation instance + from initialization up to a final success or failure +- the *participants* and their interactions, passing the baton to the software component + whose responsibility is to advance the operation in a given state + and to notify the other participants what is the new resulting state +- the *possible state sequences* so that the system can detect any stale or misbehaving operation request. + +A specific workflow rules each operation type, with specific: +- states +- message payloads +- states transitions. + +However, there are some common rules: +- There at least four states: `"init"`, `"executing"`, `"successful"` and `"failed"`. +- A new command has to be created in the `"init"` state. +- Some checks can be done before the `"executing"` state, but this one should be the first triggering updates on the system. +- There only two terminal states: `"successful"` and `"failed"`. +- The retained messages are finally cleared by the requester. + +```mermaid +flowchart LR + init --> executing --> successful + executing --> failed + successful --> done + failed --> done + + subgraph init_state["status: init"] + init(gives all the data to proceed

created by the requester
typically a mapper) + end + + subgraph executing_state["status: executing"] + executing(the command is under execution
the content is the same as for init) + end + + subgraph successful_state["status: successful"] + successful(the command has been successful
some outcome might be returned to the requester) + end + + subgraph failed_state["status: failed"] + failed(the command has failed
the reason is given to the requester) + end + + subgraph done_state[done] + done(finally cleared by the requester) + end +``` + + +### Capability declaration + +On start, a service that implements an operation for a device must: + +- know the MQTT root prefix (per default: `te`), +- know the entity topic identifier of the target device (e.g. `device/child001//`), +- know the well-known name of the operation (e.g. `software_update`), +- forge a capability message describing the operation support (e.g. `{ "types": ["apt", "docker"] }`) +- publish this capability message, with a retained flag, on the target command metadata topic (e.g. `te/device/child001///cmd/software_update`) + +### Command execution + +To request the execution of a command, a client must: +- know the MQTT root prefix (per default: `te`), +- know the entity topic identifier of the target device (e.g. `device/child001//`), +- know the well-known name of the operation (e.g. `software_update`), +- assign a unique id to its command (say `c8y-1234`) +- use these target and command identifiers to forge the topic name for its request + (e.g. `te/device/child001///cmd/software_update/c8y-1234`) +- describe the request in a JSON message as defined by the operation API +- publish, as a retained message, the request on its specific topic with a `"status"` set to `"init"` +- await on the same topic for a response message with a `"status"` set to `"executing"`, `"successful"` or `"failed"` +- ignore any message with an unknown status as the agent is free to use intermediate states +- finally, clear the command topic by sending an empty retained message + +To implement an operation on behalf of a device, a service must: +- know the MQTT root prefix (per default: `te`), +- know the entity topic identifier of the target device (e.g. `device/child001//`), +- know the well-known name of the operation (e.g. `software_update`), +- subscribe to any command requests of this type and the target device (i.e. `te/device/child001///cmd/software_update/+`) +- react to a well-formed requests by publishing the updated status when the step is done. + +### Example + +As an example, let's take software updates on a child device. + +When launched on the child device `device/child001//`, +`tedge-agent` notifies that software packages of types: `apt` and `docker` can be updated on this device: + +```sh te2mqtt formats=v1 +tedge mqtt pub --retain 'te/device/child001///cmd/software_update' '{ + "description": "Install, update and remove software packages", + "types": [ + "apt", + "docker", + ] +}' +``` + +On reception of this message, the Cumulocity mapper notifies Cumulocity of this capability. +On request from a cloud operator, the Cumulocity mapper creates a new command instance, say to update `nodered`: + +```sh te2mqtt formats=v1 +tedge mqtt pub --retain 'te/device/child001///cmd/software_update/c8y-123' '{ + "status": "init", + "updateList": [ + { + "type": "debian", + "modules": [ + { + "name": "nodered", + "version": "1.0.0", + "action": "install" + } + ] + } + ] +}' +``` + +The agent, running on `device/child001//`, notifies that it will execute the command: + +```sh te2mqtt formats=v1 +tedge mqtt pub --retain 'te/device/child001///cmd/software_update/c8y-123' '{ + "status": "executing", + "updateList": [ + { + "type": "debian", + "modules": [ + { + "name": "nodered", + "version": "1.0.0", + "action": "install" + } + ] + } + ] +}' +``` + +Then the agent proceeds, here installing a specific version of `nodered`, and notifies the mapper when done: + +```sh te2mqtt formats=v1 +tedge mqtt pub --retain 'te/device/child001///cmd/software_update/c8y-123' '{ + "status": "successful", + "updateList": [ + { + "type": "debian", + "modules": [ + { + "name": "nodered", + "version": "1.0.0", + "action": "install" + } + ] + } + ] +}' +``` + +The Cumulocity mapper, having subscribed to all software_update commands, +monitors this command instance and notifies Cumulocity of its progress upto completion. +Finally, the Cumulocity mapper clear the command topic: + +```sh te2mqtt formats=v1 +tedge mqtt pub --retain 'te/device/child001///cmd/software_update/c8y-123' '' +``` diff --git a/versioned_docs/version-1.1.0/references/agent/index.md b/versioned_docs/version-1.1.0/references/agent/index.md new file mode 100644 index 0000000..58347d3 --- /dev/null +++ b/versioned_docs/version-1.1.0/references/agent/index.md @@ -0,0 +1,61 @@ +--- +title: The Agent +tags: [Reference, Agent] +sidebar_position: 2 +--- + +import DocCardList from '@theme/DocCardList'; + +# The Agent and Operations Plugins + +The __agent__ and the __operation plugins__ are the main components used by %%te%% +to enable edge device management from various clouds on diverse hardware, +giving the cloud operators a uniform interface to install software packages, +update configuration files, inspect log files, restart devices ... + +The API between the mappers and the agent is designed to abstract device location, operating system, file system and hardware. +__Any service securely connected to the local MQTT bus can trigger commands as well as respond to commands.__ + +- The agent and the operation plugins interact with the operating system, + running the appropriate sequences of checks and actions + in response to command requests initiated by [mappers](../mappers) on behalf of cloud operators. +- `tedge-agent` is the out-of-the-box implementation of the device management APIs provided by %%te%%. + It can run on the main device as well as child devices. + It can be replaced with any other user-developed components that implement these device management APIs + addressing specific requirements or hardware. +- %%te%% also provides the tools to define, extend and combine *user-defined operation workflows* + that rule the sequence of steps applied when an *operation* is triggered by an operator or a software component. + +```mermaid +--- +title: System components enabling device management +--- +graph LR + mapper_daemon <-- JSON/MQTT --> agent_daemon -- System Calls --> device_os + mapper_start <-- JSON/MQTT --- agent_start + http_proxy <-- HTTP --- http_callback + + subgraph mapper + mapper_start(report to the cloud
available operations across devices) + mapper_daemon(trigger and monitor commands over MQTT
on behalf of cloud operators) + http_proxy(proxy HTTP requests to the cloud) + end + + subgraph device[main or child device] + subgraph agent[agent or operation plugin] + agent_start(on start, publish commands supported by this device) + agent_daemon(execute command workflow
reporting progress over MQTT) + http_callback(GET/PUT large data set over HTTP) + end + + subgraph os[device os] + device_os(apply commands) + end + end +``` + +- MQTT is used to *declare* which operations are supported by each device, + to *trigger* commands and to *monitor* their progress upto completion. +- HTTP is used to *transfer* files between devices independently of their file systems. + + \ No newline at end of file diff --git a/versioned_docs/version-1.1.0/references/agent/operation-workflow.md b/versioned_docs/version-1.1.0/references/agent/operation-workflow.md new file mode 100644 index 0000000..fe03936 --- /dev/null +++ b/versioned_docs/version-1.1.0/references/agent/operation-workflow.md @@ -0,0 +1,597 @@ +--- +title: User-defined Operation Workflows +tags: [Reference, MQTT, Operations] +sidebar_position: 7 +description: User-defined operation workflow reference guide +--- + +## Overview + +An agent developer can define application specific [operation workflows](./device-management-api.md#mqtt-driven-workflows) +to control how an operation is performed on a device. +%%te%% **tedge-agent** provides the tools to: + +- override existing workflows +- define new states and actions such as pre-requisite or post-execution checks +- introduce new transitions such as rollbacks or conditional executions +- create new workflows, combining workflows and steps + +## Example + +Here is an example where three software components participate in a `firmware_update` command. +- The **tedge-mapper** creates the initial state of the command + providing the required information to install a new version of the device firmware; + and then waits for the final outcome (in black). +- The **tedge-agent** handles the main steps (in red): downloading the file and installing it where expected. +- User-provided **scripts** handle domain-specific checks (in blue) + to timely schedule the command as well as to ensure the configuration file is not corrupted and properly installed. + +```mermaid +stateDiagram-v2 + [*] --> init + init --> install + install --> reboot + reboot --> verify + verify --> commit + commit --> successful + init --> failed + install --> failed + verify --> rollback + commit --> rollback + rollback --> failed + successful --> [*] + failed --> [*] + + classDef specific color:blue; + class init, verify specific + + classDef plugin color:red; + class install, reboot, commit, rollback plugin + + classDef mapper color:black; + class successful, failed mapper +``` + +### Key points + +Observe on the example that: + +- At any state, *one and only one* participant is responsible to move the operation forward. +- Publishing a state to the MQTT command topic, can be seen as passing the baton from one participant to another. + The mapper creates the **init** state and then lets the other components work. + The agent tells the installation has been successful by publishing the **reboot** state, + but do nothing till the domain-specific component has checked the file and move the command state to **install**. +- Each software component has to know only *some* states of the whole workflow: + - the states they are responsible for + - the states they create to pass the control. +- The compatibility of two participants, one advancing to the new state owned by the other, is only defined by the message payload: + - all the property fields, required to make progress at some state, must be provided by the previous participant. + +### Benefits + +The benefits are that: +- A participant can be substituted by another implementation as long as the substitute implementation + is ready to process at least all the states processed by the former implementation. +- Extra states and participants can be added as long as each state is owned by one participant. + - For instance, an agent developer can introduce a **rollback** state in the `firmware_update` workflow, + associated by another software component responsible for these rollbacks. +- The same executable can be used to handle operations on different targets. + For instance, the **tedge-agent** can run on the main device `te/device/main//` + as well as on a child-device identified by `te/device/child-xyz//`. +- Specific versions of the same workflow can be defined on different targets. + The main and child devices can each run their own version of a workflow for an operation. + As an example, an agent developer can define an extra rollback state on the main device but not on the child devices. +- A specific executable can be substituted on a specific target. + If for some reasons, **tedge-agent** cannot be installed on a child-device, + then a specific implementation of the `firmware_update` MQTT API can be used to serve firmware updates + on that specific hardware. + +## Operation API + +As several software components have to collaborate when executing a command, each operation must define a specific API. +This API should be based on the principles of MQTT-driven workflow and defines: +- the well-known operation name such `firmware_update` or `restart` +- user documentation of the required input and the expected outcome of an operation request +- the set of observable states for a command and the possible state sequences +- for each state: + - the well-known name such as **download** or **downloaded** + - the schema of the state payload and the required parameters to process a command at this stage + - developer documentation on the role of each parameter and the expected checks and actions +- the schema for the capability message sent when the operation is enabled on some %%te%% entity or component + - developer documentation on the role of each field of the capability message + +A workflow implementation is free to define the states a command can go through +as well as the message payload attached to each state. + +However, there are some rules and best practices. + +- Three states are mandatory: **init**, **successful** and **failed**. +- **init** must be the unique initial state of the workflow. + - This state must give all the information required to execute the command. + - Having no other initial states is important so any command requester (as the mapper) + can trigger commands in a systematic manner *even* if the workflow is updated. +- **successful** and **failed** must be the unique terminal states of the workflow. + - The payload of the **failed** state should contain a `reason` property telling what failed. + - Having no other terminal states is important so any command requester (as the mapper) + can await the termination of a command in a systematic manner *even* if the workflow is updated. +- A workflow API should define *no-op* states with no pre-defined actions + and which sole purpose is to give an agent developer the opportunity to *insert* its own logic and extra steps. + - A *no-op* state is a state which has a single direct transition to the next state. + - As an example, having a **downloaded** *no-op* state, with a direct transition to an **install** state which uses the downloaded file, + lets an agent operator override the automatic transition, say to check the downloaded file *before* moving the **install** state. + - The **init** state should be a *no-op* state. + +### Operation Implementation + +A workflow implementation for a specific operation must implement the actions specified for each non *no-op* state. +This implementation has some freedom, notably to implement extra checks and actions but also to add new states. + +However, there are some rules and best practices. + +- All the state messages must be published as retained with QoS 1. +- A workflow implementation should not reject a state message payload with unknown fields. + - It's also important to keep these unknown fields in the following states. + - This is important as we want to *extend* the workflow of an operation. + A software component added by the user might need these *extra* fields the plugin is not aware of. +- A workflow implementation must not react on *no-op* states nor terminal states. + - The transition from a *no-op* state must be handled either by %%te%% as direct transition + or overridden by the user with domain-specific checks and actions. + - The terminal states, a.k.a **successful** and **failed**, are owned by the process which created the **init** state (in practice, the mapper). + Only this process should clear the retained message state for an operation instance by sending an empty payload on command's topic. + +## User-defined Operation Workflow + +%%te%% provides a mechanism to define, extend and combine workflows. + +This mechanism is provided by the **tedge-agent** which gathers a set of user-defined workflows +and combined them with the builtin workflows implemented by the agent itself. + +Each workflow is defined using a TOML file stored in `/etc/tedge/operations`. Each specifies: +- the command name that it should trigger on + such as `firmware_update` or `restart` +- the list of states +- for each state: + - the state name as defined by the operation API + - an action to process the command at this stage, e.g. + - run a script + - trigger a sub operation + - let the agent applies the builtin behavior + - instructions on how to proceed given the successful or failed outcome of the action. + +```toml title="file: firmware_update_example.toml" +operation = "firmware_update" + +on_error = "failed" + +[init] + script = "/usr/bin/firmware_handler.sh plan" + on_success = "executing" + on_error = { status = "failed", reason = "not timely" } + +[executing] + action = "proceed" + on_success = "install" + +[install] + script = "/usr/bin/firmware_handler.sh install ${.payload.url}" + on_success = "reboot" + +[reboot] + script = "/usr/bin/firmware_handler.sh reboot" + on_exec = "verify" + +[verify] + script = "/usr/bin/firmware_handler.sh verify" + on_success = "commit" + on_error = { status = "rollback", reason = "sanity check failed" } + +[commit] + script = "/usr/bin/firmware_handler.sh commit" + on_success = "successful" + on_error = { status = "rollback", reason = "commit failed" } + +[rollback] + script = "/usr/bin/firmware_handler.sh rollback" + on_success = "failed" + +[successful] + action = "cleanup" + +[failed] + action = "cleanup" +``` + +%%te%% combines all these workflows to determine what has to be done +when a state message is published for a command on a topic matching the global topic filter for commands, +i.e. `te/+/+/+/+/cmd/+/+`. +- Each running instance of the __tedge_agent__ reacts only on commands targeting its own device. +- If a user-defined workflow has been defined for this operation, then this workflow is used to determine the required action. +- If no workflow has been defined by the user for this operation, then the built-in workflow is used. +- If there is no workflow or no defined action for the current state, + then the __tedge_agent__ simply waits for another component to take over the command. + +### Script Execution + +A script can be attached to a command state. + +``` +[state] +script = "/full/path/command [args]" +``` + +This script is given as a plain command line possibly with arguments. + +Data extracted from the command status topic and message payload can be passed as argument to the script. + +- `"/bin/new-command.sh ${.topic} ${.payload}"` passes two arguments to the `/bin/new-command.sh` program. + - The first one is the full command request topic (*e.g.* `te/device/main/cmd///restart/c8y-mapper-123`). + - The second one is the full json payload (*e.g.* `{"status': "init"}`). +- Specific path expressions can be used to pass specific excerpts. + - `${.}` is a json for the whole message including the `topic` and the `payload`. + - `${.topic}` is the command request topic (*e.g.* `te/device/main/cmd///restart/c8y-mapper-123`) + - `${.topic.target}` is the command target identity (*e.g.* `device/main/cmd//`) + - `${.topic.operation}` is the command operation (*e.g.* `restart`) + - `${.topic.cmd_id}` is the command request unique identifier (*e.g.* `c8y-mapper-123`) + - `${.payload}` is the whole command json payload (*e.g.* `{"status': "init"}`) + - `${.payload.status}` is the command current status (*e.g.* `"init"`) + - `${.payload.x.y.z}` is the json value extracted from the payload following the given `x.y.z` path if any. + - If given `${.some.unknown.path}`, the argument is passed unchanged to the script. + +The script exit status and output is used to determine the next step for the command. +- If the script cannot be launched or return a non-zero status, the command request is marked as __failed__. +- If the script successfully returns, its standard output is used to update the command state payload. + - From this output, only the excerpt between a `:::begin-tedge:::` header and a `:::end-tedge:::` trailer is decoded. + This is done to ease script authoring. A script can emit arbitrary output on its stdout, + and just have to surround its workflow updates with the `:::begin-tedge:::` and `:::end-tedge:::` markers. + - If this excerpt is a json payload, this payload is injected into the previous message payload + (adding new fields, overriding overlapping ones, keeping previous unchanged ones). + - If this excerpt is a json payload with a `status` field, then this status is used as the new status for the command. +- If the script output is empty, then the exit status of the process is used to determine the next step. + +### Next step determined by script exit status + +The exit status of the script processing a command state +can be used to determine the next state of the workflow. + +The workflow can specify for each exit code: +- the next command status +- a failure reason + +```toml +script = "/some/script.sh with some args" +on_exit.0 = "next_state" # next state for an exit status +on_exit.1 = { status = "retry_state", reason = "busy"} # next status with fields +on_exit.2-5 = { status = "fatal_state", reason = "oops"} # next state for a range of exit status +on_exit._ = "failed" # wildcard for any other non successful exit +on_kill = { status = "failed", reason = "killed"} # next status when killed +``` + +- `on_success` is syntactic sugar for `on_exit.0` +- `on_error` is syntactic sugar for `on_exit._` +- This is an error to provide more than one handler for an exit code (overlapping ranges). +- If no reason is provided for an exit code, the default reason is `"${program} exited with ${code}"`. +- If no reason is provided for the on kill handler, the reason is `"${program} killed by ${code}"`. +- The default for the `on-error` and `on-kill` status is `"failed"`. +- There is no default for the `on-success` status. + In such a case the output of the script is used to determine the next status. + +If the standard output of the script contains a JSON object surrounded by `:::begin-tedge:::` and `:::end-tedge:::` markers, +then this object is injected in the command state message payload. +There are two exceptions, though. +The `status` and `reason` fields are determined after the exit code and not the standard output. + +### Next step determined by script output + +The output of the script processing a command state +can be used to determine the next state of the workflow. + +For that to work: +- An `on_stdout` handler must be provided, listing all the states that can possibly be dictated by the script output. +- Neither `on_success` nor `on_exit.0` status must be given as this would make the next status computed after the exit code. +- The standard output of the script must emit a JSON object surrounded by `:::begin-tedge:::` and `:::end-tedge:::` markers. +- This JSON object must provide a `status` field and possibly a `reason` field. + +```toml +script = "/some/script.sh with some args" # no given `on_exit.0` status +on_error = { status = "fatal_state", reason = "fail to run the script"} # possibly some `on_error` and `on_kill` handlers +on_stdout = ["state-1", "state-2", "state-3"] # the list of status accepted as next status +``` + +- If the script is successful and its output returns some `status` and `reason` fields, these are used for the next state. +- If the script is successful but its output contains no `status` field, then `on_error` is used. +- If the script cannot be executed, is killed or return an error with no specific `on_exit` handler, + then the `status` and `reason` fields are determined by the `on_error` and `on_kill` handlers + (or if none where provided using the default `on_error` and `on_kill` handlers). +- If the script is successful or returns an expected exit code (i.e. one with a specific `on_exit` handler), + then the fields of the JSON object extracted from the script output are injected into the command state message payload. + - This is done in the successful as well as the failed cases. + - If a `status` field is also provided by the message payload, + then the `status` provided by the workflow definition trumps the value provided by the script. + - For the `reason` field, the rule is reversed: + the value provided by the script trumps the `reason` provided by the workflow definition if any. + +### Background scripts + +A workflow state can be handled using a *background script*. +When executed, as a detached process, by the __tedge_agent__ no response nor exit status is expected from the script. +In any case the workflow will be moved to the given next state. +And this move to the next state is even persisted *before* the script is executed. +This can notably be used to restart the device or the agent. +After the restart, the workflow will resume in the state specified by the workflow. + +```toml +[agent-restart] +background_script = "sudo systemctl restart tedge-agent" +on_exec = "waiting-for-restart" + +[waiting-for-restart] +script = "/some/script.sh checking restart" +on_stdout = [ "successful_restart", "failed_restart"] +``` + +Note that: +- No `on_exit` nor `on_kill` status can be provided, as the script is not monitored. +- If the script cannot be launched, the workflow will be moved to the final `"failed"` state. + +### 🚧 Sub-Operation Execution {#sub-operation-execution} + +:::info +🚧 The syntax for triggering other workflows from an existing workflow is still being finalized so please avoid using it in production environments. +::: + +An operation workflow can trigger a command defined by another workflow. + +```toml +[] +operation = "" +input_script = "/some/script/which/stdout/is/used/as/sub-operation/init/state" +input.x = "some static value" +input.y = "${.payload.y}" +on_exec = "" +``` + +Operation specific arguments can be provided to the sub-operation instance, +and data extracted from the calling command status topic and message payload can be injected into these arguments. + +- For instance, `input.url = "http://127.0.0.1:8000/tedge/file-transfer/foo.txt"` + adds an `url` property to the init message of the sub-operation. +- Similarly, `input.url = "${.payload.config_update.url}"` + adds an `url` property to the init message of the sub-operation, extracting this url from the calling command state. +- Several such arguments can be added to the sub-operation initial state payload. + - All these individual arguments are combined into a JSON object value published on the sub-operation MQTT topic. + - The calling command excerpts injected into this initial state are defined using the same path conventions as for script handlers. + - If an `input.status` value is provided, this value will be ignored and replaced by `"init"` + as this is the only valid value for the initial state of a command. + - If no value at all is provided for `input`, then the default value is `{"status" = "init"}`. +- For cases where there is no one-to-one relationship between the properties of the current operation state + and the properties of sub-operation init state, the sub-operation init state can be computed using a script. + - `input_script = "/some/script/which/stdout/is/used/as/sub-operation/init/state"` + - The input script can be passed parameters extracted from the current command state, + *e.g.* `input_script = "/bin/extra_configuration_updates.sh ${.payload.device_profile}"` + - When the input script is successful, its stdout is used to derive the init state of the sub-operation + - From this output, only the excerpt between a `:::begin-tedge:::` header and a `:::end-tedge:::` trailer is decoded. + - If this excerpt is a json payload, this payload is injected into the previous message payload + (adding new fields, overriding overlapping ones, keeping previous unchanged ones). + - If this excerpt is a json payload with a `status` field, then this status is ignored and its value replaced by `"init"`. + - When the input script fails, no subcommand is created and the command moves to the `failed` state. + - If both a script and explicit input properties are provided, the script output is applied first and the explicit properties second. +- There is no way to assign a command identifier for the new sub-operation instance. + - This identifier is generated by %%te%%, which creates, monitors and clears the MQTT topic for the sub-operation. + +From an execution perspective, triggering a sub-operation is similar to running a background script. + +- The calling command workflow is paused while the sub-operation is processed. +- The calling workflow moves to an `on_exec` state from where the sub-operation execution is watched until success, failure or timeout. +- A specific `"await-operation-completion"` action is to be used for this `on_exec` state where the sub-operation completion is awaited. +- The steps following the sub-operation completion are defined on the `on_exec` state with `on_success`, `on_error` and `on_timeout` handlers. + +For example, a configuration update can be triggered with: + +```toml +[trigger_config_update] +operation = "config_update" +input.tedgeUrl = "http://127.0.0.1:8000/tedge/file-transfer/example/config_update/mosquitto-1234" +input.type = "mosquitto" +on_exec = "waiting_for_config_update" + +[waiting_for_config_update] +action = "await-operation-completion" +timeout_second = 600 +on_timeout = "timeout_config_update" +on_success = "successful_config_update" +on_error = { status = "failed", reason = "fail to update the config"} +``` + +### Setting step execution timeout + +The execution time of the state transitions of a workflow can be limited using timeouts. +- A default timeout can be set at the level of an operation for all the transitions. +- Individual timeout can be set to each state of the workflow. + +```toml +timeout_second = 300 +on_timeout = { status = "failed", reason = "timeout" } +``` + +Some scripts cannot be directly controlled. +This is notably the case for the background scripts restarting the device. +For those any timeout has to be set on the waiting state. + +```toml +["device-restart"] +background_script = "sudo reboot" +on_exec = "waiting-for-restart" + +["waiting-for-restart"] +action = "await-agent-restart" +timeout_second = 600 +on_timeout = "timeout_restart" +on_success = "successful_restart" +``` + +### Running builtin actions + +Builtin actions can be used to control a command at some state. + +```toml +[""] +action = " " +``` + +As for scripts, these actions can be complemented with handlers dictating how to proceed when the action is successful or failed. +The set of accepted handlers for an action are the following: + +- `on_success = ""` defines the next state when the action is successful +- `on_error = { status = "", reason = ""}` defines the next state when the action failed +- `timeout_second = 3600` the number of second given to the action to execute +- `on_timeout = { status = "", reason = ""}` defines the next state when the action is not be completed within the time limit + +For some action, notably a device `restart`, the handlers are limited to one: +- `on_exec = ""` defines the next state once the action has been launched in the background. + The action outcome will have to be observed in this `on_exec` state. + +Currently, here are the available actions: + +- `await-agent-restart` awaits for **tedge-agent** to restart +- `await-operation-completion` awaits for a sub-operation to reach a success, failure or timeout +- `builtin` is used when a builtin operation is overwritten by a custom workflow and indicates that for that state + the builtin action has to be applied. +- `proceed` is a no-op action, simply proceeding to the next state, which is useful when a builtin operation is customized + but no specific behavior has to be added on a workflow extension point. +- `cleanup` marks the terminal state of the workflow where the command has been fully processed + and where the original requester is expected to clean up the command retained message storing its state. + +#### Awaiting the agent to restart + +When the expected outcome of a script is to restart the device or the agent, +this script cannot be monitored end-to-end by the agent which will die executing the script. +The success or failure of the script can only be detected when the agent resumes. + +This is done using a combination of the `background_script` directive with an additional state +that awaits the agent restart using the `await-agent-restart` action after the script is triggered. + +1. The script is declared as a `background_script` with an `on_exec` handler. + This handler tells the agent to move to the next state awaiting the agent restart after the script is executed. + The agent persists this next state on disk before launching the script that can result in a reboot. +2. The action attached to this next state is to `await-agent-restart` + with two handlers for the successful and timeout cases. +3. On a successful reboot, the agent resumes from this persisted state awaiting restart and simply moves to the successful case. +4. If for some reason, no restart happens within the given timeout window, the agent moves to the `on_timeout` operation state. + +```toml +["device-restart"] +background_script = "sudo reboot" +on_exec = "waiting-for-restart" + +["waiting-for-restart"] +action = "await-agent-restart" +timeout_second = 600 +on_timeout = "timeout_restart" +on_success = "successful_restart" +``` + +#### Awaiting Sub-Command Completion + +When a sub `operation` is executed, the operation workflow moves to a waiting state which `action` must be `await-operation-completion`. +This waiting state specifies the next step of the calling workflow on success, failure or timeout of the sub-operation +using the `on_success`, `on_error` and `on_timeout` properties respectively. + +The `await-operation-completion` action also tells how to inject the outcome of the sub-operation into the invoking command state. +- This is done by attaching values to `output` subfields, as in `output.x.y.z = "${.payload.x.y.z}"` +- Many independent subfields can be added. They will all be injected into the current payload of the command state, leaving the other fields untouched. +- The invoking command payload cannot be overridden as a whole. Subfields can be added and updated but not removed. +- Specific path expressions can be used to extract specific excerpts from the sub-operation final state. +- `${.}` denotes the whole sub-operation final message including the `topic` and the `payload`. +- `${.topic}` is the sub-operation request topic +- `${.topic.target}` is the sub command target identity +- `${.topic.operation}` is the sub command operation +- `${.topic.cmd_id}` is the sub command request unique identifier +- `${.payload}` is the whole sub command json payload +- `${.payload.status}` is the sub command final status +- `${.payload.x.y.z}` is the json value extracted from the payload following the given `x.y.z` path if any. +- If given `${.some.unknown.path}`, the subfield is assigned the unchanged expression. + +```toml +[waiting_for_config_update] +action = "await-operation-completion" +output.some.new.command.field = "${.payload.of.the.sub.command}" +timeout_second = 600 +on_timeout = "timeout_config_update" +on_success = "successful_config_update" +on_error = { status = "failed", reason = "fail to update the config"} +``` + +#### Cleanup + +Used to automatically cleanup the retained command from the MQTT broker after the workflow execution completes. +To be used only on terminal states: `successful` and `failed`. + +```toml +["successful"] +action = "cleanup" + +["failed"] +action = "cleanup" +``` + +#### Proceed and Builtin actions + +The `"proceed"` and `"builtin"` actions are useful when customizing a builtin operation +(`software_list`, `software_update`, `restart`, `config_snapshot`, `config_update`, `log_upload`). +Indeed, the first step is to start with a workflow specification which mimics the builtin behavior. + +For instance, here is the builtin workflow for the `software_update` operation: + +```toml +operation = "software_update" # an operation for which tedge-agent provides an implementation + +["init"] +action = "proceed" # open to customization +on_success = "scheduled" + +[scheduled] +action = "builtin" # delegated to the tedge-agent +on_success = "executing" + +[executing] +action = "builtin" # delegated to the tedge-agent +on_success = "successful" + +[successful] +action = "cleanup" # waiting for the mapper to clean up the command + +[failed] +action = "cleanup" # waiting for the mapper to clean up the command +``` + +The action for the `"init"` state is a `"proceed"` action, meaning nothing specific is done by the __tedge-agent__ +and that a user can provide its own implementation. + +By contrast, the actions marked as `"builtin"` are those delegated to the __tedge-agent__ +and where the main task of the operation is performed, in that case, installing software. + +Here is a customized version of the same operation. + +```toml +operation = "software_update" # a customized workflow + +[init] +script = "/usr/bin/schedule-software-update.sh ${.}" # checking is the software update command is timely +on_success = ["scheduled"] +on_error = { status = "failed", reason = "not timely" } + +[scheduled] +action = "builtin" # the software installation steps are unchanged +on_success = "executing" + +[executing] +action = "builtin" +on_success = "successful" + +[successful] +action = "cleanup" + +[failed] +action = "cleanup" +``` + diff --git a/versioned_docs/version-1.1.0/references/agent/restart-operation.md b/versioned_docs/version-1.1.0/references/agent/restart-operation.md new file mode 100644 index 0000000..43f8a5c --- /dev/null +++ b/versioned_docs/version-1.1.0/references/agent/restart-operation.md @@ -0,0 +1,98 @@ +--- +title: Restart Operation +tags: [Reference, Agent, Restart] +sidebar_position: 4 +description: Restarting the device via an operation +--- + +# Restart Operation + +%%te%% defines a `restart` operation to restart a device, being the main device or a child device. + +- A restart is typically triggered by a [mapper](../mappers/index.md) on behalf of a cloud operator. +- A restart can also be triggered from another operation (as a software update) + or service (detecting for instance some anomalies requesting a reboot). +- `tedge-agent` is the reference implementation of the `restart` operation. +- However, a custom `restart` plugin implementation can be installed on a device with specific requirements. + +## MQTT API + +The `restart` operation API follows the [generic %%te%% rules for operations](./device-management-api.md): + +- The `te//cmd/restart` topic is used to tell the device `` can be restarted. +- Each `restart` request is given a `` and a dedicated topic `te//cmd/restart/`, + where all the subsequent states of the restart command are published during its execution. +- The workflow is [generic with `"init"`, `"executing"`, `"successful"` and `"failed"` statuses](./device-management-api.md#operation-workflow). + +### restart registration + +The registration message of the `restart` operation on a device is an empty JSON object `{}`. + +```sh te2mqtt formats=v1 +tedge mqtt pub --retain 'te/device/child001///cmd/restart' '{}' +``` + +### init state + +To trigger a restart operation on a device, the requester has no information to provide. +It only has to create a new `restart` command instance topic. + +```sh te2mqtt formats=v1 +tedge mqtt pub --retain 'te/device/child001///cmd/restart/c8y-2023-09-08T18:13:00' '{ + "status": "init" +}' +``` + +### executing state + +When ready, but before actually restarting the device, +the agent or the `restart` plugin publishes the new state of the command. + +```sh te2mqtt formats=v1 +tedge mqtt pub --retain 'te/device/child001///cmd/restart/c8y-2023-09-08T18:13:00' '{ + "status": "executing" +}' +``` + +### successful state + +After a successful reboot, +the agent or the `restart` plugin publishes the new state of the command. + +```sh te2mqtt formats=v1 +tedge mqtt pub --retain 'te/device/child001///cmd/restart/c8y-2023-09-08T18:13:00' '{ + "status": "successful" +}' +``` + +### failed state + +In case the reboot failed for some reason, +the agent or the `restart` plugin publishes the new state of the command, +adding a `reason` text field with the error. + +```sh te2mqtt formats=v1 +tedge mqtt pub --retain 'te/device/child001///cmd/restart/c8y-2023-09-08T18:13:00' '{ + "status": "failed", + "reason": "The device has not restarted within 5 minutes" +}' +``` + +### Command cleanup + +As for all commands, the responsibility of closing a `restart` is on the requester. +This is done by publishing an empty retained message on the command topic. + +```sh te2mqtt formats=v1 +tedge mqtt pub --retain 'te/device/child001///cmd/restart/c8y-2023-09-08T18:13:00' '' +``` + +## Implementation Contract + +The `restart` agent state must survive a device restart. + +- The `executing` state must be published *before* a reboot is scheduled. +- The `successful` state is published *after* the reboot when the agent resumes an ongoing restart command. +- The agent needs to differentiate between a simple process restart vs the actual device restart itself. +- A simple process restart is considered as a `failed` restart. + diff --git a/versioned_docs/version-1.1.0/references/agent/running-tedge-agent.md b/versioned_docs/version-1.1.0/references/agent/running-tedge-agent.md new file mode 100644 index 0000000..eeddef7 --- /dev/null +++ b/versioned_docs/version-1.1.0/references/agent/running-tedge-agent.md @@ -0,0 +1,51 @@ +--- +title: Running the Agent +tags: [Reference, Agent] +sidebar_position: 1 +description: Running the agent for device management on main and child devices +--- + +In order to enable %%te%% device management features on a device, +being the main device or a child device, +one has to install and run the `tedge-agent` service on this device. + +## On the main device + +Per default, `tedge-agent` assumes it run on the main device. + +```sh title="running tedge-agent on the main device" +tedge-agent +``` + +## On a child-device + +To launch `tedge-agent` on a child device, +one has to configure the [topic identifier](../mqtt-api.md#group-identifier) +on this device to point to the appropriate topic identifier. + +```sh title="running tedge-agent on the child device child-007" +sudo tedge config set mqtt.device_topic_id device/child-007// +tedge-agent +``` + +The configured device topic identifier can also be overridden on the command line. + +```sh title="running tedge-agent on the child device child-007" +tedge-agent --mqtt-device-topic-id device/child-007// +``` + +## Using a custom identifier schema + +If using a [custom identifier schema](/contribute/design/mqtt-topic-design.md#using-custom-identifier-schemas), +then the device topic identifier has to be configured even for the main device. + +```sh title="running tedge-agent when using a custom identifier schema" +sudo tedge config set mqtt.topic_root acme +sudo tedge config set mqtt.device_topic_id factory01/hallA/packaging/belt001 +tedge-agent +``` + +Or, using the command line: +```sh title="running tedge-agent while using a custom identifier schema" +tedge-agent --mqtt-topic-root acme --mqtt-device-topic-id factory01/hallA/packaging/belt001 +``` diff --git a/versioned_docs/version-1.1.0/references/agent/software-management.md b/versioned_docs/version-1.1.0/references/agent/software-management.md new file mode 100644 index 0000000..6bc14b3 --- /dev/null +++ b/versioned_docs/version-1.1.0/references/agent/software-management.md @@ -0,0 +1,389 @@ +--- +title: Software Management +tags: [Reference, Agent, Software Management] +sidebar_position: 3 +description: Details of the API to manage software on a device +--- + +%%te%% software management is implemented by two operations +which give the ability to manage software packages of different types on the same device. + +- `software_list` is used to fetch a pertinent subset of the software packages installed on a device. +- `software_update` let the user install, update and remove software packages on a device. + +## software_list MQTT API + +The `software_list` operation API follows the [generic %%te%% rules for operations](device-management-api.md): +- The `te//cmd/software_list` topic is used to publish the type of software packages + that can be managed on the device with the given topic identifier. +- Each `te//cmd/software_list/+` topic is dedicated to a software list command instance, +- The workflow is [generic with `"init"`, `"executing"`, `"successful"` and `"failed"` statuses](references/agent/device-management-api.md#operation-workflow). + +### Operation registration + +The registration message of the `software_list` operation on a device: +- must provide a `types` list of the types of software package that can be installed on this device (e.g. `["apt", "docker"]`) +- can provide a description of the operation and of each supported package type. + +```sh te2mqtt formats=v1 +tedge mqtt pub --retain 'te/device/child001///cmd/software_list' '{ + "description": "List software packages installed on the device", + "types": [ + "apt", + "docker" + ] +}' +``` + +### init state + +A `software_list` command has nothing to provide beyond a `status` field. +This empty message stands for a request of the list of software currently installed. + +```sh te2mqtt formats=v1 +tedge mqtt pub --retain 'te/device/child001///cmd/software_list/c8y-2023-09-25T14:34:00' '{ + "status": "init" +}' +``` + +### executing state + +Just before starting the command execution, the agent marks the command as executing +by publishing a retained message that repeats the former command except that: + +- the `status` is set to `executing` + +### successful state + +The payload for a successful `software_list` command has two fields: + +- the `status` is set to `successful` +- a `currentSoftwareList` field is added with the *new* list of packages installed on the device + - These packages are grouped by software package type: + ```json + { + "currentSoftwareList": [ + { + "type": "apt", + "modules": [ "..." ] + }, + { + "type": "docker", + "modules": [ "..." ] + } + ] + } + ``` + - Each installed package is given: + - the package `"name"`, + - the installed `"version"`. + +As an example, here is a (simplified) status message for a successful `software_list` command on a child device: + +```sh te2mqtt formats=v1 +tedge mqtt pub --retain 'te/device/child001///cmd/software_list/c8y-2023-09-25T14:34:00' '{ + "status": "successful", + "currentSoftwareList": [ + { + "type": "debian", + "modules": [ + { + "name": "nodered", + "version": "1.0.0", + }, + { + "name": "collectd", + "version": "5.12" + } + ] + }, + { + "type": "docker", + "modules": [ + { + "name": "nginx", + "version": "1.21.0", + }, + ] + } + ] +}' +``` + +### failed state + +The payload for a failed `software_list` is made of two fields: + +- the `status` is set to `failed` +- a `reason` text field is added with the root cause of the failure + +```sh te2mqtt formats=v1 +tedge mqtt pub --retain 'te/device/child001///cmd/software_list/c8y-2023-09-25T14:34:00' '{ + "status": "failed", + "reason": "Permission denied", +}' +``` + +## software_update MQTT API + +The `software_update` operation API follows the [generic %%te%% rules for operations](device-management-api.md): +- The `te//cmd/software_update` topic is used to publish the type of software packages + that can be managed on the device with the given topic identifier. +- Each `te//cmd/software_update/+` topic is dedicated to a software update command instance, + and is used to publish the subsequent states of the command execution. +- The workflow is [generic with `"init"`, `"executing"`, `"successful"` and `"failed"` statuses](references/agent/device-management-api.md#operation-workflow). + +### Operation registration + +The registration message of the `software_update` operation on a device: +- must provide a `types` list of the types of software package that can be installed on this device (e.g. `["apt", "docker"]`) +- can provide a description of the operation and of each supported package type. + +```sh te2mqtt formats=v1 +tedge mqtt pub --retain 'te/device/child001///cmd/software_update' '{ + "description": "Install, update and remove software packages", + "types": [ + "apt", + "docker" + ] +}' +``` + +:::info +The software types are currently not sent to the cloud (e.g. Cumulocity IoT), however this is planned in a future release. +::: + +### init state + +A `software_update` command is defined by an `"updateList"` array giving the packages to install, update or remove. + +- The `"updateList"` field is a list of software update actions. + - These actions are grouped by software package type: + ```json + {"updateList": [ + { + "type": "apt", + "modules": [ "..." ] + }, + { + "type": "docker", + "modules": [ "..." ] + } + ]} + ``` + - Each action is either to `"install"` or `"remove"` a software package. + - `"action": "install"` is to be understood as a goal: + "this package must be installed with this version" whatever the actual action to reach this goal, + being a fresh install, an upgrade or a downgrade. + - `"action": "remove"` is also to be understood as a goal: "that package must not be installed". + - An action provides: + - the package `"name"` (as known by the package packager), + - optionally a `"version"` (using the same conventions as the package manager), + - optionally an `"url"` from where to download the package. + +As an example, here is a message requesting a `software_update` on a child device: + +```sh te2mqtt formats=v1 +tedge mqtt pub --retain 'te/device/child001///cmd/software_update/c8y-2023-09-25T14:53:00' '{ + "status": "init", + "updateList": [ + { + "type": "apt", + "modules": [ + { + "name": "nodered", + "version": "1.0.0", + "action": "install" + }, + { + "name": "collectd", + "version": "5.12", + "url": "https://collectd.org/download/collectd-tarballs/collectd-5.12.0.tar.bz2", + "action": "install" + } + ] + }, + { + "type": "docker", + "modules": [ + { + "name": "nginx", + "version": "1.21.0", + "action": "install" + }, + { + "name": "mongodb", + "version": "4.4.6", + "action": "remove" + } + ] + } + ] +}' +``` + +### executing state + +Just before starting the command execution, the agent marks the command as executing +by publishing a retained message that repeats the former command except that: + +- the `status` is set to `executing` + +### successful state + +The payload for a successful `software_update` command +repeats the same content as the former request except that: + +- the `status` is set to `successful`. + +As an example, here is a status message for a successful `software_update` command on a child device: + +```sh te2mqtt formats=v1 +tedge mqtt pub --retain 'te/device/child001///cmd/software_update/c8y-2023-09-25T14:53:00' '{ + "status": "successful", + "updateList": [ + { + "type": "apt", + "modules": [ + { + "name": "nodered", + "version": "1.0.0", + "action": "install" + }, + { + "name": "collectd", + "version": "5.12", + "url": "https://collectd.org/download/collectd-tarballs/collectd-5.12.0.tar.bz2", + "action": "install" + } + ] + }, + { + "type": "docker", + "modules": [ + { + "name": "nginx", + "version": "1.21.0", + "action": "install" + }, + { + "name": "mongodb", + "version": "4.4.6", + "action": "remove" + } + ] + } + ] +}' +``` + +### failed state + +The payload for a failed `software_update` command +repeats the same content as the former request except that: + +- the `status` is set to `failed` +- a `reason` text field is added with the root cause of the failure +- a `failures` array field might be added to list the errors for all the failing actions. + +```sh te2mqtt formats=v1 +tedge mqtt pub --retain 'te/device/child001///cmd/software_update/c8y-2023-09-25T14:53:00' '{ + "status": "failed", + "reason": "Partial failure: Could not install collectd and nginx", + "updateList": [ + { + "type": "apt", + "modules": [ + { + "name": "nodered", + "version": "1.0.0", + "action": "install" + }, + { + "name": "collectd", + "version": "5.12", + "url": "https://collectd.org/download/collectd-tarballs/collectd-5.12.0.tar.bz2", + "action": "install" + } + ] + }, + { + "type": "docker", + "modules": [ + { + "name": "nginx", + "version": "1.21.0", + "action": "install" + }, + { + "name": "mongodb", + "version": "4.4.6", + "action": "remove" + } + ] + } + ], + "failures":[ + { + "type":"apt", + "modules": [ + { + "name":"collectd", + "version":"5.7", + "action":"install", + "reason":"Network timeout" + } + ] + }, + { + "type":"docker", + "modules": [ + { + "name": "mongodb", + "version": "4.4.6", + "action":"remove", + "reason":"Other components dependent on it" + } + ] + } + ] +}' +``` + +## tedge-agent implementation + +### Software management plugins + +The `tedge-agent` service uses software management plugins to interact with the actual package managers. + +For each type of software package supported on the device must be provided a specific software management plugin: + +- A plugin is an executable file implementing the [software plugin API](../software-management-plugin-api.md), + to `list`, `install` and `remove` software packages of a specific type. +- These plugins are looked up by `tedge-agent` in the plugin directory (`/etc/tedge/sm-plugins` if not specified otherwise). +- `tedge-agent` uses the file name of a plugin executables as the software package type name. + +### Settings + +`tedge-agent` behavior on `software_update` commands can be configured with `tedge config`. + +- `software.plugin.default` sets the default software plugin to be used for software management on the device. +- `software.plugin.max_packages` sets the maximum number of software packages reported for each type of software package. +- `software.plugin.exclude` sets the filtering criterion that excludes software packages from the output list if they match the pattern. +- `software.plugin.include` sets the filtering criterion that includes software packages in the output list if they match the pattern. + +:::info +Include pattern takes precedence over exclude pattern, so when both are used at the same time, the software list will exclude packages according to the pattern but keep the exceptions covered by the include pattern. +::: + +## Custom implementation + +%%te%% users can implement their own support for software management to address the specificities of their devices. +- This can be done leveraging the `tedge-agent` and implementing a custom [software plugin](../software-management-plugin-api.md). +- If for some reasons the `tedge-agent` cannot run on the target hardware, + then a service must be implemented to support the `software_list` and `software_update` operation, as described below. + In this case, the service is free to choose its own mechanisms to manage software packages + and can even run on a device that is not the target hardware. + + + diff --git a/versioned_docs/version-1.1.0/references/agent/tedge-configuration-management.md b/versioned_docs/version-1.1.0/references/agent/tedge-configuration-management.md new file mode 100644 index 0000000..5f39962 --- /dev/null +++ b/versioned_docs/version-1.1.0/references/agent/tedge-configuration-management.md @@ -0,0 +1,261 @@ +--- +title: Configuration Management +tags: [Reference, Configuration] +sidebar_position: 6 +description: Details of the API to manage configuration on a device +--- + +# Configuration Management + +%%te%% implements an operation to manage device configuration files. + +* The __tedge-agent__ enables configuration management on its running device + and, combined with a cloud mapper, extends this capability to enable configuration management from the cloud. +* This management is bi-directional: + * A device can act as a reference, + with all the managed files being uploaded to the [tedge file transfer repository](../file-transfer-service.md) + and stored there as a configuration snapshot. + * A configuration update can be pushed from the tedge file transfer repository to any devices of the same type, + i.e. supporting the same kind of configuration files. +* On each device running the agent, the device owner defines the list of files to be managed + (usually configuration files, but not necessarily). +* The configuration of this feature itself can be managed both locally and from the cloud, + meaning, the device owner can update the list of files to be managed from the cloud using the cloud mapper. +* The configuration files are managed according to their type, + a name chosen by the device owner to categorize each configuration. + By default, the full path of a configuration file on the device is used as its type. +* When files are downloaded from the tedge file transfer repository to the target device, + __these files are stored in the target path with a temporary name first__. + They are atomically renamed, only after a fully successful download to avoid breaking the system with half-downloaded files. +* When a downloaded file is copied to its target, the Unix user, group and mode are preserved. +* Once an update has been downloaded from the tedge file transfer repository to the target device, + __the agent publishes an operation status update message on the local %%te%% MQTT bus__. + The device software must subscribe to these messages if any action is required, + such as checking the content of the file, to pre-processing it, or restarting a daemon. + +In summary, the responsibilities of the agent regarding configuration management are: + * to define the list of files under configuration management, + * to notify the local MQTT bus when this list is updated, + * to upload these files to the tedge file transfer repository on demand, + * to download the files pushed from the tedge file transfer repository, + * to ensure that the target files are atomically updated after a successful download, + * to notify the device software when the configuration is updated. + +By contrast, the agent is not responsible for: + * checking that the uploaded files are well-formed, + * restarting the configured processes, + * establishing any direct connection to clouds. + +A user-specific component installed on the device + can implement more sophisticated configuration use-cases by: + * listening for configuration updates on the local %%te%% MQTT bus, + * restarting the appropriate processes when needed, + * declaring intermediate files as the managed files, + to have the opportunity to check or update their content + before moving them to the actual targets. + +## Configuration + +The configuration file used by the agent for configuration management is stored by default +under `/etc/tedge/plugins/tedge-configuration-plugin.toml`. + +This [TOML](https://toml.io/en/) file defines the list of files to be managed by the agent. +Each configuration file is defined by a record with: +* The full `path` to the file. +* An optional configuration `type`. If not provided, the `path` is used as `type`. + This `type` is used to declare the supported configuration file and then to trigger operations on that file. + All the configuration `type`s are declared as the supported config list to the local MQTT bus on startup + and on changes to the `plugins/tedge-configuration-plugin.toml` file. +* Optional unix file ownership: `user`, `group` and octal `mode`. + These are only used when a configuration file pushed via a `config_update` command doesn't exist on the device, + and a new one is created with these ownership parameters. + When a configuration file is already present on the device, + the agent preserves its existing ownership, ignoring these parameters. + +```toml title="file: /etc/tedge/plugins/tedge-configuration-plugin.toml" +files = [ + { path = '/etc/tedge/tedge.toml', type = 'tedge.toml' }, + { path = '/etc/tedge/mosquitto-conf/c8y-bridge.conf', type = 'c8y-bridge' }, + { path = '/etc/tedge/mosquitto-conf/tedge-mosquitto.conf', type = 'tedge-mosquitto' }, + { path = '/etc/mosquitto/mosquitto.conf', type = 'mosquitto', user = 'mosquitto', group = 'mosquitto', mode = 0o644 } +] +``` + +On start and whenever this file is updated, the agent sends +the supported config types declaration message with a retained flag +to the `config_snapshot` and `config_update` command topics +with the set of `type`s listed in that configuration file +(implicitly adding the `tedge-configuration-plugin` type also to that set). +The message can be observed over the MQTT bus of the %%te%% device. + +Given that `mqtt.topic_root` and `mqtt.device_topic_id` are set to `te` and `device/main//` for the main device, +the message to declare the supported configuration types is as follows. + +```sh te2mqtt formats=v1 +tedge mqtt pub -r 'te/device/main///cmd/config_snapshot' '{ + "types": ["tedge-configuration-plugin", "tedge.toml", "/etc/tedge/mosquitto-conf/c8y-bridge.conf", "/etc/tedge/mosquitto-conf/tedge-mosquitto.conf", "mosquitto"] +}' +``` + +```sh te2mqtt formats=v1 +tedge mqtt pub -r 'te/device/main///cmd/config_update' '{ + "types": ["tedge-configuration-plugin", "tedge.toml", "/etc/tedge/mosquitto-conf/c8y-bridge.conf", "/etc/tedge/mosquitto-conf/tedge-mosquitto.conf", "mosquitto"] +}' +``` + +:::note +* The file `/etc/tedge/plugins/tedge-configuration-plugin.toml` itself doesn't need to be listed. + This is implied, so the list can *always* be configured. + The `type` for this self configuration file is `tedge-configuration-plugin`. +* If the file `/etc/tedge/plugins/tedge-configuration-plugin.toml` + is not found, empty, ill-formed or not-readable + then only `tedge-configuration-plugin.toml` is declared as a supported configuration type. +::: + +The behavior of the agent is also controlled by the configuration of %%te%%: + +* `tedge config get mqtt.bind.address`: the address of the local MQTT bus. +* `tedge config get mqtt.bind.port`: the TCP port of the local MQTT bus. +* `tedge config get mqtt.topic_root`: the root of the [MQTT topic scheme](../mqtt-api.md) to publish and subscribe. +* `tedge config get mqtt.device_topic_id`: the identifier of the [MQTT topic scheme](../mqtt-api.md) to publish and subscribe. + +## Handling config snapshot commands + +During a config snapshot operation, the agent uploads a requested configuration file to the tedge file transfer repository. + +The agent subscribes to config snapshot commands on the `//cmd/config_snapshot/+` MQTT topics. +For example, it subscribes to the following topic for the main device. + +```sh te2mqtt formats=v1 +tedge mqtt sub 'te/device/main///cmd/config_snapshot/+' +``` + +To start a new config snapshot with the ID "1234" on the device named "example", a component has to publish the following message over MQTT: + +```sh te2mqtt formats=v1 +tedge mqtt pub -r 'te/device/main///cmd/config_snapshot/1234' '{ + "status": "init", + "tedgeUrl": "http://127.0.0.1:8000/tedge/file-transfer/example/config_snapshot/mosquitto-1234", + "type": "mosquitto" +}' +``` + +:::note +The `tedgeUrl` is an optional field. If the user does not provide the URL, the agent will create it, and the link will be added while the operation is being processed. +::: + +Upon receiving a configuration snapshot command, the agent performs the following actions: + 1. The agent uses the `type` information (`mosquitto`) to look up the target path from the `tedge-configuration-plugin.toml` file + and retrieves the requested configuration content from the corresponding `path`(`/etc/mosquitto/mosquitto.conf`). + 2. It then performs a `PUT` request to the `tedgeUrl` specified in the command's payload to upload the content. + +Throughout the process, the agent updates the command status via MQTT by publishing a retained message +to the same `//cmd/config_snapshot/` topic where the command is received. +The payload contains all the received data along with the `path` information. + +When the agent receives a new config snapshot command, it updates the status to `executing`. +After successfully uploading the file to the file transfer repository, +the agent updates the status to `successful`. +If any unexpected error occurs, the agent updates the status to `failed`, +providing a comprehensive `reason` for the failure. + +As a result, the operation status update message for the example above looks like this: + +```sh te2mqtt formats=v1 +tedge mqtt pub -r 'te/device/main///cmd/config_snapshot/1234' '{ + "status": "successful", + "tedgeUrl": "http://127.0.0.1:8000/tedge/file-transfer/example/config_snapshot/mosquitto-1234", + "type": "mosquitto", + "path": "/etc/mosquitto/mosquitto.conf" +}' +``` + +### Flow + +```mermaid +sequenceDiagram + participant Mapper + participant Child Agent + participant Main Agent + + Mapper->>Child Agent: tedge config_snapshot command (Status: init) + Child Agent->>Mapper: Status: executing + alt No error + Child Agent->>Main Agent: File upload [HTTP] + Main Agent-->> Child Agent: Status OK [HTTP] + Child Agent->>Mapper: Status: successful + else Any error occurs + Child Agent->>Mapper: Status: failed + end +``` + +## Handling config update commands + +During a config update operation, the agent downloads a requested configuration file +from the tedge file transfer repository and moves it to the target path. + +The agent subscribes to config update commands on the `//cmd/config_update/+` MQTT topics. +For example, it subscribes to the following topic for the main device. + +```sh te2mqtt formats=v1 +tedge mqtt sub 'te/device/main///cmd/config_update/+' +``` + +To start a new config update with the ID "1234" on the device named "example", a component has to publish the following message over MQTT: + +```sh te2mqtt formats=v1 +tedge mqtt pub -r 'te/device/main///cmd/config_update/1234' '{ + "status": "init", + "tedgeUrl": "http://127.0.0.1:8000/tedge/file-transfer/example/config_update/mosquitto-1234", + "remoteUrl": "http://www.my.url", + "type": "mosquitto" +}' +``` + +Upon receiving a configuration update command, the agent performs the following actions: + 1. It performs a `GET` request to the `tedgeUrl` specified in the command to retrieve the content. + 2. The agent then uses the `type` information (`mosquitto`) to to look up the target path from the `tedge-configuration-plugin.toml` file + and applies the new configuration content to the corresponding `path`(`/etc/mosquitto/mosquitto.conf`). + +Throughout the process, the agent updates the command status via MQTT +by publishing a retained message to the same `//cmd/config_update/` topic +where the command is received. +The payload contains all the received data along with the `path` information. + +Upon receiving a new config update command, the agent updates the status to `executing`. +After successfully completing all operation steps, the agent updates the status to `successful`. +If any unexpected error occurs, +the agent updates the status to `failed` along with a comprehensive `reason` for the failure. + +As a result, the operation status update message for the example above looks like this. + +```sh te2mqtt formats=v1 +tedge mqtt pub -r 'te/device/main///cmd/config_update/1234' '{ + "status": "successful", + "tedgeUrl": "http://127.0.0.1:8000/tedge/file-transfer/example/config_update/mosquitto-1234", + "remoteUrl": "http://www.my.url", + "type": "mosquitto", + "path": "/etc/mosquitto/mosquitto.conf" +}' +``` + +### Flow + +```mermaid +sequenceDiagram + participant Mapper + participant Child Agent + participant Main Agent + + Mapper->>Main Agent: Make a target config file ready + Mapper->>Child Agent: tedge config_update command (Status: init) + Child Agent->>Mapper: Status: executing + alt No error + Child Agent->>Main Agent: GET the config file [HTTP] + Main Agent-->>Child Agent: Return the content [HTTP] + Child Agent->>Child Agent: Apply the config + Child Agent->>Mapper: Status: successful + else Any error occurs + Child Agent->>Mapper: Status: failed + end +``` diff --git a/versioned_docs/version-1.1.0/references/agent/tedge-log-management.md b/versioned_docs/version-1.1.0/references/agent/tedge-log-management.md new file mode 100644 index 0000000..878d3d3 --- /dev/null +++ b/versioned_docs/version-1.1.0/references/agent/tedge-log-management.md @@ -0,0 +1,130 @@ +--- +title: Log File Management +tags: [Reference, Log Files] +sidebar_position: 5 +description: Details of the API to manage log files on a device +--- + +%%te%% implements an operation to upload log files from the device to the cloud. + +* The log file management feature on the main or a child device is provided by the __tedge-agent__ running on that device. +* The device owner can define the list of log files using the `tedge-log-plugin.toml` configuration file of the target device. +* Each entry in this file contains a log `type` and a `path` pattern, + where the `type` is used to represent the logical group of log files matching the `path` pattern. +* Upon receiving a log file upload command for a given `type`, + the log files for that type are retrieved using the `path` pattern defined in this `tedge-log-plugin.toml`, + matched against the requested time range, search text and maximum line count. +* The agent uploads the requested log file to the tedge file transfer repository. + Its url is given by the received log upload command. +* The list of managed log files in `tedge-log-plugin.toml` can be updated locally or from cloud, for instance, by using the configuration management feature. +* However, the agent provides no direct connection to clouds, which is the responsibility of the cloud mapper. + +## Configuration + +Configuration of the log upload operation is stored by default under `/etc/tedge/plugins/tedge-log-plugin.toml`. +If it does not exist on startup, the agent creates the file with example contents. + +This [TOML](https://toml.io/en/) file defines the list of log files that should be managed by the agent running on the device. +The paths to these files can be represented using [glob](https://en.wikipedia.org/wiki/Glob_(programming)) patterns. +The `type` given to these paths are used as the log type associated to a log path. + +```toml title="file: /etc/tedge/plugins/tedge-log-plugin.toml" +files = [ + { type = "mosquitto", path = '/var/log/mosquitto/mosquitto.log' }, + { type = "software-management", path = '/var/log/tedge/agent/software-*' }, + { type = "c8y_CustomOperation", path = '/var/log/tedge/agent/c8y_CustomOperation/*' } +] +``` + +The agent parses this configuration file on startup for all the `type` values specified, +and sends the supported log types message to the MQTT local broker on the `//cmd/log_upload` topic with a retained flag. + +Given that `root.topic` and `device.topic` are set to `te` and `device/main//` for the main device, +the message to declare the supported log types is as follows. + +```sh te2mqtt formats=v1 +tedge mqtt pub -r 'te/device/main///cmd/log_upload' '{ + "types" : [ "mosquitto", "software-management", "c8y_CustomOperation" ] +}' +``` + +The agent continuously watches this configuration file for any changes and resends the JSON message with the `type`s in this file, +whenever it is updated. + +:::note +If the file `/etc/tedge/plugins/tedge-log-plugin.toml` is ill-formed or cannot be read, +then a JSON message with an empty array for the `types` field is sent, indicating no log files are tracked. +::: + +## Handling log upload commands + +The agent subscribes to log upload commands on the [`//cmd/log_upload/+` MQTT topic](../mqtt-api.md). +For example, it subscribes to the following topic for the main device. + +```sh te2mqtt formats=v1 +tedge mqtt sub 'te/device/main///cmd/log_upload/+' +``` + +A new log file upload command with the ID "1234" is published for the device named "example" by another component as below. + +```sh te2mqtt formats=v1 +tedge mqtt pub -r 'te/device/main///cmd/log_upload/1234' '{ + "status": "init", + "tedgeUrl": "http://127.0.0.1:8000/tedge/file-transfer/example/log_upload/mosquitto-1234", + "type": "mosquitto", + "dateFrom": "2013-06-22T17:03:14.000+02:00", + "dateTo": "2013-06-23T18:03:14.000+02:00", + "searchText": "ERROR", + "lines": 1000 +}' +``` + +The agent then checks the `tedge-log-plugin.toml` file for the log `type` in the incoming message (`mosquitto`), +retrieves the log files using the `path` glob pattern provided in the configuration file for log upload, +including only the ones modified within the date range(`2013-06-22T17:03:14.000+02:00` to `2013-06-23T18:03:14.000+02:00`), +with the content filtered by the search text(`ERROR`) and the maximum line count(`1000`). + +This filtered content is then uploaded to the URL received in the command as `tedgeUrl` via an HTTP PUT request. + +During the process, the agent updates the command status via MQTT +by publishing a retained message to the same `//cmd/log_upload/` topic, +where the command is received. + +On the reception of a new log file upload command, the agent updates the status to `executing`. +After successfully uploading the file to the file transfer repository, the agent updates the status to `successful`. +If any unexpected error occurs, the agent updates the status to `failed` with a `reason`. + +Thus, the operation status update message for the above example looks like below. + +```sh te2mqtt formats=v1 +tedge mqtt pub -r 'te/device/main///cmd/log_upload/1234' '{ + "status": "failed", + "reason": "The target log file for 'mosquitto' does not exist.", + "tedgeUrl": "http://127.0.0.1:8000/tedge/file-transfer/example/log_upload/mosquitto-1234", + "type": "mosquitto", + "dateFrom": "2013-06-22T17:03:14.000+02:00", + "dateTo": "2013-06-22T18:03:14.000+02:00", + "searchText": "ERROR", + "lines": 1000 +}' +``` + +### Flow + +```mermaid +sequenceDiagram + participant Mapper + participant Child Agent + participant Main Agent + + Mapper->>Child Agent: tedge log_upload command (Status: init) + Child Agent->>Mapper: Status: executing + alt No error + Child Agent->>Child Agent: Extract log + Child Agent->>Main Agent: File upload [HTTP] + Main Agent-->>Child Agent: Status OK [HTTP] + Child Agent->>Mapper: Status: successful + else Any error occurs + Child Agent->>Mapper: Status: failed + end +``` diff --git a/versioned_docs/version-1.1.0/references/cli/index.md b/versioned_docs/version-1.1.0/references/cli/index.md new file mode 100644 index 0000000..4a8842a --- /dev/null +++ b/versioned_docs/version-1.1.0/references/cli/index.md @@ -0,0 +1,30 @@ +--- +title: "The tedge command" +tags: [Reference, CLI] +sidebar_position: 4 +--- + +# The tedge command + +```sh title="tedge" +tedge is the cli tool for thin-edge.io + +USAGE: + tedge [OPTIONS] [SUBCOMMAND] + +OPTIONS: + --config-dir [default: /etc/tedge] + -h, --help Print help information + --init Initialize the tedge + -V, --version Print version information + +SUBCOMMANDS: + cert Create and manage device certificate + config Configure Thin Edge + connect Connect to connector provider + disconnect Remove bridge connection for a provider + help Print this message or the help of the given subcommand(s) + init Initialize Thin Edge + mqtt Publish a message on a topic and subscribe a topic + reconnect Reconnect command, calls disconnect followed by connect +``` diff --git a/versioned_docs/version-1.1.0/references/cli/tedge-cert.md b/versioned_docs/version-1.1.0/references/cli/tedge-cert.md new file mode 100644 index 0000000..c18bc89 --- /dev/null +++ b/versioned_docs/version-1.1.0/references/cli/tedge-cert.md @@ -0,0 +1,97 @@ +--- +title: "tedge cert" +tags: [Reference, CLI] +sidebar_position: 1 +--- + +# The tedge cert command + +```sh title="tedge cert" +tedge-cert +Create and manage device certificate + +USAGE: + tedge cert + +OPTIONS: + -h, --help Print help information + +SUBCOMMANDS: + create Create a self-signed device certificate + help Print this message or the help of the given subcommand(s) + remove Remove the device certificate + show Show the device certificate, if any + upload Upload root certificate +``` + +## Create + +```sh title="tedge cert create" +tedge-cert-create +Create a self-signed device certificate + +USAGE: + tedge cert create --device-id + +OPTIONS: + --device-id The device identifier to be used as the common name for the certificate + -h, --help Print help information +``` + +## Create-csr + +```sh title="tedge cert create-csr" +tedge-cert-create-csr +Create certificate signing request + +Usage: tedge cert create-csr [OPTIONS] + +Options: + --device-id The device identifier to be used as the common name for the certificate + --output-path Path where a Certificate signing request will be stored + --config-dir [default: /etc/tedge] + -h, --help Print help +``` + +## Show + +```sh title="tedge cert show" +tedge-cert-show +Show the device certificate, if any + +USAGE: + tedge cert show + +OPTIONS: + -h, --help Print help information +``` + +## Remove + +```sh title="tedge cert remove" +tedge-cert-remove +Remove the device certificate + +USAGE: + tedge cert remove + +OPTIONS: + -h, --help Print help information +``` + +## Upload + +```sh title="tedge cert upload" +tedge-cert-upload +Upload root certificate + +USAGE: + tedge cert upload + +OPTIONS: + -h, --help Print help information + +SUBCOMMANDS: + c8y Upload root certificate to Cumulocity + help Print this message or the help of the given subcommand(s) +``` diff --git a/versioned_docs/version-1.1.0/references/cli/tedge-config.md b/versioned_docs/version-1.1.0/references/cli/tedge-config.md new file mode 100644 index 0000000..7652461 --- /dev/null +++ b/versioned_docs/version-1.1.0/references/cli/tedge-config.md @@ -0,0 +1,89 @@ +--- +title: "tedge config" +tags: [Reference, CLI] +sidebar_position: 2 +--- + +# The tedge config command + +```sh title="tedge config" +tedge-config +Configure Thin Edge + +USAGE: + tedge config + +OPTIONS: + -h, --help Print help information + +SUBCOMMANDS: + get Get the value of the provided configuration key + help Print this message or the help of the given subcommand(s) + list Print the configuration keys and their values + set Set or update the provided configuration key with the given value + unset Unset the provided configuration key +``` + +## Get + +```sh title="tedge config get" +tedge-config-get +Get the value of the provided configuration key + +USAGE: + tedge config get + +ARGS: + Configuration key. Run `tedge config list --doc` for available keys + +OPTIONS: + -h, --help Print help information +``` + +## Set + +```sh title="tedge config set" +tedge-config-set +Set or update the provided configuration key with the given value + +USAGE: + tedge config set + +ARGS: + Configuration key. Run `tedge config list --doc` for available keys + Configuration value + +OPTIONS: + -h, --help Print help information +``` + +## List + +```sh title="tedge config list" +tedge-config-list +Print the configuration keys and their values + +USAGE: + tedge config list [OPTIONS] + +OPTIONS: + --all Prints all the configuration keys, even those without a configured value + --doc Prints all keys and descriptions with example values + -h, --help Print help information +``` + +## Unset + +```sh title="tedge config unset" +tedge-config-unset +Unset the provided configuration key + +USAGE: + tedge config unset + +ARGS: + Configuration key. Run `tedge config list --doc` for available keys + +OPTIONS: + -h, --help Print help information +``` diff --git a/versioned_docs/version-1.1.0/references/cli/tedge-connect.md b/versioned_docs/version-1.1.0/references/cli/tedge-connect.md new file mode 100644 index 0000000..8b35419 --- /dev/null +++ b/versioned_docs/version-1.1.0/references/cli/tedge-connect.md @@ -0,0 +1,90 @@ +--- +title: "tedge connect" +tags: [Reference, CLI] +sidebar_position: 3 +--- + +# The tedge connect command + +```sh title="tedge connect" +tedge-connect +Connect to connector provider + +USAGE: + tedge connect + +OPTIONS: + -h, --help Print help information + +SUBCOMMANDS: + aws Create connection to AWS + az Create connection to Azure + c8y Create connection to Cumulocity + help Print this message or the help of the given subcommand(s) +``` + +## AWS + +```sh title="tedge connect aws" +tedge-connect-aws +Create connection to AWS + +The command will create config and start edge relay from the device to AWS instance + +USAGE: + tedge connect aws [OPTIONS] + +OPTIONS: + -h, --help + Print help information + + --test + Test connection to AWS + + --offline + Ignore connection registration and connection check +``` + +## Azure + +```sh title="tedge connect az" +tedge-connect-az +Create connection to Azure + +The command will create config and start edge relay from the device to az instance + +USAGE: + tedge connect az [OPTIONS] + +OPTIONS: + -h, --help + Print help information + + --test + Test connection to Azure + + --offline + Ignore connection registration and connection check +``` + +## Cumulocity + +```sh title="tedge connect c8y" +tedge-connect-c8y +Create connection to Cumulocity + +The command will create config and start edge relay from the device to c8y instance + +USAGE: + tedge connect c8y [OPTIONS] + +OPTIONS: + -h, --help + Print help information + + --test + Test connection to Cumulocity + + --offline + Ignore connection registration and connection check +``` diff --git a/versioned_docs/version-1.1.0/references/cli/tedge-disconnect.md b/versioned_docs/version-1.1.0/references/cli/tedge-disconnect.md new file mode 100644 index 0000000..c9a74b2 --- /dev/null +++ b/versioned_docs/version-1.1.0/references/cli/tedge-disconnect.md @@ -0,0 +1,63 @@ +--- +title: "tedge disconnect" +tags: [Reference, CLI] +sidebar_position: 4 +--- + +# The tedge disconnect command + +```sh title="tedge disconnect" +tedge-disconnect +Remove bridge connection for a provider + +USAGE: + tedge disconnect + +OPTIONS: + -h, --help Print help information + +SUBCOMMANDS: + aws Remove bridge connection to AWS + az Remove bridge connection to Azure + c8y Remove bridge connection to Cumulocity + help Print this message or the help of the given subcommand(s) +``` + +## AWS + +```sh title="tedge disconnect aws" +tedge-disconnect-aws +Remove bridge connection to AWS + +USAGE: + tedge disconnect aws + +OPTIONS: + -h, --help Print help information +``` + +## Azure + +```sh title="tedge disconnect az" +tedge-disconnect-az +Remove bridge connection to Azure + +USAGE: + tedge disconnect az + +OPTIONS: + -h, --help Print help information +``` + +## Cumulocity + +```sh title="tedge disconnect c8y" +tedge-disconnect-c8y +Remove bridge connection to Cumulocity + +USAGE: + tedge disconnect c8y + +OPTIONS: + -h, --help Print help information +``` diff --git a/versioned_docs/version-1.1.0/references/cli/tedge-mqtt.md b/versioned_docs/version-1.1.0/references/cli/tedge-mqtt.md new file mode 100644 index 0000000..cab2526 --- /dev/null +++ b/versioned_docs/version-1.1.0/references/cli/tedge-mqtt.md @@ -0,0 +1,60 @@ +--- +title: "tedge mqtt" +tags: [Reference, CLI] +sidebar_position: 5 +--- + +# The tedge mqtt command + +```sh title="tedge mqtt" +tedge-mqtt +Publish a message on a topic and subscribe a topic + +USAGE: + tedge mqtt + +OPTIONS: + -h, --help Print help information + +SUBCOMMANDS: + help Print this message or the help of the given subcommand(s) + pub Publish a MQTT message on a topic + sub Subscribe a MQTT topic +``` + +## Pub + +```sh title="tedge mqtt pub" +tedge-mqtt-pub +Publish a MQTT message on a topic + +USAGE: + tedge mqtt pub [OPTIONS] + +ARGS: + Topic to publish + Message to publish + +OPTIONS: + -h, --help Print help information + -q, --qos QoS level (0, 1, 2) [default: 0] + -r, --retain Retain flag +``` + +## Sub + +```sh title="tedge mqtt sub" +tedge-mqtt-sub +Subscribe a MQTT topic + +USAGE: + tedge mqtt sub [OPTIONS] + +ARGS: + Topic to subscribe to + +OPTIONS: + -h, --help Print help information + --no-topic Avoid printing the message topics on the console + -q, --qos QoS level (0, 1, 2) [default: 0] +``` diff --git a/versioned_docs/version-1.1.0/references/configuration-files.md b/versioned_docs/version-1.1.0/references/configuration-files.md new file mode 100644 index 0000000..565da70 --- /dev/null +++ b/versioned_docs/version-1.1.0/references/configuration-files.md @@ -0,0 +1,183 @@ +--- +title: Configuration Files +tags: [Reference, Configuration] +sidebar_position: 5 +description: Using a custom configuration path +--- + +%%te%% requires config files for its operation. The `tedge init` command is used to create +the base directory and other directories inside the base directory with appropriate user and permissions. +The `tedge-mapper` and `tedge-agent` will create the config +directories/files inside the base directory that are required for their operation on startup. + +By default, the config files are created in `/etc/tedge` directory. To create the config files in +a custom base directory one has to use `--config-dir ` option. + +## Creating config files + +The config files are created using `tedge init` as below. + +```sh +sudo tedge init +``` + +All the directories will be created in the `/etc/tedge` directory. The directories layout looks as below. + +```sh +ls -l /etc/tedge +``` + +```text title="Output" +total 16 +drwxrwxr-x 2 mosquitto mosquitto 4096 Jun 10 14:49 device-certs +drwxrwxr-x 2 mosquitto mosquitto 4096 Jun 10 14:49 mosquitto-conf +drwxrwxr-x 2 tedge tedge 4096 Jun 10 14:49 operations +drwxrwxr-x 2 tedge tedge 4096 Jun 10 14:49 plugins +``` + +Use the below command to create the config directories in a custom directory. + +```sh +sudo tedge --config-dir /global/path/to/config/dir init +``` + +Now all the config directories will be created inside the `/global/path/to/config/dir` directory. + +The directories and files that are required by the `tedge-mapper` are created as below. + +```sh +sudo tedge-mapper c8y +``` + +```sh +ls -l /etc/tedge/operations/c8y +``` + +```text title="Output" +total 0 +-rw-r--r-- 1 tedge tedge 0 Jun 14 14:37 c8y_Restart +-rw-r--r-- 1 tedge tedge 0 Jun 14 14:37 c8y_SoftwareUpdate +``` + +To create these directories in a custom directory, use `--config-dir` option as below. + +```sh +sudo tedge-mapper --config-dir /global/path/to/config/dir c8y +``` + +The directories and files that are required by the `tedge-agent` are created on startup as follows: + +```sh +ls -l /etc/tedge/.agent +``` + +```text title="Output" +-rw-r--r-- 1 tedge tedge 0 Jun 15 11:51 /etc/tedge/.agent/current-operation +``` + +```sh +ls -l /var/tedge/ +``` + +```text title="Output" +drwxr-xr-x 2 tedge tedge 0 Jun 15 11:51 /var/tedge/file-transfer +``` + +To create these directories and files in a custom directory, use the `--config-dir` option as below as below. + +```sh +sudo tedge-agent --config-dir /global/path/to/config/dir +``` + +## Manage the configuration parameters + +The configuration parameters can be set/unset/list in a config file as below + +For example, the config parameter can be set as below. + +```sh +sudo tedge config set c8y.url your.cumulocity.io +``` + +Now the configuration will be added into `/etc/tedge/tedge.toml` + +Use the below command to set/unset/list configuration parameters in a config file that is present +in a custom directory. + +```sh +sudo tedge --config-dir /global/path/to/config/dir config set c8y.url your.cumulocity.io +``` + +Now the config will be set in `/global/path/to/config/dir/tedge/tedge.toml` + +## Manage the certificate + +To create/remove/upload the certificate, one can use the below command. + +```sh +sudo tedge cert create --device-id thinedge +``` + +```text title="Output" +Certificate was successfully created +``` + +Find the certificates that are created as below. + +```sh +ls -l /etc/tedge/device-certs/ +``` + +```text title="Output" +total 8 +-r--r--r-- 1 mosquitto mosquitto 638 Jun 14 14:38 tedge-certificate.pem +-r-------- 1 mosquitto mosquitto 246 Jun 14 14:38 tedge-private-key.pem +``` + +Use the below command to create/remove/upload the certificate. + +```sh +sudo tedge --config-dir /global/path/to/config/dir cert create --device-id thinedge +``` + +Find the certificates that are created as below. + +```sh +ls -l /global/path/to/config/dir/tedge/device-certs/ +``` + +```text title="Output" +total 8 +-r--r--r-- 1 mosquitto mosquitto 638 Jun 14 14:38 tedge-certificate.pem +-r-------- 1 mosquitto mosquitto 246 Jun 14 14:38 tedge-private-key.pem +``` + +## Connecting to the cloud + +Use the`tedge connect c8y/az/aws` command to connect to the cloud using the default configuration files +that are present in `/etc/tedge`. + +To connect to the cloud with config files that are present in a custom location use +the `tedge connect --config-dir c8y/az/aws` option. + +This is a two step process. + +### Step 1: Update the mosquitto.conf + +Since the bridge configuration files for Cumulocity IoT, Azure IoT Hub or AWS IoT will be created in a directory given through `--config-dir`, +the path to the bridge configuration files (tedge-mosquitto.conf, c8y/az/aws-bridge.conf) must be found by `mosquitto`. +So, the below line has to be added to your `mosquitto.conf` file manually. + +```text title="file: /etc/mosquitto/mosquitto.conf" +include_dir /global/path/to/config/dir/tedge/mosquitto-conf +``` + +### Step 2: tedge connect using a custom config directory + +Use the below command to connect to `Cumulocity IoT, Azure IoT Hub or AWS IoT` cloud using `--config-dir` + +```sh +sudo tedge --config-dir /global/path/to/config/dir connect c8y/az/aws +``` + +Here the `path/to/config/dir` is the directory where the configuration files are present. diff --git a/versioned_docs/version-1.1.0/references/cumulocity-proxy.md b/versioned_docs/version-1.1.0/references/cumulocity-proxy.md new file mode 100644 index 0000000..2039248 --- /dev/null +++ b/versioned_docs/version-1.1.0/references/cumulocity-proxy.md @@ -0,0 +1,47 @@ +--- +title: Cumulocity Proxy +tags: [ Reference, HTTP, Cumulocity ] +sidebar_position: 12 +description: Using the Cumulocity IoT Proxy for full access to Cumulocity API +--- + +The `tedge-mapper` (when running in `c8y` mode) hosts a proxy server to access the Cumulocity HTTP API from the +%%te%% device. +It automatically handles authenticating with a JWT, avoiding the need for clients to support MQTT to retrieve this +information. + +The Cumulocity HTTP API can be accessed at `http://{host}:{port}/c8y/{c8y-endpoint}`. Configuration settings +`c8y.proxy.client.host` and `c8y.proxy.client.port` are used to configure `{host}` and `{port}` parts of the base URL +which will be used by %%te%% components to make requests to the C8y Proxy. `c8y.proxy.bind.address` and +`c8y.proxy.bind.port` are read by `tedge-mapper-c8y` and used as bind address and port for the Cumulocity HTTP proxy. In +both `client` and `bind`, `127.0.0.1` and `8001` are used as defaults for the address and port, respectively. + +For example, you can access the current tenant information +at [http://127.0.0.1:8001/c8y/tenant/currentTenant](http://127.0.0.1:8001/c8y/tenant/currentTenant) +from the machine running `tedge-mapper`. +The server supports all public REST APIs of Cumulocity, and all possible request methods +(e.g. `HEAD`/`GET`/`PUT`/`PATCH`/`DELETE`). +There is no need to provide an `Authorization` header (or any other authentication method) when accessing the API. +If an `Authorization` header is provided, this will be used to authenticate the request instead of the device JWT. + +## HTTPS and authenticated access +By default, the service is unauthenticated and does not support incoming HTTPS connections +(when the request is forwarded to Cumulocity, however, this will use HTTPS). +HTTPS can be enabled by setting `c8y.proxy.cert_path` and `c8y.proxy.key_path`. +If the certificates are configured, the mapper will automatically host the proxy via HTTPS, and redirect any +HTTP requests to the equivalent HTTPS URL. +If HTTPS is enabled, the configured certificate should be installed in the OS trust store for any connected agents +in order for them to trust the connection to the mapper. + +Once HTTPS is enabled for the mapper, certificate-based authentication can also be enabled. +The directory containing the certificates that the mapper will trust can be configured using `c8y.proxy.ca_path`, +and the agent can be configured to use a trusted certificate using the `http.client.auth.cert_file` and `http.client.auth.key_file` +settings. + +## Possible errors returned by the proxy +Due to the underlying JWT handling in Cumulocity, requests to the proxy API are occasionally spuriously rejected with +a `401 Not Authorized` status code. +The proxy server currently forwards this response directly to the client, as well as all other errors responses from +Cumulocity. +If there is an error connecting to Cumulocity to make the request, a plain text response with the status +code `502 Bad Gateway` will be returned. diff --git a/versioned_docs/version-1.1.0/references/file-transfer-service.md b/versioned_docs/version-1.1.0/references/file-transfer-service.md new file mode 100644 index 0000000..46ef008 --- /dev/null +++ b/versioned_docs/version-1.1.0/references/file-transfer-service.md @@ -0,0 +1,46 @@ +--- +title: File Transfer Service +tags: [Reference, HTTP] +sidebar_position: 6 +descrioption: Interacting with the %%te%% file transfer service +--- + +The `tedge-agent` hosts a binary repository for child devices and other plugins/extensions to exchange binary files between them. +This repository is meant to be used as a temporary storage for exchanging files, and not for storing items permanently, +as the storage on %%te%% devices are typically very limited. + +Files can be uploaded, downloaded and deleted from this repository via the following HTTP endpoints: + +|Type|Method|Endpoint| +|----|------|--------| +|Upload|PUT|`http://{fts-address}:8000/tedge/file-transfer/{path}/{to}/{resource}`| +|Download|GET|`http://{fts-address}:8000/tedge/file-transfer/{path}/{to}/{resource}`| +|Delete|DELETE|`http://{fts-address}:8000/tedge/file-transfer/{path}/{to}/{resource}`| + +The `fts-address` is derived from `http.client.host` config setting with a default value of `127.0.0.1`. + +The files uploaded to this repository are stored at `/var/tedge/file-transfer` directory. +The `{path}/{to}/{resource}` specified in the URL is replicated under this directory. + +For example, a file uploaded to `http://{fts-address}/tedge/file-transfer/config_update/mosquitto/mosquitto.conf` +is stored at `/var/tedge/file-transfer/config_update/mosquitto/mosquitto.conf`. + +An existing file at a given path is replaced on subsequent uploads using the same URL path. +Unique paths must be used in the URL path to avoid such overwrites. + +All uploaded files are preserved until they are explicitly deleted with the DELETE API. +To avoid exhaustion of storage space on the %%te%% device, +users must be diligent to delete any stored files as soon as their purpose is served. + +## HTTPS and authenticated access +By default, the service is unauthenticated and does not support HTTPS connections. +HTTPS can be enabled by setting `http.cert_path` and `http.key_path`. +If the certificates are configured, the agent will automatically host the service via HTTPS, and redirect any +HTTP requests to the equivalent HTTPS URL. +If HTTPS is enabled, the configured certificate should be installed in the OS trust store for any connected agents +in order for them to trust the connection to the mapper. + +Once HTTPS is enabled for the file-transfer service, certificate-based authentication can also be enabled. +The directory containing the certificates that the agent will trust can be configured using `http.ca_path`, +and the mapper as well as the child device agents can be configured to use a trusted certificate using the +`http.client.auth.cert_file` and `http.client.auth.key_file` settings. diff --git a/versioned_docs/version-1.1.0/references/index.md b/versioned_docs/version-1.1.0/references/index.md new file mode 100644 index 0000000..9919fba --- /dev/null +++ b/versioned_docs/version-1.1.0/references/index.md @@ -0,0 +1,15 @@ +--- +title: References +tags: [Reference] +sidebar_position: 6 +--- + +import DocCardList from '@theme/DocCardList'; + +The reference guides include technical documentation about the following areas: + +- Command Line Interface +- Configuration Files +- MQTT API + + \ No newline at end of file diff --git a/versioned_docs/version-1.1.0/references/init-system-configuration.md b/versioned_docs/version-1.1.0/references/init-system-configuration.md new file mode 100644 index 0000000..8b6388a --- /dev/null +++ b/versioned_docs/version-1.1.0/references/init-system-configuration.md @@ -0,0 +1,63 @@ +--- +title: Init System Configuration +tags: [Reference, Unix, Init, Services] +sidebar_position: 6 +description: Configuring %%te%% to work with Linux init systems +--- + +To support multiple init systems and service managers, `tedge` requires the `/etc/tedge/system.toml` file. +The file contains configurations about the init system and the supported actions. + +The format of the file is: + +```toml title="file: /etc/tedge/system.toml" +[init] +name = "systemd" +is_available = ["/bin/systemctl", "--version"] +restart = ["/bin/systemctl", "restart", "{}"] +stop = ["/bin/systemctl", "stop", "{}"] +start = ["/bin/systemctl", "start", "{}"] +enable = ["/bin/systemctl", "enable", "{}"] +disable = ["/bin/systemctl", "disable", "{}"] +is_active = ["/bin/systemctl", "is-active", "{}"] +``` + +:::info +For security reasons, the `system.toml` file should not be writable by non-root users. The permissions on the file can be set using the following command: + +```sh +sudo chmod 644 /etc/tedge/system.toml +``` +::: + +## Placeholder + +`{}` will be replaced by a service name (`mosquitto`, `tedge-mapper-c8y`, `tedge-mapper-az`, `tedge-mapper-aws`, etc.). +For example, + +```toml +restart = ["/bin/systemctl", "restart", "{}"] +``` + +will be interpreted as + +```sh +/bin/systemctl restart mosquitto +``` + +## Keys + +| Property | Description | +|----------------|------------------------------------------------------------------------------------------------------| +| `name` | An identifier of the init system. It is used in the output of `tedge connect` and `tedge disconnect` | +| `is_available` | The command to check if the init is available on your system | +| `restart` | The command to restart a service by the init system | +| `stop` | The command to stop a service by the init system | +| `start` | The command to start a service by the init system | +| `enable` | The command to enable a service by the init system | +| `disable` | The command to disable a service by the init system | +| `is_active` | The command to check if the service is running by the init system | + +## Default settings + +If the `system.toml` file does not exist, then %%te%% will assume that you are using Systemd, and use `/bin/systemctl` to control the services. diff --git a/versioned_docs/version-1.1.0/references/mappers/c8y-mapper.md b/versioned_docs/version-1.1.0/references/mappers/c8y-mapper.md new file mode 100644 index 0000000..dde6b96 --- /dev/null +++ b/versioned_docs/version-1.1.0/references/mappers/c8y-mapper.md @@ -0,0 +1,1366 @@ +--- +title: Cumulocity Mapper +tags: [Reference, Mappers, Cloud] +sidebar_position: 1 +--- + +The Cumulocity mapper, referred to as `c8y-mapper` in the rest of this document, +maps data in [%%te%% format](../mqtt-api.md) into their equivalent [Cumulocity format](https://cumulocity.com/guides/reference/smartrest-two/#smartrest-two). + + +## Registration + +Cumulocity keeps the record of all the registered devices and their associated metadata in its inventory. +The `c8y-mapper` creates and maintains inventory entries for all the devices and services registered with %%te%%. +The mapper subscribes to the following topics to get a list of all registered devices and services: + +```sh +mosquitto_sub -t 'te/+' -t 'te/+/+' -t 'te/+/+/+' -t 'te/+/+/+/+' +``` + +The registration messages received for child devices and services are mapped as follows: + +### Child device + +
+ +**%%te%% (input)** + +```text title="Topic" +te/device/child01// +``` + +```json5 title="Payload" +{ + "@type": "child-device", + "name": "child01", + "type": "SmartHomeHub" +} +``` + +
+ +
+ +**Cumulocity IoT (output)** + +```text title="Topic" +c8y/s/us +``` + +```text title="Payload" +101,:device:child01,child01,SmartHomeHub +``` + +
+ +Where the `` is added as the prefix to the external id to avoid id clashes +with devices using the same name in other tedge deployments connected to the same tenant. + +### Child device with explicit id + +
+ +**%%te%% (input)** + +```text title="Topic" +te/device/child01// +``` + +```json5 title="Payload" +{ + "@type": "child-device", + "@id": "child01", + "name": "child01", + "type": "SmartHomeHub" +} +``` + +
+ +
+ +**Cumulocity IoT (output)** + +```text title="Topic" +c8y/s/us +``` + +```text title="Payload" +101,child01,child01,SmartHomeHub +``` + +
+ +Where the provided `@id` is directly used as the external id. + +### Nested child device + +
+ +**%%te%% (input)** + +```text title="Topic" +te/device/nested_child01// +``` + +```json5 title="Payload" +{ + "@type": "child-device", + "@parent": "device/child01//", + "name": "nested_child01", + "type": "BatterySensor" +} +``` + +
+ + +
+ +**Cumulocity IoT (output)** + +```text title="Topic" +c8y/s/us/:device:child01 +``` + +```text title="Payload" +101,:device:nested_child01,nested_child01,BatterySensor +``` + +
+ +### Main device service + +
+ +**%%te%% (input)** + +```text title="Topic" +te/device/main/service/nodered +``` + +```json5 title="Payload" +{ + "@type": "service", + "name": "Node-Red", + "type": "systemd" +} +``` + +
+ + +
+ +**Cumulocity IoT (output)** + +```text title="Topic" +c8y/s/us +``` + +```text title="Payload" +102,:device:main:service:nodered,systemd,Node-Red,up +``` + +
+ +### Child device service + +
+ +**%%te%% (input)** + +```text title="Topic" +te/device/child01/service/nodered +``` + +```json5 title="Payload" +{ + "@type": "service", + "name": "Node-Red", + "type": "systemd" +} +``` + +
+ + +
+ +**Cumulocity IoT (output)** + +```text title="Topic" +c8y/s/us/:device:child01 +``` + +```text title="Payload" +102,:device:child01:service:nodered,systemd,Node-Red,up +``` + +
+ +Where the unique external IDs to be used in the cloud are derived from the entity identifier subtopics, +replacing the `/` characters with `:`. + +:::note +The main device is registered with the cloud via the `tedge connect c8y` command execution +and hence there is no mapping done for main device registration messages. +Inventory data updates for the main device are handled differently. +::: + +## Auto Registration of an entity + +Before any data messages from an entity can be processed, the entity has to be registered first. +The entity can be registered either explicitly or implicitly (Auto registration). + +With auto-registration, an entity does not need to explicitly send a registration message, +and the registration is done automatically by the mapper on receipt of the first message from that entity. +But, auto-registration is allowed only when the entity follows the default topic scheme: `te/device//service/`. + +For example, sending a measurement message to `te/device/child1///m/temperature` will result in the auto-registration of the device entity with topic id: `device/child1//` and the auto-generated external id: `:device:child1`, derived from the topic id. +Similarly, a measurement message on `te/device/child1/service/my-service/m/temperature` results in the auto-registration of both +the device entity: `device/child1//` and the service entity: `device/child1/service/my-service` with their respective auto-generated external IDs, in that order. + +**Pros:** +* No need to have a separate registration message for an entity. + This would be ideal for simple devices where programming an additional registration logic is not possible ( e.g: simple sensors that can only generate telemetry messages). + +**Cons:** +* Auto-registration of all entities is not possible in complex deployments with nested/hierarchical devices, as the parent of a nested child device can't be identified solely from its topic id (e.g: `te/device/nested-child//`). +The parent information must be provided explicitly using a registration message so that any nested child devices are not wrongly auto-registered as immediate child devices of the main device. +* Auto-registration results in the auto-generation of the device external id as well. If the user wants more control over it, then an explicit registration must be done. + +Auto-registration can be enabled/disabled based on the complexity of the deployment. +For simpler deployments with just a single level child devices, following the default topic scheme, +auto-registration can be kept enabled. +For any complex deployments requiring external id customizations or with nested child devices, +auto-registration **must be disabled**. + +Auto-registration can be disabled using the following `tedge config` command: +```sh +sudo tedge config set c8y.entity_store.auto_register false +``` + +Auto-registration is enabled, by default. +When the auto registration is disabled, and if the device is not explicitly registered, +then the c8y-mapper will ignore all the data messages received from that device, +logging that error message on the `te/errors` topic indicating that the entity is not registered. + + +## Telemetry + +Telemetry data types like measurements, events and alarms are mapped to their respective equivalents in Cumulocity as follows: + +### Measurement + +
+ +**%%te%% (input)** + +```text title="Topic" +te/device/main///m/environment +``` + +```json5 title="Payload" +{ + "temperature": 23.4 +} +``` + +
+ +
+ +**Cumulocity IoT (output)** + +```text title="Topic" +c8y/measurement/measurements/create +``` + +```json5 title="Payload" +{ + "type": "environment", + "time": "2021-04-22T17:05:26.958340390+00:00", + "temperature": { + "temperature": { + "value": 23 + } + } +} +``` + +
+ + +#### Measurement without type + +
+ +**%%te%% (input)** + +```text title="Topic" +te/device/main///m/ +``` + +```json5 title="Payload" +{ + "temperature": 23.4 +} +``` + +
+ +
+ +**Cumulocity IoT (output)** + +```text title="Topic" +c8y/measurement/measurements/create +``` + +```json5 title="Payload" +{ + "type": "ThinEdgeMeasurement", + "time": "2021-04-22T17:05:26.958340390+00:00", + "temperature": { + "temperature": { + "value": 23 + } + } +} +``` + +
+ + +#### Measurement of child device + +
+ +**%%te%% (input)** + +```text title="Topic" +te/device/child01///m/ +``` + +```json5 title="Payload" +{ + "temperature": 23.4 +} +``` + +
+ +
+ +**Cumulocity IoT (output)** + +```text title="Topic" +c8y/measurement/measurements/create +``` + +```json5 title="Payload" +{ + "externalSource":{ + "externalId":":device:child01", + "type":"c8y_Serial" + }, + "type":"ThinEdgeMeasurement", + "time":"2013-06-22T17:03:14+02:00", + "temperature":{ + "temperature":{ + "value":23 + } + } +} +``` + +
+ +#### Measurement with unit + +The unit is a metadata associated with measurements which can be registered as a metadata message for a given measurement type. +If the following metadata message is registered for the `environment` measurement type: + +```sh te2mqtt formats=v1 +tedge mqtt pub -r te/device/main///m/environment/meta '{ + "units": { + "temperature": "Β°C" + } +}' +``` + +Then subsequent messages will be mapped with the registered unit value as follows. + +
+ +**%%te%% (input)** + +```text title="Topic" +te/device/main///m/environment +``` + +```json5 title="Payload" +{ + "temperature": 23.4 +} +``` + +
+ +
+ +**Cumulocity IoT (output)** + +```text title="Topic" +c8y/measurement/measurements/create +``` + +```json5 title="Payload" +{ + "type": "environment", + "time": "2021-04-22T17:05:26.958340390+00:00", + "temperature": { + "temperature": { + "value": 23, + "unit": "Β°C" + } + } +} +``` + +
+ +### Events + +
+ +**%%te%% (input)** + +```text title="Topic" +te/device/main///e/login_event +``` + +```json5 title="Payload" +{ + "text": "A user just logged in", + "time": "2021-01-01T05:30:45+00:00" +} +``` + +
+ +
+ +**Cumulocity IoT (output)** + +```text title="Topic" +c8y/s/us +``` + +```text title="Payload" +400,login_event,"A user just logged in",2021-01-01T05:30:45+00:00 +``` + +
+ +#### Event - Complex + +
+ +**%%te%% (input)** + +```text title="Topic" +te/device/main///e/login_event +``` + +```json5 title="Payload" +{ + "text": "A user just logged in", + "time": "2021-01-01T05:30:45+00:00", + "customFragment": { + "nested": { + "value": "extra info" + } + } +} +``` + +
+ +
+ +**Cumulocity IoT (output)** + +```text title="Topic" +c8y/event/events/create +``` + +```json5 title="Payload" +{ + "externalSource":{ + "externalId":"", + "type":"c8y_Serial" + }, + "type":"login_event", + "text":"A user just logged in", + "time":"2021-01-01T05:30:45+00:00", + "customFragment": { + "nested": { + "value": "extra info" + } + } +} +``` + +
+ +### Alarms + +
+ +**%%te%% (input)** + +```text title="Topic" +te/device/main///a/temperature_high +``` + +```json5 title="Payload" +{ + "severity": "critical", + "text": "Temperature is very high", + "time": "2021-01-01T05:30:45+00:00" +} +``` + +
+ +
+ +**Cumulocity IoT (output)** + +```text title="Topic" +c8y/s/us +``` + +```text title="Payload" +301,temperature_high,"Temperature is very high",2021-01-01T05:30:45+00:00 +``` + +
+ +#### Alarm - Complex + +
+ +**%%te%% (input)** + +```text title="Topic" +te/device/main///a/pressure_alarm +``` + +```json5 title="Payload" +{ + "severity": "major", + "time": "2023-01-25T18:41:14.776170774Z", + "text": "Pressure alarm", + "customFragment": { + "nested": { + "value": "extra info" + } + } +} +``` + +
+ +
+ +**Cumulocity IoT (output)** + +```text title="Topic" +c8y/alarm/alarms/create +``` + +```json5 title="Payload" +{ + "externalSource": { + "externalId": "", + "type": "c8y_Serial" + }, + "type": "pressure_alarm", + "severity": "MAJOR", + "time": "2023-01-25T18:41:14.776170774Z", + "text": "Pressure alarm", + "customFragment": { + "nested": { + "value": "extra info" + } + }, +} +``` + +
+ +### Health status + +
+ +**%%te%% (input)** + +```text title="Topic" +te/device/main/service/my-service/status/health +``` + +```json5 title="Payload" +{ + "status": "up", + "pid": "21037" +} +``` + +
+ +
+ +**Cumulocity IoT (output)** + +```text title="Topic" +c8y/s/us/ +``` + +```text title="Payload" +104,up +``` + +
+ + +## Twin + +The `twin` metadata is mapped to [inventory data](https://cumulocity.com/guides/concepts/domain-model/#inventory) in Cumulocity. + +#### Twin - Main device + +A device's digital twin model can be updated by publishing to a specific topic. + +The type part of the topic is used to group the data so it is easier for components to subscribe to relevant parts. + +
+ +**%%te%% (input)** + +```text title="Topic (retain=true)" +te/device/main///twin/device_OS +``` + +```json5 title="Payload" +{ + "family": "Debian", + "version": "11" +} +``` + +
+ +
+ +**Cumulocity IoT (output)** + +```text title="Topic" +c8y/inventory/managedObjects/update/ +``` + +```json5 title="Payload" +{ + "device_OS": { + "family": "Debian", + "version": "11" + } +} +``` + +
+ + +#### Twin - Child Device + +
+ +**%%te%% (input)** + +```text title="Topic (retain=true)" +te/device/child01///twin/device_OS +``` + +```json5 title="Payload" +{ + "family": "Debian", + "version": "11" +} +``` + +
+ +
+ +**Cumulocity IoT (output)** + +```text title="Topic" +c8y/inventory/managedObjects/update/:device:child01 +``` + +```json5 title="Payload" +{ + "device_OS": { + "family": "Debian", + "version": "11" + } +} +``` + +
+ +#### Twin - Service on Main Device + +
+ +**%%te%% (input)** + +```text title="Topic (retain=true)" +te/device/main/service/tedge-agent/twin/runtime_stats +``` + +```json5 title="Payload" +{ + "memory": 3024, + "uptime": 86400 +} +``` + +
+ +
+ +**Cumulocity IoT (output)** + +```text title="Topic" +c8y/inventory/managedObjects/update/:device:main:service:tedge-agent +``` + +```json5 title="Payload" +{ + "runtime_stats": { + "memory": 3.3, + "uptime": 86400 + } +} +``` + +
+ + +#### Twin - Service on Child Device + +
+ +**%%te%% (input)** + +```text title="Topic (retain=true)" +te/device/child01/service/tedge-agent/twin/runtime_stats +``` + +```json5 title="Payload" +{ + "memory": 3.3, + "uptime": 86400 +} +``` + +
+ +
+ +**Cumulocity IoT (output)** + +```text title="Topic" +c8y/inventory/managedObjects/update/:device:child01:service:tedge-agent +``` + +```json5 title="Payload" +{ + "runtime_stats": { + "memory": 3.3, + "uptime": 86400 + } +} +``` + +
+ + +### Twin data - Root fragments + +Data can be added on the root level of the twin by publishing the values directly to the topic with the key used as type. +The payload can be any valid JSON value other than a JSON object. +JSON objects must be published to their typed topics directly. + +
+ +**%%te%% (input)** + +```text title="Topic (retain=true)" +te/device/main///twin/subtype +``` + +```json5 title="Payload" +"my-custom-type" +``` + +
+ +
+ +**Cumulocity IoT (output)** + +```text title="Topic" +c8y/inventory/managedObjects/update/ +``` + +```json5 title="Payload" +{ + "subtype": "my-custom-type" +} +``` + +
+ +:::warning +Updating the following properties via the `twin` channel is not supported + +* `name` +* `type` + +as they are included in the entity registration message and can only be updated with another registration message. +::: + + +### Twin - Deleting a fragment + +
+ +**%%te%% (input)** + +```text title="Topic (retain=true)" +te/device/child01/service/tedge-agent/twin/runtime_stats +``` + +```json5 title="Payload" +<> +``` + +
+ +
+ +**Cumulocity IoT (output)** + +```text title="Topic" +c8y/inventory/managedObjects/update/:device:child01:service:tedge-agent +``` + +```json5 title="Payload" +{ + "runtime_stats": null +} +``` + +
+ +### Base inventory model + +The contents of `{tedge_config_dir}/device/inventory.json` are used to populate the initial inventory fragments +of the the main %%te%% device in Cumulocity. +For example, if the `inventory.json` contains the following fragments: + +```json title="inventory.json" +{ + "c8y_Firmware": { + "name": "raspberrypi-bootloader", + "version": "1.20140107-1", + "url": "31aab9856861b1a587e2094690c2f6e272712cb1" + }, + "c8y_Hardware": { + "model": "BCM2708", + "revision": "000e", + "serialNumber": "00000000e2f5ad4d" + } +} +``` + +It is mapped to the following Cumulocity message: + +```text title="Topic" +c8y/inventory/managedObjects/update +``` + +```json5 title="Payload" +{ + "c8y_Agent": { + "name": "thin-edge.io", + "url": "https://thin-edge.io", + "version": "x.x.x" + }, + "c8y_Firmware": { + "name": "raspberrypi-bootloader", + "version": "1.20140107-1", + "url": "31aab9856861b1a587e2094690c2f6e272712cb1" + }, + "c8y_Hardware": { + "model": "BCM2708", + "revision": "000e", + "serialNumber": "00000000e2f5ad4d" + } +} +``` + +Where the `c8y_Agent` fragment is auto-generated by %%te%% and appended to the contents of the file before it is published. + +The fragments in this file are also published to the `te/device/main///twin/` topics so that +the local twin metadata on the broker is also up-to-date and other components can also consume it. +For example, the above `inventory.json` would result in the following `twin` messages: + +```text title="Topic" +te/device/main///twin/c8y_Agent +``` + +```json5 title="Payload" +{ + "name": "thin-edge.io", + "url": "https://thin-edge.io", + "version": "x.x.x" +} +``` + +```text title="Topic" +te/device/main///twin/c8y_Firmware +``` + +```json5 title="Payload" +{ + "name": "raspberrypi-bootloader", + "version": "1.20140107-1", + "url": "31aab9856861b1a587e2094690c2f6e272712cb1" +} +``` + +```text title="Topic" +te/device/main///twin/c8y_Hardware +``` + +```json5 title="Payload" +{ + "model": "BCM2708", + "revision": "000e", + "serialNumber": "00000000e2f5ad4d" +} +``` + +:::warning +The following keys in the `inventory.json` file are also ignored: + +* `name` +* `type` + +as they are included in the entity registration message and can only be updated with another registration message. +::: + +### Updating entity type in inventory + +After updating the inventory with `inventory.json` file contents, +the device `type` of the main device, set using the `device.type` tedge config key, +is also updated in the inventory with the following message: + +```text title="Topic" +c8y/inventory/managedObjects/update +``` + +```json5 title="Payload" +{ + "type": "configured-device-type" +} +``` + + +## Operations/Commands + +Operations from Cumulocity are mapped to their equivalent commands in %%te%% format. + +### Supported Operations/Capabilities + +All the supported operations of all registered devices can be derived from the metadata messages +linked to their respective `cmd` topics with a simple subscription as follows: + +```sh +mosquitto_sub -v -t 'te/+/+/+/+/cmd/+' +``` + +If that subscription returns the following messages: + +``` text title="Output" +[te/device/main///cmd/restart] {} +[te/device/main///cmd/log_upload] { "types": ["tedge-agent", "mosquitto"] } +[te/device/child01///cmd/config_snapshot] { "types": ["mosquitto", "tedge", "collectd"] } +[te/device/child01///cmd/config_update] { "types": ["mosquitto", "tedge", "collectd"] } +``` + +The `cmd` metadata registered for both the `main` device and the child device `child01` +are mapped to the following supported operations messages: + +```text +[c8y/s/us] 114,c8y_Restart,c8y_LogfileRequest +[c8y/s/us/:device:child01] 114,c8y_UploadConfigFile,c8y_DownloadConfigFile +``` + +The operation specific metadata like `types` for `log_upload`, `config_snapshot` and `config_update` +are also mapped to their corresponding _supported logs_ and _supported configs_ messages as follows: + +```text +[c8y/s/us] 118,"tedge-agent", "mosquitto" +[c8y/s/us/:device:child01] 119,"mosquitto", "tedge", "collectd" +``` + +### Device Restart + +#### Request + +
+ +**Cumulocity IoT (input)** + +```text title="Topic" +c8y/s/ds +``` + +```csv title="Payload" +510, +``` + +
+ +
+ +**%%te%% (output)** + +```text title="Topic" +te/device/main///cmd/restart/ +``` + +```json5 title="Payload" +{ + "status": "init" +} +``` + +
+ +#### Response + +Even though operations on tedge can have different kinds of `status` for each operation type, +the mapper recognizes and maps only the following `status` values as follows: + + + + + + + + + + + + + + + + + + + + + + + + + +
%%te%% (input)Cumulocity IoT (output)
+ +```text title="Topic" +te/device/main///cmd/restart/ +``` + +```json5 title="Payload" +{ + "status": "executing" +} +``` + + + +```text title="Topic" +c8y/s/us +``` + +```text title="Payload" +501,c8y_Restart +``` + +
+ +```text title="Topic" +te/device/main///cmd/restart/ +``` + +```json5 title="Payload" +{ + "status": "successful" +} +``` + + + +```text title="Topic" +c8y/s/us +``` + +```text title="Payload" +503,c8y_Restart +``` + +
+ +```text title="Topic" +te/device/main///cmd/restart/ +``` + +```json5 title="Payload" +{ + "status": "failed" +} +``` + + + +```text title="Topic" +c8y/s/us +``` + +```text title="Payload" +502,c8y_Restart +``` + +
+ +All other `status` values are just ignored. + +### Restart: Child device + +
+ +**Cumulocity IoT (input)** + +```text title="Topic" +c8y/s/ds +``` + +```csv title="Payload" +510,:device:child01 +``` + +
+ +
+ +**%%te%% (output)** + +```text title="Topic" +te/device/child01///cmd/restart +``` + +```json5 title="Payload" +{} +``` + +
+ +### Software Update + +
+ +**Cumulocity IoT (input)** + +```text title="Topic" +c8y/s/ds +``` + +```csv title="Payload" +528,,nodered::debian,1.0.0,,install,collectd,5.7,,install,rolldice,1.16,,delete +``` + +
+ +
+ +**%%te%% (output)** + +```text title="Topic" +te/device/main///cmd/software_update/ +``` + +```json5 title="Payload" +{ + "status": "init", + "updateList": [ + { + "type": "debian", + "modules": [ + { + "name": "nodered", + "version": "1.0.0", + "url": "", + "action": "install" + }, + { + "name": "collectd", + "version": "5.7", + "action": "install" + }, + { + "name": "rolldice", + "version": "1.16", + "action": "remove" + } + ] + } + ] +} +``` + +
+ +Where the `collectd` binary from the `` is downloaded to the tedge file transfer repository by the mapper, +and the local `` of that binary is included in the mapped request. + +### Configuration Snapshot + +
+ +**Cumulocity IoT (input)** + +```text title="Topic" +c8y/s/ds +``` + +```csv title="Payload" +526,,collectd +``` + +
+ +
+ +**%%te%% (output)** + +```text title="Topic" +te/device/main///cmd/config_snapshot/ +``` + +```json5 title="Payload" +{ + "status": "init", + "type": "collectd", + "url": "" +} +``` + +
+ +Where the `url` is the target URL in the tedge file transfer repository to which the config snapshot must be uploaded. + +### Configuration Update + +
+ +**Cumulocity IoT (input)** + +```text title="Topic" +c8y/s/ds +``` + +```csv title="Payload" +524,,,collectd +``` + +
+ +
+ +**%%te%% (output)** + +```text title="Topic" +te/device/main///cmd/config_update/ +``` + +```json5 title="Payload" +{ + "status": "init", + "type": "collectd", + "url": "" +} +``` + +
+ +Where the `collectd` configuration binary from the `` is downloaded to the tedge file transfer repository by the mapper, +and the local `` of that binary is included in the mapped request. + +### Log Upload + +
+ +**Cumulocity IoT (input)** + +```text title="Topic" +c8y/s/ds +``` + +```csv title="Payload" +522,,tedge-agent,2013-06-22T17:03:14.000+02:00,2013-06-22T18:03:14.000+02:00,ERROR,1000 +``` + +
+ +
+ +**%%te%% (output)** + +```text title="Topic" +te/device/main///cmd/log_upload/ +``` + +```json5 title="Payload" +{ + "status": "init", + "type": "tedge-agent", + "tedgeUrl": "", + "dateFrom": "2013-06-22T17:03:14.000+02:00", + "dateTo": "2013-06-23T18:03:14.000+02:00", + "searchText": "ERROR", + "lines": 1000 +} +``` + +
+ +Where the `url` is the target URL in the tedge file transfer repository to which the config snapshot must be uploaded. diff --git a/versioned_docs/version-1.1.0/references/mappers/index.md b/versioned_docs/version-1.1.0/references/mappers/index.md new file mode 100644 index 0000000..491004f --- /dev/null +++ b/versioned_docs/version-1.1.0/references/mappers/index.md @@ -0,0 +1,18 @@ +--- +title: Mappers +tags: [Reference, Mappers] +sidebar_position: 3 +--- + +import DocCardList from '@theme/DocCardList'; + +# Thin-edge Mappers + +This section covers the specification of various mappers like cloud data mappers as well as data format mappers: + +- Cumulocity Mapper +- Azure Mapper +- AWS Mapper +- Collectd Mapper + + \ No newline at end of file diff --git a/versioned_docs/version-1.1.0/references/mappers/mqtt-topics.md b/versioned_docs/version-1.1.0/references/mappers/mqtt-topics.md new file mode 100644 index 0000000..9007589 --- /dev/null +++ b/versioned_docs/version-1.1.0/references/mappers/mqtt-topics.md @@ -0,0 +1,100 @@ +--- +title: MQTT Topics +tags: [Reference, Mappers, MQTT] +sidebar_position: 2 +--- + +# MQTT topics + +This document lists the MQTT topics that are used by the mappers. + +## Cumulocity MQTT Topics + +The topics follow the below format +`/[/