diff --git a/README.md b/README.md index 96adc1a..f2c6056 100644 --- a/README.md +++ b/README.md @@ -5,16 +5,18 @@ Welcome to Piper! Piper is open source project that aimed at providing multibran ## Table of Contents -- [Getting Started](#getting-started) -- [Reporting Issues](#reporting-issues) -- [How to Contribute](docs/CONTRIBUTING.md#how-to-contribute) -- [License](#license) +- [Piper](#piper) + - [Table of Contents](#table-of-contents) + - [Getting Started](#getting-started) + - [Reporting Issues](#reporting-issues) + - [How to Contribute](#how-to-contribute) + - [License](#license) ## Getting Started -Piper configures a webhook in git provider and listens to the webhooks sends. It will create a Workflow CRD out of branches that contains `.workflows` folder. -This folder should contain declarations of the templates and main DAG that will be running. -Finally, it will submit the Workflow as a K8s resource in the cluster. +Piper configures a webhook in git provider and listens to the webhooks sends. It will create a Workflow CRD out of branches that contains `.workflows` folder. +This folder should contain declarations of the templates and main DAG that will be running. +Finally, it will submit the Workflow as a K8s resource in the cluster. To access more detailed explanations, please navigate to the [Documentation site](https://piper.quickube.com). https://github.com/quickube/piper/assets/106976988/09b3a5d8-3428-4bdc-9146-3034d81164bf @@ -29,7 +31,6 @@ If you encounter any issues or bugs while using Piper, please help us improve by 4. Add appropriate labels to categorize the issue (e.g., bug, enhancement, question). 5. Submit the issue, and our team will review and address it as soon as possible. - ## How to Contribute If you're interested in contributing to this project, please feel free to submit a pull request. We welcome all contributions and feedback. @@ -38,6 +39,3 @@ Please check out our [Contribution guidelines for this project](docs/CONTRIBUTIN ## License This project is licensed under the Apache License. Please see the [LICENSE](LICENSE) file for details. - - - diff --git a/docs/configuration/environment_variables.md b/docs/configuration/environment_variables.md index e96f0b8..483a62a 100644 --- a/docs/configuration/environment_variables.md +++ b/docs/configuration/environment_variables.md @@ -1,49 +1,49 @@ ## Environment Variables -The environment variables used by Piper to configure its functionality. -The helm chart populate them using [values.yaml](https://github.com/quickube/piper/tree/main/helm-chart/values.yaml) file +Piper uses the following environment variables to configure its functionality. +The helm chart populates them using [values.yaml](https://github.com/quickube/piper/tree/main/helm-chart/values.yaml) file ### Git * GIT_PROVIDER - The git provider that Piper will use, possible variables: GitHub (will support bitbucket and gitlab) + The git provider that Piper will use, possible variables: GitHub . We plan to support Bitbucket and GitLab, as well. * GIT_TOKEN - The git token that will be used. + The git token that will be used to connect to the git provider. * GIT_ORG_NAME The organization name. * GIT_ORG_LEVEL_WEBHOOK - Boolean variable, whether to config webhook in organization level. default `false` + Boolean variable, whether to config webhook at the organization level. Defaults to `false`. * GIT_WEBHOOK_REPO_LIST List of repositories to configure webhooks to. * GIT_WEBHOOK_URL - URL of piper ingress, to configure webhooks. + URL of Piper ingress, to configure webhooks. * GIT_WEBHOOK_AUTO_CLEANUP - Will cleanup all webhook that were created with piper. - Notice that there will be a race conditions between pod that being terminated and the new one. + Boolean variable that, if true, will cause Piper to automatically cleanup all webhooks that it creates when they are no longer necessary. + Notice that there is a race condition between a pod that is being terminated and the new one being scheduled. * GIT_ENFORCE_ORG_BELONGING - Boolean variable, whether to enforce organizational belonging of git event creator. default `false` + Boolean variable that, if true, will cause Piper to enforce organizational belonging of git event creator. Defaults to `false`. * GIT_FULL_HEALTH_CHECK - Enables full health check of webhook. Full health check contains expecting and validating ping event from a webhook. - Doesn't work for bitbucket, because the API call doesn't exists. - + Boolean variable that, if true, enables full health checks on webhooks. A full health check contains expecting and validating ping event from a webhook. + Doesn't work for Bitbucket, because the API call doesn't exist on that platform. ### Argo Workflows Server + * ARGO_WORKFLOWS_TOKEN - The token of Argo Workflows server. + This token is used to authenticate with the Argo Workflows server. * ARGO_WORKFLOWS_ADDRESS The address of Argo Workflows Server. - + * ARGO_WORKFLOWS_CREATE_CRD - Whether to directly send Workflows instruction or create a CRD in the Cluster. + Boolean variable that deterines whether to directly send Workflows instructions or create a CRD in the Cluster. * ARGO_WORKFLOWS_NAMESPACE The namespace of Workflows creation for Argo Workflows. @@ -52,9 +52,10 @@ The helm chart populate them using [values.yaml](https://github.com/quickube/pip Used to configure Argo Workflows client with local kube configurations. ### Rookout + * ROOKOUT_TOKEN The token used to configure Rookout agent. If not provided, will not start the agent. -* ROOKOUT_LABELS - The labels to label instances at Rookout, default to "service:piper" +* ROOKOUT_LABELS + The labels to label instances at Rookout, defaults to "service:piper" * ROOKOUT_REMOTE_ORIGIN - The repo URL for source code fetching, default:"https://github.com/quickube/piper.git". \ No newline at end of file + The repo URL for source code fetching, default:"https://github.com/quickube/piper.git". diff --git a/docs/configuration/health_check.md b/docs/configuration/health_check.md index 3033526..d91dfec 100644 --- a/docs/configuration/health_check.md +++ b/docs/configuration/health_check.md @@ -1,6 +1,7 @@ ## Health Check -Health check executed every 1 minute as configured in the helm chart under `livenessProbe`, and triggered by `/healthz` endpoint: +The following examples shows a health check being executed every 1 minute as configured in the helm chart under `livenessProbe`, and triggered by `/healthz` endpoint: + ```yaml livenessProbe: httpGet: @@ -18,14 +19,13 @@ The mechanism for checking the health of Piper is: 1. Piper set health status of all webhooks to not-healthy. -2. Piper requests ping from all the webhooks configured. +2. Piper requests ping from all the webhooks configured. 3. Git Provider send ping to `/webhook` endpoint, this will set the health status to `healthy` with timeout of 5 seconds. 4. Piper check the status of all webhooks configured. Therefore, the criteria for health checking are: -1. The registered webhook exists. -2. The webhook send a ping in 5 seconds. - +1. The registered webhook exists. +2. The webhook send a ping in 5 seconds. diff --git a/docs/getting_started/installation.md b/docs/getting_started/installation.md index 5ab16c0..cb12a3a 100644 --- a/docs/getting_started/installation.md +++ b/docs/getting_started/installation.md @@ -1,16 +1,18 @@ -## Instalation +## Installation -Piper should be deployed in the cluster with Argo Workflows. -Piper will create a CRD that Argo Workflows will pick, so install or configure Piper to create those CRDs in the right namespace. +You must deploy Piper to a cluster with a pre-existing Argo Workflows deployment. +Piper will create a CRD that Argo Workflows will pick, so install or configure Piper to create those CRDs in the right namespace. Please check out [values.yaml](https://github.com/quickube/piper/tree/main/helm-chart/values.yaml) file of the helm chart configurations. To add piper helm repo run: + ```bash helm repo add piper https://piper.quickube.com ``` After configuring Piper [values.yaml](https://github.com/quickube/piper/tree/main/helm-chart/values.yaml), run the following command for installation: + ```bash helm upgrade --install piper piper/piper \ -f YOUR_VALUES_FILE.yaml @@ -22,33 +24,34 @@ helm upgrade --install piper piper/piper \ ### Ingress -Piper should listen to webhooks from your git provider. -Expose it using ingress or service, then provide the address to `piper.webhook.url` as followed: -`https://PIPER_EXPOESED_URL/webhook` +Piper works best when it is able to listen to webhooks from your git provider. +Expose Piper using an ingress or service, then provide the address to `piper.webhook.url` as follows: +`https://PIPER_EXPOSED_URL/webhook` -Checkout [values.yaml](https://github.com/quickube/piper/tree/main/helm-chart/values.yaml) +Refer to [values.yaml](https://github.com/quickube/piper/tree/main/helm-chart/values.yaml) for more information. ### Git -Piper will use git for fetching `.workflows` folder and receiving events using webhooks. +Piper will use git to fetch the `.workflows` folder and receive events using webhooks. -To pick which git provider you are using provide `gitProvider.name` configuration in helm chart (Now only supports GitHub and Bitbucket). +To pick which git provider you are using provide `gitProvider.name` configuration in helm chart (Currently we only support GitHub and Bitbucket). -Also configure you organization (Github) or workspace (Bitbucket) name using `gitProvider.organization.name` in helm chart. +You must also configure your organization (GitHub) or workspace (Bitbucket) name using `gitProvider.organization.name` in the helm chart. #### Git Token Permissions The token should have access for creating webhooks and read repositories content. -For GitHub configure `admin:org` and `write:org` permissions in Classic Token. -For Bitbucket configure `Repositories:read`, `Webhooks:read and write` and `Pull requests:read` permissions (for multiple repos use workspace token). +For GitHub, configure `admin:org` and `write:org` permissions in Classic Token. +For Bitbucket, configure `Repositories:read`, `Webhooks:read and write` and `Pull requests:read` permissions (for multiple repos use workspace token). #### Token -The git token should be passed as secret in the helm chart at `gitProvider.token`. -Can be passed as parameter in helm install command using `--set piper.gitProvider.token=YOUR_GIT_TOKEN` +The git token should be passed as secret in the helm chart at `gitProvider.token`. +The token can be passed as parameter via helm install command using `--set piper.gitProvider.token=YOUR_GIT_TOKEN` + +Alternatively, you can consume an already existing secret by configuring `piper.gipProvider.existingSecret`. +The key should have the name `token`. You can be create a Secret using this command: -Alternatively, you can consume already existing secret and fill up `piper.gipProvider.existingSecret`. -The key should be name `token`. Can be created using ```bash kubectl create secret generic piper-git-token --from-literal=token=YOUR_GIT_OKEN ``` @@ -57,17 +60,17 @@ kubectl create secret generic piper-git-token --from-literal=token=YOUR_GIT_OKEN Piper will create a webhook configuration for you, for the whole organization or for each repo you configure. -Configure `piper.webhook.url` the address of piper that exposed with ingress with `/webhook` postfix. +Configure `piper.webhook.url` with the address of Piper that you exposed using an Ingress or Service with `/webhook` postfix. -For organization level configure: `gitProvider.webhook.orgLevel` to `true`. +For organization level configuration: `gitProvider.webhook.orgLevel` to `true`. -For granular repo webhook provide list of repos at: `gitProvider.webhook.repoList`. +For granular repo webhook provide list of repos at: `gitProvider.webhook.repoList`. -Piper implements graceful shutdown, it will delete all the webhooks when terminated. +Piper implements graceful shutdown, it will delete all the webhooks when terminated. #### Status check -Piper will handle status checks for you. +Piper will handle status checks for you. It will notify the GitProvider for the status of Workflow for specific commit that triggered Piper. For linking provide valid URL of your Argo Workflows server address at: `argoWorkflows.server.address` @@ -81,4 +84,4 @@ To lint the workflow before submitting it, please configure the internal address #### Skip CRD Creation (On development) -Piper can communicate directly to Argo Workflow using ARGO_WORKFLOWS_CREATE_CRD environment variable, if you want to skip the creation of CRD change `argoWorkflows.crdCreation` to `false`. \ No newline at end of file +Piper can communicate directly to Argo Workflow using ARGO_WORKFLOWS_CREATE_CRD environment variable, if you want to skip the creation of CRD change `argoWorkflows.crdCreation` to `false`. diff --git a/docs/index.md b/docs/index.md index 65488d1..3e8f0f2 100644 --- a/docs/index.md +++ b/docs/index.md @@ -3,19 +3,17 @@

- -Welcome to Piper! -Piper is an open source project that aimed at providing multibranch pipeline functionality to Argo Workflows, allows users to create distinct Workflows based on Git branches. Supports GitHub and Bitbucket. +Welcome to Piper! -## General explanation +Piper is an open source project that aimed at providing multibranch pipeline functionality to Argo Workflows. This allows users to create distinct Workflows based on Git branches. We supports GitHub and Bitbucket. + +## General Explanation

-To achieve multibranch pipeline functionality Piper will do the hard works for us. -At initialization, it will load all configuration and create a webhook in repository or organization scope. -Then each branch that have `.workflows` folder will create a Workflow CRD out of the files in this folder. +Piper handles the hard work of configuring multibranch pipelines for us! At initialization, it will load all configuration and create a webhook in repository or organization scope. Then, for each branch that has a `.workflows` folder, Piper will create a Workflow CRD out of the files in this folder. Finally, when Piper detects changes in the repository via the webhook, it triggers the workflows that match the branch and event. -![type:video](./img/piper-demo-1080.mp4) \ No newline at end of file +![type:video](./img/piper-demo-1080.mp4) diff --git a/docs/usage/global_variables.md b/docs/usage/global_variables.md index 93e12dd..c87c227 100644 --- a/docs/usage/global_variables.md +++ b/docs/usage/global_variables.md @@ -1,4 +1,4 @@ -## Global variables +## Global Variables Piper will automatically add Workflow scope parameters that can be referenced from any template. The parameters taken from webhook metadata, and will be populated respectively to GitProvider and event that triggered the workflow. diff --git a/docs/usage/workflows_config.md b/docs/usage/workflows_config.md index 2ef0eef..146ffc0 100644 --- a/docs/usage/workflows_config.md +++ b/docs/usage/workflows_config.md @@ -5,15 +5,18 @@ Piper can inject configuration for Workflows that Piper creates. `default` config used as a convention for all Workflows that piper will create, even if not explicitly mentioned in triggers.yaml file. ### ConfigMap + Piper will mount a configMap when helm used. `piper.workflowsConfig` variable in helm chart, will create a configMap that hold set of configuration for Piper. Here is an [examples](https://github.com/quickube/piper/tree/main/examples/config.yaml) of such configuration. ### Spec -This will be injected to Workflow spec field. can hold all configuration of the Workflow. + +This will be injected to Workflow spec field. can hold all configuration of the Workflow. > :warning: Please notice that the fields `entrypoint` and `onExit` should not exist in spec. both of them are managed fields. ### onExit -This is the exit handler for each of the Workflows create by piper. -It configures a DAG that will be executed when the workflow ends. -You can provide the templates to it us in the following [Examples](https://github.com/quickube/piper/tree/main/examples/config.yaml). \ No newline at end of file + +This is the exit handler for each of the Workflows create by piper. +It configures a DAG that will be executed when the workflow ends. +You can provide the templates to it us in the following [Examples](https://github.com/quickube/piper/tree/main/examples/config.yaml). diff --git a/docs/usage/workflows_folder.md b/docs/usage/workflows_folder.md index 5ed3ca4..b0cd551 100644 --- a/docs/usage/workflows_folder.md +++ b/docs/usage/workflows_folder.md @@ -5,8 +5,9 @@ We will explain each of the files that should be included in the `.workflows` fo ### triggers.yaml (convention name) -This file holds a list of triggers that will be executed `onStart` by `events` from specific `branches`. +This file holds a list of triggers that will be executed `onStart` by `events` from specific `branches`. Piper will execute each of matching triggers, so configure it wisely. + ```yaml - events: - push @@ -17,38 +18,43 @@ Piper will execute each of matching triggers, so configure it wisely. templates: ["templates.yaml"] config: "default" ``` -Can be found [here](https://github.com/quickube/piper/tree/main/examples/.workflows/triggers.yaml). + +This example can be found [here](https://github.com/quickube/piper/tree/main/examples/.workflows/triggers.yaml). In this example `main.yaml` will be executed as DAG when `push` or `pull_request.synchronize` events will be applied in `main` branch. `onExit` will be executed `exit.yaml` when finished the workflow as exit handler. - `onExit` can overwrite the default `onExit` configuration from by reference existing DAG tasks as in the [example](https://github.com/quickube/piper/tree/main/examples/.workflows/exit.yaml). `config` field used for workflow configuration selection. the default value is `default` configuration. #### events -Events field used to terminate when the trigger will be executed. name of the event depends on the git provider. + +Events field used to terminate when the trigger will be executed. name of the event depends on the git provider. For instance, GitHub pull_request event have few action, one of them is synchronize. #### branches + For which branch that trigger will be executed. -#### onStart +#### onStart + This [file](https://github.com/quickube/piper/tree/main/examples/.workflows/main.yaml) can be named as you wish and will be referenced in `triggers.yaml` file. It will define an entrypoint DAG that the Workflow will execute. As a best practice, this file should contain the dependencies logic and parametrization of each of referenced templates. It should not implement new templates, for this, use template.yaml file. -#### onExit +#### onExit + This field used to pass verbose exitHandler to the triggered workflow. It will override the default onExit from the provided `config` or the default `config`. In the provided `exit.yaml` describes a DAG that will overwrite the default `onExit` configuration. [Example](https://github.com/quickube/piper/tree/main/examples/.workflows/exit.yaml) -#### templates -This field will have additional templates that will be injected to the workflows. +#### templates + +This field will have additional templates that will be injected to the workflows. The purpose of this field is to create repository scope templates that can be referenced from the DAGs templates at `onStart` or `onExit`. [Example](https://github.com/quickube/piper/tree/main/examples/.workflows/templates.yaml) @@ -56,11 +62,13 @@ As a best practice, use this field for template implementation and reference the [Example](https://github.com/quickube/piper/tree/main/examples/.workflows/main.yaml). ### config -configured by `piper-workflows-config` [configMap](workflows_config.md). + +configured by `piper-workflows-config` [configMap](workflows_config.md). Can be passed explicitly, or will use `deafault` configuration. ### parameters.yaml (convention name) -Will hold a list of global parameters of the Workflow. -can be referenced from any template with `{{ workflow.parameters.___ }}.` -[Example](https://github.com/quickube/piper/tree/main/examples/.workflows/parameters.yaml) \ No newline at end of file +Will hold a list of global parameters of the Workflow. +Can be referenced from any template with `{{ workflow.parameters.___ }}.` + +[Example](https://github.com/quickube/piper/tree/main/examples/.workflows/parameters.yaml) diff --git a/examples/.workflows/templates.yaml b/examples/.workflows/templates.yaml index 8324f17..f8921df 100644 --- a/examples/.workflows/templates.yaml +++ b/examples/.workflows/templates.yaml @@ -6,5 +6,5 @@ image: alpine command: [ sh ] source: | - echo "wellcome to {{ workflow.parameters.global }} - echo "{{ inputs.parameters.message }}" \ No newline at end of file + echo "welcome to {{ workflow.parameters.global }} + echo "{{ inputs.parameters.message }}" diff --git a/examples/config.yaml b/examples/config.yaml index fb0dccf..3e3a623 100644 --- a/examples/config.yaml +++ b/examples/config.yaml @@ -24,10 +24,10 @@ data: key: node_pool operator: Equal value: workflows - onExit: # optinal, will be overwritten if specifc in .wokrflows/exit.yaml. + onExit: # optional, will be overwritten if specifc in .workflows/exit.yaml. - name: github-status template: exit-handler arguments: parameters: - name: param1 - value: "{{ workflow.labels.repo }}" \ No newline at end of file + value: "{{ workflow.labels.repo }}" diff --git a/examples/workflow.yaml b/examples/workflow.yaml index 4e342b5..3c5f94c 100644 --- a/examples/workflow.yaml +++ b/examples/workflow.yaml @@ -19,7 +19,7 @@ spec: archiveLogs: true arguments: parameters: - - name: PLACHOLDER + - name: PLACEHOLDER artifactRepositoryRef: configMap: artifact-repositories onExit: exit-handler @@ -50,7 +50,7 @@ spec: image: alpine command: [sh] source: | - echo "wellcome to {{ workflow.parameters.global }} + echo "welcome to {{ workflow.parameters.global }} echo "{{ inputs.parameters.message }}" - name: exit-handler script: @@ -75,6 +75,3 @@ spec: value: step-2 dependencies: - local-step1 - - - \ No newline at end of file