diff --git a/.github/PULL_REQUEST_TEMPLATE/documentation.md b/.github/PULL_REQUEST_TEMPLATE/documentation.md new file mode 100644 index 0000000..01e58ed --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE/documentation.md @@ -0,0 +1 @@ +Resolves # diff --git a/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md b/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md deleted file mode 100644 index 90f639b..0000000 --- a/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md +++ /dev/null @@ -1,17 +0,0 @@ -# Pull Request - -## Related Issue - -Resolves # - -## Description of the Change - - - -## Prerequisites - - - -## Testing Instructions - - diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index e354fa2..a0a60a4 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -1,22 +1,7 @@ -# Pull Request - -## Related Issue +## Issue Resolves # -## Description of the Change - - - -## Prerequisites - - - -## Testing Instructions - - - - ## Checklist - [ ] Did you include necessary logging? diff --git a/docs/adrs/0001_initial_setup.md b/docs/adrs/0001_initial_setup.md index 9a185b6..4528ec0 100644 --- a/docs/adrs/0001_initial_setup.md +++ b/docs/adrs/0001_initial_setup.md @@ -30,7 +30,7 @@ The code wasn't necessarily clearly contained within a container like the `confi Here is a view from the command line at this time. -![23_12_2021_CLI_view](../assets/23_12_2021_CLI_view.png) +![23_12_2021_CLI_view](../assets/screenshots/23_12_2021_CLI_view.png) Here is some additional context on these root commands: @@ -57,5 +57,6 @@ The initial concept has been designed as a single scenario concept (due to a pro * creating the correct structure in config for components like local, gcp and any future ones * various components in the code were not prepared to be modularized in a way to easily accept extensions from new components (like AWS extensions) - -* [Piotr](https://github.com/Katolus) + + +[Piotr]: https://github.com/Katolus diff --git a/docs/adrs/0002_architecture_layout.md b/docs/adrs/0002_architecture_layout.md new file mode 100644 index 0000000..60b8d8b --- /dev/null +++ b/docs/adrs/0002_architecture_layout.md @@ -0,0 +1,137 @@ +# `functions` - architecture overview + +* Status: Proposed +* Deciders: [Piotr] +* Date: 2022-02-23 + +## Context and Problem Statement + +This document describes all the architectural components used in the layout of the app. + +It describe the assumed flow of the app under certain user stories and how different components fall into a full composition. + +The architecture of the project has been developed in iterative process of improvement to the base concept (ADR-0001) and this document will formalize the decisions and provide shape and form context. + +## Users + +We are taking into the account following user stories: + +* user has never worked with function as a service +* user has existing functions and wants to use the tool for unification and easy of use + * locally on the machine of work + * remotely in a form of a repository +* machine use, as an example being part of a CI/CD + +## Goals + +### Seamless installation + +**Problem**: We want our users to have a single line and minimal complexity installation journey. + +In the driver of *simplicity* the installation should be a quick and easy process. This should include all the ways of managing functions. + +### Hight modularity + +**Problem**: We want to allow user to decide which parts of the package are required and which one's are not. + +This means that if the user only needs the local setup, then installing `docker` should be the only thing required to run those commands. + +Some of the questions we need to answer with the solution: + +* How do we figure out which components are available? +* How do we store that information? +* How do we change that state? +* Do we remove or update that information? +* Do we run any validation? If so, then when? How often? + +### Components sync + +**Problem**: We need to have a consistent way of knowing what is happening with various parts of the application at the lowest operational cost. + +This means we want to know which functions are currently deployed, but we don't want to run a check every time a script is run, because that is very inefficient in the CLI way. + +Some of the questions we need to answer with the solution: + +* How and when do we query and store the current state of the `Docker` component? Is a function running, stopped, built? + +### Assumptions + +Our users are technically skilled individuals or organisation that have necessary skills to operate a CLI tool. + +### Constraints + +No constrains at the moment. + +## Decision Drivers + +* Implementation complexity +* Simplicity of use +* Configurability + + +## Design decision + +The architecture is build in a iterative process with improving current design. + +In the following sections we describe components, their purpose and how they may interact with other parts of the design. + +### Overview + +TBU + +### App Diagram + +*Insert a diagram representing different interactions between components.* + +![Functions app diagram](../assets/diagrams/functional_layout.png) + +### Flow + +TBU + +Example interaction for basic case of CLI command. + +1. User types a `functions` command in the terminal. +2. `functions` will evaluate available components and store this information in the config file. Command will raise exceptions if required components like `docker` are not present. +3. Next, it will evaluate the registry if information about this function is available and if said registry needs to be updated. +4. Depending on the command type, `docker` will be used to run or serve a targeted function. +5. If a command deploys resources to a cloud provider, we will use the correct provider to manipulate resources. Subject to authorization and sub-tool (like `gcloud`) availability. + +TBU + +- Internal state saved in a function registry +- Component management +- Functions Config +- Logging +- Exceptions +- Managing Docker resources +- Managing GCP resources +- Creating new resources +- Adding existing resources to the scope of the tool + - defaults + - +- Command validation and autocomplete +- Flows and Actions + + +### Disclaimer + +All these sections describe an idea components. What is real and present in the code often escapes that. These are the best ideas that we came up with trying to handle code scalability, tool's functionality while maintaining an experimental mindset. + +## Decision Impact + +| Interested Parties/Groups | Informed | +| ------------------------- | -------- | +| All Users | N/A | + +### How will this decision be communicated to the public + +There is no need to communicate it to any channels as it is a documentation written prior to any releases. It is bound to the first release. + +## Links + +* [ADR-0001](0001_initial_setup.md) + + + +[Piotr]: https://github.com/Katolus diff --git a/docs/adrs/0002_development_branch.md b/docs/adrs/0002_development_branch.md index e69de29..2c7e7ef 100644 --- a/docs/adrs/0002_development_branch.md +++ b/docs/adrs/0002_development_branch.md @@ -0,0 +1,90 @@ +# What are we using as our main development branch? + +* Status: Accepted +* Deciders: [Piotr] +* Date: 2021-11-23 + +## Context and Problem Statement + +[Describe the context and problem statement, e.g., in free form using two to three sentences. You may want to articulate the problem in form of a question.] + +## Details + + +### Constraints + +* Adhere to an understandable standard. + +## Decision Drivers + +* Keep it simple. +* Keep it flexible. +* Keep it cheap. +* Keep it safe. + +## Considered Options + +* `master`/`main` +* `development` + +## Decision Outcome + +Chosen option: `development`, because we think that it will be the easiest branch names to drive ongoing development efforts. It semantically separate from `master` or `main` and has a clear commitment. + +The idea behind this decision is a attempt to create a more accessible development stream. + +The vision being that the `master` branch should be focused on the valuable releases associate with package versions releases. + +All the tooling that needs to be run to enforce quality should be associated with the `master` branch. + +The `master` branch should be protected. + +All this should enable a more flexible work around smaller issues in a singular pool of developer(s). + +All the PRs should target the `development` branch as base and a release PR should be a combination of sub PRs on a per ready basis. + +This means that the tests for various python versions of the package and other heavy duty actions will only be trigger on merges to master. + +### Positive Consequences + +* low-cost of ongoing work. +* faster development. +* more control over release stages. + +### Negative Consequences + +* untested changes in `development` might be hard to discover prior to `master` merges. +* slightly more complicated contribution process. + +## Pros and Cons of the Options + +### `master` / `main` + +Use `master` or `main` as the main development branch. + +* Good, because it is a well understand pattern. +* Bad, because it will cost us more to run all the check in every merge event. + +### `development` + +Use `development` as the main development branch. + +* Good, because it separates development actions from main tooling branch. +* Good, because it has a clear meaning. +* Good, because it should allow for making quick changes. +* Bad, because it might be a bit confusing. +* Bad, because it might lead untested changes into the main `development` branch. + +## Decision Impact + +| Interested Parties/Groups | Informed | +| ------------------------- | -------- | +| Contributors | Yes | + +### How will this decision be communicated to the public + +This information is communication via the `CONTRIBUTING.md` document. + + + +[Piotr]: https://github.com/Katolus diff --git a/docs/adrs/0003_gcloud_vs_python_api.md b/docs/adrs/0003_gcloud_vs_python_api.md index a8626f2..bcabe9c 100644 --- a/docs/adrs/0003_gcloud_vs_python_api.md +++ b/docs/adrs/0003_gcloud_vs_python_api.md @@ -1,9 +1,81 @@ -# `gcloud` vs GCP's Python APIs +# What interface should we use it interact with GCP resources? -## Problem statement +* Status: Accepted +* Deciders: [Piotr] +* Date: 2021-11-23 -In order to work with GCP, we need to find a way of interacting with GCP resources. There are two know ways of doing that. One way is to use the `gcloud` tool. A command line tool that can be used to interact with GCP via a terminal. Second way is to use GCP's Python API libraries. A set of libraries developed by Google to make it possible to interact with GCP's components from a python scope. +## Context and Problem Statement -## Attempt +Since we want to enable users to be able to deploy functions to GCP, we need to have an interface that we can call within our code. -History attempt on using the libraries. +This interface needs to be capable of bridging us from a source files to a deployed and working cloud resource. + +### Assumptions + +The tool is not meant to be ideal, but be a working proof of concept at this point in time. + +### Constraints + +* Limited development capacity +* Scope of the initial release + +## Decision Drivers + +* Development complexity +* Short vs long term solution + +## Considered Options + +* `gcloud` +* `Python API` + +### `gcloud` + +Use the `gcloud` command line interface to move our resources from the source to cloud. + +Pros: + +* Good, because it is a simple solution that allows us to provide this functionality. +* Good, because it handles a lot of the complexity, like authorization, outputs, logging. + +Cons: + +* Bad, because it feels more like a hack than a solution. +* Bad, because it is not a long term solution. +* Bad, because it requires an external interface - `gcloud` - to handle the tasks. + +**Spike**: This works relatively well, but feels like a hacky way to solve this problem. + +### Python APIs + +Use resource [specific](https://cloud.google.com/python/docs/reference/cloudfunctions/latest) as well as [generic](https://github.com/googleapis/google-api-python-client) python SDKs to interact with GCP resources. + +Pros: + +* Good, because it gives us a lot of flexibility over our implementation. +* Good, because we don't have to rely on a user having to install a tool within the environment. +* Good, because there is place for factoring out common code from different providers and exploring unification efforts more achievable. + +Cons: + +* Bad, because it requires a lot of additional work to write logic to handle all the resources. +* Bad, because it is much more complex to do so. There are no single call APIs for cloud functions. +* Bad, because it feels like reinventing parts of the `gcloud` tool. + +**Spike**: What seems like a simple task of deploying a folder of files into a cloud function scope turns out to be more complex than that. You need to have a place to store the files, build the function and then connect all these resources. Cleaning and managing artefacts becomes our scope as well. + +## Decision Outcome + +Chosen option: `gcloud`, because the development effort is minimal and allows us to perform really powerful stuff. It offloads the effort onto another tool and allows us to test and move at relatively fast pace. At a cost of not being a future proof solution. + +## Decision Impact + +No one is impacted by this decision. Interface will stay without a change. This change does have a long term impact of being vulnerable to any `gcloud` version changes. + +### How will this decision be communicated to the public + +No need. + + + +[Piotr]: https://github.com/Katolus diff --git a/docs/adrs/0004_usage_flow.md b/docs/adrs/0004_usage_flow.md deleted file mode 100644 index 2b6ed3a..0000000 --- a/docs/adrs/0004_usage_flow.md +++ /dev/null @@ -1,143 +0,0 @@ -# `functions` high level flow - -* Status: Proposed -* Deciders: [Piotr] -* Date: 2021-12-23 - -## Context and Problem Statement - -We need to document the technical assumptions around the way this software can be used. Describe user stories through the app flow and how different components fall into full composition. - -The project has been based on iterations of improvement to the base concept, however at this point there is a need of formalizing the decisions makings to add a shape and form the way the project is developed. - -## Details - -We want to define the design of the app from different points of perspective and propose a plan of how different components will interact with each other. - -Decisions made to the flow should take into the account a user stories of: - -* user has never worked with function as a service -* user has existing functions and wants to use the tool for unification and easy of use - * locally on the machine of work - * remotely in a form of a repository -* machine use, as an example being part of a CI/CD - -## Goals - -### Seamless installation - -TBU - -### Hight modularity - -**Problem**: We want to allow user to decide which parts of the package are required and which one's are not. - -This means that if the user only needs the local setup, then installing docker should be the only thing required to run those commands. - -Some of the questions we need to answer with the solution: - -* How do we figure out which components are available? -* How do we store that information? -* How do we change that state? -* Do we remove or update that information? -* Do we run any validation? If so, then when? How often? - -### Components sync - -**Problem**: We need to have a consistent way of knowing what is happening with various parts of the application at the lowest operational cost. - -This means we want to know which functions are currently deployed, but we don't want to run a check every time a script is run, because that is very inefficient in the CLI way. - -Some of the questions we need to answer with the solution: - -* How and when do we query and store the current state of the `Docker` component? Is a function running, stopped, built? - -### Assumptions - -Our users are technically skilled individuals or organisation that have necessary skills to operate a CLI tool. - -### Constraints - -No constrains at the moment. - -## Decision Drivers - -* implementation complexity -* simplicity of use -* configurability - -## Considered Options - -[Can be either a list of options or in complicated scenarios, a reference to sub documents] - -* subcommand components -* sync subcommand -* `init` command - -## Decision Outcome - -Chosen option: "[option 1]", because [justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force force | … | comes out best (see below)]. - -### App Diagram - -*Insert a diagram representing different interactions between components.* - -### Positive Consequences - -* [e.g., improvement of quality attribute satisfaction, follow-up decisions required, …] -* … - -### Negative Consequences - -* [e.g., compromising quality attribute, follow-up decisions required, …] -* … - -## Pros and Cons of the Options - -### [option 1] - -[example | description | pointer to more information | …] - -* Good, because [argument a] -* Good, because [argument b] -* Bad, because [argument c] -* … - -### [option 2] - -[example | description | pointer to more information | …] - -* Good, because [argument a] -* Good, because [argument b] -* Bad, because [argument c] -* … - -### [option 3] - -[example | description | pointer to more information | …] - -* Good, because [argument a] -* Good, because [argument b] -* Bad, because [argument c] -* … - -## Decision Impact - -[Who may be impacted by this decision] - -| Interested Parties/Groups | Informed | -| ------------------------- | -------- | -| N/A | N/A | - -### How will this decision be communicated to the public - -It is bound to the first release. - -## Links - -* [Link type] [Link to ADR] -* … - - - -[Piotr]: https://github.com/Katolus diff --git a/docs/assets/diagrams/functional_layout.png b/docs/assets/diagrams/functional_layout.png new file mode 100644 index 0000000..9fb5858 Binary files /dev/null and b/docs/assets/diagrams/functional_layout.png differ diff --git a/docs/assets/23_12_2021_CLI_view.png b/docs/assets/screenshots/23_12_2021_CLI_view.png similarity index 100% rename from docs/assets/23_12_2021_CLI_view.png rename to docs/assets/screenshots/23_12_2021_CLI_view.png diff --git a/docs/assets/screenshots/function_name_in_docker_2022-02-27.png b/docs/assets/screenshots/function_name_in_docker_2022-02-27.png new file mode 100644 index 0000000..38cef6d Binary files /dev/null and b/docs/assets/screenshots/function_name_in_docker_2022-02-27.png differ diff --git a/docs/cli.md b/docs/cli.md deleted file mode 100644 index 05945e0..0000000 --- a/docs/cli.md +++ /dev/null @@ -1,283 +0,0 @@ -# CLI documentation - -> Here lie the descriptions of commands autogenerated from the source code of the package. - -# `functions` - -Run script to executing, testing and deploying included functions. - -**Usage**: - -```console -$ functions [OPTIONS] COMMAND [ARGS]... -``` - -**Options**: - -* `--install-completion`: Install completion for the current shell. -* `--show-completion`: Show completion for the current shell, to copy it or customize the installation. -* `--help`: Show this message and exit. - -**Commands**: - -* `build` -* `gcp`: Deploy functions in GCP -* `list`: List existing functions -* `new`: Factory method for creating new functions -* `remove` -* `run`: Start a container for a given function -* `stop` - -## `functions build` - -**Usage**: - -```console -$ functions build [OPTIONS] FUNCTION_PATH -``` - -**Arguments**: - -* `FUNCTION_PATH`: Path to the functions you want to build [required] - -**Options**: - -* `--force`: [default: False] -* `--help`: Show this message and exit. - -## `functions gcp` - -Deploy functions in GCP - -**Usage**: - -```console -$ functions gcp [OPTIONS] COMMAND [ARGS]... -``` - -**Options**: - -* `--help`: Show this message and exit. - -**Commands**: - -* `delete`: Deletes a functions deployed to GCP -* `deploy`: Deploy a functions to GCP -* `describe`: Returns information about a deployed function -* `install`: Install required libraries -* `logs`: Reads log from a deployed function -* `update`: Update required libraries - -### `functions gcp delete` - -Deletes a functions deployed to GCP - -**Usage**: - -```console -$ functions gcp delete [OPTIONS] FUNCTION_NAME -``` - -**Arguments**: - -* `FUNCTION_NAME`: Name of the function you want to remove [required] - -**Options**: - -* `--help`: Show this message and exit. - -### `functions gcp deploy` - -Deploy a functions to GCP - -**Usage**: - -```console -$ functions gcp deploy [OPTIONS] FUNCTION_DIR -``` - -**Arguments**: - -* `FUNCTION_DIR`: Path to the functions you want to deploy [required] - -**Options**: - -* `--service [cloud_function]`: Type of service you want this resource to be deploy to -* `--help`: Show this message and exit. - -### `functions gcp describe` - -Returns information about a deployed function - -**Usage**: - -```console -$ functions gcp describe [OPTIONS] -``` - -**Options**: - -* `--help`: Show this message and exit. - -### `functions gcp install` - -Install required libraries - -**Usage**: - -```console -$ functions gcp install [OPTIONS] -``` - -**Options**: - -* `--help`: Show this message and exit. - -### `functions gcp logs` - -Reads log from a deployed function - -**Usage**: - -```console -$ functions gcp logs [OPTIONS] -``` - -**Options**: - -* `--help`: Show this message and exit. - -### `functions gcp update` - -Update required libraries - -**Usage**: - -```console -$ functions gcp update [OPTIONS] -``` - -**Options**: - -* `--help`: Show this message and exit. - -## `functions list` - -List existing functions - -**Usage**: - -```console -$ functions list [OPTIONS] -``` - -**Options**: - -* `--help`: Show this message and exit. - -## `functions new` - -Factory method for creating new functions - -**Usage**: - -```console -$ functions new [OPTIONS] COMMAND [ARGS]... -``` - -**Options**: - -* `--help`: Show this message and exit. - -**Commands**: - -* `http`: Creates a new http directory -* `pubsub`: Creates a new pubsub directory - -### `functions new http` - -Creates a new http directory - -**Usage**: - -```console -$ functions new http [OPTIONS] FUNCTION_NAME -``` - -**Arguments**: - -* `FUNCTION_NAME`: Name of a function in alphabetic constrain [i.e new-function] [required] - -**Options**: - -* `--dir PATH`: Directory that will be used as a root of the new function -* `--help`: Show this message and exit. - -### `functions new pubsub` - -Creates a new pubsub directory - -**Usage**: - -```console -$ functions new pubsub [OPTIONS] FUNCTION_NAME -``` - -**Arguments**: - -* `FUNCTION_NAME`: Name of a function in alphabetic constrain [i.e new-function] [required] - -**Options**: - -* `--dir TEXT`: Directory that will be used as a root of the new function -* `--help`: Show this message and exit. - -## `functions remove` - -**Usage**: - -```console -$ functions remove [OPTIONS] FUNCTION_NAME -``` - -**Arguments**: - -* `FUNCTION_NAME`: Name of the function you want to remove [required] - -**Options**: - -* `--help`: Show this message and exit. - -## `functions run` - -Start a container for a given function - -**Usage**: - -```console -$ functions run [OPTIONS] FUNCTION_NAME -``` - -**Arguments**: - -* `FUNCTION_NAME`: Name of the function you want to run [required] - -**Options**: - -* `--help`: Show this message and exit. - -## `functions stop` - -**Usage**: - -```console -$ functions stop [OPTIONS] FUNCTION_NAME -``` - -**Arguments**: - -* `FUNCTION_NAME`: Name of the function you want to stop [required] - -**Options**: - -* `--help`: Show this message and exit. diff --git a/docs/examples.md b/docs/examples.md index 22dd2ca..8846ade 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -5,5 +5,3 @@ Here are the examples you might use to your use case. * [Simple `http` function locally](examples/http_function.md). * [Simple `pubsub` function locally](examples/pubsub_locally.md). * [Deploying a function to GCP](examples/deploying_to_gcp.md). - -# Slack integration function diff --git a/docs/examples/add_existing_function.md b/docs/examples/add_existing_function.md index e69de29..9a8e103 100644 --- a/docs/examples/add_existing_function.md +++ b/docs/examples/add_existing_function.md @@ -0,0 +1,20 @@ +# Adding an existing function resource to the registry + +If you ever found yourself wanting to add existing function to `functions` registry to simplify manipulation you can follow this example. + +## Prerequisites + +* an existing directory holding function like files. + +## Add a function + +Run the `add` command + +```console + +> functions add ~/tests/http-function/ + +Successfully added a function to the registry. The name of the functions is -> http-function +``` + +The output might differ depending on the answers you give during the terminal prompts. diff --git a/docs/examples/deploy_function_to_gcp.md b/docs/examples/deploy_function_to_gcp.md new file mode 100644 index 0000000..d4a9ce1 --- /dev/null +++ b/docs/examples/deploy_function_to_gcp.md @@ -0,0 +1,43 @@ +# Deploying a function to GCP + +A prerequisite to this tutorial is a requirement that you have an existing function in your registry. We will use a function from the `http` tutorial. + +As you will see it is very straightforward to deploy a resources to GCP using `functions`. + +Additional assumptions include: + +* you have a valid GCP account in `gcloud` scope capable of deploying cloud function resources. +* you have `GCP` available as your component. +* internet connection! + +## Run the deployment + +It is a simple as running + +```console +> functions gcp deploy http-function + +'http-function' functions has been deployed to GCP! +``` + +## Validate the deployment + +To validate the state of all of the functions run + +```console +> functions list + +Function - http-function | Local - STOPPED | GCP - DEPLOYED +``` + +## Remove a cloud function resource + +To remove a cloud function resource run + +```console +> functions gcp delete http-function + +'http-function' has been removed from GCP! +``` + +**Note**: You will not be able to remove resources not deployed with `functions`. diff --git a/docs/examples/http_function.md b/docs/examples/http_function.md index 09e7d8a..8732aa2 100644 --- a/docs/examples/http_function.md +++ b/docs/examples/http_function.md @@ -1,70 +1,94 @@ -# Http function (locally) +# Create a `http` function and run it locally -### Create a new http function directory. +HTTP functions can be created to handle HTTP requests. +They are the easiest function to validate working locally. -``` -functions new http http-function --dir ~/test/ +Follow these steps to get it to work locally. + +## Create a new `http` function using the `new` command + +```console +functions new http http-function --dir ~/tests/ ``` Expected output: -``` -Added a new http function to -> /home/{user}/test/http-function + +```console +Added a new http function to -> /home/{user}/tests/http-function ``` -### Build the newly created http function (This may take a little bit of time). +## Build the newly created http function -``` -functions build ~/test/http-function/ +In console run + +```console +functions build http-function ``` -**Note** - You can add `--show-logs` to see preview the build process. +You are building a docker image in the background using the default settings. -Expected output: -``` +**Note**: You can add `--show-logs` to see preview the build process. + +Expected output + +```console Successfully build a function's image. The name of the functions is -> http-function ``` +## Running the newly built function`http-function` -### Running `http-function`. +To the results and your newly build function, run the following -``` +```console functions run http-function ``` -Expected output: -``` +Expected output + +```console Function (http-function) has started. Visit -> http://localhost:8080 ``` -### Work validation. +### Work validation -``` +To validate that the function is running you can run + +```console functions list ``` -Expected output: -``` -Function - http-function | Status - Running +Expected output + +```console +Function - http-function | Local - RUNNING | GCP - UNKNOWN ``` -You should be able to view the newly created functions running by going to `http://localhost:8080`. +You should be able to view the newly created functions running by going to `http://localhost:8080`. ![Hello world on localhost](https://user-images.githubusercontent.com/20417569/139000266-f596a100-c018-4591-83c5-d131b778a24e.png) -### Cleaning up. +### Cleaning up -Once you done with the function, it is a practice to remove unused. +Once you done with the function, it is a practice to remove an unused resource. -``` +```console functions stop http-function ``` to stop the running function. -``` +```console functions remove http-function ``` -to remove the `http-function` from the local registry. +to remove the `http-function` from the local registry + +or + +```console +functions delete http-function +``` + +to delete the function with underlying images, containers and even files. diff --git a/docs/examples/new_gcp_functions.md b/docs/examples/new_gcp_functions.md deleted file mode 100644 index e69de29..0000000 diff --git a/docs/glossary.md b/docs/glossary.md index f821742..a2f5242 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -3,3 +3,11 @@ Should include all of the terms used in this project. ## Function + +A function is a resource defined by runtime files, a config file and a function name. + +All that we do revolves around this main entity of "function". + +## Registry + +Is a component responsible for persisting information about individual functions. diff --git a/docs/proposals/commands/config.md b/docs/proposals/commands/config.md deleted file mode 100644 index e69de29..0000000 diff --git a/docs/proposals/config.md b/docs/proposals/config.md index 581311a..c145496 100644 --- a/docs/proposals/config.md +++ b/docs/proposals/config.md @@ -1,22 +1,83 @@ -# App configuration wit the `config` module +# Cam we have a persistent state for functions? -* Status: Accepted -* Deciders: [Piotr] +* Status: Proposed +* Deciders: [Piotr] * Date: 2021-11-23 ## Work context -Architecture and features expects a to a place to store files. File like the config file, logs, etc.... In this document we are hoping to find a +In order to enable functionalities like `autocomplete`, `logging` or resources management, we need an easy and accessible place to store information about the state. ## Problem -## Research +We want to have a way of storing information between function executions. -TBU +Our goal is to maintain flexibility and ease of definition for the first implementation. ## Proposal -`functions-cli` stores valuable configurations in an app specific folder. +We look into some of the options considered in building this feature. + +### Database + +Add a database to background to store or retrieve information about the functions. + +Something like a postgres DB? + +Pros: + +* moderate scalability +* safe + +Cons: + +* big effort +* weight of the implementation +* difficulty with justifying using a database for a CLI tool + +### File storage + +Use a file to store the configuration in a unified format - `JSON` / `TOML`. + +Pros: + +* Easy implementation +* All the control +* `cat` to view the content +* One place to store it all + +Cons: + +* Questionable consistency +* "Ups I deleted the file. What now?" + +## Decision + +A `database` system for storing information about the state definitely seems like an overkill. + +It much more achievable to handle a single file write and reads. We can manipulate it, remove it, create again very easily which means we can make mistakes and fix them quickly. + +We are not worried about any consistency issues at this point in time. + +## Implementation + +We define a `config` module. + +This module will store all `functions` specific files. Current files like log files, config file or the registry files, but any other future files if necessary. + +Moreover the `config` module stores said `config.toml` file that holds information about the available components, version information and tool specific configuration. + +Module's path will evaluated in another document, but it needs to be saved in a location accessible by multiple operating systems. + +**The module is created on the first run of the `functions` terminal interface**. + +This is when the components are evaluated and the information is saved on file. If you need to reevaluate it is possible to do by deleting the file. + +## Summary + +The solution chosen should be a simple way of handling the problem enabling all the derived functionalities. If however this becomes insufficient another way should be easily configurable. + +We will continue to monitor its validity in the `Internal Alpha` release and suggest improvements and soon the problems become apparent. diff --git a/docs/proposals/config_path.md b/docs/proposals/config_path.md index c0c614b..a6602fa 100644 --- a/docs/proposals/config_path.md +++ b/docs/proposals/config_path.md @@ -1,30 +1,43 @@ # Where do we store configuration files? * Status: Proposed -* Deciders: [Piotr] +* Deciders: [Piotr] * Date: 2022-01-08 ## Work context -We want the app to have a pseudo state in which information about managed functions are stored to enable better management and enhanced development possibilities. In this document want to tackle the problem of storing all the configuration documents and future application specific files in a directory where it can be access by all components that need it. A unified strategy that works well across different operating systems and specific distributions. +We want the tool to have a easily persistent state in which information about managed functions can be stored to enable better management and enhanced development capabilities. In this document want to find a good solution of storing all the configuration files and future application specific files in a place where it can be access by all components that need it. A unified strategy that works well across different operating systems and specific OS distributions. ## Problem +Since we decided on use a file based system to persist information about the state of the tool, we face a problem of finding the base place to store it. + Unfortunately there is not a single pattern widely approved and implemented by developers working cross systems that instructs an application where to store the config data so that is best for the health of the files and the whole system. -For example linux application developers have a concept of a `SOMETHING` environment variable that usually is empty but if missing they default such actions to the `.config` file on a user's workspace. +For example linux application developers have a concept of a `XDG_CONFIG_HOME` environment variable that usually is empty but if missing they default such actions to the `.config` file on a user's workspace. This however is unique to Linux and does not apply throughout other systems, hence we need to find a way of tackling the issue and enable usage of this application on Windows and macOS systems. -## Research +## Proposal -To be done... +The solution is relatively naive and suggests using a condition based definition of a default system config path. Each OS type will point to a different config path. -## Proposal +```python +# Set system constants based on the current platform +if sys.platform.startswith("win32"): + DEFAULT_SYSTEM_CONFIG_PATH = os.path.join(os.environ["APPDATA"], "config") +elif sys.platform.startswith("linux"): + DEFAULT_SYSTEM_CONFIG_PATH = os.path.join(os.environ["HOME"], ".config") +elif sys.platform.startswith("darwin"): + DEFAULT_SYSTEM_CONFIG_PATH = os.path.join( + os.environ["HOME"], "Library", "Application Support" + ) +else: + DEFAULT_SYSTEM_CONFIG_PATH = os.path.join(os.environ["HOME"], "config") +``` -To be updated... +It is hard to make sure that all the operating systems are accounted for in this definition so an update might be required in the future. [Piotr]: https://github.com/Katolus - diff --git a/docs/proposals/development_branch.md b/docs/proposals/development_branch.md deleted file mode 100644 index f522dd0..0000000 --- a/docs/proposals/development_branch.md +++ /dev/null @@ -1,29 +0,0 @@ -# `development` as default branch - -* Status: Proposed -* Deciders: [Piotr] -* Date: 2021-11-23 - -## Proposal - -We propose to set up the main branch that the majority work from PRs and contributions are commited to be the `development` branch. - -The idea behind this decision is a attempt to create a more accessible development stream. - -The vision being that the `master` branch should be focused on the valuable releases associate with package versions releases. - -All the tooling that needs to be run to enforce quality should be associated with the `master` branch. - -The `master` branch should be protected. - -All this should enable a more flexible work around smaller issues in a singular pool of developer(s). - -All the PRs should target the `development` branch as base and a release PR should be a combination of sub PRs on a per ready basis. - -This means that the tests for various python versions of the package and other heavy duty actions will only be trigger on merges to master. - -__The details for this process are yet to be established.__ - - - -[Piotr]: https://github.com/Katolus diff --git a/docs/proposals/exceptions.md b/docs/proposals/exceptions.md index 19d7d24..578d001 100644 --- a/docs/proposals/exceptions.md +++ b/docs/proposals/exceptions.md @@ -1,8 +1,8 @@ # How do we handle exceptions? -* Status: Accepted -* Deciders: [Piotr] -* Date: 2021-11-23 +* Status: Proposed +* Deciders: [Piotr] +* Date: 2021-02-23 ## Work context @@ -12,7 +12,7 @@ We want to be smart about how we use exceptions in the app. We want to use them We know that exceptions in python can be subclassed and caught accordingly to a level of ancestory. -We suggest using this pattern to throw case specific exceptions grouped in by kind and handle the most generic case in case nothing matched. +We suggest using this pattern to throw case specific exceptions grouped in by type and handle the most generic case in case nothing matched. In order to make this happen and support this feature out of the box without having to remember about handling errors, we want to use a decorator that will wrap each respective command. @@ -26,7 +26,39 @@ class DocumentationQualityError(FunctionBaseError): msg_template = 'Message to return to the user on error' ``` +### Implementation + +Error subclasses are included in files named `errors.py`. + +Their base classes takes any key argument passed into the classes and saves it on an instance so that it is available for the templated message and stringifying - `msg_template` variable. + +This means that you can pass any key argument without the necessity of defining the arguments on a base class. + +```python +class IsNotAValidDirectory(FunctionBaseError): + code = "path.invalid_directory" + msg_template = "Path '{path}' is not a valid function directory" + +... + +raise IsNotAValidDirectory(path='/home/paris/') +``` + +This however means that if you do not pass the reference variable, an error class will throw an error. + +All `FunctionBaseError` derived error classes are handled per default handlers defined in the `error_handlers.py` script. If you need to make a default case for handling any errors, use the `error_handler` decorator as per examples in the script file. + +The `error_handler` decorator stores references to errors in a `ERROR_REGISTRY` that is used to capture and handle expected errors in the `handle_error` decorator. + +`handle_error` decorator handles or all core functionalities by default through a wrapper definition in the `FTyper` class. It is defined on the core class to avoid having to define it on all of the resources. + +You can also use `@handle_error` decorator to capture a specific class to be handle in the context of the decorator. + +```python +@handle_error(error_class=CaptureErrorOnlyHere) +def some_edge_case_method(): ... +``` + [Piotr]: https://github.com/Katolus - diff --git a/docs/proposals/flows_and_actions.md b/docs/proposals/flows_and_actions.md new file mode 100644 index 0000000..d0e2852 --- /dev/null +++ b/docs/proposals/flows_and_actions.md @@ -0,0 +1,37 @@ +# Flow and Actions + +* Status: Experimental +* Deciders: [Piotr] +* Date: 2022-02-01 + +## Context + +Throughout the functions interactions there are topics and places where a certain "action" is repeated. Say we want to add a function into the registry, we might want to as well sync the registry with components which will entitle adding sync resources in a similar way to the `add` command mentioned before. + +## Problem + +We want to ensure there is a standard way of handling user "actions" and "flows". + +With this approach we hope to avoid code complexity, edge cases, duplication and handle more cases on a larger scale. + +The difficulty comes from properly scoping out said actions and flows. The concept fits wonderfully with the idea of modularity and DRY standards. + +## Criteria + +* Keep it simple. +* Keep it readable. + +## Proposal + +We suggest grouping user actions and flows into scripts that can be easily used in other modules. + +* `actions.py` - holds logic that includes a user interaction like asking for a function name. We need to know that in more than one place. Once we specify it as an action, we can use the same action in all of the places. +* `flows.py` - holds definitions of procedures that are common throughout the code, like loading a config file, exiting if function name already exists or adding a function to the registry. + +## Our worries + +We worry that we will eventually come across a circular dependency issue that is hard to tackle without a rewrite of the code. This should be avoidable if the code layout is well structured, but that is a challenge on it's own. + + + +[Piotr]: https://github.com/Katolus diff --git a/docs/proposals/function_invocation.md b/docs/proposals/function_invocation.md index 1248577..a4124d3 100644 --- a/docs/proposals/function_invocation.md +++ b/docs/proposals/function_invocation.md @@ -1,11 +1,51 @@ # How do we invoke a function? -* Status: Accepted -* Deciders: [Piotr] -* Date: 2021-11-23 +* Status: Proposed +* Deciders: [Piotr] +* Date: 2021-02-26 +## Context -To be talked about how the function is invoked an why that way? +We want to define and document what is the process of managing function resources like `docker` containers or GCP resources. How do we call and reference any of these resources? + +## Problem + +In order to use a docker image and spin a container, we have to have a way of "tagging" the built image. This comes with additional difficulties of using `names vs ids`, `ensuring uniqueness` or `enforcing syntax`. + +## Criteria + +We want to make sure that: + +* it is a easy and intuitive for a user to invoke a function +* the results won't produce any unwanted outcomes + +## Backfilled decision + +This is one of the solutions that were defined prior to making any decisions and were never properly evaluated. It lead to a default implementation of a simple most of intuitive solution of using the function names to manage them. + +## Implementation + +We use function names as `unique` indicators for filter to a given resource. + +Function names are defined on creation or add of a function to the registry. This cannot be changed without updating the configuration file. + +A function that was added with a `http` name will invoked with same name. + +```console +functions run http +``` + +The same name will be used to tag resources in components like `docker` or `GCP`. + +![function name in docker](../assets/screenshots/function_name_in_docker_2022-02-27.png) + +## Future work + +Since the initial implementation we discovered potential problems with solutions. We list here problems which we would want to tackle in the future. + +### Define names, but use IDs + +Instead of using names as a single source of truth we might want to use `id` a filter and uniqueness indicator but invocate the functions via a name. The main problem with this solution is invocation context. On `functions run http` will face an issue if there is another function with the same name. diff --git a/docs/proposals/function_registry.md b/docs/proposals/function_registry.md index 279b3ae..8562a4a 100644 --- a/docs/proposals/function_registry.md +++ b/docs/proposals/function_registry.md @@ -1,8 +1,8 @@ # How do we store information about the managed functions? -* Status: Accepted -* Deciders: [Piotr] -* Date: 2021-11-23 +* Status: Proposed +* Deciders: [Piotr] +* Date: 2021-02-24 ## Problem statement @@ -23,11 +23,27 @@ Things to consider: Each time a CLI is triggered a file will be read and loaded into the memory so that the data about interacted functions is available in the scope of the program. The marshalling and un-marshalling will be performed in a JSON format. +## Implementation + +The core functionality of this feature is defined by the `FunctionRegistry` class. + +This class should be used to interact with `registry.json` file. This file holds information about the all the functions in the scope of the tool. + +Details of each of the `functions` are defined by an instance of the `FunctionRecord` class and a stored in a JSON format. This class is capable of loading, saving and displaying info about the function in a terminal. It's purpose is to store information about the `name`, `config` and `status`. + +### Function vs FunctionRecord + +`Function` and `FunctionRecord` can easily be confused so it is important to clearly define their purpose. + +`Function` is class that defines variables methods of a instantiated `function` in run time. Use this method to manipulate the function on runtime. + +`FunctionRecord` is a class that describes configuration and metadata about the state of the function; as far as registry information is capable. Use this class to handle, pass, update and use function's metadata. + ## Outcomes The truth is that is not the most stable solution, but it offers transparency in viewing the saved information in a file as well as direct modification as needed. -In a `user` scenario this file should be accessed and edited. +In a `user` scenario this file should not be accessed nor edited. diff --git a/docs/proposals/logging.md b/docs/proposals/logging.md index c23c051..ec7e5af 100644 --- a/docs/proposals/logging.md +++ b/docs/proposals/logging.md @@ -1,8 +1,8 @@ # Logging in `functions` -* Status: Accepted +* Status: Proposal * Deciders: [Piotr] -* Date: 2021-11-23 +* Date: 2022-02-23 ## Work context @@ -14,17 +14,31 @@ We suggest creating a storing a simple file based log using that Python's standa The files should take a lot of space and should ensure that the information is properly rotated. -The file is to be stored in the `config` module's directory path. +The file is named - `functions.log` by default and is stored in the `config` module's directory path. -### Levels of logging +### Default level -* Debug: Use this level for anything that happens in the program. +The default level for command execution is `INFO`. It means that none of the debug statements will be visible in the a command's output. -* Info: Use this level to record all actions that are user driven or system specific, such as regularly scheduled operations. +### Format -* Warning: Use this level to record all irregular or undesired actions. +A log format takes a form of three parts: -* Error: Use this level to record any error that occurs. +```console +{time} - {level} - {message} + +2022-02-27 01:08:33,367 - DEBUG - Running application in info logging level. +``` + +### Command - logs + +The file can be used by any regular means, like reading the file, output in terminal or terminal tailing. + +In order to make this more straightforward, we added a `functions logs` command that would output and tail the log file so that you don't need to do anything else. + +### Access + +You can get access to the debug logging output on command execution by passing the `--verbose` flag as an option to the root `functions` command. diff --git a/docs/templates/commands.md b/docs/templates/commands.md new file mode 100644 index 0000000..88c38a8 --- /dev/null +++ b/docs/templates/commands.md @@ -0,0 +1,33 @@ +# Title + +* Status: [ + Proposed + | Accepted + | Implemented + | Experimental + | Rejected + | Retired + | Superseded +] +* Deciders: [Piotr] +* Date: 2022-02-01 + +## Context and Problem Statement + +TBU... + +## Proposed Solution + +TBU... + +## Implementation Details + +TBU... + +## Benefits + +TBU... + + + +[Piotr]: https://github.com/Katolus diff --git a/docs/templates/proposals.md b/docs/templates/proposals.md index 0ced354..51380f4 100644 --- a/docs/templates/proposals.md +++ b/docs/templates/proposals.md @@ -1,8 +1,16 @@ # Title -* Status: Proposed +* Status: [ + Proposed + | Accepted + | Implemented + | Experimental + | Rejected + | Retired + | Superseded +] * Deciders: [Piotr] -* Date: 08-01-2022 +* Date: 2022-02-01 ## Context diff --git a/docs/troubleshooting/gcloud_issues.md b/docs/troubleshooting/gcloud_issues.md new file mode 100644 index 0000000..9fb5e08 --- /dev/null +++ b/docs/troubleshooting/gcloud_issues.md @@ -0,0 +1 @@ +# Handy document for handling common GCP issues diff --git a/functions/logs.py b/functions/logs.py index 61c5fb1..2960747 100644 --- a/functions/logs.py +++ b/functions/logs.py @@ -52,9 +52,7 @@ def filter(self, record: logging.LogRecord) -> bool: delay=True, ) f_handler.setLevel(logging.DEBUG) -f_handler.setFormatter( - logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s") -) +f_handler.setFormatter(logging.Formatter("%(asctime)s - %(levelname)s - %(message)s")) logger.addHandler(f_handler) diff --git a/functions/main.py b/functions/main.py index 477ca2e..01626c3 100644 --- a/functions/main.py +++ b/functions/main.py @@ -93,9 +93,7 @@ def build( autocompletion=autocomplete_registry_function_names, callback=check_if_function_can_be_built, ), - disable_logs: bool = typer.Option( - False, "--disable-logs", help="Disable build output" - ), + show_logs: bool = typer.Option(False, "--show-logs", help="Show build output"), ) -> None: """ Build a function. @@ -103,7 +101,7 @@ def build( # Get the absolute path function = Function(function_name) - function.build(show_logs=disable_logs) + function.build(show_logs=show_logs) user.inform( f"{styles.green('Successfully')} build a function's image."