Skip to content

Latest commit

 

History

History
646 lines (447 loc) · 17.6 KB

CONTRIBUTING.md

File metadata and controls

646 lines (447 loc) · 17.6 KB

Contributing to Besom

Thank you for contributing to help make Besom better. We appreciate the help!

Code of Conduct

Please make sure to read and observe our Contributor Code of Conduct.

Communications

We discuss features and file bugs on GitHub via Issues.

Issues

Feel free to pick up any existing issue that looks interesting to you or fix a bug you stumble across while using Besom. No matter the size, we welcome all improvements.

Before investing a lot of time, please let us know, so we can discuss the issue together.

Feature Work

For larger features, we'd appreciate it if you open a new issue before investing a lot of time, so we can discuss the feature together.

Please also be sure to browse current issues to make sure your issue is unique, to lighten the triage burden on our maintainers.

Adding examples

Start with a template that is closest to your use case, and modify it, e.g.:

mkdir examples/my-example
cd examples/my-example
pulumi --logtostderr new https://github.com/VirtusLab/besom/tree/main/templates/gcp

Branching and versioning strategy

We mostly follow the Pulumi strategy:

  • main branch contains current *-SNAPSHOT version
  • vX.Y.Z tag marks the X.Y.Z release
  • release/vX.Y.Z branch contains the X.Y.Z release
  • PRs must have a prefix with the name of the author and issue number e.g. pprazak/123-fix-bug

Note

Please make sure to tag first before creating a release branch.

Versioning is done using Semantic Versioning, with following additions:

  • x.y.z for core version, where:
    • x no guarantees are made about compatibility,
    • y should not break source compatibility,
    • z should not break binary compatibility
  • a.b.c-core.x.y for provider version, where a.b.c is the schema version
  • *-SNAPSHOT versions are used for development versions

Developing

Setting up your development environment

You will want to install the following on your machine:

  • Pulumi CLI 3.30.0 or higher
  • Scala-CLI 1.0.4 or higher
  • JDK 11 or higher (GraalVM version is recommended)
  • Scala 3.3.1 or higher
  • Go 1.20 or higher
  • protoc 24.3 or higher
  • just 1.14.0 or higher
  • git 2.37.1 or higher
  • unzip
  • coursier
  • gh

Mac OS

brew install pulumi/tap/pulumi
brew install Virtuslab/scala-cli/scala-cli
brew install coursier/formulas/coursier
brew install just
brew install java11
brew install sbt
brew install go
brew install git
brew install unzip
brew install protobuf # protoc
brew install gh

Preparing a pull request

  1. Ensure running just passes with no issues.
  2. Ensure the branch name is prefixed with your name and contains issue number, e.g. johndoe/123-fix-bug.

Understanding the repo structure

  • core contains the core of Scala SDK for Pulumi
  • besom-cats contains the cats effect integration for Scala SDK for Pulumi
  • besom-zio contains the ZIO integration for Scala SDK for Pulumi
  • codegen contains Scala code generation fo Besom (e.g. used for provider SDKs)
  • proto contains a copy of Pulumi protobuf definitions (generated by a script)
  • compiler-plugin contains the compiler plugin for Scala SDK for Pulumi
  • language-plugin contains the Golang language host provider plugin for Scala SDK for Pulumi
  • templates define starter project templates
  • examples contains examples of using Besom
  • scripts contains various scripts used for development

Working with local dependencies

There are Just targets that help rebuild all Scala and Go packages from source. For correct cross-referencing in examples and tests, Scala packages need to be installed into the local ~/.ivy repo.

The targets do not yet understand dependencies accurately, so you may need to re-run to make sure changes to Besom SDK or providers are rebuilt.

As for Go changes, Pulumi CLI will respect the version of pulumi-language-scala it finds in PATH over the default version it ships with. When testing changes to the language provider, you therefore might need to manipulate PATH to prefer the local version.

Publish locally core SDK packages

Publish locally and install necessary Besom packages:

just publish-local-all

Publish locally additional SDKs

You have to generate an SDK for a provider of your choice, use just cli, e.g.:

export GITHUB_TOKEN=$(gh auth token)
just cli packages local azure docker gcp kubernetes random tls

To generate all Provider SDKs (takes a very long time):

export GITHUB_TOKEN=$(gh auth token)
just cli packages local-all

Test locally

To test locally, you need to install the packages into your local ~/.ivy repo. This was done in previous steps.

just test-all

Working with published dependencies

Release builds of the Besom SDK are published to Maven Central, and snapshots to GitHub Packages as org.virtuslab::besom-core:X.Y.Z and org.virtuslab::besom-<package>:A.B.C-core.X.Y.

Language host provider is published to Maven as pulumi-language-scala-vX.Y.Z-OS-ARCH.tar.gz.

To use development version of the language host provider:

pulumi --logtostderr plugin install language scala $(cat version.txt) --server github://api.github.com/VirtusLab/besom

To use development version of the Besom SDKs add repository in your project.scala:

//> using repository sonatype:snapshots

or use a command line option:

--repository=sonatype:snapshots

To use development version of an example:

pulumi --logtostderr new https://github.com/VirtusLab/besom/tree/main/templates/aws

Publishing a release

After all the testing is done, you can publish a release.

just publish-local-all
just test-all

Publish fresh packages locally

It is recommended to use just power-wash before publishing a release:

just power-wash

Make sure the repository is clean and there are no uncommitted changes:

git status -s

Bump Besom version (skip for SNAPSHOT re-release)

To bump Besom version in all project.scala and version.txt files:

export GITHUB_TOKEN=$(gh auth token)
just cli version bump X.Y.Z

Publish the new version of SDKs locally to test and provide fresh dependencies for scripts:

just publish-local-all

Update dependencies versions in all project.scala files (optional for patch versions)

This is most useful for examples and templates, and integration tests:

export GITHUB_TOKEN=$(gh auth token)
just cli version update

Update versions in all other places (skip for SNAPSHOT)

Manually update versions in all other places, specifically documentation and website, using find&replace.

Look for:

  • DO NOT change historical versions in CHANGELOG.md by mistake
  • X.Y.Z - the besom version
  • core.X.Y - core part of the provider version
  • update the provider versions in README.md

Create release branch

git checkout -b release/v$(cat version.txt)

Create a release draft on GitHub and publish language host

We need this done early to be able to publish the language host provider binaries.

just upsert-gh-release

Publish language host provider binaries:

just publish-language-plugins-all

Re-publish local language host provider binary (to fix local setup):

just install-language-plugin

Publish SDKs to maven

Publishing to maven requires:

  • OSSRH_USERNAME - the Sonatype username
  • OSSRH_PASSWORD - the Sonatype password
  • PGP_KEY_ID - the signing key id (gpg --list-keys or gpg --show-keys)
  • PGP_PASSWORD - the signing key passphrase

Wait for the CI to pass Besom build and test / build before proceeding.

Publish main SDK packages to Maven:

just publish-maven-all

Publish packages (optional for patch versions)

To publish critical provider package(s):

export GITHUB_TOKEN=$(gh auth token)
just clean-out
just cli packages maven aws awsx azure gcp docker kubernetes random command tls eks
just cli packages maven aws-native google-native azure-native

Tip: to check what packages are required for examples and templates use:

just cli version summary examples
just cli version summary templates

Publish all packages:

export GITHUB_TOKEN=$(gh auth token)
just clean-out cli packages maven-all

Cation: publishing to Maven Central is irreversible. Tip: it's safer to publish the packages on-by-one or in batches due to how Maven Central behaves. Note: azure-native publishing takes a long time (1-2 hours) it is recommended to handle it separately.

In case of any issues, you can try to resolve the issues manually at https://oss.sonatype.org/index.html#stagingRepositories.

Finish the release

Finish the release on GitHub manually, make sure the changelog is correct and correct git tag was created.

According to our Git branching and versioning strategy, the release branch should be merged after the tag is created. Make sure to bump the git tag because GitHub Release probably already created the tag.

git push --set-upstream origin release/v$(cat version.txt)
git tag -f v$(cat version.txt)
git push -f origin v$(cat version.txt)

Make sure to DO NOT squash but merge release/vX.Y.Z branch into main otherwise the tag will not propagate to main. Make sure to DO NOT (auto) delete the release branch on merge, but on the other hand, let the release/vX.Y.Z-SNAPSHOT be deleted on merge to not pollute git.

After the release

After the release, you can bump the version to the next -SNAPSHOT version:

just cli version bump X.Y.Z-SNAPSHOT

Remember to release the snapshots to maven.

Testing examples locally

Every example is a valid Pulumi program that can be tested by manually doing pulumi up in the right folder.

cd examples/<example-name>
pulumi up

Here is a just helper to run the automated testing:

just test-example aws-webserver

Testing templates locally

See templates/README.md on how to manually test template changes locally.

Similarly to examples, Just helper targets are provided to automatically test templates, for example to test templates/default run:

just test-template default

Testing codegen locally

Codegen is tested by running:

just test-integration-codegen

To update test schema files from upstream:

just copy-test-schemas

Testing Protobuf/gRPC codegen locally

Protobuf/gRPC codegen is tested by running:

just cli proto all
just compile-core

Setting up the code editor

Both IDEs support rely on BSP and is experimental.

BSP setup with scala-compose

Build experimental scala-compose and place on $PATH:

git clone git@github.com:VirtusLab/scala-compose.git
cd scala-compose
cat << EOF | git apply
> diff --git a/project/publish.sc b/project/publish.sc
> index e00f81ca..619d4c99 100644
> --- a/project/publish.sc
> +++ b/project/publish.sc
> @@ -113,8 +113,9 @@ def finalPublishVersion = {
>      }
>    else
>      T {
> -      val state = VcsVersion.vcsState()
> -      computePublishVersion(state, simple = true)
> +      // val state = VcsVersion.vcsState()
> +      // computePublishVersion(state, simple = true)
> +      "1.0.4"
>      }
>  }
>
> EOF
./mill -i show scala-compose.nativeImage
cp out/scala-compose/base-image/nativeImage.dest/scala-cli ~/bin/scala-compose

Use scala-compose in besom directory:

scala-compose setup-ide --conf-dir .

IntelliJ setup

IntelliJ support is experimental.

  1. Make sure you have the latest IntelliJ
  2. Install Scala plugin and set update chanel to "Nightly Builds"
  3. Use BSP with scala-cli (also see IntelliJ documentation)

To make sure you have .bsp directories, by running:

just setup-intellij

Now open the project in IntelliJ. If neede and add modules manually using "Project Structure > Import Module" dialog.

Additionally, please set scalafmt as the formatter.

VSCode setup

If you are using VSCode:

  1. Install Metals
  2. Open the project in Metals.

Make sure you have .bsp directory before you open the project in VSCode.

This might not be enough if your infrastructure is just a part (a module) of your existing Scala project. For this to work you have to make your build tool aware of the infrastructure code, for sbt create a corresponding module:

lazy val infra = project.in(file("infrastructure")).settings(
libraryDependencies ++= Seq(
 "org.virtuslab" %% "besom-kubernetes" % "0.1.0", // or any other sdk you are using
 "org.virtuslab" %% "besom-core" % "0.1.0"
))

This just informs your IDE about the existence of the infrastructure module, DO NOT remove dependencies from project.scala, because they are necessary in both places.

Troubleshooting

If you suspect the issue is related to serialization, try to skip the preview (dry run is known to be problematic):

pulumi up --skip-preview

GitHub might be throttling your requests

If you see an error like this:

  • git failed to clone or checkout the repository
  • pulumi failed to download the provider (401) GitHub might be throttling your requests, try to authenticate:
export GITHUB_TOKEN=$(gh auth token)

Verbosity and debugging options

Pulumi has a few options that can help with debugging.

CLI command line

You can pass debug options to any pulumi CLI command, e.g.:

pulumi up -v9 --logtostderr 2> log.txt
less -R log.txt

Use the flag --logflow to apply the same log level to resource providers (but not a language provider).

Pulumi.yaml runtime.options

You can set runtime.options in Pulumi.yaml to pass options to the language host provider, e.g.:

name: example
runtime:
  name: scala
  options:
    logtostderr: true
    v: 5

Environment variables

  • PULUMI_BESOM_LOG_LEVEL - for setting Besom log level (default is WARN)
  • PULUMI_ENABLE_TRACE_LOGGING_TO_FILE - for enabling Besom trace logging to file
  • PULUMI_DEBUG_COMMANDS=1 - for activating hidden debugging Pulumi CLI commands
  • TF_LOG=TRACE - for debugging Terraform-based provider
  • PULUMI_SKIP_UPDATE_CHECK=true - to skip Pulumi update check

More environment variables can be found in Pulumi documentation.

Tracing

To collect and view a trace:

pulumi up --tracing=file:./up.trace
PULUMI_DEBUG_COMMANDS=1 pulumi view-trace ./up.trace

Downgrading Pulumi on Mac OS

As a workaround one can downgrade pulumi to a version, e.g. 3.94.2 using a commit hash:

curl -L -O https://raw.githubusercontent.com/Homebrew/homebrew-core/69b97f26bc78cf68eb30eedd0ca874b6e1914b19/Formula/p/pulumi.rb
brew install pulumi.rb
rm pulumi.rb

Compilation issues

Remove .scala-build, e.g.:

rm -rf core/.scala-build

To restart bloop compilation server:

scala-cli bloop exit

To clean the builds:

just clean-all

If a deep cleaning needed:

just power-wash

To set bloop verbosity:

scala-cli setup-ide -v -v -v .

To use a nightly version of Scala compiler:

scala-cli compile -S 3.nightly .

To increase Scala compiler verbosity:

scala-cli compile --scalac-option -verbose .

To inspect a running JVM byt its PID use jcmd, e.g.:

jcmd 25776 VM.flags
jcmd 25776 GC.heap_info

Implementation details

Serde - gRPC/Protobuf serialization and deserialization

The most important information about our serde:

  • unknown values are neither empty nor non-empty - we simply don't know
  • maps (structs) don't preserve Null protobuf value (but we do preserve unknown values)
  • upstream uses special structures signatures besom.internal.Constants.SpecialSig to encode internal metadata that Pulumi uses

Serialization main entry points:

Deserialization main entry points:

Other important files:

Getting Help

We are sure there are rough edges, and we appreciate you helping out. If you want to reach out to other folks in the Besom community please go to GitHub via Issues.