From 39d5f7d02f42d3354d4e9ee5c351d12b82dff755 Mon Sep 17 00:00:00 2001 From: Ioannis Karasavvaidis <32846251+CoMfUcIoS@users.noreply.github.com> Date: Wed, 25 Sep 2024 09:44:18 +0100 Subject: [PATCH] Adding support for legacy compilers (#448) (#495) * Adding support for legacy compilers (#448) * Adding support for legacy compilers * Adding inventoryfile to test upgrade legacy workflow * feat(documentation): add instructions for converting compilers to legacy This commit introduces a new section in the `convert.md` documentation. It provides instructions on how to convert compilers to legacy compilers for Puppet Enterprise installations using puppetlabs-peadm version 3.21 or later, as well as for versions prior to 3.21. The new section includes specific commands to run and references to other relevant documentation. * Fixing lint and regenerating referencemd * PE-38772 Node groups added for legacy compilers (#455) * Fixing typo for parameter in docs --------- Co-authored-by: Ioannis Karasavvaidis Co-authored-by: Neil Anderson Co-authored-by: Aaron Shannon * Updating groups to cover ha scenarios * PE-38768 classify compilers task added (#467) * (PE-38767) Adding legacy compilers to get_peadm_config (#456) Added legacy compilers section, and updated compilers with legacy compilers oid Co-authored-by: Neil Anderson * (PE-38770) Install Plan accepts legacy_compilers key (#474) * feat(peadm): add support for legacy compilers - Added `legacy_compilers` parameter to `peadm::install`, `peadm::subplans::install`, and `peadm::subplans::configure` plans. - Updated `peadm::assert_supported_architecture` function to handle `legacy_compilers`. - Modified various plans to include `legacy_compilers` in the installation and configuration processes. - Updated documentation to reflect the changes and added examples for `legacy_compilers`. This change allows the PEADM module to support legacy compilers in addition to the standard compilers. * feat(assert_supported_architecture): normalize and combine compiler variables - Normalize `$legacy_compilers` and `$compiler_hosts` to arrays, handling both strings and arrays. - Combine the normalized arrays into `$all_compilers`. - Set `$has_compilers` to `undef` if `$all_compilers` is empty, otherwise set it to `true`. - Update tests to cover scenarios with both `compiler_hosts` and `legacy_compilers`. This change ensures that the function correctly handles different types for compiler variables and improves robustness. * Updated REFERENCE.md to document the new parameter. Renamed the variable `legacy_targets` to `legacy_compiler_targets` across multiple plans for better clarity and consistency. This change affects the following plans: - convert_compiler_to_legacy.pp - subplans/configure.pp - subplans/install.pp The new variable name more accurately reflects its purpose, which is to target legacy compiler hosts. * docs(install): correct grammar in installation requirements section Corrected the grammar in the installation requirements section by changing "Puppets must not be installed" to "Puppet must not be installed" for clarity and accuracy. * (PE-38771) Convert plan accepts legacy compilers key in params.json (#476) * feat(convert): add support for legacy compilers - Introduced `legacy_compilers` parameter to handle legacy compiler hosts. - Added logic to filter and categorize legacy compiler targets. - Updated certificate modification steps to include legacy compiler targets with appropriate extensions. * feat(convert): ensure all targets are up-to-date post conversion - Added a step to run puppet on all targets after restarting services. - Ensures all targets are fully up-to-date after conversion. * docs(convert): update documentation for PEADM module - Corrected the module name from 'peadm' to 'PEADM' for consistency. - Added 'legacy_compilers' section in the example JSON parameters. - Improved clarity in the description of the `begin_at_step` parameter. - Fixed minor grammatical issues in the instructions for converting compilers to legacy. * - Updated REFERENCE.md to include the new parameter. * feat(convert): add check for legacy compilers (#483) - Introduced a new task `check_legacy_compilers.rb` to verify legacy compilers. - Updated `convert.pp` to run the new task and display warnings if legacy compilers are detected. * fix: update messaging for legacy compilers check - Clarified the message indicating which legacy compilers are classified as Puppet primary nodes. - Improved the remediation message to specify that the upgrade cannot proceed until the compilers are no longer classified as Puppet primary nodes. * fix: clarify legacy compilers upgrade message - Updated the message to specify that the upgrade can only proceed if the compilers are no longer recognized as Puppet primary nodes. * chore(workflow): update test-upgrade-legacy.yaml for compiler versions Updated the compiler versions in the test-upgrade-legacy.yaml GitHub workflow from 2023.6.0 and 2023.7.0 to 2023.7.0 and 2023.8.0. Also, made formatting changes for consistency. * chore(workflow): add console_password to test-upgrade-legacy.yaml Added the console_password parameter to the test-upgrade-legacy.yaml GitHub workflow to ensure the console password is passed securely from secrets. * refactor(plan): remove PuppetDB service stop commands in convert_compiler_to_legacy.pp Removed the commands to stop Puppet and PuppetDB services when the remove_pdb parameter is true in the convert_compiler_to_legacy plan. This simplifies the plan and avoids unnecessary service stops. * fix(setup): correct variable reference for internal compiler pool address - Updated the variable reference for `internal_compiler_b_pool_address` and `internal_compiler_a_pool_address` in `peadm::setup::legacy_compiler_group` class. - Ensured the correct scope is used for the variables. - Added missing newline at the end of the file. - Added logic to stop and disable `pe-puppetdb` service if `remove_pdb` is true in `convert_compiler_to_legacy` plan. * Adding internal compiler address params to convert legacy compilers, and added trusted cert along with internal compiler address * Matching internal compiler address param types * Adding require node_manager_yaml to legacy_compiler_group * Introduced a puppet run before node_groups defs * fix(setup): update puppetdb_host to use internal compiler pool addresses - Replaced `${trusted['certname']}` with `$internal_compiler_a_pool_address` and `$internal_compiler_b_pool_address` in `puppetdb_host` for `puppet_enterprise::profile::master` class. - Applied `.filter |$_| { $_ }` to ensure non-empty values. - Removed lint ignore comments for single quote string with variables. This change ensures that the correct internal compiler pool addresses are used for the `puppetdb_host` configuration. * docs: update upgrade_with_legacy_compilers.md with remove_pdb parameter info - Added information about the `remove_pdb` parameter to the documentation. - Updated `convert_compiler_to_legacy` plan to set `remove_pdb` default to `true`. This change provides clarity on how to retain PuppetDB service on converted compilers. * fix(workflow): correct indentation for ssh-debugging input in test-upgrade-legacy.yaml - Added missing `inputs` key for `ssh-debugging` under `workflow_dispatch`. This change ensures proper YAML structure for the workflow configuration. * chore(workflow): update test-upgrade-legacy.yaml name for clarity - Changed workflow name to "Upgrade PE with one legacy compiler (test)" for better identification. fix(convert_compiler_to_legacy): ensure proper handling of legacy compilers - Added architecture validation for supported architectures. - Split legacy compiler targets into groups A and B based on availability group or cluster. - Modified certificate extensions for legacy compilers. - Ensured proper stopping of PuppetDB service on legacy compilers when `remove_pdb` is true. - Updated node manager YAML and legacy compiler group setup to include internal compiler pool addresses. - Adjusted Puppet run tasks to include all compiler targets. These changes improve the handling and configuration of legacy compilers during the upgrade process. * feat(convert_compiler_to_legacy): update parameter types and improve formatting - Changed `remove_pdb` parameter type to `Optional[Boolean]` for better flexibility. - Improved alignment and formatting of parameters and variables for better readability. - Updated `$compiler_targets` assignment to use `peadm::get_targets` for consistency. * feat(workflow): enhance test-upgrade-legacy with SSH config and grouping - Added SSH configuration setup to ensure stable connections. - Grouped preparation and conversion steps for better readability in logs. * fix: handle undefined certname in convert_compiler_to_legacy plan - Updated `primary_host` to handle cases where `certname` is undefined. - Added conditional logic to default to `$primary_target` if `certname` is undefined. * fix(workflow): downgrade test-upgrade-legacy version to 2021.7.9 - Changed the `version` in the test-upgrade-legacy workflow from 2023.7.0 to 2021.7.9. - Ensured compatibility with older versions for testing purposes. * fix(workflow): update architecture in test-upgrade-legacy to large-with-dr - Changed the `architecture` in the test-upgrade-legacy workflow from `large-with-two-compilers` to `large-with-dr`. - Updated the `version` from `2021.7.9` to `2023.7.0`. * Fix Conflicts * feat(plan): run puppet agent on all nodes after installation - Added a task to run the puppet agent on all nodes after the installation plan. * fix(plan): remove redundant puppet agent run task - Removed the redundant task to run the puppet agent on all nodes after installation. - Simplified the install_test_cluster plan by eliminating unnecessary steps. * feat(plans): gather certificate extension information in convert_compiler_to_legacy plan - Added a step to gather certificate extension information from all systems using the `peadm::cert_data` task. - This information is used to filter legacy compiler targets based on availability group. This enhancement improves the accuracy of the conversion process by ensuring that the correct certificate data is used. * feat(convert_compiler_to_legacy): gather certificate extension information - Added steps to gather certificate extension information from all systems using the `peadm::cert_data` task. - Updated variable names for clarity and consistency. - Filtered legacy compiler targets based on availability group using the gathered certificate data. This enhancement improves the accuracy of the conversion process by ensuring that the correct certificate data is used. * feat(convert_compiler_to_legacy): gather certificate extension information - Updated `puppetdb_host` to include both internal compiler pool addresses. - Modified `convert_compiler_to_legacy` plan to gather certificate extension information from legacy targets using the `peadm::cert_data` task. - Adjusted filtering of legacy compiler targets based on availability group using the gathered certificate data. This enhancement improves the accuracy of the conversion process by ensuring that the correct certificate data is used. * feat(convert_compiler_to_legacy): gather certificate extension information - Added steps to apply `peadm::setup::node_manager_yaml` class on the primary target. - Ensured `primary_host` is set correctly based on the primary target's certname. - Improved the setup process by gathering certificate extension information. This enhancement ensures that the correct certificate data is used, improving the accuracy of the conversion process. * feat(convert_compiler_to_legacy): gather certificate extension information - Added steps to apply `peadm::setup::legacy_compiler_group` class with internal compiler pool addresses. - Ensured `primary_host` is set correctly based on the primary target's certname. - Removed redundant application of `peadm::setup::node_manager_yaml` and `peadm::setup::legacy_compiler_group` classes. This enhancement ensures that the correct certificate data is used, improving the accuracy of the conversion process. - Deleted `.github/workflows/test-upgrade-legacy.yaml` file. - Removed workflow for testing PE upgrade with one legacy compiler. This change simplifies the CI configuration by removing an outdated or unnecessary workflow. * docs: update documentation for converting and upgrading legacy compilers - Updated the version reference from 3.21 to 3.25 in the upgrade documentation. - Added instructions for using the `peadm::convert_compiler_to_legacy` plan. - Removed outdated steps for updating certificate extensions and converting legacy compilers. - Simplified the upgrade process description for Puppet Enterprise using puppetlabs-peadm module 3.25 or later. --------- Co-authored-by: Neil Anderson Co-authored-by: Neil Anderson Co-authored-by: Aaron Shannon --- README.md | 59 ++++--- REFERENCE.md | 44 ++++- documentation/convert.md | 31 +++- documentation/convert_compiler_to_legacy.md | 9 + documentation/install.md | 130 ++++++++------- .../upgrade_with_legacy_compilers.md | 27 +++ functions/assert_supported_architecture.pp | 30 +++- functions/oid.pp | 1 + manifests/setup/legacy_compiler_group.pp | 72 ++++++++ manifests/setup/node_manager.pp | 70 ++++++++ plans/convert.pp | 66 ++++++++ plans/convert_compiler_to_legacy.pp | 154 ++++++++++++++++++ plans/install.pp | 3 + plans/subplans/component_install.pp | 4 + plans/subplans/configure.pp | 6 + plans/subplans/install.pp | 33 +++- plans/update_compiler_extensions.pp | 25 +++ plans/upgrade.pp | 18 +- .../plans/provision_test_cluster.pp | 3 + .../assert_supported_architecture_spec.rb | 53 ++++++ spec/plans/subplans/install_spec.rb | 44 +++-- tasks/check_legacy_compilers.rb | 67 ++++++++ tasks/classify_compilers.json | 15 ++ tasks/classify_compilers.rb | 43 +++++ tasks/get_peadm_config.rb | 26 ++- 25 files changed, 910 insertions(+), 123 deletions(-) create mode 100644 documentation/convert_compiler_to_legacy.md create mode 100644 documentation/upgrade_with_legacy_compilers.md create mode 100644 manifests/setup/legacy_compiler_group.pp create mode 100644 plans/convert_compiler_to_legacy.pp create mode 100644 plans/update_compiler_extensions.pp create mode 100755 tasks/check_legacy_compilers.rb create mode 100644 tasks/classify_compilers.json create mode 100755 tasks/classify_compilers.rb diff --git a/README.md b/README.md index ef33203a..9efda46b 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ You can use PEADM to deploy and manage PE installations for standard, large, and #### Table of contents - [Puppet Enterprise Administration Module (PEADM)](#puppet-enterprise-pe-administration-adm-module) - - [Table of contents](#table-of-contents) + - [Table of contents](#table-of-contents) - [Support](#support) - [Overview](#overview) - [What PEADM affects](#what-peadm-affects) @@ -24,7 +24,6 @@ You can use PEADM to deploy and manage PE installations for standard, large, and PEADM is a supported PE module. If you are a PE customer with the standard or premium support service, you can contact [Support](https://portal.perforce.com/s/topic/0TO4X000000DbNgWAK/puppet) or your Technical Account Manager for assistance. - ## Overview This is the standard workflow for installing PEADM. @@ -32,6 +31,7 @@ This is the standard workflow for installing PEADM. 1. [Install Bolt](https://www.puppet.com/docs/bolt/latest/bolt_installing) on a jump host. 2. To create a Bolt project that includes the PEADM module, run:
`mkdir && cd && bolt project init --modules puppetlabs-peadm` 3. In the Bolt project directory, update the `inventory.yaml` file with connection information for the servers you plan to use for hosting PE infrastructure. For example: + ```yaml --- groups: @@ -55,56 +55,55 @@ groups: ### What PEADM affects -* The `peadm::install` plan adds a number of custom original identifier (OID) trusted facts to the certificates of deployed PE infrastructure nodes. These trusted facts are used by PEADM plans to identify nodes that host PE infrastructure components. -* Depending on the scale of your architecture, up to four node groups may be created to configure `puppet_enterprise` class parameters for the following PE infrastructure components: - * The primary server - * The primary server replica - * PostgreSQL nodes (database servers) - * Compilers (compiler hosts are designated as belonging to availability group A or B) +- The `peadm::install` plan adds a number of custom original identifier (OID) trusted facts to the certificates of deployed PE infrastructure nodes. These trusted facts are used by PEADM plans to identify nodes that host PE infrastructure components. +- Depending on the scale of your architecture, up to four node groups may be created to configure `puppet_enterprise` class parameters for the following PE infrastructure components: + - The primary server + - The primary server replica + - PostgreSQL nodes (database servers) + - Compilers (compiler hosts are designated as belonging to availability group A or B) ### What PEADM does not affect -* PEADM does not impact regular PE operations. After using it to deploy a new PE installation or upgrade an existing one, PEADM is not required until you want to use it to upgrade PE or expand your installation. -* Using PEADM to install PE or upgrade PE does not prevent you from using documented PE procedures such as setting up disaster recovery or performing a manual upgrade. +- PEADM does not impact regular PE operations. After using it to deploy a new PE installation or upgrade an existing one, PEADM is not required until you want to use it to upgrade PE or expand your installation. +- Using PEADM to install PE or upgrade PE does not prevent you from using documented PE procedures such as setting up disaster recovery or performing a manual upgrade. ### Requirements -* PEADM is compatible with Puppet Enterprise 2019.8.1 or newer versions. -* PEADM requires [Bolt](https://www.puppet.com/docs/bolt/latest/bolt_installing) version 3.17.0 or newer and works on operating systems supported by Bolt. -**Note:** You can use PEADM to install Puppet Enterprise (PE) onto any operating system platform that is supported by the specific PE version you want to install. To review operating system platforms supported by specific PE versions, refer to the PE documentation. -* To successfully convert your current PE installation to a PEADM-managed installation, ensure that the PE setting for editing classifier configuration data is enabled. This setting is enabled by default on new PE installations, but it could be disabled if the relevant configuration was removed from your global hiera.yaml file. See the [PE docs](https://www.puppet.com/docs/pe/latest/config_console.html#enable_console_configuration_data) for more information. +- PEADM is compatible with Puppet Enterprise 2019.8.1 or newer versions. +- PEADM requires [Bolt](https://www.puppet.com/docs/bolt/latest/bolt_installing) version 3.17.0 or newer and works on operating systems supported by Bolt. + **Note:** You can use PEADM to install Puppet Enterprise (PE) onto any operating system platform that is supported by the specific PE version you want to install. To review operating system platforms supported by specific PE versions, refer to the PE documentation. +- To successfully convert your current PE installation to a PEADM-managed installation, ensure that the PE setting for editing classifier configuration data is enabled. This setting is enabled by default on new PE installations, but it could be disabled if the relevant configuration was removed from your global hiera.yaml file. See the [PE docs](https://www.puppet.com/docs/pe/latest/config_console.html#enable_console_configuration_data) for more information. ## Usage For instructions on using PEADM plans, see the following PEADM docs: -* [Install](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/install.md) -* [Upgrade](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/upgrade.md) -* [Convert](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/convert.md) -* [Status](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/status.md) +- [Install](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/install.md) +- [Upgrade](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/upgrade.md) +- [Convert](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/convert.md) +- [Status](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/status.md) ## Reference To understand which architecture is right for you, see the following information on the Puppet documentation site: -* [PE architectures](https://puppet.com/docs/pe/latest/choosing_an_architecture.html) -* [PE multi-region reference architectures](https://puppet.com/docs/patterns-and-tactics/latest/reference-architectures/pe-multi-region-reference-architectures.html) - +- [PE architectures](https://puppet.com/docs/pe/latest/choosing_an_architecture.html) +- [PE multi-region reference architectures](https://puppet.com/docs/patterns-and-tactics/latest/reference-architectures/pe-multi-region-reference-architectures.html) To learn more about the PEADM module and its uses, see the following PEADM docs: -* [Recovery procedures](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/recovery.md) -* [Architectures](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/architectures.md) -* [Expanding deployment](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/expanding.md) -* [Classification](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/classification.md) -* [Testing](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/pre_post_checks.md) -* [Docker based examples](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/docker_examples.md) -* [Release process](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/release_process.md) +- [Recovery procedures](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/recovery.md) +- [Architectures](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/architectures.md) +- [Expanding deployment](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/expanding.md) +- [Classification](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/classification.md) +- [Testing](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/pre_post_checks.md) +- [Docker based examples](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/docker_examples.md) +- [Release process](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/release_process.md) ## Getting help -* If you find a bug, you can [create a GitHub issue](https://github.com/puppetlabs/puppetlabs-peadm/issues). -* For PE customers using PEADM and experiencing outages or other issues, [contact the Support team](https://portal.perforce.com/s/topic/0TO4X000000DbNgWAK/puppet). +- If you find a bug, you can [create a GitHub issue](https://github.com/puppetlabs/puppetlabs-peadm/issues). +- For PE customers using PEADM and experiencing outages or other issues, [contact the Support team](https://portal.perforce.com/s/topic/0TO4X000000DbNgWAK/puppet). ## License diff --git a/REFERENCE.md b/REFERENCE.md index b649f54b..df2287ba 100644 --- a/REFERENCE.md +++ b/REFERENCE.md @@ -10,6 +10,7 @@ * `peadm::setup::convert_node_manager`: Used during the peadm::convert plan * `peadm::setup::convert_pre20197`: Defines configuration needed for converting PE 2018 +* `peadm::setup::legacy_compiler_group` * `peadm::setup::node_manager`: Configures PEAdm's required node groups * `peadm::setup::node_manager_yaml`: Set up the node_manager.yaml file in the temporary Bolt confdir @@ -55,6 +56,7 @@ * [`backup_classification`](#backup_classification): A task to call the classification api and write to file * [`cert_data`](#cert_data): Return certificate data related to the Puppet agent * [`cert_valid_status`](#cert_valid_status): Check primary for valid state of a certificate +* [`classify_compilers`](#classify_compilers): Classify compilers as legacy or non-legacy * [`code_manager`](#code_manager): Perform various code manager actions * [`code_manager_enabled`](#code_manager_enabled): Run on a PE primary node to check if Code Manager is enabled. * [`code_sync_status`](#code_sync_status): A task to confirm code is in sync accross the cluster for clusters with code manager configured @@ -108,6 +110,7 @@ Supported use cases: #### Private Plans +* `peadm::convert_compiler_to_legacy` * `peadm::misc::divert_code_manager`: This plan exists to account for a scenario where a PE XL * `peadm::modify_cert_extensions` * `peadm::subplans::component_install`: Install a new PEADM component @@ -117,6 +120,7 @@ Supported use cases: * `peadm::subplans::modify_certificate` * `peadm::subplans::prepare_agent` * `peadm::uninstall`: Single-entry-point plan for uninstalling Puppet Enterprise +* `peadm::update_compiler_extensions` * `peadm::util::code_sync_status` * `peadm::util::copy_file` * `peadm::util::db_disable_pglogical` @@ -135,7 +139,7 @@ Type: Puppet Language Assert that the architecture given is a supported one -#### `peadm::assert_supported_architecture(TargetSpec $primary_host, Variant[TargetSpec, Undef] $replica_host = undef, Variant[TargetSpec, Undef] $primary_postgresql_host = undef, Variant[TargetSpec, Undef] $replica_postgresql_host = undef, Variant[TargetSpec, Undef] $compiler_hosts = undef)` +#### `peadm::assert_supported_architecture(TargetSpec $primary_host, Variant[TargetSpec, Undef] $replica_host = undef, Variant[TargetSpec, Undef] $primary_postgresql_host = undef, Variant[TargetSpec, Undef] $replica_postgresql_host = undef, Variant[TargetSpec, Undef] $compiler_hosts = undef, Variant[TargetSpec, Undef] $legacy_compilers = undef)` The peadm::assert_supported_architecture function. @@ -171,6 +175,12 @@ Data type: `Variant[TargetSpec, Undef]` +##### `legacy_compilers` + +Data type: `Variant[TargetSpec, Undef]` + + + ### `peadm::assert_supported_bolt_version` Type: Puppet Language @@ -1055,6 +1065,20 @@ Data type: `String` The certifcate name to check validation of +### `classify_compilers` + +Classify compilers as legacy or non-legacy + +**Supports noop?** false + +#### Parameters + +##### `compiler_hosts` + +Data type: `Array[String]` + +List of FQDNs of compilers + ### `code_manager` Perform various code manager actions @@ -1798,6 +1822,7 @@ The following parameters are available in the `peadm::convert` plan: * [`primary_host`](#-peadm--convert--primary_host) * [`replica_host`](#-peadm--convert--replica_host) * [`compiler_hosts`](#-peadm--convert--compiler_hosts) +* [`legacy_compilers`](#-peadm--convert--legacy_compilers) * [`primary_postgresql_host`](#-peadm--convert--primary_postgresql_host) * [`replica_postgresql_host`](#-peadm--convert--replica_postgresql_host) * [`compiler_pool_address`](#-peadm--convert--compiler_pool_address) @@ -1826,6 +1851,14 @@ Data type: `Optional[TargetSpec]` +Default value: `undef` + +##### `legacy_compilers` + +Data type: `Optional[TargetSpec]` + + + Default value: `undef` ##### `primary_postgresql_host` @@ -1911,6 +1944,7 @@ The following parameters are available in the `peadm::install` plan: * [`primary_host`](#-peadm--install--primary_host) * [`replica_host`](#-peadm--install--replica_host) * [`compiler_hosts`](#-peadm--install--compiler_hosts) +* [`legacy_compilers`](#-peadm--install--legacy_compilers) * [`primary_postgresql_host`](#-peadm--install--primary_postgresql_host) * [`replica_postgresql_host`](#-peadm--install--replica_postgresql_host) * [`console_password`](#-peadm--install--console_password) @@ -2027,6 +2061,14 @@ Data type: `Optional[TargetSpec]` +Default value: `undef` + +##### `legacy_compilers` + +Data type: `Optional[TargetSpec]` + + + Default value: `undef` ##### `primary_postgresql_host` diff --git a/documentation/convert.md b/documentation/convert.md index 6d2c8095..c0ec001f 100644 --- a/documentation/convert.md +++ b/documentation/convert.md @@ -1,6 +1,6 @@ -# Convert infrastructure for use with the peadm module +# Convert infrastructure for use with the PEADM module -The peadm::convert plan can be used to adopt manually deployed infrastructure for use with peadm, or to adopt infrastructure deployed with an older version of peadm. +The peadm::convert plan can be used to adopt manually deployed infrastructure for use with PEADM or to adopt infrastructure deployed with an older version of peadm. ## Convert an Existing Deployment @@ -14,19 +14,36 @@ Prepare to run the plan against all servers in the PE infrastructure, using a pa "pe-xl-compiler-0.lab1.puppet.vm", "pe-xl-compiler-1.lab1.puppet.vm" ], - - "compiler_pool_address": "puppet.lab1.puppet.vm", + "legacy_compilers": [ + "pe-xl-legacy-compiler-0.lab1.puppet.vm", + "pe-xl-legacy-compiler-1.lab1.puppet.vm" + ], + "compiler_pool_address": "puppet.lab1.puppet.vm" } ``` -See the [install](install.md#reference-architectures) documentation for a list of supported architectures. Note that for convert, *all infrastructure being converted must already be functional*; you cannot use convert to add new systems to the infrastructure, nor can you use it to change your architecture. +See the [install](install.md#reference-architectures) documentation for a list of supported architectures. Note that for convert, _all infrastructure being converted must already be functional_; you cannot use convert to add new systems to the infrastructure, nor can you use it to change your architecture. ``` -bolt plan run peadm::convert --params @params.json +bolt plan run peadm::convert --params @params.json ``` ## Retry or resume plan This plan is broken down into steps. Normally, the plan runs through all the steps from start to finish. The name of each step is displayed during the plan run, as the step begins. -The `begin_at_step` parameter can be used to facilitate re-running this plan after a failed attempt, skipping past any steps that already completed successfully on the first try and picking up again at the step specified. The step name to resume at can be read from the previous run logs. A full list of available values for this parameter can be viewed by running `bolt plan show peadm::convert`. +The `begin_at_step` parameter can be used to facilitate re-running this plan after a failed attempt, skipping past any steps that were already completed successfully on the first try and picking up again at the step specified. The step name to resume can be read from the previous run logs. A full list of available values for this parameter can be viewed by running `bolt plan show peadm::convert`. + +## Convert compilers to legacy + +### Puppet Enterprise installed with puppetlabs-peadm version 3.21 or later + +To convert compilers to legacy compilers use the `peadm::convert_compiler_to_legacy` plan. This plan will create the needed Node group and Classifier rules to make compilers legacy. Also will add certificate extensions to those nodes. + +```shell +bolt plan run peadm::convert_compiler_to_legacy legacy_hosts=compiler1.example.com,compiler2.example.com primary_host=primary.example.com +``` + +### Puppet Enterprise installed with puppetlabs-peadm version prior to 3.21 + +Follow Steps 1 to 3 in the [Upgrade Puppet Enterprise with legacy compilers](upgrade_with_legacy_compilers.md) documentation. diff --git a/documentation/convert_compiler_to_legacy.md b/documentation/convert_compiler_to_legacy.md new file mode 100644 index 00000000..badb4832 --- /dev/null +++ b/documentation/convert_compiler_to_legacy.md @@ -0,0 +1,9 @@ +# Convert compilers to legacy + +### Puppet Enterprise installed with puppetlabs-peadm version 3.25 or later + +To convert compilers to legacy compilers use the `peadm::convert_compiler_to_legacy` plan. This plan will create the needed Node group and Classifier rules to make compilers legacy. Also will add certificate extensions to those nodes. + +```shell +bolt plan run peadm::convert_compiler_to_legacy legacy_hosts=compiler1.example.com,compiler2.example.com primary_host=primary.example.com +``` diff --git a/documentation/install.md b/documentation/install.md index 22e9d70c..72db9afc 100644 --- a/documentation/install.md +++ b/documentation/install.md @@ -1,69 +1,68 @@ -# Install Puppet Enterprise using the peadm module +# Install Puppet Enterprise using the PEADM module -The peadm module can be used to install Puppet Enterprise on new infrastructure. Supported architectures include Standard, Large, and Extra Large. +The PEADM module can be used to install Puppet Enterprise on new infrastructure. Supported architectures include Standard, Large, and Extra Large. -The peadm install plan creates a base install. Once a base cluster is installed, you may need to continue and perform additional configuration and adjustments to reach your target state, depending on your use case. +The PEADM install plan creates a base install. Once a base cluster is installed, you may need to continue and perform additional configuration and adjustments to reach your target state, depending on your use case. ## Reference Architectures -When installing a new PE cluster using peadm, there are several different host parameters which can be specified. At a minimum, you must always specify the primary parameter. Depending on which architecture you are deploying, other host parameters may be needed as well. The following is a list of the architectures peadm can install and the required parameters. - -* Standard - - primary -* Standard with DR - - primary - - primary-replica -* Large - - primary - - compilers -* Large with DR - - primary - - primary-replica - - compilers -* Extra Large - - primary - - pdb-database - - compilers (optional) -* Extra Large with DR - - primary - - primary-replica - - pdb-database - - pdb-database-replica - - compilers (optional) - -Supplying a combination of host parameters which does not match one of the supported architectures above will result in an unsupported architecture error. +When installing a new PE cluster using PEADM, several different host parameters can be specified. At a minimum, you must always specify the primary parameter. Depending on which architecture you are deploying, other host parameters may be needed as well. The following is a list of the architectures PEADM can install and the required parameters. + +- Standard + - primary +- Standard with DR + - primary + - primary-replica +- Large + - primary + - compilers +- Large with DR + - primary + - primary-replica + - compilers +- Extra Large + - primary + - pdb-database + - compilers (optional) +- Extra Large with DR + - primary + - primary-replica + - pdb-database + - pdb-database-replica + - compilers (optional) + +Supplying a combination of host parameters that do not match one of the supported architectures above will result in an unsupported architecture error. ## Requirements -* Puppet must not be installed on any of the target PE cluster hosts prior to beginning install. +Puppet must not be installed on any of the target PE cluster hosts before beginning installation. ## Usage ### Bolt 3 usage + We will name the bolt project `large_ha_peadm` in this example but the project name can be anything. 1. Install Bolt on a jumphost. This can be the primary, or any other system. (via package) 2. Run `mkdir large_ha_peadm && cd large_ha_peadm && bolt project init large_ha_peadm --modules puppetlabs-peadm` -4. Create an inventory file with connection information. Example included below. -5. Create a parameters file. Example included below. -6. Run `bolt plan run peadm::install --params @params.json ` after the inventory and params files are created. - +3. Create an inventory file with connection information. Example included below. +4. Create a parameters file. Example included below. +5. Run `bolt plan run peadm::install --params @params.json ` after the inventory and params files are created. ### Bolt 2 usage -1. Install Bolt on a jumphost. This can be the primary, or any other system. -2. Download or git clone the peadm module and put it somewhere on the jumphost. e.g. ~/modules/peadm. -3. Download or git clone the module dependencies, and put them somewhere on the jumphost. e.g. ~/modules/stdlib, ~/modules/node\_manager, etc. -4. Create an inventory file with connection information. Example included below. -5. Create a parameters file. Example included below. -6. Run the peadm::install plan with the inputs created. Example: +1. Install Bolt on a jumphost. This can be the primary or any other system. +2. Download or git clone the PEADM module and put it somewhere on the jumphost. e.g. ~/modules/peadm. +3. Download or git clone the module dependencies, and put them somewhere on the jumphost. e.g. ~/modules/stdlib, ~/modules/node_manager, etc. +4. Create an inventory file with connection information. Example included below. +5. Create a parameters file. Example included below. +6. Run the peadm::install plan with the inputs created. Example: bolt plan run peadm::install \ --inventory inventory.yaml \ --modulepath ~/modules \ --params @params.json - Example inventory.yaml Bolt inventory file: ```yaml @@ -85,9 +84,13 @@ groups: uri: 10.234.6.45 - name: pe-xl-compiler-1.lab1.puppet.vm uri: 10.234.14.131 + - name: pe-xl-legacy-compiler-0.lab1.puppet.vm + uri: 10.234.6.46 + - name: pe-xl-legacy-compiler-1.lab1.puppet.vm + uri: 10.234.6.47 ``` -Example params.json Bolt parameters file (shown: Extra Large with DR): +example params.json bolt parameters file (shown: extra large with dr): ```json { @@ -99,9 +102,12 @@ Example params.json Bolt parameters file (shown: Extra Large with DR): "pe-xl-compiler-0.lab1.puppet.vm", "pe-xl-compiler-1.lab1.puppet.vm" ], - + "legacy_compilers": [ + "pe-xl-legacy-compiler-0.lab1.puppet.vm", + "pe-xl-legacy-compiler-1.lab1.puppet.vm" + ], "console_password": "puppetlabs", - "dns_alt_names": [ "puppet", "puppet.lab1.puppet.vm" ], + "dns_alt_names": ["puppet", "puppet.lab1.puppet.vm"], "compiler_pool_address": "puppet.lab1.puppet.vm", "version": "2021.7.0" } @@ -113,12 +119,12 @@ Example params.json Bolt parameters file (shown: Standard): { "primary_host": "pe-xl-core-0.lab1.puppet.vm", "console_password": "puppetlabs", - "dns_alt_names": [ "puppet", "puppet.lab1.puppet.vm" ], + "dns_alt_names": ["puppet", "puppet.lab1.puppet.vm"], "version": "2021.7.9" } ``` -Review the [peadm::install plan](../plans/install.pp) to learn about more advanced installation options. For example, it is possible to: supply an ssh private key and git clone URL for a control-repo as part of installation; supply the LDAP configuration data for PE; specify where the installer tarball is uploaded; and similar complete automation tie-ins. +Review the [peadm::install plan](../plans/install.pp) to learn about more advanced installation options. For example, it is possible to: supply an SSH private key and git clone URL for a control-repo as part of installation; supply the LDAP configuration data for PE; specify where the installer tarball is uploaded; and similar complete automation tie-ins. ## Offline usage @@ -126,28 +132,27 @@ The peadm::install plan downloads installation content from an online repository The default staging directory is `/tmp`. If a different staging dir is being used, it can be specified using the `stagingdir` parameter to the peadm::install plan. -The content needed is the PE installation tarball for the target version. The installation content should be in the staging dir, and should have its original name. E.g. `/tmp/puppet-enterprise-2021.4.0-el-7-x86_64.tar.gz`. +The content needed is the PE installation tarball for the target version. The installation content should be in the staging dir and should have its original name. E.g. `/tmp/puppet-enterprise-2021.4.0-el-7-x86_64.tar.gz`. Installation content can be downloaded from [https://puppet.com/try-puppet/puppet-enterprise/download/](https://puppet.com/try-puppet/puppet-enterprise/download/). -If you wish to prevent the bolt host from transferring the installer tarball to the targets you can place the installer tar files on the target hosts in the _upload directory_. This can save time over slow networks, but is best to just perform the bolt task on the target node (puppetserver). The default _upload directory_ is `/tmp`. If a different upload dir is being used, it can be specified using the `uploaddir` parameter to the peadm::install plan. With default parameters the tar file will need to exist in the directories for offline configuration: +If you wish to prevent the bolt host from transferring the installer tarball to the targets you can place the installer tar files on the target hosts in the _upload directory_. This can save time over slow networks, but is best to just perform the bolt task on the target node (puppetserver). The default _upload directory_ is `/tmp`. If a different upload dir is being used, it can be specified using the `uploaddir` parameter to the peadm::install plan. With default parameters, the tar file will need to exist in the directories for offline configuration: -* /tmp on the Bolt host -* /tmp on the primary -* /tmp on the primary PuppetDB PostgreSQL (if used) -* /tmp on the replica PuppetDB PostgreSQL (if used) +- /tmp on the Bolt host +- /tmp on the primary +- /tmp on the primary PuppetDB PostgreSQL (if used) +- /tmp on the replica PuppetDB PostgreSQL (if used) ## Online usage -The peadm::install plan can be configured to download installation content directly to hosts. To configure online installation, set the `download_mode` parameter of the `peadm::install` plan to `direct`. The direct mode is often more efficient when PE hosts have a route to the internet. +The peadm::install plan can be configured to download installation content directly to hosts. To configure the online installation, set the `download_mode` parameter of the `peadm::install` plan to `direct`. The direct mode is often more efficient when PE hosts have a route to the internet. ## Hostnames and Certificate Names The various host parameters given to the peadm::install or peadm::action::install plans will be set as Puppet certificate names. You must use the names here that you want the servers to be identified as by Puppet. While it is not required that target names match hostnames, it _is_ required that target names be resolvable. - -In the event that Bolt will reach servers by IP address or external DNS name rather than internal DNS name or desired certname, a Bolt inventory file should be used to specify URIs for each name. For example: +If Bolt will reach servers by IP address or external DNS name rather than internal DNS name or desired certname, a Bolt inventory file should be used to specify URIs for each name. For example: ```yaml --- @@ -158,7 +163,7 @@ targets: uri: 10.234.14.131 ``` -A parameters JSON file can then reference the target names, which will become the Puppet certificate names, and Bolt will still be able to reach the systems by using the IP addresses or other DNS name specified as the URIs in the inventory.yaml file. +A parameters JSON file can then reference the target names, which will become the Puppet certificate names, and Bolt will still be able to reach the systems by using the IP addresses or other DNS names specified as the URIs in the inventory.yaml file. ```json { @@ -166,7 +171,7 @@ A parameters JSON file can then reference the target names, which will become th "replica_host": "pe-xl-core-1.lab1.puppet.vm", "console_password": "puppetlabs", - "dns_alt_names": [ "puppet", "puppet.lab1.puppet.vm" ], + "dns_alt_names": ["puppet", "puppet.lab1.puppet.vm"], "compiler_pool_address": "puppet.lab1.puppet.vm", "version": "2021.7.0" } @@ -174,12 +179,11 @@ A parameters JSON file can then reference the target names, which will become th ## Implementation Reference -Provisioning can be broken down into two actions: [install](../plans/action/install.pp), and [configure](../plans/subplans/configure.pp). Installation currently requires ssh access to the un-installed nodes, but configure can be performed using the Orchestrator transport if installation has already been completed. - -Besides getting Puppet Enterprise installed, the key configuration supporting Large and Extra Large architectures is laid out in four classification groups. Links are provided below to a Markdown document that describes the groups, and also to the Puppet manifest that actually configures them: +Provisioning can be broken down into two actions: [install](../plans/action/install.pp), and [configure](../plans/subplans/configure.pp). The installation currently requires SSH access to the uninstalled nodes, but configuration can be performed using the Orchestrator transport if the installation has already been completed. -* [classification.md](classification.md) -* [peadm::setup::node\_manager class](../manifests/setup/node_manager.pp) +Besides getting Puppet Enterprise installed, the key configuration supporting Large and Extra-large architectures is laid out in four classification groups. Links are provided below to a Markdown document that describes the groups, and also to the Puppet manifest that configures them: -The reference implementation uses trusted facts to put nodes in the right groups. Because the important puppet\_enterprise::\* class parameters and data are specified in the console, it should also be safe to have a pe.conf present on both the primary, and the primary replica nodes. +- [classification.md](classification.md) +- [peadm::setup::node_manager class](../manifests/setup/node_manager.pp) +The reference implementation uses trusted facts to put nodes in the right groups. Because the important puppet_enterprise::\* class parameters and data are specified in the console, it should also be safe to have a pe.conf present on both the primary and the primary replica nodes. diff --git a/documentation/upgrade_with_legacy_compilers.md b/documentation/upgrade_with_legacy_compilers.md new file mode 100644 index 00000000..398a7bec --- /dev/null +++ b/documentation/upgrade_with_legacy_compilers.md @@ -0,0 +1,27 @@ +# Upgrade Puppet Enterprise with legacy compilers + +## What is a legacy compiler and a current compiler + +As a legacy compiler we refer to a compiler that doesn't have PuppetDB. And a current Compiler is a compiler that has PuppetDB. By default, latest versions of Puppet enterprise comes with compilers that have PuppetDB.If your primary server and compilers are connected with high-latency links or congested network segments, you might experience better PuppetDB performance with legacy compilers. + +## Who is this documentation for + +For those users that have installed Puppet Enterprise with puppetlabs-peadm prior version 3.25 and manually converted their existing complilers (all of the or at least 1) to legacy compilers. + +## Who is this documentation not for + +For those users that have installed Puppet Enterprise with PEADM with 3.25 version or later, there is no need to follow this documentation. The install process will automatically have created the necessary configurations for you and you can use the `peadm::convert_compiler_to_legacy` plan if you need a legacy compiler. example: + +```shell +bolt plan run peadm::convert_compiler_to_legacy legacy_hosts=compiler1.example.com,compiler2.example.com primary_host=primary.example.com +``` + +## How to upgrade Puppet Enterprise with legacy compilers + +### 1. Revert changes to the legacy compilers nodes + +Usually users pin the nodes in the Pe Master Node Group and then manually removing PuppetDB from compilers nodes. To revert this changes go to your Puppet Enterprise console and unpin the compilers nodes from the Group. + +### 2. Upgrade Puppet Enterprise + +You can proceed with the upgrade of Puppet Enterprise as usual using the puppetlabs-peadm module 3.25 or later and pass legacy compilers to the upgrade plan. diff --git a/functions/assert_supported_architecture.pp b/functions/assert_supported_architecture.pp index 0c705ddd..25709e86 100644 --- a/functions/assert_supported_architecture.pp +++ b/functions/assert_supported_architecture.pp @@ -5,7 +5,31 @@ function peadm::assert_supported_architecture ( Variant[TargetSpec, Undef] $primary_postgresql_host = undef, Variant[TargetSpec, Undef] $replica_postgresql_host = undef, Variant[TargetSpec, Undef] $compiler_hosts = undef, + Variant[TargetSpec, Undef] $legacy_compilers = undef, ) >> Hash { + # Normalize $legacy_compilers to an array + $legacy_compilers_array = $legacy_compilers ? { + undef => [], + String => [$legacy_compilers], + Array => $legacy_compilers, + default => fail("Unexpected type for \$legacy_compilers: ${legacy_compilers}"), + } + + # Normalize $compiler_hosts to an array + $compiler_hosts_array = $compiler_hosts ? { + undef => [], + String => [$compiler_hosts], + Array => $compiler_hosts, + default => fail("Unexpected type for \$compiler_hosts: ${compiler_hosts}"), + } + $all_compilers = $legacy_compilers_array + $compiler_hosts_array + + # Set $has_compilers to undef if $all_compilers is empty, otherwise set it to true + $has_compilers = empty($all_compilers) ? { + true => undef, + default => true, + } + $result = case [ !!($primary_host), !!($replica_host), @@ -13,13 +37,13 @@ function peadm::assert_supported_architecture ( !!($replica_postgresql_host), ] { [true, false, false, false]: { # Standard or Large, no DR - ({ 'disaster-recovery' => false, 'architecture' => $compiler_hosts ? { + ({ 'disaster-recovery' => false, 'architecture' => $has_compilers ? { undef => 'standard', default => 'large', } }) } [true, true, false, false]: { # Standard or Large, DR - ({ 'disaster-recovery' => true, 'architecture' => $compiler_hosts ? { + ({ 'disaster-recovery' => true, 'architecture' => $has_compilers ? { undef => 'standard', default => 'large', } }) @@ -44,7 +68,7 @@ function peadm::assert_supported_architecture ( <% if $replica_postgresql_host { -%> - pdb-database-replica <% } -%> - <% if $compiler_hosts { -%> + <% if $has_compilers { -%> - compilers <% } -%> diff --git a/functions/oid.pp b/functions/oid.pp index 2fc735d0..0f03a43c 100644 --- a/functions/oid.pp +++ b/functions/oid.pp @@ -4,6 +4,7 @@ function peadm::oid ( case $short_name { 'peadm_role': { '1.3.6.1.4.1.34380.1.1.9812' } 'peadm_availability_group': { '1.3.6.1.4.1.34380.1.1.9813' } + 'peadm_legacy_compiler': { '1.3.6.1.4.1.34380.1.1.9814' } 'pp_application': { '1.3.6.1.4.1.34380.1.1.8' } 'pp_cluster': { '1.3.6.1.4.1.34380.1.1.16' } 'pp_role': { '1.3.6.1.4.1.34380.1.1.13' } diff --git a/manifests/setup/legacy_compiler_group.pp b/manifests/setup/legacy_compiler_group.pp new file mode 100644 index 00000000..e3601200 --- /dev/null +++ b/manifests/setup/legacy_compiler_group.pp @@ -0,0 +1,72 @@ +# @api private +class peadm::setup::legacy_compiler_group ( + String[1] $primary_host, + Optional[String] $internal_compiler_a_pool_address = undef, + Optional[String] $internal_compiler_b_pool_address = undef, +) { + Node_group { + purge_behavior => none, + } + + node_group { 'PE Legacy Compiler': + parent => 'PE Master', + rule => ['and', + ['=', ['trusted', 'extensions', peadm::oid('peadm_legacy_compiler')], 'true'], + ['=', ['trusted', 'extensions', 'pp_auth_role'], 'pe_compiler'], + ], + classes => { + 'puppet_enterprise::profile::master' => { + 'puppetdb_host' => [$internal_compiler_a_pool_address, $internal_compiler_b_pool_address].filter |$_| { $_ }, + 'puppetdb_port' => [8081], + }, + }, + } + + node_group { 'PE Legacy Compiler Group A': + ensure => 'present', + parent => 'PE Legacy Compiler', + rule => ['and', + ['=', ['trusted', 'extensions', 'pp_auth_role'], 'pe_compiler'], + ['=', ['trusted', 'extensions', peadm::oid('peadm_availability_group')], 'A'], + ['=', ['trusted', 'extensions', peadm::oid('peadm_legacy_compiler')], 'true'], + ], + classes => { + 'puppet_enterprise::profile::master' => { + 'puppetdb_host' => [$internal_compiler_b_pool_address, $internal_compiler_a_pool_address].filter |$_| { $_ }, + 'puppetdb_port' => [8081], + }, + }, + data => { + # Workaround for GH-118 + 'puppet_enterprise::profile::master::puppetdb' => { + 'ha_enabled_replicas' => [], + }, + }, + } + + node_group { 'PE Legacy Compiler Group B': + ensure => 'present', + parent => 'PE Legacy Compiler', + rule => ['and', + ['=', ['trusted', 'extensions', 'pp_auth_role'], 'pe_compiler'], + ['=', ['trusted', 'extensions', peadm::oid('peadm_availability_group')], 'B'], + ['=', ['trusted', 'extensions', peadm::oid('peadm_legacy_compiler')], 'true'], + ], + classes => { + 'puppet_enterprise::profile::master' => { + 'puppetdb_host' => [$internal_compiler_a_pool_address, $internal_compiler_b_pool_address].filter |$_| { $_ }, + 'puppetdb_port' => [8081], + }, + }, + data => { + # Workaround for GH-118 + 'puppet_enterprise::profile::master::puppetdb' => { + 'ha_enabled_replicas' => [], + }, + }, + } + + node_group { 'PE Compiler': + rule => ['and', ['=', ['trusted', 'extensions', peadm::oid('peadm_legacy_compiler')], 'false']], + } +} diff --git a/manifests/setup/node_manager.pp b/manifests/setup/node_manager.pp index 65c69044..f74cb217 100644 --- a/manifests/setup/node_manager.pp +++ b/manifests/setup/node_manager.pp @@ -79,6 +79,12 @@ variables => { 'pe_master' => true }, } + # PE Compiler group comes from default PE and already has the pe compiler role + node_group { 'PE Compiler': + parent => 'PE Master', + rule => ['and', ['=', ['trusted', 'extensions', peadm::oid('peadm_legacy_compiler')], 'false']], + } + # This group should pin the primary, and also map to any pe-postgresql nodes # which are part of the architecture. node_group { 'PE Database': @@ -115,6 +121,7 @@ rule => ['and', ['=', ['trusted', 'extensions', 'pp_auth_role'], 'pe_compiler'], ['=', ['trusted', 'extensions', peadm::oid('peadm_availability_group')], 'A'], + ['=', ['trusted', 'extensions', peadm::oid('peadm_legacy_compiler')], 'false'], ], classes => { 'puppet_enterprise::profile::puppetdb' => { @@ -173,6 +180,7 @@ rule => ['and', ['=', ['trusted', 'extensions', 'pp_auth_role'], 'pe_compiler'], ['=', ['trusted', 'extensions', peadm::oid('peadm_availability_group')], 'B'], + ['=', ['trusted', 'extensions', peadm::oid('peadm_legacy_compiler')], 'false'], ], classes => { 'puppet_enterprise::profile::puppetdb' => { @@ -192,4 +200,66 @@ }, }, } + + node_group { 'PE Legacy Compiler': + parent => 'PE Master', + rule => ['and', + ['=', ['trusted', 'extensions', 'pp_auth_role'], 'pe_compiler'], + ['=', ['trusted', 'extensions', peadm::oid('peadm_legacy_compiler')], 'true'], + ], + classes => { + 'puppet_enterprise::profile::master' => { + 'puppetdb_host' => [$internal_compiler_a_pool_address, $internal_compiler_b_pool_address].filter |$_| { $_ }, + 'puppetdb_port' => [8081], + }, + }, + } + + # Configure the A pool for legacy compilers. There are up to two pools for DR, each + # having an affinity for one "availability zone" or the other. + node_group { 'PE Legacy Compiler Group A': + ensure => 'present', + parent => 'PE Legacy Compiler', + rule => ['and', + ['=', ['trusted', 'extensions', 'pp_auth_role'], 'pe_compiler'], + ['=', ['trusted', 'extensions', peadm::oid('peadm_availability_group')], 'A'], + ['=', ['trusted', 'extensions', peadm::oid('peadm_legacy_compiler')], 'true'], + ], + classes => { + 'puppet_enterprise::profile::master' => { + 'puppetdb_host' => [$internal_compiler_b_pool_address, $internal_compiler_a_pool_address].filter |$_| { $_ }, + 'puppetdb_port' => [8081], + }, + }, + data => { + # Workaround for GH-118 + 'puppet_enterprise::profile::master::puppetdb' => { + 'ha_enabled_replicas' => [], + }, + }, + } + + # Configure the B pool for legacy compilers. There are up to two pools for DR, each + # having an affinity for one "availability zone" or the other. + node_group { 'PE Legacy Compiler Group B': + ensure => 'present', + parent => 'PE Legacy Compiler', + rule => ['and', + ['=', ['trusted', 'extensions', 'pp_auth_role'], 'pe_compiler'], + ['=', ['trusted', 'extensions', peadm::oid('peadm_availability_group')], 'B'], + ['=', ['trusted', 'extensions', peadm::oid('peadm_legacy_compiler')], 'true'], + ], + classes => { + 'puppet_enterprise::profile::master' => { + 'puppetdb_host' => [$internal_compiler_a_pool_address, $internal_compiler_a_pool_address].filter |$_| { $_ }, + 'puppetdb_port' => [8081], + }, + }, + data => { + # Workaround for GH-118 + 'puppet_enterprise::profile::master::puppetdb' => { + 'ha_enabled_replicas' => [], + }, + }, + } } diff --git a/plans/convert.pp b/plans/convert.pp index 1995a0b0..e697e87d 100644 --- a/plans/convert.pp +++ b/plans/convert.pp @@ -10,6 +10,7 @@ # Large Optional[TargetSpec] $compiler_hosts = undef, + Optional[TargetSpec] $legacy_compilers = undef, # Extra Large Optional[Peadm::SingleTargetSpec] $primary_postgresql_host = undef, @@ -36,6 +37,7 @@ $replica_target = peadm::get_targets($replica_host, 1) $replica_postgresql_target = peadm::get_targets($replica_postgresql_host, 1) $compiler_targets = peadm::get_targets($compiler_hosts) + $legacy_compiler_targets = peadm::get_targets($legacy_compilers) $primary_postgresql_target = peadm::get_targets($primary_postgresql_host, 1) $all_targets = peadm::flatten_compact([ @@ -43,6 +45,7 @@ $replica_target, $replica_postgresql_target, $compiler_targets, + $legacy_compiler_targets, $primary_postgresql_target, ]) @@ -53,6 +56,7 @@ $primary_postgresql_host, $replica_postgresql_host, $compiler_hosts, + $legacy_compilers, ) out::message('# Gathering information') @@ -115,10 +119,36 @@ $index % 2 != 0 } } + $legacy_compiler_a_targets = $legacy_compiler_targets.filter |$index,$target| { + $exts = $cert_extensions[$target.peadm::certname()] + if ($exts[peadm::oid('peadm_availability_group')] in ['A', 'B']) { + $exts[peadm::oid('peadm_availability_group')] == 'A' + } + elsif ($exts[peadm::oid('pp_cluster')] in ['A', 'B']) { + $exts[peadm::oid('pp_cluster')] == 'A' + } + else { + $index % 2 == 0 + } + } + $legacy_compiler_b_targets = $legacy_compiler_targets.filter |$index,$target| { + $exts = $cert_extensions[$target.peadm::certname()] + if ($exts[peadm::oid('peadm_availability_group')] in ['A', 'B']) { + $exts[peadm::oid('peadm_availability_group')] == 'B' + } + elsif ($exts[peadm::oid('pp_cluster')] in ['A', 'B']) { + $exts[peadm::oid('pp_cluster')] == 'B' + } + else { + $index % 2 != 0 + } + } } else { $compiler_a_targets = $compiler_targets $compiler_b_targets = [] + $legacy_compiler_a_targets = $legacy_compiler_targets + $legacy_compiler_b_targets = [] } # Modify csr_attributes.yaml and insert the peadm-specific OIDs to identify @@ -185,6 +215,7 @@ add_extensions => { peadm::oid('pp_auth_role') => 'pe_compiler', peadm::oid('peadm_availability_group') => 'A', + peadm::oid('peadm_legacy_compiler') => 'false', }, ) }, @@ -194,6 +225,27 @@ add_extensions => { peadm::oid('pp_auth_role') => 'pe_compiler', peadm::oid('peadm_availability_group') => 'B', + peadm::oid('peadm_legacy_compiler') => 'false', + }, + ) + }, + background('modify-compilers-a-certs') || { + run_plan('peadm::modify_certificate', $legacy_compiler_a_targets, + primary_host => $primary_target, + add_extensions => { + peadm::oid('pp_auth_role') => 'pe_compiler', + peadm::oid('peadm_availability_group') => 'A', + peadm::oid('peadm_legacy_compiler') => 'true', + }, + ) + }, + background('modify-compilers-b-certs') || { + run_plan('peadm::modify_certificate', $legacy_compiler_b_targets, + primary_host => $primary_target, + add_extensions => { + peadm::oid('pp_auth_role') => 'pe_compiler', + peadm::oid('peadm_availability_group') => 'B', + peadm::oid('peadm_legacy_compiler') => 'true', }, ) }, @@ -252,6 +304,20 @@ # completion run_command('systemctl restart pe-puppetserver.service pe-puppetdb.service', $all_targets - $primary_target - $primary_postgresql_target - $replica_postgresql_target) + + # Run puppet on all targets again to ensure everything is fully up-to-date + run_task('peadm::puppet_runonce', $all_targets) + } + + if $legacy_compilers { +# lint:ignore:strict_indent + $warning_msg = run_task('peadm::check_legacy_compilers', $primary_host, legacy_compilers => $legacy_compilers.join(',') ).first.message + if $warning_msg.length > 0 { + out::message(@("WARN"/L)) + WARNING: ${warning_msg} + | WARN + } +# lint:endignore } return("Conversion to peadm Puppet Enterprise ${arch['architecture']} completed.") diff --git a/plans/convert_compiler_to_legacy.pp b/plans/convert_compiler_to_legacy.pp new file mode 100644 index 00000000..c75924bd --- /dev/null +++ b/plans/convert_compiler_to_legacy.pp @@ -0,0 +1,154 @@ +# @api private +plan peadm::convert_compiler_to_legacy ( + Peadm::SingleTargetSpec $primary_host, + TargetSpec $legacy_hosts, + Optional[Boolean] $remove_pdb = true, +) { + $primary_target = peadm::get_targets($primary_host, 1) + $convert_legacy_compiler_targets = peadm::get_targets($legacy_hosts) + + $cluster = run_task('peadm::get_peadm_config', $primary_host).first.value + $error = getvar('cluster.error') + if $error { + fail_plan($error) + } + + apply($primary_target) { + class { 'peadm::setup::node_manager_yaml': + primary_host => $primary_target.peadm::certname() ? { + undef => $primary_target, + default => $primary_target.peadm::certname(), + }, + } + + class { 'peadm::setup::legacy_compiler_group': + primary_host => $primary_target.peadm::certname() ? { + undef => $primary_target, + default => $primary_target.peadm::certname(), + }, + internal_compiler_a_pool_address => getvar('cluster.params.internal_compiler_a_pool_address'), + internal_compiler_b_pool_address => getvar('cluster.params.internal_compiler_b_pool_address'), + require => Class['peadm::setup::node_manager_yaml'], + } + } + + $replica_host = getvar('cluster.params.replica_host') + $primary_postgresql_host = getvar('cluster.params.primary_postgresql_host') + $replica_postgresql_host = getvar('cluster.params.replica_postgresql_host') + $compiler_hosts = getvar('cluster.params.compiler_hosts') + $legacy_compilers = getvar('cluster.params.legacy_hosts') + + $replica_target = peadm::get_targets($replica_host, 1) + $primary_postgresql_target = peadm::get_targets($primary_postgresql_host, 1) + $replica_postgresql_target = peadm::get_targets($replica_postgresql_host, 1) + $compiler_targets = peadm::get_targets($compiler_hosts) - $convert_legacy_compiler_targets + $legacy_targets = peadm::get_targets($legacy_compilers) + $convert_legacy_compiler_targets + + $all_targets = peadm::flatten_compact([ + $primary_target, + $replica_target, + $primary_postgresql_target, + $replica_postgresql_target, + $compiler_targets, + $legacy_targets, + ]) + + # Ensure input valid for a supported architecture + $arch = peadm::assert_supported_architecture( + $primary_host, + $replica_host, + $primary_postgresql_host, + $replica_postgresql_host, + $compiler_hosts, + $legacy_compilers, + ) + + if $arch['disaster-recovery'] { + # Gather certificate extension information from all systems + $cert_extensions = run_task('peadm::cert_data', $legacy_targets).reduce({}) |$memo,$result| { + $memo + { $result.target.peadm::certname => $result['extensions'] } + } + $legacy_compiler_a_targets = $convert_legacy_compiler_targets.filter |$index,$target| { + $exts = $cert_extensions[$target.peadm::certname()] + if ($exts[peadm::oid('peadm_availability_group')] in ['A', 'B']) { + $exts[peadm::oid('peadm_availability_group')] == 'A' + } + elsif ($exts[peadm::oid('pp_cluster')] in ['A', 'B']) { + $exts[peadm::oid('pp_cluster')] == 'A' + } + else { + $index % 2 == 0 + } + } + $legacy_compiler_b_targets = $convert_legacy_compiler_targets.filter |$index,$target| { + $exts = $cert_extensions[$target.peadm::certname()] + if ($exts[peadm::oid('peadm_availability_group')] in ['A', 'B']) { + $exts[peadm::oid('peadm_availability_group')] == 'B' + } + elsif ($exts[peadm::oid('pp_cluster')] in ['A', 'B']) { + $exts[peadm::oid('pp_cluster')] == 'B' + } + else { + $index % 2 != 0 + } + } + } else { + $legacy_compiler_a_targets = $convert_legacy_compiler_targets + $legacy_compiler_b_targets = [] + } + + wait([ + background('modify-compilers-certs') || { + run_plan('peadm::modify_certificate', $compiler_targets, + primary_host => $primary_target, + add_extensions => { + peadm::oid('peadm_legacy_compiler') => 'false', + }, + ) + }, + background('modify-compilers-a-certs') || { + run_plan('peadm::modify_certificate', $legacy_compiler_a_targets, + primary_host => $primary_target, + add_extensions => { + peadm::oid('pp_auth_role') => 'pe_compiler', + peadm::oid('peadm_availability_group') => 'A', + peadm::oid('peadm_legacy_compiler') => 'true', + }, + ) + }, + background('modify-compilers-b-certs') || { + run_plan('peadm::modify_certificate', $legacy_compiler_b_targets, + primary_host => $primary_target, + add_extensions => { + peadm::oid('pp_auth_role') => 'pe_compiler', + peadm::oid('peadm_availability_group') => 'B', + peadm::oid('peadm_legacy_compiler') => 'true', + }, + ) + }, + ]) + + if $remove_pdb { + run_command('puppet resource service puppet ensure=stopped', $convert_legacy_compiler_targets) + run_command('puppet resource service pe-puppetdb ensure=stopped enable=false', $convert_legacy_compiler_targets) + } + + run_task('peadm::puppet_runonce', $convert_legacy_compiler_targets) + run_task('peadm::puppet_runonce', $compiler_targets) + run_task('peadm::puppet_runonce', $primary_target) + run_task('peadm::puppet_runonce', $all_targets) + + if $remove_pdb { + run_command('puppet resource package pe-puppetdb ensure=purged', $convert_legacy_compiler_targets) + run_command('puppet resource user pe-puppetdb ensure=absent', $convert_legacy_compiler_targets) + + run_command('rm -rf /etc/puppetlabs/puppetdb', $convert_legacy_compiler_targets) + run_command('rm -rf /var/log/puppetlabs/puppetdb', $convert_legacy_compiler_targets) + run_command('rm -rf /opt/puppetlabs/server/data/puppetdb', $convert_legacy_compiler_targets) + } + + run_command('systemctl start pe-puppetserver.service', $convert_legacy_compiler_targets) + run_command('puppet resource service puppet ensure=running', $convert_legacy_compiler_targets) + + return("Converted host ${convert_legacy_compiler_targets} to legacy compiler.") +} diff --git a/plans/install.pp b/plans/install.pp index fb1052c5..58b512ca 100644 --- a/plans/install.pp +++ b/plans/install.pp @@ -38,6 +38,7 @@ # Large Optional[TargetSpec] $compiler_hosts = undef, + Optional[TargetSpec] $legacy_compilers = undef, # Extra Large Optional[Peadm::SingleTargetSpec] $primary_postgresql_host = undef, @@ -85,6 +86,7 @@ # Large compiler_hosts => $compiler_hosts, + legacy_compilers => $legacy_compilers, # Extra Large primary_postgresql_host => $primary_postgresql_host, @@ -123,6 +125,7 @@ # Large compiler_hosts => $compiler_hosts, + legacy_compilers => $legacy_compilers, # Extra Large primary_postgresql_host => $primary_postgresql_host, diff --git a/plans/subplans/component_install.pp b/plans/subplans/component_install.pp index df74079e..daf11688 100644 --- a/plans/subplans/component_install.pp +++ b/plans/subplans/component_install.pp @@ -22,6 +22,10 @@ peadm::oid('pp_auth_role') => 'pe_compiler', peadm::oid('peadm_availability_group') => $avail_group_letter, } + } elsif $role == 'pe_compiler_legacy' { + $certificate_extensions = { + peadm::oid('peadm_role') => $role, + } } else { $certificate_extensions = { peadm::oid('peadm_role') => $role, diff --git a/plans/subplans/configure.pp b/plans/subplans/configure.pp index bccc0503..2afa0084 100644 --- a/plans/subplans/configure.pp +++ b/plans/subplans/configure.pp @@ -27,6 +27,7 @@ # Large Optional[TargetSpec] $compiler_hosts = undef, + Optional[TargetSpec] $legacy_compilers = undef, # Extra Large Optional[Peadm::SingleTargetSpec] $primary_postgresql_host = undef, @@ -51,6 +52,7 @@ $replica_target = peadm::get_targets($replica_host, 1) $replica_postgresql_target = peadm::get_targets($replica_postgresql_host, 1) $compiler_targets = peadm::get_targets($compiler_hosts) + $legacy_compiler_targets = peadm::get_targets($legacy_compilers) $primary_postgresql_target = peadm::get_targets($primary_postgresql_host, 1) # Ensure input valid for a supported architecture @@ -60,6 +62,7 @@ $primary_postgresql_host, $replica_postgresql_host, $compiler_hosts, + $legacy_compilers, ) # Source list of files on Primary and synchronize to new Replica @@ -74,6 +77,7 @@ run_plan('peadm::util::copy_file', peadm::flatten_compact([ $replica_target, $compiler_targets, + $legacy_compiler_targets, ]), source_host => $primary_target, path => $common_content_source @@ -139,6 +143,7 @@ $primary_target, $primary_postgresql_target, $compiler_targets, + $legacy_compiler_targets, $replica_target, $replica_postgresql_target, ])) @@ -161,6 +166,7 @@ $primary_postgresql_target, $replica_postgresql_target, $compiler_targets, + $legacy_compiler_targets, ])) return("Configuration of Puppet Enterprise ${arch['architecture']} succeeded.") diff --git a/plans/subplans/install.pp b/plans/subplans/install.pp index 7d48cb8a..d349e23e 100644 --- a/plans/subplans/install.pp +++ b/plans/subplans/install.pp @@ -41,6 +41,7 @@ # Large Optional[TargetSpec] $compiler_hosts = undef, + Optional[TargetSpec] $legacy_compilers = undef, # Extra Large Optional[Peadm::SingleTargetSpec] $primary_postgresql_host = undef, @@ -79,6 +80,7 @@ $primary_postgresql_target = peadm::get_targets($primary_postgresql_host, 1) $replica_postgresql_target = peadm::get_targets($replica_postgresql_host, 1) $compiler_targets = peadm::get_targets($compiler_hosts) + $legacy_compiler_targets = peadm::get_targets($legacy_compilers) # Ensure input valid for a supported architecture $arch = peadm::assert_supported_architecture( @@ -87,6 +89,7 @@ $primary_postgresql_host, $replica_postgresql_host, $compiler_hosts, + $legacy_compilers, ) $all_targets = peadm::flatten_compact([ @@ -95,6 +98,7 @@ $replica_target, $replica_postgresql_target, $compiler_targets, + $legacy_compiler_targets, ]) $primary_targets = peadm::flatten_compact([ @@ -115,6 +119,7 @@ $agent_installer_targets = peadm::flatten_compact([ $compiler_targets, + $legacy_compiler_targets, $replica_target, ]) @@ -122,10 +127,14 @@ if $arch['disaster-recovery'] { $compiler_a_targets = $compiler_targets.filter |$index,$target| { $index % 2 == 0 } $compiler_b_targets = $compiler_targets.filter |$index,$target| { $index % 2 != 0 } + $legacy_a_targets = $legacy_compiler_targets.filter |$index,$target| { $index % 2 == 0 } + $legacy_b_targets = $legacy_compiler_targets.filter |$index,$target| { $index % 2 != 0 } } else { $compiler_a_targets = $compiler_targets $compiler_b_targets = [] + $legacy_a_targets = $legacy_compiler_targets + $legacy_b_targets = [] } $dns_alt_names_csv = $dns_alt_names.reduce |$csv,$x| { "${csv},${x}" } @@ -143,7 +152,7 @@ true } elsif $replica_host { true - } elsif $compiler_hosts { + } elsif $compiler_hosts or $legacy_compilers { true } else { $code_manager_auto_configure @@ -179,7 +188,7 @@ # puppet and are present in PuppetDB, it is not necessary anymore. $puppetdb_database_temp_config = { 'puppet_enterprise::profile::database::puppetdb_hosts' => ( - $compiler_targets + $primary_target + $replica_target + $compiler_targets + $legacy_compiler_targets + $primary_target + $replica_target ).map |$t| { $t.peadm::certname() }, } @@ -278,6 +287,7 @@ extension_requests => { peadm::oid('pp_auth_role') => 'pe_compiler', peadm::oid('peadm_availability_group') => 'A', + peadm::oid('peadm_legacy_compiler') => 'false', } ) }, @@ -286,6 +296,25 @@ extension_requests => { peadm::oid('pp_auth_role') => 'pe_compiler', peadm::oid('peadm_availability_group') => 'B', + peadm::oid('peadm_legacy_compiler') => 'false', + } + ) + }, + background('compiler-a-csr.yaml') || { + run_plan('peadm::util::insert_csr_extension_requests', $legacy_a_targets, + extension_requests => { + peadm::oid('pp_auth_role') => 'pe_compiler', + peadm::oid('peadm_availability_group') => 'A', + peadm::oid('peadm_legacy_compiler') => 'true', + } + ) + }, + background('compiler-b-csr.yaml') || { + run_plan('peadm::util::insert_csr_extension_requests', $legacy_b_targets, + extension_requests => { + peadm::oid('pp_auth_role') => 'pe_compiler', + peadm::oid('peadm_availability_group') => 'B', + peadm::oid('peadm_legacy_compiler') => 'true', } ) }, diff --git a/plans/update_compiler_extensions.pp b/plans/update_compiler_extensions.pp new file mode 100644 index 00000000..784f919e --- /dev/null +++ b/plans/update_compiler_extensions.pp @@ -0,0 +1,25 @@ +# @api private +plan peadm::update_compiler_extensions ( + TargetSpec $compiler_hosts, + Peadm::SingleTargetSpec $primary_host, + Boolean $legacy = false, +) { + $primary_target = peadm::get_targets($primary_host, 1) + $host_targets = peadm::get_targets($compiler_hosts) + + run_plan('peadm::modify_certificate', $host_targets, + primary_host => $primary_target, + add_extensions => { peadm::oid('peadm_legacy_compiler') => String($legacy) }, + ) + + run_task('peadm::puppet_runonce', $primary_target) + run_task('peadm::puppet_runonce', $host_targets) + + if $legacy { + run_command('systemctl restart pe-puppetserver.service', $host_targets) + } else { + run_command('systemctl restart pe-puppetserver.service pe-puppetdb.service', $host_targets) + } + + return("Added legacy cert with value ${legacy} to compiler hosts ${compiler_hosts}") +} diff --git a/plans/upgrade.pp b/plans/upgrade.pp index b9adcd38..63375829 100644 --- a/plans/upgrade.pp +++ b/plans/upgrade.pp @@ -167,11 +167,25 @@ == $cert_extensions.dig($primary_target[0].peadm::certname, peadm::oid('peadm_availability_group'))) } + $compiler_m1_nonlegacy_targets = $compiler_targets.filter |$target| { + ($cert_extensions.dig($target.peadm::certname, peadm::oid('peadm_availability_group')) + == $cert_extensions.dig($primary_target[0].peadm::certname, peadm::oid('peadm_availability_group'))) and + ($cert_extensions.dig($target.peadm::certname, peadm::oid('peadm_legacy_compiler')) + == 'false') + } + $compiler_m2_targets = $compiler_targets.filter |$target| { ($cert_extensions.dig($target.peadm::certname, peadm::oid('peadm_availability_group')) == $cert_extensions.dig($replica_target[0].peadm::certname, peadm::oid('peadm_availability_group'))) } + $compiler_m2_nonlegacy_targets = $compiler_targets.filter |$target| { + ($cert_extensions.dig($target.peadm::certname, peadm::oid('peadm_availability_group')) + == $cert_extensions.dig($replica_target[0].peadm::certname, peadm::oid('peadm_availability_group'))) and + ($cert_extensions.dig($target.peadm::certname, peadm::oid('peadm_legacy_compiler')) + == 'false') + } + peadm::plan_step('preparation') || { if $download_mode == 'bolthost' { # Download the PE tarball on the nodes that need it @@ -239,7 +253,7 @@ peadm::plan_step('upgrade-primary') || { # Shut down PuppetDB on CMs that use the PM's PDB PG. Use run_command instead # of run_task(service, ...) so that upgrading from 2018.1 works over PCP. - run_command('systemctl stop pe-puppetdb', $compiler_m1_targets) + run_command('systemctl stop pe-puppetdb', $compiler_m1_nonlegacy_targets) run_task('peadm::pe_install', $primary_postgresql_target, tarball => $upload_tarball_path, @@ -344,7 +358,7 @@ # Shut down PuppetDB on CMs that use the replica's PDB PG. Use run_command # instead of run_task(service, ...) so that upgrading from 2018.1 works # over PCP. - run_command('systemctl stop pe-puppetdb', $compiler_m2_targets) + run_command('systemctl stop pe-puppetdb', $compiler_m2_nonlegacy_targets) run_task('peadm::pe_install', $replica_postgresql_target, tarball => $upload_tarball_path, diff --git a/spec/acceptance/peadm_spec/plans/provision_test_cluster.pp b/spec/acceptance/peadm_spec/plans/provision_test_cluster.pp index aecd8197..4529dc9b 100644 --- a/spec/acceptance/peadm_spec/plans/provision_test_cluster.pp +++ b/spec/acceptance/peadm_spec/plans/provision_test_cluster.pp @@ -20,6 +20,9 @@ 'large': { ['primary', 'compiler'] } + 'large-with-two-compilers': { + ['primary', 'compiler', 'compiler'] + } 'large-with-dr': { ['primary', 'compiler', 'replica', 'compiler'] } diff --git a/spec/functions/assert_supported_architecture_spec.rb b/spec/functions/assert_supported_architecture_spec.rb index d48342f3..5f67a33b 100644 --- a/spec/functions/assert_supported_architecture_spec.rb +++ b/spec/functions/assert_supported_architecture_spec.rb @@ -24,6 +24,15 @@ let(:compiler_hosts) do 'pup-c1.puppet.vm' end + let(:legacy_compilers) do + 'pup-c2.puppet.vm' + end + let(:compiler_hosts_array) do + ['pup-c1.puppet.vm'] + end + let(:legacy_compilers_array) do + ['pup-c2.puppet.vm'] + end it { is_expected.to run.with_params(primary_host) @@ -59,6 +68,28 @@ 'disaster-recovery' => false, 'architecture' => 'large') end + it do + is_expected.to run.with_params(primary_host, + nil, + nil, + nil, + compiler_hosts, + legacy_compilers) + .and_return('supported' => true, + 'disaster-recovery' => false, + 'architecture' => 'large') + end + it do + is_expected.to run.with_params(primary_host, + nil, + nil, + nil, + nil, + legacy_compilers) + .and_return('supported' => true, + 'disaster-recovery' => false, + 'architecture' => 'large') + end it do is_expected.to run.with_params(primary_host, @@ -81,4 +112,26 @@ 'disaster-recovery' => false, 'architecture' => 'extra-large') end + it do + is_expected.to run.with_params(primary_host, + nil, + primary_postgresql_host, + nil, + compiler_hosts, + legacy_compilers) + .and_return('supported' => true, + 'disaster-recovery' => false, + 'architecture' => 'extra-large') + end + it do + is_expected.to run.with_params(primary_host, + nil, + primary_postgresql_host, + nil, + compiler_hosts_array, + legacy_compilers_array) + .and_return('supported' => true, + 'disaster-recovery' => false, + 'architecture' => 'extra-large') + end end diff --git a/spec/plans/subplans/install_spec.rb b/spec/plans/subplans/install_spec.rb index f4ed169e..c83a47a7 100644 --- a/spec/plans/subplans/install_spec.rb +++ b/spec/plans/subplans/install_spec.rb @@ -12,7 +12,15 @@ allow_task('peadm::precheck').return_for_targets( 'primary' => { 'hostname' => 'primary', - 'platform' => 'el-7.11-x86_64' + 'platform' => 'el-7.11-x86_64', + }, + 'compiler1' => { + 'hostname' => 'compiler1', + 'platform' => 'el-7.11-x86_64', + }, + 'compiler2' => { + 'hostname' => 'compiler2', + 'platform' => 'el-7.11-x86_64', }, ) @@ -39,9 +47,9 @@ it 'minimum variables to run' do params = { - 'primary_host' => 'primary', + 'primary_host' => 'primary', 'console_password' => 'puppetLabs123!', - 'version' => '2019.8.12', + 'version' => '2019.8.12', } expect(run_plan('peadm::subplans::install', params)).to be_ok @@ -49,10 +57,10 @@ it 'installs 2023.4 without r10k_known_hosts' do params = { - 'primary_host' => 'primary', - 'console_password' => 'puppetLabs123!', - 'version' => '2023.4.0', - 'r10k_remote' => 'git@github.com:puppetlabs/nothing', + 'primary_host' => 'primary', + 'console_password' => 'puppetLabs123!', + 'version' => '2023.4.0', + 'r10k_remote' => 'git@github.com:puppetlabs/nothing', 'r10k_private_key_content' => '-----BEGINfoo', } @@ -61,16 +69,16 @@ it 'installs 2023.4+ with r10k_private_key and r10k_known_hosts' do params = { - 'primary_host' => 'primary', - 'console_password' => 'puppetLabs123!', - 'version' => '2023.4.0', - 'r10k_remote' => 'git@github.com:puppetlabs/nothing', + 'primary_host' => 'primary', + 'console_password' => 'puppetLabs123!', + 'version' => '2023.4.0', + 'r10k_remote' => 'git@github.com:puppetlabs/nothing', 'r10k_private_key_content' => '-----BEGINfoo', - 'r10k_known_hosts' => [ + 'r10k_known_hosts' => [ { 'name' => 'test', 'type' => 'key-type', - 'key' => 'abcdef', + 'key' => 'abcdef', }, ], 'permit_unsafe_versions' => true, @@ -78,4 +86,14 @@ expect(run_plan('peadm::subplans::install', params)).to be_ok end + + it 'installs 2021.7.9 with legacy compilers' do + params = { + 'primary_host' => 'primary', + 'console_password' => 'puppetLabs123!', + 'version' => '2021.7.9', + 'legacy_compilers' => ['compiler1', 'compiler2'], + } + expect(run_plan('peadm::subplans::install', params)).to be_ok + end end diff --git a/tasks/check_legacy_compilers.rb b/tasks/check_legacy_compilers.rb new file mode 100755 index 00000000..7ca7220b --- /dev/null +++ b/tasks/check_legacy_compilers.rb @@ -0,0 +1,67 @@ +#!/opt/puppetlabs/puppet/bin/ruby +# frozen_string_literal: true + +require 'json' +require 'uri' +require 'net/http' +require 'puppet' + +# CheckLegacyCompilers task class +class CheckLegacyCompilers + def initialize(params) + @nodes = params['legacy_compilers'].split(',') if params['legacy_compilers'].is_a?(String) + end + + def execute! + pinned_nodes = [] + @nodes.each do |node| + node_classification = get_node_classification(node) + pinned = false + node_classification['groups'].each do |group| + if group['name'] == 'PE Master' + pinned_nodes << node + pinned = true + end + end + next if pinned + next unless node_classification.key?('parameters') + next unless node_classification['parameters'].key?('pe_master') + if node_classification['parameters']['pe_master'] + pinned_nodes << node + end + end + + return unless !pinned_nodes.empty? + puts 'The following legacy compilers are classified as Puppet primary nodes:' + puts pinned_nodes.join(', ') + puts 'To continue with the upgrade, ensure that these compilers are no longer recognized as Puppet primary nodes.' + end + + def https(port) + https = Net::HTTP.new('localhost', port) + https.use_ssl = true + https.cert = @cert ||= OpenSSL::X509::Certificate.new(File.read(Puppet.settings[:hostcert])) + https.key = @key ||= OpenSSL::PKey::RSA.new(File.read(Puppet.settings[:hostprivkey])) + https.verify_mode = OpenSSL::SSL::VERIFY_NONE + https + end + + def get_node_classification(certname) + pdb = https(4433) + pdb_request = Net::HTTP::Post.new('/classifier-api/v2/classified/nodes/' + certname) + pdb_request['Content-Type'] = 'application/json' + + response = JSON.parse(pdb.request(pdb_request).body) + + response + end +end + +# Run the task unless an environment flag has been set, signaling not to. The +# environment flag is used to disable auto-execution and enable Ruby unit +# testing of this task. +unless ENV['RSPEC_UNIT_TEST_MODE'] + Puppet.initialize_settings + task = CheckLegacyCompilers.new(JSON.parse(STDIN.read)) + task.execute! +end diff --git a/tasks/classify_compilers.json b/tasks/classify_compilers.json new file mode 100644 index 00000000..cb85a1dd --- /dev/null +++ b/tasks/classify_compilers.json @@ -0,0 +1,15 @@ +{ + "description": "Classify compilers as legacy or non-legacy", + "parameters": { + "compiler_hosts": { + "type": "Array[String]", + "description": "List of FQDNs of compilers" + } + }, + "implementations": [ + { + "name": "classify_compilers.rb", + "requirements": ["shell"] + } + ] +} \ No newline at end of file diff --git a/tasks/classify_compilers.rb b/tasks/classify_compilers.rb new file mode 100755 index 00000000..754f6ad9 --- /dev/null +++ b/tasks/classify_compilers.rb @@ -0,0 +1,43 @@ +#!/usr/bin/env ruby + +require 'json' +require 'open3' + +def classify_compiler(services) + if services.any? { |service| service['type'] == 'puppetdb' } + :non_legacy + else + :legacy + end +end + +params = JSON.parse(STDIN.read) +compiler_hosts = params['compiler_hosts'] + +legacy_compilers = [] +non_legacy_compilers = [] + +compiler_hosts.each do |compiler| + cmd = "puppet infra status --host #{compiler} --format=json" + stdout, stderr, status = Open3.capture3(cmd) + + if status.success? + services = JSON.parse(stdout) + classification = classify_compiler(services) + + if classification == :legacy + legacy_compilers << compiler + else + non_legacy_compilers << compiler + end + else + STDERR.puts "Error running command for #{compiler}: #{stderr}" + end +end + +result = { + 'legacy_compilers' => legacy_compilers, + 'compilers' => non_legacy_compilers +} + +puts result.to_json diff --git a/tasks/get_peadm_config.rb b/tasks/get_peadm_config.rb index 88eab7ba..e4bb7ba4 100755 --- a/tasks/get_peadm_config.rb +++ b/tasks/get_peadm_config.rb @@ -47,6 +47,7 @@ def config 'primary_postgresql_host' => postgresql[primary_letter], 'replica_postgresql_host' => postgresql[replica_letter], 'compilers' => compilers.map { |c| c['certname'] }, + 'legacy_compilers' => legacy_compilers.map { |c| c['certname'] }, 'compiler_pool_address' => groups.dig('PE Master', 'config_data', 'pe_repo', 'compile_master_pool_address'), 'internal_compiler_a_pool_address' => groups.dig('PE Compiler Group B', 'classes', 'puppet_enterprise::profile::master', 'puppetdb_host', 1), 'internal_compiler_b_pool_address' => groups.dig('PE Compiler Group A', 'classes', 'puppet_enterprise::profile::master', 'puppetdb_host', 1), @@ -63,7 +64,11 @@ def config 'compilers' => { 'A' => compilers.select { |c| c['letter'] == 'A' }.map { |c| c['certname'] }, 'B' => compilers.select { |c| c['letter'] == 'B' }.map { |c| c['certname'] }, - } + }, + 'legacy_compilers' => { + 'A' => legacy_compilers.select { |c| c['letter'] == 'A' }.map { |c| c['certname'] }, + 'B' => legacy_compilers.select { |c| c['letter'] == 'B' }.map { |c| c['certname'] }, + }, }, } end @@ -81,7 +86,24 @@ def groups # Returns a list of compiler certnames and letters, based on a PuppetDB query def compilers @compilers ||= - pdb_query('inventory[certname,trusted.extensions] { trusted.extensions.pp_auth_role = "pe_compiler" }').map do |c| + pdb_query('inventory[certname,trusted.extensions] { + trusted.extensions.pp_auth_role = "pe_compiler" and + trusted.extensions."1.3.6.1.4.1.34380.1.1.9814" = "false" + }').map do |c| + { + 'certname' => c['certname'], + 'letter' => c.dig('trusted.extensions', '1.3.6.1.4.1.34380.1.1.9813'), + } + end + end + + # Returns a list of legacy compiler certnames and letters, based on a PuppetDB query + def legacy_compilers + @legacy_compilers ||= + pdb_query('inventory[certname,trusted.extensions] { + trusted.extensions.pp_auth_role = "pe_compiler" and + trusted.extensions."1.3.6.1.4.1.34380.1.1.9814" = "true" + }').map do |c| { 'certname' => c['certname'], 'letter' => c.dig('trusted.extensions', '1.3.6.1.4.1.34380.1.1.9813'),