Skip to main content

Upgrade to Chef Automate 4.x

[edit on GitHub]

Warning

Elasticsearch support has been removed from this version (4.0.27) of Chef Automate.

Chef Automate provides an entire suite of enterprise capabilities for node visibility and compliance. Chef Automate upgrades from one minor version to another automatically. However, Chef Automate will not automatically upgrade to a major version. See the instructions below for manually upgrading Chef Automate from date-based versions to Chef Automate 4.x.

Upgrade Journey

Please choose the following upgrade journey based on your current version of Chef Automate. All of these upgrades are manual upgrades.

Your Current VersionUpgrade To
Any version before 2022032909144220220329091442
202203290914423.0.49
3.0.494.0.x

For example, if today you are on version 2021201164433, your upgrade journey should be:

  1. Manual upgrade to 20220329091442
  2. Manual upgrade to 3.0.49
  3. Manual upgrade to 4.0.x

Prerequisites

Note

If your Elasticsearch contains older indexes of version 5, please re-index them to version 6 with the same name, before proceeding with the upgrade.
  • Plan your downtime: This upgrade requires downtime. Before upgrading, set the environment to handle the downtime. Run sudo chef-automate maintenance on to turn on maintenance mode.
  • Backup Chef Automate database: This Chef Automate version upgrades ElasticSearch. Backup your data before upgrading.
  • Current version should be 3.0.49: If you are not on 3.0.49 version, do regular upgrades according to your topology.

Upgrade to Version 3.0.49

Check your current version:

sudo chef-automate version

Note

If your server Version is less than 20220329091442, refer to Upgrade to 3.0.49 page.

Note

If your server sersion is less than 3.0.49, upgrade to latest minor version in 3 series.

Normal and airgapped upgrade to 3.0.49

To upgrade to automate 3.0.49 follow the link: upgrading to 3.0.49

Upgrade Path from 3.0.49 to 4.0.x

There are four possible scenarios to upgrade from 3.0.49 to 4.0.x version.

Note

Confirm whether your installation is using an external Elasticsearch but running the chef-automate config show command. It enable=true is present in the global.v1.external.elasticsearch config setting, you are using a external Elasticsearch.

Warning

You drive should have a minimum of sixty percent of free space to start the major version upgrade.

Warning

Disable the sharding for automate running embedded Elasticsearch.

Also, accept the checklist item asking permission to disable sharding.

Chef Automate With Embedded Elasticsearch

To upgrade Chef Automate with embedded Elasticsearch, follow the steps given below:

Upgrade Chef Automate from version 3.0.49 to 4.0.x

  1. Start a major version upgrade:

Here, you will be prompted to accept multiple Pre Upgrade checklist. Accept the actions before upgrade.

sudo chef-automate upgrade run --major

Once you are done with the upgrade, follow the steps post upgrade which are:

  1. Check the upgrade status of Chef Automate:
sudo chef-automate upgrade status
  1. Turn off maintenance mode:
sudo chef-automate maintenance off
  1. All relevant configuration fields of the Elasticsearch should be copied into the OpenSearch configuration.

For Example:

If Elasticsearch configuration was:

[elasticsearch]
  [elasticsearch.v1]
    [elasticsearch.v1.sys]
      [elasticsearch.v1.sys.cluster]
        max_shards_per_node = 6000
      [elasticsearch.v1.sys.indices]
        [elasticsearch.v1.sys.indices.breaker]
          total_limit = "95%"
      [elasticsearch.v1.sys.runtime]
        max_open_files = "65536"
        max_locked_memory = "unlimited"
        heapsize = "8g" # This should be the 50% of RAM```

Then add in OpenSearch configuration as:

[opensearch]
  [opensearch.v1]
    [opensearch.v1.sys]
      [opensearch.v1.sys.cluster]
        max_shards_per_node = 6000 # Refer the value from ElasticSearch Config, If this value is not there in elastic search config, then do not patch in openseaarch.
      [opensearch.v1.sys.indices]
        [opensearch.v1.sys.indices.breaker]
          total_limit = "95%"
      [opensearch.v1.sys.runtime]
        max_open_files = "65536"
        max_locked_memory = "unlimited"
        heapsize = "8g" # This should be the 50% of RAM

Warning

Configure the OpenSearch Heap size to 50% of RAM, but not more than 32GB.

Apply this using the config patch command.

  1. Migrate your data from ElasticSearch 6.8 to OpenSearch 1.2.4:
sudo chef-automate post-major-upgrade migrate --data=es
  1. Verify whether all services are running:
sudo chef-automate status
  1. Clear the old ElasticSearch 6.8 data if all the data is present in your upgraded Chef Automate.
sudo chef-automate post-major-upgrade clear-data --data=es

Chef Automate with External ElasticSearch

To upgrade Chef Automate with external Elasticsearch, follow the steps given below:

Upgrade Chef Automate from version 3.0.49 to 4.0.x

  1. Start major version upgrade:
sudo chef-automate upgrade run --major
  1. Check upgrade status is up-to-date
sudo chef-automate status
  1. Upgrade your external ElasticSearch 6.8 to OpenSearch 1.2.4 manually. If you have configured Host, Port, Username or Password of ElasticSearch, patch the new configuration to use Chef Automate.

  2. All relevant configuration fields of the Elasticsearch should be copied into the OpenSearch configuration.

    Please refer to the elasticsearch.yml file to get the applied configuration on your external Elasticsearch. Add the releavent configuration from external Elasticsearch (elasticsearch.yml) to the opensearch.yml on your external OpenSearch.

  3. Turn off maintenance mode:

sudo chef-automate maintenance off

Chef Automate in Air-Gapped Environment With Embedded ElasticSearch

Upgrade Chef Automate from version 3.0.49 to 4.0.x

To upgrade to 4.0.x, follow the steps below:

On Internet connected machine

  1. Download latest CLI of Chef Automate.
curl https://packages.chef.io/files/current/latest/chef-automate-cli/chef-automate_linux_amd64.zip | gunzip - > chef-automate && chmod +x chef-automate
  1. Create an Airgap Installation Bundle (AIB).
sudo ./chef-automate airgap bundle create 

OR we can directly download via curl request

curl https://packages.chef.io/airgap_bundle/current/automate/latest.aib -o automate-4.x.y.aib
  1. Copy the latest Chef Automate CLI (chef-automate) and AIB (automate-4.x.y.aib) to the air-gapped machine running Chef Automate.

On Air-Gapped machine running Chef Automate

  1. Set your upgrade strategy to none in Chef Automate config. Confirm by the following code:
sudo ./chef-automate config show
  1. Upgrade using new AIB and Chef Automate CLI:
sudo ./chef-automate upgrade run --airgap-bundle automate-4.x.y.aib --major

Post Upgrade

  1. Check the upgrade status of Chef Automate, run the following command:
sudo chef-automate upgrade status
  1. Turn off the maintenance mode.
sudo chef-automate maintenance off
  1. All relevant configuration fields of the Elasticsearch should be copied into the OpenSearch configuration.

For Example:

If Elasticsearch configuration was:

[elasticsearch]
  [elasticsearch.v1]
    [elasticsearch.v1.sys]
      [elasticsearch.v1.sys.cluster]
        max_shards_per_node = 6000
      [elasticsearch.v1.sys.indices]
        [elasticsearch.v1.sys.indices.breaker]
          total_limit = "95%"
      [elasticsearch.v1.sys.runtime]
        max_open_files = "65536"
        max_locked_memory = "unlimited"
        heapsize = "8g" # This should be the 50% of RAM```

Then add in OpenSearch configuration as:

[opensearch]
  [opensearch.v1]
    [opensearch.v1.sys]
      [opensearch.v1.sys.cluster]
        max_shards_per_node = 6000 # Refer the value from ElasticSearch Config, If this value is not there in elastic search config, then do not patch in openseaarch.
      [opensearch.v1.sys.indices]
        [opensearch.v1.sys.indices.breaker]
          total_limit = "95%"
      [opensearch.v1.sys.runtime]
        max_open_files = "65536"
        max_locked_memory = "unlimited"
        heapsize = "8g" # This should be the 50% of RAM

Apply this using config patch command.

  1. Migrate your data from ElasticSearch 6.8 to OpenSearch 1.2.4:
sudo chef-automate post-major-upgrade migrate --data=es
  1. Verify whether all services are running:
sudo chef-automate status
  1. Clear the old ElasticSearch 6.8 data if all the data is present in your upgraded Chef Automate.
sudo chef-automate post-major-upgrade clear-data --data=es

Chef Automate in Air-Gapped Environment With External ElasticSearch

Upgrade Chef Automate from version 3.0.49 to 4.0.x

To upgrade to 4.0.x, follow the steps below:

On Internet connected machine

  1. Download latest CLI of Chef Automate
curl https://packages.chef.io/files/current/latest/chef-automate-cli/chef-automate_linux_amd64.zip | gunzip - > chef-automate && chmod +x chef-automate
  1. Create an Airgap Installation Bundle (AIB):
sudo ./chef-automate airgap bundle create

OR we can directly download via curl request

curl https://packages.chef.io/airgap_bundle/current/automate/latest.aib -o automate-4.x.y.aib
  1. Copy the latest Chef Automate CLI (chef-automate) and AIB (automate-4.x.y.aib) to the air-gapped machine running Chef Automate.

On Air-Gapped machine running Chef Automate

  1. Set your upgrade strategy to none in Chef Automate config. Confirm by the following code:
sudo ./chef-automate config show
  1. Upgrade using new AIB and Chef Automate CLI:
sudo ./chef-automate upgrade run --airgap-bundle automate-4.x.y.aib --major
  1. Check whether upgrade status is up-to-date
sudo chef-automate status
  1. Upgrade your external ElasticSearch 6.8 to OpenSearch 1.2.4 manually. If you have configured Host, Port, Username or Password of ElasticSearch, patch the new configuration to use Chef Automate.

  2. All relevant configuration fields of the Elasticsearch should be copied into the OpenSearch configuration.

    Please refer to the elasticsearch.yml file to get the applied configuration on your external Elasticsearch. Add the releavent configuration from external Elasticsearch (elasticsearch.yml) to the opensearch.yml on your external OpenSearch.

  3. Turn off maintenance mode using following command:

sudo chef-automate maintenance off

Note

After upgrading to version 4.x, Automate will have the configurations both for OpenSearch and Elasticsearch. It is recommended to remove the Elasticsearch configuration after upgrading to External OpenSearch.

Troubleshooting

Circuit Breaking Exception

{"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [6126524880/5.7gb], which is larger than the limit of [5988548608/5.5gb], real usage: [6126524880/5.7gb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=74975/73.2kb, in_flight_requests=0/0b, accounting=89882860/85.7mb]","bytes_wanted":6126524880,"bytes_limit":5988548608,"durability":"PERMANENT"}]
  • Update the Opensearch Config, using chef-automate config patch <config_patch.toml>
[opensearch]
  [opensearch.v1]
    [opensearch.v1.sys]
      [opensearch.v1.sys.runtime]
         heapsize = "8g" # This should be Half of RAM
      [opensearch.v1.sys.indices]
        [opensearch.v1.sys.indices.breaker]
          total_limit = "95%"

Shard Failure

[ERROR] Elasticsearch exception [type=validation_exception, reason=Validation Failed: 1: this action would add [5] total shards, but this cluster currently has [997]/[1000] maximum shards open;]

To address the issue of shard limit hitting 1000, we need to increase the max_shards_per_node Update the Opensearch Config, using chef-automate config patch <config_patch.toml>

[opensearch]
  [opensearch.v1]
    [opensearch.v1.sys]
      [opensearch.v1.sys.cluster]
         max_shards_per_node = "<NUMBER_OF_SHARD>"

Migration Fails

If Chef Automate fails to migrate your data to OpenSearch 1.2.4 while running chef-automate post-major-upgrade migrate --data=es, restore the data using:

sudo chef-automate backup restore </path/to/backups/>BACKUP_ID

To restore your Air-Gapped bundle, run the following command:

sudo chef-automate backup restore  --airgap-bundle </path/to/bundle> </path/to/backups/>BACKUP_ID

Click here to know more about the Restore methods.

To start the upgrade, use the backup ID from the backup created. In case the restore fails even after upgrading the Chef Automate version, follow the steps given below:

  1. Uninstall Chef Automate.
sudo chef-automate uninstall
  1. Install the last major version (3.0.49) using the air-gapped installation process.

  2. Restore the backup:

sudo chef-automate backup restore <backup_id>

Refer to the Chef Automate Restore documentation.

Note

Remove the /hab/svc/deployment-service/var/upgrade_metadata.json file if the migration of data has been done using backup and restore method.

Was this page helpful?

×









Search Results