GitLab upgrade from 9.5.10 to 10.8.7 unsuccessful

Question

After installing the package for 10.8.7, which is the next supported upgrade path per https://docs.gitlab.com/ee/update/#upgrade-paths , gitlab-ctl reconfigure fails with:

================================================================================
Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee/recipes/default.rb
================================================================================

NoMethodError
-------------
undefined method `[]' for nil:NilClass

Cookbook Trace:
---------------
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/secrets_helper.rb:81:in `write_to_gitlab_secrets'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab.rb:186:in `generate_secrets'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab.rb:259:in `generate_config'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/config.rb:26:in `from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:26:in `from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee/recipes/default.rb:20:in `from_file'

Relevant File Content:
----------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/secrets_helper.rb:

 74:        },
 75:        'registry' => {
 76:          'http_secret' => Gitlab['registry']['http_secret'],
 77:          'internal_certificate' => Gitlab['registry']['internal_certificate'],
 78:          'internal_key' => Gitlab['registry']['internal_key']
 79:        },
 80:        'letsencrypt' => {
 81>>         'auto_enabled' => Gitlab['letsencrypt']['auto_enabled']
 82:        },
 83:        'mattermost' => {
 84:          'email_invite_salt' => Gitlab['mattermost']['email_invite_salt'],
 85:          'file_public_link_salt' => Gitlab['mattermost']['file_public_link_salt'],
 86:          'sql_at_rest_encrypt_key' => Gitlab['mattermost']['sql_at_rest_encrypt_key']
 87:        },
 88:        'postgresql' => {
 89:          'internal_certificate' => Gitlab['postgresql']['internal_certificate'],
 90:          'internal_key' => Gitlab['postgresql']['internal_key']

Seeing the error, I attempted to add the following to /etc/gitlab/gitlab-secrets.json :

  "letsencrypt": {
    "auto_enabled": false
  },

Now I see a warning along with the error:

Ignoring section letsencrypt in /etc/gitlab/gitlab-secrets.json, does not exist in gitlab.rb

So I added the following to /etc/gitlab/gitlab.rb :

letsencrypt['auto_enabled'] = false

But now I get a different error when running gitlab-ctl reconfigure :

  *ERROR*: Encountered unsupported config key 'letsencrypt' in /etc/gitlab/gitlab.rb.

================================================================================
  ERROR: Encountered unsupported config key 'letsencrypt' in /etc/gitlab/gitlab.rb.
================================================================================

================================================================================
Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee/recipes/default.rb
================================================================================

NoMethodError
-------------
undefined method `[]=' for nil:NilClass

Cookbook Trace:
---------------
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:100:in `block in from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/config_mash.rb:28:in `auto_vivify'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:100:in `from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab.rb:125:in `block in from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/config_mash.rb:28:in `auto_vivify'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab.rb:125:in `from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/config.rb:23:in `from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:26:in `from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee/recipes/default.rb:20:in `from_file'

Relevant File Content:
----------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:

 93:    def ee_attribute(name, **config)
 94:      config = { ee: true }.merge(config)
 95:      attribute(name, **config)
 96:    end
 97:  
 98:    def from_file(_file_path)
 99:      # Allow auto mash creation during from_file call
100>>     Gitlab::ConfigMash.auto_vivify { super }
101:    end
102:  
103:    def method_missing(method_name, *arguments) # rubocop:disable Style/MethodMissing
104:      # Give better message for NilClass errors
105:      # If there are no arguments passed, this is a 'GET' call, and if
106:      # there is no matching key in the configuration, then it has not been set (not even to nil)
107:      # and we will output a nicer error above the exception
108:      if arguments.length.zero? && !configuration.key?(method_name)
109:        breaktxt = '=' * 80

Additionally, I get the following errors if I try to run gitlab-rake cache:clear :

/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/elastic.rake:61: warning: already initialized constant INDEXABLE_CLASSES
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/gitlab/elastic.rake:61: warning: previous definition of INDEXABLE_CLASSES was here
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:31: warning: already initialized constant Gitlab::VERSION
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/2_app.rb:6: warning: previous definition of VERSION was here
WARNING: Unable to check/update clone_url_prefix for Geo: undefined method `connected?' for Gitlab::Geo:Module
Did you mean?  concern
rake aborted!
NameError: uninitialized constant API::Entities::ProjectPushRule
/opt/gitlab/embedded/service/gitlab-rails/lib/api/project_push_rule.rb:28:in `block (2 levels) in <class:ProjectPushRule>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/project_push_rule.rb:27:in `block in <class:ProjectPushRule>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/project_push_rule.rb:10:in `<class:ProjectPushRule>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/project_push_rule.rb:2:in `<module:API>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/project_push_rule.rb:1:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/v3/project_git_hook.rb:23:in `block (2 levels) in <class:ProjectGitHook>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/v3/project_git_hook.rb:22:in `block in <class:ProjectGitHook>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/v3/project_git_hook.rb:13:in `<class:ProjectGitHook>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/v3/project_git_hook.rb:3:in `<module:V3>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/v3/project_git_hook.rb:2:in `<module:API>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/v3/project_git_hook.rb:1:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/api.rb:66:in `block in <class:API>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/api.rb:25:in `<class:API>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/api.rb:2:in `<module:API>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/api.rb:1:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/rack_attack.rb:12:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:11:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => cache:clear => cache:clear:redis => environment
(See full trace by running task with --trace)

Answer 1

In my case, I was able to successfully upgrade to 10.8.7 by selecting an arbitrary intermediate version (in my case, 10.5.8 ). Not sure why directly upgrading as per the upgrade path did not work. I will continue my journey to GitLab 14 now!