Puppet 7. How to let several profiles subscribe to some service of which the single config file is generated by Puppet

Question

We faced the issue with Puppet 7 on Ubuntu 20.04

Attempt to override an already evaluated resource, defined at (file: .../modules/profiles/manifests/php/fpm_check.pp, line: 2), with new values (file: .../modules/profiles/manifests/php/fpm_pool.pp, line: 122) and

Attempt to override an already evaluated resource, defined at (file: .../modules/profiles/manifests/datadog_agent/phpfpm.pp, line: 3), with new values (file: .../modules/profiles/manifests/php/fpm_pool.pp, line: 109)

So, the question is in the topic:

How to let several profiles subscribe to some service of which the single config file is generated by Puppet.

____Additional information:

modules/profiles/manifests/php/fpm_check.pp

class profiles::php::fpm_check() {
  profiles::php::fpm_check_instance{ 'singleton': urls => [] }
}

modules/profiles/manifests/datadog_agent/phpfpm.pp

class profiles::datadog_agent::phpfpm() {
 profiles::datadog_agent::integration_phpfpm {
  'singleton': instances => []
 }
}

modules/profiles/manifests/php/fpm_pool.pp

define profiles::php::fpm_pool(
  $deployroot,
  $short_fqdn,
  $fpm_port,
  $monitor_port,
  $php_value = {},
  $user = 'www-data',
  $group = 'www-data',
  $env_vars = {},
  $use_redis_for_php_sessions = false,
  $dd_apm_disable_integrations = 'guzzle',
  $max_execution_time = undef,
  ){
  require profiles::php::fpm
  require profiles::apache

  $fpm_log_dir = $profiles::php::fpm::log_dir
  $fpm_pool_dir = $profiles::php::fpm::pool_dir
  $fpm_service_name = $profiles::php::fpm::service_name

  if $use_redis_for_php_sessions {
    $redis_key_prefix = "SESSION-${regsubst(upcase($name), '\s+', '-')}:"

    $redis_session_settings = {
      'session.save_handler' => 'redis',
      'session.save_path'    => "'unix:///var/run/twemproxy.sock?timeout=120&prefix=${redis_key_prefix}'",
    }

    $php_value_final = merge($php_value, $redis_session_settings)
  } else {
    $php_value_final = $php_value
  }

  $datadog_env_vars = {
    'DD_TRACE_APP_NAME'        => $title,
    'DD_INTEGRATIONS_DISABLED' => $dd_apm_disable_integrations,
  }

  if $max_execution_time == undef {
    $php_admin_value = {}
  } else {
    $php_admin_value = {
      'max_execution_time' => $max_execution_time,
    }
  }

  phpfpm::pool{ $title:
    listen                  => "127.0.0.1:${fpm_port}",
    pm_max_children         => 150,
    pm_start_servers        => 5,
    pm_min_spare_servers    => 5,
    pm_max_spare_servers    => 10,
    service_name            => $fpm_service_name,
    pool_dir                => $fpm_pool_dir,

    access_log              => "${fpm_log_dir}/\$pool.access.log",
    access_format           => '"%R - [%t] \"%{HTTP_HOST}e\" \"%m %r%Q%q\" %s %f %{mili}d %{kilo}M %C%% %{HTTP_X_REQUEST_ID}e"',
    pm_status_path          => '/fpmstatus/$pool/status',
    ping_path               => '/fpmstatus/$pool/ping',
    ping_response           => 'pong',
    slowlog                 => "${fpm_log_dir}/\$pool.slow.log",
    request_slowlog_timeout => 60,
    php_value               => $php_value_final,
    user                    => $user,
    group                   => $group,
    env                     => merge($env_vars, $datadog_env_vars),
    php_admin_value         => $php_admin_value,
  }

  apache::vhost { "${title}-hystrix":
    manage_docroot               => false,
    docroot                      => '/var/www',
    servername                   => "${title}-hystrix.${short_fqdn}",
    port                         => 90,
    use_port_for_filenames       => true,
    use_servername_for_filenames => true,
    proxy_pass_match             => [
      {
        'path' => '^/$',
        'url'  => "fcgi://localhost:${fpm_port}${deployroot}/www/hystrix/HystrixDashboard.php",
      },
    ],
  }

  include profiles::datadog_agent::phpfpm
  include profiles::datadog_agent::phpapm

  $monitor_path = "/fpmstatus/${title}"
  $monitor_hostname = "${title}-fpm-stats"

  apache::vhost { "${title}-fpm-stats":
    manage_docroot         => false,
    docroot                => '/var/www',
    servername             => $monitor_hostname,
    port                   => $monitor_port,
    use_port_for_filenames => true,
    proxy_pass_match       => [
      {
        'path' => "^${monitor_path}/status$",
        'url'  => "fcgi://localhost:${fpm_port}${monitor_path}/status",
      },
      {
        'path' => "^${monitor_path}/ping$",
        'url'  => "fcgi://localhost:${fpm_port}${monitor_path}/ping",
      },
    ],
  }

  Profiles::Datadog_agent::Integration_phpfpm <| |> {
    instances +> [
      {
        'http_host'  => $monitor_hostname,
        'status_url' => "http://localhost:${monitor_port}${monitor_path}/status",
        'ping_url'   => "http://localhost:${monitor_port}${monitor_path}/ping",
        'tags'       => ["pool:${title}"],
      },
    ],
  }

  include profiles::php::fpm_check

  Profiles::Php::Fpm_check_instance <| |> {
    urls +> [ "http://localhost:${monitor_port}${monitor_path}/ping" ],
  }


  logrotate::rule { "${title}-fpmpool-access":
    path          => "${fpm_log_dir}/${title}.access.log",
    copytruncate  => true,
    mail          => false,
    missingok     => true,
    compress      => true,
    ifempty       => false,
    delaycompress => true,
    rotate        => 12,
    rotate_every  => 'weekly',
    postrotate    => "/usr/lib/php/${fpm_service_name}-reopenlogs",
  }

  logrotate::rule { "${title}-fpmpool-slow":
    path          => "${fpm_log_dir}/${title}.slow.log",
    copytruncate  => true,
    mail          => false,
    missingok     => true,
    compress      => true,
    ifempty       => false,
    delaycompress => true,
    rotate        => 12,
    rotate_every  => 'weekly',
    postrotate    => "/usr/lib/php/${fpm_service_name}-reopenlogs",
  }
}

Answer 1

This issue revolves around your use of resources having defined (as opposed to native) types, and your overrides of the properties of instances of those types.

When Puppet is building a catalog, it needs to evaluate the Puppet code in the body of each declared instance of each defined type, and it must do that in light of that instance's parameters. Puppet tries to defer those evaluations as late as possible to allow resource property overrides to be processed, but there is not necessarily a well-defined total order, on account of evaluation of defined-type resources generally producing declarations of additional resources.

It may happen, therefore, that some time after Puppet has evaluated the body of a particular resource instance, it evaluates an override of that instance's properties. Puppet has no particularly good alternative in that case. It cannot re-evaluate the affected resource, so it must either go forward with a resource that was evaluated with different parameters than was intended, or bails. It bails rather than risk misconfiguring the target machine.

To make these situations unlikely, minimize your use of resource overrides, especially across module boundaries. In that pursuit, it often helps to

• use Hiera and your data hierarchy for flexible customization, instead of procedural manifest code;
• make your defined-type resources fine grained; and in conjunction with that
• use multiple resources that work together instead of large monolithic resources

Alternatively, the override issue applies only to defined types, so you could consider converting some of your defined types to custom (native) types.

With respect specifically to having contributions to the same config file from multiple sources, you might find the puppetlabs/concat module to be of interest to you. At minimum, it may give you a better understanding of what I mean about fine grained resources and resources working together.

---

Separately but related, from your class naming, I take you to be intending to implement the Roles & Profiles pattern. In that case, your profile has far too much logic in it. Profiles should aggregate component classes, and little more. In particular, it is rarely appropriate for profiles to declare resources themselves -- that's the role of component classes. In the same vein, profiles generally should not be in the business of overriding resource properties, which is where your particular trouble is arising.

Instead, give your component classes the class parameters they need to declare the right resource properties in the first place, so that no override is required. My personal practice and usual recommendation is that where class parameter values need to be customized, that is done via Hiera-based data binding. If you wish, however, you can drive it from your profile class by using resource-like class declarations there (though this means that no other class included in the same catalog should declare those same component classes).