如何从现有的红移集群参数组工作负载管理设置复制？

Question

Is there a way to copy Redshift workload management settings from existing one?

I have a complex WLM configuration in my redshift parameter group, which I want to copy to another configuration. Doing it manually is error prone. Is there a way to copy it automatically?

Here is my WLM config JSON, which I copied from Redshift console's WLM config page:

[
  {
    "query_group": [
      "defaulton",
      "lab4on"
    ],
    "query_group_wild_card": 0,
    "user_group": [],
    "user_group_wild_card": 0,
    "concurrency_scaling": "off",
    "priority": "normal",
    "queue_type": "auto",
    "auto_wlm": true
  },
  {
    "query_group": [
      "lab1",
      "lab2",
      "lab3normal",
      "lab4off",
      "lab5",
      "lab6reads",
      "defaultoff"
    ],
    "query_group_wild_card": 0,
    "user_group": [],
    "user_group_wild_card": 0,
    "concurrency_scaling": "off",
    "priority": "normal",
    "queue_type": "auto",
    "auto_wlm": true
  },
  {
    "query_group": [
      "lab3highest"
    ],
    "query_group_wild_card": 0,
    "user_group": [],
    "user_group_wild_card": 0,
    "concurrency_scaling": "off",
    "priority": "highest",
    "queue_type": "auto",
    "auto_wlm": true
  },
  {
    "query_group": [
      "lab3lowest"
    ],
    "query_group_wild_card": 0,
    "user_group": [],
    "user_group_wild_card": 0,
    "concurrency_scaling": "off",
    "priority": "lowest",
    "queue_type": "auto",
    "auto_wlm": true
  },
  {
    "query_group": [
      "lab6writes"
    ],
    "query_group_wild_card": 0,
    "user_group": [],
    "user_group_wild_card": 0,
    "concurrency_scaling": "off",
    "priority": "highest",
    "queue_type": "auto",
    "auto_wlm": true
  },
  {
    "query_group": [],
    "query_group_wild_card": 0,
    "user_group": [],
    "user_group_wild_card": 0,
    "priority": "normal",
    "queue_type": "auto",
    "auto_wlm": true
  },
  {
    "short_query_queue": false
  }
]

I could copy it to a new parameter group following below steps, but would like to simplify it if possible.

1. copy the json from existing param group as above to a nodepad editor
2. remove all nextline and space characters from this json
3. replace all doublequotes( " ) with backslash doublequotes ( \\" )
4. create below json file in my local eg modify_pg.json

[
    {
        "ParameterName": "wlm_json_configuration",
        "ParameterValue": "<input_formatted_json_here>"
    }
]

5. replace <input_formatted_json_here> above with the formatted text in my editor (created in step# 3 above)
6. create a new parameter group in Redshift, eg myclusterparametergroup
7. run below CLI command to modify WLM of this parameter group with the json file

aws redshift modify-cluster-parameter-group --parameter-group-name myclusterparametergroup --parameters file://modify_pg.json

Below is my final modify_pg.json file, which I am able to use in my new parameter group. But is there a way to simplify this?

[
    {
        "ParameterName": "wlm_json_configuration",
        "ParameterValue": "[{\"query_group\":[\"defaulton\",\"lab4on\"],\"query_group_wild_card\":0,\"user_group\":[],\"user_group_wild_card\":0,\"concurrency_scaling\":\"off\",\"priority\":\"normal\",\"queue_type\":\"auto\",\"auto_wlm\":true},{\"query_group\":[\"lab1\",\"lab2\",\"lab3normal\",\"lab4off\",\"lab5\",\"lab6reads\",\"defaultoff\"],\"query_group_wild_card\":0,\"user_group\":[],\"user_group_wild_card\":0,\"concurrency_scaling\":\"off\",\"priority\":\"normal\",\"queue_type\":\"auto\",\"auto_wlm\":true},{\"query_group\":[\"lab3highest\"],\"query_group_wild_card\":0,\"user_group\":[],\"user_group_wild_card\":0,\"concurrency_scaling\":\"off\",\"priority\":\"highest\",\"queue_type\":\"auto\",\"auto_wlm\":true},{\"query_group\":[\"lab3lowest\"],\"query_group_wild_card\":0,\"user_group\":[],\"user_group_wild_card\":0,\"concurrency_scaling\":\"off\",\"priority\":\"lowest\",\"queue_type\":\"auto\",\"auto_wlm\":true},{\"query_group\":[\"lab6writes\"],\"query_group_wild_card\":0,\"user_group\":[],\"user_group_wild_card\":0,\"concurrency_scaling\":\"off\",\"priority\":\"highest\",\"queue_type\":\"auto\",\"auto_wlm\":true},{\"query_group\":[],\"query_group_wild_card\":0,\"user_group\":[],\"user_group_wild_card\":0,\"priority\":\"normal\",\"queue_type\":\"auto\",\"auto_wlm\":true},{\"short_query_queue\":false}]"
    }
]

Answer 1

I've done this several times for clients and you are nearly there. The same way you are modifying the PG using the AWS CLI, you can extract the JSON from the existing PG using the AWS CLI.

aws redshift describe-cluster-parameters ...

You will get back a JSON that has the parameter value for a specified PG. It is then just a matter of modifying the returned values to the string format you need. This requires some programming or scripting - I'll go with Linux scripting as this is a popular option. Command line tools you will likely want:

• sed
• tr
• jq (tag line for this tool is "sed for json")

The process you will follow will be something like

1. extract parameters using aws cli
2. create shell variable with the parameters you want and the values you want each to have. jq is great for this
3. create the new PG if needed
4. modify the JSON shell variable to remove \\n and spaces - replacement_text=$(echo $wlm_configuration | sed -e 's/"/\\\\"/g' | tr -d '\\n ') create top JSON for PG as a shell variable (same as your modify_pg.json)
5. run aws cli to modify the new PG - combine these 2 JSONs right on the command line by using option: --parameters $(echo $modify_pg | tr -d '\\n ' | sed -e "s/<<WLM_CONFIGURATION>>/$replacement_text/g")

Now you can do this in python using boto3 or in java or ... This is just one path.