List all possible combinations from a complex dictionary of lists of lists?

Question

I have a dictionary which contains strings, None , lists, and lists of lists as shown in the example below.

The desired result is a list of dictionaries. The number of dictionaries equal to the number of total permutations possible.

The requirements are that each of the dictionaries may have None or a list of values with the exception that varVal has a list of lists of length equal to the length of varName (ie, len(d['varVal'] = len(d['varName'] ).

I think the None issue is already taken care of and keeping the varString as a list prevents breaking the string into characters.

I've scoured many an overflow post with this one the most helpful so far though I've ran out of ideas. Any help would be greatly appreciated. Thank you.

Example

This example does not generate the desire result :(

import itertools as it

d={}
d['varString']=['ExampleText']
d['n1']=[1,100]
d['n2']=None
d['varName']=['varA','varB']
d['varVal']=[[10],[1,0]]

df = {k:v for k,v in d.items() if v is not None}
keys, values = zip(*df.items())
res = [dict(zip(keys, v)) for v in it.product(*values)]

Desired Result

Extra space added between dictionaries to help the visual presentation.

res = [
    {'n1': 1,
     'varName': ['varA', 'varB'],
     'varString': 'ExampleText',
     'varVal': [[10], [0]]},

    {'n1': 100,
     'varName': ['varA', 'varB'],
     'varString': 'ExampleText',
     'varVal': [[10], [0]]},

    {'n1': 1,
     'varName': ['varA', 'varB'],
     'varString': 'ExampleText',
     'varVal': [[10], [1]]},

    {'n1': 100,
     'varName': ['varA', 'varB'],
     'varString': 'ExampleText',
     'varVal': [[10], [1]]}
]

Answer 1

PeterE's comment got me to think about the problem a little different which led to this answer.

This answer handles the entries that are causing the issues separately and then does some post-processing to get return the required format. Essentially creating and additional step which masks the final required dictionary format which was causing the problem.

If there is a way for this to be somehow handled in a more direct method as specified in the original question I would love to know. Thanks for those who commented.

Solution

import itertools as it

def standKeys():
    # Standard input parameters for dymola.simulateExtendedModel
    keys = ['n1',
            'n2',
            'varString',
            'varName',
            'varVal']
    return keys


def initSettings():
    # Initialize the settings parameters
    simSettings = {}
    for key in standKeys():
            simSettings[key]=None

    return simSettings

def genExperimentsRaw(simSettings):
    # Remove None and generate all experiment permutations
    simSettingsfiltered = {k:v for k,v in simSettings.items() if v is not None}
    keys, values = zip(*simSettingsfiltered.items())
    experimentsRaw = [dict(zip(keys, v)) for v in it.product(*values)]

    return experimentsRaw

def genExperiments(experimentsRaw):
    # Filter experiments to generate key/value pairs for 'varName' and 'varVal'
    allKeys = standKeys()
    experiments = []
    for i, value in enumerate(experimentsRaw):
        initialNames = []
        initialValues = []
        for key, val in value.items(): 
            if key not in allKeys:
                initialNames.append(key)
                initialValues.append(val)
                value.pop(key)
        value['varName']=initialNames
        value['varVal']=initialValues

        experiments.append(initSettings())
        experiments[i].update(value)

    return experiments

if __name__ == "__main__":
    d={}
    d['n1']=[1,100]
    d['n2']=None
    d['varString']=['ExampleText']
    # Original
    # d['varName']=['varA','varB'] # Remove this
    # d['varVal']=[[10],[1,0]]

    # Separate out and treat like other dictionary keys
    d['varA']=[10]
    d['varB']=[1,0]

# Generate all experiment permutations
experimentsRaw = genExperimentsRaw(d)

# Filter experiments to generate key/value pairs for 'varName' and 'varVal'
experiments = genExperiments(experimentsRaw)

Result

[{'n1': 1,
  'n2': None,
  'varName': ['varA', 'varB'],
  'varString': 'ExampleText',
  'varVal': [10, 1]},
 {'n1': 1,
  'n2': None,
  'varName': ['varA', 'varB'],
  'varString': 'ExampleText',
  'varVal': [10, 0]},
 {'n1': 100,
  'n2': None,
  'varName': ['varA', 'varB'],
  'varString': 'ExampleText',
  'varVal': [10, 1]},
 {'n1': 100,
  'n2': None,
  'varName': ['varA', 'varB'],
  'varString': 'ExampleText',
  'varVal': [10, 0]}]