Python：字典中numpy数组的有效串联

Question

(Using Python 2.7)

Note: This is a very similar question to Concatenating dictionaries of numpy arrays of different lengths (avoiding manual loops if possible) . However, I have a slightly different use-case:

To summarize the problem:

I have a numpy array of dictionaries that are structurally all the same (meaning they all have the same keys) containing numpy arrays of variable length (including empty). Nested data structures FTW!

What I want is one "merged" dictionary where for every key, all the numpy arrays are concatenated.

For example:

source = [{"a":numpy.array([1,2,3]),"b":numpy.array(['a','b','c'])},{"a":numpy.array([4,5]),"b":numpy.array(['d','e','f','g','h'])}]
# Perform magic here into result
result = {"a":numpy.array([1,2,3,4,5]),"b":numpy.array(['a','b','c','d','e','f','g','h'])}

I could just iterate through every dictionary and use numpy.append() , but I figured that since this is Python and Numpy there should be a more elegant solution using some kind of slicing?

Difference to the similar question linked above:

It seems that in the linked question there are only a few dictionaries, and the keys are semantically linked. For example, in dataset 0 the key "a" is 1, the key "b" is 'a'", and the key "c" is NaN, and so on. However, in my case, there is no connection between "a", "b" and so on. In fact, most of the Panda Table would consist of datasets with NaN. The concatenated "a" might have ten thousand entries, while the concatenated "b" could be empty in an extreme case. Also, I might have hundreds of dictionaries that I want to "concatenate". Finally, the linked question has keys that are present in one dictionary but not in the other. This is impossible in my case.

I'm wondering, given these circumstances, if the Panda Dataframe approach is still the best way to go, considering I'd need to create a dataframe for every dictionary, and the end result would be a dataframe with LOTS of NaNs.

Thanks!

Answer 1

If it is possible to convert the NumPy arrays to Python lists then you can use collections.Counter :

In [15]: from collections import Counter                                                         

In [16]: source_ = [{"a":[1,2,3],"b":['a','b','c']}, 
                    {"a": [4,5], "b":['d','e','f','g','h']}]

In [17]: sum((Counter(x) for x in source_), Counter())                                           
Out[17]: Counter({'b': ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h'],
                  'a': [1, 2, 3, 4, 5]})

Answer 2

I would iterate over the keys instead of iterating over the dictionaries. This will allow you to use numpy.concatenate , which is more appropriate for this case than numpy.append , and I think it's easier to read. I would be surprised if there was a numpy built-in for this, and even if there was, I don't think it would do much for readability or performance.

source = [{"a":numpy.array([1,2,3]),"b":numpy.array(['a','b','c'])}, 
          {"a":numpy.array([4,5]),"b":numpy.array(['d','e','f','g','h'])}]
result = {}
for key in source[0]:
    result[key] = numpy.concatenate([d[key] for d in source])