Pandas concat 返回 NaN 值

Question

I am trying to concat two dataframes in pandas. One is JSON file and another is Excel file. I need to add values from Excel file to JSON file, so the output is updated JSON file.Note: JSON file has about 20000 lines and more than 10 languages.
Example of an Excel file: Please note that number of languages is larger. Also number of translated words may differ from one to more. Key of every translation in JSON must be in English.

-------------------------
| en     | de     |  ru |
--------+-------+--------
|Flower  |Blume   | FF|
|Chair   |Stuhl   | BB|
|Snake   |Schlange| CC  |
|Monkey  |Affe    |  DD  |
--------------------------

Here is the example of JSON input file (old JSON file which should be updated with new values from an Excel file given above):

{
    "en": {
        "Ball": "Ball",
        "Snow": "Snow"
    },
    "de": {
        "Ball": "Ball",
        "Snow": "Schnee"
    },
    "ru": {
        "Ball": "AA",
    }
}

You can see that there is no "Snow" under "ru", and it is okay. But if I concat two DF, the output looks like this

{
    "en": {
        "Ball": "Ball",
        "Snow": "Snow"
    },
    "de": {
        "Ball": "Ball",
        "Snow": "Schnee"
    },
    "ru": {
        "Ball": "AA",
        "Snow": NaN
    }
}

Here is my code

with open(json_filePath, encoding='utf-8') as f:
    old_json = json.load(f)
new_data = pd.read_excel(excel_filePath)
old_data = pd.DataFrame(old_json)
new_json = pd.concat([old_data, new_data.set_index('en', drop=False)]).to_dict()
with open(json_filePath, 'w', encoding='utf-8') as f:
    json.dump(new_json, f, ensure_ascii=False, indent=2, separators=(',', ':'))

This is the output I got:

    {
    "en": {
        "Ball": "Ball",
        "Snow": "Snow",
        "Flower": "Flower",
        "Chair": "Chair",
        "Snake": "Snake",
        "Monkey": "Monkey"
    },
    "de": {
        "Ball": "Ball",
        "Snow": "Schnee",
        "Flower": "Blume",
        "Chair": "Stuhl",
        "Snake": "Schlange",
        "Monkey": "Affe"
    },
    "ru": {
        "Ball": "AA",
        "Snow": NaN,
        "Flower": "FF",
        "Chair": "BB",
        "Snake": "CC",
        "Monkey": "DD"    
    }
}

And below is desired output:

 {
    "en": {
        "Ball": "Ball",
        "Snow": "Snow",
        "Flower": "Flower",
        "Chair": "Chair",
        "Snake": "Snake",
        "Monkey": "Monkey"
    },
    "de": {
        "Ball": "Ball",
        "Snow": "Schnee",
        "Flower": "Blume",
        "Chair": "Stuhl",
        "Snake": "Schlange",
        "Monkey": "Affe"
    },
    "ru": {
        "Ball": "AA",
        "Flower": "FF",
        "Chair": "BB",
        "Snake": "CC",
        "Monkey": "DD"    
    }
}

So the only difference is that pandas added NaN value for "Snow" under "ru". And this is happening every time, if some key is missing from one language in original JSON file. My point is that I do not want to change existing values in original JSON file, just to add new values from Excel. Key of new data (from Excel) is set to be English value. I tried to iterate through the JSON and drop the NaN values, but it drop ALL values with that key. For example if "Snow" is NaN under "ru", and has value under "en", it deletes EVERY VALUE for "Snow". Reset index was not very helpful Tried inner, outer join without success. Am a newbie, searched for some solutions, but still no success. Any idea?

Answer 1

The problem is that pandas only works with dataitems of the same type (each object must have identical list of properties) You can filter NaNs out in output json right before exporting it in a file:

...
new_json = pd.concat([old_data, new_data.set_index('en', drop=False)]).to_dict()
filtered_json = {item: {inner_item: inner_value for inner_item, inner_value in value.items()
                if not pd.isna(inner_value)}
                for item, value in new_json.items()}
with open(json_filePath, 'w', encoding='utf-8') as f:
    json.dump(filtered_json, f, ensure_ascii=False, indent=2, separators=(',', ':'))