Pandas read_json(orient="table") returns NaN if the column is an integer

Question

I have an issue when converting a DataFrame to json and back whilst using orient="table".

If a list is loaded into a DF and then exported as json using to_json(orient="table") the schema output includes the column name as an int which appears to be the cause of the issue.

Example

import pandas as pd

# List
arr = ["123"]

# Create the dataframe
dataframe = pd.DataFrame(arr)
print(dataframe)

# Get the table as a schema
dataframe_table_schema = dataframe.to_json(orient='table')
print(dataframe_table_schema)

# Load the DataFrame from the json object
dataframe = pd.read_json(dataframe_table_schema, orient='table')
print(dataframe)

Output

     0
0  123

{"schema": {"fields":[{"name":"index","type":"integer"},{"name":0,"type":"string"}],"primaryKey":["index"],"pandas_version":"0.20.0"}, "data": [{"index":0,"0":"123"}]}

     0
0  NaN

To work around the issue we can loop over the fields dataframe_table_schema.schema.fields and check if the field name is an integer, if it is cast it to a string and then dump the object to a json string.

import pandas as pd

# List
arr = ["123"]

# Create the dataframe
dataframe = pd.DataFrame(arr)
print(dataframe)

# Get the table as a schema
dataframe_table_schema = dataframe.to_json(orient='table')

# Load the schema into a dict
dataframe_table_schema_modified = json.loads(dataframe_table_schema)
# Loop over the fields
for field in dataframe_table_schema_modified.get("schema").get("fields"):
    # Get the column name
    column_name = field.get("name", "")
    if isinstance(column_name, int):
        # Cast the field name to a string
        field["name"] = str(column_name)
#  Dump the object to a string
dataframe_table_schema_modified = json.dumps(dataframe_table_schema_modified)
print(dataframe_table_schema_modified)

dataframe = pd.read_json(dataframe_table_schema_modified, orient='table')
print(dataframe)

Please could someone confirm if this is a bug or if there is a way to handle this correctly.

pd.show_versions() INSTALLED VERSIONS

commit: None python: 3.8.0.final.0 python-bits: 64 OS: Linux OS-release: 5.8.0-1041-aws machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: C.UTF-8 LOCALE: en_US.UTF-8

pandas: 0.25.2 numpy: 1.17.3 pytz: 2019.3 dateutil: 2.8.0 pip: 19.3.1 setuptools: 41.6.0 Cython: 0.29.13 pytest: 5.2.2 hypothesis: None sphinx: 2.2.1 blosc: None feather: None xlsxwriter: 1.2.2 lxml.etree: 4.4.1 html5lib: 1.0.1 pymysql: None psycopg2: 2.8.4 (dt dec pq3 ext lo64) jinja2: 2.10.3 IPython: 7.8.0 pandas_datareader: None bs4: 4.8.1 bottleneck: 1.2.1 fastparquet: None gcsfs: None lxml.etree: 4.4.1 matplotlib: 3.1.1 numexpr: 2.7.0 odfpy: None openpyxl: 3.0.0 pandas_gbq: None pyarrow: None pytables: None s3fs: None scipy: 1.3.1 sqlalchemy: 1.3.10 tables: None xarray: None xlrd: 1.2.0 xlwt: 1.3.0 xlsxwriter: 1.2.2

Answer 1

There is a mismatch between your field and data.

Notice in "fields", the column name is 0 ie an integer:

"fields":[{"name":"index","type":"integer"},{"name":0,"type":"string"}]
                                                   #^integer

But in "data", the column name is "0" ie a string:

"data": [{"index":0,"0":"123"}]
                    #^string

You can correct this by specifying the column names while constructing your DataFrame:

df = pd.DataFrame(["123"], columns=["A"])
js = df.to_json(orient="table")
df = pd.read_json(js, orient="table")

>>> df
     A
0  123