使用 pandas_schema 进行数据验证

Question

I would like to read a CSV data file using Python pandas library and create visualizations.
First, I decided to validate the data.
I want to use the pandas_schema module to validate the data at each column.
The initial data file has 26 columns.
My code:

from pandas_schema import Column, Schema 
from pandas_schema.validation import LeadingWhitespaceValidation, TrailingWhitespaceValidation, CanConvertValidation, MatchesPatternValidation, InRangeValidation, InListValidation



schema = Schema ([
    Column('Symboling', [InRangeValidation(-3,3)] ) ,  #integer from -3 to 3 
    Column('Normalized Loss', [InRangeValidation(65,256)] )  , # integer from 65 to 256
    Column('Make',[LeadingWhitespaceValidation(), TrailingWhitespaceValidation()] )  , # text 
    Column('Fuel Type', [InListValidation(['diesel', 'gas'])]), # diesel, gas
    Column('Aspiration'), # text 
    Column('Num of Doors' , [InListValidation(['two', 'four'])]), # text (two, four)
    Column('Body Style' , [InListValidation(['hardtop', 'wagon','sedan','hatchback', 'convertible'])] ), # text: hardtop, wagon, sedan, hatchback, convertible 
    Column('Drive Wheels' , [InListValidation(['4wd', 'fwd' , 'rwd'])]), # text: 4wd, fwd, rwd 
    Column('Engine Location' , [InListValidation(['front', 'rear'])]), # text: front, rear
    Column('Wheel Base' , [InRangeValidation([86.6,120.9])] ) ,  # decimal from 86.6 to 120.9 
    Column('Length' , [InRangeValidation(65,256)] )  ,  # decimal from 141.1 to 208.1
    Column('Width' , [InRangeValidation(60.3,72.3)] ) ,  # decimal from 60.3 to 72.3 
    Column('Height' , [InRangeValidation(47.8,59.8)] ) ,   # decimal from 47.8 to 59.8
    Column('Curb Weight' , [InRangeValidation(1488,4066)] ) ,   # integer from 1488 to 4066
    Column('Engine Type'),[InListValidation(['ohc', 'dohcv', 'l', 'ohc', 'ohcf', 'ohcv', 'rotor'])] , # text 
    Column('Num of Cylinders' , [InListValidation(['two','four','three','five','six','eight','twelve'])]) , # text: eight, five, four, six, three, twelve, two 
    Column('Engine Size' , [InRangeValidation(61,326)]) ,  # integer from 61 to 326 
    Column('Fuel System' , [InListValidation(['1bbl', '2bbl', '4bbl', 'idi','mfi','mpfi','spdi','spfi'])]), #string: 1bbl, 2bbl, 4bbl, idi,mfi,mpfi,spdi,spfi 
    Column('Bore' , [InRangeValidation(2.54,3.94)] ) , # decimal from 2.54 to 3.94 
    Column('Stroke', [InRangeValidation(2.07,4.17)] ) , #decimal from 2.07 to 4.17 
    Column('Compression Ratio' , [InRangeValidation(7,23)] ), #  integer: from 7 to 23 
    Column('Horsepower' , [InRangeValidation(48,288)] ),  # integer:from 48 to 288 
    Column('Peak rmp'), [InRangeValidation(4150,6600)]  , # integer: from 4150 to 6600 
    Column('City mpg'), [InRangeValidation(13,49)]  , #integer: from 13 to 49 
    Column('Highway mpg'), [InRangeValidation(16,54)] ,  # integer: 16 to 54 
    Column('Price'), [InRangeValidation(5118,45400)]  # integer from 5118 to 45400 
])

test_file = pd.read_csv(('E:\_Python_Projects_Data\Data_Visualization\Autos_Data_Set\Autos_Import_1985.csv')) 
errors = schema.validate(test_file) 
for error in errors: 
    print(error)

After I ran the code, I have the notification:

 The invalid number of columns. The schema specifies 31, but the data frame has 26

I don't actually understand how this happened: at the schema, I have 26 columns; and the data file has 26 columns. Any suggestions?
Thank you.

Answer 1

For the record: you have a few typos in your Schema definition, closing parentheses in Column definitions too early. This creates a list with 31 elements, which are interpreted as columns.

The correct definition should be:

schema = Schema([
    Column('Symboling', [InRangeValidation(-3,3)]),  #integer from -3 to 3 
    Column('Normalized Loss', [InRangeValidation(65,256)]), # integer from 65 to 256
    Column('Make', [LeadingWhitespaceValidation(), TrailingWhitespaceValidation()] ), # text 
    Column('Fuel Type', [InListValidation(['diesel', 'gas'])]), # diesel, gas
    Column('Aspiration'), # text 
    Column('Num of Doors', [InListValidation(['two', 'four'])]), # text (two, four)
    Column('Body Style', [InListValidation(['hardtop', 'wagon','sedan','hatchback', 'convertible'])]), # text: hardtop, wagon, sedan, hatchback, convertible 
    Column('Drive Wheels', [InListValidation(['4wd', 'fwd', 'rwd'])]), # text: 4wd, fwd, rwd 
    Column('Engine Location', [InListValidation(['front', 'rear'])]), # text: front, rear
    Column('Wheel Base', [InRangeValidation([86.6,120.9])]),  # decimal from 86.6 to 120.9 
    Column('Length', [InRangeValidation(65,256)]),  # decimal from 141.1 to 208.1
    Column('Width', [InRangeValidation(60.3,72.3)]),  # decimal from 60.3 to 72.3 
    Column('Height', [InRangeValidation(47.8,59.8)]),   # decimal from 47.8 to 59.8
    Column('Curb Weight', [InRangeValidation(1488,4066)]),   # integer from 1488 to 4066
    Column('Engine Type', [InListValidation(['ohc', 'dohcv', 'l', 'ohc', 'ohcf', 'ohcv', 'rotor'])]), # text 
    Column('Num of Cylinders', [InListValidation(['two','four','three','five','six','eight','twelve'])]), # text: eight, five, four, six, three, twelve, two 
    Column('Engine Size', [InRangeValidation(61,326)]),  # integer from 61 to 326 
    Column('Fuel System', [InListValidation(['1bbl', '2bbl', '4bbl', 'idi','mfi','mpfi','spdi','spfi'])]), #string: 1bbl, 2bbl, 4bbl, idi,mfi,mpfi,spdi,spfi 
    Column('Bore', [InRangeValidation(2.54,3.94)]), # decimal from 2.54 to 3.94 
    Column('Stroke', [InRangeValidation(2.07,4.17)]), #decimal from 2.07 to 4.17 
    Column('Compression Ratio', [InRangeValidation(7,23)]), #  integer: from 7 to 23 
    Column('Horsepower', [InRangeValidation(48,288)]),  # integer:from 48 to 288 
    Column('Peak rmp', [InRangeValidation(4150,6600)]), # integer: from 4150 to 6600 
    Column('City mpg', [InRangeValidation(13,49)]), #integer: from 13 to 49 
    Column('Highway mpg', [InRangeValidation(16,54)]),  # integer: 16 to 54 
    Column('Price', [InRangeValidation(5118,45400)]),  # integer from 5118 to 45400 
])

Answer 2

Dataframe columns must match the number of columns in the defined validation schema. Alternate way is define a new dataframe with list of columns that you want to compare and use that for validation. (Not really sure if this is most efficient way, but it solves the purpose)