I am making a simple reproducible example to understand how training and testing works:
Example
I want to predict Ages
based on Location
of origin:
import pandas as pd
# create a simple dataset of people
data = {'Name': ["John", "Anna", "Peter", "Linda","John","John","John"],
'Location' : ["Paris","Paris","Paris","Paris", "New York", "Berlin", "London"],
'Age' : [24, 23, 21, 24,36,34,36]
}
df = pd.DataFrame(data)
At this part below, it has a problem with the city names, therefore I decided to use dummy variables but the line with the get_dummies
is not correct. I think it needs to turn both Name
and Location
strings to dummy variables and that's what I tried but what is the proper way?
from sklearn.model_selection import train_test_split
X = df.drop('Age', axis=1)
y = df['Age']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.20)
from sklearn.svm import SVC
svclassifier = SVC(kernel='linear')
X_train = pd.get_dummies(df.columns) #<---- here is the issue probably
svclassifier.fit(X_train, y_train)
y_pred = svclassifier.predict(X_test)
svclassifier.fit(X_train, y_train)
takes an array as input, but you feed a pandas DataFrame. Try using sklearn.preprocessing.LabelEncoder
istead of pd.get_dummies
.
Edit: Example with LabelEncoder
and OneHotEncoder
:
# create a simple dataset of people
data = {'Name': ["John", "Anna", "Peter", "Linda","John","John","John"],
'Location' : ["Paris","Paris","Paris","Paris", "New York", "Berlin", "London"],
'Age' : [24, 23, 21, 24,36,34,36]
}
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
from sklearn.svm import SVC
import numpy as np
X = data['Location']
y = data['Age']
# Label
print("Label Encoded")
le = LabelEncoder()
le.fit(X)
X_enc = le.transform(X)
X_train, X_test, y_train, y_test = train_test_split(X_enc, y, test_size = 0.20, random_state=42)
svclassifier = SVC(kernel='linear')
svclassifier.fit(np.reshape(X_train,(X_train.shape[0],1)), y_train)
y_pred = svclassifier.predict(np.reshape(X_test, (X_test.shape[0],1)))
print(f"y_pred: {y_pred}, y_test: {y_test}")
# OneHot
print("OneHot Encoded")
ohe = OneHotEncoder()
ohe.fit(np.reshape(X,(len(X),1)))
X_oh = ohe.transform(np.reshape(X,(len(X),1)))
X_train, X_test, y_train, y_test = train_test_split(X_oh, y, test_size = 0.20, random_state=42)
svclassifier = SVC(kernel='linear')
svclassifier.fit(X_train, y_train)
y_pred = svclassifier.predict(X_test)
print(f"y_pred: {y_pred}, y_test: {y_test}")
Gives:
Label Encoded
y_pred: [24 24], y_test: [24, 23]
OneHot Encoded
y_pred: [24 24], y_test: [24, 23]
Not too bad.
You did not define your features (X) and target (y). Your X is where your model learn to predict your target y. Since your feature is Name and Location which is categorical then you need to use auto encoder like get_dummies.
from sklearn.model_selection import train_test_split
#features
X = pd.get_dummies(df[['Name','Location']])
#Target
y = df['Age']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.20)
from sklearn.svm import SVC
svclassifier = SVC(kernel='linear')
svclassifier.fit(X_train, y_train)
y_pred = svclassifier.predict(X_test)
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.