根据样本数据输入重现数据

Question

我有一个样本数据，我需要重现更多行数（将输入行数），这些行数将与我的样本共享几乎相同的分布，并且列值的随机组合也包括 NULL。

样本数据

gender         marital status  occupation    ethnic background

Male           Single          Doctor        Caucasian    
Male           Divorced        NA            African American
NA             Widow           Teacher       NA
Female         Married         Doctor        Caucasian    
Male           Divorced        Engineer      African American
NA             Widow           Teacher       NA

所需数据

gender         marital status  occupation    ethnic background
Male           Divorced        NA            African American
Male           Single          Doctor        Caucasian    
Male           Divorced        NA            African American
NA             Widow           Teacher       NA
NA             Widow           Teacher       NA
Female         Married         Doctor        Caucasian
Female         Married         Doctor        Caucasian    
Male           Divorced        Engineer      African American
NA             widow           Teacher       NA
Male           Single          Doctor        Caucasian    
NA             Widow           Teacher       NA
Female         Married         Doctor        Caucasian    
Male           Divorced        NA            African American
NA             Widow           Teacher       NA
Male           Divorced        Engineer      African American
NA             Widow           Teacher       NA
Male           Single          Doctor        Caucasian    
Male           Divorced        Engineer      African American

Answer 1

这个解决方案的一个想法 - 只需要替换缺失值以避免在较旧的 Pandas 版本的groupby中将其删除，然后为Series列表的每一列应用代码并最后连接在一起：

注意：分布匹配取决于行数，因此如果可能，您可以多个原始数据长度 - 这里原始数据为6 ，新数据为6*4=24

#test distibution of original
print (df.fillna('missing').apply(lambda x: pd.value_counts(x, normalize=True)))
                    gender  marital status  occupation  ethnic background
African American       NaN             NaN         NaN           0.333333
Caucasian              NaN             NaN         NaN           0.333333
Divorced               NaN        0.333333         NaN                NaN
Doctor                 NaN             NaN    0.333333                NaN
Engineer               NaN             NaN    0.166667                NaN
Female            0.166667             NaN         NaN                NaN
Male              0.500000             NaN         NaN                NaN
Married                NaN        0.166667         NaN                NaN
Single                 NaN        0.166667         NaN                NaN
Teacher                NaN             NaN    0.333333                NaN
Widow                  NaN        0.333333         NaN                NaN
missing           0.333333             NaN    0.166667           0.333333

---

df = df.fillna('missing')
nrows = len(df)
total_sample_size = 24

out = []
for c in df.columns:
    f = lambda x: x.sample(int((x.count()/nrows)*total_sample_size), replace=True)
    out.append(df.groupby(c)[c].apply(f).sample(frac=1).reset_index(drop=True))

df1 = pd.concat(out, axis=1).replace('missing', np.nan)

---

print (df1)
    gender marital status occupation ethnic background
0      NaN         Single    Teacher  African American
1     Male       Divorced    Teacher  African American
2     Male          Widow        NaN               NaN
3     Male        Married   Engineer               NaN
4      NaN       Divorced    Teacher  African American
5      NaN       Divorced     Doctor               NaN
6      NaN       Divorced    Teacher         Caucasian
7     Male          Widow    Teacher         Caucasian
8     Male       Divorced     Doctor         Caucasian
9   Female          Widow    Teacher               NaN
10     NaN          Widow   Engineer         Caucasian
11  Female         Single    Teacher         Caucasian
12  Female          Widow   Engineer  African American
13    Male        Married     Doctor  African American
14     NaN         Single     Doctor  African American
15  Female        Married   Engineer         Caucasian
16    Male       Divorced        NaN         Caucasian
17    Male          Widow        NaN  African American
18    Male         Single     Doctor               NaN
19    Male          Widow     Doctor               NaN
20     NaN          Widow    Teacher               NaN
21    Male       Divorced        NaN  African American
22     NaN        Married     Doctor               NaN
23    Male       Divorced     Doctor         Caucasian

---

#test distibution of new
print (df1.fillna('missing').apply(lambda x: pd.value_counts(x, normalize=True)))
                    gender  marital status  occupation  ethnic background
African American       NaN             NaN         NaN           0.333333
Caucasian              NaN             NaN         NaN           0.333333
Divorced               NaN        0.333333         NaN                NaN
Doctor                 NaN             NaN    0.333333                NaN
Engineer               NaN             NaN    0.166667                NaN
Female            0.166667             NaN         NaN                NaN
Male              0.500000             NaN         NaN                NaN
Married                NaN        0.166667         NaN                NaN
Single                 NaN        0.166667         NaN                NaN
Teacher                NaN             NaN    0.333333                NaN
Widow                  NaN        0.333333         NaN                NaN
missing           0.333333             NaN    0.166667           0.333333

编辑：

如果解决方案应该通过获取 N 次采样的原始数据来简化：

N = 4
df = pd.concat([df] * N, ignore_index=True).sample(frac=1)
print (df)
    gender marital status occupation ethnic background
12    Male         Single     Doctor         Caucasian
14     NaN          Widow    Teacher               NaN
4     Male       Divorced   Engineer  African American
8      NaN          Widow    Teacher               NaN
16    Male       Divorced   Engineer  African American
1     Male       Divorced        NaN  African American
7     Male       Divorced        NaN  African American
5      NaN          Widow    Teacher               NaN
15  Female        Married     Doctor         Caucasian
23     NaN          Widow    Teacher               NaN
22    Male       Divorced   Engineer  African American
17     NaN          Widow    Teacher               NaN
18    Male         Single     Doctor         Caucasian
0     Male         Single     Doctor         Caucasian
9   Female        Married     Doctor         Caucasian
19    Male       Divorced        NaN  African American
21  Female        Married     Doctor         Caucasian
20     NaN          Widow    Teacher               NaN
10    Male       Divorced   Engineer  African American
3   Female        Married     Doctor         Caucasian
11     NaN          Widow    Teacher               NaN
13    Male       Divorced        NaN  African American
6     Male         Single     Doctor         Caucasian
2      NaN          Widow    Teacher               NaN