来自 Pandas Dataframe 的 Python 对象列表

Question

What I want to do is take a data frame and turn each data frame row into a Python object namely RawData class presented. Dataframe contains 10^5 - 10^6 rows.

# Each row represents one RawData object
class RawData():
    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
    label_name = models.CharField(max_length=512, null=True
    currency = models.TextField(blank=True)
    content_id = models.TextField(blank=True)

@classmethod
def create_by_itertuples(cls, item):
    # item is namedtuple. converted to dict
    row = item._asdict()
    return (cls, row['Labels'], row['Currency'], row['Content_Id'])

@classmethod
def create_by_iterrows(cls, row):
    return (cls, row['Labels'], row['Currency'], row['Content_Id'])

@classmethod
def create_by_vectorization(cls, Labels, Currency, Content_Id):
    // How to proceed? 

I have tried iterrows and itertuples.

# sample dataframe
# initialize list of lists
data = [['T-Series', 'BDT', 'UX25437'], 
        ['Dragons Den', 'EUR', 'UF5432'], 
        ['A-Train', 'USD', 'GH5342']]
  
# Create the pandas DataFrame
df = pd.DataFrame(data, columns=['Labels', 'Currency', 'Content_Id'])

for index, row in df.iterrows():
    print(index)
    r = RawData.create(row)
    listofrows.append(r)

I did the same thing with itertuples which provided much better performance.

for item in df.itertuples():
   listofrows.append(RawData.create(row))

With the number of rows in mind, I am now trying NumPy vectorization. But having a hard time returning a list of objects from ndarrays.

listofrows = (RawData.create( df[''].to_numpy(), df['property2'].to_numpy(),
 df['property3'].to_numpy()).to_list() 

If in create() I have to iterate over the arrays, I figure there's no advantage of doing vectorization. Can this be improved by vectorization? Any help is appreciated.

Note: I am following this article .

Edit: As vectorization is allowed only for primitives, are there any better ways for such a operation?

Answer 1

df["obj"] = df.apply(lambda row: (row["property1"], row["property2"], row["property3"]), axis=1)

Or if there is complex logic involved which can't be fit in one line, then create a function and call in place of lambda .