将列添加到现有 dataframe 并将数据导入到 Python 中的新列 Pandas

Question

我正在使用 Python 将 CSV 文件读入 pandas dataframe。我想将文本文件列表读入 dataframe 的新列中。

我正在读取的原始 CSV 文件如下所示：

Name,PrivateIP
bastion001,10.238.2.166
logicmonitor001,10.238.2.52
logicmonitor002,45.21.2.13

原来的 dataframe 看起来是这样的。

代码：

hosts_list = dst = os.path.join('..', '..', 'source_files', 'aws_hosts_list', 'aws_hosts_list.csv')
fields = ["Name", "PrivateIP"]
orig_df = pd.read_csv(hosts_list, skipinitialspace=True, usecols=fields)
print(f"Orig DF: {orig_df}")

output：

Orig DF:
                       Name     PrivateIP
0               bastion001  10.238.2.166
1          logicmonitor001   10.238.2.52
2         logicmonitor002    45.21.2.13

---

文本目录中有一堆文本文件，每个文件有 memory 个读数：

---

bastion001-memory.txt              B-mmp-rabbitmq-core002-memory.txt  logicmonitor002-memory.txt    mmp-cassandra001-memory.txt  company-division-rcsgw002-memory.txt
B-mmp-platsvc-core001-memory.txt   haproxy001-memory.txt              company-cassandra001-memory.txt  mmp-cassandra002-memory.txt  company-waepd001-memory.txt
B-mmp-platsvc-core002-memory.txt   haproxy002-memory.txt              company-cassandra002-memory.txt  mmp-cassandra003-memory.txt  company-waepd002-memory.txt
B-mmp-rabbitmq-core001-memory.txt  logicmonitor001-memory.txt         company-cassandra003-memory.txt  company-division-rcsgw001-memory.txt  company-waepd003-memory.txt

---

每个文件看起来都类似于：

cat haproxy001-memory.txt
7706172

我把每个文件读入现有的dataframe。

---

rowcount == 0
text_path = '/home/tdun0002/stash/cloud_scripts/output_files/memory_stats/text/'
filelist = os.listdir(text_path)
for filename in filelist:
    if rowcount == 0:
        pass
    else:
        my_file = text_path + filename
        print(f"Adding {filename} to DF")
        try:
            orig_df = pd.update(my_file)
            print(f"Data Frame: {orif_df}")
            ++rowcount
        except Exception as e:
            print(f"An error has occurred: {e}")

---

但是当我再次尝试读取结果 dataframe 时，它还没有更新。 为清楚起见，我给新 DF 取了一个新名称。

代码：

result_df = orig_df
pd.options.display.max_rows
print(f"\nResult Data Frame:\n{result_df}\n")

output：

Result Data Frame:
                      Name     PrivateIP
0               bastion001  10.238.2.166
1          logicmonitor001   10.238.2.52
2          logicmonitor002    45.21.2.13

如何在 DF 中创建一个名为Memory的新列并将文本文件的内容添加到该列？

Answer 1

这是我希望可以使用的代码。 这有点笨拙，但你会明白的。 里面有评论。

import pandas as pd
import os
from os import listdir
from os.path import isfile, join

# get all files in the directory
# i used os.getcwd() to get the current directory
# if your text files are in another dir, then write exact dir location
# this gets you all files in your text dir
onlyfiles = [f for f in listdir(os.getcwd()) if isfile(join(os.getcwd(), f))]

# convert it to series
memory_series = pd.Series(onlyfiles)

# an apply function to get just txt files
# others will be returned as None
def file_name_getter(x):
    names = x.split(".", maxsplit=1)
    if names[1] == "txt":
        return names[0]
    else:
        return None

# apply the function and get a new series with name values
mem_list = memory_series.apply(lambda x: file_name_getter(x))

# now read first line of txt files
# and this is the function for it
def get_txt_data(x):
    if x != None:
        with open(f'{x}.txt') as f:
            return int(f.readline().rstrip())
    else:
        return 0

# apply the function, get a new series with memory values
mem_val_list = mem_list.apply(lambda x: get_txt_data(x))

# create a df where our Name and Memory data are present
# cast Memory data as int
df = pd.DataFrame(mem_val_list, columns=["Memory"], dtype="int")
df["Name"] = mem_list

# get rid of -memory now
def name_normalizer(x):
    if x is None:
        return x
    else:
        return x.rsplit("-", maxsplit=1)[0]

# apply function
df["Name"] = df["Name"].apply(lambda x:  name_normalizer(x))


# our sample orig_df
orig_df = pd.DataFrame([["algo_2", "10.10.10"], ["other", "20.20.20"]], columns=["Name", "PrivateIP"])

# merge using on, so if we miss data; that data wont cause any problem
# all matching names will get their memory values
final_df = orig_df.merge(df, on="Name")

编辑：修复了要正确返回的Name 。 (xxx-内存到xxx)