在Django中从csv到sql的熊猫：OperationalError，表x没有名为y的列

Question

我正在建立一个网站来管理家用设备产生的数据。 数据包含在.csv文件中，随着时间的流逝将有数百/数千个数据被摄取。

csv看起来像这样：

Timestamp, Address (64bit), Zone, Sensor Type, Data type, Value
12/23/16 02:05:30, some_64bit_address, 5, 0, 0, 255

我可以读取csv并将其转换为pandas数据框，但是在将其保存到sql时遇到了麻烦。 我得到一个

/ import /：表FromCsv上的OperationalError没有名为Address（64bit）的列。

我知道有几个与此问题类似的问题，但是我找不到能解决我问题的答案，也不知道下一步该怎么做。 我试图与不index_col=False在read_csv ，有和没有index=False在to_sql ，我试图删除数据库和迁移文件，并从头开始。 我正在使用python 3.5，django 1.11，pandas 0.19.2和sqlite3。

风景：

def import_data(request):
    if request.method == "POST":
        files = request.FILES.getlist("csv_files")
        dateparse = lambda x: pd.datetime.strptime(x, "%m/%d/%y %H:%M:%S")

        for fichier in files:
            # convert the file in memory to stringio
            decoded = fichier.read().decode('utf-8')
            io_string = StringIO(decoded)

            try:
                # convert to dataframe
                chunks = pd.read_csv(
                    filepath_or_buffer=io_string,
                    parse_dates=["Timestamp"],
                    date_parser=dateparse,
                    chunksize=500000
                )

                for chunk in chunks:
                    chunk.columns = [
                        "Timestamp",
                        "Address (64bit)",
                        "Zone",
                        "Sensor Type",
                        "Data type",
                        "Value"]

                    # save table in database
                    with sqlite3.connect("db.sqlite3") as conn:
                        chunk.to_sql("FromCsv", con=conn, index=False, if_exists="append")

                messages.success(request, fichier.name)

            except pd.io.common.EmptyDataError:
                pass

        return render(request, "upload_done.html")
    return render(request, "import_data.html")

该模型：

class RawData(models.Model):

    class Meta:
        db_table = "FromCsv"

    timestamp = models.DateTimeField(default=timezone.now)
    address = models.CharField(max_length=250)
    zone = models.IntegerField(default=0)
    sensor = models.IntegerField(default=0)
    data_type = models.IntegerField(default=0)
    data_value = models.IntegerField(default=0)

    def __str__(self):
        return str(self.timestamp)

追溯：

Traceback:

File "/home/smoky05/.virtualenvs/ubiosData/lib/python3.5/site-packages/django/core/handlers/exception.py" in inner
  39.             response = get_response(request)

File "/home/smoky05/.virtualenvs/ubiosData/lib/python3.5/site-packages/django/core/handlers/base.py" in _get_response
  187.                 response = self.process_exception_by_middleware(e, request)

File "/home/smoky05/.virtualenvs/ubiosData/lib/python3.5/site-packages/django/core/handlers/base.py" in _get_response
  185.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/home/smoky05/PycharmProjects/UbiosDataSite/data/views.py" in import_data
  62.                         chunk.to_sql("FromCsv", con=conn, index=False, if_exists="append")

File "/home/smoky05/.virtualenvs/ubiosData/lib/python3.5/site-packages/pandas/core/generic.py" in to_sql
  1201.                    chunksize=chunksize, dtype=dtype)

File "/home/smoky05/.virtualenvs/ubiosData/lib/python3.5/site-packages/pandas/io/sql.py" in to_sql
  470.                       chunksize=chunksize, dtype=dtype)

File "/home/smoky05/.virtualenvs/ubiosData/lib/python3.5/site-packages/pandas/io/sql.py" in to_sql
  1503.         table.insert(chunksize)

File "/home/smoky05/.virtualenvs/ubiosData/lib/python3.5/site-packages/pandas/io/sql.py" in insert
  664.                 self._execute_insert(conn, keys, chunk_iter)

File "/home/smoky05/.virtualenvs/ubiosData/lib/python3.5/site-packages/pandas/io/sql.py" in _execute_insert
  1291.         conn.executemany(self.insert_statement(), data_list)

Exception Type: OperationalError at /import/
Exception Value: table FromCsv has no column named Address (64bit)

Answer 1

使用GUI sqlite查看器，我发现了问题：创建的表使用了目标模型中的列名，因此相应地重命名数据框的列使其可以工作：

chunk.columns = [
                        "timestamp",
                        "address",
                        "zone",
                        "sensor",
                        "data_type",
                        "data_value"]

现在它与模型匹配：

class RawData(models.Model):

    class Meta:
        db_table = "FromCsv"

    timestamp = models.DateTimeField(default=timezone.now)
    address = models.CharField(max_length=250)
    zone = models.IntegerField(default=0)
    sensor = models.IntegerField(default=0)
    data_type = models.IntegerField(default=0)
    data_value = models.IntegerField(default=0)

    def __str__(self):
        return str(self.timestamp)