堆棧內存溢出
  簡體   English   中英

csv 中的數據，數據用雙引號括起來

[英]Data in csv with enclosed data in double quotation marks

 原文  2020-11-17 17:13:28  7   2    python/ pandas/ csv/ quotation-marks

問題描述

我正在嘗試從開放的在線 .csv 文件中檢索數據： http : //www.anp.gov.br/arquivos/acesso-informacao/dp/2020-producao-mar.csv

我正在使用 Anaconda + Spyder + Pandas。 我使用的命令行是這樣的：

FileList = ['http://www.anp.gov.br/arquivos/acesso-informacao/dp/2020-producao-mar.csv']
arq1 = FileList[0]
df1 = pd.read_csv(arq1, quotechar = '"')

Pandas 能夠讀取文件，但無法正確解析行。 它無法解析的行是包含雙引號內數據的行，例如：

'2020,01/2020,Bahia,Camamu,MANATI,7-MNT-3-BAS,Mar,PLATAFORMA DE MANATI 1,0,"241,729",0,"12257,70101","61,573",,,,,,,,'

我也嘗試過這種方法：

file1 = pd.read_csv(arq1,sep=',\s*',skipinitialspace=True,quoting=csv.QUOTE_ALL,engine='python')

但是這第二種方法會導致以下錯誤：

UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 212: character maps to <undefined>

你能給我建議嗎？

測試數據

  • 該文件已包含在內，但如果該文件不再可用，以下內容足以重現問題。
  • 將以下數據存儲在csv文件中。
Ano,Mês/Ano,Estado,Bacia,Campo,Poço,Ambiente,Instalação,Produção de Óleo (m³),Produção de Condensado (m³),Produção de Gás Associado (Mm³),Produção de Gás Não Associado (Mm³),Produção de Água (m³),Injeção de Gás (Mm³),Injeção de Água para Recuperação Secundária (m³),Injeção de Água para Descarte (m³),Injeção de Gás Carbônico (Mm³),Injeção de Nitrogênio (Mm³),Injeção de Vapor de Água (t),Injeção de Polímeros (m³),Injeção de Outros Fluidos (m³)
2020,01/2020,Alagoas,Alagoas,PARU,4-ALS-39-AL,Mar,Não Informado,0,0,0,0,0,,,,,,,,
"2020,01/2020,Bahia,Camamu,MANATI,7-MNT-1-BAS,Mar,PLATAFORMA DE MANATI 1,0,""265,58"",0,""17605,52003"",""74,489"",,,,,,,,"
"2020,01/2020,Bahia,Camamu,MANATI,7-MNT-2-BAS,Mar,PLATAFORMA DE MANATI 1,0,""326,366"",0,""17810,97775"",""84,152"",,,,,,,,"
"2020,01/2020,Bahia,Camamu,MANATI,7-MNT-3-BAS,Mar,PLATAFORMA DE MANATI 1,0,""241,729"",0,""12257,70101"",""61,573"",,,,,,,,"
"2020,01/2020,Bahia,Camamu,MANATI,7-MNT-4-BAS,Mar,PLATAFORMA DE MANATI 1,0,""285,911"",0,""17013,25742"",""88,015"",,,,,,,,"
"2020,01/2020,Bahia,Camamu,MANATI,7-MNT-5D-BAS,Mar,PLATAFORMA DE MANATI 1,0,""173,078"",0,""20459,1769"",""68,169"",,,,,,,,"
"2020,01/2020,Bahia,Camamu,MANATI,7-MNT-6D-BAS,Mar,PLATAFORMA DE MANATI 1,0,""178,857"",0,""24557,04732"",""75,546"",,,,,,,,"
"2020,01/2020,Bahia,Recôncavo,CANDEIAS,7-C-173D-BA,Mar,Estação Pedra Branca,""95,742"",0,""82,24558"",0,""0,194"",,,,,,,,"
2020,01/2020,Bahia,Recôncavo,CANDEIAS,7-C-174D-BA,Mar,Não Informado,0,0,0,0,0,,,,,,,,
2020,01/2020,Bahia,Recôncavo,CANDEIAS,7-C-197D-BA,Mar,Não Informado,0,0,0,0,0,,,,,,,,
2020,01/2020,Bahia,Recôncavo,CANDEIAS,7-C-201D-BA,Mar,Não Informado,0,0,0,0,0,,,,,,,,
2020,01/2020,Bahia,Recôncavo,CANDEIAS,7-C-202D-BA,Mar,Não Informado,0,0,0,0,0,,,,,,,,
2020,01/2020,Bahia,Recôncavo,CANDEIAS,7-C-203D-BA,Mar,Não Informado,0,0,0,0,0,,,,,,,,
2020,01/2020,Bahia,Recôncavo,CANDEIAS,7-C-211D-BA,Mar,Não Informado,0,0,0,0,0,,,,,,,,
2020,01/2020,Bahia,Recôncavo,CANDEIAS,7-C-212D-BA,Mar,Não Informado,0,0,0,0,0,,,,,,,,
"2020,01/2020,Bahia,Recôncavo,DOM JOÃO,7-DJM-854H-BAS,Mar,Estação Marapé (Dom João Mar),""388,00158"",0,""3,10388"",0,""3221,81179"",,,,,,,,"
"2020,01/2020,Bahia,Recôncavo,DOM JOÃO,7-DJM-856H-BAS,Mar,Estação Marapé (Dom João Mar),""318,49041"",0,""2,54778"",0,""4814,03179"",,,,,,,,"
"2020,01/2020,Bahia,Recôncavo,DOM JOÃO,7-DJM-857H-BAS,Mar,Estação Marapé (Dom João Mar),""149,19484"",0,""1,19341"",0,""2641,14209"",,,,,,,,"
"2020,01/2020,Bahia,Recôncavo,DOM JOÃO,8-DJ-811H-BAS,Mar,Estação Marapé (Dom João Mar),,,,,,0,""5816,23328"",0,0,0,0,0,0"
"2020,01/2020,Bahia,Recôncavo,DOM JOÃO,8-DJM-858H-BAS,Mar,Estação Marapé (Dom João Mar),,,,,,0,""5396,07916"",0,0,0,0,0,0"
"2020,01/2020,Bahia,Recôncavo,DOM JOÃO,8-DJM-881H-BAS,Mar,Estação Marapé (Dom João Mar),""196,46254"",0,""1,57155"",0,""2268,57935"",,,,,,,,"
"2020,01/2020,Bahia,Recôncavo,DOM JOÃO MAR,7-DJM-854H-BAS,Mar,Estação Marapé (Dom João Mar),""56,69942"",0,""0,45345"",0,""470,80921"",,,,,,,,"
"2020,01/2020,Bahia,Recôncavo,DOM JOÃO MAR,7-DJM-856H-BAS,Mar,Estação Marapé (Dom João Mar),""46,54159"",0,""0,37222"",0,""703,48321"",,,,,,,,"
"2020,01/2020,Bahia,Recôncavo,DOM JOÃO MAR,7-DJM-857H-BAS,Mar,Estação Marapé (Dom João Mar),""21,80216"",0,""0,17426"",0,""385,95491"",,,,,,,,"
2020,01/2020,Bahia,Recôncavo,DOM JOÃO MAR,7-DJM-882H-BAS,Mar,Não Informado,0,0,0,0,0,,,,,,,,
"2020,01/2020,Bahia,Recôncavo,DOM JOÃO MAR,8-DJ-811H-BAS,Mar,Estação Marapé (Dom João Mar),,,,,,0,""849,93672"",0,0,0,0,0,0"
"2020,01/2020,Bahia,Recôncavo,DOM JOÃO MAR,8-DJM-858H-BAS,Mar,Estação Marapé (Dom João Mar),,,,,,0,""788,53884"",0,0,0,0,0,0"
"2020,01/2020,Bahia,Recôncavo,DOM JOÃO MAR,8-DJM-881H-BAS,Mar,Estação Marapé (Dom João Mar),""28,70946"",0,""0,22956"",0,""331,51165"",,,,,,,,"
2020,01/2020,Ceará,Ceará,ATUM,3-AT-8-CES,Mar,Não Informado,0,0,0,0,0,,,,,,,,
2020,01/2020,Ceará,Ceará,ATUM,3-CES-83-CE,Mar,Não Informado,0,0,0,0,0,,,,,,,,
2020,01/2020,Ceará,Ceará,ATUM,3-CES-86D-CE,Mar,Não Informado,0,0,0,0,0,,,,,,,,
2020,01/2020,Ceará,Ceará,ATUM,7-AT-10D-CES,Mar,Não Informado,0,0,0,0,0,,,,,,,,
2020,01/2020,Ceará,Ceará,ATUM,7-AT-13D-CES,Mar,Não Informado,0,0,0,0,0,,,,,,,,
2020,01/2020,Ceará,Ceará,ATUM,7-AT-16D-CES,Mar,Não Informado,0,0,0,0,0,,,,,,,,
2020,01/2020,Ceará,Ceará,ATUM,7-AT-17D-CES,Mar,Não Informado,0,0,0,0,0,,,,,,,,
2020,01/2020,Ceará,Ceará,ATUM,7-AT-18D-CES,Mar,Não Informado,0,0,0,0,0,,,,,,,,
2020,01/2020,Ceará,Ceará,ATUM,7-AT-19D-CES,Mar,Não Informado,0,0,0,0,0,,,,,,,,
2020,01/2020,Ceará,Ceará,ATUM,7-AT-21DP-CES,Mar,Não Informado,0,0,0,0,0,,,,,,,,
"2020,01/2020,Ceará,Ceará,ATUM,7-AT-22DP-CES,Mar,PLATAFORMA DE ATUM 2,""328,927"",0,""23,30745"",0,""761,164"",,,,,,,,"

2 個解決方案

解決方案1
 2  2020-11-17 18:57:04

  • 有 5735 行不在"..."
  • 開頭和結尾'"'已從其他 10735 行中刪除，某些值包含在 2 組雙引號中（例如""..."" ），已修復，因為它會導致行長度不均勻的問題.
  • 創建數據框時，使用decimal=','這樣數字將被正確解析為float類型。
import pandas as pd

# file location
file = 'e:/PythonProjects/stack_overflow/data/2020-producao-mar.csv'

# open the original file to read and open a file to write to
with open(file, mode='r', encoding='utf-8') as f, open('cleaned.csv', mode='w', encoding='utf-8') as f1:
    
    # read the lines
    lines = f.readlines()
    
    # parse each line
    for line in lines:
        
        # remove newlines from the end of the rows
        line = line.strip()
        
        # find rows with beginning and ending quotes
        if (line.startswith('"') == True) and (line.endswith('"') == True):
            
            # extract the string between the beginning and end quote
            line = line[1:-1]
            
            # replace 2 double quotes for 1 double quote
            line = line.replace('""', '"')
            
        # write the rows to cleaned.csv
        f1.writelines(line + '\n')

# create a dataframe with original.csv
df1 = pd.read_csv('cleaned.csv', encoding='utf-8', quotechar='"', decimal=',')

# display(df1.head())
    Ano  Mês/Ano   Estado    Bacia   Campo         Poço Ambiente              Instalação  Produção de Óleo (m³)  Produção de Condensado (m³)  Produção de Gás Associado (Mm³)  Produção de Gás Não Associado (Mm³)  Produção de Água (m³)  Injeção de Gás (Mm³)  Injeção de Água para Recuperação Secundária (m³)  Injeção de Água para Descarte (m³)  Injeção de Gás Carbônico (Mm³)  Injeção de Nitrogênio (Mm³)  Injeção de Vapor de Água (t)  Injeção de Polímeros (m³)  Injeção de Outros Fluidos (m³)
0  2020  01/2020  Alagoas  Alagoas    PARU  4-ALS-39-AL      Mar           Não Informado                    0.0                        0.000                              0.0                              0.00000                  0.000                   NaN                                               NaN                                 NaN                             NaN                          NaN                           NaN                        NaN                             NaN
1  2020  01/2020    Bahia   Camamu  MANATI  7-MNT-1-BAS      Mar  PLATAFORMA DE MANATI 1                    0.0                      265.580                              0.0                          17605.52003                 74.489                   NaN                                               NaN                                 NaN                             NaN                          NaN                           NaN                        NaN                             NaN
2  2020  01/2020    Bahia   Camamu  MANATI  7-MNT-2-BAS      Mar  PLATAFORMA DE MANATI 1                    0.0                      326.366                              0.0                          17810.97775                 84.152                   NaN                                               NaN                                 NaN                             NaN                          NaN                           NaN                        NaN                             NaN
3  2020  01/2020    Bahia   Camamu  MANATI  7-MNT-3-BAS      Mar  PLATAFORMA DE MANATI 1                    0.0                      241.729                              0.0                          12257.70101                 61.573                   NaN                                               NaN                                 NaN                             NaN                          NaN                           NaN                        NaN                             NaN
4  2020  01/2020    Bahia   Camamu  MANATI  7-MNT-4-BAS      Mar  PLATAFORMA DE MANATI 1                    0.0                      285.911                              0.0                          17013.25742                 88.015                   NaN                                               NaN                                 NaN                             NaN                          NaN                           NaN                        NaN                             NaN

解決方案2
 0  2020-11-17 19:56:01

非常感謝大家！

代碼是這樣工作的：

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import csv
import urllib
print('\014')




FileList = ['http://www.anp.gov.br/arquivos/acesso-informacao/dp/2020-producao-mar.csv']


file = FileList[0]

urllib.request.urlretrieve (file, "inputfile.csv")    

with open('inputfile.csv', mode='r', encoding='utf-8') as f, open('cleaned.csv', mode='w', encoding='utf-8') as f1:

    # read the lines
    lines = f.readlines()

    # parse each line
    for line in lines:

        # remove newlines from the end of the rows
        line = line.strip()

        # find rows with beginning and ending quotes and write those rows to cleaned.csv
        if (line.startswith('"') == True) and (line.endswith('"') == True):

            # extract the string between the beginning and end quote
            line = line[1:-1]
            line = line.replace('""', '"')

            f1.writelines(line + '\n')

        # write the remaining rows to original.csv
        else:
            line = line.replace('""', '"')
            f1.writelines(line + '\n')


# create a dataframe with original.csv
df1 = pd.read_csv('cleaned.csv', decimal=',' , encoding='utf-8', quotechar='"')

暫無
暫無

聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.

相關問題 Python數據框導出到帶引號的csv（“） 使用Pandas從CSV解析帶引號的JSON字符串 Python，在多個雙引號之間分割數據？ 使CSV轉義雙引號 從雙引號括起來的 csv 文件格式加載配置單元數據 用下划線連接用引號引起來的多個字符串 保存的數據帶有不希望的引號 打印數據框時去掉引號 CSV文件中不需要的引號 如何使用Python轉義特定.csv列中的所有單雙引號？
相關標簽
 
粵ICP備18138465號  © 2020-2024 STACKOOM.COM