python .xml和.csv文件操作

Question

I made a conversion from an .xml file to .csv. In the .xml file there were some values ​​from the txtDescricao column of this type: "Logistics, Search and Support." Because of this, when I read the file, pandas interprets that comma after Logistics as a column separator, and throws the rest of the text forward. I am trying to work around this with the following code:

in_file = 'dados_limpos_2018.csv'
out_file = 'dados_2018.csv'
output = open(out_file, 'w')
with open(in_file, 'r') as source:
    for line in source:
    # split by semicolon
        data = line.strip().split(';')             
    # remove all quotes found
        data = [t.replace('"','') for t in data]
        for item in data[:-1]:
            item.replace(',', '')
            output.write(''.join(['', item, '',',']))
            # write the last item separately, without the trailing ';'
        output.write(''.join(['"', item, '"']))
        output.write('\n')
output.close()

however, in the line python already interprets the comma as separator and turns it into a semicolon. Here I would like to know: Is there any way I can handle this in the .csv file, or would I have to do this in .xml to .csv conversion? Example of .cs file

name, number, sgUF, txtDescricao, year
Romario, 15, RJ, Consultoria, 2018
Ronaldo, 9, RJ, Logistics, Search and Support, 2018

Example .xml file:

<?xml version="1.0" encoding="UTF-8"?>
<xml>
    <dados>
          <despesa>
                  <name>Romario</name>
                  <number>15</number>
                  <sgUF>RJ</sgUF>
                  <txtDescricao>Consultoria</txtDescricao>
                  <year>2018</year>
           </despesa>

           <despesa>
                  <name>Ronaldo</name>
                  <number>9</number>
                  <sgUF>RJ</sgUF>
                  <txtDescricao>Logistics, Search and Support</txtDescricao>
                  <year>2018</year>
           </despesa>
     </dados>
</xml>

Note: The original file is too large to open in spreadsheet editor.

Answer 1

It would be nice if you share your xml file.

Based on supplied info,

If your xml file data has , as value, use different separator(semicolon,tab,space) to form your csv file. Or Just replace , with null when its in XML file, then convert.

In both situations, you should handle this while converting from xml to csv. With csv -> csv will be hard to implement and count of , will be unpredictable.

EDIT 1:

I suggest to use objectify from lxml. Dont forget to delete <?xml version="1.0" encoding="UTF-8"?> from your xml. Solution is below.

from lxml import objectify
import csv

file_xml = open('d:\\path\\to\\xml.xml','r')
converted_csv_file = open("converted.csv","w")
xml_string = file_xml.read()
xml_object = objectify.fromstring(xml_string)
csvwriter = csv.writer(converted_csv_file, delimiter=',',lineterminator = '\n')
count = 0
for row in xml_object.dados.despesa:
    if count == 0:
        csvwriter.writerow([row.name.tag,row.number.tag,row.sgUF.tag,row.txtDescricao.tag,row.year.tag])
    csvwriter.writerow([row.name.text,row.number.text,row.sgUF.text,row.txtDescricao.text.replace(',',''),row.year.text])
    count += 1

You can install lxml by

pip install lxml

Answer 2

I modified your function to deal with those cases in the txtDescricao column.

ncols= 5
index = 3
in_file = 'dados_limpos_2018.csv'
out_file = 'dados_2018.csv'
output = open(out_file, 'w')
with open(in_file, 'r') as source:
     for line in source:
         # split by colon
         data = line.strip().split(',')
         # Change third element
         data_len = len(data)
         if  data_len > ncols:
             # Join all elements
             data[index] = ''.join(data[index:index + 1 + (data_len - ncols)])
             data[index + 1:] = data[index + 1 + data_len - ncols:]
         # Write columns
         output.write(','.join(data[:ncols]))
         output.write('\n')
 output.close()

Input file:

name, number, sgUF, txtDescricao, year
Romario, 15, RJ, Consultoria, 2018
Ronaldo, 9, RJ, Logistics, Search and Support, 2018

Output file:

name, number, sgUF, txtDescricao, year
Romario, 15, RJ, Consultoria, 2018
Ronaldo, 9, RJ, Logistics Search and Support, 2018

OBS.: I am assuming that this problem only occurs in the txtDecricao column.