stackoom
  简体   繁体   中英

convert soap log file to csv using powershell

 原文  2021-02-08 00:49:44  3   1    linux/ powershell/ csv/ soap/ logfile

Question

I have been looking for a good way to convert a log file using powershell into a csv so I can import data correctly into Teradata. I have never seen a file like this before so I am going to try to explain as best as I can.

Here is a sample from the logfile

 [XXXXX:11aa11a-123-12a4-12a3-12323aabb123:4] 2021-02-05 00:00:00,000: [DEBUG] Somesystem::Request: <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"> <soapenv:Header> <ctx:App xmlns:ctx="http://namespace"> <ctx:Session> <ctx:UN>XXXXX</ctx:UN> <ctx:SId>11aa11a-123-12a4-12a3-12323aabb123</ctx:SId> <ctx:Creation>2021-02-05T00:00:00+11:00</ctx:Creation> </ctx:Session> <ctx:IC> <ctx:TId>11aa1aa-1aa1-2a22-3aa1-aa1b1233456</ctx:TId> <ctx:Call> <ctx:BI>13</ctx:BI> <ctx:CN>Somesystem</ctx:CN> </ctx:Call> <ctx:Call> <ctx:BI>1</ctx:BI> </ctx:Call> <ctx:Call> <ctx:BI>1</ctx:BI> </ctx:Call> <ctx:Call> <ctx:BI>1</ctx:BI> </ctx:Call> <ctx:Call> <ctx:BI>1</ctx:BI> <ctx:CN>Somesystem</ctx:CN> <ctx:CL>UnknownLocation</ctx:CL> </ctx:Call> <ctx:Call> <ctx:BI>0</ctx:BI> </ctx:Call> </ctx:IC> </ctx:App> <wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"> <wsse:UT> <wsse:UN>SomesystemUSR</wsse:UN> <wsse:P>Somepassword</wsse:P> </wsse:UT> </wsse:Security> </soapenv:Header> <soapenv:Body> <ns5:ListRequest xmlns:ns5="http://Namespace/Service/V1.0" xmlns="http://www.immi.gov.au/Namespace/Enterprise/ErrorMessages/V1.0" xmlns:ns6="http://Namespace/Core/V1.0" xmlns:ns8="http://Namespace/Service/V1.0" xmlns:ns7="http://Namespace/Core/V1.0" xmlns:ns9="http://Namespace/Core/V1.0" xmlns:ns10="http://Namespace/Core/Messaging/V1.0" xmlns:ns2="http://Namespace/Enterprise/V1.0" xmlns:ns4="http://Namespace/Core/V1.0" xmlns:ns3="http://Namespace/WarningMessages/V1.0"> <ns4:FromDate>2021-01-28</ns4:FromDate> <ns4:ToDate>2021-01-28</ns4:ToDate> <ns4:Code>0123</ns4:Code> <ns4:Type>U</ns4:Type> <ns4:Record>S</ns4:Record> <ns4:AnotherCode>D</ns4:AnotherCode> </ns5:ListRequest> </soapenv:Body> </soapenv:Envelope> (jaxws.LoggingJaxWsHandler) [WContainer: 0] [XXXXX:11aa11a-123-12a4-12a3-12323aabb123BB:4] 2021-02-05 01:00:00,000: [DEBUG] Somesystem::Request: <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"> <soapenv:Header> <ctx:App xmlns:ctx="http://namespace"> <ctx:Session> <ctx:UN>XXXXX</ctx:UN> <ctx:SId>11aa11a-123-12a4-12a3-12323aabb123BB</ctx:SId> <ctx:Creation>2021-02-05T01:00:00+11:00</ctx:Creation> </ctx:Session> <ctx:IC> <ctx:TId>1as23bb3-1er2-234d-234e-bb8b20995147</ctx:TId> <ctx:Call> <ctx:BI>25</ctx:BI> <ctx:CN>Somesystem</ctx:CN> </ctx:Call> <ctx:Call> <ctx:BI>1</ctx:BranchIndex> </ctx:Call> <ctx:Call> <ctx:BI>1</ctx:BI> </ctx:Call> <ctx:Call> <ctx:BI>1</ctx:BI> </ctx:Call> <ctx:Call> <ctx:BI>1</ctx:BI> <ctx:CN>Somesystem</ctx:CN> <ctx:CL>UnknownLocation</ctx:CL> </ctx:Call> <ctx:Call> <ctx:BI>0</ctx:BI> </ctx:Call> </ctx:IC> </ctx:App> <wsse:Security xmlns:wsse="http://xt-1.0.xsd"> <wsse:UT> <wsse:UN>SomeUSR</wsse:UN> </wsse:UT> </wsse:Security> </soapenv:Header> <soapenv:Body> <s:ListResponse xmlns:s="http://Namespace/Service/V1.0" xmlns:t="http://Namespace/Core/V1.0" xmlns:m="http://Namespace/Core/Messaging/V1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://Namespace/Service/V1.0..\Service\V1.0\Response.xsd"> <s:List> <s:FL> <t:lDate>2021-01-05</t:lDate> <t:Code>N</t:Code> <t:lID>AB123</t:lID> </s:FL> <s:FL> <t:lDate>2021-01-05</t:lDate> <t:Code>G</t:Code> <t:lID>CD456</t:lID> </s:FL> <s:FL> <t:Date>2021-01-05</t:Date> <t:Code>W</t:Code> <t:lID>EF654</t:lID> </s:FL> <s:FL> <t:Date>2021-01-05</t:Date> <t:Code>P</t:Code> <t:lID>TR123</t:lID> </s:FL> <s:FL> <t:lDate>2021-01-05</t:lDate> <t:Code>N</t:Code> <t:lID>AB123</t:lID> </s:FL> <s:FL> <t:lDate>2021-01-05</t:lDate> <t:Code>N</t:Code> <t:lID>AB123</t:lID> </s:FL> <s:FL> <t:Date>2021-01-05</t:Date> <t:Code>M</t:Code> <t:lID>AB345</t:lID> </s:FL> <s:FL> <t:Date>2021-01-05</t:Date> <t:Code>C</t:Code> <t:lID>AB456</t:lID> </s:FL> </s:List> </s:ListResponse> </soapenv:Body> </soapenv:Envelope> (jaxws.LoggingJaxWsHandler) [WContainer: 3]

I want to be able to extract data in a format that is easy to import into Teradata for example

UN, SId, Creation, TId, BI, CN, FromDate, ToDate, Code, Type, Record, AnotherCode, FL, Ldate, Iid,...... so-on

I am open to any suggestions as to having a better way of getting a flat structure of out this soap tree. As you can see the second SoapEnvelope has a lot more data in soapBody than the first one. I am unable to find any question similar to this one.

Note: I do not have access to command line

1 answers

solution1
 1 ACCPTED  2021-02-08 01:48:28

As Olaf has stated it might be hard to convert xml into flat csv records. xml is usually made up of many parent/child object whereas csv is typically single object per row

To help maybe guide your progress you can try taking one of the xml envelopes and converting that to an xml object in powershell by doing the following. 

$xml = [xml]@"
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
  <soapenv:Header>
    <ctx:App xmlns:ctx="http://namespace">
      <ctx:Session>
        <ctx:UN>XXXXX</ctx:UN>
        <ctx:SId>11aa11a-123-12a4-12a3-12323aabb123</ctx:SId>
        <ctx:Creation>2021-02-05T00:00:00+11:00</ctx:Creation>
      </ctx:Session>
      <ctx:IC>
        <ctx:TId>11aa1aa-1aa1-2a22-3aa1-aa1b1233456</ctx:TId>
        <ctx:Call>
          <ctx:BI>13</ctx:BI>
          <ctx:CN>Somesystem</ctx:CN>
        </ctx:Call>
        <ctx:Call>
          <ctx:BI>1</ctx:BI>
        </ctx:Call>
        <ctx:Call>
          <ctx:BI>1</ctx:BI>
        </ctx:Call>
        <ctx:Call>
          <ctx:BI>1</ctx:BI>
        </ctx:Call>
        <ctx:Call>
          <ctx:BI>1</ctx:BI>
          <ctx:CN>Somesystem</ctx:CN>
          <ctx:CL>UnknownLocation</ctx:CL>
        </ctx:Call>
        <ctx:Call>
          <ctx:BI>0</ctx:BI>
        </ctx:Call>
      </ctx:IC>
    </ctx:App>
    <wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
      <wsse:UT>
        <wsse:UN>SomesystemUSR</wsse:UN>
        <wsse:P>Somepassword</wsse:P>
      </wsse:UT>
    </wsse:Security>
  </soapenv:Header>
  <soapenv:Body>
    <ns5:ListRequest xmlns:ns5="http://Namespace/Service/V1.0" xmlns="http://www.immi.gov.au/Namespace/Enterprise/ErrorMessages/V1.0" xmlns:ns6="http://Namespace/Core/V1.0" xmlns:ns8="http://Namespace/Service/V1.0" xmlns:ns7="http://Namespace/Core/V1.0" xmlns:ns9="http://Namespace/Core/V1.0"
      xmlns:ns10="http://Namespace/Core/Messaging/V1.0" xmlns:ns2="http://Namespace/Enterprise/V1.0" xmlns:ns4="http://Namespace/Core/V1.0" xmlns:ns3="http://Namespace/WarningMessages/V1.0">
      <ns4:FromDate>2021-01-28</ns4:FromDate>
      <ns4:ToDate>2021-01-28</ns4:ToDate>
      <ns4:Code>0123</ns4:Code>
      <ns4:Type>U</ns4:Type>
      <ns4:Record>S</ns4:Record>
      <ns4:AnotherCode>D</ns4:AnotherCode>
    </ns5:ListRequest>
  </soapenv:Body>
</soapenv:Envelope>
"@

From there you can examine what the object looks like by drilling down into the properties

    PS Env:\> $xml

Envelope
--------
Envelope

PS Env:\> $xml.Envelope

soapenv                                   Header Body
-------                                   ------ ----
http://schemas.xmlsoap.org/soap/envelope/ Header Body

PS Env:\> $xml.Envelope.Body

ListRequest
-----------
ListRequest

PS Env:\> $xml.Envelope.Body.ListRequest

ns5         : http://Namespace/Service/V1.0
xmlns       : http://www.immi.gov.au/Namespace/Enterprise/ErrorMessages/V1.0
ns6         : http://Namespace/Core/V1.0
ns8         : http://Namespace/Service/V1.0
ns7         : http://Namespace/Core/V1.0
ns9         : http://Namespace/Core/V1.0
ns10        : http://Namespace/Core/Messaging/V1.0
ns2         : http://Namespace/Enterprise/V1.0
ns4         : http://Namespace/Core/V1.0
ns3         : http://Namespace/WarningMessages/V1.0
FromDate    : 2021-01-28
ToDate      : 2021-01-28
Code        : 0123
Type        : U
Record      : S
AnotherCode : D

That's all I got. I don't know of any way to convert this type of multidimensional object into flat csv other than picking out what you want from the xml object and creating a custom object with those properties. I'm sure there's someone smarter out there that might be able to offer more:)

UPDATE: So to take this idea a step further and explain what I meant by creating the custom object I've created this function which will take in the xml object and convert it to a custom object with properties I've picked out of the xml. After that you can use either the ConvertTo-Csv or Export-Csv cmdlets to generate the csv/csv file

function Convert-SoapXmlToCustomObject
{
    [cmdletbinding()]
    param(
        [Parameter(ValueFromPipeline = $true, Mandatory = $true)]
        [xml[]]$XmlObject
    )

    Process
    {
        if ($XmlObject)
        {
            foreach ($xml in $XmlObject)
            {
                [pscustomobject][ordered]@{
                    Sid         = $xml.Envelope.Header.App.Session.SId
                    Creation    = $xml.Envelope.Header.App.Session.Creation
                    Tid         = $xml.Envelope.Header.App.IC.TId
                    Calls       = @() #if each call needs to be separate record maybe use loop to create
                    Username    = $xml.Envelope.Header.Security.ut.UN
                    FromDate    = $xml.Envelope.Body.ListRequest.FromDate
                    ToDate      = $xml.Envelope.Body.ListRequest.ToDate
                    Code        = $xml.Envelope.Body.ListRequest.Code
                    Type        = $xml.Envelope.Body.ListRequest.Type
                    Record      = $xml.Envelope.Body.ListRequest.Record
                    AnotherCode = $xml.Envelope.Body.ListRequest.AnotherCode

                }
            }
        }
    }
}

$xml, $xml2 | Convert-SoapXmlToCustomObject | ConvertTo-Csv -NoTypeInformation

Output looks like this 

"Sid","Creation","Tid","Calls","Username","FromDate","ToDate","Code","Type","Record","AnotherCode"
"11aa11a-123-12a4-12a3-12323aabb123","2021-02-05T00:00:00+11:00","11aa1aa-1aa1-2a22-3aa1-aa1b1233456","System.Object[]","SomesystemUSR","2021-01-28","2021-01-28","0123","U","S","D"
"11aa11a-123-12a4-12a3-12323aabb123BB","2021-02-05T01:00:00+11:00","1as23bb3-1er2-234d-234e-bb8b20995147","System.Object[]","SomeUSR",,,,,,

暂无
暂无

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

Related Question Convert postgres text log file to csv file Convert linux log into csv Convert JSON file to CSV file using jq Create a CSV file from a specific log file using shell script How to convert xlsx file to csv file using shell script? How do I convert a .csv file to .db file using python? Convert a json file to csv using shell script without using jq? Converting the bash log file to CSV How to convert a log file to JSON format using shell script? Is there a way to convert one column in csv file to upper using shell commands?
Related Tags
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM