使用 powershell 将 soap 日志文件转换为 csv

Question

I have been looking for a good way to convert a log file using powershell into a csv so I can import data correctly into Teradata. I have never seen a file like this before so I am going to try to explain as best as I can.

Here is a sample from the logfile

 [XXXXX:11aa11a-123-12a4-12a3-12323aabb123:4] 2021-02-05 00:00:00,000: [DEBUG] Somesystem::Request: <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"> <soapenv:Header> <ctx:App xmlns:ctx="http://namespace"> <ctx:Session> <ctx:UN>XXXXX</ctx:UN> <ctx:SId>11aa11a-123-12a4-12a3-12323aabb123</ctx:SId> <ctx:Creation>2021-02-05T00:00:00+11:00</ctx:Creation> </ctx:Session> <ctx:IC> <ctx:TId>11aa1aa-1aa1-2a22-3aa1-aa1b1233456</ctx:TId> <ctx:Call> <ctx:BI>13</ctx:BI> <ctx:CN>Somesystem</ctx:CN> </ctx:Call> <ctx:Call> <ctx:BI>1</ctx:BI> </ctx:Call> <ctx:Call> <ctx:BI>1</ctx:BI> </ctx:Call> <ctx:Call> <ctx:BI>1</ctx:BI> </ctx:Call> <ctx:Call> <ctx:BI>1</ctx:BI> <ctx:CN>Somesystem</ctx:CN> <ctx:CL>UnknownLocation</ctx:CL> </ctx:Call> <ctx:Call> <ctx:BI>0</ctx:BI> </ctx:Call> </ctx:IC> </ctx:App> <wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"> <wsse:UT> <wsse:UN>SomesystemUSR</wsse:UN> <wsse:P>Somepassword</wsse:P> </wsse:UT> </wsse:Security> </soapenv:Header> <soapenv:Body> <ns5:ListRequest xmlns:ns5="http://Namespace/Service/V1.0" xmlns="http://www.immi.gov.au/Namespace/Enterprise/ErrorMessages/V1.0" xmlns:ns6="http://Namespace/Core/V1.0" xmlns:ns8="http://Namespace/Service/V1.0" xmlns:ns7="http://Namespace/Core/V1.0" xmlns:ns9="http://Namespace/Core/V1.0" xmlns:ns10="http://Namespace/Core/Messaging/V1.0" xmlns:ns2="http://Namespace/Enterprise/V1.0" xmlns:ns4="http://Namespace/Core/V1.0" xmlns:ns3="http://Namespace/WarningMessages/V1.0"> <ns4:FromDate>2021-01-28</ns4:FromDate> <ns4:ToDate>2021-01-28</ns4:ToDate> <ns4:Code>0123</ns4:Code> <ns4:Type>U</ns4:Type> <ns4:Record>S</ns4:Record> <ns4:AnotherCode>D</ns4:AnotherCode> </ns5:ListRequest> </soapenv:Body> </soapenv:Envelope> (jaxws.LoggingJaxWsHandler) [WContainer: 0] [XXXXX:11aa11a-123-12a4-12a3-12323aabb123BB:4] 2021-02-05 01:00:00,000: [DEBUG] Somesystem::Request: <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"> <soapenv:Header> <ctx:App xmlns:ctx="http://namespace"> <ctx:Session> <ctx:UN>XXXXX</ctx:UN> <ctx:SId>11aa11a-123-12a4-12a3-12323aabb123BB</ctx:SId> <ctx:Creation>2021-02-05T01:00:00+11:00</ctx:Creation> </ctx:Session> <ctx:IC> <ctx:TId>1as23bb3-1er2-234d-234e-bb8b20995147</ctx:TId> <ctx:Call> <ctx:BI>25</ctx:BI> <ctx:CN>Somesystem</ctx:CN> </ctx:Call> <ctx:Call> <ctx:BI>1</ctx:BranchIndex> </ctx:Call> <ctx:Call> <ctx:BI>1</ctx:BI> </ctx:Call> <ctx:Call> <ctx:BI>1</ctx:BI> </ctx:Call> <ctx:Call> <ctx:BI>1</ctx:BI> <ctx:CN>Somesystem</ctx:CN> <ctx:CL>UnknownLocation</ctx:CL> </ctx:Call> <ctx:Call> <ctx:BI>0</ctx:BI> </ctx:Call> </ctx:IC> </ctx:App> <wsse:Security xmlns:wsse="http://xt-1.0.xsd"> <wsse:UT> <wsse:UN>SomeUSR</wsse:UN> </wsse:UT> </wsse:Security> </soapenv:Header> <soapenv:Body> <s:ListResponse xmlns:s="http://Namespace/Service/V1.0" xmlns:t="http://Namespace/Core/V1.0" xmlns:m="http://Namespace/Core/Messaging/V1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://Namespace/Service/V1.0..\Service\V1.0\Response.xsd"> <s:List> <s:FL> <t:lDate>2021-01-05</t:lDate> <t:Code>N</t:Code> <t:lID>AB123</t:lID> </s:FL> <s:FL> <t:lDate>2021-01-05</t:lDate> <t:Code>G</t:Code> <t:lID>CD456</t:lID> </s:FL> <s:FL> <t:Date>2021-01-05</t:Date> <t:Code>W</t:Code> <t:lID>EF654</t:lID> </s:FL> <s:FL> <t:Date>2021-01-05</t:Date> <t:Code>P</t:Code> <t:lID>TR123</t:lID> </s:FL> <s:FL> <t:lDate>2021-01-05</t:lDate> <t:Code>N</t:Code> <t:lID>AB123</t:lID> </s:FL> <s:FL> <t:lDate>2021-01-05</t:lDate> <t:Code>N</t:Code> <t:lID>AB123</t:lID> </s:FL> <s:FL> <t:Date>2021-01-05</t:Date> <t:Code>M</t:Code> <t:lID>AB345</t:lID> </s:FL> <s:FL> <t:Date>2021-01-05</t:Date> <t:Code>C</t:Code> <t:lID>AB456</t:lID> </s:FL> </s:List> </s:ListResponse> </soapenv:Body> </soapenv:Envelope> (jaxws.LoggingJaxWsHandler) [WContainer: 3]

I want to be able to extract data in a format that is easy to import into Teradata for example

UN, SId, Creation, TId, BI, CN, FromDate, ToDate, Code, Type, Record, AnotherCode, FL, Ldate, Iid,...... so-on

I am open to any suggestions as to having a better way of getting a flat structure of out this soap tree. As you can see the second SoapEnvelope has a lot more data in soapBody than the first one. I am unable to find any question similar to this one.

Note: I do not have access to command line

Answer 1

As Olaf has stated it might be hard to convert xml into flat csv records. xml is usually made up of many parent/child object whereas csv is typically single object per row

To help maybe guide your progress you can try taking one of the xml envelopes and converting that to an xml object in powershell by doing the following.

$xml = [xml]@"
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
  <soapenv:Header>
    <ctx:App xmlns:ctx="http://namespace">
      <ctx:Session>
        <ctx:UN>XXXXX</ctx:UN>
        <ctx:SId>11aa11a-123-12a4-12a3-12323aabb123</ctx:SId>
        <ctx:Creation>2021-02-05T00:00:00+11:00</ctx:Creation>
      </ctx:Session>
      <ctx:IC>
        <ctx:TId>11aa1aa-1aa1-2a22-3aa1-aa1b1233456</ctx:TId>
        <ctx:Call>
          <ctx:BI>13</ctx:BI>
          <ctx:CN>Somesystem</ctx:CN>
        </ctx:Call>
        <ctx:Call>
          <ctx:BI>1</ctx:BI>
        </ctx:Call>
        <ctx:Call>
          <ctx:BI>1</ctx:BI>
        </ctx:Call>
        <ctx:Call>
          <ctx:BI>1</ctx:BI>
        </ctx:Call>
        <ctx:Call>
          <ctx:BI>1</ctx:BI>
          <ctx:CN>Somesystem</ctx:CN>
          <ctx:CL>UnknownLocation</ctx:CL>
        </ctx:Call>
        <ctx:Call>
          <ctx:BI>0</ctx:BI>
        </ctx:Call>
      </ctx:IC>
    </ctx:App>
    <wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
      <wsse:UT>
        <wsse:UN>SomesystemUSR</wsse:UN>
        <wsse:P>Somepassword</wsse:P>
      </wsse:UT>
    </wsse:Security>
  </soapenv:Header>
  <soapenv:Body>
    <ns5:ListRequest xmlns:ns5="http://Namespace/Service/V1.0" xmlns="http://www.immi.gov.au/Namespace/Enterprise/ErrorMessages/V1.0" xmlns:ns6="http://Namespace/Core/V1.0" xmlns:ns8="http://Namespace/Service/V1.0" xmlns:ns7="http://Namespace/Core/V1.0" xmlns:ns9="http://Namespace/Core/V1.0"
      xmlns:ns10="http://Namespace/Core/Messaging/V1.0" xmlns:ns2="http://Namespace/Enterprise/V1.0" xmlns:ns4="http://Namespace/Core/V1.0" xmlns:ns3="http://Namespace/WarningMessages/V1.0">
      <ns4:FromDate>2021-01-28</ns4:FromDate>
      <ns4:ToDate>2021-01-28</ns4:ToDate>
      <ns4:Code>0123</ns4:Code>
      <ns4:Type>U</ns4:Type>
      <ns4:Record>S</ns4:Record>
      <ns4:AnotherCode>D</ns4:AnotherCode>
    </ns5:ListRequest>
  </soapenv:Body>
</soapenv:Envelope>
"@

From there you can examine what the object looks like by drilling down into the properties

    PS Env:\> $xml

Envelope
--------
Envelope

PS Env:\> $xml.Envelope

soapenv                                   Header Body
-------                                   ------ ----
http://schemas.xmlsoap.org/soap/envelope/ Header Body

PS Env:\> $xml.Envelope.Body

ListRequest
-----------
ListRequest

PS Env:\> $xml.Envelope.Body.ListRequest

ns5         : http://Namespace/Service/V1.0
xmlns       : http://www.immi.gov.au/Namespace/Enterprise/ErrorMessages/V1.0
ns6         : http://Namespace/Core/V1.0
ns8         : http://Namespace/Service/V1.0
ns7         : http://Namespace/Core/V1.0
ns9         : http://Namespace/Core/V1.0
ns10        : http://Namespace/Core/Messaging/V1.0
ns2         : http://Namespace/Enterprise/V1.0
ns4         : http://Namespace/Core/V1.0
ns3         : http://Namespace/WarningMessages/V1.0
FromDate    : 2021-01-28
ToDate      : 2021-01-28
Code        : 0123
Type        : U
Record      : S
AnotherCode : D

That's all I got. I don't know of any way to convert this type of multidimensional object into flat csv other than picking out what you want from the xml object and creating a custom object with those properties. I'm sure there's someone smarter out there that might be able to offer more:)

UPDATE: So to take this idea a step further and explain what I meant by creating the custom object I've created this function which will take in the xml object and convert it to a custom object with properties I've picked out of the xml. After that you can use either the ConvertTo-Csv or Export-Csv cmdlets to generate the csv/csv file

function Convert-SoapXmlToCustomObject
{
    [cmdletbinding()]
    param(
        [Parameter(ValueFromPipeline = $true, Mandatory = $true)]
        [xml[]]$XmlObject
    )

    Process
    {
        if ($XmlObject)
        {
            foreach ($xml in $XmlObject)
            {
                [pscustomobject][ordered]@{
                    Sid         = $xml.Envelope.Header.App.Session.SId
                    Creation    = $xml.Envelope.Header.App.Session.Creation
                    Tid         = $xml.Envelope.Header.App.IC.TId
                    Calls       = @() #if each call needs to be separate record maybe use loop to create
                    Username    = $xml.Envelope.Header.Security.ut.UN
                    FromDate    = $xml.Envelope.Body.ListRequest.FromDate
                    ToDate      = $xml.Envelope.Body.ListRequest.ToDate
                    Code        = $xml.Envelope.Body.ListRequest.Code
                    Type        = $xml.Envelope.Body.ListRequest.Type
                    Record      = $xml.Envelope.Body.ListRequest.Record
                    AnotherCode = $xml.Envelope.Body.ListRequest.AnotherCode

                }
            }
        }
    }
}

$xml, $xml2 | Convert-SoapXmlToCustomObject | ConvertTo-Csv -NoTypeInformation

Output looks like this

"Sid","Creation","Tid","Calls","Username","FromDate","ToDate","Code","Type","Record","AnotherCode"
"11aa11a-123-12a4-12a3-12323aabb123","2021-02-05T00:00:00+11:00","11aa1aa-1aa1-2a22-3aa1-aa1b1233456","System.Object[]","SomesystemUSR","2021-01-28","2021-01-28","0123","U","S","D"
"11aa11a-123-12a4-12a3-12323aabb123BB","2021-02-05T01:00:00+11:00","1as23bb3-1er2-234d-234e-bb8b20995147","System.Object[]","SomeUSR",,,,,,