Reading json Payload as part of a CSV file in dataweave

Question

I have a web service which exports a CSV file which has one of the columns as a JSON payload. After the execution of the web service, I store the values in a local variable for transformation. Every time when I read the values from that column I am missing the values and only "}" is returned. Not sure why this is happening. I want to preserve the JSON payload as is and persist to a file after some processing. Please advise

I am using the code below to get the value of the attribute column and it always returns a "}". The rest of the contents are ignored

CSV Fragment
-------------
 id,name,attributes
 1,name1,{"Choice Number":"0","Campaign Run Id":"234"}
 2,name2,{"Choice Number":"1","Campaign Run Id":"235"}
 3,name3,{"Choice Number":"2","Campaign Run Id":"236"}
 4,name4,{"Choice Number":"3","Campaign Run Id":"236"}


Code
----
 %dw 1.0
 %output application/java
 ---
 flowVars.activityData map ((actData) -> {
      "playerId": actData.id,
      "name": actData.name,
      "attributes": actData.attributes
  })

I was expecting that the full JSON payload from the attributes column will be returned and that is not the case. One thing that I noticed here is that there is no escaping of characters in the JSON payload in the input. But I don't have any control on that as well. How do I extract the information from the attribute column in this case

---

Since I cannot share the whole project, created a sample project and using the inputs from @machaval with http object receiving the csv file. Marked the mimetype as text/csv and sending the payload

<?xml version="1.0" encoding="UTF-8"?>

<mule xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns:tracking="http://www.mulesoft.org/schema/mule/ee/tracking" xmlns:file="http://www.mulesoft.org/schema/mule/file" xmlns:dw="http://www.mulesoft.org/schema/mule/ee/dw" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
    xmlns:spring="http://www.springframework.org/schema/beans" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/file http://www.mulesoft.org/schema/mule/file/current/mule-file.xsd
http://www.mulesoft.org/schema/mule/ee/dw http://www.mulesoft.org/schema/mule/ee/dw/current/dw.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/ee/tracking http://www.mulesoft.org/schema/mule/ee/tracking/current/mule-tracking-ee.xsd">
    <http:listener-config name="HTTP_Listener_Configuration" host="0.0.0.0" port="8081" doc:name="HTTP Listener Configuration"/>
    <flow name="CSVFlow">
        <http:listener config-ref="HTTP_Listener_Configuration" path="/process" doc:name="HTTP"/>
        <object-to-string-transformer doc:name="Object to String"/>
        <dw:transform-message doc:name="Transform Message">
            <dw:set-payload><![CDATA[%dw 1.0
%output application/json
---
//First remove the header
(payload splitBy "\n")[1 to -1] 
    map ((item, index) -> using(commas = item find ",")             
        {
            id: item[0 to commas[0] - 1],
            name: item[commas[0] + 1 to commas[1] - 1],
            attributes: item[commas[1] + 1 to -1]
        }
    )]]></dw:set-payload>
        </dw:transform-message>
    </flow>

</mule>

Answer 1

Hi Rajeev the problem with your data is that is not a CSV the json needs to be escaped somehow. So the way I solved your problem was by handling your input as a String (this can be easily done by using the object to string mp) and parsing it by hand. My assumption is that there are not more tricks on your format.

%dw 1.0
output application/json  
---
//First remove the header
(payload splitBy "\n")[1 to -1] 
    map ((item, index) -> using(commas = item find ",")             
        {
            id: item[0 to commas[0] - 1],
            name: item[commas[0] + 1 to commas[1] - 1],
            attributes: item[commas[1] + 1 to -1]
        }
    )

Answer 2

there is one problem - special characters in csv(, ") are surrounded by quotes. if there are more than one columns with multiple quotes and commas in the values the above solution will fail. i took the liberty to modify the solution and add some tweaks to it:

%dw 2.0
output application/java
import * from dw::core::Arrays
var headerRow = (data) -> ((data splitBy "\n")[0]) splitBy ","
var dataRows = (data) -> (data splitBy "\n")[1 to -1]
---
dataRows(payload) map (dataRow,index) -> 
do 
{
    var commas = dataRow find ","
    var indices = flatten(dataRow find /(?<!")"(?!")/)
    var quoteposition = 

            indices map 
            (
                (item, index) -> 
                (
                    {
                        (start:item) if isEven(index),
                        (end:indices[index + 1]) if isEven(index)
                    }
                ) 

            ) filter $ != null and $ != {}

    fun removeCommasinQuotes(c: Array<Number>, q: Array<{|start?: Number, end?: Number|}>) =  c filter (item,index) -> !(q some (item > $.start and item < $.end))
    var separator = removeCommasinQuotes(commas,quoteposition)
    ---
    {
        (headerRow(payload)[0]): dataRow[0 to separator[0] - 1],
        (headerRow(payload)[1]):dataRow[separator[0] + 1 to separator[1] - 1],
        (headerRow(payload)[2]):dataRow[separator[1] + 1 to separator[2] - 1],
        (headerRow(payload)[3]):dataRow[separator[2] + 1 to separator[3] - 1],
        (headerRow(payload)[4]):dataRow[separator[3] + 1 to separator[4] - 1],
        (headerRow(payload)[5]):dataRow[separator[4] + 1 to separator[5] - 1],
        (headerRow(payload)[6]):dataRow[separator[5] + 1 to separator[6] - 1],
        (headerRow(payload)[7]):dataRow[separator[6] + 1 to separator[7] - 1],
        (headerRow(payload)[8]):dataRow[separator[7] + 1 to -1]

    }

}