如何处理Haskell中的CSV记录字段列表

Question

The goal is to read in a text file and convert it to a csv. The input text file lines will always have some fields, but there will be other fields that appear zero, 1, or more times. The problem is how to handle those fields with varying numbers of items.

Example of the problem:

I can parse the text file to get a list of "events" that might look like the following where each of the data constructors if for a particular type TrialEvents:

For instance,

trialRecord1 = [ Trial {time = 123, trialNum = 1}
               , Efix {eye ='R' ,start = 123, stop = 234, x = 222, y = 123}
               , RewStart {time = 234}, RewEnd {time = 345} ]

trialRecord2 = [ Trial {time = 123, trialNum = 1}
               , RewStart {time = 234}, RewEnd {time = 345} ]

trialRecord3 = [ Trial {time = 123, trialNum = 1}
               , Efix {eye ='R' ,start = 123, stop = 234, x = 222, y = 123}
               , Efix {eye ='R' ,start = 223, stop = 334, x = 100, y = 222}
               , RewStart {time = 234}, RewEnd {time = 345} ]

These lists of events, one for each trial, will always have a trial number and time, but may have 0, 1, or more of other values, such as Efix in this example.

My confusion is about how to generate a csv file from such data where I can have column heads such as trialTime, trialNumber, fixationStartTime, ... , rewStartTime, and RewEndTime. To be able to write the fixationStartTimes I thought I could use a list that could be empty, have one value, or have multiple values.

But when using the cassava package and writing my own instance of ToField (just testing with a list of integers) with pack $ show I noticed I get quotes and escapes characters when the list is greater than length 1, but not for 1 or zero length lists.

"123,234,\"[1,2]\",345,456\r\n"
"123,234,[1],345,456\r\n"

This will present problems when I try to read the csv file into an analysis program and I will have to handle these varying cases.

Can anyone advise how I might approach this issue of a variable length list in a conversion to csv and how to make it maximally friendly on reading the CSV into another environment like R?

Thanks.

Answer 1

Can anyone advise how I might approach this issue of a variable length list in a conversion to csv and how to make it maximally friendly on reading the CSV into another environment like R?

Since this seems to be the core question, and the remainder of the post seems to be an XY problem that assumes CSV as the preferred format, here's an isolated answer to that:

How about using JSON instead? You can write your data type to a JSON structure that is isomorphic to the one you've already got, and R supports JSON through jsonlite . Then you can have your variable-length lists in R without needing to encode them into a column layout (and back again?).

---

If, however, you'd prefer to have that column layout, here's an answer to that:

events will always have a trial number and time, but may have 0, 1, or more of other values, such as Efix [, RewStart and RewEnd [?] ] in this example.

Then having

data Event = Trial { time :: Int, trialNum :: Int }
           | Efix { eye :: Char, start :: Int, stop :: Int, x :: Int, y :: Int }
           | RewStart { time :: Int }
           | RewEnd { time :: Int }

type Events = [Event]

does not seem to quite model what you're saying. How about, instead,

data Event p = Trial { time :: Int, trialNum :: Int, points :: [p] }
data Point = Efix { eye :: Char, start :: Int, stop :: Int, x :: Int, y :: Int }
           | RewStart { time :: Int }
           | RewEnd { time :: Int }

type Events = [Event Point]

Then your records would look like

trialRecord1 = Trial { time = 123, trialNum = 1, points =
                 [ Efix { eye = 'R', start = 123, stop = 234, x = 222, y = 123 }
                 , RewStart { time = 234 }
                 , RewEnd { time = 345 } ] }

trialRecord2 = Trial { time = 123, trialNum = 1, points =
                 [ RewStart { time = 234 }
                 , RewEnd { time = 345 } ] }

trialRecord3 = Trial { time = 123, trialNum = 1, points =
                 [ Efix { eye = 'R', start = 123, stop = 234, x = 222, y = 123 }
                 , Efix { eye = 'R', start = 223, stop = 334, x = 100, y = 222 }
                 , RewStart { time = 234 }
                 , RewEnd { time = 345 } ] }

How do I generate a csv file from such data where I can have column heads such as trialTime , trialNum , fixationStartTime , ..., rewStartTime , and rewEndTime .

Since you can only be sure of trialTime and trialNum , those are the only two columns you can hardcode. The rest of the columns have to be indented according to what points are present in the other events. For example, rendering trialRecord1 , trialRecord2 and trialRecord3 in a table layout should (probably?) give something like

+-----------+-----------+----------+--------------+--------------------+-------------------+------------+------------+--------------+--------------------+-------------------+------------+------------+---------------+-------------+
| recordNum | trialTime | trialNum | fixationEye1 | fixationStartTime1 | fixationStopTime1 | fixationX1 | fixationY1 | fixationEye2 | fixationStartTime2 | fixationStopTime2 | fixationX2 | fixationY2 | rewStartTime1 | rewEndTime1 |
+-----------+-----------+----------+--------------+--------------------+-------------------+------------+------------+--------------+--------------------+-------------------+------------+------------+---------------+-------------+
|         1 |       123 |        1 |            R |                123 |               234 |        222 |        123 |              |                    |                   |            |            |           234 |         345 |
|         2 |       123 |        1 |              |                    |                   |            |            |              |                    |                   |            |            |           234 |         345 |
|         3 |       123 |        1 |            R |                123 |               234 |        222 |        123 |            R |                223 |               334 |        100 |        222 |           234 |         345 |
+-----------+-----------+----------+--------------+--------------------+-------------------+------------+------------+--------------+--------------------+-------------------+------------+------------+---------------+-------------+

You could write a function align :: [Event Point] -> [Event (Maybe Point)] that inserts Nothing s in case of missing data points. (A Nothing may correspond to a variable amount of rows depending on what Point is being translated to columns, so you could also consider a function of type [Event Point] -> [Event (Either NumEmptyColumns Point)] where type NumEmptyColumns = Int .)

Running align [ trialRecord1, trialRecord2, trialRecord3 ] could then give the value

[ Trial { time = 123, trialNum = 1, points =
    [ Just $ Efix { eye ='R', start = 123, stop = 234, x = 222, y = 123 }
    , Nothing
    , Just $ RewStart { time = 234 }
    , Just $ RewEnd { time = 345 } ] }

, Trial { time = 123, trialNum = 1, points =
    [ Nothing
    , Nothing
    , Just $ RewStart { time = 234 }
    , Just $ RewEnd { time = 345 } ] }

, Trial { time = 123, trialNum = 1, points =
    [ Just $ Efix { eye ='R', start = 123, stop = 234, x = 222, y = 123 }
    , Just $ Efix { eye ='R', start = 223, stop = 334, x = 100, y = 222 }
    , Just $ RewStart { time = 234 }
    , Just $ RewEnd { time = 345 } ] }
]

Turning this regular (non-jagged) list of lists into a regular csv should be more straight-forward.

with pack $ show I noticed I get quotes and escapes characters when the list is greater than length 1

 "123,234,\\"[1,2]\\",345,456\\r\\n" 

As @DarthFennec says, this is because the value [1,2] contains a comma which is a special character in your csv. The only kind of escape character you have here is the " s -- the \\ s are Haskell escape codes to show a string that contains a quote:

GHCi> putStrLn "123,234,\"[1,2]\",345,456\r\n"
123,234,"[1,2]",345,456

This is how the string actually looks like.

But having multiple values in a Haskell-syntax list literal in a csv file is probably not "maximally friendly". If you're doing that, then perhaps JSON is a better alternative.