在F＃中使用FileHelperAsyncEngine

Question

I am trying to load rows from a csv files to an Elasticsearch database in f# using FileHelpers to read the csv. Everything is working for small test files with the code snippet below reading all records at once

let readRows<'T>(filePath:string) =
    let engine = FileHelperEngine(typeof<'T>)

    engine.ReadFile(filePath)
    |> Array.map (fun row -> row :?> 'T)

Unfortunately it needs to be able read larger files, of which many columns are discarded later, line by line. The function FileHelperAsyncEngine.BeginReadFile returns an IDisposable.

let readRowsAsync<'T>(filePath:string) =
    let engine = new FileHelperAsyncEngine(typeof<'T>)

    engine.BeginReadFile(filePath:string)
    |> ...

How can I further process this object to an Array of <'T>s?

Answer 1

According to the documentation , after you call BeginReadFile , the engine itself becomes an enumerable sequence over which you can iterate (which is a very strange design decision). So you can just build your own sequence on top of it:

let readRowsAsync<'T>(filePath:string) = 
  seq {
    let engine = new FileHelperAsyncEngine(typeof<'T>)
    use disposable = engine.BeginReadFile(filePath)

    for r in engine do
      if not (shouldDiscard r) then yield (map r)
  }

Note that I'm using the use binding, rather than let . This will ensure that the disposable is disposed after the sequence ends or the consumer stops iterating over it.

Note that the following will not work, even though it will compile:

let readRowsAsync<'T>(filePath:string) = 
  let engine = new FileHelperAsyncEngine(typeof<'T>)
  use disposable = engine.BeginReadFile(filePath)

  engine |> Seq.filter (not << shouldDiscard) |> Seq.map map

If you do it this way, the disposable will be disposed after the function returns, but before the resulting enumeration is iterated over, thus closing the file before its time. To ensure that the disposable is correctly disposed, you must enclose the whole thing in a seq expression.

If you really want to use Seq.filter / Seq.map instead of for / yield , you can still do this, but inside the seq expression, like this:

let readRowsAsync<'T>(filePath:string) = 
  seq {
    let engine = new FileHelperAsyncEngine(typeof<'T>)
    use disposable = engine.BeginReadFile(filePath)

    yield! engine |> Seq.filter (not << shouldDiscard) |> Seq.map map
  }

You can also bring the filtering and mapping out of the seq expression (which would make your function more reusable), but the seq expression itself must remain in place, because it controls the disposing part:

let readRowsAsync<'T>(filePath:string) = 
  seq {
    let engine = new FileHelperAsyncEngine(typeof<'T>)
    use disposable = engine.BeginReadFile(filePath)

    yield! engine
  }

let results = 
  readRowsAsync<SomeType>( "someFile.txt" )
  |> Seq.filter (not << shouldDiscard) 
  |> Seq.map map

Finally, it must be noted that you should be careful handling this sequence, because it's holding on to an unmanaged resource (ie open file): don't hold it open for a long time, don't employ blocking operations while processing it, etc.