Apache Arrow or feather plugin?

Question

I'd like to use local feather-files as a sources in Intake. Is the plugin for feather/arrow not yet existing or am I missing something?

Answer 1

You are right, there is not currently a feather or arrow (ie, framed buffers) driver for Intake, as far as I know. Neither is supported by Dask either.

Given that pandas does explicitly support feather, it would be easy to build a driver for Intake, supporting multiple remote files, and these could even be loaded in parallel with Dask, without adding code to dask.

However, first I'd like to ask: why not parquet? That seems to be the standard format, at least partly because its reach goes well beyond python/arrow. That format is supported by Intake and any other tabular data engine.