A file format writable by python, readable as a Dataframe in Spark
Question
I have python scripts (no Spark here) producing some data files, that I want to be readable easily as Dataframes in a scala/spark application.
What's the best choice ?
1 answers
solution1
1 2015-06-30 09:12:47
If your data doesn't have newlines in then a simple text-based format such as TSV is probably best.
If you need to include binary data then a separated format like protobuf makes sense - anything for which a hadoop InputFormat exists should be fine.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
Convert any file (Readable as Text File) to Excel Format (.xlsx) virtually then convert as dataframe -Python
User-readable file format for python list of lists
Decode pickle file in human-readable format in python
Cleaning up a messy data file to a more readable format in Python?
Colossal Cave Adventure File: Decode From Binary (?) to readable format (Python)
pickle a dictonary in python in a readable format
Write spark dataframe to file using python and '|' delimiter
How to format a timestamp in python to a readable format?
Output python into python-readable format
Spark Dataframe-Python
Related Tags