Python JSON API for linked data, with flat files

Question

We're creating gamma-cat , an open data collection for gamma-ray astronomy, and are looking for advice (here, or links to resources, formats, tools, packages) how to best set it up.

The data we have consists of measurements for different sources, from different papers. It's pretty heterogeneous, sometimes there's data for multiple sources in one paper, for each source there's usually several papers, sometimes there's no spectrum, sometimes one, sometimes many, ...

Currently we just collect the data in an input folder as YAML and CSV files, and now we'd like to expose it to users. Mainly access from Python, but also from Javascript and accessible from a static website.

The question is what format and organisation we should use for the data, and if there's any Python packages that will help us generate the output files as a set of linked data, as well as Python and Javascript packages that will help us access it?

We would like to get multiple "views" or simple "queries" of the data, eg "list of all sources", "list of all papers", "list of all spectra for source X", "spectrum A from paper B for source C".

For format, probably JSON would be a good choice? Although YAML is a bit nicer to read, and it's possible to have comments and ordered maps. We're storing the output files in a git repo, and have had a lot of meaningless diffs for JSON files because key order changes all the time.

To make the datasets discoverable and linked, I don't know what to use. I found eg http://jsonapi.org/ but that seems to be for REST APIs, not for just a series of flat JSON files on a static webserver? Maybe it could still be used that way? I also found http://json-ld.org/ which looks relevant, but also pretty complex. Would either of those or something else be a good choice?

And finally, we'd like to generate the linked and discoverable files in output from just a bunch of somewhat organised YAML and CSV files in input using Python scripts. So far we just wrote a bunch of Python classes or scripts based on Python dicts / lists and YAML / JSON files. Is there a Python package that would help with that task of generating the linked data files?

Apologies for the long and complex question! I hope it's still in scope for SO and someone will have some advice to share.

Answer 1

Judging from the breadth of your question, you are new to linked data. The least "strange" format for you might be the Data Package . In the most common case it's just a zip archive of a CSV file and JSON metadata. It has a Python package .

If you have queries to the data, you should settle for a database (triplestore) with a SPARQL endpoint. Take a look at Fuseki . You can then use Turtle or RDF/XML for file export.

If the data comes from some kind of a tool, you can model the domain it represents using Eclipse Lyo ( tutorial ).

These tools are maintained by 3 different communities, you can reach out to their user mailing lists separately if you have further questions about them.