How can I get the json object which represents a Yahoo! pipe

Question

It seems that Yahoo pipes are represented using JSON. I want to download these JSON objects for some research purpose. Usually a Yahoo pipe is rendered in a browser editor thru a url like this: http://pipes.yahoo.com/pipes/pipe.edit?_id=XgRo96h13BGtJWvS8SvLAg , but you can't get the corresponding JSON object to this Yahoo pipe. Does anyone know how to get JSON objects representing Yahoo pipes and store them in any persistent form?

Answer 1

It is possible to get hold of a JSON description of a Yahoo Pipe using a URL of the form:

http://pipes.yahoo.com/pipes/pipe.info?_out=json&_id=PIPE_ID

The pipe2py python library demonstrates how to grab the JSON description of a pipe and "compile" it to a Python equivalent that can be run on your own server.

The post Exporting Yahoo Pipe Definitions, Compiling Them to Python, and Running Them in Scraperwiki describes how you can use pipe2py in the Scraperwiki environment to compile and execute pipes on Scraperwiki using pipe definitions imported directly from Yahoo Pipes, or exported from Yahoo Pipes and then stored locally in a Scraperwiki database table.

Answer 2

Use pipe.run?[your pipe id here]&_render=json as opposed to pipe.edit

So in your case to get the json it would be - http://pipes.yahoo.com/pipes/pipe.run?_id=XgRo96h13BGtJWvS8SvLAg&_render=json

I guess how you implement the client is dependent on what you like writing in/what other functionality you need.

You could also do it the other way around and use the web service service module to post the data to a script that can extract the json and persist it to a database. You could check out json.org .

Answer 3

When I load that page in a browser I can see that it makes an ajax request for: http://pipes.yahoo.com/pipes/ajax.pipe.load?id=XgRo96h13BGtJWvS8SvLAg&_out=json&modinfo=true&rnd=7560&.crumb=MjvGjpzhPLl

That's your object but I'm not sure if I'm answering your question of how to "get it". If you need to get it through a program you would need a script that loges into pipes and extracts that url.

Answer 4

A quick way, while not automated, is to use an HTTP analyzer. Here's a process for getting the object using HttpFox (I use v0.8.9) for Firefox. With the analyzer running, load the edit page for a pipe, like the one you linked:

http://pipes.yahoo.com/pipes/pipe.edit?_id=XgRo96h13BGtJWvS8SvLAg

Look at the request with a URL that starts with:

http://pipes.yahoo.com/pipes/ajax.pipe.load?id= ....

Next, explore the content of the request (there's a 'Content' tab in HttpFox). That's the JSON object representing the pipe structure.