模仿Twitter的流API

Question

Twitter implements a streaming API for serving tweets one at a time as JSON blobs. Further, libraries such as Python's requests can consume from it like so,

import json
import requests

r = requests.get('http://httpbin.org/stream/20', stream=True)

for line in r.iter_lines():

    # filter out keep-alive new lines
    if line:
        print json.loads(line)

How might I implement, server-side, something that can be consumed the same way? Ideally, using something like bottle ?

Answer 1

One can aim to make a module that rolls streams of data based on a uri - without making it RESTful that is.

For example //site.org/stream/40/

Theres several ways to communicate with an HTTP library like curl or requests. A basic approach would be to accept the recipient's POST data, store it in a database with a unique id and wait for a response. And you can use a javascript's xhr to find out if the response is ready.

All in all on the backend it should be a python module whether you use django, bottle, etc. is your call.

Hope that helps.

Answer 2

server.py

import gevent.monkey; gevent.monkey.patch_all()

import itertools
import json
import random
import time

import bottle


@bottle.get("/foo")
def foo():
  for i in range(5):
    o = {"i": i, "r": random.random()}
    print 'sending: {}'.format(o)
    yield json.dumps(o) + "\n"
    time.sleep(1)


if __name__ == '__main__':
  bottle.run(
    host="localhost",
    port=1234,
    server="gevent"
  )

client.py

import json
import pprint
import sys

import requests


def main(url):
  print 'Reading from {}'.format(url)
  for line in requests.get(url, stream=True).iter_lines(chunk_size=1):
    pprint.pprint(json.loads(line))
  print 'All done.'


if __name__ == '__main__':
  main(sys.argv[1])

usage,

$ python server.py &
$ python client.py http://localhost:1234/foo