How to connect from server-based Jupyter Notebook to service in localhost?

Question

Note: This is the reverse of what most StackOverflow users ask

I have a Jupyter notebook running on a remote server (via a browser) and a test service running on my localhost . I'd like to call the local service from the remote server.

The problem is that connecting to localhost when running on the remote server connects to the remote server's localhost, not my own PC (in which the browser is running).

I'm expecting that maybe I'd have to drop into Javascript to do this?? I'm guessing this would be an AJAX-like call, with a callback involved. I think this complicates things somewhat. Is there a simpler approach?

Does anyone have any experience with this?? Maybe some sample code??

Answer 1

all --

OK ... I have tried this every which way, and have gotten a decent education along the way. The bottom line is that it is possible to call a localhost-based service from a Jupyter Notebook (aka IPython) running on a remote server, and it may also be possible to get a result back so it can be used in Python (though, with restrictions). This is important because it enables biological workflows to call the REST-based automation interface of the Cytoscape desktop application.

I'm writing this up so successor-me can see this and profit, or give up. I have been helped by tons of other StackOverlow and free standing articles ... too many to mention, most prominently:

ipython-notebook-javascript-python-communication

Execute Discussion

workaround-for-wrapping-a-js-function-in-python-in-jupyter-notebook

There are three parts to this problem:

• Execute an HTTP operation (... suppose GET, but POST, PUT, etc work, too)
• Get the HTTP status and result back
• Use the result in Python

There are multiple challenges:

• Pass a URL and payload (for PUT/POST) into the HTTP caller
• Return to caller only when an HTTP result is available (and not before)
• Make the HTTP result available via a Python variable

A major problem exists because Python is a traditional synchronous language executing (partially) in a browser-based (IPython) environment. Normally, this isn't a problem, as the remote Jupyter server executes a full cell of Python and then another cell. The problem arises because the HTTP call must be in Javascript (ie, in a %%js or %%javascript cell or as an argument to an IPython.display.HTML() call), which is executed in the local browser (instead of the remote server). A base principal of browser-based Javascript is that long (eg, HTTP) operations should be done asynchronously (eg, Fetch() using Promises, generators using Yield(), or normal XMLHttpRequest calls) to maintain highly interactive browser performance. For a workflow waiting on an answer, asynchronous execution would put a poor bioinformatics Python programmer into shock via callback hell. Additionally, such workflows are expected to be compute-heavy, and interactivity isn't expected ... causality must be preserved.

In Javascript, the gist of the solution is to use synchronous XMLHttpRequest calls. This gets a synchronous answer. Next, use the undocumented IPython.notebook.kernel.execute() to store a value into a Python variable on the remote server. This is an asynchronous operation, and must be done last. (You can see the source to execute() only by using the Development mode in a Chrome browser: F12->static/notebook/js/services/kernels/kernel.js, and find kernel.js/Kernel.prototype.execute). If this is done at the end of a cell, IPython will pause while it gets the next cell ready or waits for user input. During either of these times, it appears that the Python kernel services the statement queue created by execute(). If that's what's really happening, it should be safe to expect that the Python variable containing the HTTP result will be ready to read in the follow-on cell. It's a fair bet it won't be ready if the execute() and Python variable fetch are performed in the same cell.

A synchronous XMLHttpRequest call is created when you pass "false" to the XMLHttpRequest.open() function. Some browsers may complain about this, as they think that synchronous calls are evil and are deprecated. Best to use a browser (eg, Chrome) that doesn't complain.

Weirdness: If you call HTML() to execute the Javascript, it should be the last call in a cell; otherwise, the HTML() call will fail. If you wrap the HTML() call in a IPython.display.display() call, you can put additional Python statements after it in the same cell ... just don't look for the HTTP result variable to have been set until IPython transitions to the next cell. Note that workarounds like Timeout() are ineffective here, as it doesn't appear to force processing of the Execute() statement queue, so you really do have to wait until the next cell before accessing the HTTP result.

This Javascript cell creates the Javascript functions that will do the work. Note the Cy* naming convention, which makes collision with IPython browser code unlikely.

%%js

function CyHttpGet(url) {
  var xhr = new XMLHttpRequest()
  xhr.open('GET', url, false)
  xhr.send(null);
  if (xhr.status === 200) {
    return xhr.responseText;
  }
  else {
    return "bad news:" + xhr.status
  }
}

function CySetKernelVar(varValue) {
  varValue = varValue.replace(/'/g, "/'");  // Double any single quotes
        
  var command = "httpResult = '"+ varValue + "'";
        
  var kernel = IPython.notebook.kernel;
  kernel.execute(command);
}

This Python cell makes the actual HTTP call:

from IPython.display import HTML

javascript = """
<script type="text/Javascript">

// This is what actually executes the HTTP and sets the result as a Python variable
CySetKernelVar(CyHttpGet('http://localhost:1234'));

</script>
"""

HTML(javascript)

This Python cell makes use of the HTTP result (and must not be part of the previous cell):

print(httpResult)

httpResult='xxx' # clear variable so we're not fooled if the Javascript code doesn't do what we thought it would

Note that the ability to pass a value from Javascript to Python via Execute() really does depend on the Python kernel clearing the Execute queue between cells. If this isn't really happening, we would expect to see "xxx" in the print(httpResult) from time to time - very bad news.

Note that some web posts suppose that a value can be passed from Javascript to Python by having the Javascript store the value as the innerHTML of a DIV statement, and then having an HTML() or Javascript() function read the innerHTML and return it. I haven't gotten this to work, but if it can work, the Execute() may not be necessary.