How to yield from an async callback called from a generator function?

Question

I'm just learning node.js generators, and I'd like to have a generator yield a value from the callback of an asynchronous method called by the generator. The following code is my current example. The first generator (get_urls) returns a series of URLs for some pages for a website. It works fine.

The second generator should use unirest to retrieve each of those web pages so the generator can return the page contents. However, I can't figure out how to yield from the callback of the unirest.end() function.

There are probably libraries that do this more comprehensively (and with better error handling), but I'm trying to understand the lowest level so I can get more out of the libraries when I get smart enough to use them.

#!/usr/local/bin/node
// -*- mode: Javascript; -*-

var unirest = require( 'unirest' );

for (var url of get_urls( 'http://www.example.com/generated-page?n=1' ))
    console.log( 'Get url', url );

for (var page of get_pages( 'http://www.example.com/generated-page?n=1' ))
    console.log( 'Got page', page );

function* get_urls( url ) {
    do {
        yield url;
        var rx = url.match( /^(.*?n=)([0-9]+)$/ );
        if (rx) {
            if (rx[2] >= 3) break;
            url = rx[1] + (parseInt(rx[2]) + 1).toString( );
        }
    } while (rx);
}

function* get_pages( url ) {
    do {
// *** This is what I want to do, but it's not the right way to do it! ***
//      unirest.get( url ).end( function (rsp) { yield rsp; } );
        var rx = url.match( /^(.*?n=)([0-9]+)$/ );
        if (rx) {
            if (rx[2] >= 3) break;
            url = rx[1] + (parseInt(rx[2]) + 1).toString( );
        }
    } while (rx);
}

Answer 1

yield can only yield to the generator function* that immediately contains it. Therefore, you can't yield from a callback.

What you can do from a callback is resolve a Promise, which your generator function can yield.

Your answer has quite a bit of extraneous code (plus why use unirest when there's native Fetch etc.), so here's a sample (from another answer of mine) that illustrates the concept of yielding a promise repeatedly:

 async function* fetchUrls() { let i = 0; while (i < 10) yield new Promise((resolve, reject) => { setTimeout(() => resolve(i++), 200); }); } (async function main() { // for-await-of syntax for await (const result of fetchUrls()) console.log(result); }()); 

The new for-await-of syntax has been available since Node v9.2, and can be used in Node v10 or later without any flags.