BigQuery + PHP客户端中的令牌分页不起作用

Question

I'm trying to implement a simple pagination mechanism with BigQuery.

this is my query function:

function query($sql, $max_results = null, $page_token = null) {
    $request = new Google_Service_Bigquery_QueryRequest();
    $request->setQuery($sql);
    $response = $this->service->jobs->query(PROJECT_ID, $request);
    $job_id = $response->getJobReference()->getJobId();

    $optParams = ($max_results) ? array(
        'pageToken' => $page_token,
        'maxResults' => $max_results,
    ) : array();
    $response = $this->service->jobs->getQueryResults(PROJECT_ID, $job_id, $optParams);
    if (!$response->getJobComplete()) {
        return null;
    }
    $rowsJson = $this->rowsJson($response->getRows());
    if ($max_results) {
        return array(
            "rows" => $rowsJson,
            "token" => $response->getPageToken()
        );
    }
    return $rowsJson;
}

query("select url, CEIL(AVG(total)) as avg, count(id) as count from $table_id " .
      "where created > $date_start and created < $date_end group by url order by $order_by desc",
      10, $page_token
);

The first query seems like it's working. it returns only 10 results and a token (there are definitely more than 10 results). But when I call the function later with the token I got, the results are empty... I have no idea why. I see no error message in the logs or anything...

Answer 1

In order for BigQuery "pagination" to work you shold follow below steps (simplified but enough for this answer):

1. Execute query job
2. Retrieve jobid
3. Execute getQueryResults with that jobid and retrieve result and page_token
4. If page_token null - you done, otherwise continue with next step
5. Execute getQueryResults ( still with jobid from #2 ) and retrieve result and page_token
6. Jump to step #4

Now, you most likely will see that your code doesn't follow this and each time executes query as new job - which totally breaks condition highlighted in #5 ( still with jobid from #2 ).

The reason why first time it returns result is that first call actually meet above steps, but in consequitive calls you actually passing page_token and force execution of NEW job with some arbitrary (from that new job prospective) page_token
And also, this explains why your current code works with startIndex

Try to rewrite your code to follow above steps and it should work

Also,
PS page_token is alive and should be reusable for as long as underlying temp table (that holds your result) is alive - which is approximately 24 hours

Answer 2

Solved it by using startIndex instead of tokens.

Maybe tokens are short-lived and don't survive the 4-5 seconds delay, I'm not really sure.

Answer 3

Using a page token to get paged results are recommended because page token provides a view of the table at the first time it is retrieved. By just using startIndex, you may see different results as the table may change.

Response size is what may cause you get less than maxResults. We are limited by Apiary to return results of certain size. https://cloud.google.com/bigquery/docs/data#paging And we also have a max field counts to be less than 350,000 limit per call.

You should try to get the next results by using the page token it passed back to you.

Answer 4

function get_quote($page = 1){
global $database_connection;
    $start = 0;
    $limit = 5;
    $start = ($page - 1) * $limit;

    //select from database with limit $start, $limit
    //display results

    $sql_get_quote =  "SELECT * FROM quote_table ORDER BY date_added ASC LIMIT $start, $limit";
    $query_get_quote = mysqli_query($database_connection, $sql_get_quote);
    while($fetch_quote = mysqli_fetch_array($query_get_quote)){
        echo "<br>
        <div class='w3-container w3-pale-green w3-bottombar w3-border-green w3-border'>
        <p>".$fetch_quote['quote']."</p><hr><p class='w3-left-align'>".$fetch_quote['quote_by']."</p>
        </div>";
    }

    //do another query, get the num_rows
    $quote_rows= mysqli_num_rows (mysqli_query($database_connection ,"SELECT * FROM quote_table"));
    $total = ceil($quote_rows / $limit);
if (isset($page)){
    echo "<div class='w3-container'>";
    echo "<ul class='pager'>";
    if($page > 1) {
        echo "<a href='?page=".($page - 1)."' style='float: left;' class=' w3-sand w3-btn w3-margin w3-round'>Previous</a>";
    }
    for($i = 1; $i <= $total; $i++) {
        if($i == $page) { echo "<li class='active current'>".$i."</li>"; }

        else { echo "<li><a href='?page=".$i."'>".$i."</a></li>"; }
    }
    if($page != $total) {
        echo "<a href='?page=".($page + 1)."' class='w3-btn w3-margin w3-sand w3-round'>Next</a>";
    }
    echo "</ul></div>";
}

}

This works for me... Hope you can build around this