基于XML文件（PHP，Jquery）的页面生成

Question

So I have XML data that I need to take and use in my PHP page, as search results.
Problem - the XML is structured horribly (one "index" file with links that lead to other XML files) and the size of the whole thing is huuuge (from 1000 to 20+K XML files, 10 and more MBs in total).
There are lots of different tools I researched into: XMLreader, XML Parser and a bit of JQuery. But I'm not sure which one will be better for that task.
What, I think would be the best way of solving it is a facebook-style "press to load more" kind of page that loads itself, loads the "index" XML (maybe in a hidden input field or a div so it can be read by JQuery), then starts actively reading XML files that are listed in the index and generate the results dynamically on the page. And I do need all the data in some sort of memory, since I will have to do analytics on it as well.
Question: Which is better to use for it and any techniques I'd benefit from? Or am I looking at it from a wrong side completely?

I tried strait PHP reading using XMLreader and parser, as well as SimpleXMLElement + for=loop, but once I put a second read (from the "Index") into the equation the page just breaks from too big of loading times, and that's with 30MB/s internet. I don't have much experience with JQuery, so that's why I'm asking for advice.


PS I'm taking XML from http://www.clinicaltrials.gov
Example of a smaller "Index": http://www.clinicaltrials.gov/search?term=attack&count=1856&displayxml=true
If you add "?displayxml=true" to each of the "url"s there it will be an XML file that I need to read.

Answer 1

What I would do is:

Since the site provides some helpful query strings like &count= , take this advantage.

This means you really don't need to process and query tens of thousands of rows.

So normally, you just query the external site like this:

http://www.clinicaltrials.gov/search?term=heart%20attack&count=10&displayxml=true&pg=1

So just limit every request. Example: 10 at a time.

Then start building the server side.

The client site is up to you to decide, this is just personal preference but I would use DataTables in this example.

The code below just recreates the same structure in the sample url above:

$search_term = 'attack';
$count = 10;
$query = http_build_query(array(
    'term' => $search_term,
    'count' => $count,
    'displayxml' => 'true',
    'pg' => $draw,
));
$main_url = 'http://www.clinicaltrials.gov/search?' . $query;

After building the correct URL, just request the XML needed. An then ultimately, after you have gathered all the data you need (the chunked data). Present it on the client-side.

Full example: Also Sample Fiddle

index.php

if($_SERVER['REQUEST_METHOD'] == 'POST') {
    $draw = isset($_POST['draw']) ? $_POST['draw'] : 1;
    $search_term = 'attack';
    $count = 10;
    $query = http_build_query(array(
        'term' => $search_term,
        'count' => $count,
        'displayxml' => 'true',
        'pg' => $draw,
    ));
    $main_url = 'http://www.clinicaltrials.gov/search?' . $query;
    $contents = file_get_contents($main_url);
    $xml = simplexml_load_string($contents);
    $total_results = (string) $xml->attributes()['count'];
    $data = array();

    $data['draw'] = $draw;
    $data['recordsTotal'] = $total_results;
    $data['recordsFiltered'] = $total_results;
    foreach($xml->clinical_study as $entry) {
        $data['data'][] = json_decode(json_encode($entry), true);

    }

    echo json_encode($data);
    exit;
}

?>

<link rel="stylesheet" type="text/css" href="http://cdn.datatables.net/1.10.2/css/jquery.dataTables.css" />
<table border="1" class="display dataTable" cellspacing="0" width="100%">
    <thead>
        <tr>
            <th>Order</th>
            <th>Score</th>
            <th>Nct ID</th>
            <th>URL</th>
            <th>Title</th>
            <th>Status</th>
            <th>Condition Summary</th>
            <th>Last Changed</th>
        </tr>
    </thead>
    <tfoot>
        <tr>
            <th>Order</th>
            <th>Score</th>
            <th>Nct ID</th>
            <th>URL</th>
            <th>Title</th>
            <th>Status</th>
            <th>Condition Summary</th>
            <th>Last Changed</th>
        </tr>
    </tfoot>
</table>

<script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
<script src="http://cdn.datatables.net/1.10.2/js/jquery.dataTables.min.js"></script>
<script type="text/javascript">
$(document).ready(function(){

    $('.display').dataTable({
        'processing': true,
        'serverSide': true,
        'ajax': {
            'url': document.URL,
            'type': 'POST',
        },
        "columns": [
            { "data": "order" },
            { "data": "score" },
            { "data": "nct_id" },
            { "data": "url" },
            { "data": "title" },
            { "data": "status" },
            { "data": "condition_summary" },
            { "data": "last_changed" },
        ],
        bFilter: false, bInfo: false, bSort: false,
    });

});
</script>

So the basic idea here really is that you really don't need to request that thousand rows immediately. You can just call them in chunks instead.

Answer 2

You can use XSL to take the clinicaltrials.gov XML and convert it into a sane XML format, including HTML. XSL is a language for transforming XML.

PHP even has a built in XSL processor: http://php.net/manual/en/book.xsl.php

On a side note, I use XSL to convert DocBook XML files (a semantic markup language) into Twitter Bootstrap HTML.

For example, using the example you've provided ( http://www.clinicaltrials.gov/search?term=attack&count=1856&displayxml=true ), if you wanted to display the titles of all of the clinical studies as a list, the following XSL stylesheet would do the job:

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output method="xml" omit-xml-declaration="yes" indent="no"/>
  <xsl:template match="/">
    <ul>
      <xsl:apply-templates/>
    </ul>
  </xsl:template>
  <xsl:template match="search_results">
    <xsl:apply-templates/>
  </xsl:template>
  <xsl:template match="clinical_study">
    <li><xsl:value-of select="title"/></li>
  </xsl:template>
</xsl:stylesheet>

The XSL stylesheet enters the source XML document at the root of the document. It then traverses the tree. Anytime that it finds an element which matches a defined template, it executes that template. Pretty cool stuff! It takes a while to orient yourself into the XSL paradigm of programming, but it is quite powerful once you get the hang of it.

Note that I just wrote that as a toy example off the top of my head. I'm not sure if that will actually execute properly.

Edit 1:

(OP asks about performing analysis, eg counting all elements of a specific type)

Looking at your example XML results, it looks like the only way to determine if a trial is in Phase 3 is to check the text of the <title> element. This is still easily within the capabilities of XSL (with some help from XPath).

<xsl:variable name="countPhase3">
   <xsl:value-of select="count(//title[text() = 'Phase 3' | text() = 'Phase III']"/>
</xsl:variable>

I'll warn you again that this is just an example off the top of my head.