删除多个文件

Question

The following code is working but extremely slow. Up till the search function all goes well. First, the search function returns a sequence and not an array (why?!). Second, the array consists of nodes and I need URI's for the delete. And third, the deleteDocument function takes a string and not an array of URI's.

What would be the better way to do this? I need to delete year+ old documents.

Here I use xdmp.log in stead of document.delete just te be safe.

var now      = new Date();
var yearBack = now.setDate(now.getDate() - 365); 

var date = new Date(yearBack);
var b    = cts.jsonPropertyRangeQuery("Dtm", "<", date);
var c    = cts.search(b, ['unfiltered']).toArray();

for (i=0; i<fn.count(c); i++) {
  xdmp.log(fn.documentUri(c[i]), "info");
};

Answer 1

Doing the same with cts.uris :

var now      = new Date();
var yearBack = now.setDate(now.getDate() - 365);

var date = new Date(yearBack);
var b    = cts.jsonPropertyRangeQuery("Dtm", "<", date);
var c    = cts.uris("", [], b);

while (true) {
    var uri = c.next();

    if (uri.done == true){
        break;
    }

   xdmp.log(uri.value, "info");
 }

HTH!

Answer 2

Using toArray will work but is most likely were your slowness is. The cts.search() function returns an iterator. So All you have to do is loop over it and do your deleting until there is no more items in it. Also You might want to limit your search to 1,000 items. A transaction with a large number of deletes will take a while and might time out.

Here is an example of looping over the iterator

var now      = new Date();
var yearBack = now.setDate(now.getDate() - 365);

var date = new Date(yearBack);
var b    = cts.jsonPropertyRangeQuery("Dtm", "<", date);
var c    = cts.search(b, ['unfiltered']);

while (true) {
    var doc = c.next();

    if (doc.done == true){
        break;
    }

   xdmp.log(fn.documentUri(doc), "info");
 }

here is an example if you wanted to limit to the first 1,000.

fn.subsequence(cts.search(b, ['unfiltered']), 1, 1000);

Answer 3

Several things to consider. 1) If you are searching for the purpose of deleting or anything that doesnt require the document body, using a search that returns URIs instead of nodes can be much faster. If that isnt convenient then getting the URI as close to the search expression can achieve similar results. You want to avoid having the server have to fetch and expand the document just to get the URI to delete it.

2) While there is full coverage in the JavaScript API's for all MarkLogic features, the JavaScript API's are based on the same underlying functions that the XQuery API's use. Its useful to understand that, and take a look at the equivalent XQuery API docs to get the big picture. For example Arrays vs Iterators - If the JS search API's returned Arrays it could be a huge performance problem because the underlying code is based on 'lazy evaluation' of sequences. For example a search could return 1 million rows but if you only look at the first one the server can often avoid accessing the remaining 999,999,999 documents. Similarly, as you iterate only the in scope referenced data needs to be in available. If they had to be put into an array then all results would have to be pre-fetched and put put in memory upfront.

3) Always keep in mind that operations which return lists of things may only be bounded by how big your database is. That is why cts.search() and other functions have built in 'pagination'. You should code for that from the start. By reading the users guides you can get a better understanding of not only how to do something, but how to do it efficiently - or even at all - once your database becomes larger than memory. In general its a good idea to always code for paginated results - it is a lot more efficient and your code will still work just as well after you add 100 docs or a million.

4) take a look at xdmp.nodeUrl https://docs.marklogic.com/xdmp.nodeUri , This function, unlike fn.documentUri(), will work on any node even if its not document node. If you can put this right next to the search instead of next to the delete then the system can optimize much better. The examples in the JavaScript guide are a good start https://docs.marklogic.com/guide/getting-started/javascript#chapter

In your case I suggest something like this to experiment with both pagination and extracting the URIs without having to expand the documents ..

var uris = []
for (var result of fn.subsequence(cts.search( ... ), 1 , 100   )
  uris.push(xdmp.nodeUri(result))

for( i in uris ) 
  xdmp.log( uris[i] )