Liferay 7.3.5GA6 custom index search and ddmFieldArray

Question

I'm trying to develop a custom web content search portlet for Liferay 7.3.5 GA6, using SearchContext, IndexSearcherHelperUtil and all the other stuff.

I have some DDMStructure with different fields and, from what I've seen on the elasticsearch index, this fields are indexed in a nested document, like this:

"ddmFieldArray": [
{
"ddmFieldName": "ddm__text__37702__nome_it_IT",
"ddmValueFieldName": "ddmFieldValueText_it_IT",
"ddmFieldValueText_it_IT": "Nome esempio",
"ddmFieldValueText_it_IT_String_sortable": "nome esempio"
}
,
{
"ddmFieldName": "ddm__text__37702__descrizione_breve_it_IT",
"ddmValueFieldName": "ddmFieldValueText_it_IT",
"ddmFieldValueText_it_IT": "Esempio di descrizione breve da indicizzare",
"ddmFieldValueText_it_IT_String_sortable": "esempio di descrizione breve da indicizzare"
}
]

which is different from the old way I used to know, where the custom fields were indexed like ddm__[keyword/text]__[structure_id]__[field_name]

Now, I understand this different way of indexing is due to an improvement to avoid elastic issues (Limit of total fields has been exceeded) but...After executing the search, there is no ddmFieldArray in the com.liferay.portal.kernel.search.Document .getFields , so I'm unable to get the ddmstructure fields values from the elastic search index.

Here's the code:

long journalArticleClassId = ClassNameLocalServiceUtil.getClassNameId(JournalArticle.class.getName());

SearchContext searchContext = new SearchContext();
searchContext.setClassTypeIds(new long[] {journalArticleClassId});
searchContext.setCompanyId(companyId);
searchContext.setStart(QueryUtil.ALL_POS);
searchContext.setEnd(QueryUtil.ALL_POS);
        
BooleanQuery query = new BooleanQueryImpl();
            
        
MatchQuery approvedQuery = new MatchQuery(Field.STATUS, String.valueOf(WorkflowConstants.STATUS_APPROVED));

query.add(approvedQuery, BooleanClauseOccur.MUST.getName());


Hits resultHits = IndexSearcherHelperUtil.search(searchContext, query);

for (Document doc: resultHits.getDocs()) {          
    doc.getFields().forEach((k, v) -> _log.debug(k)); //No ddm structure field
}

Is this still an improvement or just an unexpected behaviour?

Any way to solve or extend this?

Thanks

Answer 1

The "ddmFieldArray" field is a nested field that it is not returned by default. You have to fetch it from the document source field that is stored in an internal "_source" field in Elasticsearch (see https://www.elastic.co/guide/en/elasticsearch/reference/7.9/search-fields.html )

In order to do this in Liferay, you have to use some search methods that aren't available in the old portal-kernel search classes, you have to use the new search classes that are available in the portal-search-api module that is located in modules/apps/portal-search

These are the changes you have to apply to your code:

• Before executing the search, you have to add the "fetchSource" flag to your search adding this code:

    searchRequestBuilderFactory.builder(
        searchContext
    ).fetchSource(
        true
    ).build();

• After executing the search, you have to get the Document objects from the SearchResponse object. This SearchResponse is available in the searchContext using this code:

    /* Execute search */
    IndexSearcherHelperUtil.search(searchContext, query);
    
    /* Get results from search response */
    SearchResponse searchResponse = searchContext.getAttribute("search.response");
    List<SearchHit> resultHits = searchResponse.getSearchHits().getSearchHits();
    
    /* Iterate */
    for (SearchHit searchHit : resultHits)  {
        Document doc = searchHit.getDocument();
    
        ...your stuff...
    }

You have some examples about fetching source and gettings SearchResponse in following Liferay classes:

• com.liferay.document.library.search.test.DLSearchFixture , see method searchOnlyOneSearchHit where fetchSource(true) is setted and the SearchResponse object is got from the searchContext
• com.liferay.account.internal.retriever.AccountOrganizationRetrieverImpl where some searchs are done using SearchRequest and SearchResponse.
• com.liferay.asset.internal.util.AssetHelperImpl where searchRequestBuilderFactory is used.

I have also implemented a groovy script example that you can execute it from: Control Panel => Server Administration => Script

import com.liferay.registry.*;
import com.liferay.portal.kernel.search.*;
import com.liferay.portal.kernel.search.generic.*;
import com.liferay.portal.search.legacy.searcher.SearchRequestBuilderFactory;
import com.liferay.portal.search.searcher.SearchResponse;
import com.liferay.portal.search.hits.SearchHit;
import com.liferay.portal.search.document.Document;

/* Get SearchRequestBuilderFactory reference using RegistryUtil, because we cannot use "@Reference" in a groovy script */
Registry registry = RegistryUtil.getRegistry();
SearchRequestBuilderFactory searchRequestBuilderFactory = registry.getService(registry.getServiceReference(SearchRequestBuilderFactory.class.getName()));

/* Create SearchContext */
SearchContext searchContext = new SearchContext();
searchContext.setCompanyId(com.liferay.portal.kernel.util.PortalUtil.getCompany(actionRequest).getCompanyId());
searchContext.setStart(-1);
searchContext.setEnd(-1);

/* Line to fetch stored source of documents */
searchRequestBuilderFactory.builder(
    searchContext
).fetchSource(
    true
).build();

/* Get journal articles that are approved (status = 0) */
MatchQuery approvedQuery = new MatchQuery(Field.STATUS, String.valueOf(0));
MatchQuery journalArticleQuery = new MatchQuery("entryClassName", com.liferay.journal.model.JournalArticle.class.getName());

BooleanQuery query = new BooleanQueryImpl();
query.add(approvedQuery, BooleanClauseOccur.MUST.getName());
query.add(journalArticleQuery, BooleanClauseOccur.MUST.getName());

/* Execute search */
IndexSearcherHelperUtil.search(searchContext, query);

/* Get results from search response */
SearchResponse searchResponse = searchContext.getAttribute("search.response");
List<SearchHit> resultHits = searchResponse.getSearchHits().getSearchHits();

/* Iterate */
for (SearchHit searchHit : resultHits)  {
    Document doc = searchHit.getDocument();
    out.println("entryClassPK: " + doc.getValue("entryClassPK"));
    out.println("ddmFieldArray: " + doc.getValues("ddmFieldArray"));
    out.println("");
}

In your code you should replace the RegistryUtil usage with the "@Reference" annotation.

Let me know if you have any problem with my example.