ElasticSearch和Python：搜索功能問題

Question

我正在嘗試將ElasticSearch 6.4首次用於以Python/Django編寫的現有Web應用程序。 我有一些問題，我想了解為什么以及如何解決這些問題。

###########

＃現有：

###########

在我的應用程序中，可以上傳文檔文件（例如.pdf或.doc）。 然后，我在我的應用程序中有了一個搜索功能，該功能可以在上載ElasticSearch索引的文檔時進行搜索。

文檔標題始終以相同的方式書寫：

YEAR - DOC_TYPE - ORGANISATION - document_title.extension

例如 ：

1970_ANNUAL_REPORT_APP-TEST_1342 - loremipsum.pdf

搜索功能始終在doc_type = ANNUAL_REPORT 。 因為存在多個doc_type（ANNUAL_REPORT，OTHERS等）。

##################

＃我的環境：＃

##################

根據我的ElasticSearch部分，這是一些數據。 我也在學習ES命令。

$ curl -XGET http://127.0.0.1:9200/_cat/indices?v
health status index uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   app  5T0HZTbmQU2-ZNJXlNb-zg   5   1        742            2    396.4kb        396.4kb

所以我的索引是app

對於上面的示例，如果我搜索以下文檔： 1970_ANNUAL_REPORT_APP-TEST_1342 - loremipsum.pdf ，則有：

$ curl -XGET http://127.0.0.1:9200/app/annual-report/1343?pretty
{
  "_index" : "app",
  "_type" : "annual-report",
  "_id" : "1343",
  "_version" : 33,
  "found" : true,
  "_source" : {
    "attachment" : {
      "date" : "2010-03-04T12:08:00Z",
      "content_type" : "application/pdf",
      "author" : "manshanden",
      "language" : "et",
      "title" : "Microsoft Word - Test document Word.doc",
      "content" : "some text ...",
      "content_length" : 3926
    },
    "relative_path" : "app_docs/APP-TEST/1970_ANNUAL_REPORT_APP-TEST_1342.pdf",
    "title" : "1970_ANNUAL_REPORT_APP-TEST_1342 - loremipsum.pdf"
  }
}

現在，將搜索部分放入Web應用程序中，我希望通過以下搜索找到該文檔： 1970 。

def search_in_annual(self, q):
    try:
        response = self.es.search(
            index='app', doc_type='annual-report',
            q=q, _source_exclude=['data'], size=5000)
    except ConnectionError:
        return -1, None

    total = 0
    hits = []
    if response:
        for hit in response["hits"]["hits"]:
            hits.append({
                'id': hit['_id'],
                'title': hit['_source']['title'],
                'file': hit['_source']['relative_path'],
            })

        total = response["hits"]["total"]

    return total, hits

但是當q=1970 ，結果為0

如果我寫：

response = self.es.search(
                index='app', doc_type='annual-report',
                q="q*", _source_exclude=['data'], size=5000)

它返回我的文檔，但標題或文檔內容中也包含許多沒有1970的文檔。

#################

＃我的全局代碼：＃

#################

這是管理索引功能的全局類：

class EdqmES(object):
    host = 'localhost'
    port = 9200
    es = None

    def __init__(self, *args, **kwargs):
        self.host = kwargs.pop('host', self.host)
        self.port = kwargs.pop('port', self.port)

        # Connect to ElasticSearch server
        self.es = Elasticsearch([{
            'host': self.host,
            'port': self.port
        }])

    def __str__(self):
        return self.host + ':' + self.port

    @staticmethod
    def file_encode(filename):
        with open(filename, "rb") as f:
            return b64encode(f.read()).decode('utf-8')

    def create_pipeline(self):
        body = {
            "description": "Extract attachment information",
            "processors": [
                {"attachment": {
                    "field": "data",
                    "target_field": "attachment",
                    "indexed_chars": -1
                }},
                {"remove": {"field": "data"}}
            ]
        }
        self.es.index(
            index='_ingest',
            doc_type='pipeline',
            id='attachment',
            body=body
        )

    def index_document(self, doc, bulk=False):
        filename = doc.get_filename()

        try:
            data = self.file_encode(filename)
        except IOError:
            data = ''
            print('ERROR with ' + filename)
            # TODO: log error

        item_body = {
            '_id': doc.id,
            'data': data,
            'relative_path': str(doc.file),
            'title': doc.title,
        }

        if bulk:
            return item_body

        result1 = self.es.index(
            index='app', doc_type='annual-report',
            id=doc.id,
            pipeline='attachment',
            body=item_body,
            request_timeout=60
        )
        print(result1)
        return result1

    def index_annual_reports(self):
        list_docs = Document.objects.filter(category=Document.OPT_ANNUAL)

        print(list_docs.count())
        self.create_pipeline()

        bulk = []
        inserted = 0
        for doc in list_docs:
            inserted += 1
            bulk.append(self.index_document(doc, True))

            if inserted == 20:
                inserted = 0
                try:
                    print(helpers.bulk(self.es, bulk, index='app',
                                       doc_type='annual-report',
                                       pipeline='attachment',
                                       request_timeout=60))
                except BulkIndexError as err:
                    print(err)
                bulk = []

        if inserted:
            print(helpers.bulk(
                self.es, bulk, index='app',
                doc_type='annual-report',
                pipeline='attachment', request_timeout=60))

當他提交時，我的文檔已被索引，感謝帶信號的Django表單：

@receiver(signals.post_save, sender=Document, dispatch_uid='add_new_doc')
def add_document_handler(sender, instance=None, created=False, **kwargs):
    """ When a document is created index new annual report (only) with Elasticsearch and update conformity date if the
    document is a new declaration of conformity

    :param sender: Class which is concerned
    :type sender: the model class
    :param instance: Object which was just saved
    :type instance: model instance
    :param created: True for a creation, False for an update
    :type created: boolean
    :param kwargs: Additional parameter of the signal
    :type kwargs: dict
    """

    if not created:
        return

    # Index only annual reports
    elif instance.category == Document.OPT_ANNUAL:
        es = EdqmES()
        es.index_document(instance)

Answer 1

這是我所做的，並且似乎可以正常工作：

def search_in_annual(self, q):
    try:
        response = self.es.search(
            index='app', doc_type='annual-report', q=q, _source_exclude=['data'], size=5000)

        if response['hits']['total'] == 0:

            response = self.es.search(
                index='app', doc_type='annual-report',
                body={
                    "query":
                        {"prefix": {"title": q}},
                }, _source_exclude=['data'], size=5000)

    except ConnectionError:
        return -1, None

    total = 0
    hits = []
    if response:
        for hit in response["hits"]["hits"]:
            hits.append({
                'id': hit['_id'],
                'title': hit['_source']['title'],
                'file': hit['_source']['relative_path'],
            })

        total = response["hits"]["total"]
    return total, hits

它可以搜索標題，前綴和內容以找到我的文檔。