Elasticsearch - PHP Bulk

Question

I am trying to bulk - import data into elasticsearch via cURL from PHP.

For start, I would like to add that I copied the import data format generated by the PHP and pasted it into Sense and the bulk import works just fine. But by sending the same data via cURL to the same link, with the same method that I used in Sense, I am receiving the following error message:

{"_index":"product","_type":"pid","_id":"_bulk","found":false}

OR, if I do not specify the _index and _type via link and I specify it via the json I send, I get the following error

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"No endpoint or operation is available at [_bulk]"}],"type":"illegal_argument_exception","reason":"No endpoint or operation is available at [_bulk]"},"status":400}

The way I am creating the cURL request is the following

protected $curl_opts = array(
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_CUSTOMREQUEST => "GET",
    CURLOPT_TIMEOUT => 10
);

......

public function sendcURL() {
    $this->curl->create('http://localhost:9200/_bulk';

    foreach($this->curl_opts as $option => $option_value)
        $this->curl->option($option, $option_value);

    $this->curl->http_header('Content-Type', 'application/json');
    $this->curl->option(CURLOPT_BINARYTRANSFER, true);
    $this->curl->option(CURLOPT_HEADER, true);
    $this->curl->post($json_data);
    $this->execute();
}

Consider that the $json_data is correctly formatted, as well, consider that I am using the correct link / method.

As well, I know of the elasticsearch-php github repo (even searched how they do bulk in there, and it is similar to my method), but I would prefer writing my own methods & libraries for the moment as what I need at the moment won't require a complete elastic-php library.

What am I doing wrong?

Answer 1

Have you considered that as is written in the ES docs, When sending requests to the _bulk endpoint the Content-Type header should be set to application/x-ndjson

ES bulk Docs

Other thing that could be is that you are using CURLOPT_CUSTOMREQUEST => "GET" but the real request to ES is a POST.

According to DOCS it could cause libcurl to send invalid requests and it could possibly confuse the remote server badly

CURL_OPT DOCS

if one of this approach works, let me know it, and i will edit the answer to just the correct one

Answer 2

CURLOPT_CUSTOMREQUEST => 'GET' can be used with php curl:

$requests = '';
foreach ($queries as $query) {
    list ($data, $headers) = $query;
    $requests .= json_encode($headers) . "\n";
    $requests .= json_encode($data) . "\n";
}

$ch = $this->_ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $requests);
curl_setopt($ch, CURLOPT_HTTPHEADER, ['Content-Type: application/x-ndjson']);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
$response = @curl_exec($ch);