Elasticsearch Bulk

Component that can talk to Elasticsearch and utilizes bulk API endpoint. Requires Elasticsearch 8.4.x or higher.

Configuration

Bulk Size

Number specifying the number of documents in each bulk request.

Bulk Timeout

Timeout in milliseconds before triggering a request in case the bulk size is not reached.

Connection

Connection URL

Connection URL to the API endpoint

Example: https://localhost:9200/

Authentication

Username As required by the API

Example: elastic

Password As required by the API

Example: changeMe

HTTP(s)

The path to the elasticsearch certificate

Example: Home/Documents/Elasticsearch/config/certs/http_ca.crt

Rate limit

Request limit

The max amount of requests during the interval.

Interval

The interval in milliseconds in which the requests happen. Should be a multiple of 250.

Max concurrent

The maximum concurrent executions.

Inputs

• Name: Input

  • Description:

    Performs multiple operations in a single API call. This reduces overhead and can greatly increase indexing speed. As an input, this component takes any operations and combines them until reaching the bulk size or until the bulk timeout is reached and then triggers the request.

    On successful execution, the output will be the response of Elasticsearch.

    On error, the default error handling applies.

  • Example: A message containing the operation and, depending on the type of operation, the data.

    {
      "operation":{},
      "data":{}
    }

  • Example: Most common types of operations:

    Index operation

    {
      "operation":{ "index" : { "_index" : "test", "_id" : "1" } },
      "data":{ "field1" : "value1" }
    }

    Delete operation

    {
      "operation":{ "delete" : { "_index" : "test", "_id" : "2" } },
      "data":{}
    }

    Update operation

    {
      "operation":{ "update" : {"_id" : "1", "_index" : "test"} },
      "data":{ "doc" : {"field2" : "value2"} }
    }

Outputs

• Name: Output

  • Description: The incoming message with the response of Elasticsearch.