Elasticsearch API Authentication: How to Set Up with Examples

Elasticsearch is a powerful distributed search and analytics engine widely used for logging, monitoring, and data analysis. To protect your data and ensure secure access, setting up API authentication is essential.

This article will guide you through the process of configuring Elasticsearch API authentication with detailed examples and outputs. We will cover basic authentication, API keys, and role-based access control (RBAC).

API authentication in Elasticsearch is crucial for several reasons:

• Data Security: Prevent unauthorized access to your data.
• Data Integrity: Ensure that only authenticated users can modify data.
• Auditability: Track who accessed or modified data.
• Compliance: Meet regulatory requirements for data protection.

Prerequisites

Before setting up API authentication, ensure you have the following:

• Elasticsearch is installed and running.
• Kibana is installed and running (for managing users and roles via the UI).
• Basic knowledge of Elasticsearch and its REST API.

By default, Elasticsearch security features are disabled. To enable them, you need to configure Elasticsearch and restart it.

Step 1: Update the Configuration

Open the elasticsearch.yml configuration file and add the following settings:

xpack.security.enabled: true

Step 2: Generate Certificates

Elasticsearch requires transport and HTTP layer encryption. Use the elasticsearch-certutil tool to generate the necessary certificates.

bin/elasticsearch-certutil ca
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

Follow the prompts to generate the certificates.

Step 3: Configure the Keystore

Add the generated certificates to the Elasticsearch keystore:

bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

Step 4: Restart Elasticsearch

Restart Elasticsearch to apply the changes.

bin/elasticsearch

Basic authentication uses usernames and passwords to control access to the Elasticsearch API.

Step 1: Create a User

You can create users using the Kibana UI or the Elasticsearch REST API.

Using Kibana

• Open Kibana and go to Management > Security > Users.
• Click Create user.
• Fill in the username, password, and assign roles (e.g., superuser).

Using the REST API

Alternatively, you can create a user using the REST API:

curl -X POST "localhost:9200/_security/user/my_user" -H 'Content-Type: application/json' -d'
{
  "password" : "mypassword",
  "roles" : [ "superuser" ],
  "full_name" : "John Doe",
  "email" : "john.doe@example.com"
}'

Step 2: Authenticate API Requests

To authenticate API requests, include the username and password in the request header.

Example: Indexing a Document

curl -u my_user:mypassword -X POST "localhost:9200/myindex/_doc/1" -H 'Content-Type: application/json' -d'
{
  "name": "John Doe",
  "age": 30,
  "city": "New York"
}'

Output

The response indicates that the document is indexed successfully:

{
  "_index": "myindex",
  "_id": "1",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  }
}

API keys provide an alternative method for authenticating API requests without using usernames and passwords.

Step 1: Create an API Key

Create an API key using the Elasticsearch REST API:

curl -u my_user:mypassword -X POST "localhost:9200/_security/api_key" -H 'Content-Type: application/json' -d'
{
  "name": "my_api_key",
  "role_descriptors": {
    "my_role": {
      "cluster": ["all"],
      "index": [
        {
          "names": ["*"],
          "privileges": ["all"]
        }
      ]
    }
  }
}'

Step 2: Authenticate API Requests with the API Key

Include the API key in the request header.

Example: Indexing a Document

curl -H "Authorization: ApiKey <base64-encoded-api-key>" -X POST "localhost:9200/myindex/_doc/1" -H 'Content-Type: application/json' -d'
{
  "name": "Jane Doe",
  "age": 25,
  "city": "San Francisco"
}'

To base64-encode the API key, use the following command (replace id:key with your actual API key):

echo -n "id:key" | base64

Output

The response indicates that the document is indexed successfully:

{
  "_index": "myindex",
  "_id": "1",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  }
}

RBAC allows you to define roles with specific permissions and assign these roles to users and API keys.

Step 1: Define Roles

Create roles that define specific permissions using the Kibana UI or the REST API.

Using Kibana

• Open Kibana and go to Management > Security > Roles.
• Click Create role.
• Define the role name and permissions (e.g., read access to specific indices).

Using the REST API

Alternatively, create a role using the REST API:

curl -u my_user:mypassword -X PUT "localhost:9200/_security/role/my_role" -H 'Content-Type: application/json' -d'
{
  "cluster": ["all"],
  "indices": [
    {
      "names": ["myindex"],
      "privileges": ["read"]
    }
  ]
}'

Step 2: Assign Roles to Users

Assign the created role to a user using the Kibana UI or the REST API.

Using Kibana

• Open Kibana and go to Management > Security > Users.
• Edit the user and assign the role.

Using the REST API

Assign a role to a user using the REST API:

curl -u my_user:mypassword -X POST "localhost:9200/_security/user/my_user/_roles" -H 'Content-Type: application/json' -d'
{
  "roles": ["my_role"]
}'

Step 3: Authenticate API Requests

Authenticated API requests will now have access based on the assigned roles.

Example: Querying an Index with Role-Based Permissions

curl -u my_user:mypassword -X GET "localhost:9200/myindex/_search" -H 'Content-Type: application/json' -d'
{
  "query": {
    "match_all": {}
  }
}'

Output

The response will include documents from the myindex index:

{
  "took": 10,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1.0,
    "hits": [
      {
        "_index": "myindex",
        "_id": "1",
        "_score": 1.0,
        "_source": {
          "name": "Jane Doe",
          "age": 25,
          "city": "San Francisco"
        }
      }
    ]
  }
}

Setting up API authentication in Elasticsearch is essential for securing access to your data and ensuring that only authorized users can interact with your Elasticsearch clusters. This article covered the basics of enabling security features, setting up basic authentication, using API keys, and implementing role-based access control (RBAC).

By following these steps, you can enhance the security of your Elasticsearch deployment and provide controlled access to your data, helping to maintain data integrity and comply with security requirements. Experiment with different configurations and techniques to tailor the authentication setup to your specific needs and environment.