Missing Aggregation in Elasticsearch

Elasticsearch is a powerful tool for full-text search and data analytics, and one of its core features is the aggregation framework. Aggregations allow you to summarize and analyze your data flexibly and efficiently.

Among the various types of aggregations available, the “missing” aggregation is particularly useful for dealing with incomplete data. This guide will explain what missing aggregation is, how it works, and provide detailed examples to help you understand its usage.

Missing aggregation in Elasticsearch is used to find documents that do not contain a value for a specified field. This type of aggregation is useful when you want to count or analyze documents that are missing certain data. For instance, if you have an index of products and some of the products do not have a price, you can use a missing aggregation to find out how many products are missing this information.

Missing aggregation is particularly useful in scenarios where:

• You need to ensure data completeness by identifying missing fields.
• You want to perform an analysis on incomplete records.
• You need to improve data quality by identifying and filling in missing information.

Let’s consider an Elasticsearch index called products with documents like this:

{
  "product_id": 1,
  "name": "Laptop",
  "category": "electronics",
  "price": 1000,
  "quantity_sold": 5
},
{
  "product_id": 2,
  "name": "T-shirt",
  "category": "clothing",
  "quantity_sold": 20
},
{
  "product_id": 3,
  "name": "Book",
  "category": "books",
  "price": 15
}

In this dataset, the second product (T-shirt) is missing the price field.

To use missing aggregation, you need to specify the field you want to check for missing values. Here is a step-by-step guide on how to do this.

Step 1: Indexing the Data

First, make sure you have indexed your data in Elasticsearch. If you haven’t done so already, you can use the following command to index the example dataset:

POST /products/_bulk
{ "index": { "_id": 1 } }
{ "product_id": 1, "name": "Laptop", "category": "electronics", "price": 1000, "quantity_sold": 5 }
{ "index": { "_id": 2 } }
{ "product_id": 2, "name": "T-shirt", "category": "clothing", "quantity_sold": 20 }
{ "index": { "_id": 3 } }
{ "product_id": 3, "name": "Book", "category": "books", "price": 15 }

Step 2: Running the Missing Aggregation Query

Now, let’s run a missing aggregation query to find out how many products are missing the price field.

Query

GET /products/_search
{
  "size": 0,
  "aggs": {
    "missing_price": {
      "missing": {
        "field": "price"
      }
    }
  }
}

Output

{
  "took": 10,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "missing_price": {
      "doc_count": 1
    }
  }
}

In this example, the aggregation named missing_price shows that there is 1 document (product) missing the price field.

Missing aggregation can be combined with other aggregations to perform more complex analyses. For instance, you can use a terms aggregation to group products by category and then use a missing aggregation to count the number of products missing the price field in each category.

Query

GET /products/_search
{
  "size": 0,
  "aggs": {
    "categories": {
      "terms": {
        "field": "category.keyword"
      },
      "aggs": {
        "missing_price": {
          "missing": {
            "field": "price"
          }
        }
      }
    }
  }
}

Output

{
  "took": 15,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "categories": {
      "buckets": [
        {
          "key": "electronics",
          "doc_count": 1,
          "missing_price": {
            "doc_count": 0
          }
        },
        {
          "key": "clothing",
          "doc_count": 1,
          "missing_price": {
            "doc_count": 1
          }
        },
        {
          "key": "books",
          "doc_count": 1,
          "missing_price": {
            "doc_count": 0
          }
        }
      ]
    }
  }
}

In this example, the products are grouped by category, and within each category, the number of products missing the price field is counted.

Data Quality Checks

One of the primary use cases for missing aggregation is to perform data quality checks. By identifying missing fields, you can ensure that your data is complete and consistent. This is particularly useful in scenarios where data completeness is critical, such as financial reporting or compliance monitoring.

Data Cleaning

Missing aggregation can also be used as part of a data-cleaning process. Once you identify documents with missing fields, you can take corrective actions to fill in the missing information. This can involve updating the documents with the correct values or flagging them for further review.

Monitoring Data Completeness

In applications where data is collected over time, such as logging or IoT data, it’s important to monitor data completeness. Missing aggregation can be used to regularly check for missing fields and alert you when data completeness falls below a certain threshold.

Advanced Example: Nested Aggregations

In some cases, you might want to perform missing aggregations on nested fields. For example, consider a product index where each product has a nested reviews field:

{
  "product_id": 1,
  "name": "Laptop",
  "category": "electronics",
  "reviews": [
    {
      "reviewer": "John",
      "rating": 4
    },
    {
      "reviewer": "Jane",
      "rating": 5
    }
  ]
},
{
  "product_id": 2,
  "name": "T-shirt",
  "category": "clothing",
  "reviews": [
    {
      "reviewer": "Alice",
      "rating": 3
    }
  ]
},
{
  "product_id": 3,
  "name": "Book",
  "category": "books",
  "reviews": []
}

To find products with missing reviews, you can use a nested aggregation combined with a missing aggregation.

Query

GET /products/_search
{
  "size": 0,
  "aggs": {
    "products_with_missing_reviews": {
      "nested": {
        "path": "reviews"
      },
      "aggs": {
        "missing_reviews": {
          "missing": {
            "field": "reviews.reviewer"
          }
        }
      }
    }
  }
}

Output

{
  "took": 20,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "products_with_missing_reviews": {
      "doc_count": 0,
      "missing_reviews": {
        "doc_count": 0
      }
    }
  }
}

In this example, the nested aggregation focuses on the reviews field, and the missing aggregation identifies products where the reviews. reviewer field is missing.

Missing aggregation in Elasticsearch is a powerful tool for identifying and analyzing documents that lack certain data. By understanding and using missing aggregation, you can improve data quality, perform data completeness checks, and gain insights into incomplete records. Whether you’re working on data analytics, reporting, or data cleaning, missing aggregation provides a flexible and efficient way to handle missing data in Elasticsearch. By combining it with other aggregations, you can perform complex analyses and ensure your data is complete and accurate.