ElasticSearch Pagination: 3 Options#
1. From/Size Pagination#
Uses from + size = offset method to load data into memory on-demand. Supports up to 10,000 results.
The index.max_result_window option can be used to load more than 10,000 results, but this is not recommended.
Features
- Simple implementation
- Limited to 10,000 results maximum
- Performance degradation with deep pagination
2. Search After#
Overcomes the pagination limitation of 10,000 results. Similar to typical cursor-based approaches.
Uses the sort condition field of search results as a key value to retrieve subsequent results.
Drawbacks
Using search-after alone may result in inconsistent responses if indexing is updated during pagination.
Using with PIT (Point In Time)
To address this, use PIT (Point In Time)
POST /my-index-000001/_pit?keep_alive=1m
The above request creates a snapshot of the index at the current point in time, which can then be used with the id value as follows:
{
"query": {},
"size": 100,
"sort": {
"my_sort": "desc"
},
"search_after": {},
"pit": {
"id": "{{pit_value}}"
}
}
Even if there are changes to the index, results are returned based on the snapshot.
keep_alive represents the validity period of the PIT. It’s recommended to manage PIT based on the latest point in time.
Features
- Can paginate more than 10,000 results
- Cursor-based approach
- Guarantees consistent results when used with PIT
- Efficient for deep pagination
3. Scroll#
Note: According to ES official documentation, Search After method is recommended instead of Scroll method
Features
- For bulk data extraction
- Not suitable for real-time search
- Search After + PIT combination is now recommended
