Elasticsearch 7.0.1 - Trying to create too many scroll contexts. Must be less than or equal to: [500]

See original GitHub issue

Good afternoon,

I upgraded my stack to 7.0.1 and I am using elastalert>=0.2.0b. When I run Elasticsearch and elastalert, it alls seems to be working fine. However, the moment I ingest data, and the rules start hitting the indices, I get the following messages:

ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 [status:500 request:0.016s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-endpoint-winevent-security-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 [status:500 request:0.007s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-endpoint-winevent-sysmon-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 [status:500 request:0.007s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-endpoint-winevent-security-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 [status:500 request:0.005s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 [status:500 request:0.010s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')

I updated the max_open_scroll_context to 5000 and I still get the same errors.

curl -X PUT localhost:9200/_cluster/settings -H 'Content-Type: application/json' -d'{
    "persistent" : {
        "search.max_open_scroll_context": 5000
    },
    "transient": {
        "search.max_open_scroll_context": 5000
    }
}
'

Issue Analytics

  • State:open
  • Created 4 years ago
  • Comments:30 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
Qmandocommented, May 17, 2019

Next to --verbose is fine (order doesn’t matter). You should see a bunch of stuff in there after running elastalert for a short time. Just note it prints out things with curl localhost:9200 despite whatever settings you have, a “feature” of the elasticsearch library. 😉

1reaction
Qmandocommented, May 17, 2019

Hmm…

One thing you should check is that your version of the elasticsearch Python library is > 7. Maybe this is caused by some API change.

You can see from GET http://helk-elasticsearch:9200/logs-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 that it’s setting each scroll to a 30 second timeout. 7.0 changed a bunch of APIs so maybe that assumption is no longer right.

I really don’t have much expertise here but I’d probably check to see whether there are other APIs you can use to see what scrolls are active, then you could manually clear them or verify that they are coming from elastalert and not something else.

From https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html

You can check how many search contexts are open with the nodes stats API:
GET /_nodes/stats/indices/search
Read more comments on GitHub >

github_iconTop Results From Across the Web

Trying to create too many scroll contexts. Must be less than or ...
I am new to elasticsearch. Im getting this error."Trying to create too many scroll contexts. Must be less than or equal to: [500]....
Read more >
How to resolve Trying to create too many scroll contexts. Must ...
Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting. status: 500, displayName ...
Read more >
Search Troubleshooting | K2View Support
Must be less than or equal to: [500]. ... running migration that involves search queries on the Elasticsearch: "Trying to create too many...
Read more >
Running Elasticsearch 7.5.2 with Bitbucket 7.1+ can result in ...
2 with Bitbucket 7.1+ can result in too many scroll contexts created. Closed: Assignee: Priority: Resolution: Closed. Wolfgang ...
Read more >
Elasticsearch Trying to create too many scroll contexts. Must ...
Elasticsearch Trying to create too many scroll contexts. Must be less than or equal to 500. qq_34412985 于 2021-12-30 10:17:32 发布 993 收藏...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found