[graylog2] Unable to execute search: all shards failed / org.elasticsearch.action.search.SearchPhaseExecutionException

Oliver Schrenk

2016-09-19 13:22:31 UTC

Hi,

I have troubles setting up a basic graylog instance with a single UDP GELF
input.

ElasticSearch: 2.4.0
Graylog: 2.1.1

There is no load balancer or reverse proxy. and I'm going with default
configuration and changed these settings:

```

rest_listen_uri = http://127.0.0.1:9000/api/

< rest_listen_uri = http://0.0.0.0:12900/

#web_listen_uri = http://127.0.0.1:9000/

< web_listen_uri = http://0.0.0.0:9000/

#web_endpoint_uri =

< web_endpoint_uri = http://staging.acme.nl:12900/
```

I successfully logged in, created UDP Gelf input source, was able to see a
sample message I pushed from my Scala application (upon creating the input,
not while searching it).

I'm pushing the data with this config

```
<appender name="GRAYLOG"
class="com.github.pukkaone.gelf.logback.GelfAppender">
<graylogHost>udp:staging.acme.nl</graylogHost>
<graylogPort>12201</graylogPort>
<originHost>my.machine.example.com</originHost>
<levelIncluded>true</levelIncluded>
<loggerIncluded>true</loggerIncluded>
<markerIncluded>false</markerIncluded>
<threadIncluded>false</threadIncluded>
<facility>gelf-java</facility>
<additionalField>application=scala-api</additionalField>
<additionalField>environment=development</additionalField>
</appender>
```

But when I want to actually search for something the UI fails.

The javascript console shows

```
client.js:960 GET
http://staging.acme.nl:12900/search/universal/relative/histogram?query=%2A&range=300&interval=minute
400 (Bad Request)d.end @ client.js:960(anonymous function) @ index.js:61i @
bluebird.js:4594i._resolveFromResolver @ bluebird.js:2671i @
bluebird.js:2259r.promise @ index.js:60r.then @ index.js:95value @
FetchProvider.js:57i @ FetchProvider.js:96histogram @
UniversalSearchStore.js:43promise.promise.isCancelled.promise.P.search.then.t.additional.t.additional.status.e.setState.error
@ SearchPage.jsx:99i @ bluebird.js:4594i._settlePromiseFromHandler @
bluebird.js:2698i._settlePromiseAt @ bluebird.js:2772i._settlePromises @
bluebird.js:2888n._drainQueue @ bluebird.js:175n._drainQueues @
bluebird.js:185drainQueues @ bluebird.js:67
FetchProvider.js:17 There was an error fetching a resource: cannot GET
http://staging.acme.nl:12900/search/universal/relative/histogram?query=%2A&range=300&interval=minute
(400). Additional information: Unable to execute searcht @
FetchProvider.js:17(anonymous function) @ FetchProvider.js:82i @
bluebird.js:4594i._settlePromiseFromHandler @
bluebird.js:2698i._settlePromiseAt @ bluebird.js:2772i._settlePromises @
bluebird.js:2888n._drainQueue @ bluebird.js:175n._drainQueues @
bluebird.js:185drainQueues @ bluebird.js:67
bluebird.js:953 Unhandled rejection Error: cannot GET
http://staging.acme.nl:12900/search/universal/relative/histogram?query=%2A&range=300&interval=minute
(400)
```

The network tab shows that the request (here copied as Curl)

```
curl
'http://staging.acme.nl:12900/search/universal/relative/histogram?query=%2A&range=300&interval=minute'
-H 'Authorization: Basic
ZjkxZTk4YjktMGUxZC00MzM3LWJmYTktYzI2M2JmMTNiMGUzOnNlc3Npb24=' -H 'Origin:
http://staging.acme.nl:9000' -H 'Accept-Encoding: gzip, deflate, sdch' -H
'Accept-Language: en-GB,en-US;q=0.8,en;q=0.6' -H 'User-Agent: Mozilla/5.0
(Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/54.0.2837.0 Safari/537.36' -H 'Content-Type: application/json' -H
'Accept: application/json' -H 'Referer: http://staging.acme.nl:9000/search'
-H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' -H 'DNT:
1' --compressed
```

fails with

```
{"query":"*","begin_column":null,"begin_line":null,"end_column":null,"end_line":null,"message":"Unable
to execute
search","exception_name":"org.elasticsearch.action.search.SearchPhaseExecutionException"}
```

I don't see anything in elasticsearch log and the tab
http://staging.acme.nl:9000/system/indices shows no error for
elasticsearch. But graylog logs shows this

```
2016-09-19T15:17:14.564+02:00 INFO [InputStateListener] Input [GELF
UDP/57dfbbcbc0cf5374e79b8c0a] is now STARTING
2016-09-19T15:17:14.662+02:00 WARN [NettyTransport] receiveBufferSize
(SO_RCVBUF) for input GELFUDPInput{title=Gelf UDP Test,
type=org.graylog2.inputs.gelf.udp.GELFUDPInput,
nodeId=5e5806ce-0922-4739-93af-0212d8b43772} should be 262144 but is 212992.
2016-09-19T15:17:14.676+02:00 INFO [InputStateListener] Input [GELF
UDP/57dfbbcbc0cf5374e79b8c0a] is now RUNNING
2016-09-19T15:17:18.748+02:00 WARN [transport]
[graylog-5e5806ce-0922-4739-93af-0212d8b43772] Transport response handler
not found of id [220]
2016-09-19T15:17:18.749+02:00 WARN [SearchResource] Unable to execute
search: all shards failed
```

I presume it's a configuration error, but with these error messages I don't
get far. Any idea?

Cheers,
Oliver

--
You received this message because you are subscribed to the Google Groups "Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/cec349b1-3e0f-4ec4-ab79-4018d43614c6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.