[graylog2] How to solve this alert? Notification condition [NO_MASTER] has been fixed.

[Aitor Mendoza]

Hello,

Since I configured my Graylog server sometimes appears this alert:
*Notification condition [NO_MASTER] has been fixed.*
I have *one Graylog server* with *two ElasticSearch nodes*. I already check
the server.conf to verify that is_master is correct.

Yesterday I found that it could also be because I did not have synchronized
time (I always had it correctly), but I also installed NTP and configured
the 3 hosts with the same NTP servers configuration.
But it seems that the warning still appears ...

I appreciate any help, thank you in advance!

[Jochen Schalanda]

Hi Aitor,

please post the logs of your Graylog node, your Graylog configuration
(including JVM settings), and some details about the hardware of the
machine running Graylog.

Cheers,
Jochen

Post by Aitor Mendoza
Hello,
*Notification condition [NO_MASTER] has been fixed.*
I have *one Graylog server* with *two ElasticSearch nodes*. I already
check the server.conf to verify that is_master is correct.
Yesterday I found that it could also be because I did not have
synchronized time (I always had it correctly), but I also installed NTP and
configured the 3 hosts with the same NTP servers configuration.
But it seems that the warning still appears ...
I appreciate any help, thank you in advance!

[Aitor Mendoza]

Hi,

<Loading Image...>

*graylog.conf on GRAYLOG SERVER:*
is_master = true
node_id_file = /var/opt/graylog/graylog-server-node-id
rest_listen_uri = http://0.0.0.0:9000/api
web_listen_uri = http://0.0.0.0:9000/
elasticsearch_shards = 4
elasticsearch_replicas = 1
elasticsearch_discovery_zen_ping_unicast_hosts =
192.168.1.XX:9300,192.168.1.XX:9300,192.168.1.XX:9300
elasticsearch_cluster_discovery_timeout = 5000
elasticsearch_network_host = 0.0.0.0

*http://192.168.1.xx:9200/_cluster/state?human&pretty*

"nodes" : {
"VqnZug3bTe-SYeYEJTxbbg" : {
"name" : "Servidor Graylog",
"transport_address" : "192.168.1.xx:9300",
"attributes" : {
"data" : "false",
"master" : "true"
}
},
"HD4nGhhfTNuj323-4vzJ8A" : {
"name" : "Servidor Elasticsearch 01",
"transport_address" : "192.168.1.xx:9300",
"attributes" : {
"master" : "true"
}
},
"A6tbkcVMQC6X3ogr7LBCBw" : {
"name" : "graylog-d84e9b91-9e4e-4ca9-a13f-09e824f26e0b",
"transport_address" : "192.168.1.xx:9350",
"attributes" : {
"client" : "true",
"data" : "false",
"master" : "false"
}
},
"z1xsgIoQSDqie4Wj3xT10w" : {
"name" : "Servidor Elasticsearch 02",
"transport_address" : "192.168.1.xx:9300",
"attributes" : {
"master" : "true"
}
}
},


*/etc/elasticsearch/elasticsearch.yml on ES1:*

<Loading Image...>

cluster.name: graylog
node.name: "Servidor Elasticsearch 01"
node.master: true
node.data: true

network.host: 192.168.1.x1
network.bind_host: 192.168.1.x1
network.publish_host: 192.168.1.x1
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["192.168.1.x1:9300" ,
"192.168.1.x2:9300" , "192.168.1.x3:9300"]

index.number_of_replicas: 1
index.number_of_shards: 2


Thanks


El jueves, 2 de febrero de 2017, 10:30:04 (UTC+1), Jochen Schalanda

Post by Jochen Schalanda
Hi Aitor,
please post the logs of your Graylog node, your Graylog configuration
(including JVM settings), and some details about the hardware of the
machine running Graylog.
Cheers,
Jochen

Post by Aitor Mendoza
Hello,
*Notification condition [NO_MASTER] has been fixed.*
I have *one Graylog server* with *two ElasticSearch nodes*. I already
check the server.conf to verify that is_master is correct.
Yesterday I found that it could also be because I did not have
synchronized time (I always had it correctly), but I also installed NTP and
configured the 3 hosts with the same NTP servers configuration.
But it seems that the warning still appears ...
I appreciate any help, thank you in advance!

[Jochen Schalanda]

Hi Aitor,

please post the logs of your Graylog and ES
nodes: http://docs.graylog.org/en/2.1/pages/configuration/file_location.html#deb-package

Cheers,
Jochen

[Aitor Mendoza]

Sorry,

*For example: (/var/log/graylog/elasticsearch/graylog.log)*

<Loading Image...>

<Loading Image...>
[2017-01-30 01:49:24,310][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_56] update_mapping [message]
[2017-01-30 01:49:24,591][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_56] update_mapping [message]
[2017-01-30 02:00:18,260][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_56] update_mapping [message]
[2017-01-30 02:24:34,321][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_56] update_mapping [message]
[2017-01-30 02:24:34,568][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_56] update_mapping [message]
[2017-01-30 05:44:38,282][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_56] update_mapping [message]
[2017-01-30 06:10:19,257][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_56] update_mapping [message]
[2017-01-30 06:25:32,267][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_56] update_mapping [message]
[2017-01-30 07:26:44,297][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_56] update_mapping [message]
[2017-01-30 07:26:48,272][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_56] update_mapping [message]
[2017-01-30 10:38:00,252][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_56] update_mapping [message]
[2017-01-30 16:19:33,278][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_56] update_mapping [message]
[2017-01-30 17:25:35,716][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_56] update_mapping [message]
[2017-01-30 18:03:43,270][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_56] update_mapping [message]
[2017-01-30 18:03:45,291][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_56] update_mapping [message]
[2017-01-30 18:04:17,286][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_56] update_mapping [message]
[2017-01-31 01:00:05,015][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_57] creating index, cause [api], templates
[graylog-internal], shards [4]/[1], mapping$
[2017-01-31 01:00:06,298][INFO ][cluster.routing.allocation] [Servidor
Graylog] Cluster health status changed from [RED] to [YELLOW] (reason:
[shards started [[graylog_57][0]$
[2017-01-31 01:00:07,427][INFO ][cluster.routing.allocation] [Servidor
Graylog] Cluster health status changed from [YELLOW] to [GREEN] (reason:
[shards started [[graylog_57][$
[2017-01-31 01:00:08,251][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_57] update_mapping [message]
[2017-01-31 01:00:08,254][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_57] update_mapping [message]
[2017-01-31 01:00:08,470][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_57] update_mapping [message]
[2017-01-31 01:00:08,737][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_57] update_mapping [message]
[2017-01-31 01:00:09,019][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_57] update_mapping [message]
[2017-01-31 01:00:09,403][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_57] update_mapping [message]
[2017-01-31 01:00:09,657][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_57] update_mapping [message]
[2017-01-31 01:00:10,278][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_57] update_mapping [message]
[2017-01-31 01:00:10,477][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_57] update_mapping [message]
[2017-01-31 01:00:18,255][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_57] update_mapping [message]
[2017-01-31 01:00:22,259][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_57] update_mapping [message]
[2017-01-31 01:00:34,326][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_57] update_mapping [message]
[2017-01-31 01:00:34,531][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_57] update_mapping [message]
[2017-01-31 01:00:34,535][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_57] update_mapping [message]
[2017-01-31 01:00:34,821][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_57] update_mapping [message]
[2017-01-31 01:00:34,824][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_57] update_mapping [message]
[2017-01-31 01:00:34,828][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_57] update_mapping [message]
[2017-01-31 01:00:35,055][INFO ][cluster.metadata ] [Servidor
Graylog] [graylog_57] update_mapping [message]





El jueves, 2 de febrero de 2017, 12:57:03 (UTC+1), Jochen Schalanda

Post by Jochen Schalanda
Hi Aitor,
http://docs.graylog.org/en/2.1/pages/configuration/file_location.html#deb-package
Cheers,
Jochen

[Jochen Schalanda]

Hi Aitor,

Post by Aitor Mendoza
*For example: (/var/log/graylog/elasticsearch/graylog.log)*

Please post the *complete* logs of your Graylog and Elasticsearch nodes as
text (for example as an attachment to this discussion).

Did you run out of disk space? There is at least 1 corrupted Elasticsearch
index (graylog_58) according to your logs.

Cheers,
Jochen

[Aitor Mendoza]

Here you have. Ignore the space problem from today. Yes, I had a problem
with the datastore.

Always is running perfectly but somtimes appears this alert "NO MASTER" on
graylog.

Thanks!

El jueves, 2 de febrero de 2017, 14:29:46 (UTC+1), Jochen Schalanda

Post by Jochen Schalanda
Hi Aitor,

Post by Aitor Mendoza
*For example: (/var/log/graylog/elasticsearch/graylog.log)*

Please post the *complete* logs of your Graylog and Elasticsearch nodes
as text (for example as an attachment to this discussion).
Did you run out of disk space? There is at least 1 corrupted Elasticsearch
index (graylog_58) according to your logs.
Cheers,
Jochen

[Jochen Schalanda]

Hi Aitor,

these logs clearly show that your Elasticsearch cluster is not healthy: It
ran out of disk space multiple times and it can't keep up with indexing
messages sent by Graylog (full task queues etc.).

You'll have to provide more hardware (esp. more memory, at least 4 GiB) to
your Elasticsearch nodes.

Cheers,
Jochen

[Aitor Mendoza]

Hello Jochen,

But the problem of disk space is from yesterday because a vmware datastore
problem that is already solved. But I want to solve the alert "NO MASTER
fixed" that appears till the first day...

Thanks

El jueves, 2 de febrero de 2017, 15:22:58 (UTC+1), Jochen Schalanda

Post by Jochen Schalanda
Hi Aitor,
these logs clearly show that your Elasticsearch cluster is not healthy: It
ran out of disk space multiple times and it can't keep up with indexing
messages sent by Graylog (full task queues etc.).
You'll have to provide more hardware (esp. more memory, at least 4 GiB) to
your Elasticsearch nodes.
Cheers,
Jochen

[Jochen Schalanda]

Hi Aitor,

as I already mentioned, your ES cluster doesn't have enough hardware
resources to keep up with the data ingestion from Graylog. Assign at least
4 GiB of memory for each Elasticsearch node.

Cheers,
Jochen

Post by Aitor Mendoza
Hello Jochen,
But the problem of disk space is from yesterday because a vmware datastore
problem that is already solved. But I want to solve the alert "NO MASTER
fixed" that appears till the first day...
Thanks
El jueves, 2 de febrero de 2017, 15:22:58 (UTC+1), Jochen Schalanda

Post by Jochen Schalanda
Hi Aitor,
It ran out of disk space multiple times and it can't keep up with indexing
messages sent by Graylog (full task queues etc.).
You'll have to provide more hardware (esp. more memory, at least 4 GiB)
to your Elasticsearch nodes.
Cheers,
Jochen

[Aitor Mendoza]

Hi Jochen,

Always both ElasticSearch nodes are using 900 MB of 2GB RAM... but I will
try it with 4GB if it is necessary.

Thanks ;)

El viernes, 3 de febrero de 2017, 10:07:57 (UTC+1), Jochen Schalanda

Post by Jochen Schalanda
Hi Aitor,
as I already mentioned, your ES cluster doesn't have enough hardware
resources to keep up with the data ingestion from Graylog. Assign at least
4 GiB of memory for each Elasticsearch node.
Cheers,
Jochen

Post by Aitor Mendoza
Hello Jochen,
But the problem of disk space is from yesterday because a vmware
datastore problem that is already solved. But I want to solve the alert "NO
MASTER fixed" that appears till the first day...
Thanks
El jueves, 2 de febrero de 2017, 15:22:58 (UTC+1), Jochen Schalanda

Post by Jochen Schalanda
Hi Aitor,
It ran out of disk space multiple times and it can't keep up with indexing
messages sent by Graylog (full task queues etc.).
You'll have to provide more hardware (esp. more memory, at least 4 GiB)
to your Elasticsearch nodes.
Cheers,
Jochen