[graylog2] Nodes with too long GC pauses

Discussion:

Nitzan Haimovich

2017-02-02 16:18:12 UTC

Hi all,

I'm getting this message (*Nodes with too long GC pauses*) on my Graylog
cluster. I saw many people were posting about it but not a single thread
with solutions for how to solve/fix/approach it.
I would be glad for any help.

My cluster - 3 Graylog instances, each one with 8 cores and 16GB memory
(heap size is configured to be : Xms - 1GB , Xmx - 8GB).
If you need any more details please let me know.

Thanks!!

Nitzan

--
You received this message because you are subscribed to the Google Groups "Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/fd53a040-cecb-45eb-ba32-20c2fbf2dac7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Jochen Schalanda

2017-02-02 16:38:57 UTC

Permalink

Hi Nitzan,

please post the configuration and logs of all Graylog nodes and a
description of your hardware.

Cheers,
Jochen

Post by Nitzan Haimovich
Hi all,
I'm getting this message (*Nodes with too long GC pauses*) on my Graylog
cluster. I saw many people were posting about it but not a single thread
with solutions for how to solve/fix/approach it.
I would be glad for any help.
My cluster - 3 Graylog instances, each one with 8 cores and 16GB memory
(heap size is configured to be : Xms - 1GB , Xmx - 8GB).
If you need any more details please let me know.
Thanks!!
Nitzan

--
You received this message because you are subscribed to the Google Groups "Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/67436c8a-659b-4dce-b24a-3fec658852ec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Nitzan Haimovich

2017-02-05 06:55:24 UTC

Permalink

Hi Jochen,

Thanks for you help. I attached one file of the configuration - The
configuration is the same in all 3 Graylog machines in our cluster (except
is_master parameter and the IP parameter).
I also attached one file of the logs, again - same logs are all around the
cluster (except the master periodical duties).

Our cluster:
3 Amazon instances, each one sized as m4.2xlarge - 8 cores and 32GB mem.
With Disks of 2.5TB.

Let me know if you need anything else!

Nitzan

Post by Jochen Schalanda
Hi Nitzan,
please post the configuration and logs of all Graylog nodes and a
description of your hardware.
Cheers,
Jochen

--
You received this message because you are subscribed to the Google Groups "Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/942c1187-99bf-492a-b226-e91d89f99392%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Jochen Schalanda

2017-02-06 09:44:26 UTC

Permalink

Hi Nitzan,

you've configure a very high number of processbuffer_processors and
outputbuffer_processors in your Graylog configuration which usually
counterproductive.

Please revert to the default values and only increase these values very
conservatively.

Cheers,
Jochen

Post by Nitzan Haimovich
Hi Jochen,
Thanks for you help. I attached one file of the configuration - The
configuration is the same in all 3 Graylog machines in our cluster (except
is_master parameter and the IP parameter).
I also attached one file of the logs, again - same logs are all around the
cluster (except the master periodical duties).
3 Amazon instances, each one sized as m4.2xlarge - 8 cores and 32GB mem.
With Disks of 2.5TB.
Let me know if you need anything else!
Nitzan

Post by Jochen Schalanda
Hi Nitzan,
please post the configuration and logs of all Graylog nodes and a
description of your hardware.
Cheers,
Jochen

Post by Nitzan Haimovich
Hi all,
I'm getting this message (*Nodes with too long GC pauses*) on my
Graylog cluster. I saw many people were posting about it but not a single
thread with solutions for how to solve/fix/approach it.
I would be glad for any help.
My cluster - 3 Graylog instances, each one with 8 cores and 16GB memory
(heap size is configured to be : Xms - 1GB , Xmx - 8GB).
If you need any more details please let me know.
Thanks!!
Nitzan

--
You received this message because you are subscribed to the Google Groups "Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/276bc8b7-10be-4388-9a5b-c2ae2cef7f99%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Nitzan Haimovich

2017-02-16 13:20:33 UTC

Permalink

I reverted, no change.
It doesn't affect the GC long pauses.
Where do I configure the threshold for this?

Thanks

Post by Jochen Schalanda
Hi Nitzan,
you've configure a very high number of processbuffer_processors and
outputbuffer_processors in your Graylog configuration which usually
counterproductive.
Please revert to the default values and only increase these values very
conservatively.
Cheers,
Jochen

Post by Nitzan Haimovich
Hi Jochen,
Thanks for you help. I attached one file of the configuration - The
configuration is the same in all 3 Graylog machines in our cluster (except
is_master parameter and the IP parameter).
I also attached one file of the logs, again - same logs are all around
the cluster (except the master periodical duties).
3 Amazon instances, each one sized as m4.2xlarge - 8 cores and 32GB mem.
With Disks of 2.5TB.
Let me know if you need anything else!
Nitzan

Post by Jochen Schalanda
Hi Nitzan,
please post the configuration and logs of all Graylog nodes and a
description of your hardware.
Cheers,
Jochen

Post by Nitzan Haimovich
Hi all,
I'm getting this message (*Nodes with too long GC pauses*) on my
Graylog cluster. I saw many people were posting about it but not a single
thread with solutions for how to solve/fix/approach it.
I would be glad for any help.
My cluster - 3 Graylog instances, each one with 8 cores and 16GB memory
(heap size is configured to be : Xms - 1GB , Xmx - 8GB).
If you need any more details please let me know.
Thanks!!
Nitzan

--
You received this message because you are subscribed to the Google Groups "Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/d7483bff-11c0-4044-a16a-ebdf366452e8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Jochen Schalanda

2017-02-16 13:25:18 UTC

Permalink

Hi Nitzan,

Post by Nitzan Haimovich
Where do I configure the threshold for this?

You can configure this with the gc_warning_threshold
setting: https://github.com/Graylog2/graylog2-server/
blob/2.2.0/misc/graylog.conf#L527-L529

But be aware that you're merely covering up the problem and not solving
anything with increasing this threshold.

Cheers,
Jochen

--
You received this message because you are subscribed to the Google Groups "Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/f2d21cb2-19fe-4047-a3d3-8a4f07777540%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Continue reading on narkive:

Search results for '[graylog2] Nodes with too long GC pauses' (Questions and Answers)

replies

Briefly describe the Microsoft's 2000 DNS management?

started 2006-08-17 22:05:37 UTC

computer networking