[graylog2] Huge gaps between the time stamp on Graylog server and actual logged items

[Kevin Johnson]

I’m running a single node of Graylog v1.1.2 with Elasticsearch, MongoDB and
Graylog-web-interface v.1.1.2 on an Amazon EC2 instance (m4.large). I have
four sources and running a total of nine inputs. All of my inputs are
RAW/PLAINTEXT UDP. I’m using a script to push my logs to Graylog via
separate ports. My Graylog server is running at optimum performance. For
some reason there is a huge gap in time between the Graylog time stamp and
the actual time of the log items. Can anyone please tell me why?

[Jochen Schalanda]

Hi Kevin,

by default (and especially on Raw/Plaintext inputs) message timestamps are
stored in UTC. Maybe you didn't change the timezone settings of your
user(s) to match the actual timezone in your place?


Cheers,
Jochen

Post by Kevin Johnson
I’m running a single node of Graylog v1.1.2 with Elasticsearch, MongoDB
and Graylog-web-interface v.1.1.2 on an Amazon EC2 instance (m4.large). I
have four sources and running a total of nine inputs. All of my inputs
are RAW/PLAINTEXT UDP. I’m using a script to push my logs to Graylog via
separate ports. My Graylog server is running at optimum performance. For
some reason there is a huge gap in time between the Graylog time stamp and
the actual time of the log items. Can anyone please tell me why?

[Kevin Johnson]

Thank you for your reply Jochen. All of my server are in the same region
and have the same timezone settings.

Post by Jochen Schalanda
Hi Kevin,
by default (and especially on Raw/Plaintext inputs) message timestamps are
stored in UTC. Maybe you didn't change the timezone settings of your
user(s) to match the actual timezone in your place?
Cheers,
Jochen

Post by Kevin Johnson
I’m running a single node of Graylog v1.1.2 with Elasticsearch, MongoDB
and Graylog-web-interface v.1.1.2 on an Amazon EC2 instance (m4.large). I
have four sources and running a total of nine inputs. All of my inputs
are RAW/PLAINTEXT UDP. I’m using a script to push my logs to Graylog via
separate ports. My Graylog server is running at optimum performance. For
some reason there is a huge gap in time between the Graylog time stamp and
the actual time of the log items. Can anyone please tell me why?

[Jochen Schalanda]

Hi Kevin,

did you also set root_timezone (
https://github.com/Graylog2/graylog2-server/blob/1.1.6/misc/graylog2.conf#L28-L31)
in the configuration files of your Graylog server nodes?


Cheers,
Jochen

Post by Kevin Johnson
Thank you for your reply Jochen. All of my server are in the same region
and have the same timezone settings.

Post by Jochen Schalanda
Hi Kevin,
by default (and especially on Raw/Plaintext inputs) message timestamps
are stored in UTC. Maybe you didn't change the timezone settings of your
user(s) to match the actual timezone in your place?
Cheers,
Jochen

Post by Kevin Johnson
I’m running a single node of Graylog v1.1.2 with Elasticsearch, MongoDB
and Graylog-web-interface v.1.1.2 on an Amazon EC2 instance (m4.large).
I have four sources and running a total of nine inputs. All of my
inputs are RAW/PLAINTEXT UDP. I’m using a script to push my logs to Graylog
via separate ports. My Graylog server is running at optimum
performance. For some reason there is a huge gap in time between the
Graylog time stamp and the actual time of the log items. Can anyone
please tell me why?

[Kevin Johnson]

I set the root_timezone to EST, which all my servers are set to. Restarted Graylog. Once again there is a huge gap in time between the Graylog time stamp and the actual time of the logs. The time between them is well over 24 hrs. When creating alerts, I receive them while after the fact. Is there anything I tweak on the Graylog server to alleviate the huge gap in time?

[Jochen Schalanda]

Hi Kevin,

could you please post some of the messages you send to Graylog and how they
are being parsed?


Cheers,
Jochen

Post by Kevin Johnson
I set the root_timezone to EST, which all my servers are set to.
Restarted Graylog. Once again there is a huge gap in time between the
Graylog time stamp and the actual time of the logs. The time between them
is well over 24 hrs. When creating alerts, I receive them while after the
fact. Is there anything I tweak on the Graylog server to alleviate the huge
gap in time?

[Kevin Johnson]

Hi Jochen,

Below is a screen shot of some of the messages:



I'm running the following script to send the log to the Graylog server.

#!/bin/bash

tail -F -q /u02/logs/php_error.log |

while read -r line ;

do echo 192.1681.1 $line |

nc -w 1 -u 192.168.1.12 12409;

done;

Post by Jochen Schalanda
Hi Kevin,
could you please post some of the messages you send to Graylog and how
they are being parsed?
Cheers,
Jochen

Post by Kevin Johnson
I set the root_timezone to EST, which all my servers are set to.
Restarted Graylog. Once again there is a huge gap in time between the
Graylog time stamp and the actual time of the logs. The time between them
is well over 24 hrs. When creating alerts, I receive them while after the
fact. Is there anything I tweak on the Graylog server to alleviate the huge
gap in time?

[Jochen Schalanda]

Hi Kevin,

did you create an extractor (e. g. a grok or a regex extractor) to parse
those access logs and use the recognized date as the message timestamp? If
so, how do those extractors look like?


Cheers,
Jochen

Post by Kevin Johnson
Hi Jochen,
I'm running the following script to send the log to the Graylog server.
#!/bin/bash
tail -F -q /u02/logs/php_error.log |
while read -r line ;
do echo 192.1681.1 $line |
nc -w 1 -u 192.168.1.12 12409;
done;

Post by Jochen Schalanda
Hi Kevin,
could you please post some of the messages you send to Graylog and how
they are being parsed?
Cheers,
Jochen

Post by Kevin Johnson
I set the root_timezone to EST, which all my servers are set to.
Restarted Graylog. Once again there is a huge gap in time between the
Graylog time stamp and the actual time of the logs. The time between them
is well over 24 hrs. When creating alerts, I receive them while after the
fact. Is there anything I tweak on the Graylog server to alleviate the huge
gap in time?

[Kevin Johnson]

Hi Jochen,

I did not create an extractor to parse the access logs. I have setup
extractors on other inputs. How do I use the recognized date as the
message timestamps?

Post by Jochen Schalanda
Hi Kevin,
did you create an extractor (e. g. a grok or a regex extractor) to parse
those access logs and use the recognized date as the message timestamp? If
so, how do those extractors look like?
Cheers,
Jochen

Post by Kevin Johnson
Hi Jochen,
I'm running the following script to send the log to the Graylog server.
#!/bin/bash
tail -F -q /u02/logs/php_error.log |
while read -r line ;
do echo 192.1681.1 $line |
nc -w 1 -u 192.168.1.12 12409;
done;

Post by Jochen Schalanda
Hi Kevin,
could you please post some of the messages you send to Graylog and how
they are being parsed?
Cheers,
Jochen

Post by Kevin Johnson
I set the root_timezone to EST, which all my servers are set to.
Restarted Graylog. Once again there is a huge gap in time between the
Graylog time stamp and the actual time of the logs. The time between them
is well over 24 hrs. When creating alerts, I receive them while after the
fact. Is there anything I tweak on the Graylog server to alleviate the huge
gap in time?

[Jochen Schalanda]

Hi Kevin,

you can extract the date from the log messages with a regex extractor and
afterwards use a date or flexdate converter (see
http://docs.graylog.org/en/1.1/pages/extractors.html#normalization) to
convert it to an actual timestamp which you store in the timestamp field of
your message. Otherwise the receive date of the raw text message will be
used as message timestamp.

Cheers,
Jochen

Post by Kevin Johnson
Hi Jochen,
I did not create an extractor to parse the access logs. I have setup
extractors on other inputs. How do I use the recognized date as the
message timestamps?

Post by Jochen Schalanda
Hi Kevin,
did you create an extractor (e. g. a grok or a regex extractor) to parse
those access logs and use the recognized date as the message timestamp? If
so, how do those extractors look like?
Cheers,
Jochen

Post by Kevin Johnson
Hi Jochen,
I'm running the following script to send the log to the Graylog server.
#!/bin/bash
tail -F -q /u02/logs/php_error.log |
while read -r line ;
do echo 192.1681.1 $line |
nc -w 1 -u 192.168.1.12 12409;
done;

Post by Jochen Schalanda
Hi Kevin,
could you please post some of the messages you send to Graylog and how
they are being parsed?
Cheers,
Jochen

Post by Kevin Johnson
I set the root_timezone to EST, which all my servers are set to.
Restarted Graylog. Once again there is a huge gap in time between the
Graylog time stamp and the actual time of the logs. The time between them
is well over 24 hrs. When creating alerts, I receive them while after the
fact. Is there anything I tweak on the Graylog server to alleviate the huge
gap in time?