[graylog2] Overwriting Timestamp field using Pipeline rules

[Al Reynolds]

Hello all,

I'm attempting to switch our logging infrastructure from the ELK stack to
Graylog, but I'm running into an issue with the pipeline rules and
replacing the timestamp field. Rule below:

rule "WO-CS-RAS"
when

contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
then
set_field("WO_Log_Source","RAS-CS");
let matches = grok(pattern: "%{WO_CS_RAS_CS_MESSAGE}", value:
to_string($message.message));
set_fields(matches);
let date = parse_date(to_string($message.WO_Timestamp), "YYYY-MM-dd
HH:mm:ss,sss");
let new_date = format_date(date,"YYYY-MM-DD'T'HH:mm:ss.SSS");
set_field("$timestamp", new_date);
route_to_stream("WideOrbit Logs");
end

I've tried without the date formatter as well--no luck there either. The
rule will error out and not replace the timestamp field. Everything else
works perfectly. Any suggestions as to where I might be going wrong? If I
use an extractor I can replace the timestamp field, but I'd like to keep
everything in one place if possible.

Thanks!

Cheers,
Al

[Jochen Schalanda]

Hi Al,

the "timestamp" field has to be a Date object and not a string.
Additionally, the first parameter of your set_field() call seems odd
("$timestamp" instead of "timestamp").

This rule might work, although I haven't tested it:

rule "WO-CS-RAS"
when

contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
then
set_field("WO_Log_Source","RAS-CS");
let matches = grok(pattern: "%{WO_CS_RAS_CS_MESSAGE}", value:
to_string($message.message));
set_fields(matches);
let date = parse_date(to_string($message.WO_Timestamp), "YYYY-MM-dd
HH:mm:ss,sss");
set_field("timestamp", date);
route_to_stream("WideOrbit Logs");
end


Cheers,
Jochen

Post by Al Reynolds
Hello all,
I'm attempting to switch our logging infrastructure from the ELK stack to
Graylog, but I'm running into an issue with the pipeline rules and
rule "WO-CS-RAS"
when
contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
then
set_field("WO_Log_Source","RAS-CS");
to_string($message.message));
set_fields(matches);
let date = parse_date(to_string($message.WO_Timestamp), "YYYY-MM-dd
HH:mm:ss,sss");
let new_date = format_date(date,"YYYY-MM-DD'T'HH:mm:ss.SSS");
set_field("$timestamp", new_date);
route_to_stream("WideOrbit Logs");
end
I've tried without the date formatter as well--no luck there either. The
rule will error out and not replace the timestamp field. Everything else
works perfectly. Any suggestions as to where I might be going wrong? If I
use an extractor I can replace the timestamp field, but I'd like to keep
everything in one place if possible.
Thanks!
Cheers,
Al

[Al Reynolds]

Jochen,

Thanks for the reply! I'm guessing my problem is that the source field (in
this case WO_Timestamp) is not a date object, as I'm not having any luck
with your example either. I was under the impression that using the
"parse_date" function would create a Date object?

As for "$timestamp" instead of "timestamp", I was trying different
configurations, and thought that since the message field is referenced as
"$message" I would try that format. What does the "$" indicate?

Thanks!

Cheers,
Al

Post by Jochen Schalanda
Hi Al,
the "timestamp" field has to be a Date object and not a string.
Additionally, the first parameter of your set_field() call seems odd
("$timestamp" instead of "timestamp").
rule "WO-CS-RAS"
when
contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
then
set_field("WO_Log_Source","RAS-CS");
to_string($message.message));
set_fields(matches);
let date = parse_date(to_string($message.WO_Timestamp), "YYYY-MM-dd
HH:mm:ss,sss");
set_field("timestamp", date);
route_to_stream("WideOrbit Logs");
end
Cheers,
Jochen

Post by Al Reynolds
Hello all,
I'm attempting to switch our logging infrastructure from the ELK stack to
Graylog, but I'm running into an issue with the pipeline rules and
rule "WO-CS-RAS"
when
contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
then
set_field("WO_Log_Source","RAS-CS");
to_string($message.message));
set_fields(matches);
let date = parse_date(to_string($message.WO_Timestamp), "YYYY-MM-dd
HH:mm:ss,sss");
let new_date = format_date(date,"YYYY-MM-DD'T'HH:mm:ss.SSS");
set_field("$timestamp", new_date);
route_to_stream("WideOrbit Logs");
end
I've tried without the date formatter as well--no luck there either. The
rule will error out and not replace the timestamp field. Everything else
works perfectly. Any suggestions as to where I might be going wrong? If I
use an extractor I can replace the timestamp field, but I'd like to keep
everything in one place if possible.
Thanks!
Cheers,
Al

[Jochen Schalanda]

Hi Al,

I was under the impression that using the "parse_date" function would
create a Date object?

It does,
see http://docs.graylog.org/en/2.1/pages/pipelines/functions.html#parse-date
for reference. But your date pattern may be wrong
(see http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html
for reference).

Please share some example messages, so that we can validate your rule.

As for "$timestamp" instead of "timestamp", I was trying different
configurations, and thought that since the message field is referenced as
"$message" I would try that format. What does the "$" indicate?

The $ character is simply part of the variable name containing the current
message (which is "$message"). It doesn't have a special meaning.

Cheers,
Jochen

[Al Reynolds]

Jochen,

MSG 01 Sample:
WO_CS_RAS_CS_MESSAGE
2017-02-08 09:41:40,674 INFO [Index 'ATLANTA-FS' FS timer.1]
FriendshipIndexServiceImpl = No index changes detected since last update at
09:10:19, for Friendship Server ATLANTA-FS.
WO_LogLevel
INFO
WO_Log_Source
RAS-CS
WO_Message
No index changes detected since last update at 09:10:19, for Friendship
Server ATLANTA-FS.
WO_Process
Index 'ATLANTA-FS' FS timer.1
WO_SubProcess
FriendshipIndexServiceImpl
WO_Timestamp
2017-02-08 09:41:40,674
facility
filebeat
file
d:\centralserver\ras-server\log\ras_cs_WO-ATL-CS.log
input_type
log
message
2017-02-08 09:41:40,674 INFO [Index 'ATLANTA-FS' FS timer.1]
FriendshipIndexServiceImpl = No index changes detected since last update at
09:10:19, for Friendship Server ATLANTA-FS.
name
WO-ATL-CS
offset
6719671
source
WO-ATL-CS
timestamp
2017-02-08T14:41:47.617Z
type
log

MSG 02 Sample:
WO_CS_RAS_CS_MESSAGE
2017-02-08 09:42:30,056 INFO [Device message listener] GpioRoleImpl = gpi
axia_pathfinder_-1856806973 pin 1582 set on
WO_LogLevel
INFO
WO_Log_Source
RAS-CS
WO_Message
gpi axia_pathfinder_-1856806973 pin 1582 set on
WO_Process
Device message listener
WO_SubProcess
GpioRoleImpl
WO_Timestamp
2017-02-08 09:42:30,056
facility
filebeat
file
d:\centralserver\ras-server\log\ras_cs_WO-ATL-CS.log
input_type
log
message
2017-02-08 09:42:30,056 INFO [Device message listener] GpioRoleImpl = gpi
axia_pathfinder_-1856806973 pin 1582 set on
name
WO-ATL-CS
offset
7014109
source
WO-ATL-CS
timestamp
2017-02-08T14:42:32.698Z
type
log


Those messages are with the date parsing disabled. I'm attempting to
replace "timestamp" with the "WO_Timestamp" field.

Side note: The full_message field is empty on my filebeat inputs--is that
expected behavior?


Thanks!

Cheers,
Al

Post by Jochen Schalanda
Hi Al,

I was under the impression that using the "parse_date" function would
create a Date object?

It does, see
http://docs.graylog.org/en/2.1/pages/pipelines/functions.html#parse-date
for reference. But your date pattern may be wrong (see
http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html
for reference).
Please share some example messages, so that we can validate your rule.

As for "$timestamp" instead of "timestamp", I was trying different
configurations, and thought that since the message field is referenced as
"$message" I would try that format. What does the "$" indicate?

The $ character is simply part of the variable name containing the current
message (which is "$message"). It doesn't have a special meaning.
Cheers,
Jochen

[Jochen Schalanda]

Hi Al,

Post by Al Reynolds
WO_Timestamp
2017-02-08 09:42:30,056
Those messages are with the date parsing disabled. I'm attempting to
replace "timestamp" with the "WO_Timestamp" field.

The string in WO_Timestamp doesn't match the pattern "YYYY-MM-dd
HH:mm:ss,sss" used in parse_date().
See http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html
for details.

Hint: 's' and 'S' are not the same thing.


Side note: The full_message field is empty on my filebeat inputs--is that

Post by Al Reynolds
expected behavior?

Yes, that's expected.

What would you expect to find in the (optional) full_message field?

Cheers,
Jochen

[Al Reynolds]

That's what I get for typing it out...thank you for catching that!
Unfortunately, even after correcting for the incorrect milliseconds value,
it's still not replacing timestamp value. I sent the parsed date to a new
field (in this case, "log_timestamp") to verify that the output data was in
the correct format, which it is now, but it still won't replace the
timestamp field.

Message sample with "log_timestamp" field:
WO_CS_RAS_CS_MESSAGE
2017-02-08 11:00:34,980 WARN [Task 'ATLANTA-FS' FS timer.1]
FriendshipTasksServiceImpl = Could not obtain task info for: 2c95ac8e-57e3-
91b2-0158-495b880b24e8REQUEST FAILED ==> STATUS CODE: 404, RESPONSE BODY:
WO_LogLevel
WARN
WO_Log_Source
RAS-CS
WO_Message
Could not obtain task info for: 2c95ac8e-57e3-91b2-0158-495b880b24e8REQUEST
FAILED ==> STATUS CODE: 404, RESPONSE BODY:
WO_Process
Task 'ATLANTA-FS' FS timer.1
WO_SubProcess
FriendshipTasksServiceImpl
WO_Timestamp
2017-02-08 11:00:34,980
facility
filebeat
file
d:\centralserver\ras-server\log\ras_cs_WO-ATL-CS.log
input_type
log
log_timestamp
2017-02-08T11:00:34.980Z
message
2017-02-08 11:00:34,980 WARN [Task 'ATLANTA-FS' FS timer.1]
FriendshipTasksServiceImpl = Could not obtain task info for: 2c95ac8e-57e3-
91b2-0158-495b880b24e8REQUEST FAILED ==> STATUS CODE: 404, RESPONSE BODY:
name
WO-ATL-CS
offset
2372156
source
WO-ATL-CS
timestamp
2017-02-08T16:00:35.864Z
type
log

Corrected rule:
rule "WO-CS-RAS"
when

contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
then
set_field("WO_Log_Source","RAS-CS");
let matches = grok(pattern: "%{WO_CS_RAS_CS_MESSAGE}", value:
to_string($message.message));
set_fields(matches);
let date = parse_date(to_string($message.WO_Timestamp), "YYYY-MM-dd
HH:mm:ss,SSS");
set_field("timestamp", date);
route_to_stream("WideOrbit Logs");
end

Thanks!

Cheers,
Al

Post by Jochen Schalanda
Hi Al,

Post by Al Reynolds
WO_Timestamp
2017-02-08 09:42:30,056
Those messages are with the date parsing disabled. I'm attempting to
replace "timestamp" with the "WO_Timestamp" field.

The string in WO_Timestamp doesn't match the pattern "YYYY-MM-dd
HH:mm:ss,sss" used in parse_date(). See
http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html
for details.
Hint: 's' and 'S' are not the same thing.
Side note: The full_message field is empty on my filebeat inputs--is that

Post by Al Reynolds
expected behavior?

Yes, that's expected.
What would you expect to find in the (optional) full_message field?
Cheers,
Jochen

[Al Reynolds]

Figured it out--parse_date needed the timestamp . New rule looks like this:
rule "WO-CS-RAS"
when

contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
then
set_field("WO_Log_Source","RAS-CS");
let matches = grok(pattern: "%{WO_CS_RAS_CS_MESSAGE}", value:
to_string($message.message));
set_fields(matches);
let date = parse_date(to_string($message.WO_Timestamp), "YYYY-MM-dd
HH:mm:ss,SSS", "EST");
set_field("timestamp", date);
route_to_stream("WideOrbit Logs");
end

I was under the impression that the timezone was optional?

Thanks for all your help with this Jochen--it's greatly appreciated!

Cheers,
Al

Post by Al Reynolds
That's what I get for typing it out...thank you for catching that!
Unfortunately, even after correcting for the incorrect milliseconds value,
it's still not replacing timestamp value. I sent the parsed date to a new
field (in this case, "log_timestamp") to verify that the output data was in
the correct format, which it is now, but it still won't replace the
timestamp field.
WO_CS_RAS_CS_MESSAGE
2017-02-08 11:00:34,980 WARN [Task 'ATLANTA-FS' FS timer.1]
2c95ac8e-57e3-91b2-0158-495b880b24e8REQUEST FAILED ==> STATUS CODE: 404,
WO_LogLevel
WARN
WO_Log_Source
RAS-CS
WO_Message
Could not obtain task info for: 2c95ac8e-57e3-91b2-0158-
WO_Process
Task 'ATLANTA-FS' FS timer.1
WO_SubProcess
FriendshipTasksServiceImpl
WO_Timestamp
2017-02-08 11:00:34,980
facility
filebeat
file
d:\centralserver\ras-server\log\ras_cs_WO-ATL-CS.log
input_type
log
log_timestamp
2017-02-08T11:00:34.980Z
message
2017-02-08 11:00:34,980 WARN [Task 'ATLANTA-FS' FS timer.1]
2c95ac8e-57e3-91b2-0158-495b880b24e8REQUEST FAILED ==> STATUS CODE: 404,
name
WO-ATL-CS
offset
2372156
source
WO-ATL-CS
timestamp
2017-02-08T16:00:35.864Z
type
log
rule "WO-CS-RAS"
when
contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
then
set_field("WO_Log_Source","RAS-CS");
to_string($message.message));
set_fields(matches);
let date = parse_date(to_string($message.WO_Timestamp), "YYYY-MM-dd
HH:mm:ss,SSS");
set_field("timestamp", date);
route_to_stream("WideOrbit Logs");
end
Thanks!
Cheers,
Al

Post by Jochen Schalanda
Hi Al,

Post by Al Reynolds
WO_Timestamp
2017-02-08 09:42:30,056
Those messages are with the date parsing disabled. I'm attempting to
replace "timestamp" with the "WO_Timestamp" field.

The string in WO_Timestamp doesn't match the pattern "YYYY-MM-dd
HH:mm:ss,sss" used in parse_date(). See
http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html
for details.
Hint: 's' and 'S' are not the same thing.
Side note: The full_message field is empty on my filebeat inputs--is that

Post by Al Reynolds
expected behavior?

Yes, that's expected.
What would you expect to find in the (optional) full_message field?
Cheers,
Jochen

[Al Reynolds]

I've noticed another error. The timestamp field is being replaced
correctly, but the "gl2_processing_error" field is showing the following
error (on all messages):
For rule 'WO-CS-RAS': In call to function 'parse_date' at 8:15 an exception
was thrown: Invalid format: "2017-02-08 15:05:59,170" is malformed at "17-02-08
15:05:59,170"


It doesn't seem to have any adverse effects, but I'm curious as to what
might be causing it?

Post by Al Reynolds
rule "WO-CS-RAS"
when
contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
then
set_field("WO_Log_Source","RAS-CS");
to_string($message.message));
set_fields(matches);
let date = parse_date(to_string($message.WO_Timestamp), "YYYY-MM-dd
HH:mm:ss,SSS", "EST");
set_field("timestamp", date);
route_to_stream("WideOrbit Logs");
end
I was under the impression that the timezone was optional?
Thanks for all your help with this Jochen--it's greatly appreciated!
Cheers,
Al

Post by Al Reynolds
That's what I get for typing it out...thank you for catching that!
Unfortunately, even after correcting for the incorrect milliseconds value,
it's still not replacing timestamp value. I sent the parsed date to a new
field (in this case, "log_timestamp") to verify that the output data was in
the correct format, which it is now, but it still won't replace the
timestamp field.
WO_CS_RAS_CS_MESSAGE
2017-02-08 11:00:34,980 WARN [Task 'ATLANTA-FS' FS timer.1]
2c95ac8e-57e3-91b2-0158-495b880b24e8REQUEST FAILED ==> STATUS CODE: 404,
WO_LogLevel
WARN
WO_Log_Source
RAS-CS
WO_Message
Could not obtain task info for: 2c95ac8e-57e3-91b2-0158-
WO_Process
Task 'ATLANTA-FS' FS timer.1
WO_SubProcess
FriendshipTasksServiceImpl
WO_Timestamp
2017-02-08 11:00:34,980
facility
filebeat
file
d:\centralserver\ras-server\log\ras_cs_WO-ATL-CS.log
input_type
log
log_timestamp
2017-02-08T11:00:34.980Z
message
2017-02-08 11:00:34,980 WARN [Task 'ATLANTA-FS' FS timer.1]
2c95ac8e-57e3-91b2-0158-495b880b24e8REQUEST FAILED ==> STATUS CODE: 404,
name
WO-ATL-CS
offset
2372156
source
WO-ATL-CS
timestamp
2017-02-08T16:00:35.864Z
type
log
rule "WO-CS-RAS"
when
contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
then
set_field("WO_Log_Source","RAS-CS");
to_string($message.message));
set_fields(matches);
let date = parse_date(to_string($message.WO_Timestamp), "YYYY-MM-dd
HH:mm:ss,SSS");
set_field("timestamp", date);
route_to_stream("WideOrbit Logs");
end
Thanks!
Cheers,
Al

Post by Jochen Schalanda
Hi Al,

Post by Al Reynolds
WO_Timestamp
2017-02-08 09:42:30,056
Those messages are with the date parsing disabled. I'm attempting to
replace "timestamp" with the "WO_Timestamp" field.

The string in WO_Timestamp doesn't match the pattern "YYYY-MM-dd
HH:mm:ss,sss" used in parse_date(). See
http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html
for details.
Hint: 's' and 'S' are not the same thing.
Side note: The full_message field is empty on my filebeat inputs--is

Post by Al Reynolds
that expected behavior?

Yes, that's expected.
What would you expect to find in the (optional) full_message field?
Cheers,
Jochen

[Jochen Schalanda]

Hi Al,

you might want to try to use your pattern with lower case 'y' for the year
component of the date pattern.

Cheers,
Jochen

Post by Al Reynolds
I've noticed another error. The timestamp field is being replaced
correctly, but the "gl2_processing_error" field is showing the following
For rule 'WO-CS-RAS': In call to function 'parse_date' at 8:15 an
exception was thrown: Invalid format: "2017-02-08 15:05:59,170" is
malformed at "17-02-08 15:05:59,170"
It doesn't seem to have any adverse effects, but I'm curious as to what
might be causing it?

Post by Al Reynolds
rule "WO-CS-RAS"
when
contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
then
set_field("WO_Log_Source","RAS-CS");
to_string($message.message));
set_fields(matches);
let date = parse_date(to_string($message.WO_Timestamp), "YYYY-MM-dd
HH:mm:ss,SSS", "EST");
set_field("timestamp", date);
route_to_stream("WideOrbit Logs");
end
I was under the impression that the timezone was optional?
Thanks for all your help with this Jochen--it's greatly appreciated!
Cheers,
Al

Post by Al Reynolds
That's what I get for typing it out...thank you for catching that!
Unfortunately, even after correcting for the incorrect milliseconds value,
it's still not replacing timestamp value. I sent the parsed date to a new
field (in this case, "log_timestamp") to verify that the output data was in
the correct format, which it is now, but it still won't replace the
timestamp field.
WO_CS_RAS_CS_MESSAGE
2017-02-08 11:00:34,980 WARN [Task 'ATLANTA-FS' FS timer.1]
2c95ac8e-57e3-91b2-0158-495b880b24e8REQUEST FAILED ==> STATUS CODE: 404,
WO_LogLevel
WARN
WO_Log_Source
RAS-CS
WO_Message
Could not obtain task info for: 2c95ac8e-57e3-91b2-0158-
WO_Process
Task 'ATLANTA-FS' FS timer.1
WO_SubProcess
FriendshipTasksServiceImpl
WO_Timestamp
2017-02-08 11:00:34,980
facility
filebeat
file
d:\centralserver\ras-server\log\ras_cs_WO-ATL-CS.log
input_type
log
log_timestamp
2017-02-08T11:00:34.980Z
message
2017-02-08 11:00:34,980 WARN [Task 'ATLANTA-FS' FS timer.1]
2c95ac8e-57e3-91b2-0158-495b880b24e8REQUEST FAILED ==> STATUS CODE: 404,
name
WO-ATL-CS
offset
2372156
source
WO-ATL-CS
timestamp
2017-02-08T16:00:35.864Z
type
log
rule "WO-CS-RAS"
when
contains(to_string($message.file),"centralserver\\ras-server\\log\\ras_cs_")
then
set_field("WO_Log_Source","RAS-CS");
to_string($message.message));
set_fields(matches);
let date = parse_date(to_string($message.WO_Timestamp), "YYYY-MM-dd
HH:mm:ss,SSS");
set_field("timestamp", date);
route_to_stream("WideOrbit Logs");
end
Thanks!
Cheers,
Al

Post by Jochen Schalanda
Hi Al,

Post by Al Reynolds
WO_Timestamp
2017-02-08 09:42:30,056
Those messages are with the date parsing disabled. I'm attempting to
replace "timestamp" with the "WO_Timestamp" field.

The string in WO_Timestamp doesn't match the pattern "YYYY-MM-dd
HH:mm:ss,sss" used in parse_date(). See
http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html
for details.
Hint: 's' and 'S' are not the same thing.
Side note: The full_message field is empty on my filebeat inputs--is

Post by Al Reynolds
that expected behavior?

Yes, that's expected.
What would you expect to find in the (optional) full_message field?
Cheers,
Jochen