[graylog2] How to log from a java application using logback to a file in a way that won't need grok filtering before getting to graylog

[Joseph DJOMEDA]

Hello All,

I am having a issue with search accuracy because, I suspect of the way we
log see below an example:

*03 Jan 2015 ;13:51:40.767
[myActorSystem-akka.actor.default-dispatcher-3859] INFO
c.d.m.c.m.l.MessageListenerImpl -
[MsgReceived][email:***@clientemail.com][account:700774779919551][batchId:1420209848423][msgId:1420209848425][recipient:xxxxxxxxxxxx]*

The block : "
* [MsgReceived][email:***@clientemail.com][account:700774779919551][batchId:1420209848423][msgId:1420209848425]*"
is inconsistent and might have more or less "[ ]" block based on the type
of request is been logges. That has posed a little issue for me writing a
grok filter so I had to treat all as loginfo to be sure nothing is left out.

Below is the grok filter I am using currently :

* if [type] == "my-app" {*

* grok {*
* match => { "message" => "%{MONTHDAY:day} %{MONTH:month}
%{YEAR:year} ;%{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second}
\[%{GREEDYDATA:thread}\] %{LOGLEVEL:loglevel} %{GREEDYDATA:classinfo}
-%{GREEDYDATA:loginfo}" }*
* }*
* }*

The purpose of this post is to have some kind of guidelines to I can advise
our developers how to log: I was thinking whether splitting the above entry
to the following one would work.

*date: "03 Jan 2015 ;13:51:40.767" thread:
myActorSystem-akka.actor.default-dispatcher-3859 loglevel: INFO logger:
c.d.m.c.m.l.MessageListenerImpl requesttype: MsgReceived accountemail:
***@clietnemail.com accountid: 700774779919551 batchId:1420209848423
msgId:1420209848425 recipient: xxxxxxxxx*

So I am aware of some gelf appenders for logback but it looks to me that
they are directly logging into graylog. What I would like to have is
leveraging on my centralized log management system having
logstash-forwarder use the selfsigned SSL to the central logstash. So is
there a way to log into file in a way that is gelf compatible?

Best Regards,

[Jochen Schalanda]

Hello Joseph,

you can use the Logback JSON encoder for Logstash (
https://github.com/logstash/logstash-logback-encoder) with any Logback
appender to write JSON which is compatible with logstash.


Cheers,
Jochen

Post by Joseph DJOMEDA
Hello All,
I am having a issue with search accuracy because, I suspect of the way we
*03 Jan 2015 ;13:51:40.767
[myActorSystem-akka.actor.default-dispatcher-3859] INFO
<javascript:>][account:700774779919551][batchId:1420209848423][msgId:1420209848425][recipient:xxxxxxxxxxxx]*
<javascript:>][account:700774779919551][batchId:1420209848423][msgId:1420209848425]*"
is inconsistent and might have more or less "[ ]" block based on the type
of request is been logges. That has posed a little issue for me writing a
grok filter so I had to treat all as loginfo to be sure nothing is left out.
* if [type] == "my-app" {*
* grok {*
* match => { "message" => "%{MONTHDAY:day} %{MONTH:month}
%{YEAR:year} ;%{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second}
\[%{GREEDYDATA:thread}\] %{LOGLEVEL:loglevel} %{GREEDYDATA:classinfo}
-%{GREEDYDATA:loginfo}" }*
* }*
* }*
The purpose of this post is to have some kind of guidelines to I can
advise our developers how to log: I was thinking whether splitting the
above entry to the following one would work.
batchId:1420209848423 msgId:1420209848425 recipient: xxxxxxxxx*
So I am aware of some gelf appenders for logback but it looks to me that
they are directly logging into graylog. What I would like to have is
leveraging on my centralized log management system having
logstash-forwarder use the selfsigned SSL to the central logstash. So is
there a way to log into file in a way that is gelf compatible?
Best Regards,

[Joseph DJOMEDA]

Thanks Jochen

From your suggestion I understand that I can leverage on this logback json
appender to write logs that would meet graylogs gelf specifications as in
make sure the gelf required fields are part of each entry and that the
custom fields have a prepending underscore.

Correct me if you think I got you wrong.
Best Regards

Sent from mobile device... Please excuse the typos

Post by Jochen Schalanda
Hello Joseph,
you can use the Logback JSON encoder for Logstash (
https://github.com/logstash/logstash-logback-encoder) with any Logback
appender to write JSON which is compatible with logstash.
Cheers,
Jochen

Post by Joseph DJOMEDA
Hello All,
I am having a issue with search accuracy because, I suspect of the way we
*03 Jan 2015 ;13:51:40.767
[myActorSystem-akka.actor.default-dispatcher-3859] INFO
c.d.m.c.m.l.MessageListenerImpl -
The block : "
is inconsistent and might have more or less "[ ]" block based on the type
of request is been logges. That has posed a little issue for me writing a
grok filter so I had to treat all as loginfo to be sure nothing is left out.
* if [type] == "my-app" {*
* grok {*
* match => { "message" => "%{MONTHDAY:day} %{MONTH:month}
%{YEAR:year} ;%{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second}
\[%{GREEDYDATA:thread}\] %{LOGLEVEL:loglevel} %{GREEDYDATA:classinfo}
-%{GREEDYDATA:loginfo}" }*
* }*
* }*
The purpose of this post is to have some kind of guidelines to I can
advise our developers how to log: I was thinking whether splitting the
above entry to the following one would work.
msgId:1420209848425 recipient: xxxxxxxxx*
So I am aware of some gelf appenders for logback but it looks to me that
they are directly logging into graylog. What I would like to have is
leveraging on my centralized log management system having
logstash-forwarder use the selfsigned SSL to the central logstash. So is
there a way to log into file in a way that is gelf compatible?
Best Regards,
--

You received this message because you are subscribed to a topic in the
Google Groups "graylog2" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/graylog2/axWa2AJ9ybY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
For more options, visit https://groups.google.com/d/optout.

[Jochen Schalanda]

Hello Joseph,

I think we had a small misunderstanding here. The Logback JSON encoder for
Logstash can generate log messages which are compatible with logstash but
these are not valid GELF messages. You could grab those files with
logstash, though, and send the messages to your central logstash server
with minimal post-processing (since they are already in a format which
logstash understands).

I am not aware of any Encoder <http://logback.qos.ch/manual/encoders.html>
for Logback which supports the GELF message format.


Cheers,
Jochen

Post by Joseph DJOMEDA
Thanks Jochen
From your suggestion I understand that I can leverage on this logback json
appender to write logs that would meet graylogs gelf specifications as in
make sure the gelf required fields are part of each entry and that the
custom fields have a prepending underscore.
Correct me if you think I got you wrong.
Best Regards
Sent from mobile device... Please excuse the typos

Post by Jochen Schalanda
Hello Joseph,
you can use the Logback JSON encoder for Logstash (
https://github.com/logstash/logstash-logback-encoder) with any Logback
appender to write JSON which is compatible with logstash.
Cheers,
Jochen

Post by Joseph DJOMEDA
Hello All,
I am having a issue with search accuracy because, I suspect of the way
*03 Jan 2015 ;13:51:40.767
[myActorSystem-akka.actor.default-dispatcher-3859] INFO
c.d.m.c.m.l.MessageListenerImpl -
The block : "
is inconsistent and might have more or less "[ ]" block based on the type
of request is been logges. That has posed a little issue for me writing a
grok filter so I had to treat all as loginfo to be sure nothing is left out.
* if [type] == "my-app" {*
* grok {*
* match => { "message" => "%{MONTHDAY:day} %{MONTH:month}
%{YEAR:year} ;%{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second}
\[%{GREEDYDATA:thread}\] %{LOGLEVEL:loglevel} %{GREEDYDATA:classinfo}
-%{GREEDYDATA:loginfo}" }*
* }*
* }*
The purpose of this post is to have some kind of guidelines to I can
advise our developers how to log: I was thinking whether splitting the
above entry to the following one would work.
msgId:1420209848425 recipient: xxxxxxxxx*
So I am aware of some gelf appenders for logback but it looks to me that
they are directly logging into graylog. What I would like to have is
leveraging on my centralized log management system having
logstash-forwarder use the selfsigned SSL to the central logstash. So is
there a way to log into file in a way that is gelf compatible?
Best Regards,
--

You received this message because you are subscribed to a topic in the
Google Groups "graylog2" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/graylog2/axWa2AJ9ybY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
For more options, visit https://groups.google.com/d/optout.