[graylog2] Remove field using extractos

[Rui Goncalves]

Hi all.

I'm receiving messages following the pattern key=value. I'd like to set the
value of two of the received keys on graylog standard fields, namely
"message" and "timestamp" and discard the original fields completely.

I can use the "key=value" converter, then "copy" the original field's
content and set the value on graylog fields, however the unnecessary fields
(msg, and time) remain on the log. Using the cut option causes the field
value to be set with the value "fullyCutByExtractor".

The question is: Why it's not possible to remove a field from the received
message using extractors? I believe having to drop fields is something
usual.?! I've checked the docs (haven't tried yet), and it's possible to
accomplish what I want using the pipelines feature. However it's in an
experimental phase (with potential stability and performance issues) and it
seems overkill for doing something so simple as dropping a field.

Thanks.

[Jochen Schalanda]

Hi Rui,

Why it's not possible to remove a field from the received message using
extractors?

This was a deliberate decision at the time to prevent people from wondering
why some field didn't exist anymore due to stacked or complicated
extractors.

However it's in an experimental phase (with potential stability and
performance issues) and it seems overkill for doing something so simple as
dropping a field.

The message processing pipelines aren't experimental anymore in Graylog
2.2.0.

Cheers,
Jochen

[Rui Goncalves]

Thanks Jochen.

I'm looking at graylog pipelines docs, but I think I'm really confused :-/

I've created a pipeline with one rule that extracts key=value pairs:

rule "Extract K=V"
when true
then
set_fields(key_value(to_string($message.message)));
end

Then I've created a stream of messages, where only messages with the
intended format will pass through. After that, I've connected the stream to
the pipeline. Executing the "Simulate processing" function, fields get
extracted as expected. However, graylog keeps indexing the original message
into elasticsearch!

What am I missing? I think there must be something that I'm missing,
because we can route the same message to multiple streams. If it worked the
way I'm thinking, we would end up with duplicated messages on elasticsearch.

I've also looked at stream outputs, but there's no ES output. :-/

Can you shed some light here please?

Thanks.

Post by Jochen Schalanda
Hi Rui,

Why it's not possible to remove a field from the received message using
extractors?

This was a deliberate decision at the time to prevent people from
wondering why some field didn't exist anymore due to stacked or complicated
extractors.

However it's in an experimental phase (with potential stability and
performance issues) and it seems overkill for doing something so simple as
dropping a field.

The message processing pipelines aren't experimental anymore in Graylog
2.2.0.
Cheers,
Jochen

[Jochen Schalanda]

Hi Rui,

Post by Rui Goncalves
What am I missing? I think there must be something that I'm missing,
because we can route the same message to multiple streams.

This sounds like an incorrect message processor order. Check the order of
your message processors on the System / Configurations page.

Cheers,
Jochen