[graylog2] Copy input extractor failure

[Rayees Namathponnan]

Hi All,

I created “Copy Input” extractor to get key value pair, here is my message, trying to extract level and status from all the messages



2016-09-28 19:21:52,466 level=INFO tag="run_workflow.py" msg="Run complete for appname=cils, job_date=20160912, status=Passed starttime=Wed Sep 28 19:15:25 2016, endtime=Wed Sep 28 19:21:47 2016, duration=0:6:21, inputs=[{"path": "/esss/srg/20160912_1473688239855_4f29bbb6efdb3c39", "tag": "cilPurge", "stats": {"size": "13.47MB"}}, {"path": "/compressed/cil/20160912", "tag": "cil", "stats": {"size": "580.16MB"}}], outputs=[{"path": "/processed/test/parse//cil", "tag": "cil.output.folder", "stats": {"diffSize": "645.78MB", "newFiles": [], "endSize": "18.93GB", "startSize": "18.30GB"}}]”



Its failing with below error

0]: index [graylog_8], type [message], id [c0e7ea80-f572-11e6-b21e-5254007b267d], message [java.lang.IllegalArgumentException: Document contains at least one immense term in field="level" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[70, 101, 98, 32, 49, 55, 32, 49, 57, 58, 51, 57, 58, 51, 53, 32, 115, 106, 101, 108, 107, 51, 51, 32, 115, 121, 115, 116, 101, 109]...', original message: bytes can be at most 32766 in length; got 34944]
2017-02-17T19:39:56.795-05:00 ERROR [Extractor] Could not apply converter [tokenizer] of extractor [77e451d0-f3b9-11e6-b21e-5254007b267d].
java.lang.IllegalArgumentException: Multiple entries with same key: id=4038, and id=4038,
at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:136) ~[graylog.jar:?]
at com.google.common.collect.RegularImmutableMap.checkNoConflictInKeyBucket(RegularImmutableMap.java:98) ~[graylog.jar:?]
at com.google.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:84) ~[graylog.jar:?]
at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:295) ~[graylog.jar:?]
at org.graylog2.inputs.converters.TokenizerConverter.convert(TokenizerConverter.java:55) ~[graylog.jar:?]
at org.graylog2.plugin.inputs.Extractor.runConverters(Extractor.java:242) [graylog.jar:?]
at org.graylog2.plugin.inputs.Extractor.runExtractor(Extractor.java:228) [graylog.jar:?]
at org.graylog2.filters.ExtractorFilter.filter(ExtractorFilter.java:73) [graylog.jar:?]
at org.graylog2.messageprocessors.MessageFilterChainProcessor.process(MessageFilterChainProcessor.java:100) [graylog.jar:?]
at org.graylog2.buffers.processors.ServerProcessBufferProcessor.handleMessage(ServerProcessBufferProcessor.java:56) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.dispatchMessage(ProcessBufferProcessor.java:82) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:61) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:35) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
2017-02-17T19:39:57.677-05:00 ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:
[0]: index [graylog_8], type [message], id [c560a160-f572-11e6-b21e-5254007b267d], message [java.lang.IllegalArgumentException: Document contains at least one immense term in field="level" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[70, 101, 98, 32, 49, 55, 32, 49, 57, 58, 51, 57, 58, 52, 53, 32, 115, 106, 101, 108, 107, 51, 51, 32, 115, 121, 115, 116, 101, 109]...', original message: bytes can be at most 32766 in length; got 34944]
^Z

[Jochen Schalanda]

Hi Rayees,

which version of Graylog and Elasticsearch are you running?

Cheers,
Jochen

Post by Rayees Namathponnan
Hi All,
I created “Copy Input” extractor to get key value pair, here is my
message, trying to extract *level* and *status* from all the messages
2016-09-28 19:21:52,466 level=INFO tag="run_workflow.py" msg="Run complete
for appname=cils, job_date=20160912, status=Passed starttime=Wed Sep 28
19:15:25 2016, endtime=Wed Sep 28 19:21:47 2016, duration=0:6:21,
inputs=[{"path": "/esss/srg/20160912_1473688239855_4f29bbb6efdb3c39",
"/compressed/cil/20160912", "tag": "cil", "stats": {"size": "580.16MB"}}],
outputs=[{"path": "/processed/test/parse//cil", "tag": "cil.output.folder",
"stats": {"diffSize": "645.78MB", "newFiles": [], "endSize": "18.93GB",
"startSize": "18.30GB"}}]”
Its failing with below error
0]: index [graylog_8], type [message], id
[c0e7ea80-f572-11e6-b21e-5254007b267d], message
[java.lang.IllegalArgumentException: Document contains at least one immense
term in field="level" (whose UTF8 encoding is longer than the max length
32766), all of which were skipped. Please correct the analyzer to not
produce such terms. The prefix of the first immense term is: '[70, 101,
98, 32, 49, 55, 32, 49, 57, 58, 51, 57, 58, 51, 53, 32, 115, 106, 101, 108,
107, 51, 51, 32, 115, 121, 115, 116, 101, 109]...', original message: bytes
can be at most 32766 in length; got 34944]
2017-02-17T19:39:56.795-05:00 ERROR [Extractor] Could not apply converter
[tokenizer] of extractor [77e451d0-f3b9-11e6-b21e-5254007b267d].
java.lang.IllegalArgumentException: Multiple entries with same key: id=4038, and id=4038,
at
com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:136)
~[graylog.jar:?]
at
com.google.common.collect.RegularImmutableMap.checkNoConflictInKeyBucket(RegularImmutableMap.java:98)
~[graylog.jar:?]
at
com.google.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:84)
~[graylog.jar:?]
at
com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:295)
~[graylog.jar:?]
at
org.graylog2.inputs.converters.TokenizerConverter.convert(TokenizerConverter.java:55)
~[graylog.jar:?]
at org.graylog2.plugin.inputs.Extractor.runConverters(Extractor.java:242) [graylog.jar:?]
at org.graylog2.plugin.inputs.Extractor.runExtractor(Extractor.java:228) [graylog.jar:?]
at org.graylog2.filters.ExtractorFilter.filter(ExtractorFilter.java:73) [graylog.jar:?]
at
org.graylog2.messageprocessors.MessageFilterChainProcessor.process(MessageFilterChainProcessor.java:100)
[graylog.jar:?]
at
org.graylog2.buffers.processors.ServerProcessBufferProcessor.handleMessage(ServerProcessBufferProcessor.java:56)
[graylog.jar:?]
at
org.graylog2.shared.buffers.processors.ProcessBufferProcessor.dispatchMessage(ProcessBufferProcessor.java:82)
[graylog.jar:?]
at
org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:61)
[graylog.jar:?]
at
org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:35)
[graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at
com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66)
[graylog.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
2017-02-17T19:39:57.677-05:00 ERROR [Messages] Failed to index [1]
messages. Please check the index error log in your web interface for the
[0]: index [graylog_8], type [message], id
[c560a160-f572-11e6-b21e-5254007b267d], message
[java.lang.IllegalArgumentException: Document contains at least one immense
term in field="level" (whose UTF8 encoding is longer than the max length
32766), all of which were skipped. Please correct the analyzer to not
produce such terms. The prefix of the first immense term is: '[70, 101,
98, 32, 49, 55, 32, 49, 57, 58, 51, 57, 58, 52, 53, 32, 115, 106, 101, 108,
107, 51, 51, 32, 115, 121, 115, 116, 101, 109]...', original message: bytes
can be at most 32766 in length; got 34944]
^Z