Tagged: ruby

Logstash: Failed to flush outgoing items UndefinedConversionError workaround

If you have ever seen an error similar to this in Logstash it can be frustrating and can take your whole pipeline down (blocks). It appears that there are some outstanding tickets on this, one of which is here. This error can occur if you have an upstream input where the charset is defined as US-ASCII (such as from an ASCII file input), where that file contains some extended chars, and then being sent to an output where it needs to be converted (in my case it was elasticsearch_http). This was w/ Logstash 1.4.1

:message=>"Failed to flush outgoing items", :outgoing_count=>1,
:exception=>#<Encoding::UndefinedConversionError: ""\xC2"" from ASCII-8BIT to UTF-8>

For anyone else out there, here is a simple workaround fix I put in my filters which pre-processes messages from “known” upstream ASCII inputs. This fixed it for me: (using the Ruby code filter)

ruby {
code => "begin; if !event['message'].nil?; event['message'] = event['message'].force_encoding('ASCII-8BIT').encode('UTF-8', :invalid => :replace, :undef => :replace, :replace => '?'); end; rescue; end;"
}

To create a little log file to reproduce this try:

perl -e 'print "\xc2\xa0\n"' > test.log

Processing ModSecurity audit logs with Fluentd

Recently had a need to take tons of raw ModSecurity audit logs and make use of them. First used Logstash and then attempted with Apache Flume (see previous articles). Next in line was Fluentd which is what this article is about, long story short I ended up just having to write a Fluentd output plugin to take the output from the tail multiline plugin and then format it into a more structured first class object that looks like the below example.

The Modsecurity Fluentd plugin is located here on Github: https://github.com/bitsofinfo/fluentd-modsecurity

  1. Get some audit logs generated from modsecurity and throw them into a directory
  2. Edit your fluentd config file and customize its input to use the tail multiline plugin and then the modsecurity plugin, an example is here.
  3. Customize your output(s)
  4. On the command line:  “fluentd ./fluent.conf -vv” 

This was tested against the latest version of Fluentd available at the time of this article.

The end result of it is that with this configuration, your raw Modsec audit log entries, will end up looking something like this JSON example below. Again this is just how I ended up structuring the fields via the filters, you can fork and modify the plugin as you see fit to output a different format, or even make it more configurable

EXAMPLE JSON OUTPUT, using https://github.com/bitsofinfo/fluentd-modsecurity

{
"modsec_timestamp":"08/Nov/2013:06:22:59 --0400",
"uniqueId":"C5g8kkk0002012221222",
"sourceIp":"192.168.1.22",
"sourcePort":"34156",
"destIp":"192.168.0.2",
"destPort":"80",
"httpMethod":"GET",
"requestedUri":"/myuri/x",
"incomingProtocol":"HTTP/1.1",
"myCookie":"myCookie=testValue",
"requestHeaders":{
"Host":"accts.x4.bitsofinfo2.com",
"Connection":"keep-alive",
"Accept":"*/*",
"User-Agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) Safari/537.36",
"Referer":"https",
"Accept-Encoding":"gzip,deflate,sdch",
"Accept-Language":"en-US,en;q=0.8",
"Cookie":"myCookie=testValue; myapp_sec_7861ac9196050da; special=ddd",
"Incoming-Protocol":"HTTPS",
"X-Forwarded-For":"192.1.33.22"
},
"XForwardedFor":"192.1.33.22",
"XForwardedFor-GEOIP":{
      "country_code":"TW",
      "country_code3":"TWN",
      "country_name":"Taiwan",
      "region":"03",
      "region_name":"T'ai-pei",
      "city":"Taipei",
      "latitude":25.039199829101562,
      "longitude":121.5250015258789
   },
"serverProtocol":"HTTP/1.1",
"responseStatus":"200 OK",
"responseHeaders":{
"Vary":"Accept-Encoding",
"Expires":"Fri, 08 Aug 2014 10",
"Cache-Control":"public, max-age=31536000",
"Content-Encoding":"deflate",
"Content-Type":"application/x-javascript; charset=UTF-8",
"Set-Cookie":"zippy=65.sss31; path=/; domain=accts.x4.bitsofinfo2.com",
"Connection":"close",
"Transfer-Encoding":"chunked"
},
"auditLogTrailer":{
"Apache-Handler":"proxy-server",
"Stopwatch":"1375957379601874 39178 (989 4992 -)",
"Producer":"ModSecurity for Apache (http://www.modsecurity.org/); core ruleset",
"Server":"Apache (party6)",
"messages":[
{
"info":"Warning 1. Operator EQ matched 0 at GLOBAL.",
"file":"/etc/d4/modsechttp_policy.conf",
"line":"120",
"id":"960903",
"msg":"ModSecurity does not support content encodings",
"severity":"WARNING"
},
{
"info":"Warning 2. Operator EQ matched 0 at GLOBAL.",
"file":"/etc/d4/modsechttp_policy.conf",
"line":"120",
"id":"960903",
"msg":"ModSecurity does not support content encodings",
"severity":"WARNING"
}
]
},
"event_date_microseconds":1.375957379601874e+15,
"event_date_milliseconds":1375957379601.874,
"event_date_seconds":1375957379.601874,
"event_timestamp":"2013-08-08T10:22:59.601Z",
"secRuleIds":[
"960011",
"960904",
"960903"
],
"matchedRules":[
"SecRule \"REQUEST_METHOD\" \"@rx ^(?:GET|HEAD)$\" ",
"SecRule \"&REQUEST_HEADERS:Content-Type\" \"@eq 0\" \"phase:2,deny,status:406,t:lo",
"SecRule \"REQUEST_FILENAME|ARGS|ARGS_NAMES|REQUEST_HEADERS|XML:/*|!REQUEST_HEADERS:Ref",
"SecAction \"phase:2,status:406,t:lowercase,t:replaceNulls,t:compres",
"SecRule \"&GLOBAL:alerted_960903_compression\" \"@eq 0\" \"phase:2,log,deny,status:406,t:lower"
]
}