Tagged: apache

Deserializing Modsecurity Audit logs with Apache Flume

This post will be updated in the coming days/weeks, however when looking at using Apache Flume to ingest some ModSecurity Audit logs, it quickly became apparent that Flume’s SpoolingDirectorySource lacked the ability to de-serialized “events” from a file that spanned many “new lines” (\n). Lacking this support, and seeing that an outstanding ticket already existed on a related subject at https://issues.apache.org/jira/browse/FLUME-1988 I went ahead and coded one up.

Please see RegexDelimiterDeSerializer and its corresponding unit test attached to FLUME-1988. Hopefully this can be included in an actual Flume release. In the meantime you should be able to include this and the related classes in a local copy of the Flume source and do your own build to get this functionality. The net result of using this regex patch is that each ModSecurity audit log entry (that spans many lines) will be summarized into *one* flume message. What you do next is up to you, however the next best thing is to pump this into the Flume Morphline Interceptor to then begin grokking and parsing the raw multi-lined modsec event. Note there are some possible synergies and re-use of regexes once you start using Morphlines and the Grok patterns we came up with for use with my Logstash based solution.

a) clone the official Flume source code

b) Drop in the files attached to FLUME-1988 into your cloned source of Flume

c) Follow the instructions located here to modify the source so that you can have a flume snapshot distro, that contains all the dependencies for Morphline (https://groups.google.com/a/cloudera.org/d/msg/cdk-dev/7T4pTebdWN4/sBHGkoS70LkJ)

d) From the root of the flume project run “mvn install -DskipTests=true” and take the tarball generated in “flume-ng-dist/target” and copy it somewhere else. (this is the freshly built Flume dist w/ the regex deserializer support)

e) Go to where you extracted the distro, widdle up your own flume config file and morphline config using the snippets below and then run “bin/flume-ng agent –conf conf –conf-file conf/flume.conf -Dflume.root.logger=DEBUG,console -n agent”

Here is a sample flume config snippet that uses this:


agent.sources = src1
agent.channels = memoryChannel
agent.sinks = loggerSink

# For each one of the sources, the type is defined
agent.sources.src1.type = spooldir
agent.sources.src1.channels = memoryChannel
agent.sources.src1.spoolDir = /path/to/my_modsec_logs
agent.sources.src1.deserializer = REGEX
agent.sources.src1.deserializer.outputCharset = UTF-8
agent.sources.src1.deserializer.eventEndRegex = --[a-fA-F0-9]{8}-Z--
agent.sources.src1.deserializer.includeEventEndRegex = true

agent.sources.src1.interceptors = morphlineinterceptor
agent.sources.src1.interceptors.morphlineinterceptor.type = org.apache.flume.sink.solr.morphline.MorphlineInterceptor$Builder
agent.sources.src1.interceptors.morphlineinterceptor.morphlineFile = /path/to/conf/morphline.conf
agent.sources.src1.interceptors.morphlineinterceptor.morphlineId = morphline1

 

Next is a sample “morphline.conf” configuration which will just emit each ModSecurity message from the audit log to standard out when running Flume. You can do the rest from there (have fun parsing). Please refer to the following morphlines documentation:

morphlines : [
  {
    id : morphline1
    importCommands : ["com.cloudera.**"]

    commands : [
      {
        readMultiLine {
          regex: ".*"
          charset : UTF-8
        }
      }

      # log the record at DEBUG level to SLF4J
      { logDebug { format : "output record: {}", args : ["@{}"] } }

    ]
  }
]
Advertisements

Securing Foscam IP camera access over SSL with Apache reverse proxying

UPDATED: 5/7/14 Important security vulnerabilities for Foscam cameras

UPDATED: 9/27/13  (The solution below does not include audio support; for audio over stunnel please see this post over at warped.org)

Recently I was assisting a local business setup their Foscam IP cameras and make them remotely accessible for monitoring purposes from anywhere in the world. The particular models they had installed are the FI8910W line. These camera’s are pretty cool and for ~$100 retail they are a pretty good deal in my opinion. The cameras can be accessed via a browser over HTTP and also support a rudimentary HTTP/CGI API. However one of the biggest issues with these cameras security wise is the lack of SSL support. The embedded webserver on these things only supports HTTP and basic auth in the clear which, outside of your local network is not a good thing if your requirements is to be able to view/manage them remotely from over the internet.

One solution for this is to simply front all access to your cameras with a SSL secured reverse proxy. We did this using Apache’s mod_proxy. I’m not going to go into every detail of how to do this below, but the point is to give the reader a starting point. You can lookup the details on all these apache configuration specifics elsewhere on the web, there are tons of examples out there.

The example below would be for securing access to 2 (two) Foscam IP cameras on your local network, living on an example subnet 192.168.1.0. It assumes the local network is fronted by a router that supports port forwarding, which most consumer/business routers do. The end objective here is that when you access https://myproxy.host.com:10001 you will be accessing CAM1 and when you access https://myproxy.host.com:10002 you will be accessing CAM2.

Secondarily you can also set it up so that you could hit CAM1 at https://myproxy.host.com:10000/cam1/ and CAM2 at https://myproxy.host.com:10000/cam2/

  1. CAM1 IP = 192.168.1.100 listening on port 80
  2. CAM2 IP = 192.168.1.101 listening on port 80
  3. Reverse Proxy Server = 192.168.1.50 listening on ports 10000, 10001, 10002
  4. Router IP address: 192.168.1.1  configured with port forwarding as follows: Port 10000 -> 192.168.1.50:10000, 10001 -> 192.168.1.50:10001 and 10002 -> 192.168.1.50:10002

OVERVIEW

  • First off you need to setup a computer/server running Apache. The Apache webserver is available for almost every operating system known to man from linux to windows, to os-x. This server’s IP address is 192.168.1.50 and ensure that name based virtual host support is enabled as well as mod_ssl.
  • Next ensure that apache is listening on all the necessary ports (the 3 mentioned above). You will want to have Apache listen on a separate unique port for each IP Camera it is proxying access to, or at least one unique port if you are proxying the cameras of of a sub-path: For this example we are assigning port 10000 -> [CAM1 & CAM2 via sub-dir proxies], port 10001->CAM1 only and 10002->CAM2 only. Within your apache configuration you will want to ensure that you have statements like the following configured:
NameVirtualHost *:10000
NameVirtualHost *:10001
NameVirtualHost *:10002
Listen 10000
Listen 10001
Listen 10002
  • Now that Apache is configured to listen on the necessary ports, we need to configure the actual virtual hosts and the reverse proxying directives within each host, see the example below:
###############################
# Reverse proxy config for BOTH
# CAMs (1 & 2) via sub-paths
# @ 192.168.1.100
###############################
<VirtualHost 192.168.1.50:10000>
 ProxyRequests Off
 ProxyPreserveHost On
 ProxyVia On
 <Proxy *>
 Order deny,allow
 Allow from all
 </Proxy>

 # CAM1 (note trailing / is important)
 ProxyPass /cam1/ http://192.168.1.100:80/
 ProxyPassReverse /cam1/ http://192.168.1.100:80/

 # CAM2 (note trailing / is important)
 ProxyPass /cam2/ http://192.168.1.101:80/
 ProxyPassReverse /cam2/ http://192.168.1.101:80/

 CustomLog /path/to/apachelogs/access_cam1.log combined
 ErrorLog /path/to/apachelogs/error_cam1.log
 ServerName cam3

 SSLEngine On
 SSLCertificateFile /path/to/sslcert/mysslcert.crt
 SSLCertificateKeyFile /path/to/sslkey/sslkey.key

 <FilesMatch "\.(cgi|shtml|phtml|php)$">
 SSLOptions +StdEnvVars
 </FilesMatch>
 <Directory /usr/lib/cgi-bin>
 SSLOptions +StdEnvVars
 </Directory>

 BrowserMatch "MSIE [2-6]" \
 nokeepalive ssl-unclean-shutdown \
 downgrade-1.0 force-response-1.0
 # MSIE 7 and newer should be able to use keepalive
 BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown
</VirtualHost>

###############################
# Reverse proxy config for CAM1
# @ 192.168.1.100
###############################
<VirtualHost 192.168.1.50:10001>
 ProxyRequests Off
 ProxyPreserveHost On
 ProxyVia On
 <Proxy *>
 Order deny,allow
 Allow from all
 </Proxy>
 ProxyPass / http://192.168.1.100:80/
 ProxyPassReverse / http://192.168.1.100:80/
 CustomLog /path/to/apachelogs/access_cam1.log combined
 ErrorLog /path/to/apachelogs/error_cam1.log
 ServerName cam3

 SSLEngine On
 SSLCertificateFile /path/to/sslcert/mysslcert.crt
 SSLCertificateKeyFile /path/to/sslkey/sslkey.key

 <FilesMatch "\.(cgi|shtml|phtml|php)$">
 SSLOptions +StdEnvVars
 </FilesMatch>
 <Directory /usr/lib/cgi-bin>
 SSLOptions +StdEnvVars
 </Directory>

 BrowserMatch "MSIE [2-6]" \
 nokeepalive ssl-unclean-shutdown \
 downgrade-1.0 force-response-1.0
 # MSIE 7 and newer should be able to use keepalive
 BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown
</VirtualHost>

###############################
# Reverse proxy config for CAM2
# @ 192.168.1.101
###############################
<VirtualHost 192.168.1.50:10002>
 ProxyRequests Off
 ProxyPreserveHost On
 ProxyVia On
 <Proxy *>
 Order deny,allow
 Allow from all
 </Proxy>
 ProxyPass / http://192.168.1.101:80/
 ProxyPassReverse / http://192.168.1.101:80/
 CustomLog /path/to/apachelogs/access_cam2.log combined
 ErrorLog /path/to/apachelogs/error_cam2.log
 ServerName cam3

 SSLEngine On
 SSLCertificateFile /path/to/sslcert/mysslcert.crt
 SSLCertificateKeyFile /path/to/sslkey/sslkey.key

 <FilesMatch "\.(cgi|shtml|phtml|php)$">
 SSLOptions +StdEnvVars
 </FilesMatch>
 <Directory /usr/lib/cgi-bin>
 SSLOptions +StdEnvVars
 </Directory>

 BrowserMatch "MSIE [2-6]" \
 nokeepalive ssl-unclean-shutdown \
 downgrade-1.0 force-response-1.0
 # MSIE 7 and newer should be able to use keepalive
 BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown
</VirtualHost>
  • Ok, so before you start up apache, you need to generate your own self-signed SSL certificate/key. See those lines above in the configuration for “SSLCertificateFile” and “SSLCertificateKeyFile”? You will need to generate your own SSL private key, certificate request, and then self sign it. The results of those openssl commands yield files that you point to on your local proxy server. You can read here for an example on how to generate the necessary files
  • Next ensure the router that sits in front of your proxy server @ 192.168.1.50 has port forwarding enabled and forwards traffic going to port 10000, 10001 and 10002 to your proxy server.
  • Start up apache, work out the kinks and you should be ready to go. If you are outside of your normal network you will need to find your router’s WAN public IP address and go to https://my.external.router.ip:10001 and https://my.external.router.ip:10002 and you will be accessing CAM1 and CAM2 respectively over SSL from anywhere in the world. OR secondarily you can also go to https://my.external.router.ip:10000/cam1/ and https://my.external.router.ip:10000/cam2/ to hit the cameras. Please note that traffic from your browser to your proxy server is encrypted with SSL, however the SSL encryption will terminate at the proxy server. Network traffic from your proxy server to CAM1 and CAM2 is unencrypted but only running over your local network. This article assumes you trust who is on your local network not to be sniffing packets.
  • You will also want to ensure that your proxy server has a firewall on it, permits IP forwarding, limits access to only the necessary ports and is configured securely. You can handle that on your own and that is outside of the scope of this article.
  • Hopefully this helps someone out there who wants to securely access their IP cameras over the internet. Note that what is described above should work with any IP camera on the market that only supports HTTP, however the general procedure described above was only tested to work with Foscam model FI8910W

SOFTWARE FOR VIEWING YOUR PROXIED CAMERAS

I’ve received many questions regarding which apps out there support talking to Foscam’s behind a SSL secured proxy and unfortunately the few I’ve used all fall short in one way or another. Proxying http/https based resources on a network (via ports, sub-paths or other methods) is a technology that has been around for a long time and from an client apps perspective it need not know it is even there. Secondarily, the Foscam camera APIs will work just fine regardless of how they are proxied (from the root url or off of a sub-path in the proxy). Regardless here are some apps I’ve used below with some notes

  • iOS: FoscamPro: Cool app, works great when you are on your internal network, but fails miserably if you try to use it from outside your network when your cameras are behind an SSL secured proxy as described above. Why? The FoscamPro application simply DOES NOT support SSL. (FoscamPro devs: PLEASE IMPLEMENT THIS!) The only way to use FoscamPro in the setup above is if you have a VPN server running behind your router; you then connect to your home VPN which lets you appear “internal” to your local network when you are outside of your network, and then access your cameras directly, bypassing the proxy. The VPN itself is what is encrypting all of your communications.
  • iOS: Live Cams Pro: Cool app, works very similar to FoscamPro but supports other manufacturers and more devices, generic url streams etc. They DO support SSL which works with the proxied setup described above. However they DO NOT support specifying a relative path off of the base IP that you are connecting to a Foscam camera with. This effectively eliminates your ability to proxy your cameras via sub-dirs (i.e. https://my.net/cam1/) which is CRITICAL if you have a lot of cams but your router limits the number of port forwards you can have! (Live Cams Pro devs: PLEASE IMPLEMENT THIS!)
  • Android: tinyCam Monitor PRO: Cool app, I must admit I am pretty sure this supports HTTPS as I was testing with this earlier this summer for the port-> cam based config. I have not tested with the sub-dir path setup. If someone can shoot me an update on this I’ll appreciate it. (I’ve switched to all iOS)