I’ve done quite a bit of WS development both the on client side and server side; interacting with both SOAP, REST and variants of XML/JSON RPC hybrid services. For my latest project I need to expose a set of REST services to a myriad of clients: phones, fat JS clients etc. This application also needs to talk to other nodes or “agents” that are also doing work in a distributed cloud environment. The core engine of this application really has no need for a bloated higher level MVC/gui supporting stack and bringing that into this library would just be a pain. I’ve always like the simplicity of being able to skip the whole JEE/Spring/Tomcat based container stack and just do a plain old “java -jar”… run my application…. but the reality of being able to do that has been lacking… until now.

In looking at the available options for picking the framework to use (such as Restlet, Spring-MVC REST, Spring Data REST etc and others), I immediately became discouraged when looking at examples for setting them up; they are full of complexity and lots of configuration and generally require a full application server container to run within, which just adds further complexity to your setup.

Then I stumbled across Dropwizard by the folks at Yammer. I encourage everyone reading this to just try the simple Hello World example they have on their site. If you have any experience in this space and an appreciation and vision for decoupling; you will immediately recognize the beauty of this little framework and the power it can bring to the table from a deployment standpoint. Build your core app engine back-end library as you normally would, toss in Dropwizard, expose some REST services to extend your interfaces to the outside world; throw it up on a server and “java -jar myapp server myconfig.yml” and you are ready to rock. (they make this possible by in-lining Jetty). Create a few little JS/HTML files for a fat JS client, (i’d recommend Angular) and hook into your REST services and you will have an awesome little decoupled application.

I have a need to do the following. My app uses the Log4J API deployed on JBoss 6.1.x

A) Correlate log statements with a given thread of execution, logically identified by an arbitrary generated “request id” that my application sets at the entry point for a request, doing this using ThreadLocals. No, this is not a discussion for NDC/MDC.

B) These log statements will end up wherever the log system is configured to send them. I expect to be able to use that logging system’s documented APIs/configuration to add additional targets for log entries (appenders in log4j lingo)

C) Due to (A) and the sheer volume, I have the need to create a highly customized appender for log events that has access to my application’s context/services. The intent is to log these off asynchronously to a queue or a NoSql DB leveraging connections that my application already has handles on and configured itself. Hence I am not going to be able to reply on an instance of something that is instantiated and is outside of my application’s context. (i.e. I don’t want to configure this appender via the container’s default configuration mechanism, xml or whatever it is). I need to do this at runtime. My appender will need access to queues, and configuration, sources of connectivity that my application provides (not the deployment environment)

D) The logical approach would be to just use Log4J’s API and do something like:

Logger.getLogger("my.package.root").addAppender(new MySuperCustomizedAppender(configFromMyApp));

The above runs great in a unit test locally, I’m ready to go, lets deploy to JBoss!……oops wait, once deployed there nothing is happening.. whats going on??

E) When I do a Logger.getLogger() I get a nice instance of org.jboss.logmanager.log4j.BridgeLogger , looking at the source I see this little gem:

public void addAppender(final Appender newAppender) {
   // ignored
}

No warning. No Exception. No Message. Just silently discarded, letting you go ahead and waste your time trying to figure out what the hell is going on.

F) “Ok, someone else has certainly encountered this, lets google it”…. yep, doing a search on “BridgeLogger jboss addAppender” will yield you a whole lot of results and no definitive solution that really seems to do the job. Several forum posts going on and on with the lack of a satisfactory answer; several JIRA tickets documenting this issue etc. Overall the “workarounds” seem like way more trouble then they are worth (fiddling w/ jars, build/deploy routines, configurations etc on all app servers etc)

G) “Hmm, maybe I shouldn’t be using Log4J, how about Slf4J?”….. ok well, Slf4J provides a nice facade to logging. I’ll have to refactor a lot of my logging statements to use this API but it might be manageable. Then I can specifically target Log4j and bypass Jboss’s logging system…… oops. wait. no. JBoss also includes an SLF4J adapter of its own in the parent server level classpath, so if I create my own, since SLF4j does not permit multiple bindings, it will be a crap shoot for which one it picks…. Jboss’ or the one you want it to. Again the “solutions” for getting SLF4J to use your binding of choice over the one provided by JBoss seem like more “forum threads” with the lack of clear documented solution that does not involve your system administrators having to make farm wide config changes… @see F above. Forget it

H) Now what? Create my own “extension” of SLF4J or my own cloned interface of SLF4J that gives me some sort of hook that permits me to route log events over my customized “appender”? Yet more abstractions.

Recently I was assisting a local business setup their Foscam IP cameras and make them remotely accessible for monitoring purposes from anywhere in the world. The particular models they had installed are the FI8910W line. These camera’s are pretty cool and for ~$100 retail they are a pretty good deal in my opinion. The cameras can be accessed via a browser over HTTP and also support a rudimentary HTTP/CGI API. However one of the biggest issues with these cameras security wise is the lack of SSL support. The embedded webserver on these things only supports HTTP and basic auth in the clear which, outside of your local network is not a good thing if your requirements is to be able to view/manage them remotely from over the internet.

One solution for this is to simply front all access to your cameras with a SSL secured reverse proxy. We did this using Apache’s mod_proxy. I’m not going to go into every detail of how to do this below, but the point is to give the reader a starting point. You can lookup the details on all these apache configuration specifics elsewhere on the web, there are tons of examples out there.

The example below would be for securing access to 2 (two) Foscam IP cameras on your local network, living on an example subnet 192.168.1.0. It assumes the local network is fronted by a router that supports port forwarding, which most consumer/business routers do. The end objective here is that when you access https://myproxy.host.com:10001 you will be accessing CAM1 and when you access https://myproxy.host.com:10002 you will be accessing CAM2.

Secondarily you can also set it up so that you could hit CAM1 at https://myproxy.host.com:10000/cam1/ and CAM2 at https://myproxy.host.com:10000/cam2/

1. CAM1 IP = 192.168.1.100 listening on port 80
2. CAM2 IP = 192.168.1.101 listening on port 80
3. Reverse Proxy Server = 192.168.1.50 listening on ports 10000, 10001, 10002
4. Router IP address: 192.168.1.1  configured with port forwarding as follows: Port 10000 -> 192.168.1.50:10000, 10001 -> 192.168.1.50:10001 and 10002 -> 192.168.1.50:10002

OVERVIEW

• First off you need to setup a computer/server running Apache. The Apache webserver is available for almost every operating system known to man from linux to windows, to os-x. This server’s IP address is 192.168.1.50 and ensure that name based virtual host support is enabled as well as mod_ssl.

• Next ensure that apache is listening on all the necessary ports (the 3 mentioned above). You will want to have Apache listen on a separate unique port for each IP Camera it is proxying access to, or at least one unique port if you are proxying the cameras of of a sub-path: For this example we are assigning port 10000 -> [CAM1 & CAM2 via sub-dir proxies], port 10001->CAM1 only and 10002->CAM2 only. Within your apache configuration you will want to ensure that you have statements like the following configured:

NameVirtualHost *:10000
NameVirtualHost *:10001
NameVirtualHost *:10002
Listen 10000
Listen 10001
Listen 10002

• Now that Apache is configured to listen on the necessary ports, we need to configure the actual virtual hosts and the reverse proxying directives within each host, see the example below:

###############################
# Reverse proxy config for BOTH
# CAMs (1 & 2) via sub-paths
# @ 192.168.1.100
###############################
<VirtualHost 192.168.1.50:10000>
 ProxyRequests Off
 ProxyPreserveHost On
 ProxyVia On
 <Proxy *>
 Order deny,allow
 Allow from all
 </Proxy>

 # CAM1 (note trailing / is important)
 ProxyPass /cam1/ http://192.168.1.100:80/
 ProxyPassReverse /cam1/ http://192.168.1.100:80/

 # CAM2 (note trailing / is important)
 ProxyPass /cam2/ http://192.168.1.101:80/
 ProxyPassReverse /cam2/ http://192.168.1.101:80/

 CustomLog /path/to/apachelogs/access_cam1.log combined
 ErrorLog /path/to/apachelogs/error_cam1.log
 ServerName cam3

 SSLEngine On
 SSLCertificateFile /path/to/sslcert/mysslcert.crt
 SSLCertificateKeyFile /path/to/sslkey/sslkey.key

 <FilesMatch "\.(cgi|shtml|phtml|php)$">
 SSLOptions +StdEnvVars
 </FilesMatch>
 <Directory /usr/lib/cgi-bin>
 SSLOptions +StdEnvVars
 </Directory>

 BrowserMatch "MSIE [2-6]" \
 nokeepalive ssl-unclean-shutdown \
 downgrade-1.0 force-response-1.0
 # MSIE 7 and newer should be able to use keepalive
 BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown
</VirtualHost>

###############################
# Reverse proxy config for CAM1
# @ 192.168.1.100
###############################
<VirtualHost 192.168.1.50:10001>
 ProxyRequests Off
 ProxyPreserveHost On
 ProxyVia On
 <Proxy *>
 Order deny,allow
 Allow from all
 </Proxy>
 ProxyPass / http://192.168.1.100:80/
 ProxyPassReverse / http://192.168.1.100:80/
 CustomLog /path/to/apachelogs/access_cam1.log combined
 ErrorLog /path/to/apachelogs/error_cam1.log
 ServerName cam3

 SSLEngine On
 SSLCertificateFile /path/to/sslcert/mysslcert.crt
 SSLCertificateKeyFile /path/to/sslkey/sslkey.key

 <FilesMatch "\.(cgi|shtml|phtml|php)$">
 SSLOptions +StdEnvVars
 </FilesMatch>
 <Directory /usr/lib/cgi-bin>
 SSLOptions +StdEnvVars
 </Directory>

 BrowserMatch "MSIE [2-6]" \
 nokeepalive ssl-unclean-shutdown \
 downgrade-1.0 force-response-1.0
 # MSIE 7 and newer should be able to use keepalive
 BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown
</VirtualHost>

###############################
# Reverse proxy config for CAM2
# @ 192.168.1.101
###############################
<VirtualHost 192.168.1.50:10002>
 ProxyRequests Off
 ProxyPreserveHost On
 ProxyVia On
 <Proxy *>
 Order deny,allow
 Allow from all
 </Proxy>
 ProxyPass / http://192.168.1.101:80/
 ProxyPassReverse / http://192.168.1.101:80/
 CustomLog /path/to/apachelogs/access_cam2.log combined
 ErrorLog /path/to/apachelogs/error_cam2.log
 ServerName cam3

 SSLEngine On
 SSLCertificateFile /path/to/sslcert/mysslcert.crt
 SSLCertificateKeyFile /path/to/sslkey/sslkey.key

 <FilesMatch "\.(cgi|shtml|phtml|php)$">
 SSLOptions +StdEnvVars
 </FilesMatch>
 <Directory /usr/lib/cgi-bin>
 SSLOptions +StdEnvVars
 </Directory>

 BrowserMatch "MSIE [2-6]" \
 nokeepalive ssl-unclean-shutdown \
 downgrade-1.0 force-response-1.0
 # MSIE 7 and newer should be able to use keepalive
 BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown
</VirtualHost>

• Ok, so before you start up apache, you need to generate your own self-signed SSL certificate/key. See those lines above in the configuration for “SSLCertificateFile” and “SSLCertificateKeyFile”? You will need to generate your own SSL private key, certificate request, and then self sign it. The results of those openssl commands yield files that you point to on your local proxy server. You can read here for an example on how to generate the necessary files

• Next ensure the router that sits in front of your proxy server @ 192.168.1.50 has port forwarding enabled and forwards traffic going to port 10000, 10001 and 10002 to your proxy server.

• Start up apache, work out the kinks and you should be ready to go. If you are outside of your normal network you will need to find your router’s WAN public IP address and go to https://my.external.router.ip:10001 and https://my.external.router.ip:10002 and you will be accessing CAM1 and CAM2 respectively over SSL from anywhere in the world. OR secondarily you can also go to https://my.external.router.ip:10000/cam1/ and https://my.external.router.ip:10000/cam2/ to hit the cameras. Please note that traffic from your browser to your proxy server is encrypted with SSL, however the SSL encryption will terminate at the proxy server. Network traffic from your proxy server to CAM1 and CAM2 is unencrypted but only running over your local network. This article assumes you trust who is on your local network not to be sniffing packets.

• You will also want to ensure that your proxy server has a firewall on it, permits IP forwarding, limits access to only the necessary ports and is configured securely. You can handle that on your own and that is outside of the scope of this article.

• Hopefully this helps someone out there who wants to securely access their IP cameras over the internet. Note that what is described above should work with any IP camera on the market that only supports HTTP, however the general procedure described above was only tested to work with Foscam model FI8910W

SOFTWARE FOR VIEWING YOUR PROXIED CAMERAS

I’ve received many questions regarding which apps out there support talking to Foscam’s behind a SSL secured proxy and unfortunately the few I’ve used all fall short in one way or another. Proxying http/https based resources on a network (via ports, sub-paths or other methods) is a technology that has been around for a long time and from an client apps perspective it need not know it is even there. Secondarily, the Foscam camera APIs will work just fine regardless of how they are proxied (from the root url or off of a sub-path in the proxy). Regardless here are some apps I’ve used below with some notes

• iOS: FoscamPro: Cool app, works great when you are on your internal network, but fails miserably if you try to use it from outside your network when your cameras are behind an SSL secured proxy as described above. Why? The FoscamPro application simply DOES NOT support SSL. (FoscamPro devs: PLEASE IMPLEMENT THIS!) The only way to use FoscamPro in the setup above is if you have a VPN server running behind your router; you then connect to your home VPN which lets you appear “internal” to your local network when you are outside of your network, and then access your cameras directly, bypassing the proxy. The VPN itself is what is encrypting all of your communications.

• iOS: Live Cams Pro: Cool app, works very similar to FoscamPro but supports other manufacturers and more devices, generic url streams etc. They DO support SSL which works with the proxied setup described above. However they DO NOT support specifying a relative path off of the base IP that you are connecting to a Foscam camera with. This effectively eliminates your ability to proxy your cameras via sub-dirs (i.e. https://my.net/cam1/) which is CRITICAL if you have a lot of cams but your router limits the number of port forwards you can have! (Live Cams Pro devs: PLEASE IMPLEMENT THIS!)

• Android: tinyCam Monitor PRO: Cool app, I must admit I am pretty sure this supports HTTPS as I was testing with this earlier this summer for the port-> cam based config. I have not tested with the sub-dir path setup. If someone can shoot me an update on this I’ll appreciate it. (I’ve switched to all iOS)

Some background regarding this Jboss issue at https://issues.jboss.org/browse/AS7-308

Errors that can be fixed by doing the above:

java.lang.SecurityException: JCE cannot authenticate the provider BC: org.jasypt.exceptions.EncryptionInitializationException: java.lang.SecurityException: JCE cannot authenticate the provider BC

OR

Caused by: java.util.jar.JarException: Cannot parse vfs: /path/to/your/bouncycastle.jar

Doing this, I came across a situation trying to use the Netflix Astyanax client API to talk to Cassandra, while Cassandra was running with th custom IAuthenticator and IAuthorities that I made. When testing the initializations of connections to Cassandra, while specifying invalid credentials (intentionally) instead of seeing some sort of AuthenticationException dumped to my client Astyanax log file, I was getting “PoolTimeoutException“s instead…. which was odd. I scratched my head on this for a while as I cloned Astyanax from GitHub and began digging into the source. I suspected that the Thrift AuthenticationException might  be supressed somewhere…. well after reading the source, I realized it wasn’t being suppressed per-say, but rather sent to Astyanax’s ConnectionPoolMonitor, which is something you can configure programatically when you are defining your client code’s AstyanaxContext object which manages all connectivity to Cassandra. Out of the box Astyanax ships with a few ConnectionPoolMonitor implementations, one is the CountingConnectionPoolMonitor (does no logging, just collects stats) and the second is the Slf4jConnectionPoolMonitorImpl (logs to SLF4J). Depending on which one you specify in your context’s configuration you may or may not see AuthenticationException information in your client’s logs/console.

In my case, I was specifying the CountingConnectionPoolMonitor which was receiving the AuthenticationException, but not doing anything with it other than incrementing some counter, effectivly hiding it from me. The pool ran out of connections (could not create any) and the code waiting on getting a connection just threw a PoolTimeoutException, adding to my confusion.

To correct this, as I was using Log4J, I just created a custom ConnectionPoolMonitor which logged everything to Log4J instead. (@see Astyanax’s SLF4J monitor implementation as an example for how to create one for Log4j) See below for how to specify the monitor. Creating your own ConnectionPoolMonitor implementation is easy and pretty self explanatory.

Below is an example of setting up an AstyanaxContext and how you specify the ConnectionPoolMonitor that should be used. Once I used the correct monitor for my needs, I was able to see the true source of the PoolTimeoutExceptions (i.e. the AuthenticationExceptions) because now my monitor was logging them. (NOTE: the example below is just a test context, not something for a robust setup)

AstyanaxContext context = new AstyanaxContext.Builder()
 .forCluster(clusterName)
 .forKeyspace(keyspaceName)
 .withAstyanaxConfiguration(new AstyanaxConfigurationImpl()
 .setDiscoveryType(NodeDiscoveryType.NONE)
 )

.withConnectionPoolConfiguration(new ConnectionPoolConfigurationImpl(clusterName+"-"+keyspaceName+"_CONN_POOL")
 .setPort(defaultConnectionPoolHostPort)
 .setInitConnsPerHost(1)
 .setMaxConnsPerHost(2)
 .setSeeds(connectionPoolSeedHosts)
 .setAuthenticationCredentials(
 new SimpleAuthenticationCredentials(new String(principal), new String(credentials)))
 )
 .withConnectionPoolMonitor(new Log4jConnPoolMonitor())
 .buildKeyspace(ThriftFamilyFactory.getInstance());

context.start();

This project was originally started to create a framework for loading data files from the USPS AIS suite of data products (zipPlus4, cityState). The project has not been worked on in a while but I figured I’d open-source it and maybe some folks would like to team up to work on it further, if so let me know! Throwing it out there under the Apache 2.0 license. Some of the libs need updating etc as well, for instance it was originally developed w/ Spring 2.5.

USPS AIS data files are fixed length format records. This framework was created to handle bulk loading/updating this data into a structured/semi-structured data store of address data (i.e. MySql or HBase). It is wired together using Spring and built w/ Maven. A key package is the “org.bitsofinfo.util.address.usps.ais” package which defines the pojos for the records, and leverages a custom annotation which binds record properties to locations within the fixed length records which contain the data being loaded.

Initial loader implementations include both a single JVM multi-threaded version as well as a second one that leverages Hadoop Mapreduce to split the AIS files up across HDFS and process them in parallel using Hadoop mapreduce nodes to ingest the data much faster then just on one box. Both of these obviously operate asynchronously given a load job submission. Ingestion times are significantly faster using Hadoop.

This project also had a need for a Hadoop InputFormat/RecordReader that could read from fixed length data files (none existed), so I created it for this project (FixedLengthInputFormat). This was also contributed as a patch to the Hadoop project. This source is included in here and updated for Hadoop 0.23.1 (not yet tested), however the patch that was submitted to the Hadoop project is still pending and was compiled under 0.20.x. The 0.20.x version in the patch files was tested and functionally running on a 4 node Hadoop and Hbase cluster.

You can read more about the fixed length record reader patch @

http://bitsofinfo.wordpress.com/2009/11/01/reading-fixed-length-width-input-record-reader-with-hadoop-mapreduce/

https://issues.apache.org/jira/browse/MAPREDUCE-1176 

The USPS AIS products have some sample data-sets available online at the USPS website, however for the full product of data-files you need to pay for the data and/or subscription for delta updates. Some of the unit-tests reference files from the real data-sets, they have been omitted, you will have to replace them with the real ones. Other unit tests reference the sample files freely available via USPS or other providers.

Links where USPS data files can be purchased:

https://www.usps.com/business/address-information-systems.htm

http://www.zipinfo.com/products/natzip4/natzip4.htm

Turns out you can enable this by enabling port forwarding locally on the computer you want to connect from. This is on OS-X:

• First ensure that your SSH key is available for use by the command we will run: “ssh-add /path/to/your/openshift/ssh.key” This should be the key that you created when you initially signed up for Openshift

• Attempt to run the port forwarding enable command “rhc port-forward -a [yourAppName] -l [yourOpenShiftLoginId]“, it will prompt you for your openshift credentials, then will likely output something like the below:

Does ANYONE OUT THERE have a proven, easy to implement, somewhat universal way, to simple set connect and read timeouts for JAX-WS web-service clients? I can’t seem to find them, and neither can tons of others out there by the looks for dozens of posts/blogs I have read in forums etc.

This is one of the most insidious problems with an application. Timeouts, or the lack thereof causing a thread in your application to mysteriously (or not) hang up sitting there forever. Is my app deadlocked? Why is this thread hung? Then after a while your app’s thread pool may be exhausted etc leading to harder to diagnose and worthless artifact error symptoms being spewed into a log file.

In an enterprise app, one of the common things we do is use a web-service to send or receive data from some other application out there… and a lot of people use some sort of JAX-WS generated variant of clients.

Here is my question. HOW can I reliably set a jax-ws client timeout for both connect and read operations? I seem to see various answers to this, often time they don’t work for folks. Other times I suspect folks are getting confused with regards to the actual JAX-WS client implementation executing their code (jboss, glassfish, websphere etc).. and I guess these settings vary based on where you code is deployed, which is SUPER convenient because there is not abstract definition of this in the spec itself to allow developers to specify their timeouts in a portable way??

Hey Oracle: Why can’t Java just support a simple solution to this stupid time-wasting problem?? @see Axis clients and how easy it is to do there!

PLEASE POST YOUR SOLUTIONS IN THE COMMENTS SECTION OF THIS POST TO HELP OTHERS (and me!)

Here are some examples I’ve seen out there of threads that have random answers that often go unanswered.

http://www.coderanch.com/t/537394/Web-Services/java/JBoss-Jax-ws-Timeout-Configuration

https://community.jboss.org/message/607864

http://stackoverflow.com/questions/2592303/how-do-i-set-the-jax-ws-client-request-timeout-programatically-on-jboss

http://stackoverflow.com/questions/3130913/setting-jax-ws-client-timeout

http://www.java.net/node/676074

http://www.websphere-world.com/modules.php?name=News&file=article&sid=2058

http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=%2Fcom.ibm.websphere.express.doc%2Finfo%2Fexp%2Fae%2Frwbs_trbclientruntime.html

http://chamerling.org/2009/09/23/setting-timeout-on-generated-jaxws-cxf-clients/

https://forums.oracle.com/forums/thread.jspa?threadID=1666087