Tagged: Spring

Dropwizard Java REST services

To sum it up; Dropwizard rocks.

I’ve done quite a bit of WS development both the on client side and server side; interacting with both SOAP, REST and variants of XML/JSON RPC hybrid services. For my latest project I need to expose a set of REST services to a myriad of clients: phones, fat JS clients etc. This application also needs to talk to other nodes or “agents” that are also doing work in a distributed cloud environment. The core engine of this application really has no need for a bloated higher level MVC/gui supporting stack and bringing that into this library would just be a pain. I’ve always like the simplicity of being able to skip the whole JEE/Spring/Tomcat based container stack and just do a plain old “java -jar”… run my application…. but the reality of being able to do that has been lacking… until now.

In looking at the available options for picking the framework to use (such as Restlet, Spring-MVC REST, Spring Data REST etc and others), I immediately became discouraged when looking at examples for setting them up; they are full of complexity and lots of configuration and generally require a full application server container to run within, which just adds further complexity to your setup.

Then I stumbled across Dropwizard by the folks at Yammer. I encourage everyone reading this to just try the simple Hello World example they have on their site. If you have any experience in this space and an appreciation and vision for decoupling; you will immediately recognize the beauty of this little framework and the power it can bring to the table from a deployment standpoint. Build your core app engine back-end library as you normally would, toss in Dropwizard, expose some REST services to extend your interfaces to the outside world; throw it up on a server and “java -jar myapp server myconfig.yml” and you are ready to rock. (they make this possible by in-lining Jetty). Create a few little JS/HTML files for a fat JS client, (i’d recommend Angular) and hook into your REST services and you will have an awesome little decoupled application.

USPS AIS bulk data loading with Hadoop mapreduce

Today I pushed up some source to Github for a utility I was previously working on to load data from USPS AIS data files into HBase/Mysql using Hadoop mapreduce and simpler data loaders. Source @ https://github.com/bitsofinfo/usps-ais-data-loader

This project was originally started to create a framework for loading data files from the USPS AIS suite of data products (zipPlus4, cityState). The project has not been worked on in a while but I figured I’d open-source it and maybe some folks would like to team up to work on it further, if so let me know! Throwing it out there under the Apache 2.0 license. Some of the libs need updating etc as well, for instance it was originally developed w/ Spring 2.5.

USPS AIS data files are fixed length format records. This framework was created to handle bulk loading/updating this data into a structured/semi-structured data store of address data (i.e. MySql or HBase). It is wired together using Spring and built w/ Maven. A key package is the “org.bitsofinfo.util.address.usps.ais” package which defines the pojos for the records, and leverages a custom annotation which binds record properties to locations within the fixed length records which contain the data being loaded.

Initial loader implementations include both a single JVM multi-threaded version as well as a second one that leverages Hadoop Mapreduce to split the AIS files up across HDFS and process them in parallel using Hadoop mapreduce nodes to ingest the data much faster then just on one box. Both of these obviously operate asynchronously given a load job submission. Ingestion times are significantly faster using Hadoop.

This project also had a need for a Hadoop InputFormat/RecordReader that could read from fixed length data files (none existed), so I created it for this project (FixedLengthInputFormat). This was also contributed as a patch to the Hadoop project. This source is included in here and updated for Hadoop 0.23.1 (not yet tested), however the patch that was submitted to the Hadoop project is still pending and was compiled under 0.20.x. The 0.20.x version in the patch files was tested and functionally running on a 4 node Hadoop and Hbase cluster.

You can read more about the fixed length record reader patch @

https://bitsofinfo.wordpress.com/2009/11/01/reading-fixed-length-width-input-record-reader-with-hadoop-mapreduce/

https://issues.apache.org/jira/browse/MAPREDUCE-1176 

The USPS AIS products have some sample data-sets available online at the USPS website, however for the full product of data-files you need to pay for the data and/or subscription for delta updates. Some of the unit-tests reference files from the real data-sets, they have been omitted, you will have to replace them with the real ones. Other unit tests reference the sample files freely available via USPS or other providers.

Links where USPS data files can be purchased:

https://www.usps.com/business/address-information-systems.htm

http://www.zipinfo.com/products/natzip4/natzip4.htm

Integrating Restlet with Spring

For those of you out there who would like to get Restlet 2.0 (currently the M5) release up integrated with your existing Spring application, hopefully this post will be of some help. I recently had to do this and unfortunately the documentation related to Spring integration on the Restlet site is scattered across various docs and some of it appears out of date. What I am describing below worked with source code straight from the Restlet SVN trunk just before the M5 release so you should be good to go if you use the M5 release (JEE edition)

First off, I am assuming you have an existing web application with a web.xml file and are using Spring. Secondly I am just trying to give you some working web.xml and the corresponding Spring configuration to get up and running. I am not explaining the details of how Restlet works as you can find that on the Restlet site.

First you will want to make sure you have the Restlet JEE 2.0 M5 edition. Make sure you grab the JEE version and not the JSE version as the latter does not include the Spring integration extension. Once downloaded extract the ZIP to a location on your drive. The JEE zip package contains a ton of Restlet Jar files. The three we care about are org.restlet.jar, org.restlet.ext.spring.jar, org.restlet.ext.servlet.jar

If you are using Maven you can add the following repository and dependencies to your POM by using the repository instructions on the Restlet site. NOTE as of today: the Restlet repository currently does NOT have the M5 release up there so you are going to manually have to add the M5 jar to your repository by doing the following for each of the 3 jars.

mvn install:install-file -Dfile=/PATH/TO/RESTLET-m5-ZIP-EXTRACT-DIR/lib/org.restlet.jar -DgroupId=org.restlet -DartifactId=org.restlet -Dversion=2.0-SNAPSHOT-M5 -Dpackaging=jar

mvn install:install-file -Dfile=/PATH/TO/RESTLET-m5-ZIP-EXTRACT-DIR/lib/org.restlet.ext.spring.jar -DgroupId=org.restlet -DartifactId=org.restlet.ext.spring -Dversion=2.0-SNAPSHOT-M5 -Dpackaging=jar

mvn install:install-file -Dfile=/PATH/TO/RESTLET-m5-ZIP-EXTRACT-DIR/lib/org.restlet.ext.servlet.jar -DgroupId=org.restlet -DartifactId=org.restlet.ext.servlet -Dversion=2.0-SNAPSHOT-M5 -Dpackaging=jar

The above 3 commands will manually install the three Jars into your Maven repository. Next you can configure your POM to add the official Maven repository plus the dependencies to the 3 Jars you installed above. Note that the repository entry is sort of meaningless at this point because you manually installed the jars above. It is IMPORTANT that the version elements in your dependencies below MATCH exactly the versions you specified in the commands above!

	<repository>  
    	<id>maven-restlet</id>  
    	<name>Public online Restlet repository</name>  
    	<url>http://maven.restlet.org</url>  
	</repository>

	<dependency>
    	<groupId>org.restlet</groupId>
    	<artifactId>org.restlet</artifactId>
    	<version>2.0-SNAPSHOT-M5</version>
	</dependency>

	<dependency>
    	<groupId>org.restlet</groupId>
    	<artifactId>org.restlet.ext.spring</artifactId>
    	<version>2.0-SNAPSHOT-M5</version>
	</dependency>

	<dependency>
    	<groupId>org.restlet</groupId>
    	<artifactId>org.restlet.ext.servlet</artifactId>
    	<version>2.0-SNAPSHOT-M5</version>
	</dependency>

Ok, great. Next we need to configure your web.xml, open it up and add the following entries in the appropriate spots:

  	<servlet>
      	<servlet-name>myRESTApi</servlet-name>
      	<servlet-class>org.restlet.ext.spring.SpringServerServlet</servlet-class>
      	 <init-param>
                <param-name>org.restlet.component</param-name>
                 <!-- this value must match the bean id of the Restlet component you will configure in Spring (below) -->
                <param-value>restletComponent</param-value>
         </init-param>
  	</servlet>

  	<servlet-mapping>
        <servlet-name>myRESTApi</servlet-name>
        <url-pattern>/my/REST/api/*</url-pattern>
  	</servlet-mapping>

Now your web.xml is configured to take all requests to /my/REST/api/* and send those to a Restlet Component which you will wire up in your Spring configuration. So... bring up your applicationContext.xml or whatever you have it named and add the following entries:


<!-- our SpringComponent which binds us to the Restlet servlet configured above -->
<bean id="restletComponent" class="org.restlet.ext.spring.SpringComponent">
         <!-- the defaultTarget for this component is our Restlet Application -->
	<property name="defaultTarget" ref="myRestletApplication" />
</bean>

<!-- your Restlet application. This class extends "org.restlet.Application" -->
<bean id="myRestletApplication" class="my.restlet.MyRestletApplication">
         <!-- all requests to this Application will be sent to myPath2BeanRouter -->
	<property name="root" ref="myPath2BeanRouter"/>
</bean>

<!-- This router automagically routes requests to beans that extend org.restlet.resource.ServerResource or org.restlet.Restlet who's name starts with a "/" slash which matches the request-->
<bean name="myPath2BeanRouter" class="org.restlet.ext.spring.SpringBeanRouter"/>
 
<!-- This extension of org.restlet.resource.ServerResource bean will handle all requests to made to /my/REST/api/myResource (GET/POST/PUT etc) 
This class extends "org.restlet.Restlet" or "org.restlet.resource.ServerResource" -->
 <bean name="/myResource" autowire="byName" scope="prototype" 
    		class="my.restlet.package.resources.MyResourceBean">
    		
    	<property name="somePropertyOfMine" ref="someOtherSpringBean"/>
 </bean>

Ok, well if you were having trouble trying to get Spring working with Restlet I hope this helped get you rolling. Restlet is a cool project that works great and can get a REST API up and running pretty quickly (granted you are good at crawling through somewhat scattered documentation) Here are a few other links which you may want to reference:

Restlet 2.0 Extensions API
Restlet 2.0 JEE API

Also, I am posting the following error messages that troubled me when trying to get this to work. The configuration I show above was the result of getting beyond the below errors by using the correct fixed releases. To AVOID the errors below, ENSURE you are using Restlet 2.0 M5 or a custom build from the trunk. Prior to 9/25/09 people were getting the errors below.

Message ID
No target class was defined for this finder
Complete Message org.restlet.ext.spring.SpringFinder $$ EnhancerByCGLIB

AccessDecisionManager does not support secure object class…

This post will be short and sweet, but for those of you using Spring Security and come across this exception, hopefully this post will be of some help to you. Here is the exception:

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name '_filterChainList': Cannot resolve reference to bean '_filterSecurityInterceptor' while setting bean property 'filters' with key [3]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name '_filterSecurityInterceptor': Invocation of init method failed; nested exception is java.lang.IllegalArgumentException: AccessDecisionManager does not support secure object class: class org.springframework.security.intercept.web.FilterInvocation 
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:275) 
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:104) 
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveManagedList(BeanDefinitionValueResolver.java:287) 
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:126)

I received this exception today when I began to add some http/web level security into my application which already had some pre-existing custom configuration for my own AuthenticationManager, AccessDecisionManager (w/ custom voters) and AuthenticationProvider. When I added the following configuration to my Spring configuration the above exception was thrown on the next Maven test run:


... custom auth provider, auth manager, voters above....

<!-- my custom accessDecisionManager config -->
<bean id="accessDecisionManager" 
	class="org.springframework.security.vote.AffirmativeBased">
	<property name="decisionVoters">
		<list>
			<ref bean="myCustomAccessVoter"/>
		</list>
	</property>
</bean>

<!-- the NEW entry I added which triggered the exception -->
<security:http access-decision-manager-ref="accessDecisionManager">
    	<security:intercept-url pattern="/**" access="ROLE_USER" />
    	<security:form-login/>
</security:http>

The reason for the above exception is because my AffirmativeBased AccessDecisionManager did not have any decisionVoters who responded with “true” when passed a “FilterInvocation” object to their supports(Class clazz) method.

If you have a custom AccessDecisionVoter like I did above, you need to begin returning true in calls to support(Class clazz) when passed instances of “FilterInvocation” objects, while still only returning true for only the ConfigAttributes you care about when support(ConfigAttribute attr) is called on your voter. Secondly you should add an RoleVoter to your list of decisionVoters for your AccessDecisionManager configuration.

There is more to it than just the quick fix listed above, however that is for you to implement in your application. I just wanted to post this to give people a pointer in the right direction as to what is causing this exception. Hope it helped!

AbstractTransactionalJUnit4SpringContextTests Failing to Rollback in MySQL?

Ok so today I was working on some JUnit tests within Spring using AbstractTransactionalJUnit4SpringContextTests (yes, wow what a long class name). For those of you unfamiliar with this class, basically if you extend your unit test class from this class, every @Test annotated test method will run within a transaction, with the default behavior being that a rollback is called after each @Test method’s execution (unless you declare so otherwise). Pretty convenient.

My simple test case was leveraging a DAO that derived from HibernateDaoSupport and the test inserted a few records. When the test was complete, another method annotated with @AfterTransaction would verify that the data did NOT exist in the table, which was expected due to the rollback that occurs after each test case…… well my asserts were failing because Hibernate created the MySQL tables using MyISAM, which does not support transactions. If you encounter this kind of issue, all you have to do is change your Hibernate dialect to use the MySQL5InnoDBDialect (for InnoDB storage which does transactions) within your session factory configuration as such:

...
<property name="hibernateProperties">
 <props>
<prop key="hibernate.dialect">org.hibernate.dialect.MySQL5InnoDBDialect</prop>
...

Review: Spring Recipes

springrecipesThis is a review of the book “Spring Recipes” by Gary Mak

I picked up this book as a supplement to the “Spring in Action” 1st/2nd editions that I have. This book takes the same approach as the O’Reilly Cookbook series and covers Spring 2.5. Basically it is a task/problem oriented book. Each little section addresses a small specific task such as “Unit testing Spring MVC Controllers” or ” Checking Properties with Dependency Checking”. Despite the task oriented recipes approach, the book is still structured in a progressive manner whereby a newcomer to the framework could start reading from the start to the end, learning the basics to the more complex. Each recipe is structured with a “Problem/Solution/How it works” section which presents a consistent approach to all content within the book. The examples are clear and in many places the author provides helpful diagrams to help the reader visualize the relationships involved in a particular problem.

I’ve gone through about 1/2 the book and stopped until I had an actual need for a specific recipe, however I do recommend the book and it is something I have cracked open as I’ve encountered problems.

Recommended: YES
Skills: Java / Beginner’s all the way to Advanced

Review: Spring in Action

siaThis is a review for “Spring in Action” 2nd Edition by Craig Walls

Spring in Action covers 2.0. The book is excellent and walks the reader through the basics of IoC, bean injection, AOP and then on to virtually every major component of this framework. For the database side of things, they cover basic JDBC usage, Hibernate, iBATIS, and JPA etc. Beyond that, Spring-WS, JMS, EJB 2.x and 3, JNDI and JMX integration is given excellent coverage. Then on to the client side it covers Spring MVC, Webflow, how to integrate w/ Struts, WebWork, JSF and even some DWR.

The book is huge covering over 700 pages and mine is currently tagged with about 50 page flags marking off all the topics I need quick access to. I read the book mainly to get up to speed on components I was unfamiliar with and would recommend this to anyone who does anything with Spring. The content in here is excellent and walks you through the entire framework in a logical approach, starting with the simpler stuff up to the advanced.

Recommend: YES
Skills: Java, Intermediate to Advanced. Excellent desk side reference for the heavy user.