Turnleaf Design

Log4j 4 Enterprise

Billy Korando — Wed, 06 Oct 2010 06:24:39 +0000

Logging is an important part of any application. Logging provides debugging information, the state of the application, and a record of what happened when the application failed. As applications increase in size so does the demands and complexity of logging. Without a proper logging system it can become difficult to determine from where in an application a log statement is being executed and without an easy way of changing logging behavior it can have a negative effect on an application's performance.

The Apache Foundation maintains the log4j project which is an easy to use system that provides a large array of really cool features for logging. This tutorial, part of my dev environment series, is an introduction into using log4j in an enterprise setting. Topics that I will cover include; implementing log4j in a project, customizing the log statement, usage of appenders, creating loggers and logger inheritence, and setting up log4j to work on an application server. This will cover many of the basic needs of enterprise logging.

A few simple steps before we begin...

As with all my tutorials that involve coding you can download the source code here. In this example I will be using the project HttpExample, starting on revision 24. If you want to work on this code locally you will also need the log4j library which can be found here, and then add it to your project's classpath by right clicking on your project > build path > add external libraries and then navigate to the library's location (if you are using eclipse).

Log4j, it's better than bad, it's good!

Implementing log4j in a project is surprisingly simple. I already covered the first step of adding log4j to the project's classpath. The second step is setting up the configuration file. If you look in the project you will see I already created one, log4j.xml, which looks like this:

The final step is to add a logger to a class and log statements, which I have also already done so in NewUserFormController:

com.tld.form.controller.NewUserFormController

public final class NewUserFormController {
private static Logger LOG = Logger.getLogger(NewUserFormController.class);

...

protected static String createUser(UserFormBean bean) {
LOG.trace("trace");
LOG.debug("debug");
LOG.info("info");
LOG.warn("warn");
LOG.error("error");
LOG.fatal("fatal");

...

}

In a previous article I created some unit tests for this project, if you try running one of the unit tests under TestNewUserFormController the following should be printed to the console:

DEBUG - debug
INFO - info
WARN - warn
ERROR - error
FATAL - fatal

Not much more advanced than using the standard system.out. There seems to be a problem however, one of the log statments, LOG.trace("trace");, was not printed to the console. This actually is not a problem and brings me to the first feature of log4j I will go over.

Not all log statements are created equal

All applications more or less go through the loosely defined stages of; development, qa testing, and finally production. The demands of logging change as much as what occurs in each of these stages. In response to these demands log4j offers six default levels of logging (from lowest to highest); trace, debug, info, warn, error, fatal, these levels give developers relatively fine grained control of which log statements are executed. This aids developers by allowing us to change the amount of logging an application is executing easily and without making a binary change.

To change the logging level of our application we will need to go into the configuration file and add a "level" tag within our "root" tag. Within this tag is the attribute "value" which we set with the minimum level at which to begin logging. So if we set the level to "debug" (the default) all log statements of the level of "debug" or higher will be executed. Here is an example:

With the logging level set to trace, if we run one of the unit tests again all six statements should appear.

Customizing the logging message

So log4j has given us control over which statements are printed, currently we are using the SimpleLayout which, though quick, doesn't offer a lot of information about the statements being printed. Log4j offers the PatternLayout class which gives the developer a wide degree of latitude in customizing his log statements. Lets get our feet wet with this system by adding a timestamp and the fully qualified name and line number to our log statements. Open up log4j.xml again and make the following changes to the appender tag:

Run an unit test and the output should look something like this:

Sep-29 00:39:33 TRACE Sep-29 00:45:57 TRACE com.tld.form.controller.NewUserFormController.createUser(NewUserFormController.java:18) - trace

Definitely a lot more useful as we know exactly where the statement is being executed from and when the statement was executed (not particularly useful here, but absolutely necessary for a deployed application). There is a lot more options for customization, for more information check out this link. It's important to consider the performance impact modifications may have as well, adding the fully qualified name to a log statement has a significant performance impact. The link above notes if an option has a significant impact on performance.

Appender? I barely knew her!

Right now all of our log statements are printed out to the console. This is great for when debugging in an IDE, but eventually we will deploy this application to the server and we don't want these log statements to be sent to the standard server log. A lot of information is already dumped to this log and if we deploy multiple applications to a server it would be very difficult to differentiate which application is printing what statement. Again log4j comes through by making it simple to change where a log statement is printed. Again open up log4j.xml and add the following lines (note this is a new appender tag):

Then add the appender to the root tag like so:

Finally rerun one of the unit tests, refresh your project and you should see a file titled "myLogFile.log" with the log statements contained within.

Clearly this is an unorthadox (and not particularly useful) way of handling logging. Later in this tutorial we will send this output to a more reasonable location, but for now on to loggers.

Setting up separate logs and logger inheritence

So far we have been using the root logger (the root tag in the log4j.xml file). For a small application with few dependencies like I am demonstrating, it is difficult to show the issues that arise from only using the root logger. As an application increases in complexity it often pulls in additional libraries, just like a developer may setup step filters in a debugger to ignore portions of an application or libraries he doesn't care about, we can setup loggers to ignore log statements from certain parts of an application or direct log statments to a different location. Lets explore this practice as best we can, given the limitations of this example, once again open up log4j.xml remove the "root" tag and make the following changes:

If you run an unit you will see the output was the same as before. Now go add couple of log statements to UserDaoImpl like so:

public class UserDaoImpl implements UserDao {

private static final Logger LOG = Logger.getLogger(UserDaoImpl.class);

@Override

public boolean doesUserExist(UserFormBean bean) {

LOG.debug("Debug");
LOG.error("Error");
...
}
...
}

Again run the unit test and you will see these two log statements added to the original six. Now go back into log4j.xml and add the following:

Rerun the unit test and you will see only the "error" log statement was printed from with in UserDaoImpl.

There are two important and closely related concepts shown in the examples above; logger retrieval and logger inheritance:

1. The way I have been retrieving logger by passing the class is the industry standard. When you call "Logger.getLogger(class)" log4j checks its list of declared loggers and returns the closest ancestor to the passed in name, if none are found than the root logger is returned. Above the logger for the class "NewUserFormController" uses logger "com.tld.form" because its logger name is "com.tld.form.controller.NewUserFormController" where as the the class "com.tld.form.dao.UserDaoImpl" uses the logger"com.tld.form.dao" because that is its closest ancestor.

2. Just like Java offers inheritance so a developer can reuse attributes in like ancestor classes, log4j supports inheritance for the same reason. In the above example we didn't have to redefine an appender for loggers "com.tld.form" and "com.tld.form.dao" because both inherited the appender from the root logger (which all loggers inherit from). Also like in Java attributes inherited from a more recent ancestor override those of an older ancestor. For a more in depth explanation of how inheritance works check here.

Server?! I barel... never mind

So log4j is working all fine and dandy in Eclipse, but eventually we will move this application to a server and we will still want logging to take place. To do this right we will have to make several changes.

The first change is correcting a mistake I made earlier which involves the location of the log4j config file. One of the strengths of log4j is the ability to make a change to the logging behavior of an application without making a binary change (the change does not involve recompiling and redploying code). While not absolutely necessary we will want to move the log4j out from under the src folder and create a new folder called "resources" and place it in here. This will show other developers that log4j.xml is a separate deployable from the source code.

[Note: you will also want to include a copy of log4j.xml under test, so logging can still take place while running unit tests]

Since log4j is no longer on the classpath of our project we will need to tell our application where to find the configuration file out on the server, however this should only happen once and at application start up. To do this we will need to create a servlet listener. If you look under com.tld.form, you will see I already wrote one:

public class LoggerListener implements ServletContextListener {
private static final String LOG4J_CONFIG = "log4jConfig";

@Override
public void contextDestroyed(ServletContextEvent arg0) {
}

@Override
public void contextInitialized(ServletContextEvent arg0) {
DOMConfigurator.configure(arg0.getServletContext()
.getInitParameter(LOG4J_CONFIG));
}
}

A servlet listener is hook for a developer to run actions during an application start up or shut down ( contextInitialized and contextDestroyed respectively). They are quite handy in scenarios like this where you need to load global properties or configure something. Also I didn't cover this before, but there are two formats to log4j configuration files, in this tutorial we covered using the xml format, but there is also the properties format which is a text file that follows a specific syntax. This information is significant because if you are using the former like we are you will need to use DOMConfigurator, if you are using the latter however you will need to use PropertyConfigurator to configure log4j.

Next we need to update our web.xml to tell it we have a listener and to give the location of the log4j.xml file.


...

log4jConfig
/etc/glassfishv3/glassfish/domains/domain1/config/log4j.xml


com.tld.form.LoggerListener

...

The "context-param" element tells the listener where the log4j.xml file is located on the server. The listener tag is obviously designating a class in the project as being a servlet listener. Location of elements matter in a web-xml, so put these elements near the top of your web.xml file.

Next we will need to make a change to the build file. Since we are now referencing log4j classes in our application we will need to make sure these classes are available on the classpath of our application when we deploy it to the sever. To do this go into the build.xml file and add the "lib" tag to the war tag like so:

Now when we build the project the log4j library will be included within the distributable (the ear file).

Finally we will need to update the log4j.xml under the resources folder to get setup handle logging in a server environment. I made the following changes:

We want our application to be writing to its own log for readability and convenience purposes. I also made some changes to the log statement formatting; the fully qualified name and line number has been removed as it is a performance hog and I added a literal "" to delineate between log statements.

And that is it! Just make sure you place the log4j.xml file in the correct folder and everything should be working correctly! There are a few features like rolling logs that log4j offers which may be more useful in situation like this, but for brevity of an already long article I will cover that and a few other topics in a future article. If you have any question or comments let me know!

Sources:
http://logging.apache.org/log4j/1.2/manual.html
http://wiki.apache.org/logging-log4j/Log4jXmlFormat
http://logging.apache.org/log4j/companions/extras/apidocs/org/apache/log4j/EnhancedPatternLayout.html

Technorati Tags: Java, Logging, Web development

6 traits of successful developers

Billy Korando — Thu, 16 Sep 2010 04:40:47 +0000

While perhaps easier to get into than it should be, software development is a difficult and demanding field to be successful in over a long period of time. A lot is asked of developers as we are often given nebulous business requirements and asked to make a functioning system. In my experience here are six traits all successful developers seem to poses.

1.Pupil

For a developer to be successful over any period of time a continued commitment to learning is absolutely vital. The field is too large, too quickly changing, and too in depth for any developer to ever "know it all." Luckily there are a lot of resources a developer can use to learn; books, co-workers, classes, blogs, among many others.

In my personal experience I have found one of the best resources for learning have been my co-workers. Books, classes, and most online resources are largely impersonal and geared toward large audiences, they simply cannot provide the help that a co-worker familiar with your issue or habits can provide. Which leads me to my next important trait...

2.Mentor

As a developer understands he does not know it all, he will also realize that this same quality exists in all developers. It is important for every developer to act up this knowledge and be willing to help other developers learn. It is particularly important when dealing with junior developers, those fresh out of school. Often these developers know just enough to be dangerous and do not realize this yet, so it is important to teach junior developers good practices, to prevent them from picking up (and potentially propagating) bad practices.

There is a responsibility in being a mentor and that is ensuring the knowledge being passed on is good. I have taken this responsibility for granted in the past, at the time I didn't understand the importance of getters and setters and I irresponsibly suggested not to use getter setters. Mentors who take this responsibility seriously however end up helping themselves just as much they help others. In order to understand why a practice is good a deep understanding is often required and understanding the fundamentals of a concept in my experience offers the most benefit.

3.Flexible

As alluded to under the pupil trait, software development is a rapidly changing discipline. Developers must constantly be looking at and reviewing their practices to see where improvements can be made. Much of the time the improvements are small and incremental, but occasionally entire upheavals must take place. A developer must be willing to accept that one day practices and technologies once relied upon have become obsolete and entirely new technologies and practices must be learned. Clearly this isn't an endorsement to blindly jump on every new trend, but to be aware of how the industry is evolving, familiar with the emerging technologies and practices, and willing to learn them should your job or the industry demand it.

4. Tester

Testing isn't simply a responsibility of the QA team. It is a core responsibility of a software developer. Testing isn't just part of the development process, but even the design process, from both a business and system perspective. The system perspective is somewhat obvious, but testing can also play a role in change business requirements as well; scenarios could be discovered that were not previously accounted for, or a technology may be found to be inadequate to meet business needs.

Testing, specifically unit testing, is also an important part of writing quality and maintainable code. In the same way a structural engineer writes mathematical models to test his design and eventually creates scale models, a software developer writes unit tests to test his code. Unit tests give a developer instant feedback on rather the changes made broke the system, or fixed an issue. Just like it is a responsibility of a mentor to pass on good knowledge it is the responsibility of developers to write good tests. Good unit tests replicate real world scenarios as well as hit fringe scenarios and negative scenarios. Obviously such a task would be extremely laborious to write for an entire system, so it is important for developers to recognize when this practice must be applied.

5. Farmer

Just as a crop is a farmer's livelihood code is a developer's. A lot goes into growing crops, a farmer must; pick good seeds, ensure the fields are fertile, and be ever vigilant against pests. Similarly it is every developer's responsibility to; ensure the code he adds to is of good quality, ensure there are ways of monitoring it's continued quality, and remove bad code when found.

6. Reporter

Every so once in a while a picture comes along to perfectly describe a point. Few things are more infuriating to a developer than to be told there is a problem and given no additional information. It considerably increases the time turnaround time to fixing an issue (if one actually exists) as a developer must track down what he thinks is the problem.

This issue isn't just relegated to when a problem arises or just non-developers (which is why this trait is included). As a company grows its developers become increasingly specialized and responsible for individual systems within a collective whole. Before asking a question a developer should read up on a system's documentation (a lot of times a question need not be asked), good questions lead to good answers and ultimately a lot less frustration on both sides. Conversely a developer should also provide good documentation on the system(s) he is responsible for.

So there are six traits every successful developer should have. If you have suggestions for traits I should add or disagree with the ones I listed leave a comment.

Technorati Tags: Best practices

Web developers should hate http

Billy Korando — Mon, 23 Aug 2010 13:00:32 +0000

I don't mean that literally, what I do mean is a web developer should hate when Java's http specification is directly accessed in the code. This is a topic I briefly touched on in my 8 signs your code sucks post awhile back. I decided to return to this topic because it touches on a very important and fundamental programming practice which is limiting dependencies in code. This article represents the initial entry into a new series I plan on covering called "Java don't care." As aforementioned this series will focus on reducing dependencies in code, as well as making your code less aware of the environment it is in, and how following these practices will help you as a developer. But back to the task at hand; why referencing Java's http specification is bad.

The problems

A long list likely could be made as to why you should not directly reference http in your code. Here are four of the more important:

Testing

Probably the most readily noticable and debately the most important; referencing http in code greatly increases the difficultly of writing tests. To even begin writing a unit test you will need a tool to mock the http request/session. This increase in the difficulty of writing unit tests leads to two problems; decreased unit test coverage and increased unit test complexity.

Contract

A somewhat overlooked issue, but important none the less, is the lack of a contract between the view and the backend code that supports it. There is no central location to figure out what is contained in the request/session. The HttpServletRequest does not (and could not) provide this information. This lack of a contract can make it difficult to understand what is happening in the application as well as what the developer has access to.

Coupling

Directly accessing the http specification makes the code more aware of a view's internal implementations than it should be. The controller, and certainly the model, should not know the names of the various fields and variables in the view. It creates a mud ball of an application where changes to one level unduly necessitate changes to other levels.

You also introduce an unnecessary requirement into the application. Obviously requiring a web based application to include the httpservlet library in its classpath isn't particularly burdesom, but in other areas it could represent an issue. In general it is a bad programming practice to force requirements on an application.

Maintainability

Fundamentally this is more of a meta-issue, it is the inevitable outcome of the above three issues (as well as others). When you have an application that is difficult to write unit test for, hard to figure out what is accessible where, and has unnecessary requirements to work, you have an application that is difficult to maintain. I would argue that writing a maintainable application is the second most important requirement of an application (the first of course being one that it meets business requirements).

How to fix it

There are many differnet ways to fix and/or prevent this issue from occurring. Many frameworks, like Spring, (can) handle the session and request for you. This would also typically be the best way of addressing the issue. Let's assume, for the purpose of this article, that such an option is not available. Below is the controller of a simple application that I wrote that makes extensive use of the http request.

public class NewUserFormController extends HttpServlet {

@Override
protected void doPost(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
String nextPage = createUser(req);
req.getRequestDispatcher(nextPage).forward(req, resp);
}

private String createUser(HttpServletRequest req) {
List errorMsg = new ArrayList();
UserDao dao = new UserDaoImpl();
validateUserInput(req,
errorMsg);
if(!dao.doesUserExist(req, errorMsg)){
dao.insertUser(req, errorMsg);
}
if(errorMsg.isEmpty()){
return "Complete.jsp";
} else{
req.setAttribute("errorMsg", errorMsg);
setRequestAttributes(req);
return "Form.jsp";
}
}

private void setRequestAttributes(HttpServletRequest req) {
String fName = req.getParameter("fName");
String lName = req.getParameter("lName");
String eMail = req.getParameter("eMail");
req.setAttribute("fName", fName);
req.setAttribute("lName", lName);
req.setAttribute("eMail", eMail);
}

private void validateUserInput(HttpServletRequest req, List errorMsg) {
String fName = req.getParameter("fName");
String lName = req.getParameter("lName");
String eMail = req.getParameter("eMail");
String password = req.getParameter("password");
String confirmPassword = req.getParameter("confirmPassword");
if (isEmpty(fName)) {
errorMsg.add("Must enter a first name!");
}
if (isEmpty(lName)) {
errorMsg.add("Must enter a last mame!");
}
if (isEmpty(eMail)) {
errorMsg.add("Must enter an email address!");
}
if (isEmpty(password)) {
errorMsg.add("Must enter a password!");
}
if (isEmpty(confirmPassword)) {
errorMsg.add("Must confirm password!");
}
if(!password.equals(confirmPassword)){
errorMsg.add("Passwords must match!");
}
}

private boolean isEmpty(String string) {
return string == null || string.trim().equals("");
}
}

Entire source of the project can be viewed here

As you can see the access to the HttpServletRequest permeates near the entireity of the code base. As a result many of the issues I brought up are problems in this code (coincedence?). It would be difficult to write unit tests for this code, it's somewhat difficult to know what information is on the request, and I am dependant on the httpservlet in my code.

The fix

So clearly this code needs help. Our refacotring of the code has two major goals:
1. Create a contract between the view and the rest of the application
2. Reduce the depence on the http specification in the code

Typically in this situation I create a bean that holds all the fields in the request/session as well as a few additional meta fields (such as in this example the next page). This creates the previously mentioned contract. A developer can look at the bean class and know what fields are accessible in the application.

The usage of this bean will help accomplish goal number two. Instead of accessing the http request to get the data we need, instead I will dump all the data into the bean and access the bean and then reference it instead.

So here is what the new controller class looks like

public final class NewUserFormController {

private NewUserFormController(){

}

protected static String createUser(UserFormBean bean) {
UserDao dao = new UserDaoImpl();
validateUserInput(bean);
if(!dao.doesUserExist(bean)){
dao.insertUser(bean);
}
if(bean.getErrorMsg().isEmpty()){
return FormConstants.COMPLETE_PAGE;
} else{
return FormConstants.FORM_PAGE;
}
}

private static void validateUserInput(UserFormBean bean) {
if (isEmpty(bean.getfName())) {
bean.addErrorMsg("Must enter a first name!");
}
if (isEmpty(bean.getlName())) {
bean.addErrorMsg("Must enter a last mame!");
}
if (isEmpty(bean.geteMail())) {
bean.addErrorMsg("Must enter an email address!");
}
if (isEmpty(bean.getPassword())) {
bean.addErrorMsg("Must enter a password!");
}
if (isEmpty(bean.getConfirmPassword())) {
bean.addErrorMsg("Must confirm password!");
}
if(!bean.getPassword().equals(bean.getConfirmPassword())){
bean.addErrorMsg("Passwords must match!");
}
}

private static boolean isEmpty(String string) {
return string == null || string.trim().equals("");
}
}

As can be seen the HttpServlet dependency has been completely removed from this code. The code is a bit simpler as I removed pretty much all local variables and use the bean instead. However the best part is I can much more easily write unit tests (which I did).

So what does this mean to me?

Well these changes fix much of the four problems I mentioned above. However there is a couple other addtional benefits:

Increased modularity

If a new form was created that had a lot the same fields, but a few additional ones (say city and address), instead of writing a bunch of new code to handle this form, instead these classes; the bean and the controller (removing the final modifier) could be reused. The developer then only needs to write code and test for the new fields.

Separation of responsibilities

With a contract in place there is less of a requirement for the developer writing the backend to be involved in writing the view portion of the application (or vice versa). This allows developers (or designers) to better play to their respective strengths and makes parallel development easier.

You can view all the source code for the entire project here (revision 13 is the last revision relevant to this article). The project also requires the Java servlet package (2.3+)

Technorati Tags: Best practices, Java, Programming, Unit testing

An intro to writing ANT Scripts

Billy Korando — Sat, 14 Aug 2010 19:58:26 +0000

What is Ant?

A(nother) N(eat) T(ool) is a tool developed by the Apache Foundation for automating tasks in a project. It is primarily, though not exclusively used for Java projects. Most developers interaction with ant is through the xml specification, which is used to define specific operations.

The objective of this tutorial

Ant was designed with adaptability and expandability in mind, as a consequence it is used for many tasks and in many systems. As we get deeper in the dev environment series we will rely heavily on ant to make our lives easier and to carry out many important functions. For this tutorial however I will go cover what ant is primarily used for, building a project and creating a deliverable, in this case a jar file. This will provide a good intro into the usage of ant and how it works.

A simple ant script

So let go ahead and jump in, here is an ant script I wrote for one of my projects:

This is a simple build file, for a very (very) simple project which you can view here [http://turnleafdesign.googlecode.com/svn/!svn/bc/7/trunk/Ant%20Project/trunk/Ant%20Project/]. The script is fairly self explanatory but since this is a tutorial...

Project:

The default attribute refers to the target that is run by default when you run the ant build file. So if you were running in command line you would only need to type “ant” in the directory in which the build.xml file resides and the target”build-jar” would be ran. More relevant, from Eclipse (and I assume most other IDEs) you need only click the icon representing the build file to run the default target.

The basedir attribute represents the path from which ant builds off. The path is relative and follows the same conventions common to navigating file systems through the command line. A period represents the current directory in which the ant scripts resides (which is also the default).

Property:

A tag used to create variables, somewhat similar to Java variables in which you give them a name (though this is not required), a type, and a value. While only the location attribute is shown in the example above, there are many different attributes available.

Target:

Represents an executable operation in the ant script which can be ran from the command line (ant [target name]) or within an IDE. A target can run multiple tasks if needed, for example I could include mkdir, javac, and jar (all of which are considered tasks in ant) within the “init” target.

The depends attribute represents a target that must be run prior to executing the current target, so for the target “compile” to execute the target “init” must be executed. Multiple targets (separated by commas) can be included in the depends attribute with them running in order from left to right.

Mkdir, javac, jar

As briefly mentioned above, these represent tasks in ant. These are predefined operations that ant executes to carry out a task. Unsurprisingly there are a lot of tasks in the ant project. You can create your own as well [http://ant.apache.org/manual/develop.html].

A bit more about Jar and the jar task

The main purpose of this script is to build a jar, most with experience with Java know what a jar is but since this is a tutorial...

Jar is a file format specification developed by Sun which is based upon the zip format. The purpose of a jar is to aggregate many files, usually Java classes, into one archive. All jar files must contain a META-INF directory, and a manifest.mf file within that directory. The specification predominantly concerns what can be contained within the META-INF directory and the files there within. Java projects are often built into war and ear files, which when we get to that point I will go over.

Compared to the other tasks in the project there is a bit more going on with the jar task. So to better explain...

Jarfile represents the location where the jar file will be written

basedir represents the root directory of all the files that will be contained within the jar file

The manifest task is used to add additional metadata to the manifest.mf file, which is explained in more detail below.

The Manifest task and manifest.mf

Another task that is being executed which I did not touch on is the manifest task. In this example it is designating the main-class which is required in order to allow a jar to be executable (often jars are used as a source library for another project, and don't have (or designate) a main-class so they are not executable).

If you open up the jar file you see two directories; com and META-INF. META-INF as explained earlier is a standard directory in all jars, “com” represents a top level package name. Any number of other folder or files could be in here, depending up how the jar is built. But getting back on track; if you open up the META-INF directory you will see manifest.mf. This file contains metadata about the jar; manifest-version, ant-version, and created-by. All these are standard and are there to provide information as to how the jar was built. Also included is “Main-Class: com.ant.HelloWord,” this is really important because if you attempt to execute the jar the JVM will actually read this file looking for this property.

As we get deeper into this series we will utilize the manifest task more heavily to provide important metadata about the jars, wars, ears we will be building.

Tip of the iceberg

This is merely the tip of the proverbial iceberg. As suggested earlier we will increasingly rely on ant as we get deeper into the dev environment series. This tutorial will hopefully provide a good base for explaining what ant is doing for future reference.

Sources:

http://en.wikipedia.org/wiki/Apache_Ant

http://en.wikipedia.org/wiki/JAR_%28file_format%29

http://download-llnw.oracle.com/javase/1.5.0/docs/guide/jar/jar.html

http://ant.apache.org/manual/

Technorati Tags: ANT, Automation, Java

Setting up SVN over http on Linux

Billy Korando — Wed, 04 Aug 2010 14:35:48 +0000

One project I was planning on starting before I went on my extended hiatus was setting up a development environment, so it only seems natural that I start there again. I already went over how to setup a SVN repository, however in that example I used VisualSVN which require elements of Visual Studio and since I primarily focus on Java development, VisualSVN isn't a particularly good choice. So today I will go over how to setup a svn server in Linux (Ubuntu 10.04 to be exact).

Getting everything we need

From the terminal run the following command to get all the different packages we will need:

sudo apt-get install apache2 subversion subversion-tools libapache2-svn

Creating a repository

So everything is now downloaded and installed (wasn't that easy?!), let's get subversion setup.

First lets create a new folder to hold our repository.

sudo mkdir /repos

Now create an actual svn repository

sudo svnadmin create /repos

And just like that we have a svn repository. But we are not near done yet. You can look around in repos if you want, however we will be letting Apache handle the authorization and authentication so there is not much here for us.

Accessing subversion over http

Since most code shops don't have everybody working on one computer, to really make svn useful we need it accessible over http, something that apache does very well. So let's get started.

At the bottom of (/etc/apache2/)apache2.conf add the following:

DAV svn
SVNPath /repos
AuthType Basic
AuthName “SVN Repository”
AuthUserFile security/svnpasswd
Require valid-user

Make sure you are currently in your /etc/apache2 directory and run the following command:

sudo mkdir security

Navigate to the newly created security directory and run the following command:

sudo htpasswd -c svnpasswd [username]

You will then be prompted to enter (and reenter) a new password for the user. This will be the user you use to login into svn. If you want to add addition users drop the “-c” (which creates/overwrites a file). You can also add the password directly if need ([username] [password]) however, for security reasons, this is not recommended.

Setting up Permissions

We are getting closer, you should be able to navigate to the repository, however you will likely be getting a 500 error when you login. These errors are related to incorrect permissions settings so let's fix that.

Run the following command:

sudo useradd -U www

This will create the group “www” with the user “www.”

In httpd.conf add the following lines:

User www
Group www

This will specify the user and group that Apache will use

Finally go back to the root directory and run the following:

chown -R www:www repos

Be sure to navigate into your repos directory and run the following command (otherwise the command will be run against your entire file system [the -R means recursive]):

sudo chmod -R /repos 764 *

These last two commands put the the www user and group in control of the svn repo along with giving the proper permissions to read/write/execute. And just like that you have fully operational svn server accessible over http! Pretty sweet eh?

If you are still unable to access and/or update the repository check the apache error.log located under /var/logs/apache2. If you still cannot figure it out ask and I shall try to answer.

This tutorial is merely the beginning, for (much) more information on how to configure your svn server go here.

Sources:
http://www.section6.net/wiki/index.php/Setting_up_Subversion_in_Linux
http://blogs.usask.ca/slb534/

Tested and verified on Ubuntu 10.04

Technorati Tags: Subversion, SVN

Mocking JDBC Connections with MockRunner

Billy Korando — Wed, 11 Nov 2009 05:54:11 +0000

Last week I published an article, An Intro into Test Driven Development with Junit4, which got somewhat mixed reviews. One particular commenter on reddit suggested I didn't understand that unit tests shouldn't talk to its data source. I decided to do some investigating into his claims, you can check out my findings here. I was however intrigued none the less and started looking for a tool that would allow me to easily mock a JDBC connection. I found Mockrunner to be the easiest to use of the ones I have found. Below is a brief tutorial on how to use Mockrunner. (Keep in mind that I am still fairly new to using Mockrunner)

In the class I am testing I am explicitly calling the columns I am retrieving by name. So I need to add these column names to my mock ResultSet.

result.addColumn("Date");
result.addColumn("High");
result.addColumn("Low");
result.addColumn("Condition");
result.addColumn("ID");

Obviously I also retrieve data from a result set so I must also insert data into the mock ResultSet. There are several ways to do this, but the easiest I found was using a list. You must add elements to the list in the same order as you added columns to the ResultSet.

List rowItems = new ArrayList();
rowItems.add(new Date(0));
rowItems.add(72);
rowItems.add(56);
rowItems.add("Cloudy");
rowItems.add(1);

In my actual test case I can verify that I am actually gathering the contents of the result set correctly and that the correct SQL statement was run.

prepareRS();
ForecastDAO dao = new ForecastDaoImpl(getConnection());
List forecasts = dao.getAllForecasts();

assertTrue(!forecasts.isEmpty());
verifySQLStatementExecuted("SELECT * FROM forecast");
assertEquals(73, forecasts.get(1).getHigh());

This is but scratching the surface of MockRunner's potential and using mocks at large. I hope this brief tutorial was helpful in getting you started using MockRunner. You can checkout the entire test class here and the class that I am testing here. If you want to get an entire working version of the project be sure to read the technical guide. Let me know if you have any questions or want to add your own comments.

Technorati Tags: Mock, Programming, Testing, Unit testing

Ramblings: Should unit tests talk to a data source?

Billy Korando — Thu, 05 Nov 2009 04:48:40 +0000

Last week I published an article on test driven development. One person who read my article (briefly) suggested I did not know that unit tests shouldn't talk to their data source. I plan on covering how to mock JDBC connections later this week, however I wanted to do some research to see if what my critic says is an industry standard or a philosophical choice.

Surprisingly there seems to be relatively little information on this subject, or I have been incapable of finding information. From what I have gathered though, the theory that data sources should be mocked is sound, but its may not always be practical to implement it. Below is two reasons why data source connections should be mocked and three reasons why they should not.

Why you should mock connections

Speed – Talking to a data source is one the primary operations that slows an application's performance. Most data sources require some sort of network connection and retrieving data is often a processor intensive task. Obviously performing these tasks take time. Unit tests are most useful when they are run automatically and frequently to ensure changes being made are not breaking the application. Unit tests that are run automatically should be somewhat speedy, by mocking a data source connection, this can reduce considerably the amount of time it takes to run unit tests.

Remove the reliance on the data source logic – Unit tests should be testing relatively small parts of code, individual methods and/or small collections of methods. By connecting to a data source you are depending upon the logic of not only how you are connecting, but how the data source operates. This goes beyond the scope of what a unit test should be testing.

Why you shouldn't mock connections

It's time consuming – Writing mocks can be a quite laborious task. I often avoid writing mocks like the plague, because even relatively simple mocks can require a considerable amount of time to get running and running correctly. Even my very simple example of running a single query against a five column table took several hours to setup. Granted this was my first time creating a JDBC mock and future mocks would be easier and quicker to write, but it would still take longer than simply connecting to my database.

It's ugly – It takes a lot of typing to get even relatively simple mocks working. A lot of the rules that apply to how real business code should be written need not be followed when writing unit tests, that said looking at the code of a unit test shouldn't make your eyes bleed. Unit tests that cannot be easily maintained often become ignored when they break and broken unit test have no value (if ignored).

It's not real – Mocks don't really care what you put into or take out of them. Real data sources are often not so forgiving. If I misspelled a column or table name in a query, a mock would not pick up on this (you can verify the sql statements run, but again that could be misspelled), where as a real database would. By actually connecting to the real data source you can be more confident that the application will perform its intended tasks.

Why you should use your head

Even within the same project there will be instances where mocking a connection to a data source is the right choice and instances when you should actually connect to a real data source. In the early stages of a project using mocks can be more practical as the structure of the data source is more abstract and subject to change. Rewriting a mock is often quicker and easier than restructuring a data source. However as a project matures and the unit tests become more complex, connecting to the real data source may be more practical as it is not only (more) well defined, but writing the mocks begins to require more time. As with all practices, you need decide which one is best to follow (or not) based upon your requirements and restrictions. Though I think it is worth noting standards are standards for a reason. Feel free to weigh in with your own thoughts on the subject.

Additional reading:
http://www.javaranch.com/journal/2003/12/UnitTestingDatabaseCode.html
http://www.buunguyen.net/blog/unit-testing-the-data-access-layer.html

Technorati Tags: Best practices, Mock, Programming, Testing, Unit testing

Link dump 11/3

Billy Korando — Tue, 03 Nov 2009 06:07:01 +0000

http://www.developerart.com – A new blog like my own. While not a whole lot yet, the content that is on his site is of high quality.

Kind of a smallish link dump I know. I plan on getting some new articles posted this week, namely an update to my TDD article I posted last Monday. So check back soon.

Technorati Tags: Best practices, Programming

An Intro into Test Driven Development with JUnit4

Billy Korando — Tue, 27 Oct 2009 04:26:16 +0000

Please read the technical guide before starting this tutorial.

This article will mark the first of a long-term series covering professional software development. For the lowdown on this project check out this article. Be sure to give me your feedback as it will be vital in helping me develop better tutorials in the future.

Test driven development seemed like a natural choice as a lead off to my series of tutorials as I had to explain why I am writing all these tests. It is also a very good development methodology that will actually save a lot of time by reducing the amount of time spent debugging. For this tutorial and the entire project, I will be using Junit4. For a synopsis on test driven development you can check out the wikipedia article here. If you need a brief refresher on JUnit you can read my tutorial here (written in Junit3).

The goal of this tutorial, from a project stand point, is to get the connection to our (MySQL) database working. To do this we are going to need several things: a MySQL server, a client to access the server, and a JDBC driver to allow our application to access the server.

The MySQL server:
I will be using MySQL server 5.0 for this project, you can download it here. The tutorial instructions will be based upon the user being “root,” the password “turnleaf,” and the url “http://localhost:3306.”

The MySQL client:
I will be using HeidiSQL to query and manipulate the database. Once the client is installed import this sql file to setup the database and table.

The JDBC Driver:
You can download the JDBC driver here. Please include it on your projects build path.

To begin this project, please checkout revision 7 from the code repository.

Once you have downloaded the project go ahead and navigate around it a little bit. You will notice I already have several source files; Forecast, ForecastDao, and ForecastDaoImpl. Forecast is a bean for holding all weather information for a specific date (this bean will likely be modified in the future). We also have ForecastDao which is an interface and ForecastDaoImpl which implements ForecastDao. I plan on covering interfaces in more detail in the future, but if you are unfamiliar with the concept you can read a brief description here.

Update to revision 8 and you will see I added in my first unit test. The unit test is checking to see if I get any returns when I run getAllForecasts(). Here is the code:

@Test

public void testGetAllForecasts(){

List forecasts = dao.getAllForecasts();

Assert.assertTrue(!forecasts.isEmpty());

}

If you attempt to run the test you should get a null point error when you attempt to check if forecasts is empty. If you look at the implementation of getAllForecasts() in ForecastDaoImpl it is obvious why, getAllForecasts() is returning null. This is one of the tenants of TDD, write fail first tests.

Go ahead and update to revision 9. You will see I have added connection information to getAllForecasts(). Don't worry about it being messy we will refactor it later. Go ahead and run the unit test again. You will still fail, but in the console shows we are successfully connecting to the database.

Update to revision 10. I have made several more changes, most noticeably if you run the test it should now pass! (If not check to make sure your database is setup correctly) Before we start celebrating too much, lets actually check the contents of the list to make sure we are getting the right data. Update to revision 11, I have added a few more checks, and we are in fact getting the correct data, awesome!

Author note: For some dumb reason when you create a new Junit4 test class in Eclipse it does not automatically inherit TestClass. Anyways I inherit that class now so you can just do assertEquals(expected, actual) instead of Assert.assertEquals(expected, actual).
If you checked the implementation of getAllForecasts() before updating to revision 10, you will noticed I am only setting the date value of my forecast bean. My new tests initially failed (I originally thought I was setting all fields), thus the importance of not only writing unit tests, but writing good unit tests.

So now it is time to start refactoring. With our unit test we now have a good baseline of how the system should behave. This way when we are making changes we can be confident we are not breaking the system because we will be getting the same output for the same input.

Update to revision 13. You will see that I have created the getConnection() method and that contains the database connection logic.

private Connection getConnection() {

Connection conn;

try {

String userName = "root";

String password = "turnleaf";

String url = "jdbc:mysql://localhost/weather";

Class.forName("com.mysql.jdbc.Driver").newInstance();

conn = (Connection) DriverManager.getConnection(url, userName,

password);

} catch (Exception e) {

throw new DataAccessException(e);

}

return conn;

}

I pretty much just copied and pasted (one of the VERY few times copying and pasting is ok) the connection logic into this method. The most noticeable change I made is in the catch clause. There is a bunch of different exceptions that could be thrown when attempting to connect to the database, sine they all end in the same scenario, unable to connect to the database, I kept the same generic catch(Exception). However you should never just throw exception so I created my own custom exception called DataAccessException. Inside the super(Throwable) constructor I set it up to add a message stating “Unable to connect to data source,” I also have it extend RunTimeException turning it into an unchecked exception.

***Philosophical warning***
This is philosophical decision. As projects become more complex it can become difficult to catch and/or throw an exception at every level. However, you should catch the exception at some point to give the user a reasonable error message and let you know when the application is throwing exceptions. There is another school of thought that all exceptions should be checked as it lets a developer know what exceptions and method could throw, and other various reasons.

Between the message and the name of the exception, it should be pretty clear that when this exception is thrown it means the application failed to connect to the database. Running the unit test we will see everything is still passing so we can be confidant that the refactoring did not break the system.

Update to revision 14 and you will see I refactored out the closing of the connection. Exceptions should never be ignored, personally I would say being unable to close a database connection could be a major issue. So instead of catching Exception I changed it to only catch SQLException and throw the DataAccessException except I will have my own message in their stating the connection could not be closed. Here is what the closeConnection() method looks like:

private void closeConnection(Connection conn) {

if (conn != null) {

try {

conn.close();

} catch (SQLException e) {

throw new DataAccessException("Could not close connection", e);

}

}

}

From what it originally looked like, the getAllForecast() method is starting to look a lot better as well as the ForecastDaoImpl class in general. Adding new methods that retrieve data from the database will be easier as all I have to do to get a connection is call getConnection() and to close it closeConnection(Connection). On top of that if I need to change my database connection information I only have to do it in one area.

Despite this refactoring, this is hardly an ideal data access layer. With the test unit already created go ahead and continue to refactor getAllForecast(), a good place to start would be how I create a new Forecast object. I would also suggest adding some new functionality yourself a couple of examples might be; getForecastByDate(Date) or getForecastByCondition(String condition) (Remember create unit tests!). I will continue to refactor and update this work myself. So you can run updates and compare your work to my own.

Addendum and thanks:
Because I am an idiot I couldn't remember how to write a JDBC connection. My initial connection method is predominantly based off of the code in this article: http://www.kitebird.com/articles/jdbc.html
For being able to go to a specific revision with a url:
http://www.perhammer.com/2008/07/subversion-in-url-revision-browsing.html

Technorati Tags: Best practices, Java, Programming, Unit

The start of a long journey

Billy Korando — Tue, 27 Oct 2009 04:03:22 +0000

A couple of weeks ago I posted about my ideas for the future of this site. Well today I start making good on those lofty promises. I will be posting my first tutorial, which will cover test driven development, shortly. Before I do that I want to give a bit of an explanation of this project, why I am doing it, what I hope to accomplish, a few other thoughts, and finally some technical notes.

The what

As I said in my previous post, this project's main goal is to check to see how accurate meteorologists really are with their predictions. NOAA offers a web service for checking forecast information and I will be using that to gather my data. Just following the exact requirements of those project will give plenty of opportunities to learn new things, but I also intend to meander around a bit so we can learn a lot more. So when I start adding user sign in and other seemingly odd/unnecessary features, now you know why.

The why

I intend for this project to serve a duo purpose, to teach others and to learn a thing or two myself. One of my main goals with this site is to help teach other developers. There where many important subjects that I did not fully grasp in my early career and it really held me back. Luckily I was given the opportunity to learn from my mistakes as well as those around me. I hope to repay some of this debt by helping others.

There is a lot to software development, a lot. Sometimes it is hard to know where to begin, but it is said necessity is the mother of invention and I hope both intentionally and unintentionally this project forces me to learn things as it is the only way to (effectively) solve a problem.

A few words on practices

Occasionally there will be bad practices in this project. It will be a mixture of both intentional and unintentional. I plan on making example of why those practices are so bad, and sometimes the only effective way to do so is by actually showing the technical debt taken on by those practices. When I do intentionally make a bad design choice, I plan on explicitly stating so and I will also show how to fix the issues resulting from those practices.

That said, I am both human and a junior developer. I am going to make mistakes, lots of them. You don't need to burn me at the stake. If you see a problem in my code, please leave a comment and if possible offer a solution. If I think you are correct I will make amendments to my articles to correct any problems.

There will be philosophical differences. When I am cognizant of differing views points on a subject, I will state my design choice is philosophical. If you are of an opposing view please speak up and say why your choice is better, but it has to be something more substantive than “you are wrong.”

Lastly all practices are guidelines. For every good, great, or best practice, there is going to be a few examples out there that contradict the practice. Thus they are practices and not rules. Understand that there is no possible way I could be familiar with every person's business domain or systems the you work on or use. I have to write with a certain degree of generality.

I need you

I can only fix what I know to be broken. If you see a problem, want me to cover something in more detail, or anything else, please let me know. Your feedback is absolutely vital in the process of making this site and the content I publish here better. I will do the best I can to self correct, but I can only do so much alone.

Technical notes

All coding work will be done utilizing Eclipse 3.5 (Galileo), you can download it here. I will be using subclipse as my primary version control client. If you need help using subclipse check out my subclipse primer. I will be utilizing several other technologies as this project because more advanced. As I do I'll let you know what I am using and where to get it. Hopefully in the future I will be able to do this all automatically, though network/financial issues may prevent this.

I'm still contemplating on how to get a live version of this project hosted. I think I may be able to use Google's app engine, but I am still unsure if it will fully mesh with the goals of this project. If you know of free or cheap Java hosting let me know.

This project is published under the MIT license. Which as far as my understanding goes allows you to do pretty much anything with the source code. Rather that be the case or not you do have my permission to use the source code in this project, at least the code I personally write, to whatever purposes you see fit (I'd prefer good). Attribute me if you want, I don't really care.

Well anyways I hope you like this series of tutorials. It's going to be a long, but I hope a very fun and enlightening, journey.