LogMX

In a previous post I described some benefits of using Apache Chainsaw to help you analyze log output from log4j.  Despite some significant quality problems, I still found a useful tool while working with log4j output (hey what do you expect for free!).  I have search many times for a log viewer that is better, and there are some very good ones out there (I like LogExpert in particular).  But sadly most of them tend to be to general purpose to excel as a Java debugging tool.

FINALLY the other day I founds something better than Chainsaw.  Way better.  It's called LogMX:

LogMX that includes the nice things that I like in Chainsaw, in addition to a lot of other useful and unique features.  And importantly, LogMX is polished, rock solid, zippy, and somewhat easier to use.  Take a look at the features and screenshots.

I've only tried LogMX with log4j, but it claims it is compatible with all the major logging frameworks, in addition to any "simple" log file format.  To use LogMX with log4j, you just need to configure an appender with the HTMLLayout or XMLLayout.  For example:

        <appender name="logmx" class="org.apache.log4j.FileAppender">
                <param name="File" value="target/logmx.log" />
                <param name="Append" value="true" />
                <param name="Threshold" value="TRACE" />
                <layout class="org.apache.log4j.xml.XMLLayout"/>
        </appender>

Then from within LogMX you just do File -> Open and point it to your log file.  

One important  thing to be aware of is LogMX is a commericial program, but you can use it for free for non-commericial use, and the commericial license is not unreasonable at $99.

Apache Chainsaw Jumpstart

It's a tedious job to filter and sift through log files to try to understand the complex behaviour of large systems.  Tail and grep are a good place to start, but there are many other more powerful GUI based options designed specifically for log analysis that make the job a little easier.  For java, one log viewer that I find to be useful is the venerable Apache Chainsaw.  As a GUI app,  first impressions are not great.  Chainsaw presents itself with the very unappealing classic Swing look and feel that will probably drive most users away immediately.  However, putting aesthetics aside, Chainsaw is a well thought-out app with some nice features.  


One feature I particularly like is the tree control that corresponds to the heirarichal "categories" of the log4j output, which in turn correspond to java packages.  Providing a tree control to help filter based category/package is just awesome, and I'm suprised there aren't more log viewers that apply this idea.


Chainsaw provides a number of different integration options.  I have had mixed results with SocketAppender and "Zero Touch Config" but using plain old log files works great.  The only hurdle to overcome is writing the log files in a format that Chainsaw can understand, and then configuring Chainsaw to read the file. Using a file based approach it should also be possible configure Chainsaw to read log files produced by other logging frameworks such as jdk-logging or logback.  


Here is the info you need to quickly get you started with log4j and chainsaw:


This the log4j appender configuration I use for Chainsaw. (just paste this into your log4j.xml)  This will create an additional logfile named "chainsaw.log" containing all the fields that Chainsaw expects: 

<appender name="chainsaw" class="org.apache.log4j.DailyRollingFileAppender">
    <param name="File" value="chainsaw.log" />
    <param name="DatePattern" value="'.'yyyy-MM-dd" />
    <param name="Append" value="true" />
    <param name="Threshold" value="DEBUG" />
    <layout class="org.apache.log4j.PatternLayout">
        <!-- date|category|class|file|lineno|method|priority|thread|message -->
        <param name="ConversionPattern" value="%d{ISO8601} %c %C %F %L %M %p %t %m%n" />
    </layout>
</appender>

Here's the corresponding chainsaw configuration file.  Unfortunately you have to edit this file to provide the absolute path to the chainsaw.log file.  Then from within Chainsaw you can load this configuration by doing "Load -> Load Log4j File" from the  menu.  If everything is configured properly you should see a new tab in the Chainsaw UI for your logfile.

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration >
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/" debug="true">

       <appender name="A2" class="org.apache.log4j.ConsoleAppender">
              <layout class="org.apache.log4j.SimpleLayout" />
       </appender>

       <plugin name="LogFileReceiver" class="org.apache.log4j.varia.LogFilePatternReceiver">
              <param name="fileURL" value="file:///c:/--path-to--/chainsaw.log" />
              <param name="timestampFormat" value="yyyy-MM-dd HH:mm:ss,SSS" />
              <param name="logFormat" value="TIMESTAMP LOGGER CLASS FILE LINE METHOD LEVEL THREAD MESSAGE" />
              <param name="name" value="sampleLogFilePatternReceiver" />
              <param name="tailing" value="true" />
       </plugin>

       <root>
              <level value="debug" />
       </root>
</log4j:configuration>

Better Tab Completions for GroovyShell

GroovyShell provides a text console based environment for interacting with Groovy code. It allows you to type in and execute groovy statements and see the results immediately.  To make the shell a more confortable environment for users to work in, GroovyShell enlists the help of JLine to provide powerful line editing, command history and tab-completion. 

One area in which I feel GroovyShell could use some improvement is tab-completion.  When you hit [tab], the list of "candidates" that GroovyShell provides to JLine are currently limited to the build in shell commands (such as exit, import, inspect, etc).  This is a good start, but the thing that would make the environment immensely more usable is for the shell to supply candidates for variable and method names within the current shell context. 


For example suppose you define a variable in the shell:


     inputFile = new File("data.csv")


When you type in the shell:

    in[tab]   

the shell could go ahead and complete the variable name "inputFile" for you.  Other modern shells and IDE do an excellent job of providing this style of interaction.  When using GroovyShell I find myself compulsively slamming the [tab] key and wondering why the heck it's not working! 


So, I set out to see what was involved to pimp up GroovyShell in this way.  It turns out the task is quite simple.  All the raw functionality is already present in the the shell and JLine, it's just a matter of wiring it all up.

First I took a look at JLine to find out how to add additional completors.  The relevant extension point is the jline.Completor interface, which contains a single  method to be overridden:

int complete(String buffer, int cursor, List candidates)

When invoked,  buffer contains the current contents of the line the user is editing, cursor is the cursor position within that line, and candidates is what the completor must fill up with completions based on the current buffer and cursor position. Easy!

Next, I took a loot at the GroovyShell source to see how I could come up with the candidates.  There are basically only two kinds of completions I was hoping for.  The first is for global variables.  Once you have a reference to GroovyShell's "Shell" instance, these can be retrieved via:  shell.interp.context.variables, which gets you a map of all the globals  currently defined in the shell context.


The other kind of completion is when you already have one or more dots in the expression, optionally followed by some additional characters. For example, suppose you type in


    inputFile.isF[tab]


First, the shell is used to everything to the left of the final dot to resolve it to an object:


    def instance = shell.interp.evaluate(["inputFile"])


Once you have an instance, you can use reflection to look up all public fields and methods that match the prefix (in this case, "isF" which is a prefix for the "isFile" method.).


The last task is to add the new Completor to the JLine object within GroovyShell, without disrupting the functionality of the existing Completors.  I looked through the GroovyShell code, but unfortunately it appears there is no way to do this without making a code change.  Fortunately though the code change is very minimal.  In he InteractiveShell constructor:


    this.reader = new ConsoleReader(
        shell.io.inputStream, 
        new PrintWriter(shell.io.outputStream, 
        true))
    reader.addCompletor(new ReflectionCompletor(shell))
    this.completor = new CommandsMultiCompletor()

Here is source code for the ReflectionCompletor class:

package org.codehaus.groovy.tools.shell
import java.util.List;

import jline.Completor


/**
 * Implements the Completor interface to provide competions for
 * GroovyShell by using reflection on global variables.
 *
 * @version $Id$
 * @author Marty Saxton
 */
class ReflectionCompletor implements Completor {

    private Shell shell;

    ReflectionCompletor(Shell shell) {
        this.shell = shell
    }

    int complete(String buffer, int cursor, List candidates) {

        int identifierStart = findIdentifierStart(buffer, cursor)
        String identifierPrefix = identifierStart != -1 ? buffer.substring(identifierStart, cursor) : ""
        int lastDot = buffer.lastIndexOf('.')

        // if there are no dots, and there is a valid identifier prefix
        if (lastDot == -1 ) {
            if (identifierStart != -1) {
                List myCandidates = findMatchingVariables(identifierPrefix)
                if (myCandidates.size() > 0) {
                    candidates.addAll(myCandidates)
                    return identifierStart

                }
            }
        }
        else {
            // there are 1 or more dots
            // if ends in a dot, or if there is a valid identifier prefix
            if (lastDot == cursor-1 || identifierStart != -1){
                // evaluate the part before the dot to get an instance
                String instanceRefExpression = buffer.substring(0, lastDot)
                def instance = shell.interp.evaluate([instanceRefExpression])
                if (instance != null) {
                    // look for public methods/fields that match the prefix
                    List myCandidates = getPublicFieldsAndMethods(instance, identifierPrefix)
                    if (myCandidates.size() > 0) {
                        candidates.addAll(myCandidates)
                        return lastDot+1
                    }
                }
            }
        }
        
        // no candidates
        return -1  
    }

    /**
     * Parse a buffer to determine the start index of the groovy identifier
     * @param buffer the buffer to parse
     * @param endingAt the end index with the buffer
     * @return the start index of the identifier, or -1 if the buffer
     * does not contain a valid identifier that ends at endingAt
     */
    int findIdentifierStart(String buffer, int endingAt) {
        // if the string is empty then there is no expression
        if (endingAt == 0)
            return -1
        // if the last character is not valid then there is no expression
        char lastChar = buffer.charAt(endingAt-1)
        if (!Character.isJavaIdentifierPart(lastChar))
            return -1
        // scan backwards until the beginning of the expression is found
        int startIndex = endingAt-1
        while (startIndex > 0 && Character.isJavaIdentifierPart(buffer.charAt(startIndex-1)))
            --startIndex
        return startIndex
    }


    /**
     * Build a list of public fields and methods for an object
     * that match a given prefix.
     * @param instance the object
     * @param prefix the prefix that must be matched
     * @return the list of public methods and fields that begin with the prefix
     */
    List getPublicFieldsAndMethods(Object instance, String prefix) {
        def rv = []
        instance.class.fields.each {
            if (it.name.startsWith(prefix))
                rv << it.name
        }
        instance.class.methods.each {
            if (it.name.startsWith(prefix))
                rv << it.name + (it.parameterTypes.length == 0 ? "()" : "(")
        }
        return rv.sort().unique()
    }

    /**
     * Build a list of variables defined in the shell that
     * match a given prefix.
     * @param prefix the prefix to match
     * @return the list of variables that match the prefix
     */
    List findMatchingVariables(String prefix) {
        def matches = []
        for (String varName in shell.interp.context.variables.keySet())
            if (varName.startsWith(prefix))
                matches << varName
        return matches
    }
}