End of last year I wrote about my first proof-of-concept with AuthorDoclet, a documentation tool I've been working on. It took me a while to get it into shape for a first release and I also had to make some conceptual changes once it was mature enough to compile its own manual.

What surprised me was how well that worked and how handy it is to have tested documentation examples. I frequently had to go back and make changes to a code snippet and it was a real timesaver to edit it in one place only, the actual runnable unit test. I also tried to implement and use features such as inline images, tables, automatic chapter/section numbering, automatic generation of a Table Of Contents etc. No problems so far and except for code example callouts (the numbered bullets inside code snippets you often see in books), I've all the features my older Docbook XML based toolchain provided.

So if you have to document some testable Java software, try the attached alpha release[1]. The code is still quite raw but I'm happy with the overall design of pipelines, processors, readers, etc. There is almost no documentation inside the core code though, I'll add that next. If you'd like to write an improved TOC generator or anything else that fits, you are more than welcome.

I'm on my way home from the Seam community meeting in Antwerp this Friday, where I managed to talk to two or three people about the Javadoc-based documentation toolset I've been working on, but there was no opportuntity to talk about it in more detail or to look at some actual examples.

The preliminary name is AuthorDoclet and I'm not attached to it - it's the first thing I entered into the project name box in IntelliJ. So I've been using AuthorDoclet to write the AuthorDoclet manual, which was a bit recursive and confusing. Especially because I'm still redesigning certain aspects of the software while I'm writing the manual.

And this is actually the most important idea behind AuthorDoclet: Stop writing anemic and synthetic unit test code. Write unit tests that people want to read, to learn your software. Write lots of quality Javadoc for your unit test classes and methods. Then press a button in AuthorDoclet and you'll get documentation automatically generated, with validated and tested examples.

This kind of approach has already been evaluated and implemented by other people, for example, there is JCite. However, the tools I've found all assume that you write documentation text in some text editor and include the code examples by putting placeholders in the text. These placeholders are then replaced with code from your Java classes by the documentation processing tool.

Validating the source code used in examples is not the only problem you'll have to deal with when you work on software documentation. The other major issue is maintaining the code while you go through many iterations of the same text, and of course many versions of the example code. If you manually copy and paste code lines from your working application into the document you are writing, you'll at some point no longer know which code needs to be updated and where, after some refactoring.

The solution offered by JCite is simple: The second time it is processing your documentation, it is going to find all the citations (included code snippets) that changed and it will show you a list of items to approve or decline.

With AuthorDoclet, you should not even have this problem because instead of referencing the code snippets from the text for inclusion, you write the text into the same file as the code, as Javadoc comments. So when the code changes, you immediately see the text that describes that code example. When you change the name of a class, method, or field, any references from your Javadoc comments (in all source files!) will be updated automatically as well. (I'm assuming that your IDE supports Javadoc refactoring.)

You still need an external master template file in AuthorDoclet which describes your documentation structure. The following example will make this easier to understand. Create an XHTML file as your master template:

<html xmlns="http://www.w3.org/1999/xhtml">
<head>
    <title>Hello World Source</title>
</head>
<body>
    <div>
        <a class="citation"
           href="example/helloworld/HelloWorld.java"/>
    </div>
</body>
</html>

When processed by AuthorDoclet, this XHTML file will be transformed into the result XHTML file, and all tags that are known to the AuthorDoclet processing pipeline (that's an implementation detail) are going to be handled. The citation anchor is going to trigger the inclusion of the source of HelloWorld.java as source within that <div>. You can organize your <div>'s into chapters and (sub-)sections.

Now, this was not an example of Javadoc inclusion, just the simpler case of Java source inclusion. This is Javadoc citation:

<html xmlns="http://www.w3.org/1999/xhtml">
<head>
    <title>Hello World Documentation</title>
</head>
<body>
    <div>
        <a class="citation"
           href="javadoc://example.helloworld.HelloWorldTest#testHello()"/>
    </div>
</body>
</html>

You'll probably recognize the syntax of this javadoc:// reference: It's the same you use in Javadoc with @link and @see tags. Your IDE can detect these strings and refactor the href value when the class or method name changes.

The Javadoc, which is also XHTML, is going to be included between the <div> element. If there are any <a class="citation"/> within that Javadoc - again, we are talking about the Javadoc of the testHello() method - they are going to be processed as well:

/**
* Testing Hello World
* <p>
* Let's assume you want to say "Hello World". The following <em>example</em> shows you how
* to do that:
* </p>
* <a class="citation" href="javacode://example.helloworld.HelloWorld#testHello()"/>
* <p>
* This was easy, right?
* </p>
*/
@Test
public void testHello() {
    System.out.println("Hello World!");
}

Note the scheme of that URI: The javacode:// prefix is enabling the dot-notation for package, class, and method names (Javadoc @see syntax). Without this scheme, you'd reference the .java file directly as shown earlier.

Processing this Javadoc comment is a recursive operation: If it would contain an anchor that cites another Javadoc comment - be it package, class, or method level - that Javadoc comment would also be transformed and so on.

AuthorDoclet currently also supports various syntaxes for inclusion/exclusion of source lines, and a few special options I'm going to show some other time.

The output document is XHTML markup, and styling, or printing this document is outside of the scope of AuthorDoclet. You can get an easy preview if you write a CSS file and open it in the browser - remember that you control the XHTML element identifiers and CSS classes directly:

<html xmlns="http://www.w3.org/1999/xhtml">
<head>
    <title>Hello World Documentation</title>
</head>
<body>
<div>
    <div class="citation javadoc" id="javadoc.example.helloworld.HelloWorldTest-testHello__">
        <div class="title">Testing Hello World</div>
        <div class="content">
            <p>
                Let's assume you want to say "Hello World". The following <em>example</em> shows you how
                to do that:
            </p>

            <div class="citation javacode" id="javacode.example.helloworld.HelloWorldTest-testHello__">
                <div class="content"><![CDATA[
                @Test
                public void testHello() {
                    System.out.println("Hello World!");
                }
                ]]></div>
            </div>
            <p>
                This was easy, right?
            </p>
        </div>
    </div>
</div>
</body>
</html>

AuthorDoclet also supports linking, e.g. you can write {@link example.helloworld.HelloWorld#testHello()} in any of your Javadoc comments, and a cross-reference (a regular link...) to the citation will be generated in the output.

Writing an XSL:FO utility that converts XHTML to nice-looking PDFs is going to be easy. I guess iText would work as well, although I've not spend any time so far on the output transformation for consumption.

A much-requested feature on this website has been automatic syntax highlighting of code snippets. It is now available and I thought I'd document it here instead of sending everyone an e-mail. The reason why it took so long to implement is that I didn't know how to best integrate it with the Seam wiki text syntax. As you know, we wrap a code block in backticks - that doesn't leave any room for syntax highlighting options. Other wiki text parsers use something like <code syntax="java">...</code> but I wanted to keep the superfast backtick syntax and have it highlighted.

So the way it works now is with an optional line of parameters that follows the opening backtick:

`[brush: java; gutter: true;] 
String text = "Hello World"; // Some comment...
System.out.println(text);
`

The options are enclosed in square brackets and the closing square bracket needs to be followed by a newline. See this page for a list of options. I've installed all brushes, though most of you will probably use 'java', 'xml', and 'sql'.

The example will then render like this:

String text = "Hello World"; // Some comment...
System.out.println(text);

If you do not specify a brush or if you do not include an option line, the code block will be rendered as a grey box, as before. Oh, and the preview when you edit a document will always show the non-highlighted version, it's only applied when you save or update the document.

(No, it will not work on the seamframework.org forums. This has to be tested here first.)

Following up on my last blog entry about the next edition of the Hibernate bible, in the comments, Will Iverson (sorry Will, I hope that is really you, first Google hit) said that he would write ALL the code examples as JUnit test cases. Well, AFAIR that is what Will was trying with his Hibernate book a few years ago.

For my own writing, from the first day, I wasn't sure how to treat code examples. It doesn't really matter if you are writing a tutorial or a reference book with 1000 pages, the question simply is: Did you verify that all code examples really work?

On the other side of the equation is your publisher and their procedures and formats. The publisher I worked with, for example, required that authors submit their text in some MSFT Word template. I've heard from other authors that some publishers are happy with a Docbook XML or SGML file. Well, the only advice I can give you is that you best ignore what the publisher wants and you use what works best for you. (Seriously, you are doing all the work, all they do is import it into Framemaker and pay a typesetter by the hour. If they can't import what you produce, find another publisher.)

So what you have to do is find a toolset that delivers what the publisher wants, but also allows you to verify code examples automatically.

For the first two Hibernate books I had my own toolset based on Docbook XML, with XML, PDF, and HTML output. This toolchain has been re-used by a few open source projects for documentation. I wrote all of the text in XML in IntelliJ IDEA. Unfortunately, all the code examples were copy/pasted lines from real working code. So when the code had to be changed because it had a bug, the book text wasn't updated automatically.

For the next edition of Java Persistence with Hibernate I do not want to bother with this and I want most of the code examples to be verified automatically, I want to reference the executable source that has been tested from within the text, without duplicating it.

I've been prototyping my new toolchain for a few weeks now and it's almost ready for a wider audience. It's based on Javadoc and XHTML and I'm going to blog about it soon.

Well, all I really wanted to show you today is a single class that helped me with unit testing and running my prototype. If you have ever written a Javadoc doclet you'll probably understand why this is useful:

import com.sun.javadoc.RootDoc;
import com.sun.javadoc.ClassDoc;
import com.sun.tools.javac.util.Context;
import com.sun.tools.javac.util.ListBuffer;
import com.sun.tools.javac.util.Options;
import com.sun.tools.javadoc.JavadocTool;
import com.sun.tools.javadoc.ModifierFilter;
import com.sun.tools.javadoc.PublicMessager;

import java.io.File;
import java.io.IOException;
import java.io.PrintWriter;
import java.io.Writer;
import java.util.Arrays;
import java.util.logging.Level;
import java.util.logging.Logger;

public class EasyDoclet {

    final private Logger log = Logger.getLogger(EasyDoclet.class.getName());

    final private File sourceDirectory;
    final private String[] packageNames;
    final private File[] fileNames;
    final private RootDoc rootDoc;

    public EasyDoclet(File sourceDirectory, String... packageNames) {
        this(sourceDirectory, packageNames, new File[0]);
    }

    public EasyDoclet(File sourceDirectory, File... fileNames) {
        this(sourceDirectory, new String[0], fileNames);
    }

    protected EasyDoclet(File sourceDirectory, String[] packageNames, File[] fileNames) {
        this.sourceDirectory = sourceDirectory;
        this.packageNames = packageNames;
        this.fileNames = fileNames;

        Context context = new Context();
        Options compOpts = Options.instance(context);

        if (getSourceDirectory().exists()) {
            log.fine("Using source path: " + getSourceDirectory().getAbsolutePath());
            compOpts.put("-sourcepath", getSourceDirectory().getAbsolutePath());
        } else {
            log.info("Ignoring non-existant source path, check your source directory argument");
        }

        ListBuffer<String> javaNames = new ListBuffer<String>();
        for (File fileName : fileNames) {
            log.fine("Adding file to documentation path: " + fileName.getAbsolutePath());
            javaNames.append(fileName.getPath());
        }

        ListBuffer<String> subPackages = new ListBuffer<String>();
        for (String packageName : packageNames) {
            log.fine("Adding sub-packages to documentation path: " + packageName);
            subPackages.append(packageName);
        }

        new PublicMessager(
                context,
                getApplicationName(),
                new PrintWriter(new LogWriter(Level.SEVERE), true),
                new PrintWriter(new LogWriter(Level.WARNING), true),
                new PrintWriter(new LogWriter(Level.FINE), true)
        );

        JavadocTool javadocTool = JavadocTool.make0(context);

        try {
            rootDoc = javadocTool.getRootDocImpl(
                    "",
                    null,
                    new ModifierFilter(ModifierFilter.ALL_ACCESS),
                    javaNames.toList(),
                    new ListBuffer<String[]>().toList(),
                    false,
                    subPackages.toList(),
                    new ListBuffer<String>().toList(),
                    false,
                    false,
                    false);
        } catch (Exception ex) {
            throw new RuntimeException(ex);
        }

        if (log.isLoggable(Level.FINEST)) {
            for (ClassDoc classDoc : getRootDoc().classes()) {
                log.finest("Parsed Javadoc class source: " + classDoc.position() + " with inline tags: " + classDoc.inlineTags().length );
            }
        }
    }

    public File getSourceDirectory() {
        return sourceDirectory;
    }

    public String[] getPackageNames() {
        return packageNames;
    }

    public File[] getFileNames() {
        return fileNames;
    }

    public RootDoc getRootDoc() {
        return rootDoc;
    }

    protected class LogWriter extends Writer {

        Level level;

        public LogWriter(Level level) {
            this.level = level;
        }

        public void write(char[] chars, int offset, int length) throws IOException {
            String s = new String(Arrays.copyOf(chars, length));
            if (!s.equals("\n"))
                log.log(level, s);
        }

        public void flush() throws IOException {}
        public void close() throws IOException {}
    }

    protected String getApplicationName() {
        return getClass().getSimpleName() + " Application";
    }

}

If you want to test your doclet or run it programmatically you have to use the javadoc command line tool or the evil Main class provided with tools.jar - it's evil because it calls System.exit() when it is done, not usable in unit tests. So what I did here is dig through the JDK source code to figure out how to start a Doclet programmatically without all the baggage. (Again, you most likely won't understand what this is about until you try to write your own Doclet. The API is very old and very bad.)

Oh, and you also need this:

package com.sun.tools.javadoc;

import com.sun.tools.javac.util.Context;

import java.io.PrintWriter;

/**
 * Protected constructors prevent the world from exploding!
 */
public class PublicMessager extends Messager {

    public PublicMessager(Context context, String s) {
        super(context, s);
    }

    public PublicMessager(Context context, String s, PrintWriter printWriter, PrintWriter printWriter1, PrintWriter printWriter2) {
        super(context, s, printWriter, printWriter1, printWriter2);
    }
}

This is how you use it in your unit test (or whatever code):

EasyDoclet doclet = new EasyDoclet(new File("/my/source"), "some.package", "another.package");
RootDoc doc = doclet.getRootDoc();
...

I'm going to write more about my prototype toolset next week and I hope that it's going to be useful not only for myself and the next book but also for other projects, like the last toolset.

I've picked up Wicket in Action last week and I've been reading without interrupting myself so far. So now I'm reading chapter 6 and I haven't written a single line of Wicket code. It's not the first time this happened, most of my books I've read once and never tried any of the code samples.

Last week Gavin called me and we talked about the next edition of Hibernate in Action, which actually would be the second edition of Java Persistence with Hibernate. Now that JPA2 is almost done, and the first beta release with JPA2 features of Hibernate is out, updating the text is inevitable in the near future.

What I need to decide soon is if this update is going to emphasize the tutorial aspect of the book, or if I'm going to add more reference material. I don't think that decision has much to do with the length of the book (JPwH is >900 pages). It's actually all about the code examples. Of course you can not write a 1000 page tutorial, when you pass the 200 page marker, you will have to switch from tutorial mode into reference mode.

Well, because that doesn't happen automatically, you constantly ask yourself the same question: Do readers expect code that works out-of-the-book? Are they going to write that code or copy/paste it, and then expect that it will run? Will it run within the project/product setup I've explained step-by-step up to this point?

For the first two Hibernate books I always considered the answer to that question to be: Yes, maybe the readers want to try most code examples immediately and they probably will have the book open on their desk while reading, next to the keyboard, and they will try the product you are describing in Action. That was actually what the publisher expected from an in Action series book and we had endless and exhausting discussions about it. In the end, it was a lot of work and I'm sure it's not quite perfect. At some point a tutorial approach just doesn't make sense anymore and you have to break the flow and continue with point-by-point reference material. Some readers will not be able to make that jump. The reviews of the books show that, you have a few people who haven't been able to follow the text and examples and got lost at some point. They probably expected the tutorial to continue for another 800 pages.

And here I am, asking myself if I would ever do this again and why I had so much trouble doing it before. I just realized that when I read a book, I don't try the code. I'm not a newbie and I have some Java and JEE specs/framework experience, and I think it's a waste of my time to try the Hello World example in a framework book I'm reading. I'll continue reading until I hit that barrier when it's obvious to me that I need try the code I'm reading. I'll actually not continue reading a book when all the practical details are getting in my way and I've to skip pages because they are full of trivial copy this JAR here, then edit the properties file there explanations. So I'm obviously not the target audience of my own books because they start with: This is how you create your working directory, and here is how you do that on your Windows computing machine. :)

So why can't you have both in one book? I've been paying extra attention to how other writers resolve that issue. In Wicket in Action, for example, the writers obviously do not expect the reader to stop and try the examples immediately. They do not even include the product configuration and initial setup steps in the main text and instead refer to the appendix. I'm somewhat surprised they got this past the Manning in Action guidelines, btw. ;)

I'd considered this for JPwH, moving all of the setup stuff into an appendix. Don't waste 50 pages on basic setup instructions (especially JEE vs. !JEE container) but cater to those readers who have some experience and expect to pick up new stuff quickly in a day or two, without the interruption of real world problems. As the title and subtitle are probably going to be Java Persistence with Hibernate, Second Edition, I'm not really worried about what the publisher has to say.

Still, I'm afraid we're going to have many angry newbies who expect all the setup/configuration steps in chapter 1 or 2, and if one little detail is missing, they are not going to continue reading. On the other hand, what's so bad about If you don't know how to create a directory and copy a JAR file, you need to take a break and read this appendix?

So should the next edition be more like Teach yourself Hibernate in 24 hours although your shoes have 'L' and 'R' on them or should it be The Hibernate Bible, Next Edition?

P.S. Whatever happens, the next edition of JPwH will not be 900 pages. As far as I can see, the .hbm.xml and org.hibernate.Session examples will be be removed whenever they duplicate JPA functionality, so without any other changes, that's going to be 150 pages gone already.