In Relation To David Lloyd
In Relation To David Lloyd
I had the pleasure of gabbing about JBoss Modules a couple times at the Red Hat/JBoss booth at JavaOne this year, drawing frankly more interest than I thought I'd see. As a part of my talk (the video for which you can download here [well, soon anyway]), I chatted a bit about what the future may hold for Java in terms of modular runtime, modular development lifecycle, and distribution management. Some things seem certain - that the current paradigm won't hold, for example - and some are very uncertain - such as what, if anything, will replace Maven as the ubiquitous modular build/distribution system of the future.
There are many things that we, as a community, have learned over time about how projects are built, deployed, maintained, and integrated. However, every popular build and deployment system in existence today is crafted from only a limited subset of this shared knowledge, usually from the point of view of that project's particular czar. The advent of modularity in common usage, and ultimately the JDK, is the final call to arms for the community to step up and share their knowledge.
Because Oracle continues to turn a deaf ear to the community when it comes to Jigsaw, it is left to the community itself to establish and understand the true nature of the problem of program and library distribution for Java. The requirements published thus far by Oracle are presented from the perspective of a company whose expertise lies not in the development of the applications that you write, but in the sales and marketing of the platform that they develop. This is why there is a community process, which is unfortunately thus far being largely bypassed in the development of Jigsaw and its supporting infrastructure. History has clearly shown that the success of any given Java specification depends largely on community involvement and empowerment. So it seems clear to me that the success of the modularity initiative will also depend on the community.
With that in mind, members of the jigsaw-dev mailing list recently discussed the idea of holding a community meeting for the purpose of brainstorming requirements for modularity in Java. I am hoping that we can discuss topics ranging from run-time modular resolution requirements and technical details to build dependency resolution to library distribution paradigms; if the chat goes well we could even evolve it into a regular or semi-regular event.
The chat is scheduled to be held at 10:00 A.M. US Pacific time on Friday, October 21, 2011 in the ##java-modularity channel at irc.freenode.net (web chat link). If you have any experience you'd like to share, either as a user or a developer of an existing build or modularity system, or you just want to see what happens, please join in the discussion. It's undecided at the moment but the discussion may be moderated, depending on attendance. Note! Make sure you have a registered nickname ahead of time so you can join in without any issues when the time arrives.
I look forward to chatting with you!
Today Oracle's Mark Reinhold published the requirements for comment a public draft of the Java Module system being proposed for inclusion into the Java SE 8 platform. The requirements as given are fairly high-level yet comprehensive, and many of these requirements align well with the goals and specifications of our own JBoss Modules system, which is not only a strong validation of our own design but also what I think is a good sign for the future of the Java platform.
The requirements as posted however contain some things that are specifically noteworthy and some things which in my view should definitely be changed, based on my own experience with implementing the JBoss Modules environment as well as applying that environment to JBoss AS 7 and 6 (which we did on an experimental basis a while back), as well as feedback from my colleagues responsible for JBoss OSGi, which is presently implemented atop JBoss Modules. Being very sizable and very different projects, these efforts have taught us a lot about the practical nature of modularity in Java.
Since this is a big document, I'm going to comment on it a bit at a time and see where we end up.
The document, which may be found here, is broken up into a number of sections, which I will attempt address approximately in order.
The fundamentals paragraph items all indicate a strong sense that the Jigsaw project is on the right general track. However I found a couple sections noteworthy:
Resolution in all phases — Build-time, install-time, and run-time module resolution and linking must all be supported. UNIX has all three models and they are all still in use 40 years later.
coupled with the following paragraph:
Fidelity across all phases — The set of modules seen by a library or application must be computed by the same algorithm at build time, install time, and run time.
and this one:
Versioning — The module system must support common version-string schemes [...]. It must be possible to specify a range of allowable versions when declaring a module dependence.
These seemingly reasonable statements hide a particularly tricky little goblin; namely, build reproducibility. At build-time, an artifact (to use the Maven term) is identified in part by its version. And that identity has a specific meaning. If I check out version 1.2.3 of jboss-cobwobble from source control and build it, I should get an artifact which is equivalent to one which anyone else who checks out and builds the same version of the same project gets.
However, that means that such builds cannot use version ranges. By definition this creates a way for two parties building the same project to get different output, thus blurring the meaning of what a version is. In the first paragraph, Mark emphasises the similarities to the shared library model and recommends that similar practices be followed to that paradigm, however there is a key difference in that when one builds against a shared library the linkage is much less invasive. A change to a compiled library used by a project does not generally affect the compilation results of that project, whereas in Java this is much easier to do, since class files are a much richer format than a shared object symbol table.
Therefore it is very important to allow (better yet, require) builds to use specific dependency versions when building, and allow version ranges only at install and run time, where version ranges are a much better fit.
Another very important consideration is that after an artifact is produced, the versions of its dependencies that it is known to be compatible with evolves over time as new versions of various things are released. Thus dependency version ranges with a closed upper end cannot be a fixed part of the module's internal metadata, or there is a risk over time that the module will have to be repackaged to accomodate new version compatibilities. My personal recommendation in fact is that modules only ever support a lower bound to version ranges for package/run-time constraining.
I admit I do not understand the purpose of this paragraph, which reads as follows:
Target constraints on modules — On a particular target platform it must be possible to declare that only modules with specific properties can be installed, e.g., particular authors, publishers, or licenses. The resolution algorithm must ignore any module that does not satisfy the target platform’s module constraints.
If the purpose of this is security, which I can very much get behind, then I think the only sensible approach is to constrain module loading to signed modules. Otherwise, if the goal is simply to create a mechanism by which administrators can annoy each other, then I guess this fits the bill; the restriction can be bypassed by changing the metadata, which means that it provides neither security nor convenience.
Overall I think this mechanism also risks fragmentation of the module ecosystem that would (hopefully) arise around this enhancement. I think Perl for example benefits greatly from a single repository of modules which is yet open to contributors with relatively little restriction; yet operating system distributors (know any of those?) have established best practices for the distribution of these modules. And of course today's Java developers often use the Maven central repository and expect it to contain
everything in reasonable locations. However by creating these filtering mechanisms, the problem of many publishers is being solved at the wrong place, and it allows for some potentially very surprising behavior (I can see the FAQ now:
I installed a module but Java says it's not found... what gives?).
This section basically says that modules need to support native code somehow, though it seems to imply that the native library has to live inside the module itself, which I think is probably unneccessary (though of course it would have to exist within the module packaging, else there would be no way to actually install it). In JBoss Modules, we simply require that native libraries exist in filesystem-backed resource loaders (as opposed to jar-backed resource loaders which one would normally use for classes). In addition the filesystem resource loader uses the name of the OS and hardware platform to allow for packaging modules which contain native bits for more than one platform (actually not unlike how Perl's DynaLoader does it, if I recall correctly).
These sections I have a serious problem with:
Package subsets — In support of platform modularization, it must be possible for the types defined in a Java package to be provided by more than one module yet still be loaded by the same class loader at run time. This is required so that CDC-sized subsets of large legacy packages such as java.util can be defined.
Shared class loaders — In support of platform modularization, it must be possible to declare that the types defined in a specific set of modules must be loaded by the same class loader.
I am very much against this notion. One module, one class loader, period, end of story. If you are splitting a single class loader among many modules, you haven't created modules, you've created one module with a class path of several JARs, of which some stuff may be missing and others not, and a hell of a lot of useless complexity, all for the privilege of saying
sure, we modularized this. There is no isolation of any sort between them beyond your standard access controls provided by the Java language. Without a strong definition of a module as an encapsulating vehicle for a class loader, the whole idea is weakened - visibility is only sensibly enforced in terms of whole class loaders (yes, while it is probably possible to use a different unit of division like class, it will definitely have a negative performance impact).
Rather than thus weakening the definition of a module, I recommend that modules be allowed to be packaged in portions, which may be separately shipped and installed - perhaps a module/submodule concept could be introduced, where the contract of a submodule is clearly defined as sharing the parent module's class loader. In other words, the label should be clearly marked, so to speak. There is no reasonable expectation that a package may be split between modules, and doing so is definitely a violation of the Principle of Least WTF.
Though there are various paragraphs which allude to it, there is no specific section addressing security, which I think is a somewhat serious oversight. I would personally like to see some enhancements to the standard security provider mechanism to accomodate modules and module signers more conveniently than the mechanisms at our disposal today, and I commented earlier about module signing, to say nothing about modular security provider discovery (okay, this kind of thing is currently listed as a non-requirement to be fair - but it will always be in the back of my mind, and you can bet that it will show up in JBoss AS 7.x at some point if I have anything to say about it).
That's about it for the first chapter. Overall the Fundamentals leave me with a sense that things are somewhat more under control in Jigsaw-land... in particular, it's good news that the brakes are apparently being gently applied to implementation in order to come up with real requirements, which are essential to ensuring that a project is firmly on the correct planetary surface. Just ask any of my colleagues, they'll tell you how I like a good requirements document!
Next time I'll try to cover a little more ground, as the subsequent chapters are mostly rather shorter than Fundamentals is.
I will be giving two talks about JBoss Modules next week at JUDCon 2011 (Boston) - the first on Monday at 9:00 am (ouch, I know) about JBoss Modules in JBoss AS 7, and the second on Monday at 2:30pm about using JBoss Modules itself. Also, I'll be around for most of the week (including during JBoss World/Red Hat Summit), and will be willing to Chat about Stuff when I can.
See you there!
JBoss Marshalling 1.3.0 is about to land! The ninth (and probably final) release candidate for 1.3.0 has been made available for download. New in this version:
- A complete object cloner API which can clone any object which is marshallable by the marshalling API
- Version 3 of the River protocol, featuring even more compact representations of collection classes
- Many more useful stream-oriented base classes which can be used by your application
- The ability to override serializability for any class or object
- New module-system-ready API for constructing MarshallerFactorys and finding implementations
- An OSGi bundle containing the Marshalling API and implementations
- Support for ObjectInput constructors in Externalizable classes
JBoss Marshalling is an object serialization library for Java which is fully compatible with Java serialization, yet outperforms it significantly and offers a vast array of additional customization and performance features. In addition it features the advanced River serialization protocol which is API-compatible with Java serialization yet far more space- and computation-efficient. It can be found in use within the excellent Infinispan project as well as finding heavy use in the upcoming JBoss AS 7 release.
The project can be found at JBoss.org. This release and others are available on the downloads page and in the JBoss Public Maven repository. The JavaDocs are online here. Give it a try and file and bugs you may find in JIRA. I plan to put out the 1.3.0 final release after JBossWorld next week.
It was almost a year ago that Mark Reinhold (one of the top Java engineers over at Sun/Oracle) announced
The classpath is dead! at JavaONE, accompanied by a series of blog posts like this one, proclaiming that the future of Java is a modular one. In the mean time, the JDK 7 timeline has slipped dramatically and we may not see modules in Java proper until 2012 or later. All the while, the folks on JSR 294 have stalled, started, and stalled again, slowly inching their way towards a tightly-Java-integrated modularization standard, possibly requiring significant language, bytecode, and packaging changes to support it.
I've long maintained that it would take far less than this to introduce a solid, usable modularity standard. And as part of our JBoss AS 7 proof-of-concept work, we've proven it with the creation of JBoss Modules.
If you haven't been following the developments around JDK 7, you may not even be aware of what you're missing.
A module is any collection of classes and resources, associated with a single class loader. Modules may express a dependency on other modules. The exported classes and resources from these dependencies are visible to classes in the dependent module. Visibility is the ability of a class from one class loader to
see a class from another class loader. To export a class or resource means to make such a class or resource visible to dependents.
So what a module system provides is a way to package up classes and resources into modules, and establish a graph, at run time, of class loaders such that all the expressed dependencies are satisfied.
The traditional method of running an application in Java uses the well-known
classpath mechanism. This basically creates a single application class loader with all those JARs' resources concatenated together in one big blob.
Now imagine you have a very, very large and complex application with dozens of JARs. Imagine that some of those JARs are not used all the time, or that some of those JARs represent multiple, conflicting versions of the same classes and packages due to conflicting dependencies. This problem, and other issues along these lines, are known popularly as Jar Hell.
Modules can greatly alleviate this problem. When all JARs are bundled as modules, a JAR will never
see a conflicting version of a dependency class, or pick up resources from other JARs inadvertently. Also, a module which is never used will never be loaded which can improve startup time of large applications.
JBoss Modules is a standalone module system for Java 6 and up. It supports the following features (and more):
- An memory-efficient, high-performance, concurrent/thread-safe delegating class loader implementation, capable of loading any class or resource in O(1) time
- An extensible module loader system which allows users to plug in alternative module definition/loading strategies
- An easy-to-use bundled local module loader, which loads modules (packaged as standard JARs or as exploded directories) from the filesystem in a simple, predictable structure (think Maven, but simpler)
- An easy bootstrap process (see below)
- A useful runtime API for loading modules, getting a class loader for modules, creating modules at run-time, extending the JDK service loader mechanism to be module-aware, and more
- Management of platform-specific native code
A modularized program is started like this:
java -jar jboss-modules.jar -mp path/to/modules my.main.module.name
In your module path (
-mp) you specify the root directory (or directories) which will be searched by the default module loader for module definitions. A module is defined using a simple XML descriptor, like this:
<module xmlns="urn:jboss:module:1.0" name="org.jboss.msc"> <main-class name="org.jboss.msc.Version"/> <resources> <resource-root path="jboss-msc-1.0.0.Beta3.jar"/> </resources> <dependencies> <module name="org.jboss.logging"/> <!-- Optional deps --> <module name="javax.inject.api" optional="true"/> <module name="org.jboss.threads" optional="true"/> <module name="org.jboss.vfs" optional="true"/> </dependencies> </module>
There is a full schema bundled in jboss-modules.jar for this descriptor format, so integration with your favorite IDE is easy. There are also many extended capabilities available which allow tight control over what packages are exported or imported, and you can also selectively exclude content from your resource JARs (making it even easier to use prepackaged JARs).
This simple module system has several advantages over JSR 294 as it stands today.
- It's available now. There is no telling when JDK modules will become available - maybe in Java 7 in 2012, maybe in Java 8 or later. This is a project you can use - and contribute to - today.
- No invasive language, bytecode, or compiler changes are needed, and no JCP approval is necessary.
- As things stand today, JBoss Modules should be compatible with JSR 294, and we will strive to maintain compatibility if and when Jigsaw solidifies and becomes part of OpenJDK.
- It is much simpler; one JAR is all that is needed to get your modular application off the ground.
- It is minimal: there is no services layer, or any other higher level functions that OSGi provides. It does exactly one thing, and it does it well.
- At the same time, it is very powerful, and could form the basis of a class loading framework which an OSGi implementation could use (more on this in the future, stay tuned!).
JBoss Modules is available now via Maven as
org.jboss.modules:jboss-modules. There is also a JIRA Instance available for reporting problems. Discussion relating to Modules in AS 7 happens here or on #jboss-msc on FreeNode IRC (join via webchat with your registered nickname). The source code is hosted on GitHub.
So... what are you waiting for?