See subject. ;-)
In Relation To Aleš Justin
In Relation To Aleš Justin
Yes, we finally moved the CapeDwarf runtime over to new WildFly codebase! It took time to get all the components aligned, all our provided patches merged and all issues resolved. But it was worth it!
Apart from new runtime with WildFly, we also switched to new Infinispan version 6.x and new UnderTow web server. With new Infinispan version 6.x you get much faster file store impl, while the move to UnderTow allowed us to implement Channel API on top of WebSockets.
The full release notes can be found here:
Download the full .zip distribution:
To name a few new features or resolved issues - we finally have initial OAuth impl, we’re now using original Endpoints resources from GAE for our Endpoints support, Marko found and fixed a nasty namespace bug in our Datastore, and we fixed a bunch of small issues exposed by the GAE TCK.
The old codebase based on JBossAS7 can be found under 1.0 branch in our GitHub repos.
Give this new release a spin. Feedback is welcome as always!
As you can see from the title, and from a bit late release, we’ve again been very busy.
Let’s just go over all new features.
There is a bit of limitation on our end here, as there is no (easy / nice) dynamic way (to my knowledge) to scale up / down virtual servers (at runtime during deployment phase) in JBossWeb that comes with JBossAS7 (which is what we use). In order to use Modules with CapeDwarf, you must use “standalone-capedwarf-modules.xml” configuration when you startup CapeDwarf AS instance.
You must also configure the number of virtual servers to match your application’s modularization. See web subsystem configuration in “standalone-capedwarf-modules.xml” for more details.
(or simply ping me for any Modules questions - via forums, email, Twitter or Freenode IRC #capedwarf)
We’ve managed to map GAE’s Endpoints annotations to JAXRS/RestEasy annotations, and we also support custom serialization.
In our case / implementation we only need .api files as markers, and not to read any information from it -- you can have a single .api dummy file if you like.
SQL was of course supported out-of-the-box for us. We we just had to hack GAE’s Driver class to connect against selected DataSource:
FileService is being deprecated in favor of Google Cloud Storage (GCS) API. At the moment dev / local implementation of the GCS client library still goes against FileService implementation, so supporting this was easy, as we already fully implemented FileService API.
We re-implemented Channel API.
Also thanks to Karel for stepping up and fixing Arquillian Drone so we can finally automagically test this.
Don’t miss my presentation at JavaOne -- Tuesday 24th September at 11:30am, Parc55. ;-)
Or my previously mentioned DC JBUG talk: http://www.meetup.com/DC-JBug/events/128068042/
Markus Eisele posted a nice overview of CapeDwarf:
Anyway, enjoy the new release! Feedback welcome as always.
After successful Google I/O and Red Hat Summit presentations, and some vacation, it's finally time to do a new release.
Regarding the release issues, they've been mostly driven by TCK failures, hence I'll let the release notes speak for themselves.
Note: for the Windows users, an issue I just discovered after releasing the .zip, and it's unfortunately not as trivial as it looks like, hence I didn't want to stale the release even more. So the fix for this is coming in the next release.
The simple workaround is to run the full AS start command:
standalone.bat -c standalone-capedwarf.xml.
(and a bit more work if you wanna use GAE API jar as a module)
Apart from classic coding, it's been busy three months with a lot of news and new things around CapeDwarf project.
For those who missed my tweets about GAE TCK, here is the link again
It looks like we're getting more and more traction on the blog / article scene
- Ludo's blog, Google Cloud Platform
- The H Open
- Yefim Natis, Gartner
- Peter Magnusson, Google
Since JavaOne notifications are out, I can happily say that my Implementing your own Google App Engine talk got accepted. And I'll prolong my US stay with my visit to DC JBUG. Welcome to drop by!
Continuing with our monthly releases, we’re happy to announce new 1.0.0.Beta4 release.
- all logging data was moved into separate cache,
- enabled metadata handling,
- default cache is now “SYNC” by default (which made MapReduce tests very happy :-)),
- we put back legacy factory transformers - which means older GAE API versions now again work to some extent,
- Velocity is jarjar-ed - so it doesn’t conflict with your version anymore,
- we support static files handling - you don’t hit your filters and listeners anymore,
- and a bunch of other small tweaks.
CapeDwarf team will be at this year’s Google I/O’ Sandbox! Make sure you drop by for a chat, demo and a nice T-shirt.
Also be careful you don’t miss the exciting surprise announcement! ;-)
Download it here:
As you can see from the JIRA release notes, it’s been super busy 2 months.
Another new sub-project: CapeDwarf Shared.
It’s where we keep the shared code between our GAE API implementation (Blue) and JBossAS integration.
There has been a lot of detail polishing on Blobstore, Logging and TaskQueue. And we also added BlackList / WhiteList support, getting even closer to GAE behavior this way.
And yes, we finally simplified how you start the whole thing - thanks Lazo. ;-)
- ./capedwarf.sh or capedwarf.bat (in JBOSS_HOME/bin/) will do
As always, feedback welcome!
With first CapeDwarf release out of the way, it’s now time to address initial user issues. Since setting up CapeDwarf environment has become so simple -- just unzip the distribution, and CapeDwarf already running on OpenShift, we got a few useful feedbacks; keep ‘em coming!
Quick look at profiling showed we’re too eager on our logging. GAE has a LogService, hence we need to tap into JBoss Application Server logging in order to properly catch all application’ logs. But if you do this too eagerly, you might end-up catching all logs -- which is exactly what happened here. Well, this is now all fixed, resulting in huge performance boost on GAE logging now.
As expected, another very useful source of information are actual user applications. Either users themselves tried to deploy the app and failed or we actually got our hands on their .war binaries. This helped flush out some OAuth, JDO, threading, backends and web components initialization bugs.
Something I forgot to mention in previous CapeDwarf blog, and it’s quite important, is that at the moment, we only support latest version of GAE API (and/or any that is 100% compatible with it; at the time of writing this blog, this is GAE 1.7.4).
So, if you haven’t yet updated to the latest GAE API version, you might run into some issues. In order to flush them out easily, I created a new CapeDwarf sub-project over the holidays.
As you can see, this allows for Blue tests to run against selected GAE API version.
I also added initial support for API versioning to CapeDwarf AS integration, but the actual support is not there (yet) -- due to how we intend to support this. Each non-latest GAE API version requires a new CapeDwarf Blue branch, from which you get the compatible binaries to distribute them with CapeDwarf distribution. Then you simply place these compatible binaries into JBoss Application Server’ modules/ dir - under org/jboss/capedwarf/<version> directory. But this is more of a task for contributors, as we encourage people to update their GAE API jars.
Also a slight update on MapReduce tests in cluster. They work, but you have to make CapeDwarf’ default cache execute operations SYNC, instead of ASYNC. We have a workaround for this in place now. ;-)
To cut the long story short, here is the new 1.0.0.Beta2 release:
Feedback welcome as always!!
After a year in the making, I’m finally able to announce first CapeDwarf release!
As our jboss.org page says, it’s a Google AppEngine (GAE) API implemented with JBoss and other open source libraries. Making it possible to run unmodified GAE applications directly in JBoss application server.
The project is split into few sub-projects:
- CapeDwarf Blue - heart of the project aka actual GAE API implementation (https://github.com/capedwarf/capedwarf-blue)
- CapeDwarf JBoss-AS - JBossAS7 extension / subsystem (https://github.com/capedwarf/capedwarf-jboss-as)
- CapeDwarf Testsuite - testing environment (https://github.com/capedwarf/capedwarf-testsuite)
All of the sub-projects are “Mavenized”, and at the moment you also need latest JBossAS upstream binaries in your local Maven repository.
- JBossAS - https://github.com/jbossas/jboss-as/
Download CapeDwarf zip distribution and unpack it to your prefered location - JBOSS_HOME. It contains current JBossAS7 upstream master snapshot binaries, plus CapeDwarf JBossAS7 subsystem and a few extra modules (aka libs or jars).
Now simply go to JBOSS_HOME/bin/ directory and run the whole thing.
- on OSX ./standalone.sh -c standalone-capedwarf.xml -b your_ip
- on Linux ./standalone.sh -c standalone-capedwarf.xml
- on Windows standalone.bat -c standalone-capedwarf.xml
Now just drop the app to JBOSS_HOME/standalone/deployment/ dir and you should already see the app being deployed. Or you could use new JBossAS7 Admin Console or CLI to deploy the app - see JBossAS7 docs on how to do this.
12:49:21,651 INFO [org.jboss.as.capedwarf.deployment.CapedwarfInitializationProcessor] (MSC service thread 1-12) Found GAE / CapeDwarf deployment: deployment "capedwarf-tests.war”
As being modular is the way to go these days, we also provide a way to run GAE apps in fully modular fashion. What does this mean? It means that instead of packing all required GAE jars / libs, you can simply not include them in your app - making the app distribution smaller this way. It is CapeDwarf that will recognize the app being a GAE app - via appengine-web.xml descriptor file, and add appropriate modules / jars to you app’ classpath. But this comes with a small gotcha.
As we need to do some bytecode enhancement to GAE API jar in order to override some services instantiation behavior, while on the other hand you’re not allowed to ship with modified binaries as per license, you will need to run a script to do this bytecode enhancement before running the application server for the first time. The script is located in usual JBossAS bin/, capedwarf-bytecode.sh or capedwarf-bytecode.bin.
Apps that bundle all GAE libs - which is the usual GAE way, do not need to run this script, as the bytecode enhancement will take place during runtime.
Without going too much into details, let me just mention few main components and how they are roughly implemented.
For most of the stuff, we take JBoss’ Infinispan and Hibernate Search projects to their limits. We use them for datastore, memcache, search, prospective search, filestore and blobstore. JMS with HornetQ is used to implement taskqueue.
Where Infinispan is not just plain cache implementation, but a whole data grid with good distributed task framework. Combine that with Hibernate Search and its nice Lucene extensions, you get a powerful clusterable environment to tackle complex problems that GAE API tries to hide from the user, but of course solve it underneath.
Yes, we also support admin console for each app. While we do lack features, we did try to add some useful functionality to this initial version.
JPA / JDO support is exactly the same as in any GAE app. We went an extra mile to support this, providing patches to DataNucleus project, and taking special care of JPA / JDO deployments in our JBossAS7 CapeDwarf subsystem.
At the moment we have a working test for standalone runtime, where we still have some issues in clustered environment. Hint: the test exists, but fails, and CapeDwarf is open source, wink. ;-)
We most certainly do. ;-)
We take great pride in trying to test this as much as possible. And there is a bunch of interesting things we do to make this happen, trying to squeeze Arquillian as much as possible.
You first need a custom JBossAS + CapeDwarf instance. You can either enhance / modify an existing JBossAS instance with CapeDwarf-JBoss-AS sub-project or you can use the JBossAS instance from CapeDwarf-Testsuite sub-project. Or of course use the custom distribution you just downloaded. :-)
As expected, each sub-module (a sub-module per GAE API split) in CapeDwarf (Blue) has its own set of tests.
If you already have a running CapeDwarf-JBoss-AS instance, you can run these tests against it with “mvn clean install -Premote”. This uses JBossAS7 Arquillian Remote container to deploy the tests.
CapeDwarf-Testsuite sub-project builds its own JBossAS instance, and then uses JBossAS7 Managed Arquillian container to run the CapeDwarf Blue’ tests.
We have all of this setup as TeamCity builds (thanks JetBrains for the license!), so drop me an email if you need to test some CapeDwarf pull-request you just hacked. ;-)
Whole CapeDwarf implementation tries to be scalable, as is of course GAE, avoiding a single point of contention. There are of course pieces of code where we know could use better approach. And this is definitely something we’ll do before Final release. Which is also where community help is highly appreciated; to either point out the issues or perhaps even contribute some better implementations to some bottleneck problems.
At the moment we have separate module in Blue which contains cluster tests. There is a README on how to exactly run those tests.
-Pcluster is the Maven profile to enable those tests.
What we did here is develop an Arquillian container just for GAE.
So what this means is that we can run the same set of tests against real GAE, and all of the (non-impl detailed) tests should pass. btw: we only have a handful of impl details tests, all others are impl agnostic
So “mvn clean install -Dappengine.sdk.root=Your GAE SKD location” runs Blue’ tests against your local GAE.
We also support embedded and remote GAE Arquillian container, but these are to be used with care. :-) Embedded - aka inJVM - is quite hacky, where remote deploys against AppSpot hence it’s a bit slow and less stable to use.
There aren’t many existing public GAE tests out there. The only ones we could find are the ones in GAE DataNucleus (DN) plugin project.
And there is a nice hack on how to make these tests run against CapeDwarf. ;-)
These GAE DN tests are pure Maven driven JUnit tests, where we need an Arquillian tests to run them against JBossAS instance. What to do? As we’re already bytecode hacking things, it’s obvious what can be done.
We add a custom Maven plugin which knows how to transform classes just after they are compiled:
And it turns a plain JUnit test into an Arquillian GAE DN test:
This now makes GAE DN tests run against CapeDwarf instead of default GAE API impl.
btw: we pass all of the non-impl detailed GAE DN tests ;-)
A custom CapeDwarf OpenShift cartridge is in the making as we speak, so you should be able to develop GAE apps against OpenShift and CapeDwarf in no time. ;-)
CapeDwarf User forum:
- #capedwarf on Freenode
Or shoot me an email directly.
Enjoy using CapeDwarf as we enjoy developing it, and do let us know any issues or feedback you might have.
I’m just on my way back from JUDCon China in Beijing. I must say it was an interesting week. I mean what else can you say for a country when you don’t understand a word they are saying, or any written word for that matter. :-) But thankfully to our Chinese co-workers there was no problem for us to go around. The food and the view is spectacular - the new olympic stadium and aquatic center are just amazing.
The conference was well organised. It could probably have a bit more attendees, but it’s a start, right. I gave a talk about our current work on CapeDwarf. Was hoping to do a release just before, but the great firewall prevented me in that intention. So stay tuned these days. ;-) The second talk was about Weld-OSGi, the work done by our friends at SERLI. I borrowed Mat(t)hieu slides, as they are just perfect to show what the new stuff can do.
Last day we had some time to do some sightseeing - the Forbidden city and the Great wall. It was freezing cold to go around, but the view made it all worth. Waiting for Ray to post some photos. :-)
With plenty of work being done on CapeDwarf, I now finally found some time to blog about previous week’ visits and a cloud oriented conference.
If you follow Mark’ blog or tweets, you could catch that he spent a day here with us - Slovenia’s JBoss team - in a beautiful town of Bled. But he wasn’t the only Red Hat / JBoss visit we got, Tristan Tarrant, from Infinispan team, joined us as well.
The main reason for us being in Bled, apart from great view and food, was this year’s CLASS conference. Mark had a short keynote about Red Hat’ vision wrt middleware and cloud, Tristan shared his view on how Infinispan data grid fits into the cloud, and I presented our latest work on CapeDwarf.
CapeDwarf has grown a lot since I last gave a presentation on it. Lately we specially focused a lot on testing, with plenty of interesting ideas and approaches on how to squeeze the maximum out of existing tests and examples. Which is what the presentation was mostly about.
But more about this in my next blog posts, stay tuned! ;-)