This post summarizes my current approach to error handling when designing Storm Trident topologies. I focus here on code design, not on deployment good practices like supervision nor redundancy.
Because of the real-time stream nature of Storm, when facing most kinds of error we’ll ultimately have to move on to the next piece of data. Error handling in that context boils down to reporting this error (or not) and retrying to process the failed input data later (or not). Read the rest of this entry »
Here are a set of instructions to build and package from source either the storm-0.8.2.jar or the complete storm-0.8.2.zip (with all dependencies). I assume packaging later versions will be similar, just be careful about dependencies versions.
Read the rest of this entry »
In this post, I illustrate how to maintain in DB the current state of a real time event-driven process in a scalable and lock free manner thanks to the Storm framework.
Storm is an event based data processing engine. Its model relies on basic primitives like event transformation, filtering, aggregation… that we assemble into topologies. The execution of a topology is typically distributed over several nodes and a storm cluster can also execute several instances of a given topology in parallel. At design time, it’s thus important to have in mind which Storm primitives execute with partition scope, i.e. at the level of one cluster node, and which ones are cluster-wide Read the rest of this entry »
Play 2 has been announced a few weeks ago.
I am currently working on a project based on version 1 of this framework, as I am envisaging a migration, I decided to write two blog posts about this experience. The text below contains my personal thoughts on Play 1, I’ll try at a later stage to hunt for some time to use this as a comparison point for my feed-back on Play 2 and the migration process (if I confirm it should happen…)
Here are some major characteristics of Play (just check their online docs for more details, that is not the point of this post):
- Web framework for Java and/or Scala
- Not based on servlet, although a project can be converted to a war package
- No server-side session
- Java controller methods are public static void
My app is mostly based on JSF2/jquery/Spring/Jackson + the AWS-specific APIs for storing data into AWS SDB and S3.
The migration to Cloudfoundry is the most straightforward migration experience I’ve seen up to now, it boiled down to: Read the rest of this entry »