Random thoughts about IT

Posts Tagged ‘storm

Error handling in Storm Trident topologies

with 3 comments

This post summarizes my current approach to error handling when designing Storm Trident topologies. I focus here on code design, not on deployment good practices like supervision nor redundancy.

Because of the real-time stream nature of Storm, when facing most kinds of error we’ll ultimately have to move on to the next piece of data. Error handling in that context boils down to reporting this error (or not) and retrying to process the failed input data later (or not). Read the rest of this entry »


Written by Svend

February 5, 2014 at 6:11 pm

How to compile Storm 0.8.2 on Mac OS X

with 3 comments

Here are a set of instructions to build and package from source either the storm-0.8.2.jar or the complete (with all dependencies). I assume packaging later versions will be similar, just be careful about dependencies versions.
Read the rest of this entry »

Written by Svend

September 4, 2013 at 4:43 pm

Posted in Uncategorized

Tagged with , ,

Scalable real time state update with Storm groupBy / persistentAggregate / IBackingMap

with 24 comments

In this post, I illustrate how to maintain in DB the current state of a real time event-driven process in a scalable and lock free manner thanks to the Storm framework.

Storm is an event based data processing engine. Its model relies on basic primitives like event transformation, filtering, aggregation… that we assemble into topologies. The execution of a topology is typically distributed over several nodes and a storm cluster can also execute several instances of a given topology in parallel. At design time, it’s thus important to have in mind which Storm primitives execute with partition scope, i.e. at the level of one cluster node, and which ones are cluster-wide Read the rest of this entry »

Written by Svend

July 30, 2013 at 1:01 am

Posted in Uncategorized

Tagged with , , , ,