Random thoughts about IT

Posts Tagged ‘python

Scalable real time state update with Storm groupBy / persistentAggregate / IBackingMap

with 24 comments

In this post, I illustrate how to maintain in DB the current state of a real time event-driven process in a scalable and lock free manner thanks to the Storm framework.

Storm is an event based data processing engine. Its model relies on basic primitives like event transformation, filtering, aggregation… that we assemble into topologies. The execution of a topology is typically distributed over several nodes and a storm cluster can also execute several instances of a given topology in parallel. At design time, it’s thus important to have in mind which Storm primitives execute with partition scope, i.e. at the level of one cluster node, and which ones are cluster-wide Read the rest of this entry »


Written by Svend

July 30, 2013 at 1:01 am

Posted in Uncategorized

Tagged with , , , ,