The Stashmetrics data pipeline was driven by a custom task queue built on MongoDB, supporting DAG-style task dependencies and priority-based dispatch.
Design
The core processing engine was Apache Storm, which handled event streaming across a topology of bolts. This allowed fan-out (splitting work across parallel sub-tasks) and fan-in (aggregating results before triggering downstream steps), making it natural to model multi-stage workflows - for example, fetching multiple user profiles in parallel and triggering a report update once they had all completed.
The MongoDB task queue sat alongside the Storm topology to handle two cases that didn’t fit cleanly into a streaming graph: rate-limited tasks that needed to be held back and dispatched at a controlled pace, and high-priority tasks (such as real-time user-triggered searches) that needed to jump ahead of pending background indexing work. Tasks could declare dependencies on other tasks and remained queued until all dependencies had completed, allowing the pipeline to chain async operations and API calls across Storm and queue boundaries.
More recent work
I’m still tinkering with some of these ideas in TQM, a serverless reimplementation using DynamoDB and Lambda. TQM preserves the DAG dependency model and priority ordering while adding rate limiting, backpressure, and partition sharding - capabilities that were not needed at Stashmetrics’ scale but are the primary motivation for the newer design.