Task Queue Manager (TQM) is an AWS-native task queue service built on DynamoDB and Lambda, deployable via Terraform. Influenced by SQS, Celery, BullMQ, and other task queuing systems, it’s designed for workloads where SQS falls short: specifically where you need fine-grained control over dispatch rate to downstream services, approximate priority ordering, or DAG-style task dependencies.
SQS’s Lambda integration handles polling for you, but this makes it hard to implement backpressure and circuit breaking without tuning concurrency and batch size - which is highly sensitive to Lambda execution time. TQM keeps you in control of dispatch, with rate limits defined by token buckets which can be used independently, or combined and shared by multiple queues.
Design
The data model builds on the DynamoDB priority queueing pattern described by the AWS Database Blog “Implementing priority queueing with Amazon DynamoDB”, using sparse GSIs and optimistic locking to implement visibility timeouts and approximate priority ordering without a dedicated queue service. Optional partition sharding supports higher-throughput queues. Tasks can be requeued at their original priority and timestamp rather than being penalised by going to the back of the queue - important for long-lived queues where jobs may wait days before dispatch.
Dependency handling follows a DAG model: tasks stay queued until their dependencies complete, with failure propagating up the tree or triggering a configurable failure handler.
Queue management is accessible either via a thin API layer or directly through the underlying tables, depending on how much abstraction you need.
This is an iteration on an earlier task queue implemented on top of MongoDB at Stashmetrics around 2015. This supported the DAG-style dependency triggered workflow and priority-based queueing, but not the serverless architecture possible with DynamoDB and Lambda.