Should we replace Kafka with Temporal Standalone Activities? Our ADR
TL;DR
We run Kafka in production for mass operations and we’re weighing whether to replace it with Temporal Standalone Activities. Written as an ADR, this weighs three candidates (Kafka + KIP-932, a plain queue, Temporal) against nine weighted criteria and shows why we lean toward Temporal - and exactly what it costs us. It’s not a “Kafka is bad” manifesto; it’s a record of a decision that won’t be final until a PoC settles the one open risk: persistence load at a million tasks.
Part 1 - The problem (for anyone who makes systems decisions)
“Batch” means something different every day
Every large organization has a class of operations that run in bulk across thousands or millions of objects.
- Granting a promotion to a customer segment.
- Fixing a misconfigured offer.
- Pushing a campaign.
- Sending a wave of email or SMS.
In telco it’s a daily occurrence, and the same pattern shows up in banking, insurance, retail. Here’s the scene that defines the whole architecture. A ticket comes in, flagged urgent: a broken offer applied the wrong discount to a few tens of thousands of customers, and it has to be fixed before the nightly billing run. At the same time the system is chewing through a scheduled campaign against the full base - millions of records. The question that decides everything downstream is simple. Does the urgent fix jump the queue, or does it wait a few hours behind the campaign? If it waits, the customer gets a wrong invoice.
Three requirements never go away:
Auditability. For every single record we need to know whether the action ran, when, with what result, and if it failed, why. Per record, not in aggregate.
Control over state and priority. An operator has to see progress, pause, resume, and let an urgent job cut ahead of a scheduled one.
Safe retries. Failed records have to be re-runnable without double-executing the action. Idempotency, in one word.
Sounds like a batch. That assumption is exactly what we built the first generation on, and it turned out to be a batch in name only.
Two generations behind us
Generation 1 - Spring Batch. Classic, proven, no extra infrastructure. The problem: the same feature processes 10 records one day and a million the next, and Spring Batch doesn’t scale in the cloud without bolting on another broker and a fair amount of configuration gymnastics. Plus a flat job queue, so no real prioritization.
Generation 2 - Kafka. This is what’s in production. A reader pulls records from the database and publishes them to topics; consumers process independently. We got the decoupling of ingest from processing, and we got consumer scaling. But after a few years of running it, the honest read is that a lot of what Kafka gives us we don’t need, and a lot of what we need we’ve had to build ourselves on top of it. The load-bearing observation: we assumed events would flow from many sources, and in practice there’s exactly one. We’re paying the full price of a pub/sub architecture for what is, underneath, a prioritized task queue.
Which is where the real subject of this article starts. If the problem is a task queue rather than a stream, what fits best? We put a slate of candidates against the same set of criteria.
Part 2 - The ADR: choosing an architecture for mass operations ( Technical analisys )
A note on scale. All of this only matters above roughly a few hundred thousand records per operation, across a few hundred distinct task types. Below that, the choice is mostly taste - any of these will do. The differences get real when either the volume of a single operation or the cardinality (how many independent task types you run) starts to hurt. Two different axes, and I’ll come back to both.
Decision context
Status: Proposed. Generation 2 is in production.
The business code is Java + Spring Boot. The action logic - remediation, promotion, dispatch - is simple and portable between architectures, which matters, because the cost of moving the logic itself is low in every option. The difference is the infrastructure around it. And the data source is one Oracle database, which quietly kills the main argument for pub/sub.
The criteria, and what each is worth
Each criterion carries a weight that comes from our context, not from a textbook: Critical, Important, or Nice-to-have. The weight is what carries the decision, not a point total. I’m not summing scores, and there’s a note further down on why.
| # | Criterion | Weight | Why this weight |
|---|---|---|---|
| K1 | Per-record audit (status, attempts, last error, queryable) | Critical | Regulatory and operational. Non-negotiable |
| K2 | Prioritization (urgent jumps scheduled) | Nice-to-have | Real scenario, but we cope without it today |
| K3 | Retry with backoff + error handling (DLQ) | Critical | Without it a failing record grinds in a loop |
| K4 | Horizontal scaling without a partition ceiling / rebalance | Critical | Task cardinality grows; topology can’t be the brake |
| K5 | Long-running tasks (no fighting a broker timeout) | Critical | Some actions take minutes, not milliseconds |
| K6 | Operational cost / infrastructure complexity | Important | The pain that started this whole analysis |
| K7 | Path to a multi-step process | Nice-to-have | We have a few chained jobs today, but they’re the minority |
| K8 | Maturity and risk | Important | A regulated client will judge this in their own review |
| K9 | Lock-in / exit cost | Nice-to-have | Domain logic stays agnostic anyway |
One thing to keep straight before the candidates. These criteria score what the queueing mechanism gives you out of the box. Where we’ve already built something on top of today’s Kafka to satisfy a criterion, I’ll flag it separately - because that code doesn’t vanish from the ledger. It works, and it’s ours to maintain. The whole point of even looking at Temporal is to delete that code, not relocate it.
Candidate A - stay on Kafka (possibly + KIP-932 Share Groups)
The cheapest decision is to change nothing. And since Kafka 4.2 there’s a genuinely strong argument for staying: Share Groups (KIP-932) give you native queue semantics and break the coupling between parallelism and partition count. That addresses our single biggest partition headache.
Where it still doesn’t close:
K1, audit (Critical). Kafka itself gives you nothing here. Delivery count and in-flight state are broker internals, not a queryable “what’s the status of record X” API. What we have today: audit built on our own state tables. It works and it meets the requirement - but it’s our code, our schema, our maintenance.
K3, retry/DLQ (Critical). Kafka lets you release a record for redelivery, but with no backoff (delayed retries are still just on the roadmap), and native DLQ is KIP-1191, aimed at Kafka 4.4 - not here yet. What we have today: a simple, reliable mechanism - a retry counter on the record plus a cron that flips statuses and re-runs based on max attempts. Reliable, and again, ours to keep alive.
K2, priorities (Nice-to-have). No native priorities in share groups, so you’re back to separate topics or your own logic. We don’t have this today and we manage.
K5, long tasks. A record is handed out under a time lock (about 30s by default); anything longer means renewing the lock. Same timeout fight, different name.
K6, cost. Goes up, not down. Share groups add a share coordinator and an internal state topic to the broker; the core pain (running a Kafka cluster) stays intact. And getting to 4.2 means an upgrade path - for a cluster on 3.x, a ZooKeeper-to-KRaft migration first, which in an enterprise is a project of its own.
K8, maturity. Kafka itself is proven, but share groups specifically are new (GA in 4.2), so this exact variant is younger than it looks. Classic Kafka without share groups: solid.
K9, lock-in. Low. Kafka is a standard with many implementations.
Verdict: KIP-932 solves exactly one of our problems (K4, the partition ceiling). Everything on the state side - audit, priorities, backoff, DLQ, long tasks - stays something we build and run ourselves, and the cluster gets more complex, not less. It’s still dispatch without task-state management. And task-state management is the code we’re trying to delete.
Candidate B - a plain queue (RabbitMQ / Oracle AQ / SELECT … FOR UPDATE SKIP LOCKED)
If the problem is a task queue, a plain queue is the obvious move. Three realistic flavors.
RabbitMQ gives you competing consumers with no notion of partitions, and consumer scaling on Kubernetes is about as clean as it gets - KEDA on queue depth, 0 to N with no rebalance. It nails K4. Oracle AQ is tempting because the source is already in Oracle, so it’s zero new infrastructure. And SKIP LOCKED is a queue built directly on the database we already have, no additional system at all - the simplest thing that could possibly work.
Where it doesn’t close, across the flavors:
K1, audit (Critical). A queue hands out a task and forgets it after the ack. Per-record status, attempt count, last error - back to your own state store. The one thing worth saying here: our current state tables and retry+cron would port to this variant almost unchanged, SKIP LOCKED especially, where the state already lives in a table, so audit and retry are a natural extension rather than a bolt-on. That makes B cheap to migrate to. It does not delete our code - it entrenches it.
K3, retry/backoff. In RabbitMQ, backoff is the dead-letter-exchange-plus-TTL dance; doable, but it’s your infra code again. AQ has delayed retry natively, which is a real point in its favor.
K5, long tasks. RabbitMQ’s consumer_timeout (30 min by default) kills the channel if the ack doesn’t come, and there’s no heartbeat equivalent, so multi-hour actions need workarounds.
K2, priorities. RabbitMQ - mirrored queues are gone in 4.0, and quorum queues give you two priority levels. Enough for “urgent vs scheduled,” not for anything richer. AQ and SKIP LOCKED handle priority with a query or a column.
K7, process path. None of them offer anything here.
There’s also a RabbitMQ broker-scaling wrinkle: a single queue lives on a single node, so one queue’s throughput is capped by one broker. At our cardinality (a few hundred queues) it happens to spread across the cluster, so it’s a smaller problem for us specifically, but worth noting.
K8, maturity is the highest of the whole field - SKIP LOCKED is plain SQL that’s been proven for decades, Rabbit and AQ have long enterprise track records. Easiest thing to get past a bank’s architecture review. K9, lock-in: SKIP LOCKED is zero (it’s SQL on a database you already run), Rabbit/AQ low.
Verdict: a plain queue solves dispatch (K4) cleanly and cheaply. It doesn’t take task state off your hands (K1, K3, K5, K7). Everything we wanted to walk away from - our own audit, retry, error-handling code - we’d rebuild somewhere else. SKIP LOCKED is the interesting one here as the minimum that works, if simplicity were the only thing we cared about.
Candidate C - Temporal Standalone Activities
We already run Temporal, for orchestrating long-lived processes (workflows). Standalone Activities, announced in 2026, let you run a single Activity as its own thing, without wrapping it in a workflow. The client submits a task, the server durably records it before acknowledging, a worker executes, and the result, attempt count and last error are visible in the API and Web UI.
I’ll go through this one with less charity than the others, not more - it’s the strongest candidate, which is exactly why it deserves the hardest look.
K1, audit (Critical). Native, with an asterisk. Status, retry count, last error are queryable, with no state store of ours. But at our scale “queryable” means standing up Elasticsearch or OpenSearch as the visibility backend - another cluster to run - and completed activities are subject to retention, so they age out of history. Audit that has to survive for years still means an export to our own store. So it’s a yes, conditional on ES plus an archival policy. Not free.
K2, priorities (Nice-to-have). Weighted priority tiers on one task queue, with starvation protection - and because of that starvation protection it is not a strict priority queue: under saturation, some of the lower tier still gets through. If the business ever demands “urgent is absolute, full stop,” you split into separate task queues and worker pools, which drags topology management back in. And pause-per-task only lands at GA. Realistically a partial.
K3, retry/backoff (Critical). Declarative retry policy with backoff, error handling in the box. Clean yes.
K4, scaling (Critical). This is where the hidden cost lives. On the consumption side, the parallelism limit is your worker count, no partition ceiling. But the bottleneck moves to the Temporal server’s persistence: every submission, every state transition, every heartbeat is a write. For a straight grind through a million records, Temporal is heavier than an append to a Kafka log - you’re paying I/O for every durability guarantee. This isn’t “scale the workers and you’re done.” It’s “size, and possibly shard, the Temporal database.” Partial, and it’s the one partial on a critical criterion.
K5, long tasks (Critical). Heartbeats plus checkpointing, designed for exactly this. Temporal’s sweet spot. Yes.
K7, process path (Nice-to-have). The same Activity runs standalone and as a workflow step with no code change. We already have a handful of chained jobs where one triggers the next - a minority, but a real one, and it’ll grow. When a mass operation turns into a process (promotion, then notification, then settlement), we wrap the existing Activities instead of rewriting. No other candidate offers this.
K8, maturity (Important). This is a new feature (Public Preview), and I’ll say so plainly. But without the theatrics: Temporal itself has been solid for us and has never let us down, and a Standalone Activity is, mechanically, the same pattern as a workflow with a single step. It’s a narrower API on a mature core, not a new engine. The risk is real but bounded to the freshness of the API surface, not the foundation. A partial, not a fail.
K9, lock-in (Nice-to-have). Low, and that’s down to discipline rather than anything Temporal does for you. The action logic lives as plain Java/Spring inside the Activity (applyPromotion(input)), and Temporal is just the queueing-and-orchestration layer around it. Exit means swapping that layer, same as swapping a Kafka consumer or a SKIP LOCKED loop; the domain logic stays untouched. The caveat: this holds only while we stick to Standalone Activities. The moment we step into K7 and write actual workflows, the workflow code becomes Temporal-shaped - determinism, signals, versioning - and that doesn’t port so easily. The same process path that’s an argument for is also where the lock-in shows up. Eyes open.
The honest caveats, because without them this is marketing and not an ADR: the Public Preview status and the missing native pause per task until GA (workaround: pause at the reader by not submitting new work - operationally fine, not the same thing). Sizing for a million tasks - a million standalone activities is real load on server persistence, and submission has to be rate-limited so we don’t DDoS our own platform. And at-least-once semantics mean Activities must be idempotent, exactly like Kafka consumers, so for us nothing changes there.
Verdict: it wins where the weight is highest - task state on the critical criteria (K3 retry, K5 long tasks, K1 audit from the platform instead of our code). Its losses sit lower: K8 (Important, but blunted by the maturity of the core) and K2/K9 (Nice-to-have). The one worry on a critical criterion is K4, scaling and persistence - and that’s precisely what a PoC has to settle before any binding call. Not a candidate that wins everything. A candidate that wins on what matters most to us and pays where we have more slack.
Side by side
Legend: ● native / ◐ partial or via our own code / ○ none. The Weight column carries the context - it decides, not the count of dots.
| Criterion | Weight | A. Kafka (+KIP-932) | B. Plain queue | C. Temporal Std. Activities |
|---|---|---|---|---|
| K1 Per-record audit | Crit. | ◐¹ | ◐¹ | ●⁶ |
| K2 Priorities | nice | ◐ | ◐ | ◐⁷ |
| K3 Retry + backoff / DLQ | Crit. | ◐¹ | ◐² | ● |
| K4 Scaling, no partition ceiling | Crit. | ● | ● | ◐⁸ |
| K5 Long-running tasks | Crit. | ◐ | ◐ | ● |
| K6 Operational cost | important | ○³ | ●⁴ | ◐⁹ |
| K7 Path to a process | nice | ○ | ○ | ● |
| K8 Maturity / risk | important | ● | ● | ◐¹⁰ |
| K9 Lock-in | nice | ● | ● | ●¹¹ |
How to read it: the decision isn’t in the count of checkmarks, it’s in how they land against weight. On the four critical criteria - audit (K1) and retry (K3) we have today, but as our own code we want gone; long tasks (K5) only C does natively; scaling (K4) is where C is actually weaker, and it’s the thing the PoC exists to verify. C’s losses are all on lower-weighted criteria. It wins where the weight is heaviest and loses where it’s lightest. That’s the justification, not arithmetic.
On method. This is an ADR in the spirit of MADR, with a considered-options section and a rating per option. I’m deliberately using a qualitative rating with explicit weights instead of a weighted score with a point total. When you’re choosing an architecture, the weights and the ratings are the author’s judgment anyway, so a final number (7.2 vs 5.8) implies a precision that isn’t there and invites an argument about the second decimal place instead of about the thing. The spread of marks against weight carries the same information without faking objectivity.
Decision
We lean toward candidate C, Temporal Standalone Activities, with both eyes open about the bill.
It’s not that Temporal wins everywhere. It doesn’t - it’s heavier on K4 and pricier on K6. It’s how its strengths land on the critical criteria. Audit (K1) and retry (K3): today we satisfy those with our own state tables and a retry+cron, which works, but it’s our code to maintain, and Temporal hands both to us from the platform and lets us delete that code. Long tasks (K5): only Temporal does them natively, without the timeout fight. On top of that there’s a bonus on a nice-to-have that’s going to gain weight over time - the process path (K7), where we already have a few chained jobs and the same Activity runs standalone or as a workflow step without a rewrite. Context seals it: we already have the Temporal skills and the server, and the core is proven here. A Standalone Activity is a narrower API on the same engine, not a new one.
The costs we’re knowingly signing up for: Public Preview (K8, blunted by the maturity of the core, but real), heavier persistence under a grind (K4), server re-sizing and probably ES for visibility (K6). If the only goal were to reproduce today’s state for the least money, SKIP LOCKED would honestly be the better pick - our current state tables and retry would move over almost verbatim, zero new infrastructure, zero lock-in. The difference is that SKIP LOCKED entrenches our state-management code and Temporal removes it, while also giving us K5 and K7 that database-as-a-queue never will. And if many event sources and independent consumers of the same data ever did materialize, Kafka comes back into play. This isn’t an ADR that says Temporal is best. It’s one that says for our set of weights Temporal comes out ahead, and here’s exactly what we pay for it.
Consequences
On the plus side we delete our own audit/retry/state-store code, we get long-task handling from the platform, we get a path to workflows without rewriting logic, and the domain logic stays agnostic (low lock-in for today’s scope).
What we accept: a dependency on a Public Preview feature, including no native pause-per-task until GA (mitigated by pausing at the reader). Temporal persistence as a new bottleneck, needing sizing and rate-limited submission - the backpressure is ours to write. A likely need for ES/OpenSearch behind visibility, plus a separate archival policy for multi-year regulatory audit. Priorities that are soft (weighted) rather than strict - if the business ever wants strict, that’s separate task queues and worker pools. And lock-in that grows the moment we enter workflows (K7): the thing that’s a benefit is also the trap on the way out.
Next step is a PoC at representative volume - a million records of one type, plus a few hundred types at a few dozen each - measuring, above all, the load on Temporal persistence and the real throughput while audit is being written at the same time. That’s where the risk of this architecture lives, not in the workers. No binding decision before the PoC.
A note on how we work
None of this is a one-off. Writing the key technology decisions down like this - candidates, weighted criteria, the costs we’re signing up for - is how we run them as a matter of course. It keeps the reasoning on the table instead of in someone’s head, it lets the next step be taken in the open rather than defended after the fact, and when we revisit the call in a year we can see what we actually knew at the time. The output isn’t the decision. It’s a decision you can audit, the same way we expect to audit every record that passes through the system we’re arguing about.
¹ the platform doesn’t give this, but we already have it built in our own code (state tables + retry counter/cron) - works, but it’s our maintenance burden. ² Oracle AQ has delayed retry natively; our current mechanism ports straight over. ³ cluster grows by a share coordinator + the 4.2 upgrade. ⁴ AQ/SKIP LOCKED especially - zero new infrastructure. ⁶ queryable from the platform (deletes our state store), but needs ES for visibility + an archival policy for multi-year audit. ⁷ weighted tiers, not a strict queue; pause-per-task only at GA. ⁸ bottleneck moves to Temporal persistence; heavier than a log append. ⁹ lower than hand-building state, but needs server re-sizing and ES. ¹⁰ Public Preview, but on Temporal’s mature core (same pattern as a workflow, one step). ¹¹ logic in agnostic Java; lock-in only appears once you enter workflows (K7).