Start with the invariant
Two workers attempting the same job do not automatically require a distributed lock. First define what must remain true: one invoice per order, one active import per account, or one writer updating a shared file. A database uniqueness constraint can enforce the first invariant more directly than an application lock.
Imagine two requests inserting invoice rows for order 42. A check-then-insert sequence races: both checks can see no invoice. A unique constraint on order_id makes the database arbitrate the conflict. The loser can load the existing invoice instead of creating another. This design keeps correctness near the durable state.
Compare the mechanisms
| Mechanism | Suitable boundary | Failure to consider |
|---|---|---|
| Unique constraint | Durable row invariant | Handling the rejected insert |
| PostgreSQL transaction advisory lock | Cooperative work in one database transaction | Participants that ignore the lock |
| PostgreSQL session advisory lock | Session-scoped coordination | Connection pools retaining a lock |
| Redis lease | Temporary cross-worker ownership | Owner continuing after lease expiry |
PostgreSQL documents both session and transaction advisory locks. They are cooperative: taking an advisory lock does not prevent unrelated SQL from changing the protected rows. Prefer transaction-scoped ownership when work and database updates fit the same transaction. See PostgreSQL explicit locking.
The lease expires while a worker is still alive
Imagine a nightly export that takes a lease for thirty seconds. Worker A starts reading records, then pauses during a long runtime stall. Its lease expires; worker B acquires the same name and begins a new export. When A resumes, it still has ordinary program variables saying that it owns the task. The cache cannot erase that belief. If both workers upload the same destination file, acquisition succeeded twice at different times but the business operation still overlaps.
The practical lesson is to distinguish a lease from permission to commit a result. For correctness-sensitive writes, use a storage rule that rejects obsolete ownership, such as a monotonically increasing fencing token checked by the destination. Where the entire operation fits one database transaction, a row lock or a uniqueness constraint may express the actual invariant more directly. Before choosing an implementation, write down what must never happen: duplicate invoice creation, concurrent inventory reservation, or merely redundant thumbnail rendering. Those failures justify different levels of coordination and different recovery costs.
Choose the invariant before the lock service
Consider two workers creating a daily invoice for the same account. A unique database key on account and billing period protects the final record even if coordination fails. A lock alone protects only participating code paths: an administrator script or a new worker implementation can bypass it. For an update to an existing balance, a short transaction with an appropriate row lock keeps the protected read and write together. For a broader cooperative job, an advisory lock can be useful, but every participant must follow the convention.
A Redis lease becomes attractive when a task spans machines and the critical section cannot live inside a database transaction. That convenience introduces expiry, renewal and network-partition decisions. Give each acquisition an unguessable ownership value and release only if that value still matches. A plain delete can remove a successor's lock. Keep the lease duration and maximum processing time visible in metrics; a timeout selected without observing work duration is only an assumption. Avoid keeping a database transaction open across slow remote API calls merely to make a diagram look atomic.
A lease is a timed claim
A Redis lock commonly uses a unique owner token and an expiry. Release must compare ownership before deleting the key. Otherwise an expired owner can delete a newer owner's lock. Redis documents this ownership check and the assumptions behind distributed locking in its locking guide.
Our illustrative failure timeline is: worker A acquires a lease; A pauses; the lease expires; B acquires ownership; A resumes. Both can now attempt the external action. Renewing leases reduces this risk but does not make an already-issued side effect disappear. A fencing token works only when the protected resource rejects older tokens. An external API that ignores tokens cannot be protected by naming a counter a fence.
A practical decision flow
Start with a constraint for row uniqueness. Use a database transaction for related row changes. Add cooperative locks when the protected workflow needs serialization. Use leases for temporary coordination across processes, and keep external effects idempotent. Avoid holding database connections during slow remote calls when a durable job record can express ownership instead.
Test your design by pausing an owner beyond its timeout and restarting a worker after the external effect but before its completion record. Those two scenarios reveal more than a successful happy-path run. This is a design guide, not a throughput benchmark.
For the companion retry design, read Idempotency keys. Practice the trade-offs in AiFolder interviews.
Recover without deleting another owner’s lock
When a queue appears stuck, first identify the business job, current lease owner and destination state. Deleting a key is not a neutral repair: it can allow a second worker to run while the first is still active. Prefer stopping the original worker or waiting for a documented lease boundary, then replaying with an idempotent destination rule. If manual intervention is unavoidable, record why ownership was considered obsolete and which external effects already happened.
A useful rehearsal pauses one worker beyond its lease and lets another continue. The acceptance criterion is that the destination admits only a valid result, not simply that the lease eventually disappears. Also rehearse a crash after the external effect but before the completion marker; this catches duplication that normal acquisition tests miss. Keep acquisition latency, renew failures, lease age and rejected stale writes separate in dashboards. They answer different questions. A rising renew-failure rate can indicate coordination trouble even while jobs finish; repeated stale-write rejections can show that the safety boundary works while the operational timing needs attention.
