When your system is fast, scalable — and still wrong

Most production bugs are not caused by slowness.
They are caused by multiple things happening at the same time.

Concurrency bugs exist:

At low traffic
On one machine
Even with perfect hardware

They are logic bugs, not capacity bugs.

1️⃣ What “Correctness” Actually Means

A system is correct if it behaves as expected under all valid interleavings of operations.

Not:

“Works on my machine”
“Passes unit tests”
“Works at low traffic”

But:

Works when operations overlap in unpredictable ways.

2️⃣ Why Concurrency Breaks Correctness

Because time is not linear in distributed systems.

You think this happens:

A → B → C

Reality:

A ↘
     B
A ↗     C

Operations interleave.

3️⃣ The Simplest Concurrency Bug: Lost Update

❌ Naive Code

async function likePost(postId) {
  const post = await db.get(postId);
  post.likes += 1;
  await db.save(post);
}

What you expect

2 likes → likes + 2

What actually happens

2 likes → likes + 1 ❌

Concurrency overwrote correctness.

4️⃣ Why This Happens (Key Insight)

The bug is not “missing a lock”.
The bug is read-modify-write without protection.

This pattern is always unsafe under concurrency.

5️⃣ Naive Fix #1 — Application Locks ❌

await mutex.lock();
const post = await db.get(postId);
post.likes += 1;
await db.save(post);
await mutex.unlock();

Why this fails

Only works in one process
Fails with multiple instances
Kills throughput

6️⃣ Correct Fix #1 — Atomic Operations (Best)

UPDATE posts
SET likes = likes + 1
WHERE id = ?

Why this works

Atomic at DB level
No read-modify-write
Scales correctly

Push correctness down to the lowest layer possible.

7️⃣ Correct Fix #2 — Optimistic Concurrency Control (OCC)

Version-based update

const post = await db.get(postId);

await db.update(
  { id: postId, version: post.version },
  { likes: post.likes + 1, version: post.version + 1 }
);

If version mismatches → retry.

8️⃣ Correctness Bug #2 — Check-Then-Act

❌ Broken Logic

if (balance >= amount) {
  balance -= amount;
}

Why it fails

Condition can change
Assumption becomes false

9️⃣ Correct Fix — Combine Check + Act

UPDATE accounts
SET balance = balance - 100
WHERE id = ? AND balance >= 100

Rows affected:

1 → success
0 → insufficient funds

🔟 Correctness Bug #3 — Idempotency

❌ Duplicate Request Bug

createOrder(order);
createOrder(order); // retry

Result:

Two orders ❌

1️⃣1️⃣ Correct Fix — Idempotency Keys

if (seen(idempotencyKey)) return cachedResult;

const result = createOrder(order);
store(idempotencyKey, result);

Correctness is about recognizing repeated intent.

1️⃣2️⃣ Why Correctness Is Harder Than Performance

Performance	Correctness
Gradual failure	Binary failure
Can degrade	Cannot be “almost right”
Measurable	Often invisible
Testable	Needs reasoning

1️⃣3️⃣ Why Load Doesn’t Matter Here

Concurrency bugs:

Happen at 2 requests
Reproduce rarely
Pass load tests
Fail in prod

This is why senior engineers fear them.

🎯 Golden Rules for Correctness Under Concurrency

Avoid read-modify-write
Use atomic primitives
Prefer database-enforced correctness
Make operations idempotent
Assume retries happen
Design for reordering

📘 Correctness Under Concurrency