The most expensive Salesforce failures are often not the loud ones. A transaction succeeds most of the time. A retry fixes some records. The same user action behaves differently on Tuesday than it did on Monday. Each team sees a small, plausible defect inside its own boundary, while the actual problem lives between those boundaries.
01 · The situation
A dependable process had become a collection of workarounds.
The client sold configurable equipment with recurring service. Salesforce managed the commercial process; an ERP owned fulfillment and finance. When an opportunity reached a qualified state, Salesforce assembled an order request, sent it through middleware, received the ERP identifier, and exposed fulfillment status back to sales and operations.
The process had evolved over several years. Administrators had added record-triggered Flows for new business rules. A legacy Apex trigger still handled validations and integration preparation. A managed package updated related records. Middleware retried requests after timeouts. Each decision had once been reasonable in isolation.
No single component was obviously broken. The business still had a serious problem: operations could not tell whether a failed-looking transaction was safe to retry.
02 · Why it looked random
Three symptoms pointed at three different teams.
Users reported duplicate ERP orders, Salesforce records stuck in “Sending,” and occasional UNABLE_TO_LOCK_ROW errors. Support treated them as separate incidents:
- Duplicate orders appeared to be middleware retry behavior.
- Stale statuses appeared to be a Salesforce callback mapping issue.
- Row locks appeared to be ordinary batch-load contention.
The failures clustered during higher-volume periods, but did not occur on every large transaction. Replaying the same record often succeeded. That made the first hypothesis—bad data—feel convincing.
Visible failures were downstream effects. The first incorrect decision happened earlier, when two automation paths independently claimed ownership of the same business event.
03 · The investigation
We stopped debugging tickets and reconstructed the transaction.
A support ticket describes what a user noticed. It rarely describes the complete execution path. We worked from a single affected transaction outward and used several techniques to test competing explanations.
1. Build an automation and ownership inventory
We retrieved the metadata and mapped every write on Opportunity, Order Request, Order Line, and Account: record-triggered Flows, Apex triggers, invocable actions, package automation, validation rules, scheduled jobs, and middleware callbacks. For each component we recorded entry criteria, fields read, fields written, transaction boundary, retry behavior, and named owner.
This revealed two publishers. A newer after-save Flow created an integration request when the commercial record became ready. The legacy trigger also enqueued a job when a related order request changed. On normal records, a guard condition prevented the second path. On amended deals, a field update later in the transaction made both conditions true.
2. Correlate logs across asynchronous boundaries
A single Salesforce debug log was not enough because the transaction crossed Flow, a queueable job, a platform event subscriber, middleware, and a callback. We added a correlation ID at the first durable handoff and carried it through every log entry and message. Historical requests were reconstructed by joining Salesforce job times, event payloads, middleware request logs, and ERP response records.
For the Apex segment, targeted trace flags and Replay Debugger made variable state and branch selection visible without filling logs with unrelated users. Salesforce documents Replay Debugger as suitable for Apex classes, triggers, anonymous Apex, and log files; its limitation to one log at a time is exactly why an external correlation key matters for async work.
3. Reproduce concurrency, not just data
The original test fixture created one order against one account. Production created many child records against a smaller set of parent accounts, and the package maintained rollups on those parents. We built a bulk test matrix that varied:
- single-record versus bulk entry;
- distinct parent accounts versus a shared “hot” parent;
- serial versus parallel requests;
- first delivery, delayed response, retry, and duplicate callback;
- new sale versus amendment paths.
The lock error appeared only when concurrent child updates converged on the same parent while the package rollup and callback were also writing there. Salesforce record locking is additive: child relationships, rollups, lookups, and sharing behavior can all expand the lock footprint. Testing only isolated records had hidden the actual shape of the problem.
4. Use controlled failure injection
We introduced a safe test-mode delay in the middleware response and forced a timeout after the ERP accepted the request but before middleware recorded the response. That reproduced the duplicate immediately. The retry used a new request identifier, and the ERP endpoint interpreted it as a new order. Both systems were behaving according to their local rules; the end-to-end contract was wrong.
Diagnostic principle
Intermittent does not mean random.
It usually means the missing input is time, order, concurrency, or state. Add those dimensions to the test before changing the code.
04 · The root cause
There was no single bug. There was a broken reliability contract.
The compound failure required four conditions:
- 01Split ownership
Flow and Apex could both translate one business transition into an outbound request.
- 02Delivery before stable completion
One event path could become visible before the full source transaction had reached its intended durable state.
- 03Retry without idempotency
A timeout created uncertainty, and the retry looked like a brand-new command to the ERP.
- 04An oversized lock footprint
Parallel work updated child records that converged on the same parent, increasing contention and extending processing time.
The row lock was not the cause of the duplicate. It increased latency, which made timeouts and retries more likely. The duplicate publisher was not enough by itself either; an idempotent consumer would have neutralized it. Reliability failed because no layer owned the invariant: one accepted commercial transition must create at most one ERP order.
05 · The solution
One orchestrator, one durable command, safe retries.
We did not rewrite every automation. We narrowed responsibility and introduced a stable boundary between the Salesforce transaction and the external side effect.
Consolidate the business transition
The record-triggered Flow remained the administrator-owned entry point because its qualification rules changed with the business. It no longer published or called integration logic directly. Instead, it invoked one bulk-safe Apex action responsible for validating the transition and creating a durable integration command.
The legacy trigger’s publishing branch was removed after dependency tests proved that no other process relied on it. The trigger retained only the validations that could not yet be retired. A custom metadata switch allowed the old path to be disabled by transaction type during the rollout, giving operations a controlled rollback lever.
Create a durable outbox record in the source transaction
The orchestrator inserted an Integration_Command__c record in the same transaction as the business change. If the transaction rolled back, the command rolled back. A post-commit worker published only committed, ready commands. This made the handoff observable and gave support a place to see state, attempts, last error, and next action.
// Illustrative pattern; names are generalized.
String key = OrderKey.forTransition(orderRequest);
Integration_Command__c command = new Integration_Command__c(
Idempotency_Key__c = key,
Operation__c = 'CREATE_ORDER',
Source_Record__c = orderRequest.Id,
Status__c = 'READY'
);
Database.upsert(command, Integration_Command__c.Idempotency_Key__c, false);The idempotency key was deterministic for the business operation—not generated per HTTP attempt. A unique external-ID field prevented Salesforce from creating two commands for the same transition.
Make the receiving side idempotent
Middleware forwarded the same key on every attempt. Before creating an ERP order, the consumer checked whether that key had already been accepted. A repeat request returned the original outcome rather than creating a second order. The callback used upsert semantics and ignored state regressions, so a late “accepted” message could not overwrite “fulfilled.”
This changed retry from a business risk into a transport behavior. Timeouts could still happen. The system could now retry safely.
Reduce and sequence contentious writes
We removed an unnecessary Account touch from the callback and moved nonessential rollup work out of the critical path. Work for records sharing a hot parent was grouped and processed serially; unrelated parents could still run in parallel. Queries used a consistent parent-first locking order where explicit locks were required.
Design the event contract deliberately
For event publication, we used post-commit behavior so downstream consumers could not act on a source change that later rolled back. The payload carried the command ID, immutable idempotency key, operation, schema version, occurred-at time, and correlation ID—enough to process and trace the event, without copying the full Salesforce record into the message.
Salesforce’s event-driven architecture guidance specifically warns that publish timing can introduce row locks or race conditions and recommends deliberate standards for event payloads and consumers. We treated that guidance as a contract, not a checkbox.
06 · How we verified it
The happy path was the smallest part of the test plan.
Verification covered behavior across failure and recovery states:
Mixed create and amendment batches stayed within limits and produced one command per transition.
Hot-parent tests completed without competing writes in the critical path.
Repeated messages and callbacks returned the original result without another ERP order.
A failed source transaction left no publishable outbound command.
An accepted-but-unacknowledged request retried with the same idempotency key.
Support could replay a failed command without editing the commercial record.
We deployed in stages, first emitting comparison telemetry without changing the downstream action. We compared old-path and new-path decisions, then enabled the new orchestrator for a limited transaction segment, expanded coverage, and finally removed the obsolete publisher. Dashboards tracked commands by state, age, attempts, and terminal error—not just Apex exceptions.
The visible outcome was straightforward: duplicate creation stopped, lock-related failures fell out of the critical path, and operations gained a clear recovery process. More importantly, the technical and business teams shared one definition of what “sent,” “accepted,” and “complete” meant.
07 · Practices to reuse
What developers, admins, and architects can take from this.
Treat automation ownership as architecture.
Before adding a record-triggered Flow, identify every existing automation on the object and the fields it writes. Define one owner for each business transition. Use entry criteria and bypass controls deliberately, and keep fault paths visible to operations.
Test time and concurrency, not only values.
Bulkify the code, but do not stop there. Test shared parents, parallel delivery, callbacks arriving out of order, timeouts after side effects, and duplicate messages. Carry correlation IDs across jobs and prefer deterministic idempotency keys over request IDs.
Define invariants and failure ownership.
Write down which system owns each state, what “accepted” means, how long uncertainty may last, and who can replay work. Choose synchronous, asynchronous, batch, or event-driven patterns from the business consistency requirement—not from tool preference.
Reliable architecture does not prevent every timeout or lock. It prevents those conditions from creating an ambiguous business outcome.
Official references
These Salesforce resources are useful companions when applying the patterns above:
Have a similar failure pattern?
Bring us the transaction nobody can explain.
A SteadNorth QuickScan can focus on one defined Salesforce issue and produce likely causes, risks, and prioritized next actions.
Start with a QuickScan