Several jobs a day were pure data entry against a web interface. Picking an order meant walking a handheld terminal through the same sequence of screens for every single unit, and orders could run to thousands of units. Bulk uploads meant logging into a portal, keying a one-time passcode, and uploading a workbook one sheet at a time. Wave planning meant logging into a third system, running its planner, waiting for it, running its scheduler, waiting again, then filtering a grid and exporting it — a sequence somebody had to stay logged in for.
It was slow, but the real cost was that it was boring. Boring work done by hand produces errors, and errors against live inventory are expensive to unwind.
A queue-driven automation service that drives the real interfaces the way an experienced operator would, with a bounded pool of concurrent sessions and a job record for every run.
Everything in the design assumes it will fail at the worst possible moment, because eventually it does.
- Jobs are claimed in the database before they start, so two workers can never pick the same order
- Jobs stuck mid-run when a server restarts are returned to the queue on boot, with a log line explaining why
- Cancellation is checked before every step, so stopping a job preserves and reports the partial work already done
- A stalled session restarts only if nothing was picked yet — never after real work, where a retry would double-count
- Cases the automation shouldn't handle alone stop and escalate with the specific item, location and reason attached
The most valuable work on this project was unglamorous. An earlier version reported success when uploads had actually failed — the error was raised inside a block that caught and discarded it, so the caller always saw a green result. Rewriting those paths to fail closed, and to treat an unreadable confirmation page as a failure rather than an assumption, mattered more than any speed gain.
One portal returned HTTP 200 on failure, with the real outcome buried in a status field. One vendor template had a spelling error in a column header that the system matched on, so the typo had to be preserved deliberately and documented. Retries wait past the one- time-passcode window, because retrying inside it just re-sends the same rejected code.
Where an ordering decision could cause damage, the safe choice was taken explicitly. Inbound requests are marked handled before any upload runs: a crash mid- upload leaves a visible flag someone can retry, whereas the alternative risks creating duplicate orders in the customer's system. A stuck flag is recoverable. A duplicate order is not.
Work that consumed hours of attention runs from a queue with a full audit trail. More importantly, the failure modes are now visible: when something doesn't work, it says so, in a place someone is looking.
