Operational and Security Risks of Automation

10.1 Forty-five minutes

In August 2012, a major US trading firm deployed new software to its trading systems. An old piece of code, accidentally left active on one server, began sending unintended orders into the stock market. In about 45 minutes, before the firm could stop it, it lost roughly $440 million. The firm had to be rescued and was later acquired.

"That was a professional firm with engineers and risk teams," said Hash. "If automation can fail that badly for them, it can fail for anyone. When code trades for you, the code becomes one of your biggest risks."

10.2 What can go wrong

Risk	Example
Logic bugs	A rule inverted by mistake: buying when it should sell
Runaway loops	A bot repeatedly placing orders because it doesn't detect that previous orders filled
Wrong units or decimals	Ordering 100 BTC instead of 1.00 (Module 6, Chapter 11)
Stale data	Acting on old prices after a connection drops
Exchange outages	Bot can't exit positions during a crash
API changes	The exchange updates its API; the bot breaks silently
Key compromise	Stolen API keys or session keys (Chapters 6 and 8)
Server problems	The machine running the bot crashes or loses internet
Dependency failures	A data provider, oracle, or automation service fails
Human error	Deploying the wrong version, or testing with real funds by accident

10.3 Kill switches

A kill switch stops all automated activity immediately. Every automated system should have one, and you should know how to use it without thinking.

Kill switch level	Action
Stop the bot	Shut down the program or pause the platform
Cancel open orders	Remove all pending orders on the exchange
Revoke access	Delete or disable the API key; revoke on-chain approvals and session keys (Chapter 8)
Flatten positions	Close open positions, if appropriate

Write down the steps and practise them on a testnet. In a real emergency, you won't want to be searching documentation.

10.4 Built-in safety limits

Good automated systems include limits that stop them even if the strategy logic goes wrong:

Safety limit	Example
Maximum order size	Never place a single order above a set amount
Maximum position size	Never hold more than a set exposure
Maximum orders per period	Stops runaway loops
Daily loss limit	Shut down after losing a set amount (Module 12, Chapter 10)
Price sanity checks	Reject orders far from the current market price
Stale data checks	Pause if data hasn't updated recently
Balance checks	Confirm expected balances before and after trades

"These limits are like the circuit breakers in your risk plan," said Node, "but for the machine."

"Exactly," said Hash. "The strategy decides what you want. The limits decide what you'll never allow, no matter what."

Chapter 10 · Automation safety limits

An automation kill switch works in layers: stop the bot, cancel open orders, revoke API or session keys, and flatten positions. Alongside it, hard limits cap maximum order size, position size, orders per minute, and daily loss, and sanity-check prices and stale data to catch runaway behaviour.

10.5 Monitoring and alerts

Automation doesn't mean ignoring things. Set up alerts for:

1. Every trade executed, at least at first.
2. Errors and rejected orders.
3. The bot going offline or failing to report.
4. Unusual activity: too many orders, unexpected positions.
5. Account balance or drawdown thresholds.
6. API key or permission changes.

Alerts should reach you in a way you'll notice, but not so many that you start ignoring them (Module 12, Chapter 12).

10.6 Testing changes safely

Practice	Why
Test changes on a testnet or demo account first	Catch bugs without real money
Change one thing at a time	Know what caused any new behaviour
Keep previous versions	Roll back quickly if needed
Deploy changes at quiet times, while watching	Not before sleeping or during major events
Start new versions with small size	Same principle as going live (Chapter 5)

10.7 Security checklist for automation

Area	Check
API keys	Trade-only permissions, IP whitelisted, stored securely, rotated (Chapter 6)
Capital exposure	Bot uses a sub-account or separate wallet with limited funds
On-chain permissions	Narrow approvals and session keys; reviewed regularly (Chapter 8)
Server and devices	Updated, secured, with minimal software installed (Module 2, Chapter 12)
Code	Reviewed, tested, never containing secrets in public places
Third-party services	Reputable, with limited access
Kill switch	Documented and practised

10.8 Try it yourself

Write a one-page operational plan for an automated strategy you might run: its kill switch steps, safety limits, alerts, and security checklist. Then, on a testnet, practise the full kill switch: stop the bot, cancel orders, revoke the API key, and close positions. Time how long it takes.

Key takeaways from this chapter

1. Automation introduces risks from bugs, runaway loops, unit errors, stale data, outages, API changes, key compromise, servers, dependencies, and human error.
2. Even professional firms have suffered huge losses from automation failures, like the $440 million loss in 2012.
3. Every automated system needs a kill switch: stop, cancel orders, revoke access, and flatten positions.
4. Built-in safety limits on order size, position size, order frequency, losses, prices, data freshness, and balances protect against logic failures.
5. Monitor with meaningful alerts, without overwhelming yourself.
6. Test changes safely: testnets first, one change at a time, rollback options, and small initial size.