Fix an erpc worker failure

Find the start of the message erpc worker printed, and do what its row says.
This page describes ERPC CLI v0.1.0-rc.4, a pre-release.

Error messages and what to do

Worker failures use erpc's codes and exit statuses: see Errors and exit codes.
Message (start of it)WhyWhat to do
<app> cannot start without <SETTING>run.requires names a setting nothing setserpc worker secret put <SETTING>, set it in [vars], or deploy with the line that sets it (--grpc, --unlimited)
your key was added but does not get inThe SSH agent did not sign: 1Password locked, or its request not allowedUnlock 1Password, allow the request, and run the same command again. It carries on
1Password could not …: authorization timeout1Password asked for approval and nobody answeredAllow the request in 1Password, and run it again
<NAME> is empty; nothing was keptsecret put read an empty standard input: the shell is not a terminalRun it in a terminal, or pipe the value in
the keystore is locked, and nobody is at a terminal …A command without a terminal needs the keystore, and no unlock is kepterpc worker secret unlock at a terminal, then the command again
nothing is kept under keystore:… on this machineThe value was kept in another machine's keystoreerpc worker secret put <NAME> --replace here; for the key, bring that machine's whole keystore directory and its passphrase
release <id> failed: …; the server went back to <previous>The new release did not pass its health checkSee erpc worker logs. The previous release still runs
… is locked: the trading key has not been handed overThe server rebooted, the agent failed or was killed and systemd started it again, or stop --lockerpc worker start
port <n> is taken: …Another worker that is kept running has the same health-check port, or another program listens on itGive the app a port of its own in erpc.toml, then deploy again
release <id> runs, and its WAF did not come up: …The WAF did not start listening. Most often, another program is on waf.porterpc worker logs --waf, then free the port or change waf.port, and deploy again. The app runs meanwhile
waf.<key> … (from check, deploy)A [waf] key is outside its bounds, or waf.port is below 1024, run.port's, SSH's or the health check'sFix erpc.toml. erpc worker check lists every problem at once
the WAF's port <port> was not opened to … (from firewall allow, deny, sync)The change would let someone new in under WAF rules the server is not running yeterpc worker deploy: it puts the new rules in place, then opens the port
<kind> line <id> did not become active in <n> minutesThe line is still being set upDeploy again to wait more
OUT_OF_STOCKNo server of that plan is free in that regionDeploy with --region <another>
the start may or may not have gone through: …The request left and no answer came back, or the answer was 5xx, 408 or 429Run the same command again: it sends the same request
this purchase, or another of the account's, is still being made …The first request is still at work, or another purchase of the account's holds its lockRun the same command again shortly: it sends the same request
… changed in 1Password since this command read itAnother machine wrote the inventory item meanwhileRun the command again
the shared RPC takes none of the account's API keys from this machine (<n> tried) (from secret put --from-account, erpc rpc call, erpc webhook run)Each of the account's keys was refused from this machine: the keys have ended, the account has no active shared RPC plan, or the keys are limited to other addresses, such as a server'sThe ERPC Web Dashboard shows a key's end date and the plan, and makes a new key. For a key limited to the server's address: erpc worker secret put ERPC_API_KEY, typed or piped in
shared: … HTTP 401 (bench)The shared RPC refused the key: no active shared RPC plan, or a wrong keyCheck the plan on the dashboard, or set ERPC_API_KEY
shared: … HTTP 429 (bench)The plan's rate limitUse --interval 150 or more

A purchase that erpc cannot put on record

When a server or a line may have been started by an earlier request that erpc could not put on record, look at your account's servers and lines in the dashboard. Take a server up with erpc worker deploy --vps <ID>, or end what is not wanted. Then remove the entry from [env.<name>.starting] in erpc.toml by hand: erpc has no flag that drops it, and until then the same command sends the same request.

If the server may have been taken over

  1. Run erpc worker backup off first, to revoke the server's backup access key. off does not undo what the server already read.
  2. Move funds off its trading key, and replace its secrets.
  3. Rebuild the server from the ERPC Web Dashboard, and deploy again.
If the rebuild changed the server's SSH host key, erpc refuses the server until you update the pinned key. A deploy brings back the code, not the app's state, which only a backup can restore (Back up the app's state).

What erpc never does

  • Put a key, a secret, or a URL that carries one on a command line.
  • Write a trading key or an app's secret to a file, with one exception: its keystore on your machine, where each is encrypted under your passphrase. On a server, erpc worker backup on leaves the server's own backup keys in a root-only file.
  • Start anything without showing its price.
  • Leave a server open: the firewall is in force before the app first starts.
  • Reach a server whose host key does not match the pinned one.
The worker guide lists every message, including the WAF's.