fix(chord): recover worker links after a sleep or a lost link

The heartbeat no longer counts time the process did not run (a laptop's sleep, a blocked event loop) as silence: after such a pause each open link pings at once and closes only if its peer stays silent for close - ping.

A running worker's lost link is now a stop instead of a failure: worker_link_lost is reported and the held route starts a new worker. A second lost link within a minute of the last recovery still fails the address.
This commit is contained in:
Armin Ronacher 2026-11-03 18:15:53 +01:00
parent 7aed50ae8b
commit 9f7d909f8b
14 changed files with 180 additions and 55 deletions

View file

@ -332,7 +332,8 @@ export class SessionWorkers implements SessionWorkerControl {
/**
* A worker's start, crash or failure: an owner notice (redacted), with the stderr tail as Pi Mobile; a crash is
* "Worker exited (signal): tail" in the notice and the status error too, as in v1.
* "Worker exited (signal): tail" in the notice and the status error too, as in v1. A lost link Chord recovers from
* is only a log event.
*/
#failed(error: WorkersError): void {
if (error.kind === "worker_exited") {
@ -342,6 +343,15 @@ export class SessionWorkers implements SessionWorkerControl {
this.#crashed(session);
return;
}
if (error.kind === "worker_link_lost") {
// Chord stopped the worker and its holders start a new one: an event, no notice.
const session = error.address.slice("session|".length);
const reason = processLogs.redactor.redact(
error.error instanceof Error ? error.error.message : String(error.error),
);
processLogs.append(`${SESSION_PREFIX}${session}`, "event", `Worker lost its link (${reason}); restarting it`);
return;
}
if (error.kind !== "worker_failed") {
console.error("p2 workers:", error);
return;