After a rescan, the env daemon's file watcher sent every changed path in
a single `change` event. When a big batch of changes made that event
larger than the 16 MiB frame limit, `write_frame` rejected it. The
daemon's only output writer then shut down. The daemon process kept
running but sent no more replies or pings. Requests hung until the
client's 30 s silence limit dropped the connection and reconnected.
Now the watcher builds the event with `change_event`. If the serialized
path list would not fit in a frame (`MAX_PAYLOAD`), it sends the
existing `{ kind: "change", overflow: true }` event in its place. The
client already maps that to an `overflow` file change, so the wire
format does not change and the connection keeps working.
Testing:
- Added Rust unit tests in `watch.rs`. One checks that a small path list
is sent as `paths`. The other checks that a list larger than
`MAX_PAYLOAD` becomes `overflow`.
- `cargo fmt --check`, `cargo clippy --locked --all-targets -- -D
warnings` and `cargo test --locked` pass in `packages/env/daemon`.
- The `packages/env` vitest suite passes.
- Reproduced with the real daemon: I watched a directory and renamed in
a directory of 6,000 files with long paths, about 20 MB of path JSON.
Before the change, `fileInfo` requests timed out from the flush until
the client reconnected about 30 s later. After the change, the watcher
reports `overflow` right away and `fileInfo` keeps working. With 3,000
files, which fits in a frame, the paths are still reported as before.
Fixes#10516
Requested by @badlogic
Co-authored-by: triagent <triagent@localhost>
Connection.close() ends stdin and sends SIGTERM at once. The daemon did not
handle SIGTERM, so it died before its end-of-input kill_all, and running
commands were reparented to init. The daemon now blocks SIGTERM, SIGINT and
SIGHUP before starting any thread and waits for them on its own thread, which
kills every process group it started and exits. On Windows the job object
already kills commands with the daemon.
The pi-env daemon started an `exec`'s stdout and stderr readers and its
waiter with `thread::spawn`. `thread::spawn` panics when no thread can
be created, for example when a cgroup `TasksMax` or `RLIMIT_NPROC` limit
is reached. When that happened after the child was already running:
- only the request thread unwound, so the exec never got a reply;
- its control channel was dropped, so `cancel` and `kill` reached
nobody;
- the child kept running, even after the daemon exited.
`pty` started its reader, input writer and waiter threads the same way.
## Changes
- `exec.rs`: `spawn_reader` now uses `thread::Builder::spawn` and
returns an `io::Result`. If a reader or the waiter cannot start, `run`
sets `stop`, interrupts any readers already started, kills the process
group, reaps the child, removes it from `groups`, and returns
`spawn_error`.
- The waiter thread now gets the `Child` through a channel once it is
running. If the waiter cannot start, the child stays with the caller,
which can still reap it. Before, the failed thread's closure dropped the
child without waiting for it.
- `pty.rs`: the same handling for the reader, writer and waiter threads.
On failure the terminal is also removed from `terminals`.
## Testing
- Manual reproduction with a script that runs the daemon as `prlimit
--nproc=N -- pi-env`. It starts 30 `sleep 30` execs 200 ms apart, aborts
them all, then closes the connection. N ran from 37 to 70.
- Before: for most N, one or two execs stayed pending forever after
abort, and a `sleep 30` kept running after the daemon was closed.
- After: every exec settles, either `aborted` or `spawn_error`, and no
`sleep` is left running.
- `cargo build`, `cargo clippy --all-targets`, `cargo fmt --check` and
`cargo test` all pass. The `packages/env` vitest suite passes (78
passed), and `npm run check` passes.
- There is no automated regression test. A thread-creation failure
cannot be triggered reliably in a test: `RLIMIT_NPROC` counts all of the
user's tasks and does not apply to root, and the daemon has no
fault-injection hook.
- The Windows build was not compiled. The `exec.rs` change uses only
existing cross-platform calls, and the `pty.rs` change is Unix-only.
Fixes#11433
Requested by @petrroll
---------
Co-authored-by: triagent <triagent@localhost>
Co-authored-by: Petr Houška <houskape@gmail.com>
Lists a directory tree like fd --hidden --exclude .git without following
symbolic links, with the ignore crate of fd and ripgrep for git's ignore
rules. Records travel as payload (kind byte, relative / path, NUL); the
listing stops at maxEntries, maxBytes or timeoutMs with truncated, and a
cancel stops it between entries. hello reports features: ["listFiles"]
so clients can tell older daemons apart.
The process group in front of the pty when it is not the program's own (a job its shell runs), with the leader's name on macOS and Linux. Exposed as RemoteTerminal.foreground().
The daemon gets pty, ptyWrite and ptyResize: raw terminal output as events, input in arrival order, SIGWINCH on resize, and the process group killed on cancel or a lost connection. openTerminal() is the client, localDaemon() the daemon for this machine. Unix only; Windows answers ENOTSUP.
- RemoteExecutionEnv: a Durable ExecutionEnv on another machine whose
results match NodeExecutionEnv there; Connection to the pi-env daemon
- packages/env/daemon: the Rust daemon for Linux, macOS, Android and
Windows, following Node and libuv semantics
- env.yml runs Durable's env tests, the daemon's checks and pi-env's
conformance and differential tests on Linux, macOS and Windows
