When a program started by the process operation exits on its own, the
daemon now ends its process group (Unix) or its own job object (Windows)
before reaping it, so a wrapper like npx does not leave its server
running. The pid leaves the tracked groups before it is reaped, so a
racing close cannot signal a reused pid.
The daemon's process operation runs a long-lived program with piped stdin, stdout and stderr, for programs that speak a protocol on stdin and stdout such as MCP servers. processWrite and processCloseStdin feed stdin in arrival order; output arrives as raw bytes per stream; cancelling or a lost connection kills the process group. hello reports it in features, and openProcess() is the client.
On Windows, readers and writers of child pipes now run their own overlapped operations and look at stop every 100 ms. The standard library opens these pipes for overlapped I/O, so CancelSynchronousIo never reached a blocked read: exec's readers stayed blocked while a descendant held a pipe open.
Spawning, reaping, argv parsing and target lookup are shared by exec, pty and process.
src/next is src (the main export), src/next-node is src/node with short subpaths (/http, /workers, /plugins,
/plugins/runtime, /app, /surfaces, /web, /react, /tui, /tui/launcher); the foundations get /json and /watch;
Chord v1 moves to src/old (/old, /old/node) until p2 runs on v2; tests and docs move likewise (test/, docs/,
test/old, docs/old). Durable and env import the foundations from their subpaths.
Replace build-time shell utilities with Node filesystem operations and retire the native sandbox example in favor of existing isolation options. Upgrade glob matching and worker-test dependencies and refresh the lockfiles.
After a rescan, the env daemon's file watcher sent every changed path in
a single `change` event. When a big batch of changes made that event
larger than the 16 MiB frame limit, `write_frame` rejected it. The
daemon's only output writer then shut down. The daemon process kept
running but sent no more replies or pings. Requests hung until the
client's 30 s silence limit dropped the connection and reconnected.
Now the watcher builds the event with `change_event`. If the serialized
path list would not fit in a frame (`MAX_PAYLOAD`), it sends the
existing `{ kind: "change", overflow: true }` event in its place. The
client already maps that to an `overflow` file change, so the wire
format does not change and the connection keeps working.
Testing:
- Added Rust unit tests in `watch.rs`. One checks that a small path list
is sent as `paths`. The other checks that a list larger than
`MAX_PAYLOAD` becomes `overflow`.
- `cargo fmt --check`, `cargo clippy --locked --all-targets -- -D
warnings` and `cargo test --locked` pass in `packages/env/daemon`.
- The `packages/env` vitest suite passes.
- Reproduced with the real daemon: I watched a directory and renamed in
a directory of 6,000 files with long paths, about 20 MB of path JSON.
Before the change, `fileInfo` requests timed out from the flush until
the client reconnected about 30 s later. After the change, the watcher
reports `overflow` right away and `fileInfo` keeps working. With 3,000
files, which fits in a frame, the paths are still reported as before.
Fixes#10516
Requested by @badlogic
Co-authored-by: triagent <triagent@localhost>
Connection.close() ends stdin and sends SIGTERM at once. The daemon did not
handle SIGTERM, so it died before its end-of-input kill_all, and running
commands were reparented to init. The daemon now blocks SIGTERM, SIGINT and
SIGHUP before starting any thread and waits for them on its own thread, which
kills every process group it started and exits. On Windows the job object
already kills commands with the daemon.
The pi-env daemon started an `exec`'s stdout and stderr readers and its
waiter with `thread::spawn`. `thread::spawn` panics when no thread can
be created, for example when a cgroup `TasksMax` or `RLIMIT_NPROC` limit
is reached. When that happened after the child was already running:
- only the request thread unwound, so the exec never got a reply;
- its control channel was dropped, so `cancel` and `kill` reached
nobody;
- the child kept running, even after the daemon exited.
`pty` started its reader, input writer and waiter threads the same way.
## Changes
- `exec.rs`: `spawn_reader` now uses `thread::Builder::spawn` and
returns an `io::Result`. If a reader or the waiter cannot start, `run`
sets `stop`, interrupts any readers already started, kills the process
group, reaps the child, removes it from `groups`, and returns
`spawn_error`.
- The waiter thread now gets the `Child` through a channel once it is
running. If the waiter cannot start, the child stays with the caller,
which can still reap it. Before, the failed thread's closure dropped the
child without waiting for it.
- `pty.rs`: the same handling for the reader, writer and waiter threads.
On failure the terminal is also removed from `terminals`.
## Testing
- Manual reproduction with a script that runs the daemon as `prlimit
--nproc=N -- pi-env`. It starts 30 `sleep 30` execs 200 ms apart, aborts
them all, then closes the connection. N ran from 37 to 70.
- Before: for most N, one or two execs stayed pending forever after
abort, and a `sleep 30` kept running after the daemon was closed.
- After: every exec settles, either `aborted` or `spawn_error`, and no
`sleep` is left running.
- `cargo build`, `cargo clippy --all-targets`, `cargo fmt --check` and
`cargo test` all pass. The `packages/env` vitest suite passes (78
passed), and `npm run check` passes.
- There is no automated regression test. A thread-creation failure
cannot be triggered reliably in a test: `RLIMIT_NPROC` counts all of the
user's tasks and does not apply to root, and the daemon has no
fault-injection hook.
- The Windows build was not compiled. The `exec.rs` change uses only
existing cross-platform calls, and the `pty.rs` change is Unix-only.
Fixes#11433
Requested by @petrroll
---------
Co-authored-by: triagent <triagent@localhost>
Co-authored-by: Petr Houška <houskape@gmail.com>
Chord's shared watch: listeners take a tagged call, { kind: "value", value, ops, retired } or
{ kind: "failed", error }; a recoverable failure keeps the watch open and the next value arrives as
[["r", value]]; WatchEnd is tagged by kind, and an aborted context ends a watch "aborted".
Durable: observers take the watch's shapes (advance(call, ctx), end(end)); onReport(report) with
report.kind and report.error, session_failed is storage_failed; subscribeCommits is observeCommits;
SessionEnd and ReportFrom are tagged by kind; FileChange is tagged and file watchers stop with
stop(ctx); CompactionFailure and CompactionStatus name their trigger (pi.live version 2 with a
migrate); the task_outdated report uses the scheduler's reasons; TaskInspection's ready is pending;
DocumentCommitChange is tagged. Stored formats are unchanged.
When the daemon exits before `hello`, the error has only the exit code.
For example, with `loginShell: true` and a profile that aborts on a
missing file:
```
pi-env exited with code 3 before it was ready
```
`sshConnection`'s probe and deploy steps already report ssh's stderr, so
this is about the launch itself: a login profile, a daemon that can't
execute, or a `Connection` created with its own command, which has no
probe or deploy step, so auth and network errors also end up here.
This keeps a short stderr tail until `hello` and adds it to startup
errors:
```
pi-env exited with code 3 before it was ready
remote stderr:
profile: /opt/toolchain/env.sh: No such file or directory
```
The tail is capped at 20 lines / 2000 chars, with terminal escapes and
control characters stripped. Ready sessions are unchanged.
I hit it while testing `loginShell` against an sshd in Docker. Tests are
in `test/connection.test.ts`, and `npm run check` and `./test.sh` pass.
Durable publishes a stored commit inside Storage.commit's adopt: the
telemetry events, its own observers (document watches, conversation
views, task graph, scheduler, waits), and a dispatch of host
subscribeCommits listeners one microtask later, before every wait and
abort the commit triggers. A commit the storage rejects after adoption is
published before the Session fails; a throwing telemetry adapter is
reported as host_error "telemetry" instead of rejecting a durable commit.
Chord's watch listeners take one change object, { value, ops, retired }
({ value } for native watches); retirement is a flag, not an end.
Durable's document watches follow their address, and a task document's
watch ends after its retirement call. New synchronous observers for
adapters: observeDoc, observeConversation, observeTaskGraph; the watches
are built on them. The env file watcher's change type is renamed
FileChange.
The fake tool API lacked retainedOutput, so every bash case threw on both sides
and the test only compared the error text. Its stdout+stderr case also depended
on the order of two concurrently read pipes and failed intermittently in CI.
The API now provides retainedOutput, each bash case writes to one stream, and
bash cases must return a result.
- Give the listing cancel test 30 s: creating 10,000 files takes over 5 s
on Windows CI.
- Accept an x64 daemon on arm64 macOS: the darwin-x64 release daemon runs
through Rosetta in CI.
RemoteExecutionEnv.listFiles resolves the path and lists through the
daemon; a daemon without the listFiles feature yields not_supported.
LocalExecutionEnv is NodeExecutionEnv plus listFiles through a daemon on
the same machine, so local and SSH sessions list with the same code.
nodePlatform maps hello's Rust names to Node's (windows -> win32,
macos -> darwin, x86_64 -> x64, aarch64 -> arm64).
Lists a directory tree like fd --hidden --exclude .git without following
symbolic links, with the ignore crate of fd and ripgrep for git's ignore
rules. Records travel as payload (kind byte, relative / path, NUL); the
listing stops at maxEntries, maxBytes or timeoutMs with truncated, and a
cancel stops it between entries. hello reports features: ["listFiles"]
so clients can tell older daemons apart.
The process group in front of the pty when it is not the program's own (a job its shell runs), with the leader's name on macOS and Linux. Exposed as RemoteTerminal.foreground().
The daemon gets pty, ptyWrite and ptyResize: raw terminal output as events, input in arrival order, SIGWINCH on resize, and the process group killed on cancel or a lost connection. openTerminal() is the client, localDaemon() the daemon for this machine. Unix only; Windows answers ENOTSUP.
- connectSsh detects the remote system, deploys the packaged daemon
verified by SHA-256 before it runs, and connects over ssh with strict
host keys under a fixed alias; scanHostKey and acceptHostKey
- Windows detection and deployment through PowerShell
- release builds the daemon for every supported remote system and
ships them in @earendil-works/pi-env's bin/
- end-to-end tests against a disposable sshd
- RemoteExecutionEnv: a Durable ExecutionEnv on another machine whose
results match NodeExecutionEnv there; Connection to the pi-env daemon
- packages/env/daemon: the Rust daemon for Linux, macOS, Android and
Windows, following Node and libuv semantics
- env.yml runs Durable's env tests, the daemon's checks and pi-env's
conformance and differential tests on Linux, macOS and Windows
