3a7bb6f48c
## Problem
`read` loads the whole file before it applies `offset`/`limit`:
`ops.readFile` -> `toString("utf-8")` -> `split("\n")`. A one-line read
still holds the whole file three times: as a Buffer, as a string and as
an array of lines.
Measured on main with a 200 MB log (2.36M lines):
- `offset: 1, limit: 1` peaks at 638 MB RSS. With
`--max-old-space-size=256` it crashes in `Runtime_StringSplit` (OOM).
- Files over about 512 MB fail with `Cannot create a string longer than
0x1fffffe8 characters`. Files over 2 GB fail with
`ERR_FS_FILE_TOO_LARGE`.
- Aborting only rejects the promise; the read keeps running in the
background.
## Solution
- `ReadOperations` gets a new optional `createReadStream(absolutePath, {
signal })`, which returns an `AsyncIterable<Uint8Array>` (a Node
`Readable` works). The default local operations open the file once,
`fstat` the handle, and stream from it. Like `fs.readFile`, they stop a
regular file at its size when opened: text appended during the read is
not counted, and a file that keeps growing still finishes. A file that
reports size 0 (such as `/proc` files) and other file types (FIFOs,
devices) are read until EOF, as `fs.readFile` reads them. No existing
member changed.
- The text path decodes the chunks with a streaming UTF-8 `TextDecoder`
(non-fatal, `ignoreBOM: true`). That matches `Buffer.toString("utf-8")`,
including invalid bytes and a UTF-8 byte order mark kept as U+FEFF.
- It counts every line but stores only the selected lines `truncateHead`
can show. Storing stops once the stored lines are past
`DEFAULT_MAX_LINES` or `DEFAULT_MAX_BYTES`, plus one more line, and the
existing `truncateHead` runs on the stored lines.
`totalLines`/`totalBytes` are then patched from the counts, as grep and
find do on main.
- The extra line is needed because `truncateHead` drops a trailing empty
line. Example: 2001 short lines, then an empty line, then more text. If
storing stopped at the empty line, the stored text would end in `\n`.
`truncateHead` would count 2000 lines and report "not truncated", but
the full text is truncated by lines.
- A line over 50KB is never stored whole; only its size is tracked. A
first line of 1 GB therefore still gives today's `[Line 1 is ...,
exceeds 50.0KB limit ...]` notice.
- Abort: the signal is passed to the stream, and the loop also checks it
between chunks. Leaving the loop closes the stream.
- The whole-file path stays unchanged for custom operations without
`createReadStream`, and for fractional `offset`/`limit` or negative
`limit`. Array slicing defines what those odd ranges select; for
example, `limit: -1` selects all lines but the last.
- Images are unchanged.
Output is the same as main's, byte for byte, including line counts,
notices, details (including key order) and errors. A file appended to
during the read gives the same output too: both read a regular file up
to its size when opened. Out of scope: the trailing-newline off-by-one
in line counts from #7329 is kept as is.
## Testing
- `test/read-streaming.test.ts` keeps a copy of the old text algorithm
as the reference. It compares the streaming path against it on text,
details and error messages, with chunks of 1 and 4096 bytes and offsets
and limits around the boundaries. The cases are: empty file, trailing
newline or not, CRLF, multi-byte characters and a UTF-8 BOM, 2001 lines,
and a first line over 50KB. There are also edge cases where the line or
byte cap falls exactly on an empty line, and the 50KB + trailing newline
case that `truncateHead` reports as `truncatedBy: "lines"`. Further
tests:
- details key order
- multi-byte characters split across chunks, and invalid UTF-8
- the `readFile` fallback, including fractional and negative ranges
- local files through the default stream, including EISDIR
- abort: the stream is closed and no more chunks are pulled
- a local file appended to mid-read (a decode hook in the test appends
after the first chunk): `limit: 1` still says `[1 more lines ...]`. This
test fails with an unbounded `fs.createReadStream`.
- bounded memory: a child process with `--max-old-space-size=64` reads a
200 MB single line, generated in memory as 64 KB chunks from a custom
`createReadStream`, and must print the size notice. It takes about 0.8 s
and uses 17 MB of heap. If over-long lines are stored whole, it runs out
of memory. 64 KB chunks matter here: V8 keeps decoded strings of 1 MB
outside the heap limit, so with 1 MB chunks the test would not catch
that regression.
- With main's `read.ts`, 5 of the 8 tests fail: matrix, key order,
chunked decoding, bounded memory and abort. The fallback, local-file and
append tests pass on main because they check behavior that did not
change. Removing the one-line-past-the-cap rule, or the
trailing-empty-line adjustment of `totalLines`, makes the matrix fail.
- `test/tools.test.ts`, `read-line-range-display`,
`tool-execution-component`, `block-images` and `image-resize-callers`
pass. `npm run check` is clean.
Measured with an ad-hoc script on a 200 MB log. Baseline RSS after
loading modules is about 121 MB.
| Case | main | this PR |
| --- | --- | --- |
| offset 1, limit 1 | 1028 ms, 638 MB peak RSS | 478 ms, 131 MB |
| offset 2000000, limit 10 | 926 ms, 637 MB | 486 ms, 132 MB |
| no offset/limit (counts all lines) | 1561 ms, 1010 MB | 524 ms, 132 MB
|
| offset 1, limit 1, 256 MB heap | OOM crash | 131 MB |
| 1.1 GB single-line file | string length error | 2.2 s, 130 MB,
line-size notice |
Remaining difference: custom operations that only implement `readFile`
keep the whole-file memory use, and their reads continue after an abort,
as before.
This PR is AI-generated.
|
||
|---|---|---|
| .github | ||
| .husky | ||
| .pi | ||
| apps/p2 | ||
| nix | ||
| packages | ||
| scripts | ||
| .gitattributes | ||
| .gitignore | ||
| .npmrc | ||
| AGENTS.md | ||
| biome.json | ||
| CONTRIBUTING.md | ||
| flake.lock | ||
| flake.nix | ||
| LICENSE | ||
| package-lock.json | ||
| package.json | ||
| pi-test.bat | ||
| pi-test.ps1 | ||
| pi-test.sh | ||
| README.md | ||
| SECURITY.md | ||
| test.sh | ||
| tsconfig.base.json | ||
| tsconfig.json | ||
| vitest.base.ts | ||
New issues and PRs from new contributors are auto-closed by default. Maintainers review auto-closed issues daily. See CONTRIBUTING.md.
Pi
Pi is a minimal, extensible agent harness that you can make your own.
Adapt Pi to your workflows, not the other way around. Customize Pi with extensions, skills, prompt templates, and themes. Bundle them as Pi packages and share via npm or git.
Pi ships with powerful defaults but skips features like sub-agents and plan mode. Ask Pi to build what you want, or install a package that does it your way.
Use Pi interactively, automate it in print or JSON mode, control it over RPC, or build apps with the Pi TypeScript SDK. See OpenClaw for a real-world integration.
Getting started
Install the command-line interface:
curl -fsSL https://pi.dev/install.sh | sh
On Windows:
powershell -c "irm https://pi.dev/install.ps1 | iex"
The installer pins all dependencies and updates Pi with pi update. Alternatively, install directly with npm, which does not pin transitive dependencies:
npm install -g --ignore-scripts @earendil-works/pi-coding-agent
Pi requires Node.js 22.19 or newer. The macOS, Linux, and Windows installers can install it if needed. Pi does not require dependency lifecycle scripts for a normal npm installation.
Start Pi in the directory where you want it to work:
cd /path/to/project
pi
For a built-in AI provider, run /login inside Pi to connect a subscription or API key. Then give Pi a task.
See the documentation for full setup and usage instructions, or visit pi.dev for demos.
Run with Nix
nix run github:earendil-works/pi/stable
stable points at the latest release. Install it with nix profile add github:earendil-works/pi/stable and update with nix profile upgrade pi. Use a release tag such as github:earendil-works/pi/v1.0.0 to pin a version, or github:earendil-works/pi for unreleased changes on main. Nix builds Pi from source.
Supports ARM64 and x86-64 on Linux and macOS. Use nix build . or nix run . to build or run your checkout.
Nix builds are offline, so the bundled model data comes from a pi.dev model catalog revision pinned in nix/model-catalog.json. At runtime, Pi still overlays newer catalog data from pi.dev as usual. The Nix workflow replaces the pin on main when it no longer matches the checkout, for example after a provider is added or gains a new model type. To refresh it by hand:
npm run update:model-catalog-pin
Packages
This monorepo contains the Pi CLI and its supporting libraries.
| Package | Description |
|---|---|
| @earendil-works/chord | Object runtime for apps across processes: contracts, documents, a store, links, plugins with hot reload, HTTP, workers, web and terminal clients |
| @earendil-works/pi-telemetry | Vendor-neutral telemetry contracts, reference adapter, conformance tests, and typed schemas |
| @earendil-works/pi-ai | Unified multi-provider LLM API (OpenAI, Anthropic, Google, etc.) |
| @earendil-works/pi-durable | Durable conversation, task, and document runtime |
| @earendil-works/pi-agent-core | Agent runtime with tool calling and state management |
| @earendil-works/pi-coding-agent | Interactive coding agent CLI |
| @earendil-works/pi-tui | Terminal UI library with differential rendering |
For Slack/chat automation and workflows see earendil-works/pi-chat.
Permissions & Containerization
Pi does not include a built-in permission system for restricting filesystem, process, network, or credential access. By default, it runs with the permissions of the user and process that launched it.
If you need stronger boundaries, containerize or sandbox Pi. See packages/coding-agent/docs/containerization.md for three patterns:
- Gondolin extension: keep
piand provider auth on the host while routing built-in tools and!commands into a local Linux micro-VM. - Plain Docker: run the whole
piprocess in a local container for simple isolation. - OpenShell: run the whole
piprocess in a policy-controlled sandbox.
Contributing
See CONTRIBUTING.md for contribution guidelines and AGENTS.md for project-specific rules (for both humans and agents). Longer term plans for Pi can also be found in RFCs.
Development
npm install --ignore-scripts # Install all dependencies without running lifecycle scripts
npm run build # Refresh model data, then build all packages
npm run build:offline # Rebuild using existing model data without network access
npm run check # Lint, format, and type check
./test.sh # Run tests (skips LLM-dependent tests without API keys)
./pi-test.sh # Run pi from sources (can be run from any directory)
Using local packages outside the monorepo
Build every public package into one coherent local artifact set:
npm run pack:packages -- --out .artifacts/pi-packages
This refreshes model data before building pi-ai. To avoid network access when
model data is already hydrated, pass --offline-model-data.
Then configure an external project to consume one package and resolve all of its Pi dependencies from the same artifact set. npm is the default:
node scripts/use-local-packages.mjs \
--manifest .artifacts/pi-packages/manifest.json \
--consumer ../my-project \
--package @earendil-works/pi-durable \
--package @earendil-works/pi-agent-core
cd ../my-project
npm install --ignore-scripts
For a pnpm project, point --consumer at the workspace root:
node scripts/use-local-packages.mjs \
--manifest .artifacts/pi-packages/manifest.json \
--consumer ../my-project \
--package @earendil-works/pi-agent-core \
--package-manager pnpm
cd ../my-project
pnpm install --ignore-scripts
Repeat --package for each direct dependency. The command updates the
consumer's package.json with content-addressed local file: references. It
writes transitive overrides to package.json for npm or pnpm-workspace.yaml
for pnpm. Keep the artifact directory available while installing or updating
the consumer. Re-run both commands after changing Pi source.
Building standalone binaries from release source
GitHub releases include a versioned source archive covered by the release's SHA256SUMS file. Extract it and run the same build script used for the official standalone binaries:
VERSION="<release-version>"
tar -xzf "pi-${VERSION}-source.tar.gz"
cd "pi-${VERSION}"
./scripts/build-binaries.sh --offline-model-data --platform linux-x64 --out "$PWD/out"
The archive includes release model data and native prebuilds. --offline-model-data uses that model data without refreshing provider catalogs. The script installs dependencies and builds the executable with its runtime assets; pass --skip-install if dependencies are already provided.
Supply-chain hardening
We treat npm dependency changes as reviewed code changes.
- Direct external dependencies are pinned to exact versions. Internal workspace packages remain version-ranged.
.npmrcsetssave-exact=trueandmin-release-age=2to avoid same-day dependency releases during npm resolution.package-lock.jsonis the dependency ground truth. Pre-commit blocks accidental lockfile commits unlessPI_ALLOW_LOCKFILE_CHANGE=1is set.npm run checkverifies pinned direct deps, native TypeScript import compatibility, and the generated coding-agent install lock.- The pi.dev installer installs from
packages/coding-agent/install-lock/, generated from the root lockfile, to pin transitive deps. The npm package does not pin transitive deps. - Local release smoke tests and npm publication use the same tarball packer; npm publishes the validated tarballs rather than repacking workspace directories.
- Local release installs, documented npm installs, and
pi update --selfuse--ignore-scriptswhere supported. - CI installs with
npm ci --ignore-scripts, and a scheduled GitHub workflow runsnpm audit --omit=devplusnpm audit signatures --omit=dev. - Install lock generation has an explicit allowlist for dependency lifecycle scripts; new lifecycle-script deps fail checks until reviewed.
Share your OSS coding agent sessions
If you use Pi or other coding agents for open source work, please share your sessions.
Public OSS session data helps improve coding agents with real-world tasks, tool use, failures, and fixes instead of toy benchmarks.
For the full explanation, see this post on X.
To publish sessions, use badlogic/pi-share-hf. Read its README.md for setup instructions. All you need is a Hugging Face account, the Hugging Face CLI, and pi-share-hf.
You can also watch this video, where I show how I publish my pi-mono sessions.
I regularly publish my own pi-mono work sessions here:
License
MIT
