1 commit

Author	SHA1	Message	Date
Petr Houška	3a7bb6f48c	fix(coding-agent): stream the read tool instead of loading whole files (#11750) ... ## Problem `read` loads the whole file before it applies `offset`/`limit`: `ops.readFile` -> `toString("utf-8")` -> `split("\n")`. A one-line read still holds the whole file three times: as a Buffer, as a string and as an array of lines. Measured on main with a 200 MB log (2.36M lines): - `offset: 1, limit: 1` peaks at 638 MB RSS. With `--max-old-space-size=256` it crashes in `Runtime_StringSplit` (OOM). - Files over about 512 MB fail with `Cannot create a string longer than 0x1fffffe8 characters`. Files over 2 GB fail with `ERR_FS_FILE_TOO_LARGE`. - Aborting only rejects the promise; the read keeps running in the background. ## Solution - `ReadOperations` gets a new optional `createReadStream(absolutePath, { signal })`, which returns an `AsyncIterable<Uint8Array>` (a Node `Readable` works). The default local operations open the file once, `fstat` the handle, and stream from it. Like `fs.readFile`, they stop a regular file at its size when opened: text appended during the read is not counted, and a file that keeps growing still finishes. A file that reports size 0 (such as `/proc` files) and other file types (FIFOs, devices) are read until EOF, as `fs.readFile` reads them. No existing member changed. - The text path decodes the chunks with a streaming UTF-8 `TextDecoder` (non-fatal, `ignoreBOM: true`). That matches `Buffer.toString("utf-8")`, including invalid bytes and a UTF-8 byte order mark kept as U+FEFF. - It counts every line but stores only the selected lines `truncateHead` can show. Storing stops once the stored lines are past `DEFAULT_MAX_LINES` or `DEFAULT_MAX_BYTES`, plus one more line, and the existing `truncateHead` runs on the stored lines. `totalLines`/`totalBytes` are then patched from the counts, as grep and find do on main. - The extra line is needed because `truncateHead` drops a trailing empty line. Example: 2001 short lines, then an empty line, then more text. If storing stopped at the empty line, the stored text would end in `\n`. `truncateHead` would count 2000 lines and report "not truncated", but the full text is truncated by lines. - A line over 50KB is never stored whole; only its size is tracked. A first line of 1 GB therefore still gives today's `[Line 1 is ..., exceeds 50.0KB limit ...]` notice. - Abort: the signal is passed to the stream, and the loop also checks it between chunks. Leaving the loop closes the stream. - The whole-file path stays unchanged for custom operations without `createReadStream`, and for fractional `offset`/`limit` or negative `limit`. Array slicing defines what those odd ranges select; for example, `limit: -1` selects all lines but the last. - Images are unchanged. Output is the same as main's, byte for byte, including line counts, notices, details (including key order) and errors. A file appended to during the read gives the same output too: both read a regular file up to its size when opened. Out of scope: the trailing-newline off-by-one in line counts from #7329 is kept as is. ## Testing - `test/read-streaming.test.ts` keeps a copy of the old text algorithm as the reference. It compares the streaming path against it on text, details and error messages, with chunks of 1 and 4096 bytes and offsets and limits around the boundaries. The cases are: empty file, trailing newline or not, CRLF, multi-byte characters and a UTF-8 BOM, 2001 lines, and a first line over 50KB. There are also edge cases where the line or byte cap falls exactly on an empty line, and the 50KB + trailing newline case that `truncateHead` reports as `truncatedBy: "lines"`. Further tests: - details key order - multi-byte characters split across chunks, and invalid UTF-8 - the `readFile` fallback, including fractional and negative ranges - local files through the default stream, including EISDIR - abort: the stream is closed and no more chunks are pulled - a local file appended to mid-read (a decode hook in the test appends after the first chunk): `limit: 1` still says `[1 more lines ...]`. This test fails with an unbounded `fs.createReadStream`. - bounded memory: a child process with `--max-old-space-size=64` reads a 200 MB single line, generated in memory as 64 KB chunks from a custom `createReadStream`, and must print the size notice. It takes about 0.8 s and uses 17 MB of heap. If over-long lines are stored whole, it runs out of memory. 64 KB chunks matter here: V8 keeps decoded strings of 1 MB outside the heap limit, so with 1 MB chunks the test would not catch that regression. - With main's `read.ts`, 5 of the 8 tests fail: matrix, key order, chunked decoding, bounded memory and abort. The fallback, local-file and append tests pass on main because they check behavior that did not change. Removing the one-line-past-the-cap rule, or the trailing-empty-line adjustment of `totalLines`, makes the matrix fail. - `test/tools.test.ts`, `read-line-range-display`, `tool-execution-component`, `block-images` and `image-resize-callers` pass. `npm run check` is clean. Measured with an ad-hoc script on a 200 MB log. Baseline RSS after loading modules is about 121 MB. | Case | main | this PR | | --- | --- | --- | | offset 1, limit 1 | 1028 ms, 638 MB peak RSS | 478 ms, 131 MB | | offset 2000000, limit 10 | 926 ms, 637 MB | 486 ms, 132 MB | | no offset/limit (counts all lines) | 1561 ms, 1010 MB | 524 ms, 132 MB | | offset 1, limit 1, 256 MB heap | OOM crash | 131 MB | | 1.1 GB single-line file | string length error | 2.2 s, 130 MB, line-size notice | Remaining difference: custom operations that only implement `readFile` keep the whole-file memory use, and their reads continue after an abort, as before. This PR is AI-generated.	2026-11-04 23:14:52 +01:00