List commands silently truncate at a default limit and never surface total-count or pagination #107

New issue

Closed

opened 2026-06-11 00:01:28 +00:00 by stephen · 2 comments

stephen commented

2026-06-11 00:01:28 +00:00

Owner

Observation

fj's list commands silently cap results at a default limit and never surface the total count or that more pages exist, so a script cannot tell a complete result set from a truncated one.

fj repo list defaults to 30 (src/cli/repo.rs:309-347 handler; --limit default 30), yet the server reports the real total. Reproduced against a host with more than 30 repos:

$ fj repo list --json | jq length
30
$ fj api '/repos/rasterstate/fj/issues?limit=2&state=all' --include | grep -iE 'x-total-count|^link:'
link: <.../issues?limit=2&page=2&state=all>; rel="next", <.../issues?limit=2&page=52&state=all>; rel="last"
x-total-count: 104

The pagination metadata is parsed and then thrown away. Every field of Page<T> that carries it is dead code (src/client/pagination.rs:5-19):

pub struct Page<T> {
    pub items: Vec<T>,
    #[allow(dead_code)]
    pub next: Option<String>,
    #[allow(dead_code)]
    pub prev: Option<String>,
    #[allow(dead_code)]
    pub last: Option<String>,
    #[allow(dead_code)]
    pub first: Option<String>,
    #[allow(dead_code)]
    pub total: Option<u64>,
}

from_headers faithfully reads X-Total-Count and Link: rel=next (src/client/pagination.rs:38-60), but the list handlers only ever consume page.items. In repo list the JSON path is serde_json::to_value(&page.items) (src/cli/repo.rs:322-324); total and next are never read by any caller, hence the #[allow(dead_code)] on all of them.

The default limits compound the trap: pr list, issue list, repo list, and release list all default to 30 (src/cli/pr.rs:77, src/cli/issue.rs:74, src/cli/repo.rs, src/cli/release.rs). And the only pagination escape hatch, --paginate (follow Link: rel=next and concatenate), exists solely on fj api (fj api --help), not on any purpose-built list command.

$ fj api --help    | grep -i paginate
      --paginate    Follow `Link: rel=next` and concatenate all pages ...
$ fj repo list --help | grep -i paginate
(nothing)

Why it matters

Silent truncation is the most dangerous failure mode for automation because it looks like success. An agent that runs fj issue list --json to "get all open issues" receives 30, exits 0, and acts on a partial set with no signal that 74 more exist. There is no total in the JSON envelope, no next cursor, no warning on stderr, and no --paginate to force completeness short of guessing a large --limit. gh solves exactly this: gh issue list --json plus an explicit --limit, and its api --paginate is mirrored by list-level paging. For a tool whose headline pitch is JSON "so scripts and AI agents" can consume it, returning a silently-capped slice undercuts the core promise.

Possible directions (sketches)

(sketch) Add --paginate to the list commands (reusing the get_all / Link: rel=next machinery already behind fj api --paginate and the opts.limit > 50 branch in src/api/pull_core.rs:117-131), so a script can opt into completeness.
(sketch) In --json mode, emit an envelope ({ "total_count": N, "items": [...] }) or at least surface X-Total-Count, so a consumer can compare returned vs. total and decide whether to page. The total field is already parsed; it just needs threading through.
(sketch) When a human-facing list is truncated (returned == limit < total), print a dim footer like Showing 30 of 104. Use --limit or --paginate for more., matching gh's behavior.
(sketch) At minimum, document the default --limit value in each command's help and note that results may be capped.

Confidence

High. Truncation reproduced (fj repo list --json | jq length = 30 with x-total-count: 104 on the host), the discard is structural (#[allow(dead_code)] on every metadata field of Page<T>, src/client/pagination.rs:7-18, and handlers reading only page.items, e.g. src/cli/repo.rs:322), and the --paginate/list asymmetry is verified from --help. The envelope-vs-footer choice is a design call.

## Observation `fj`'s `list` commands silently cap results at a default limit and never surface the total count or that more pages exist, so a script cannot tell a complete result set from a truncated one. `fj repo list` defaults to 30 (`src/cli/repo.rs:309-347` handler; `--limit` default 30), yet the server reports the real total. Reproduced against a host with more than 30 repos: ``` $ fj repo list --json | jq length 30 $ fj api '/repos/rasterstate/fj/issues?limit=2&state=all' --include | grep -iE 'x-total-count|^link:' link: <.../issues?limit=2&page=2&state=all>; rel="next", <.../issues?limit=2&page=52&state=all>; rel="last" x-total-count: 104 ``` The pagination metadata is parsed and then thrown away. Every field of `Page<T>` that carries it is dead code (`src/client/pagination.rs:5-19`): ```rust pub struct Page<T> { pub items: Vec<T>, #[allow(dead_code)] pub next: Option<String>, #[allow(dead_code)] pub prev: Option<String>, #[allow(dead_code)] pub last: Option<String>, #[allow(dead_code)] pub first: Option<String>, #[allow(dead_code)] pub total: Option<u64>, } ``` `from_headers` faithfully reads `X-Total-Count` and `Link: rel=next` (`src/client/pagination.rs:38-60`), but the list handlers only ever consume `page.items`. In `repo list` the JSON path is `serde_json::to_value(&page.items)` (`src/cli/repo.rs:322-324`); `total` and `next` are never read by any caller, hence the `#[allow(dead_code)]` on all of them. The default limits compound the trap: `pr list`, `issue list`, `repo list`, and `release list` all default to 30 (`src/cli/pr.rs:77`, `src/cli/issue.rs:74`, `src/cli/repo.rs`, `src/cli/release.rs`). And the only pagination escape hatch, `--paginate` (follow `Link: rel=next` and concatenate), exists solely on `fj api` (`fj api --help`), not on any purpose-built `list` command. ``` $ fj api --help | grep -i paginate --paginate Follow `Link: rel=next` and concatenate all pages ... $ fj repo list --help | grep -i paginate (nothing) ``` ## Why it matters Silent truncation is the most dangerous failure mode for automation because it looks like success. An agent that runs `fj issue list --json` to "get all open issues" receives 30, exits 0, and acts on a partial set with no signal that 74 more exist. There is no total in the JSON envelope, no `next` cursor, no warning on stderr, and no `--paginate` to force completeness short of guessing a large `--limit`. `gh` solves exactly this: `gh issue list --json` plus an explicit `--limit`, and its `api --paginate` is mirrored by list-level paging. For a tool whose headline pitch is JSON "so scripts and AI agents" can consume it, returning a silently-capped slice undercuts the core promise. ## Possible directions (sketches) - *(sketch)* Add `--paginate` to the `list` commands (reusing the `get_all` / `Link: rel=next` machinery already behind `fj api --paginate` and the `opts.limit > 50` branch in `src/api/pull_core.rs:117-131`), so a script can opt into completeness. - *(sketch)* In `--json` mode, emit an envelope (`{ "total_count": N, "items": [...] }`) or at least surface `X-Total-Count`, so a consumer can compare returned vs. total and decide whether to page. The `total` field is already parsed; it just needs threading through. - *(sketch)* When a human-facing `list` is truncated (returned == limit < total), print a dim footer like `Showing 30 of 104. Use --limit or --paginate for more.`, matching gh's behavior. - *(sketch)* At minimum, document the default `--limit` value in each command's help and note that results may be capped. ## Confidence High. Truncation reproduced (`fj repo list --json | jq length` = 30 with `x-total-count: 104` on the host), the discard is structural (`#[allow(dead_code)]` on every metadata field of `Page<T>`, `src/client/pagination.rs:7-18`, and handlers reading only `page.items`, e.g. `src/cli/repo.rs:322`), and the `--paginate`/list asymmetry is verified from `--help`. The envelope-vs-footer choice is a design call.

stephen added the

opportunity

converted

labels

2026-06-11 00:01:44 +00:00

stephen commented

2026-06-11 00:09:22 +00:00

Author

Owner

Converted to backlog item rasterstate/fj#111 (p1, size M).

Tracked there with task / priority / reason / acceptance / dependencies. Keeping this open with the converted label as the originating opportunity.

Converted to backlog item `rasterstate/fj#111` (p1, size M). Tracked there with task / priority / reason / acceptance / dependencies. Keeping this open with the `converted` label as the originating opportunity.

stephen commented

2026-06-11 00:46:10 +00:00

Author

Owner

Derived backlog item rasterstate/fj#111 is merged. Closing this opportunity per the issue state machine.

stephen closed this issue

2026-06-11 00:46:10 +00:00

stephen referenced this issue

2026-06-11 00:51:51 +00:00

fj run list cannot filter by status/branch/event/workflow, forcing scripts to grep JSON #128

stephen referenced this issue

2026-06-11 00:57:41 +00:00