# pdf-finder Agent Guide Base URL: `https://pdf-finder.aisloppy.com` ## Purpose Find PDF files anywhere on the public web from an arbitrary text query, and report whether each result URL is actually a reachable PDF right now. Discovery uses the Google Custom Search JSON API restricted to `fileType=pdf`. Every returned link is then probed with a ranged `GET` (first 4 KB) to confirm the `%PDF` magic bytes, capture the byte size, and separate live files from paywalls, 403s, and dead links. ## Endpoints ### `GET /api/search` Search the web for PDFs. Public, no auth. | Param | Type | Default | Meaning | |---|---|---|---| | `q` | string | required | Query. Google operators work (`site:`, `intitle:`, `filetype` is already forced to pdf). Max 500 chars. | | `page` | int | `1` | 10 results per page. Pages 1–10 only (Google returns at most 100 results per query). | | `verify` | `0`/`1` | `1` | Probe each result link. Set `0` for a faster, unverified list. | | `refresh` | `0`/`1` | `0` | Bypass the 24h search cache and spend a fresh CSE query. | ```json { "query": "attention is all you need", "page": 1, "total_estimate": 94900000, "has_next": true, "from_cache": false, "cached_at": 1786128165.9, "verified": true, "cse_calls_today": 2, "results": [ { "url": "https://arxiv.org/pdf/1706.03762", "title": "arXiv:1706.03762v7 [cs.CL] 2 Aug 2023", "snippet": "Attention Is All You Need. Ashish Vaswani...", "host": "arxiv.org", "mime": "application/pdf", "status": "pdf", "http_status": 206, "bytes": 2215244, "pages": null, "via": "direct", "detail": "Verified PDF", "checked_at": 1786128165.9 } ] } ``` `status` values: | Value | Meaning | |---|---| | `pdf` | Confirmed PDF (`%PDF` magic bytes, or `application/pdf` with a real body) | | `not_pdf` | Reachable, but the server returned HTML or another type — landing page or paywall | | `blocked` | Host refused an automated request: HTTP 401/403/429, an empty body, or a bot-shield interstitial (AWS WAF, Cloudflare, Imperva, DataDome, PerimeterX). **The file usually still opens in a browser** — `detail` names the shield. | | `missing` | HTTP 404/410 | | `error` | DNS failure, timeout, TLS error, or 5xx; see `detail` | | `unknown` | 2xx with no usable content type | | `unchecked` | `verify=0` was requested | ### `GET /api/verify?url=` Re-probe one URL, ignoring the 7-day liveness cache. Returns `{status, http_status, content_type, bytes, pages, via, detail, checked_at}`. Add `escalate=1` to retry a refused link through Text Handling's escalating fetch — a residential exit, then a real browser that executes the JavaScript challenge. `via` names the rung that produced the bytes (`direct`, `residential`, `browser`), so a link that only works through a proxy is never presented as ordinarily reachable. Escalation costs seconds and proxy traffic, so it is a per-link action and never runs across a results page. ### `GET /api/inspect?url=` Deep-read one PDF and return its own metadata — use this to tell a whole book from an article about it. Add `refresh=1` to bypass the 30-day cache. ```json { "url": "https://arxiv.org/pdf/1706.03762", "pages": 15, "bytes": 2215244, "encrypted": 0, "title": null, "author": null, "producer": "pdfTeX-1.40.25", "created": "D:20240410211143Z", "via": "direct", "from_cache": false } ``` Size alone is available for free on every result (from `Content-Range` on the verification probe — nothing is downloaded), and page count comes free from *linearized* PDFs, which carry `/N ` in their first bytes. Most PDFs are not linearized: their page tree only resolves through the cross-reference table at the end of the file, so `/api/inspect` fetches the whole file and parses it — through Text Handling, which owns the escalating fetch path and the shared PDF parser. It handles **one URL per call** and never runs as part of a results page. ### `POST /api/inspect-batch` Body: `{"urls": [...]}` — deep-read a whole results page so page count becomes a comparable column. Bounded to one page of URLs, a 90s deadline, and a total download budget. ```json {"results": {"": {"pages": 5, "bytes": 243429, "title": "...", "...": "..."}}, "skipped": {"": "Source URL returned 404: ..."}, "requested": 10} ``` Everything it could not read appears in `skipped` with the reason. A bounded batch must never read as a complete one. A cheap page count is not available: a tail range request finds sub-tree `/Count` values that look like the answer and are not (a 15-page paper reports 11), so this reads each file once rather than reporting a wrong number. ### `POST /api/rank` Body: `{"query": "...", "results": [{url, title, host, snippet, pages, bytes, inspected?}]}` Judges which candidates are likely the document the query was after. Search rank answers "does this page match these words"; it cannot separate a work from writing about the work, and the wanted document is often neither the longest nor the best-linked result. ```json {"rankings": {"": {"likelihood": 30, "kind": "excerpt or sample", "reason": "Title and plot snippet match, but 16 pages and an embedded author of Jalene Laski suggest a sample, not the full novel."}}, "model": "...", "judged": 10, "requested": 10, "from_cache": false} ``` `kind` is one of `the work itself`, `excerpt or sample`, `about the work`, `mentions the work`, `unrelated`. Every judgement carries its `reason`, and the UI shows it — a ranking that cannot be argued with is not usable evidence. Runs through BrightWrapper with a 90s deadline and per-poll timeouts; cached 7 days per (query, result set). Cold runs take ~30s. A terminal failure returns an error, never an empty ranking that would read as "nothing matched". ### `GET /api/download?url=` Streams a remote PDF through this origin as an `attachment`. Exists because most PDF hosts send neither CORS nor `Content-Disposition` headers. The retrieval is Text Handling's: redirect and address validation, the 250 MiB cap, and the escalating fetch path all live there. The response carries `X-Source-Fetch-Path` naming the rung used. A host that refuses every rung fails with its own status and message — a bot-shield page is never handed back as a `.pdf`. ### `GET /api/availability?q=` Where the work itself can be obtained, when the web index does not have it — which for anything in copyright is the normal case. ```json {"entries": [{"source": "Open Library", "title": "The Peripheral", "author": "William Gibson", "year": 2014, "access": "restricted", "match": 1.0, "note": "Digital copy exists, restricted to print-disabled readers", "url": "https://archive.org/details/...", "download_url": null}], "considered": 6, "failed": {}, "from_cache": false} ``` Sources: Open Library (editions and Internet Archive lending status), Project Gutenberg (public-domain full texts), OpenAlex (scholarly works and their legal open-access copies). Queried in parallel with 8s timeouts and cached 24 hours; a source that fails is named in `failed`, never dropped. `access` is what the reader can actually do: `download` (public domain or open access), `borrow` (library loan), `restricted` (digital copy exists, access limited), `record` (catalogue entry only). Entries are gated on `match` — the share of the query's meaningful words the entry accounts for — because catalogue search returns papers about "peripheral tissues" for a novel called *The Peripheral*. ### `GET /api/recent` Recently searched queries and today's CSE call count. No user identifiers are stored. ### Standard endpoints - `GET /api/health`, `GET /api/version`, `GET /api/me` (auth-protected) - `GET /agent-guide.md`, `GET /agent-guide` ## Service boundaries This app owns **discovery**: the Google CSE query, ranking, the results UI, and the fast per-result liveness probe. It does not own **retrieval or parsing** — [Text Handling](https://text-handling.aisloppy.com/agent-guide.md) does, and `/api/download`, `/api/inspect`, and `escalate=1` all call it over HTTP. That is where the header realism, the residential/browser escalation ladder, and the PDF structure reader live, so every app on the fleet gets the same behaviour. ## Limits and failure modes - **Google CSE free tier is 100 queries/day.** Successful searches are cached for 24 hours; link probes for 7 days. Cached queries do not consume quota. When the quota is gone the API returns `429` with an explicit message — it never returns stale or empty results as if they were fresh. - Upstream errors propagate with their real status and message in `{"error": ..., "detail": ...}`. There is no silent fallback. - Timeouts: 12s per CSE call, 8s per link probe, 90s for a delegated fetch. The web client applies a 45s search deadline, 100s for an escalated retry, and 90s for an inspection, each surfacing cancellation as a visible error. - A `blocked` result means our probe was refused, not that the file is missing. The UI offers a per-link **Retry**, which is `escalate=1`. - `status` reports whether a link is **reachable**, never whether it is the document you wanted. Size and page count are the cheap relevance signals: a 5-page, 240 KB result for a novel is a review of it. The UI keeps both in an aligned column with a magnitude bar (one-hue sequential ramp, bright = large, absolute bands so a colour means the same thing across searches), sorts on either, and `POST /api/rank` judges intent when size alone cannot. - **`total_estimate` is Google's estimate of the whole index, not what you can reach.** A query reporting 133,000 matches still yields at most 100 results — CSE stops at `start=91`. Ranking and sorting operate on what was returned. - **In-copyright works are largely absent, by design of the index.** Google removes URLs from results on copyright notice, and CSE inherits those removals, so a search for a recent novel returns criticism, reviews, and samples rather than the book. That is an index property, not a fetch failure — no amount of escalation changes it. ## Integration rules - Fail fast on errors; do not silently degrade. - Use network APIs between services; no cross-app imports. - Send explicit timeouts on every call.