BOOK A CALL

All projects

Case study

House Search

A self-hosted house-hunting board for the two of us. It reads three listing sites every weekday, keeps what sits inside a walkable district, scores each home against sixteen requirements, and never loses a mark either of us made.

Platform
Web · self-hosted
Stack
Node · Fastify · SQLite · Puppeteer · Docker · Cloudflare Access
Status
Retired September 2026 · the search ended with a house
Scale
6,000 lines · three sources · sixteen requirements · one container

The problem

Two people looking for a house in the same city keep the same list in four places: two listing apps, a shared note, and whatever each of us remembers. The apps disagree with each other about what is for sale, none of them knows what we actually need, and a home one of us flagged can vanish from all of them the day it goes under contract.

The board exists to answer one question every morning — what is new, what still fits, and what did we say about it — and to be thrown away when the search is over. It is temporary by design, with a retirement date written into its own documentation.

The shape of the answer

One container on a Mac in the house, behind a Cloudflare tunnel and an Access login. A scheduler inside it pulls every weekday morning, scores what it finds, and publishes a board both of us mark up. A realtor sheet exports as a PDF with every unanswered question printed on it, because that is what a viewing is for.

The decision worth defending

Unknown is a real answer

Every one of the sixteen requirements is scored pass, fail or unknown, and unknown is not a shrug. Listing text can rule a garage out but never in — nobody publishes garage dimensions — so that criterion can never pass. “Kitchen and primary already up to date” needs the listing to say so about both, so most homes stay unknown and want eyes on them. A home we cannot count to five usable rooms stays unknown rather than failing, because the difference between a den and a third office is a thing only a visit can settle.

Unknowns print onto the realtor sheet as “Confirm on site”. The board is not trying to decide; it is trying to make the visit shorter.

Decisions

  1. A vanished home has not necessarily sold

    The search API returns active listings only. A home that goes under contract does not come back labelled pending — it simply stops appearing — and so does a home whose request failed that morning. Those want opposite treatment, so every home missing from a sweep gets its own detail page read and lands as pending, gone, or unknown. Unknown never retires a home. A failed request is not news about a house.

  2. The scraper cannot clobber a mark

    Marks and notes are columns the scraper has no write path to — a property of the schema, not a promise in a function. A home that leaves the market is deleted only if nobody marked it; otherwise it stays, badged, and sorted last.

  3. A bad pull is rejected, not published

    Fewer than ten homes in the rings, or a drop of more than half from the previous run, and the run is marked rejected and yesterday’s board stands. A run also prunes only the ZIPs it actually read — found the hard way when a partial test run deleted eighteen homes whose requests had failed. A stale board beats an empty one, and the board says so when it is stale.

  4. One source is not enough

    A four-bedroom inside two qualifying rings, under the ceiling, was on two of the three sites and simply absent from the third’s own search. So the board reads all three and merges on normalised address — which only started working once the ZIP was derived from the address tail, because the primary source leaves it blank on its list cards. Before that fix, a hundred duplicates were one commit from the board.

  5. The ring is wider where the inventory is thin

    Every anchor is a walkable district with a ring drawn around it. One village gets a wider ring and a lower bedroom floor, because the location is the whole point there and the relaxation adds four homes rather than the thirty-three it would add everywhere else. That was measured before it was decided.

  6. One request at a time, and one run at a time

    Requests are serialised four to five seconds apart, and a PID lock allows a single pull. During development four overlapping runs, each politely pacing at five seconds, produced an effective rate three times faster than any of them intended — per-process pacing is meaningless without the lock.

The container never asks a listing site anything

Two of the three sites will not talk to a server. Every anonymous request this project ever made to them came back 403 or 429, while the same pages loaded normally in my own browser. After one morning burned nearly three hours of back-off learning nothing, the transport layer was changed to refuse outbound requests to every listing domain. It is the design, not a safety net: the container has no browser, no signed-in session and no reputation.

So a pull is two halves at different times, and only the first needs a person. The harvest is done by an AI agent driving my own everyday browser through an extension, walking each search page at human pace and writing the results to a drop file. The container then merges, scores, sanity-checks and publishes on its own schedule. A drop file is good for eighteen hours; a stale or missing one is a logged skip and the run proceeds on the one source the container can read — never a failure.

Getting the harvest out of the browser took four dead ends: the extension cannot reach loopback, the sites’ own policies block in-page fetches to it, and both downloads and the clipboard are gesture-gated. What works is writing each page’s results into local storage on the site’s own origin, reading it back through the extension, and letting the agent write the file over SSH. A dedicated automation browser was built first and rejected — the sites bot-check a fresh profile, and nobody wanted a second browser to babysit.

Engineering notes

  1. The one site that does answer is behind a challenge

    Its firewall returns a challenge page that a plain fetch cannot solve. A real browser solves it once, the resulting token is harvested, and plain fetches carry it for the rest of the run. Its own rate limiter answers differently — a “robot?” page — and the run backs off and then aborts rather than hammering.

  2. Most mornings no detail page is read at all

    Listing remarks do not change day to day, so a home already in the database at the same price keeps its stored score. A typical run is a dozen list pages plus whatever is genuinely new; a price change forces a re-read for that home alone. This is what keeps the daily run under the rate limiter.

  3. Cards without coordinates get geocoded

    One source’s search cards carry no location, so a home found only there is geocoded through a public service, cached, at one request a second with a contact address. Without it the radius filter would drop those homes silently.

  4. The schedule lives in the server

    The server spawns the scraper as a child process at three times each weekday morning, the later two exiting immediately if the first landed. One container, no cron daemon, and the retries exist for the mornings the source says no.

Where it stands

Retired. We found the house in September 2026, two months ahead of the November the code had written into its own documentation, and the board was switched off the same week: container, hostname and login gone, the database and every mark archived. It ran for two weeks, watched ninety-four homes, and took sixteen of them onto a tour sheet.