Project / 02Selected Work

Intelligent ingestion + experimentation / QA

AppyHour Scraper

Turning unstructured venue data into a measurable, human-reviewed ingestion pipeline.

Engagement
Research + ingestion system
MVP boundary
HTML-first
Quality
Automated grading + human review
Target
Overnight market enrichment

Thirty-second read

Problem

Venue offers and events were costly to gather at market scale.

Owned

Coverage research, MVP boundary, pipeline, and quality model.

Key decision

HTML-first

Proof

~95% lower API cost and 70–80% fewer tokens.

53%

Usable HTML path

Of scored venues in the studied pooled sample

~95%

Lower API cost

After extraction optimization

70–80%

Fewer tokens

Supporting overnight enrichment

01 / Context

Problem / opportunity

Offers and events lived across inconsistent pages and formats. Manual collection—and undifferentiated extraction—was too expensive to scale.

The first decision was not how to scrape everything. It was what an evidence-based v1 should cover.

53% refers only to scored venues in the studied pooled sample—not the whole web.

Before scaling extraction, I used stratified random sampling to measure what an HTML-first MVP could realistically cover. That evidence shaped the product boundary: HTML became the v1 path, while PDF, image, and JavaScript-heavy sources were measured and deliberately deferred.

02 / Ownership

What I owned

  1. 01Designed a stratified random study to measure realistic HTML coverage.
  2. 02Set the HTML-first v1 boundary and documented deferred source types.
  3. 03Designed the ingestion, grading, normalization, validation, and sync pipeline.
  4. 04Kept human review in control while AI explained count disagreement on demand.

03 / System

System / product architecture

  1. 01

    Discover

  2. 02

    Crawl

  3. 03

    Clean

  4. 04

    Extract

  5. 05

    Grade

  6. 06

    Normalize

  7. 07

    Validate

  8. 08

    Sync

  9. 09

    Mobile

A staged pipeline converts uncertain source material into reviewed, mobile-ready catalog data.

1

Scrape confidence measures extraction reliability; catalog usefulness measures product value.

2

Human reviewers make quality decisions. AI tuning notes explain disagreement but never decide pass/fail.

3

Extraction and prompt changes cut API cost by approximately 95% and token use by 70–80%.

04 / Evidence

Product evidence

Scrapability study dashboard showing a 53% headline share of scored venues with a usable HTML path, plus status mix and reason tag breakdowns.
Research evidence / 0153% of scored venues in the studied pooled sample had a usable HTML path. That evidence shaped the HTML-first v1 boundary.
QA review workspace with a venue list, per-field pass/fail grading, scrape confidence and catalog usefulness scores, and the AI tuning note.
QA evidence / 02Human field-level grading sits alongside separate scrape-confidence and catalog-usefulness signals. An on-demand AI note explains why the scrape and existing catalog disagree without acting as a pass/fail gate.

05 / Outcome

What changed

The study turned technical uncertainty into a measurable product decision.

A deliberately constrained v1, explicit expansion paths, and lower extraction costs created a practical route to overnight market enrichment.

Let's discover what's possible.

Every business has hidden opportunities.

Some require better decisions. Some require automation. Some require entirely new software.

Let's find yours.

Schedule a Strategy Session