Paste this into Cursor agents as the working brief. Repo/scaffold: click-scrape (Chrome MV3, zero-build vanilla JS). Zip for Load unpacked already exists alongside the folder.
What: Freemium Chrome MV3 point-and-click visual scraper.
Free MVP (this plan only):
chrome.storage.local)Paid later (explicitly out of MVP): schedule, cloud recipes, accounts. Target later: ~$29\u201339/mo.
Wedge vs Instant Data Scraper: nested lists + pagination-friendly picks + clean preview \u2014 not \u201canother free auto-detect scraper.\u201d
Distribution: Chrome Web Store, unlisted first. Product name + Store account still open \u2014 do not block the build.
| Area | Owns | Notes |
|---|---|---|
src/content/ |
pick, highlight, pagination walk, overlay UI | content.js, picker-overlay.js, highlighter.css — hover also paints .click-scrape-similar peers |
src/shared/ |
pure helpers | selectors.js (incl. findSimilarPeers / getSimilarScopeRoot), extract.js, export.js, storage.js, pagination.js, columns.js — keep testable, no DOM chrome |
src/popup/ |
start picking + recipe list | thin; preview/export live primarily on overlay for MVP |
src/background/service-worker.js |
messaging only | no scrape logic |
demo.html |
local fixture | first acceptance surface |
manifest.json |
MV3 | permissions stay minimal: activeTab, scripting, storage, host as needed |
Agent handoff: see README.md for feature status + explore playbook, and AGENTS.md for post-MVP contracts (similar-peer hover, column model, pagination, soft cap) and the feature → code map for Grok/QA bots.
Data path: page \u2192 content extract \u2192 in-memory rows \u2192 export / local storage. Never network for scraped data.
Roles:
Owner: Developer (Web Design: highlight clarity only)
demo.html and 2 live card-style sitesOwner: Developer
Owner: Web Design (Developer: wire extract/export to edited column model)
#cs-preview)Owner: Web Design (copy/UX) + Developer (local counter)
Current overlay (picker-overlay.js) already has: column name input, field list, undo, export CSV/JSON, save recipe, stop, basic HTML table preview.
MVP UX gaps to close in step 3\u20134:
#cs-fields)
popup.html)
highlighter.css)Do not introduce a design-system package, React, or build step in MVP. Stay vanilla JS + CSS in the existing files.
chrome.storage.local)chrome://extensions \u2192 Developer mode \u2192 Load unpacked \u2192 select click-scrape (folder with manifest.json)demo.html from the folderchrome:// pages)| Agent focus | Files | Steps |
|---|---|---|
| Selectors + pagination | src/shared/selectors.js, extract.js, src/content/content.js |
1\u20132 |
| Preview + overlay UX | picker-overlay.js, highlighter.css, overlay styles in content CSS |
3 |
| Popup + soft cap + zip | popup/*, storage.js, packaging |
4 |
| Export contract | export.js + callers |
3 (must honor edited columns) |
When implementing: prefer small PRs/commits per build-order step; prove each step on demo.html before live sites.
Paste this section with the brief when Cursor agents touch data shapes or messaging.
chrome.storage.local key clickScrapeRecipes){
id: string, // crypto.randomUUID()
name: string,
createdAt: number, // Date.now()
pageUrl: string, // location.href at save
rootSelector: string, // list root (cssPath)
itemSelector: string, // item tag / selector under root
fields: Array<{
name: string, // column key in preview + export
relativeSelector: string // path from item \u2192 field (:scope ok)
}>
// Step 3 may add optional: columnOrder?: string[], hiddenColumns?: string[]
// Prefer deriving export columns from an explicit ordered list of field names
// so rename/drop/reorder does not require rewriting relativeSelector.
}
Hard cap already in storage.saveRecipe: keep at most 50 recipes in the array. Soft free-tier nudge (step 4) is separate UX on top of this.
Record<fieldName, string>.export.exportCsv(rows, columns) / preview take ordered columns: string[] \u2014 that order is the contract for step 3 (rename = change fields[].name + keys; drop/reorder = change columns / field list, not the DOM walk).export.exportJson(rows) currently dumps row objects; after step 3 it must emit objects with only visible columns in preview order (or an array-of-arrays + header \u2014 pick one and keep CSV/JSON consistent).| Type | Direction | Purpose |
|---|---|---|
CLICK_SCRAPE_INJECT |
popup \u2192 SW | inject scripts + CSS, then start |
CLICK_SCRAPE_START |
SW \u2192 content | start picker |
CLICK_SCRAPE_STOP |
\u2192 content | stop picker |
CLICK_SCRAPE_RUN_ON_TAB |
popup \u2192 SW | inject + run saved recipe |
CLICK_SCRAPE_RUN_RECIPE |
SW \u2192 content | { recipe } extract + preview |
CLICK_SCRAPE_TOGGLE |
\u2192 content | toggle (optional) |
Do not put scrape logic, storage writes, or export in the service worker. SW = inject + relay only.
fetch / XHR / remote logging of scraped rows or recipes.shared/* stays DOM-chrome-free where possible (export may use document for download links; keep that isolated).activeTab, scripting, storage (+ host as already declared). Do not add identity / cookies / webRequest for MVP.demo.html before live sites.Owner: Developer. Do not start step 3 until 1\u20132 acceptance passes on demo.html.
src/shared/selectors.js, extract.js, content.js):scope \u2026) from a detected list root; avoid absolute cssPath that breaks when ads/chrome shift.relativeSelector.findSimilarPeers(el) outlines sibling peers in a scoped list region (not site-wide class matches). Wired from content.js hover → .click-scrape-similar. Keep scope + peer logic in selectors.js; see AGENTS.md.demo.html plus 2 live card-style sites (e.g. a product grid and a results/cards list). Same recipe after reload must rematch the same columns.shared/ where possible (selector generation + extract given a fixture DOM string or jsdom-free helpers); prove interactively on demo.html either way. Tests for peers: tests/fixtures/noisy-bullets.html.content.js + shared extract)<a rel="next">, link/button text \u2248 Next / \u203a / \u2192, and simple numbered page links \u2014 best-effort, no site-specific plugins in MVP.exportCsv(rows, columns) / JSON must take an ordered columns: string[] matching preview.Blob + download link). No fetch of row data.Schedule, cloud, accounts, bundlers, new permissions, phone-home soft-cap enforcement.
Use this so Cursor agents understand why the MVP cut exists. Do not expand scope from this section.
| Tool | Role | Notes |
|---|---|---|
| Instant Data Scraper | Primary free rival | ~1M Chrome installs, ~4.9 / 7k+ ratings; free local CSV/Excel; data stays in-browser. No longer maintained by original developer (Web Robots). Guides call out fragile pagination and no scheduling. |
| Web Scraper (webscraper.io) | Free local + paid cloud | Powerful sitemap model; steep learning curve for beginners. Cloud from ~$50/mo (Project, annual billing). |
| Simplescraper | Paid entry | Cloud/schedule plans from ~$39/mo. |
| Octoparse | Desktop + cloud | Paid from ~$69/mo. |
| ParseHub | Visual desktop | Free very limited; Standard ~$189/mo includes scheduling \u2014 too expensive as a comparison target for our paid tier. |
Failures that matter for free MVP (in priority order):
Not the free-MVP pain: \u201cI need a scheduler.\u201d Schedule shows up when users outgrow free and look at cloud tools \u2014 that is the paid wedge ($29\u201339/mo for schedule + cloud recipes later).
Owner: Web Design. Start after Developer steps 1\u20132 pass on demo.html. Honor Architecture contracts (row/column model, soft-cap defaults, no new frameworks).
picker-overlay.js + highlight/overlay CSS)columns: string[] (+ optional columnOrder / hiddenColumns on recipe) \u2014 do not re-walk the DOM to reorder or drop.fields[].name and row keys together; preview headers stay the editable source of truth.#cs-preview; full set still exports.Design-system packages, React/build tooling, Store branding/name work that blocks engineering, paid/schedule UI.