Before using Playwright for ingestion:
☐ Is there an official API, SFTP or export feed?
☐ Did I check the browser's network tab for a JSON endpoint?
☐ Is automated access allowed by the terms or by agreement?
☐ Is personal data involved? If yes, stop and check policies.
☐ Who maintains this when the page changes?
Only proceed if a browser is truly required and access is permitted.

pip install playwright pandas
playwright install chromium
# 'playwright install' downloads the browser binaries Playwright controls

from playwright.sync_api import sync_playwright
import pandas as pd

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://quotes.toscrape.com/js/")   # practice site
    page.wait_for_selector(".quote")
    texts = page.locator(".quote .text").all_inner_texts()
    authors = page.locator(".quote .author").all_inner_texts()
    browser.close()

df = pd.DataFrame({"quote": texts, "author": authors})
print(df.head())

import sqlite3
from datetime import date

df["loaded_on"] = date.today().isoformat()
with sqlite3.connect("staging.db") as con:
    df.to_sql("quotes_raw", con, if_exists="append", index=False)
    print(con.execute("SELECT COUNT(*) FROM quotes_raw").fetchone()[0], "rows in staging")

import os
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto(os.environ["PORTAL_URL"])
    page.fill("#username", os.environ["PORTAL_USER"])
    page.fill("#password", os.environ["PORTAL_PASS"])
    page.click("button[type=submit]")
    with page.expect_download() as dl:
        page.click("text=Export CSV")
    dl.value.save_as("landing/daily_export.csv")
    browser.close()
# selectors above are examples; match them to the real portal

import sqlite3
from playwright.sync_api import sync_playwright

with sqlite3.connect("warehouse.db") as con:
    expected = con.execute("SELECT ROUND(SUM(amount)) FROM orders WHERE order_date = DATE('now','-1 day')").fetchone()[0]

with sync_playwright() as p:
    page = p.chromium.launch().new_page()
    page.goto("http://localhost:8050/daily")      # your internal dashboard
    shown = page.locator("[data-test=revenue]").inner_text()
    page.context.browser.close()

shown_value = float(shown.replace(",", "").replace("₹", ""))
assert abs(shown_value - expected) < 1, f"Dashboard {shown_value} != warehouse {expected}"

Browser Ingestion Job Spec
Source: ______________
Why not an API? ______________
Permission/terms checked on: ________ by ________
Fields collected: ______________
Schedule: daily at ____
Landing: folder/table ______________
Credentials stored in: environment variables / secret manager
Owner: ______________

Run Monitor
Date     Rows   Duration   Status   Screenshot on failure?
_____    ____   ________   ______   ☐
Alert if rows drop more than 30% from the 7-day average.
On failure, save page.screenshot(path='fail.png') for debugging.

Dashboard Verification Checklist
☐ KPI tiles match warehouse totals (tolerance defined)
☐ Date filter shows the expected latest date
☐ No error banners on the page
☐ Page loads within agreed time
☐ Check runs after each scheduled refresh
☐ Failure alerts the data team, not just a log