Scrape MMC doctor records by APC year and log matches to Google Sheets

Description

Quick overview
This workflow manually scrapes the MMC MeRITS registered doctor directory, filters doctors by a target APC year, and writes matched records to Google Sheets while rate-limiting requests and saving a resume state so it can continue from the last processed page and row.

How it works
Starts when you run the workflow manually and sets the target APC year and the maximum number of matches to collect in a run.
Loads the last saved page and row index from n8n workflow static data to resume where the previous run stopped.
Requests the current results page from the MMC MeRITS registeredDoctor search and parses the HTML to extract doctor IDs, rows, and the total number of pages.
Iterates through each row on the page, fetching each doctor’s detail page from MMC MeRITS and extracting fields like full name, qualification, provisional registration date, registration number, APC year, and place of practice.
If the extracted APC year matches the configured target year, appends or updates the doctor record in Google Sheets.
After each doctor, increments and persists progress, waits briefly to rate-limit requests, and stops when the daily match target is reached; otherwise it advances to the next page until all pages are exhausted.

Setup
Add a Google Sheets OAuth2 credential and select the target spreadsheet and sheet where matched doctor details should be appended/updated.
Update the configuration values for the target APC year and the per-run match limit to suit your filtering needs.
Run the workflow once to initialize the saved resume state, and clear the workflow’s static data if you want to restart scraping from page 1.

Requirements
n8n instance (self-hosted or cloud)
Google Sheets account with OAuth2 credential connected
A destination spreadsheet with headers: Name, Qualification, Date of Provisional Registration, Full Registration Number, APC Year, Place of Practice
No login/API key needed for MMC MeRITS itself — it's a public register

Customization
Target APC year to filter by (editable in the Config node, e.g. 2026)
Daily match limit — how many matching doctors to collect per run before stopping (default 10)
Rate-limit delay between requests (default 0.7s) — increase if you hit errors
Resume position is saved automatically between runs, so re-running continues from where it left off with no duplicates

Additional info
Scrapes Malaysia's MMC MeRITS public doctor register (merits.mmc.gov.my), scanning the
full list page by page since the site has no server-side year filter. Each doctor's detail
page is checked against a target APC year; matches are written to Google Sheets. Designed
to run incrementally (e.g. once daily) progress is persisted via workflow static data, so
each run picks up exactly where the last one stopped without rescanning or duplicating records.
Total register size is ~197,000 doctors, so a full sweep for a given year may take multiple runs.