Lawful scraping at scale — Prospect-Pro AI field notes

Per-source token bucket, identified user-agent, robots.txt respect, business dedup keys, timestamped snapshots and right to erasure: 14 months of continuous scraping without incident.

Scraping Pages Jaunes, Pappers, GoAfrica and Google Maps lawfully and respectfully, at thousands of requests per day. Field report from Prospect-Pro AI.

1. The context

Prospect-Pro AI, a B2B prospecting platform, must continuously build a qualified contact base from four public sources: Pages Jaunes (French professional directory), Pappers (French company open data), GoAfrica (pan-African business directory), Google Maps (establishment listings). The challenge: feed the AI pipeline with fresh data without crossing into illegality nor saturating sources.

This article isn't a "how to bypass an anti-bot" tutorial. It's a field report on how to scrape properly: legal compliance, technical respect, production robustness.

2. The basic rule — what's legal

First reflex: know the legal terrain. Three criteria clarify 95% of cases:

Public data — accessible without authentication, not behind a paywall or explicit no-scraping ToS
Non-personal or pseudonymized data — corporate data (company name, registration number, business phone) isn't covered by GDPR if it stays in the B2B sphere
Respect ToS — if a site explicitly forbids scraping in its terms, we abstain

Concrete cases:

Pages Jaunes: public pro data, scraping tolerated for reasonable B2B use
Pappers: public API provided by the State (SIRENE open data), scraping pointless if you use their API
GoAfrica: pan-African directory, public pro data
Google Maps: public listings, but Google has an official API (Places API) — preferred

Always prefer the official API when it exists. Scraping is a last resort, not a shortcut.

3. Pipeline architecture

On Prospect-Pro, each source is wrapped in a typed connector. Skeleton:

TypeScript

1// src/scraping/Connector.ts
2export interface ScrapingConnector {
3  readonly source: SourceName;
4  readonly rateLimit: { rpm: number; concurrent: number };
5  readonly robotsTxtPolicy: "respect" | "ignore"; // always "respect" in prod
6  search(query: SearchQuery): AsyncIterable<RawRecord>;
7  hydrate(record: RawRecord): Promise<EnrichedRecord>;
8}

Each connector declares:

Its source
Its own rate-limit (negotiated with the source if possible)
Its robots.txt policy
A search method that yields raw records (async iterable for streaming)
A hydrate method to enrich a record (details, address, etc.)

4. Respectful rate-limiting

Saturating a server is pirate behavior, not pro behavior. Three combined techniques:

4.1 Per-source token bucket

Each source has its own bucket, sized conservatively (typically 60 req/min).

TypeScript

1// src/scraping/RateLimiter.ts
2export class TokenBucket {
private tokens: number;
private readonly capacity: number;
private readonly refillRate: number;
private lastRefill: number;
7 
constructor(capacity: number, refillRate: number) {
  this.capacity = capacity;
  this.tokens = capacity;
  this.refillRate = refillRate;
  this.lastRefill = Date.now();
}
14 
async acquire(): Promise<void> {
  this.refill();
  if (this.tokens >= 1) {
    this.tokens -= 1;
    return;
  }
  const waitMs = ((1 - this.tokens) / this.refillRate) * 1000;
  await sleep(waitMs);
  return this.acquire();
}
25 
private refill(): void {
  const now = Date.now();
  const elapsed = (now - this.lastRefill) / 1000;
  this.tokens = Math.min(this.capacity, this.tokens + elapsed * this.refillRate);
  this.lastRefill = now;
}
32}

Each connector calls await bucket.acquire() before each HTTP request.

4.2 Bounded concurrency

No more than N concurrent requests per source. Simplest: a queue with p-limit:

TypeScript

1import pLimit from "p-limit";
2 
3const limit = pLimit(3);
4const results = await Promise.all(
5  queries.map((q) => limit(() => connector.search(q)))
6);

4.3 Exponential backoff on error

429 Too Many Requests or 503 Service Unavailable: wait, retry with growing delay.

TypeScript

1async function withRetry<T>(fn: () => Promise<T>, maxAttempts = 5): Promise<T> {
let lastError: unknown;
for (let attempt = 0; attempt < maxAttempts; attempt += 1) {
  try {
    return await fn();
  } catch (error) {
    lastError = error;
    const delay = Math.min(30_000, 1000 * 2 ** attempt) + Math.random() * 1000;
    await sleep(delay);
  }
}
throw lastError;
13}

The + Math.random() (jitter) prevents all workers from retrying at the same time after an outage.

5. Honest identification — truthful user-agent

Hiding behind a standard Chrome user-agent is dangerous: legally, it suggests intent to deceive. Best practice:

Plain Text

1	User-Agent: ProspectProBot/1.0 (+https://prospect-pro.example.com/bot)

The URL points to a page explaining the project, contact for questions, how to opt out. Serious sources respect this transparency. Sources that block an identified bot signal they refuse scraping — and we abstain.

6. Respecting robots.txt

Before each session, load the source's robots.txt and respect its Disallow.

TypeScript

1import robotsParser from "robots-parser";
2 
3async function loadRobots(host: string): Promise<robotsParser.Robot> {
4  const text = await fetch(`https://${host}/robots.txt`).then((r) => r.text());
5  return robotsParser(`https://${host}/robots.txt`, text);
6}
7 
8async function shouldFetch(url: string, robots: robotsParser.Robot): Promise<boolean> {
9  return robots.isAllowed(url, "ProspectProBot");
10}

If robots.txt forbids it, we don't go. Period. Even if technically we could.

7. Deduplication and data quality

A source may return the same contact twice under slightly different labels. Without dedup, your base pollutes fast.

Strategy: business dedup key combining stable fields.

TypeScript

1function leadKey(lead: RawLead): string {
2  if (lead.registrationNumber) return `reg:${lead.registrationNumber}`;
3  const normalized = lead.name.toLowerCase().replace(/[^a-z0-9]/g, "");
4  return `hash:${sha256(`${normalized}|${lead.city}`)}`;
5}

In DB, unique constraint on this key. INSERT ... ON CONFLICT DO UPDATE updates recent fields without duplication.

8. Storage and history

Scraped data ages. A company moves, changes phone, closes. Store timestamped snapshots rather than a single state:

SQL

1CREATE TABLE leads (
id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
key         VARCHAR(120) NOT NULL UNIQUE,
name        TEXT NOT NULL,
registration_number VARCHAR(20) NULL,
city        TEXT NULL,
first_seen  TIMESTAMP NOT NULL DEFAULT NOW(),
last_seen   TIMESTAMP NOT NULL DEFAULT NOW()
9);
10 
11CREATE TABLE lead_snapshots (
id            UUID PRIMARY KEY DEFAULT gen_random_uuid(),
lead_id       UUID NOT NULL REFERENCES leads(id),
source        VARCHAR(40) NOT NULL,
payload       JSONB NOT NULL,
scraped_at    TIMESTAMP NOT NULL DEFAULT NOW()
17);

leads holds the current state, lead_snapshots the raw history. Lets you trace a contact, understand evolution, and purge old snapshots without losing leads.

If you scrape B2C or B2B data touching PII (a director's first + last name for example), GDPR applies. Three minimum requirements:

Information: public page explaining collection, duration, rights
Right to erasure: an endpoint letting a person request deletion
Minimization: store only what's necessary — no blind "just in case" scraping

TypeScript

1// src/controllers/rightToErasure.ts
2export async function requestErasure(req: Request, res: Response) {
3  const email = req.body.email;
4  const found = await Lead.findOne({ where: { contactEmail: email } });
5  if (!found) return res.status(404).json({ ok: false });
6 
7  await found.update({
8    redactedAt: new Date(),
9    name: "[REDACTED]",
10    contactEmail: null,
11    contactPhone: null,
12  });
13  await LeadSnapshot.destroy({ where: { leadId: found.id } });
14 
15  return res.status(200).json({ ok: true, redactedAt: found.redactedAt });
16}

10. Observability

Production scrapers must emit metrics on:

Requests per second (per source)
HTTP error rate (per code)
Average and p95 latency
New vs updated lead volume
Dedup rate

A Grafana dashboard on these signals catches early:

A source changing HTML (sudden drop in scraped leads)
An anti-bot kicking in (rise of 403s)
A bottleneck (p95 latency exploding)

11. Pitfalls to avoid

Pitfall	Symptom	Fix
Lying user-agent	Legal risk, ethical ban	Identified UA + project URL
No rate-limit	Source saturation, IP ban	Per-source token bucket
Unlimited concurrency	Too many connections, OOM	`p-limit` or bounded pool
Ignoring robots.txt	Illegal per ToS + unethical	Parse and respect
No dedup	Polluted base	Business key + UPSERT
Single snapshot	No history	Timestamped snapshots table
No right to erasure	GDPR risk	Erasure endpoint
No monitor	Source changes → silence	Dashboard on rates and codes

12. Closing

Scraping at scale means playing the role of an intensive user without penalizing the source. Three keywords: identification, moderation, respect.

Identification (honest user-agent, project page) protects legally. Moderation (rate-limit, concurrency, backoff) protects the source technically. Respect (robots.txt, GDPR, right to erasure) protects your brand and your users.

On Prospect-Pro, this framework allowed continuous scraping for 14 months with no legal incident, no ban, and stable data quality. No black magic, just discipline.

Per-source token bucket, identified user-agent, robots.txt respect, business dedup keys, timestamped snapshots and right to erasure: 14 months of continuous scraping without incident.

Scraping Pages Jaunes, Pappers, GoAfrica and Google Maps lawfully and respectfully, at thousands of requests per day. Field report from Prospect-Pro AI.

1. The context

This article isn't a "how to bypass an anti-bot" tutorial. It's a field report on how to scrape properly: legal compliance, technical respect, production robustness.

2. The basic rule — what's legal

First reflex: know the legal terrain. Three criteria clarify 95% of cases:

Public data — accessible without authentication, not behind a paywall or explicit no-scraping ToS
Non-personal or pseudonymized data — corporate data (company name, registration number, business phone) isn't covered by GDPR if it stays in the B2B sphere
Respect ToS — if a site explicitly forbids scraping in its terms, we abstain

Concrete cases:

Pages Jaunes: public pro data, scraping tolerated for reasonable B2B use
Pappers: public API provided by the State (SIRENE open data), scraping pointless if you use their API
GoAfrica: pan-African directory, public pro data
Google Maps: public listings, but Google has an official API (Places API) — preferred

Always prefer the official API when it exists. Scraping is a last resort, not a shortcut.

3. Pipeline architecture

On Prospect-Pro, each source is wrapped in a typed connector. Skeleton:

TypeScript

1// src/scraping/Connector.ts
2export interface ScrapingConnector {
3  readonly source: SourceName;
4  readonly rateLimit: { rpm: number; concurrent: number };
5  readonly robotsTxtPolicy: "respect" | "ignore"; // always "respect" in prod
6  search(query: SearchQuery): AsyncIterable<RawRecord>;
7  hydrate(record: RawRecord): Promise<EnrichedRecord>;
8}

Each connector declares:

Its source
Its own rate-limit (negotiated with the source if possible)
Its robots.txt policy
A search method that yields raw records (async iterable for streaming)
A hydrate method to enrich a record (details, address, etc.)

4. Respectful rate-limiting

Saturating a server is pirate behavior, not pro behavior. Three combined techniques:

4.1 Per-source token bucket

Each source has its own bucket, sized conservatively (typically 60 req/min).

TypeScript

1// src/scraping/RateLimiter.ts
2export class TokenBucket {
private tokens: number;
private readonly capacity: number;
private readonly refillRate: number;
private lastRefill: number;
7 
constructor(capacity: number, refillRate: number) {
  this.capacity = capacity;
  this.tokens = capacity;
  this.refillRate = refillRate;
  this.lastRefill = Date.now();
}
14 
async acquire(): Promise<void> {
  this.refill();
  if (this.tokens >= 1) {
    this.tokens -= 1;
    return;
  }
  const waitMs = ((1 - this.tokens) / this.refillRate) * 1000;
  await sleep(waitMs);
  return this.acquire();
}
25 
private refill(): void {
  const now = Date.now();
  const elapsed = (now - this.lastRefill) / 1000;
  this.tokens = Math.min(this.capacity, this.tokens + elapsed * this.refillRate);
  this.lastRefill = now;
}
32}

Each connector calls await bucket.acquire() before each HTTP request.

4.2 Bounded concurrency

No more than N concurrent requests per source. Simplest: a queue with p-limit:

TypeScript

1import pLimit from "p-limit";
2 
3const limit = pLimit(3);
4const results = await Promise.all(
5  queries.map((q) => limit(() => connector.search(q)))
6);

4.3 Exponential backoff on error

429 Too Many Requests or 503 Service Unavailable: wait, retry with growing delay.

TypeScript

1async function withRetry<T>(fn: () => Promise<T>, maxAttempts = 5): Promise<T> {
let lastError: unknown;
for (let attempt = 0; attempt < maxAttempts; attempt += 1) {
  try {
    return await fn();
  } catch (error) {
    lastError = error;
    const delay = Math.min(30_000, 1000 * 2 ** attempt) + Math.random() * 1000;
    await sleep(delay);
  }
}
throw lastError;
13}

The + Math.random() (jitter) prevents all workers from retrying at the same time after an outage.

5. Honest identification — truthful user-agent

Hiding behind a standard Chrome user-agent is dangerous: legally, it suggests intent to deceive. Best practice:

Plain Text

1	User-Agent: ProspectProBot/1.0 (+https://prospect-pro.example.com/bot)

6. Respecting robots.txt

Before each session, load the source's robots.txt and respect its Disallow.

TypeScript

1import robotsParser from "robots-parser";
2 
3async function loadRobots(host: string): Promise<robotsParser.Robot> {
4  const text = await fetch(`https://${host}/robots.txt`).then((r) => r.text());
5  return robotsParser(`https://${host}/robots.txt`, text);
6}
7 
8async function shouldFetch(url: string, robots: robotsParser.Robot): Promise<boolean> {
9  return robots.isAllowed(url, "ProspectProBot");
10}

If robots.txt forbids it, we don't go. Period. Even if technically we could.

7. Deduplication and data quality

A source may return the same contact twice under slightly different labels. Without dedup, your base pollutes fast.

Strategy: business dedup key combining stable fields.

TypeScript

1function leadKey(lead: RawLead): string {
2  if (lead.registrationNumber) return `reg:${lead.registrationNumber}`;
3  const normalized = lead.name.toLowerCase().replace(/[^a-z0-9]/g, "");
4  return `hash:${sha256(`${normalized}|${lead.city}`)}`;
5}

In DB, unique constraint on this key. INSERT ... ON CONFLICT DO UPDATE updates recent fields without duplication.

8. Storage and history

Scraped data ages. A company moves, changes phone, closes. Store timestamped snapshots rather than a single state:

SQL

1CREATE TABLE leads (
id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
key         VARCHAR(120) NOT NULL UNIQUE,
name        TEXT NOT NULL,
registration_number VARCHAR(20) NULL,
city        TEXT NULL,
first_seen  TIMESTAMP NOT NULL DEFAULT NOW(),
last_seen   TIMESTAMP NOT NULL DEFAULT NOW()
9);
10 
11CREATE TABLE lead_snapshots (
id            UUID PRIMARY KEY DEFAULT gen_random_uuid(),
lead_id       UUID NOT NULL REFERENCES leads(id),
source        VARCHAR(40) NOT NULL,
payload       JSONB NOT NULL,
scraped_at    TIMESTAMP NOT NULL DEFAULT NOW()
17);

leads holds the current state, lead_snapshots the raw history. Lets you trace a contact, understand evolution, and purge old snapshots without losing leads.

If you scrape B2C or B2B data touching PII (a director's first + last name for example), GDPR applies. Three minimum requirements:

Information: public page explaining collection, duration, rights
Right to erasure: an endpoint letting a person request deletion
Minimization: store only what's necessary — no blind "just in case" scraping

TypeScript

1// src/controllers/rightToErasure.ts
2export async function requestErasure(req: Request, res: Response) {
3  const email = req.body.email;
4  const found = await Lead.findOne({ where: { contactEmail: email } });
5  if (!found) return res.status(404).json({ ok: false });
6 
7  await found.update({
8    redactedAt: new Date(),
9    name: "[REDACTED]",
10    contactEmail: null,
11    contactPhone: null,
12  });
13  await LeadSnapshot.destroy({ where: { leadId: found.id } });
14 
15  return res.status(200).json({ ok: true, redactedAt: found.redactedAt });
16}

10. Observability

Production scrapers must emit metrics on:

Requests per second (per source)
HTTP error rate (per code)
Average and p95 latency
New vs updated lead volume
Dedup rate

A Grafana dashboard on these signals catches early:

A source changing HTML (sudden drop in scraped leads)
An anti-bot kicking in (rise of 403s)
A bottleneck (p95 latency exploding)

11. Pitfalls to avoid

Pitfall	Symptom	Fix
Lying user-agent	Legal risk, ethical ban	Identified UA + project URL
No rate-limit	Source saturation, IP ban	Per-source token bucket
Unlimited concurrency	Too many connections, OOM	`p-limit` or bounded pool
Ignoring robots.txt	Illegal per ToS + unethical	Parse and respect
No dedup	Polluted base	Business key + UPSERT
Single snapshot	No history	Timestamped snapshots table
No right to erasure	GDPR risk	Erasure endpoint
No monitor	Source changes → silence	Dashboard on rates and codes

12. Closing

Scraping at scale means playing the role of an intensive user without penalizing the source. Three keywords: identification, moderation, respect.

On Prospect-Pro, this framework allowed continuous scraping for 14 months with no legal incident, no ban, and stable data quality. No black magic, just discipline.

Lawful scraping at scale — Prospect-Pro AI field notes

1. The context

2. The basic rule — what's legal

3. Pipeline architecture

4. Respectful rate-limiting

4.1 Per-source token bucket

4.2 Bounded concurrency

4.3 Exponential backoff on error

5. Honest identification — truthful user-agent

6. Respecting robots.txt

7. Deduplication and data quality

8. Storage and history

10. Observability

11. Pitfalls to avoid

12. Closing

Reader reactions

Turn this reading into a product decision.

Lawful scraping at scale — Prospect-Pro AI field notes

1. The context

2. The basic rule — what's legal

3. Pipeline architecture

4. Respectful rate-limiting

4.1 Per-source token bucket

4.2 Bounded concurrency

4.3 Exponential backoff on error

5. Honest identification — truthful user-agent

6. Respecting robots.txt

7. Deduplication and data quality

8. Storage and history

10. Observability

11. Pitfalls to avoid

12. Closing

Reader reactions

Turn this reading into a product decision.

1	// src/scraping/Connector.ts
2	export interface ScrapingConnector {
3	readonly source: SourceName;
4	readonly rateLimit: { rpm: number; concurrent: number };
5	readonly robotsTxtPolicy: "respect" \| "ignore"; // always "respect" in prod
6	search(query: SearchQuery): AsyncIterable<RawRecord>;
7	hydrate(record: RawRecord): Promise<EnrichedRecord>;
8	}

1	// src/scraping/RateLimiter.ts
2	export class TokenBucket {
3	private tokens: number;
4	private readonly capacity: number;
5	private readonly refillRate: number;
6	private lastRefill: number;
7
8	constructor(capacity: number, refillRate: number) {
9	this.capacity = capacity;
10	this.tokens = capacity;
11	this.refillRate = refillRate;
12	this.lastRefill = Date.now();
13	}
14
15	async acquire(): Promise<void> {
16	this.refill();
17	if (this.tokens >= 1) {
18	this.tokens -= 1;
19	return;
20	}
21	const waitMs = ((1 - this.tokens) / this.refillRate) * 1000;
22	await sleep(waitMs);
23	return this.acquire();
24	}
25
26	private refill(): void {
27	const now = Date.now();
28	const elapsed = (now - this.lastRefill) / 1000;
29	this.tokens = Math.min(this.capacity, this.tokens + elapsed * this.refillRate);
30	this.lastRefill = now;
31	}
32	}

1	import pLimit from "p-limit";
2
3	const limit = pLimit(3);
4	const results = await Promise.all(
5	queries.map((q) => limit(() => connector.search(q)))
6	);

1	async function withRetry<T>(fn: () => Promise<T>, maxAttempts = 5): Promise<T> {
2	let lastError: unknown;
3	for (let attempt = 0; attempt < maxAttempts; attempt += 1) {
4	try {
5	return await fn();
6	} catch (error) {
7	lastError = error;
8	const delay = Math.min(30_000, 1000 * 2 ** attempt) + Math.random() * 1000;
9	await sleep(delay);
10	}
11	}
12	throw lastError;
13	}

1	import robotsParser from "robots-parser";
2
3	async function loadRobots(host: string): Promise<robotsParser.Robot> {
4	const text = await fetch(`https://${host}/robots.txt`).then((r) => r.text());
5	return robotsParser(`https://${host}/robots.txt`, text);
6	}
7
8	async function shouldFetch(url: string, robots: robotsParser.Robot): Promise<boolean> {
9	return robots.isAllowed(url, "ProspectProBot");
10	}

1	function leadKey(lead: RawLead): string {
2	if (lead.registrationNumber) return `reg:${lead.registrationNumber}`;
3	const normalized = lead.name.toLowerCase().replace(/[^a-z0-9]/g, "");
4	return `hash:${sha256(`${normalized}\|${lead.city}`)}`;
5	}

1	CREATE TABLE leads (
2	id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
3	key VARCHAR(120) NOT NULL UNIQUE,
4	name TEXT NOT NULL,
5	registration_number VARCHAR(20) NULL,
6	city TEXT NULL,
7	first_seen TIMESTAMP NOT NULL DEFAULT NOW(),
8	last_seen TIMESTAMP NOT NULL DEFAULT NOW()
9	);
10
11	CREATE TABLE lead_snapshots (
12	id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
13	lead_id UUID NOT NULL REFERENCES leads(id),
14	source VARCHAR(40) NOT NULL,
15	payload JSONB NOT NULL,
16	scraped_at TIMESTAMP NOT NULL DEFAULT NOW()
17	);

1	// src/controllers/rightToErasure.ts
2	export async function requestErasure(req: Request, res: Response) {
3	const email = req.body.email;
4	const found = await Lead.findOne({ where: { contactEmail: email } });
5	if (!found) return res.status(404).json({ ok: false });
6
7	await found.update({
8	redactedAt: new Date(),
9	name: "[REDACTED]",
10	contactEmail: null,
11	contactPhone: null,
12	});
13	await LeadSnapshot.destroy({ where: { leadId: found.id } });
14
15	return res.status(200).json({ ok: true, redactedAt: found.redactedAt });
16	}

Lawful scraping at scale — Prospect-Pro AI field notes

1. The context

2. The basic rule — what's legal

3. Pipeline architecture

4. Respectful rate-limiting

4.1 Per-source token bucket

4.2 Bounded concurrency

4.3 Exponential backoff on error

5. Honest identification — truthful user-agent

6. Respecting robots.txt

7. Deduplication and data quality

8. Storage and history

9. GDPR for in-scope data

10. Observability

11. Pitfalls to avoid

12. Closing

Turn this reading into a product decision.

Get the next technical notes.

Lawful scraping at scale — Prospect-Pro AI field notes

1. The context

2. The basic rule — what's legal

3. Pipeline architecture

4. Respectful rate-limiting

4.1 Per-source token bucket

4.2 Bounded concurrency

4.3 Exponential backoff on error

5. Honest identification — truthful user-agent

6. Respecting robots.txt

7. Deduplication and data quality

8. Storage and history

9. GDPR for in-scope data

10. Observability

11. Pitfalls to avoid

12. Closing

Turn this reading into a product decision.

Get the next technical notes.