This article was co-authored with generative AI. Facts have been checked against public documentation where feasible, but errors may remain. Please verify primary sources before relying on this for important decisions.

TL;DR

In my previous post, A record of handling a 10M PV/month bot scraping attack with AWS WAF, I blocked access with a Singapore Geo Block
One week later, the attacker pivoted through HK (Tencent Cloud) → VN (residential dispersion) → ID (Telkom Indonesia) → DE (Tencent Cloud Frankfurt)
WAF log analysis showed that 90% of the traffic was concentrated on /sparql and /snorql/ (the Linked Data API)
I gave up on the piecemeal approach of blocking one country at a time and switched to a Geo allowlist of 45 developed countries + default deny
I then added the observed Tencent/Alibaba CIDRs to an IPSet → but a DE pivot succeeded through a CIDR I had missed, so I ultimately moved to pulling every prefix based on ASN (Autonomous System Number)
Lesson: "manually listing observed IPs as CIDRs in an IPSet" structurally leaks. The right answer is to pull ASN announcements from a BGP (Border Gateway Protocol) feed

One Week Since Last Time

At the end of my previous article, I wrote:

Attack and defense are a cat-and-mouse game, but the "one level up" preparation is to automate re-detection with CloudWatch alarms

I had set up SNS notifications (waf-blocked-attack-sources-spike, waf-sg-allowed-spike) and established an operational foundation for detecting recurrence.

But the attacker came back through a route I did not expect.

Signs in GA4 Realtime

One day, while watching the by-country breakdown of realtime users in GA4, I noticed that the main property's breakdown was:

Vietnam:    41
Hong Kong:  25
Japan:      16

Japan being third for a Japanese-language content archive is odd.

There had been no notable alerts in the preceding days, so just to be safe I used the GA4 Data API to aggregate the 6 days after the Geo Block was applied (from 4/25 onward) across multiple axes (country / browser / OS / time of day / landing page):

PV/session = 1.04   (2-5 for humans, near 1.0 = bot)
sessions/user = 1.01
average session = 13.1 seconds
bounce rate = 95.1%
Chrome 94% / Windows 84% / Desktop 98.6%
direct (no referer) 92.8%

Exactly the same fingerprint as the previous SG attack. I concluded that the same scraper had pivoted its origin.

By country:

China:     9,009 (44%)  ← Already Geo Blocked (I had added CN since last time)
Vietnam:   2,675 (13%)  ← New
Japan:     1,893 (9%)   ← Legitimate
Hong Kong: 1,086 (5%)   ← New
Singapore:   997        ← Stragglers after the Geo Block

And by day:

4/25:   613 sessions
4/26: 8,949   ← burst
4/27:   321
4/28:   692
4/29: 8,709   ← burst
4/30: 1,184 (in progress)

A pattern of running the scraper in batches every 2-3 days.

Checking the Details in WAF Logs — 100x the Scale of GA4

Because GA4's web stream measurement presupposes gtag.js execution (unless you separately send via the Measurement Protocol), API requests and Linked Data fetches are in principle invisible in GA4. To grasp the full picture of the attack, you need to look at both the WAF logs (aws-waf-logs-<acl-name>) and GA4.

Aggregating by country over the past 3 hours in CloudWatch Logs Insights:

Country	ALLOW	BLOCK
US	245,979	-
VN	149,191	-
CN	-	54,830 (geo-block-cn hits)
JP	30,750	-
HK	27,766	-
ID	19,535	-
DE	18,810	-
IN	18,169	-
SG	-	17,034 (geo-block-sg hits)
BR	16,350	-

VN was 281 sessions/day in GA4, but at the WAF layer it was 150,000 req in 3 hours. The three-order-of-magnitude difference shows that the majority is API scrapers that do not execute JS.

Checking the top IPs behind US's 240,000 ALLOWs, most were 66.249.66.x (Googlebot) and 40.77.x / 52.167.x / 207.46.13.x / 157.55.x (Bing/Microsoft). This is legitimate crawling, so I kept it.

Identifying the Target: The Attack Is a Bulk Harvest of All LOD Metadata

When it is hard to distinguish based only on client IP ranges and UAs, the fastest approach is to look at what they are coming to fetch. Aggregating by host header and URI from the WAF logs:

host	uri	Country	reqs (3h)
ld.example.jp	/sparql	VN	89,437
ld.example.jp	/snorql/	US	32,850
ld.example.jp	/sparql	US	23,350
ld.example.jp	/snorql/	VN	23,518
ld.example.jp	/sparql	HK	16,302
ld.example.jp	/snorql/	HK	4,116
ld.example.jp	/sparql	ID	9,354
ld.example.jp	/snorql/	ID	2,765

90% of VN/HK/ID traffic was concentrated on the Linked Data API's SPARQL endpoint. The attacker's true goal was to retrieve all RDF for bibliographic records, people, and related links (on the order of 10M nodes). The HTML frontend's /book/<id> was hit as an afterthought (during the previous SG attack, that was the main target).

Three Origin Patterns

Pulling the actual top IPs, the same attacker was using three types of resources in combination.

(1) HK: Tencent Cloud HK Cloud-IP Concentration Type

119.28.24.33   106 req/3h
129.226.12.157 101
43.132.249.185 100
43.154.198.51   99
43.129.239.15   96
101.32.9.206    96
150.109.48.35   95

119.28.x 129.226.x 43.129-159.x 101.32.x 150.109.x — all Tencent Cloud (AS132203) HK ranges. Each IP does 25-35 req/5min, completely evading the WAF rate limit (5,000/5min). The same type as the previous AWS Singapore attack.

(2) VN: Fully Distributed Residential Type

14.x, 113.x, 116.x, 117.x, 171.x, 183.x  — VNPT, Viettel
Combining the Top 20 IPs still yields only 178 req (0.1% of the total)

Thinly distributed across 1,000+ IPs. Each IP does only a few req. Typical of a residential proxy network (BrightData / Soax / Oxylabs, etc.). IP-based blocking is impossible; Geo Block is the only option.

(3) ID: A Single Telkom Indonesia /16

163.7.13.x, 163.7.14.x, 163.7.15.x, 163.7.16.x  - each IP 38-62 req/3h

Concentrated in a single range, 163.7.0.0/16. Somewhere between HK and VN.

The Limits of Ad Hoc Geo Blocking

Here I changed my approach.

If I keep doing per-country Geo Blocks like this, the attacker will just pivot to the next country. It came SG → CN → HK → VN → ID, and next it will be one of IN, PH, BD, NP, BR, MX.

Adding per-country blocks structurally cannot keep up. Organizing the options:

Strategy	Effect	Side effect
Add more per-country Geo Blocks	△ Perpetual chase	Operational cost
AWS WAF Bot Control (managed, paid)	◎ Automatic	$100+/month, tuning false positives
Close only the Linked Data endpoint	△ Pivots to the main site	Limited
default-deny + developed-country allowlist	◎ Robust against future pivots	Legitimate users from non-allowed countries are blocked

Considering the site's mission (a Japanese cultural archive), legitimate access from non-allowed countries is a minority, and those people can be served instead by providing an RDF dump in the future. I chose to accept the trade-off.

Switching to default-deny + a 45-Country Allowlist

The allowlist I ultimately adopted:

East Asia (3): JP KR TW
North America (2):     US CA
Europe (35):    GB IE FR DE IT ES NL BE LU AT CH
              SE NO DK FI IS PT GR PL CZ SK HU
              SI HR EE LV LT RO BG MT CY
              AD MC SM LI VA
Oceania (2):   AU NZ
Middle East (2):   IL AE

Not included:

Asia: CN HK SG VN ID IN PH TH MY BD NP PK
Latin America: BR MX AR
Europe/Russia: RU UA TR
Africa: all

HK and SG are formally developed regions, but I excluded them because Tencent Cloud / AWS cloud IPs have become the attack origin. This is a judgment that their "misuse as bot infrastructure" outweighs their "human access demand."

The WAF Rule Structure

AWS WAF can express "Block anything not in the list" with NotStatement(GeoMatchStatement):

{
  "Name": "geo-allowlist",
  "Priority": 0,
  "Action": {"Block": {}},
  "Statement": {
    "NotStatement": {
      "Statement": {
        "GeoMatchStatement": {
          "CountryCodes": ["JP","KR","TW","US","CA",...]
        }
      }
    }
  },
  "VisibilityConfig": {
    "SampledRequestsEnabled": true,
    "CloudWatchMetricsEnabled": true,
    "MetricName": "geo-allowlist"
  }
}

I deployed this at Priority 0 (highest priority). The existing per-country geo-block-sg/cn/hk/vn rules were subsumed by the allowlist, so I deleted them.

Gotchas When Applying

If you include an empty-string Description or an empty-dict AssociationConfig in update-web-acl --cli-input-json, it fails with:

Parameter validation failed:
Invalid length for parameter Description, value: 0, valid min length: 1

You need to strip the empty fields with jq (del(.Description) | del(.AssociationConfig)) before submitting.

Also, an IPSet's Description cannot use parentheses () (constrained by the regex ^[\w+=:#@/\-,\.][\w+=:#@/\-,\.\s]+[\w+=:#@/\-,\.]$). No Japanese, ASCII only.

The Remaining Problem: "Cloud IPs" Within the Allowlist

After applying the allowlist, I checked country by country within the allowlist to see "what they were hitting," and the following became clear:

Of US's 240,000 ALLOWs, Tencent Cloud US region + Alibaba Cloud US accounted for 50,000-80,000 req
Of DE's 18,000 ALLOWs, Tencent Cloud Frankfurt was the large majority
Of TW's 7,000 ALLOWs, 92% were /sparql and /snorql/ (including Datacamp proxy)
Of JP's 30,000 ALLOWs, 57% were /sparql (some from university research, some scrapers via JP residential)

In other words, even though they are US/DE/JP/TW in GeoIP terms, the reality is Chinese-affiliated operators scraping using different cloud regions. The allowlist does not stop this.

Adding Tencent / Alibaba CIDRs to the IPSet — But They Leaked

Judging that "I can just enumerate the observed IP ranges in the IPSet," I first added 18 entries:

Tencent Cloud (14 entries):
  43.110.0.0/15, 43.130.0.0/15, 43.132.0.0/14,
  43.153.0.0/16, 43.154.0.0/15, 43.159.0.0/16,
  43.166.0.0/15, 43.173.0.0/16, 49.51.0.0/16,
  101.32.0.0/16, 119.28.0.0/16, 129.226.0.0/16,
  150.109.0.0/16, 170.106.0.0/16

Alibaba Cloud (4 entries):
  47.74.0.0/15, 47.76.0.0/14, 47.250.0.0/15, 47.252.0.0/14

Checking the WAF Traffic overview → the block-aggressive-scrapers rule fired 50,000 times / 30 min. It was working.

But 30 minutes later, on re-inspection, 4,890 /sparql ALLOWs from DE were still coming through. Looking at the top IPs:

43.157.1.36   ALLOW   ← !!!
162.62.213.74 ALLOW   ← !!!
43.157.x      ALLOW   many
47.245.x      ALLOW   ← Alibaba
8.209.x       ALLOW   ← Alibaba
8.211.x       ALLOW   ← Alibaba

43.157 is Tencent Cloud Frankfurt, 162.62 is a relatively new Tencent allocation, and 47.245 and 8.208-215 are Alibaba. All of them had leaked through my IPSet.

I added 4 more entries:

43.156.0.0/14   ← 43.156-159, includes Tencent Frankfurt
162.62.0.0/15   ← New Tencent allocation
47.245.0.0/16   ← Alibaba
8.208.0.0/13    ← Alibaba 8.208-215

Lesson: Observation-Based IPSets Structurally Leak

Here is an important reflection.

The approach of "manually listing observed IP ranges in an IPSet" leaks the moment it depends on your own memory and experience.

My procedure:

[Observe] Check top IPs from WAF logs
   ↓
[Judge] Eyeball "this looks like Tencent" / "looks like Alibaba"
   ↓
[Enumerate] List "Tencent's main CIDRs" from memory   ← this is where leaks breed
   ↓
[Deploy] Add to the IPSet

"Listing from memory" is not authoritative. Both Tencent and Alibaba officially BGP-announce their prefixes for their own AS, so if you pull that list you can cover them authoritatively.

ASN-Based Pull Is the Right Answer

Each company's ASN:

ASN	Organization
AS132203	Tencent Cloud (for international, RIPE registration is the China entity)
AS133478	Tencent Cloud Computing (Beijing) (more mainland-oriented, IPv6 includes Singapore)
AS45090	Tencent China (mainland)
AS45102	Alibaba (international)
AS37963	Alibaba China (includes some Singapore prefixes)

How to obtain the full list of announced prefixes for these (RADB whois works even in a CI environment where DNS is not permitted):

# 1. RADB (Routing Assets Database) whois
whois -h whois.radb.net '!gAS132203' > /tmp/asn-132203.txt

# The result is a single line with space-separated prefixes lined up:
# A236332
# 43.162.176.0/22 43.174.144.0/20 43.171.160.0/19 ...
# C

# 2. Hurricane Electric BGP (web-based)
# https://bgp.he.net/AS132203#_prefixes

# 3. RIPE Stat (JSON API)
curl -s "https://stat.ripe.net/data/announced-prefixes/data.json?resource=AS132203"

# 4. bgpview.io API (if DNS is permitted. As of May 2026, its availability seems unstable on some days)
curl -s https://api.bgpview.io/asn/132203/prefixes

The numbers I pulled this time (via RIPE Stat):

ASN	Announced prefixes on RIPE
AS132203 (Tencent)	1,547
AS133478 (Tencent)	8
AS45102 (Alibaba)	514

A total of 2,069. The 18 entries I listed by hand were merely the tip of the iceberg. Because RIPE Stat is based on a snapshot of the RIS feed, the count varies with the timing of retrieval.

The AWS WAF IPSet limit is 10,000 entries, so there is plenty of room. Aggregating with ipaddress.collapse_addresses():

import ipaddress, json, sys
nets = []
for f in sys.argv[1:]:
    with open(f) as h:
        d = json.load(h)
    for p in d.get("data", {}).get("prefixes", []):
        try:
            n = ipaddress.ip_network(p["prefix"], strict=False)
            if n.version == 4:
                nets.append(n)
        except (ValueError, KeyError): pass
collapsed = list(ipaddress.collapse_addresses(nets))
for c in sorted(collapsed, key=lambda n: (n.network_address, n.prefixlen)):
    print(c)

2,069 → 345 entries compressed (adjacent /24s are merged into /22 or /20).

Preventive Additions of Oracle Cloud + Vultr (Toward 96% Defense)

At this point I had covered all of Tencent/Alibaba's BGP prefixes, but the attacker had already switched from AWS to Tencent/Alibaba. What is the next pivot target?

Confidence	Cloud	ASN	Reason
High	Hetzner (DE)	AS24940	Cheapest in Europe, DE is in the allowlist
High	OVH (FR)	AS16276	Same as above
Medium	DigitalOcean (US)	AS14061	US is in the allowlist, $5/month droplet
Medium	Vultr (US/DE/multi-region)	AS20473	Instant deploy, cheap, a scraping regular
Medium	Oracle Cloud	AS31898	Free-tier abuse, close to scraping-only
Low	Akamai Connected Cloud (formerly Linode)	AS63949	AP-centric, globally distributed

Of these, I decided to add Oracle Cloud and Vultr preventively. Reasons:

Both are free/dirt-cheap and have a strong "scraping-only cloud" character
The probability that a legitimate cultural-archive viewer comes from here is essentially zero
The side-effect (collateral damage) risk is extremely low

On the other hand, I held off on Hetzner / OVH / DigitalOcean. These may have:

SPARQL clients hosted by universities and research institutions
OSS federated query tools
Individual developers' research servers

and other legitimate use cases, so I will add them on an observation basis.

The IPSet After Application

ASN	Aggregated prefix count
Tencent (AS132203 + AS133478)	~340
Alibaba (AS45102)	~5
Oracle Cloud (AS31898)	~200
Vultr (AS20473)	~1,500 (many /24 subdivisions across multiple regions)
Standalone IPs (pre-existing scrapers)	3
Total	2,058

There is plenty of room up to the WAF IPSet limit of 10,000. There is also room to add Hetzner / OVH / DigitalOcean later.

"Is This Complete?" — No, It Is Structurally Impossible

ASN-based pull is powerful, but it does not reach complete defense.

Remaining Holes

1. Other Cloud Providers

AS24940  Hetzner (DE)         ← Attacker could pivot, on hold
AS16276  OVH (FR)             ← Same as above, on hold
AS14061  DigitalOcean (US)    ← On hold
AS63949  Akamai/Linode (US)   ← On hold

These have a higher risk of legitimate use (universities, research, OSS tools), so I will add them on an observation basis. Since I preventively added Oracle / Vultr today, I will deal with them sequentially the next time a new pivot arrives.

Caution:

AS16509 (AWS): the origin lives here, cannot be casually blocked
AS15169 (Google): Googlebot is here, absolutely never block
AS8075 (Microsoft): Bingbot is here, never block

ASN-based approaches also require human judgment to determine "which ASNs are safe to block."

2. New ASN Acquisitions

If Tencent / Alibaba open a new region next year under a different ASN, it will leak at that point. An operation of re-fetching the ASN list monthly is necessary.

3. Residential Proxy Providers

BrightData, Soax, Oxylabs, IPRoyal: rent millions of residential IPs to scrape
Advertised as legitimate ISP IPs of VNPT/Viettel/NTT/KDDI, etc., so ASN blocks cannot cut them off
The realistic countermeasure: AWS WAF Bot Control (paid)

4. Future Emerging Clouds

The mainstream threat in 5 years may differ from now.

The Spectrum of Completeness

0%      No Tencent/Alibaba observation, do nothing
       ↓
50%     Add observed IPs as /32
       ↓
80%     Add main CIDRs on an observation basis
       ↓
85%     Notice leaks and add
       ↓
95%     Pull all Tencent/Alibaba ASN prefixes
       ↓
96%     ★ Also preventively add Oracle / Vultr ASNs ← today's achievement point
       ↓
98%     Also pull Hetzner / OVH / DigitalOcean ASNs (on hold, needs observation)
       ↓
99%     Introduce AWS WAF Bot Control (handles residential proxies)
       ↓
100%    Unachievable (fully handling emerging clouds / residential proxies is structurally impossible)

Today's achievement point is 96%. It is "not complete" but "the most reasonable right now." The remaining 4% has a large side-effect and cost trade-off, and is overinvestment under the current threat model.

The Web ACL Configuration After Application

The final form:

Priority	Rule	Action
0	geo-allowlist (NOT in 45 countries)	Block
5	block-aggressive-scrapers (IPSet, Tencent + Alibaba CIDRs)	Block
6	block-attacker-ja3 (JA3 fingerprint)	Block
7	block-bot-uas (UA regex)	Block
10	AWSManagedRulesCommonRuleSet	Count override on some
20	AWSManagedRulesKnownBadInputsRuleSet	Block
30	AWSManagedRulesAmazonIpReputationList	Block
100	rate-limit-per-ip (5,000/5min)	Block

The old geo-block-sg/cn/hk/vn were deleted because they were subsumed by the allowlist.

Aside: Could Cloudflare Have Blocked This by Default?

So far I have hand-built a two-tier defense with AWS WAF + IPSet, but the question "would switching to Cloudflare have been much easier?" remains. Let me organize what the reality is.

Cloudflare's Bot-Countermeasure Tiers

For the latest pricing, refer to the official Cloudflare plans page. Here I only organize the tier structure of the bot-countermeasure features.

Plan	Anti-bot features
Free	Bot Fight Mode (basic) — issues challenges to simple bots and headless browsers from cloud hosts
Pro	Super Bot Fight Mode — blocks automated bots by ML score, keeps verified bots (Googlebot/Bingbot)
Business	Super Bot Fight Mode (advanced version including sophisticated bot detection)
Enterprise	Full Bot Management, JA4 fingerprint, ASN scoring, custom rules

The Pro plan's Super Bot Fight Mode appears to be able to block roughly 70-80% of this attack by default. Specifically:

Automatic cloud-host IP detection: scores Tencent / Alibaba / Oracle / Vultr / Hetzner, etc. as "likely automated" → can Block
Automatic verified-bot allow: Googlebot / Bingbot / IndexNow, etc. pass through without ML
JA3 / TLS fingerprint: identifies headless Chrome
Cookie / JS challenge: triggers as needed (this attack's traffic should fail almost 100%)
IP threat score: also detects residential proxy providers (BrightData, Datacamp, etc.), reflected by ML

In other words, Cloudflare Pro takes over the whole job of "listing the Tencent/Alibaba CIDRs yourself" in a managed way.

Should You Still Switch from AWS to Cloudflare?

In the short term it is not worth switching — that is the honest answer:

Aspect	AWS WAF (current)	Cloudflare Pro
Monthly estimate (at this scale)	Only the basic WAF fee	Pro plan fee (see official)
Handling Tencent/Alibaba	Build the IPSet yourself (done today)	Effective by default
Fine-grained behavior control	Fully manual (CIDR / Country / UA / JA3 / rate)	Pro leaves it to ML, Business+ for custom rules
WAF + CDN integration	Tightly coupled with CloudFront	Cloudflare itself is the CDN
Migration cost	-	DNS switchover, SSL reconfiguration, origin protection design, discarding existing IPSet/WebACL, rewriting CloudFront-related items
Management UI transparency	All rules are visible	"Bot Fight Mode" scoring is a black box

The biggest factor is the migration cost. The current AWS configuration has concrete artifacts built up since April: CloudFront origin verification (X-Origin-Verify header), SNS alerts, Logs Insights queries, IPSet auto-update scripts, and more. Replacing these with Cloudflare would be a full from-scratch rewrite.

Conclusion: "Cloudflare Does It by Default" Is Half True, Half a Lie

If you are launching a new site from scratch, Cloudflare Pro is strongly recommended (80% of the work I did today appears to be taken over as managed)
Whether to switch a site already operating on an AWS stack becomes a cost-benefit calculation that includes the migration cost. At this scale (10M PV), a few days to a week of work = several hundred thousand yen of labor cost against a monthly difference of only a few tens of dollars, so it seems likely to take a long time to recoup and hard to call rational
If you continue on AWS, the procedure in this article (allowlist + ASN-based IPSet) can achieve most of the defense equivalent to Cloudflare Pro within the basic AWS WAF fee

In terms of "ease," Cloudflare; in terms of "optimizing on top of an existing stack," AWS WAF + this article's procedure — that is the division.

Incidentally, What About Paid AWS WAF Bot Control?

AWS WAF also has a paid managed rule called AWSManagedRulesBotControlRuleSet. In addition to a $10/month subscription per Web ACL, per-request charges apply (see the official AWS WAF pricing for the latest).

Common Bot Control has a free tier for the first 10M req/month, so at the 10M PV scale, additional charges basically do not occur. On the other hand, if you enable Targeted Bot Control (comprehensive judgment of ASN/JA3/behavior, equivalent to Cloudflare's Super Bot Fight Mode), the free tier is up to 1M req/month, and overage charges apply to the portion exceeding 10M req. It is an option for the use case of "I do not want to leave the AWS stack but want managed bot detection," but the manual build in this article can also be kept within the basic fee, and it was practical at the scale of a cultural archive.

Reflection: The Scraper Is Choosing to Be "Ill-Mannered"

In the LOD community, crawling an entire SPARQL endpoint is considered technically inferior. Because:

CONSTRUCT queries to /sparql are heavy, one request at a time
If you want all the RDF, you can download a bulk dump (gzipped TTL/N-Triples) and be done in one file
Bulk dumps benefit from CDN cache, so the operator's load is also almost zero

In other words, a decent scraper first checks for the presence of void:dataDump, and even if there is none, reads robots.txt and asks about the existence of a dump via the contact form.

This time's attacker did the opposite:

Ignored robots.txt
Spoofed the User-Agent as Chrome/Windows
Distributed across 1,000+ IPs (evading rate-limit detection)
Ran flat around the clock
Did not retain cookies/sessions
Repeated direct GETs (ignoring cache headers)

This is not "not knowing the manners"; it is deliberate detection evasion. Operating by using commercial residential proxy networks and multiple cloud regions in combination shows clear intent.

Since this is an opponent that cannot be stopped by "requests," in the end only technical blocking works — that is the reality.

The "Well-Mannered Response" We Can Provide

If you cannot expect good manners from the attack side, the royal road is for the operator to prepare a "well-mannered route":

Publish /dump.ttl.gz: provide all RDF in one file. CDN cache, Range Request support
Place a void:Dataset self-description at https://ld.example.jp/.well-known/void
State the license clearly: if CC0 or CC-BY, on the premise that it will be copied openly
An update feed (RSS/Atom) to notify "when the dump was updated"

With this, technically sophisticated scrapers (including LLM training pipelines) find it overwhelmingly cheaper to come fetch the dump, so they naturally stop full SPARQL crawling. You can create the structure where "people who scrape ill-manneredly are, from the start, technically inferior people."

Lessons (Addendum to Last Time)

1. "It Stopped with a Block" Is Short-Term Memory. Always Watch for Pivots

Previously peaked 4/14 → SG Geo Block applied 4/24. After about a week of quiet, it resumed from VN/HK/ID/DE on 4/30. The attacker did not "give up because it stopped" but was "preparing the next resource." In addition to alarms, weekly human review of whether GA4's by-country ratios have broken down is necessary.

2. You Need Both WAF Metrics and GA4

Because GA4's web stream measurement presupposes JS execution, unless you use the Measurement Protocol, API scraping is transparent in GA4. This time too, VN was 281 sessions/day in GA4, but pulling the WAF logs showed 150,000 req in 3 hours coming to the API layer. It is essential to operate in two tiers — GA4 as "the first detector of anomalies," WAF logs as "the device to confirm the reality" — because with only one, you misjudge the scale of the attack.

3. The Limits of Per-Country Blocking and the Judgment to Switch to default-deny

Per-country blocking works for the first 1-2 times. From the third onward, it is the decision point of "chase forever / or flip to an allowlist." This time I switched before adding the fifth country's block. It is difficult for SaaS / business-type sites, but for a site whose main purpose is content delivery, an allowlist is surprisingly realistic.

4. "Manually Enumerating Observed IPs in an IPSet" Structurally Leaks

This is the biggest lesson this time. Visually identifying cloud IPs → listing from memory is no good. Unless you pull ASN announcements from an authoritative source (BGP feed), newly allocated CIDRs will certainly leak.

The right answer is:

ASN → get all announced CIDRs from BGP feed → aggregate → deploy to IPSet
(re-fetch monthly for differential updates)

5. The Allowlist Also Leaks — via Cloud IPs + Commercial Proxies

What a country allowlist cannot fully handle is "cloud regions or commercial proxies whose exit is an allowed country." Tencent Cloud Frankfurt is DE in GeoIP, and Datacamp / BrightData / Oxylabs have exits in every country. A two-tier defense of allowlist (Country) and IPSet (ASN/CIDR) is essential.

6. For LOD Endpoints, "Providing the Correct Alternative Route" Over Defense

Scraping attacks have a distant cause in "not providing a dump." If RDF can only be obtained via SPARQL, the one and only choice for a malicious scraper is full SPARQL crawling. Publishing a dump to make "the state where well-mannered people can use it" is the operator's proper responsibility.

Summary

4/14 SG attack peak, 920,000 sessions/day
4/24 SG Geo Block applied, blocked
4/25-29 quiet period
4/30 pivot attack begins from VN/HK/ID — corroborated with WAF logs
4/30 switched to default-deny + 45-country allowlist
4/30 added Tencent/Alibaba CIDRs to the IPSet (18 entries) → leaks discovered
4/30 changed course to ASN-based full-prefix pull (95% coverage)
Going forward  - provide dump.ttl.gz
      - monthly automatic ASN list update
      - add other cloud ASNs (observation-based)
      - evaluate AWS WAF Bot Control (residential proxy support)

Whether per-country Geo Block or an allowlist is the right answer depends on the site, but if these three are all present:

Content is aimed at Japanese/a specific language/a specific region
There is no strict need to open the API/RDF to the entire world
You have experienced 2 or more pivots of an attack

then I think it is worth seriously considering going to an allowlist. Your visibility improves dramatically.

And if you use an IPSet in combination, build it ASN-based from the start. The strategy of manually listing CIDRs on an observation basis leaks with high probability, as in this article.

Appendix A: How to Investigate WAF Logs (Collection of Logs Insights Queries)

In CloudWatch Logs Insights, targeting aws-waf-logs-<acl name>.

By-Country Allowed/Blocked List

fields @timestamp
| stats count() as reqs by httpRequest.country, action
| sort reqs desc
| limit 50

The Contents of a Specific Country

filter httpRequest.country in ["VN","HK","ID","DE"]
| stats count() as reqs by httpRequest.headers.0.value, httpRequest.uri
| sort reqs desc
| limit 30

httpRequest.headers.0.value is typically the Host header.

The Origins of /sparql

filter httpRequest.uri="/sparql" or httpRequest.uri="/snorql/"
| stats count() as reqs by httpRequest.country, httpRequest.clientIp
| sort reqs desc
| limit 30

Breakdown of Block-Firing Rules

filter action="BLOCK"
| stats count() as reqs by terminatingRuleId
| sort reqs desc
| limit 20

Appendix B: Obtaining ASN Announced Prefixes

⚠️ Source Selection Matters — Use RIPE Stat, Not RADB

At first I obtained them with RADB whois (whois -h whois.radb.net '!gAS132203'):

ASN	RADB	RIPE Stat
AS132203 (Tencent)	15,047	1,547
AS133478 (Tencent)	543	8
AS45102 (Alibaba)	50,955	514

Because RADB returns all past registered route objects, it mixes in routes that are already invalid and routes registered by third parties. In fact, the RADB data contained ranges that Tencent/Alibaba do not actually own, such as 43.0.0.0/9 (a /9 block that includes NTT Communications and others) and 8.128.0.0/10, and deploying these to the IPSet would block legitimate ISPs as collateral.

The correct source is RIPE Stat, which returns the BGP feed as-is (only currently announced routes):

curl -s "https://stat.ripe.net/data/announced-prefixes/data.json?resource=AS132203" \
  | jq -r '.data.prefixes[].prefix' > /tmp/tencent-prefixes.txt

bgp.tools (https://bgp.tools/as/132203) also returns only BGP-active routes, so it can be used the same way.

Aggregation

# /tmp/extract_ripe.py
import ipaddress, json, sys
nets = []
for f in sys.argv[1:]:
    with open(f) as h:
        d = json.load(h)
    for p in d.get("data", {}).get("prefixes", []):
        try:
            n = ipaddress.ip_network(p["prefix"], strict=False)
            if n.version == 4:
                nets.append(n)
        except (ValueError, KeyError): pass
collapsed = list(ipaddress.collapse_addresses(nets))
for c in sorted(collapsed, key=lambda n: (n.network_address, n.prefixlen)):
    print(c)

# Get the JSON for the 3 ASNs
for asn in 132203 133478 45102; do
  curl -s "https://stat.ripe.net/data/announced-prefixes/data.json?resource=AS$asn" \
    -o /tmp/ripe-$asn.json
done

# Aggregate
python3 /tmp/extract_ripe.py /tmp/ripe-132203.json /tmp/ripe-133478.json /tmp/ripe-45102.json \
  > /tmp/aggregated.txt

wc -l /tmp/aggregated.txt
# → 345 lines (all Tencent + Alibaba prefixes aggregated into 345 CIDRs)

Aggregate in Python

import ipaddress, sys
nets = []
for line in sys.stdin:
    line = line.strip()
    if line:
        try: nets.append(ipaddress.ip_network(line))
        except ValueError: pass
collapsed = list(ipaddress.collapse_addresses(nets))
for c in collapsed: print(c)

cat /tmp/tencent-prefixes.txt | python3 aggregate.py > /tmp/tencent-aggregated.txt
wc -l /tmp/tencent-aggregated.txt
# → compressed to a few hundred lines

Appendix C: WAF IPSet Update Commands

# Get (LockToken)
aws wafv2 get-ip-set --name block-aggressive-scrapers \
  --scope CLOUDFRONT --id <id> --region us-east-1 \
  > /tmp/ipset-current.json

LOCK=$(jq -r '.LockToken' /tmp/ipset-current.json)

# Build update payload
jq --arg lock "$LOCK" \
   --argjson new "$(jq -R . /tmp/tencent-aggregated.txt | jq -s .)" \
   '{
     Name: "block-aggressive-scrapers",
     Scope: "CLOUDFRONT",
     Id: "<id>",
     Description: "Tencent + Alibaba ASN announced ranges",
     Addresses: (.IPSet.Addresses + $new),
     LockToken: $lock
   }' /tmp/ipset-current.json > /tmp/ipset-update.json

# Apply
aws wafv2 update-ip-set --cli-input-json file:///tmp/ipset-update.json \
  --region us-east-1