Build any automated news collector and you will hit sites that publish no feed. The awkward part is that the absence of a feed never surfaces as an error. The 404 comes back as HTML, the parser returns zero items, and zero items is indistinguishable from no news. Based on measurements taken on 4 August 2026, here is how to design around that misdiagnosis.
curl on 4 August 2026. Site configurations change, so none of this is a permanent specification. The reproduction steps are at the end of the article.
Starting from "there must be a feed" fails quietly
When building a news collector, the first thing anyone tries is guessing conventional paths like /feed/ or /rss.xml. When it works you are done in minutes. The problem is what comes back when it does not.
Here is what guessing conventional feed URLs against the newsrooms of several Japanese household and food goods manufacturers produced.
| URL tried | Status | Content-Type | Body size |
|---|---|---|---|
www.lion.co.jp/ja/newsroom/feed/ | 404 | text/html | 5,938 B |
www.lion.co.jp/ja/rss.xml | 404 | text/html | 5,938 B |
www.kao.com/jp/newsroom/rss.xml | 404 | text/html | 20,304 B |
jp.pg.com/newsroom/feed/ | 404 | text/html | 170,141 B |
www.unicharm.co.jp/ja/rss.xml | 404 | text/html | 20,163 B |
Look at the body sizes. A 404 does not mean an empty response. Each returned a designed "page not found" HTML document, the largest of them 170,141 bytes. That is bigger than three of the four real feeds measured later in this article.
Judge success on three signals
Check these three in order before you look at the contents. Skip any one of them and you fall into the trap above.
- The HTTP status. Record anything other than 2xx as an immediate failure. Never fold a 404 into "no new items".
- The Content-Type. In these measurements, real feeds returned
application/rss+xmlorapplication/xmlwhile the 404 pages returnedtext/html. Some sites do not set it correctly, so in practice use it as a warning rather than as a hard rejection. - The root element. Does the body start with
<rss>,<feed>or<rdf:RDF>? HTML starts with<!DOCTYPE html>or<html>.
# Capture all three signals in a single request curl -sL --max-time 25 -o /tmp/f.xml \ -w 'status=%{http_code} type=%{content_type} bytes=%{size_download}\n' \ "$FEED_URL" # Check the root element (the first 200 bytes are enough) head -c 200 /tmp/f.xml
"Not found" and "does not exist" are different
Stop before over-claiming here. A 404 from a guessed conventional URL is not proof that a site has no feed. It may just mean your guess was wrong.
The correct way to look is feed autodiscovery in the HTML. Sites that publish a feed normally put a link like this in <head>.
<link rel="alternate" type="application/rss+xml"
href="https://example.com/index.rdf" title="Example Feed">
Here is what counting those links on the newsroom top pages produced.
| Page | Status | Autodiscovery links |
|---|---|---|
www.lion.co.jp/ja/newsroom/ | 200 | 0 |
www.kao.com/jp/newsroom/ | 200 | 0 |
jp.pg.com/newsroom/ | 200 | 0 |
prtimes.jp/ | 200 | 1 |
www.ryutsuu.biz/ | 200 | 0 |
The pages themselves fetched fine with a 200, and carried no autodiscovery link at all. That is stronger evidence than a failed guess, but it is still not proof of absence. In fact www.ryutsuu.biz/ advertises no feed on its top page, yet /feed returned 200 with 50 items. So no autodiscovery does not mean no feed either.
The accurate claim is therefore limited to this: the conventional paths returned 404 and the newsroom top pages carried no autodiscovery link, so no feed was findable by mechanical means. For collector design that is enough to conclude that a feed-first approach will not cover these sources.
Fallback routes, and what they miss
When the official site has no feed, the practical fallback is a press-release distribution service or a trade publication. Four of them, measured the same day:
| Feed | Status | Content-Type | Items |
|---|---|---|---|
prtimes.jp/index.rdf | 200 | application/xml | 200 |
www.ssnp.co.jp/feed/ | 200 | application/rss+xml | 30 |
www.ryutsuu.biz/feed | 200 | application/rss+xml | 50 |
diamond-rm.net/feed/ | 200 | application/rss+xml | 10 |
Which brings the second trap. An aggregate feed is a window, not the whole stream. For the 200-item feed above, here are the date range it covered and the counts for consumer-goods keywords.
- Date range: 2026-07-29T10:40:41+09:00 to 2026-08-04T09:15:17+09:00 (roughly six days)
- Titles containing each term, out of 200: detergent 0, fabric softener 0, laundry 0, toothpaste 0, shampoo 0, beer 0, coffee 0 (searched with the Japanese terms)
In other words, if you track a specific category by keyword-filtering a general aggregate feed, it is entirely normal for the current window to contain not a single item in that category. All seven terms scored zero. Reading that as "there was no news in that category" is a misdiagnosis. The correct reading is "it was not in this window".
The width of the window matters too. Two hundred items covering about six days means that if your collection interval is slower than the window turns over, whatever falls off the end is invisible forever. Work the required polling interval backwards from the feed's item limit and its throughput.
Split "zero items" into three
Every trap above comes from three different states wearing the same "zero items" face. Record them separately.
| State | What is actually happening | How to tell |
|---|---|---|
| Fetch failed | 404, timeout, rate limit, DNS failure and so on | HTTP status, exceptions, Content-Type, root element |
| Fetched but nothing matched | The feed is fine, but no item matches your filter | Always record the pre-filter count |
| Genuinely no new items | The feed itself is empty, or nothing changed since last time | Pre-filter count is zero, or the latest timestamp is unchanged |
The implementation point is simple: record two counts, before and after filtering. Logging only the post-filter count makes the three indistinguishable.
If you end up scraping a site with no feed
Sometimes the answer is to fetch an HTML listing page periodically and diff it. If so, observe at least the following.
- Read robots.txt and the terms of service. If it is prohibited, do not do it.
- Leave a generous interval. Newsrooms do not update often. There is almost never a reason to poll every few minutes.
- Use conditional requests. Send
If-Modified-SinceorIf-None-Matchand skip the body on a 304. It is kinder to both sides. - Put a contact address in your User-Agent so the site can reach you if something goes wrong.
- Make the diff stable. Key the comparison on something durable such as the article URL or ID, so that a cosmetic HTML change does not mark every item as new. Hashing the rendered body is a fragile choice.
- Do not swallow failures. When a structural change makes your selectors match nothing, raise an error, not zero items.
Checklist
- Does your collection log record status, Content-Type, byte count, pre-filter count and post-filter count?
- Is anything other than 2xx kept distinct from "no new items"?
- Do you know each feed's item limit and window length, and have you derived the polling interval from them?
- Did you check for a feed via autodiscovery, not just by guessing conventional paths?
- Do you fetch one control feed through the same path?
- Can you detect a selector or parser that matched nothing as an error rather than as zero items?
How to reproduce the measurements (4 August 2026)
- For each candidate feed URL, ran
curl -sL --max-time 25 -o /tmp/f.xml -w 'status=%{http_code} ctype=%{content_type} bytes=%{size_download}'and counted<item>and<entry>elements in the saved body - Fetched each newsroom top page and counted
<link rel="alternate">elements whose type contains rss or atom - For the aggregate feed, took the minimum and maximum
<dc:date>and counted titles containing specific terms
All figures are as of the time of measurement. Treat them not as proof that a given company publishes no feed, but as the measured result that no feed was findable by mechanical means.
Frequently Asked Questions
When a feed URL returns 404, is the body empty?
Not necessarily. In measurements taken on 4 August 2026, every 404 from a guessed feed URL came back as text/html, with body sizes ranging from 5,938 bytes to 170,141 bytes. A collector that judges success by whether a body arrived, rather than by the status code, will accept that HTML as a feed.
How should I decide whether a feed fetch succeeded?
Check three things in order: that the HTTP status is 2xx, that the Content-Type is something like application/rss+xml or application/xml, and that the root element of the body is rss, feed or rdf:RDF. Some sites set the Content-Type incorrectly, so use it as a warning rather than as a hard rejection.
How can I confirm that a site has no feed?
A 404 from guessed conventional paths is not proof of absence. Check the HTML for autodiscovery, meaning a link element with rel=alternate whose type contains rss or atom. Even then, feeds do exist on sites that advertise no autodiscovery link, so the strongest honest claim is that no feed was findable by mechanical means.
Can I track a specific category by keyword-filtering an aggregate feed?
You will miss items. An aggregate feed is a window of the most recent entries, not the full stream. A 200-item feed measured on 4 August 2026 covered about six days, and all seven consumer-goods keywords matched zero titles. Reading that zero as an absence of news is a misdiagnosis.