The rules for fetching someone else's recipes
Most of the recipes in my private collection were written by someone else. Recipe bloggers and meal-kit companies spend real time on their text and photographs, and a script that copies them is taking something. I wanted rules for that, and I wanted the rules to hold on a day when I am tired and in a hurry. The project is described in a private recipe database in three repos.
I come from compliance work, so my instinct is that a control enforced by the system beats a policy I am supposed to remember.
Scraped text cannot be published
The recipe table has a column, ImportedFromUrl, that is set when the text was fetched verbatim
from a website. It also has a flag, IsPublished, that means "may this leave the house." A
CHECK constraint ties them together:
CHECK (ImportedFromUrl IS NULL OR IsPublished = 0)
If a recipe has a source URL, it cannot be marked published. The database refuses the row. Any export that only reads published recipes therefore cannot contain fetched text, however I get the flags wrong.
The obvious way around it is to clear the URL and then publish. That would work, and it would be the worst thing I could do with the column, because the URL is the record of whose words these are. Clearing it does not change the words. It only erases the evidence. The comment in the schema says the legitimate reason to clear it is that I rewrote the recipe in my own words, and that this is a deliberate act. That is the only exit I wanted to leave.
Showing my own collection is not publishing
The first version had one flag, and it caused trouble when I built the web app. The site is reachable only from my home network, behind an internal certificate authority. Showing my own recipe box to my own household is not the same act as putting it on the internet, but a single flag could only answer one question.
So there are two. IsPublished keeps its strict meaning and its constraint. A second flag,
IsHouseholdVisible, means "may appear on my own site" and defaults to on, since the whole
collection is for this household. A second constraint says nothing can be published while
hidden at home. The web app reads the household flag. Nothing was weakened, a looser gate was
added beside the strict one.
Read robots.txt as the agent that obeys it
The fetcher checks robots.txt before touching a page. For a while it used Python's built-in
urllib.robotparser, whose read() method fetches the file through urllib. That identifies
itself as Python-urllib, not as my tool.
On a site behind a CDN, those are two different visitors. I measured it on one recipe blog: the
same URL, in the same second. My fetcher, using its own User-Agent, got a 200. urllib got a
403. The parser treated the 403 as "the whole site is disallowed," which is correct under the
robots.txt standard, and the tool told me the site forbade me. It did not. The site was serving
me its rules and permitting the page.
That error was in the safe direction, but it was still an error, and the kind that ends up in a notes table as though the site had said it. Now the file is fetched with the same HTTP library and the same User-Agent that will fetch the page, then handed to the parser. The lesson I took is to read the rules as the visitor who has to follow them.
A server error is not an answer
The fix also forced me to write down what each status means. A 200 means parse the rules. A 401 or 403 means treat the whole site as disallowed. Any other 4xx means no robots.txt exists, so nothing is restricted. A 5xx, or a request that never completes, means the rules are unknown.
Unknown is its own state, not allowed and not disallowed. Before the fix, a 5xx left both internal flags false, and that read as a disallow, contrary to what the docstring claimed. The skip message now says which fact it found, because "the file returned 403" and "a rule names this path" are different things. When a site refuses me, the message points to saving the page by hand and loading that, never to an option that overrides the rule.
Two small bugs from the same day
Some recipe pages publish their data as JSON-LD, which scrapers read. One WordPress plugin, WP Recipe Maker, has a print template with the full recipe and no JSON-LD at all. On the one site I measured, it was the only page a script could reach. The fetcher now falls back to it, but only when the normal route finds no title, so existing sites are not quietly re-read a new way.
Durations were worse. The code took the first integer in a string, so "1 hour" became one minute. It was not an error and not a null, just a plausible wrong number in the column the time filter reads. It now sums the units. I checked whether it had already damaged anything, and it had not, since no current source wrote durations as prose.
What I took from it
Some of these rules protect other people, and some protect me from my own tools. The ones that mattered most were the ones the database and the fetcher enforce. The honest exception is the household flag, which is a decision I made, and it lives in a column I can change.