SEO 101 for Fishing Guides
Builds guide sites for a livingPublished July 21, 2026Last updated July 26, 202612 min read
- Search runs in three stages and Google states that none of crawling, indexing or serving is promised.
- Google states plainly that payment cannot buy a faster crawl or a higher rank, and that anybody saying otherwise is wrong.
- An unreachable robots.txt means the crawler must assume complete disallow; a merely missing one opens the site.
- The specification says robots.txt is not access authorization, and that listing a path exposes it publicly.
- Indexing clusters similar pages and picks one canonical, so ten templated town pages produce one entry and nine alternates.
If your web server returns a 500 error on one specific file, every well-behaved crawler on the internet is required to treat your entire site as forbidden. Not slowed down. Forbidden.
The file is robots.txt, the requirement is in an IETF standards-track document, and the word in the specification is MUST. That single provision is worth more to a working guide than most of what gets sold as search optimisation, because it is the difference between a site that exists and one that quietly does not. What follows is the actual machinery, in three stages, from the two organisations that publish how it works: what a crawler does, what it is allowed to do, and which parts of the outcome nobody can buy. The neighbouring material is indexed on the getting-booked hub.
| Stage | What happens | Promised |
|---|---|---|
| Crawling | Text, images and video are downloaded | No |
| Indexing | The page is analysed and stored | No |
| Serving | Results are returned for a query | No |
What are the three stages?
Crawling, indexing and serving, and not every page survives them.
Google describes Search as a fully automated search engine using software known as web crawlers that explore the web regularly to find pages to add to its index.
It states that the vast majority of pages listed in results are not manually submitted for inclusion but are found and added automatically as those crawlers explore.
Crawling is described as downloading text, images and videos from pages found on the internet with automated programs.
Indexing is analysing the text, images and video files on the page and storing the information in the index, described as a large database.
Serving is returning information relevant to a user's query when somebody searches.
The documentation is explicit that not all pages make it through each stage, which is the sentence most SEO advice skips.
Google publishes it as its in-depth guide to how Search works.
What carries no promise?
All three stages, stated plainly and more than once.
The guide says Google offers no guarantee that it will crawl, index or serve a page, even where the page follows the Search Essentials.
It repeats the point at the indexing stage, saying there is no guarantee of indexing and that not every page Google processes will be indexed.
That is a more honest position than most of the industry adopts on its behalf, and it changes what a reasonable expectation looks like.
A guide with eight pages should expect some of them to sit outside the index, and should treat that as normal rather than as a fault to be chased.
What it also means is that any promise of inclusion, from anybody, is a promise about something the operator of the system declines to promise.
Which is a useful test to apply to a proposal before signing it.
The map-side version of the same problem is worked through in the map pack piece.
What cannot be bought?
Crawl frequency and rank, and the wording leaves no room.
Before explaining anything else, the guide states that Google does not accept payment to crawl a site more frequently, or rank it higher, and adds that if anyone tells you otherwise, they are wrong.
It repeats the point at the serving stage, saying Google does not accept payment to rank pages higher and that ranking is done programmatically.
Those two sentences dispose of an entire category of sales conversation, and they are published by the company doing the ranking.
What can legitimately be bought is advertising, which is a separate and clearly labelled part of the results page.
What cannot be bought is the position of an ordinary result, at any price, through any relationship.
Anybody offering it is describing something the documentation says does not exist.
The directory equivalent of the same offer is assessed in the directories piece.
How does Google find a page at all?
Three routes, and one of them is yours to control.
The documentation notes there is no central registry of all web pages, so Google must constantly look for new and updated pages, a process it calls URL discovery.
Some pages are known because Google has already visited them.
Others are discovered when Google extracts a link from a known page to a new page, giving the example of a hub page such as a category page linking to a new blog post.
Still others are discovered when you submit a list of pages, a sitemap, for Google to crawl.
For a guide with a small site, the middle route is the one that matters, because it means an unlinked page is an undiscovered page.
A trip page nothing links to is, from the crawler's point of view, not there.
How the pages should link to each other is set out in the water and city pages piece.
What stops a crawl?
Three things, and two of them are your hosting.
The documentation lists common issues with crawler access as problems with the server handling the site, network issues, and robots.txt rules preventing access.
It also explains that crawlers are programmed not to crawl a site too fast, to avoid overloading it, and that the mechanism responds to what the site returns, giving the example that HTTP 500 errors mean slow down.
So a site that intermittently errors is teaching the crawler to visit less often, which is a hosting problem presenting as a search problem.
The guide also notes that during the crawl Google renders the page and runs any JavaScript it finds, using a recent version of Chrome, because sites often rely on JavaScript to bring content onto the page.
A page whose content only appears after a script runs is a page whose content depends on that rendering step succeeding.
None of which is exotic. It is the argument for boring, fast, server-rendered pages.
The on-page side of that is described in the on-page basics piece.
What is the file that can switch a site off?
robots.txt, and it is a published internet standard.
The Robots Exclusion Protocol was originally defined by Martijn Koster in 1994 and was standardised by the IETF as a proposed standard in September 2022.
The specification requires the rules to be accessible in a file named robots.txt, all lowercase, in the top-level path of the service, encoded as UTF-8 and served as plain text.
It defines a group as one or more user-agent lines followed by one or more rules, terminated by another user-agent line or the end of the file.
Matching must start with the first octet of the path, the most specific match must be used, and where an allow rule and a disallow rule are equivalent the allow rule should win.
Where no rule in a matching group applies, the URI is allowed, and the robots.txt path itself is implicitly allowed.
Crawlers may cache the file but should not use a cached copy for more than twenty-four hours unless it is unreachable.
The standard is published as RFC 9309 by the RFC Editor, with the same document at the IETF Datatracker.
What happens when that file errors?
The two failure modes point in opposite directions.
Where the server responds with a status indicating the file is unavailable, which in HTTP means the four hundred range, the specification says the crawler may access any resources on the server.
Where the file is unreachable because of server or network errors, which in HTTP means the five hundred range, the specification says the file is undefined and the crawler must assume complete disallow.
Read those two together. A missing robots.txt opens the site; a broken one closes it entirely.
The specification softens it slightly, saying that if the file stays undefined for a reasonably long period, given as an example of thirty days, crawlers may treat it as unavailable or keep using a cached copy.
It also asks crawlers to follow at least five consecutive redirects when fetching it, and to impose a parsing limit of at least five hundred kibibytes.
The practical instruction for a guide is short: know what that file says, and know that it returns a two hundred.
Checking that is one of the four checks below.
What the failure actually costs. Suppose a hosting change breaks robots.txt so it returns a server error, and nobody notices for a month. Under the specification every conforming crawler must assume complete disallow for that period, and the specification's own thirty-day example is the point at which they may start treating it as merely unavailable instead. Set against that, checking the file returns a two hundred takes about fifteen seconds and can be done from any browser. The thirty days is quoted from the specification; the month-long delay in noticing is a stated assumption used to describe the shape of the risk, and no figure for lost traffic, rankings or bookings is asserted here for any site in any market.
What does indexing decide?
Which of your near-identical pages is the real one.
After a page is crawled, Google tries to understand what it is about, processing and analysing the text and key content tags and attributes, naming title elements and alt attributes among them.
During indexing it determines whether a page is a duplicate of another page on the internet, or canonical.
The canonical is the page that may be shown in results, and it is chosen by first clustering pages with similar content and then selecting the one most representative of the group.
The others become alternate versions that may be served in different contexts, such as a mobile device or a very specific query.
For a guide with a page per town, written from the same template with the place name swapped, that clustering step is the whole risk.
Ten near-identical pages do not produce ten entries. They produce one canonical and nine alternates.
Writing them so they are genuinely different is the subject of the blogging piece.
Why might an indexed page not appear?
Three published reasons, and none is mysterious.
The documentation lists, for a page reported as indexed but not visible, that the content may be irrelevant to users' queries, that the quality of the content may be low, or that robots meta rules may prevent serving.
It separately lists common indexing issues as low content quality, robots meta rules disallowing indexing, and a site design that makes indexing difficult.
Notice how often quality appears in a document that is otherwise entirely mechanical.
It also notes that relevancy at the serving stage is determined by hundreds of factors, which could include the user's location, language and device.
Its own example is that searching for bicycle repair shops shows different results in Paris and in Hong Kong, which is the same location effect a guide sees every time they check their own rankings.
Which is why checking your own position from your own kitchen tells you very little.
A saner way to track it is described in the rank tracking piece.
Is robots.txt a security measure?
No, and the specification says so in two places.
The protocol definition states directly that these rules are not a form of access authorization.
The security considerations section goes further: the protocol is not a substitute for valid content security measures, and listing paths in the file exposes them publicly and thus makes the paths discoverable.
Read that twice if you have ever disallowed a folder because you did not want people finding it.
The specification's own advice is to use a security measure appropriate to the application layer, giving HTTP authentication as the example for the web.
For a guide the practical version is that a page you genuinely want private needs a password, not a line in a text file that anybody can read.
And the line in the text file is itself a signpost to the thing you were hiding.
The structured data that sits alongside all this is covered in the schema markup piece.
What should a guide actually do?
Six things, and none takes longer than an afternoon.
Confirm robots.txt exists, returns successfully, and says what you think it says.
Make sure every page you care about is linked from somewhere the crawler can already reach, because discovery runs on links.
Submit a sitemap, since the documentation names it as one of the three discovery routes.
Keep the server reliable, because errors teach the crawler to come less often.
Make the pages genuinely different from one another rather than templated, so the clustering step does not collapse them.
Then stop, because the remaining levers are content and links, both of which are slow.
Those links are the subject of the backlinks piece.
Where do guides go wrong here?
Seven ways, and two are invisible until you look.
Never once opening robots.txt, and therefore not knowing what it says.
Disallowing a folder as a privacy measure, which the specification says is not what it is for and which advertises the folder.
Publishing pages nothing links to and waiting for them to appear.
Running a site whose content only exists after JavaScript executes, and assuming that is free.
Producing ten town pages from one template and expecting ten results rather than one canonical.
Buying a service that promises inclusion or position, both of which the documentation declines to promise.
And checking rankings from a single device in a single place, when the documentation says location is among the factors.
The keyword side of the work is enumerated in the keyword list piece.
What do operators not expect?
That the whole thing is documented and free to read.
Two documents, one from Google and one from the IETF, contain the entire foundation, and between them they take under an hour.
The second surprise is how much of the machinery is refusal: no guarantee of a crawl, no guarantee of indexing, no guarantee of a result, and no price that changes any of it.
The third is that a broken file closes the site while a missing one opens it, which is the reverse of most people's intuition.
The fourth is that robots.txt is public, so it is a list of the places you would rather people did not look.
The fifth is that quality appears repeatedly in a document about plumbing, which suggests the plumbing is not where the remaining gains are.
Once the mechanics are right, the work is writing, and the writing is the slow part.
The local layer built on top of these foundations is set out in the local search piece.
How would you check your own site?
Four checks, twenty minutes, from a phone if necessary.
Type your domain followed by a slash and robots dot txt into a browser and read what comes back, including whether it loads at all.
Turn off JavaScript in the browser and reload your key trip page, then see whether the words are still there.
Open your own site map or menu and count how many pages you can reach by clicking from the home page, since anything unreachable is undiscovered.
Then compare two of your town pages side by side and ask whether a person could tell them apart without the place name.
Anything that fails those four is worth fixing before anybody is paid to do anything more sophisticated.
Nothing in that list requires a tool, a subscription or an agency.
The profile that sits beside the site is set up in the profile setup piece.
The foundations, in order
Reachable, discoverable, distinct, then patient.
Check robots.txt loads and permits what you intend, remembering that a server error there means complete disallow.
Never treat that file as privacy, because the specification says it is not access authorization and that listing a path exposes it.
Link every page you care about from somewhere already known, and submit a sitemap as well.
Keep the hosting boring and reliable, since errors slow the crawl by design.
Make sure the words exist without JavaScript having to run first.
Write pages that are genuinely different from each other, so clustering does not pick one and shelve the rest.
Expect some pages never to be indexed, because the documentation says as much.
And ignore anybody selling crawl frequency or position, on the authority of the people who do the crawling.
No traffic or ranking figures appear here. This page states no traffic number, ranking position, index rate, click share or booking effect for any site, guiding or otherwise, in any market, and nothing has been estimated where such a figure would sit. It does not describe how ranking works beyond what the documentation itself publishes. Google's documentation was read on 26 July 2026 and carries a stated last update of 18 December 2025; it is revised without notice. RFC 9309 is a proposed standard published in September 2022 and the RFC Editor records that errata exist for it, so read the current document rather than this summary. Nothing here is legal advice.
How this was checked. The three-stage account is quoted from Google Search Central, In-depth guide to how Google Search works, at developers.google.com/search/docs/fundamentals/how-search-works, read in full on 26 July 2026, page footer showing a last update of 18 December 2025. Quoted from it: that Search is a fully automated search engine using software known as web crawlers that explore the web regularly to find pages to add to the index; that the vast majority of pages listed in results are not manually submitted but found and added automatically; that Google does not accept payment to crawl a site more frequently or rank it higher, and that if anyone tells you otherwise they are wrong; that Google gives no guarantee it will crawl, index or serve a page even if the page follows the Search Essentials; the three stages of crawling, indexing and serving with the note that not all pages make it through each stage; that there is no central registry of all web pages and that finding new and updated pages is called URL discovery, with the three routes of prior visits, links extracted from known pages, and submitted sitemaps; that Googlebot uses an algorithmic process to determine which sites to crawl, how often and how many pages to fetch; that crawlers try not to crawl a site too fast to avoid overloading it, based on the responses of the site, with HTTP 500 errors given as meaning slow down; that Google renders the page and runs any JavaScript it finds using a recent version of Chrome; that common crawl access issues are server problems, network issues and robots.txt rules; that indexing includes processing and analysing textual content and key content tags and attributes such as title elements and alt attributes; that duplicate and canonical determination proceeds by clustering pages with similar content and selecting the most representative, with the others becoming alternate versions served in different contexts; that there is no guarantee of indexing and not every page processed will be indexed; that common indexing issues are low content quality, robots meta rules disallowing indexing, and website design; that Google does not accept payment to rank pages higher and ranking is done programmatically; that relevancy is determined by hundreds of factors which could include the user's location, language and device, with the bicycle repair shops example contrasting Paris and Hong Kong; and that a page reported as indexed may not appear because the content is irrelevant to queries, the quality is low, or robots meta rules prevent serving. The robots.txt provisions are quoted from RFC 9309, Robots Exclusion Protocol, a proposed standard on the IETF standards track published September 2022 by Koster, Illyes, Zeller and Sassman, read in full at the RFC Editor on 26 July 2026. Quoted from it: that the protocol was originally defined by Martijn Koster in 1994; that these rules are not a form of access authorization; that a group is one or more user-agent lines followed by one or more rules, terminated by a user-agent line or end of file; that matching must start with the first octet of the path, the most specific match must be used, and where an allow rule and a disallow rule are equivalent the allow rule should be used; that where no match is found in a matching group the URI is allowed and the robots.txt URI is implicitly allowed; that the rules must be accessible in a file named robots.txt, all lowercase, in the top-level path of the service, UTF-8 encoded and of media type text plain; that crawlers should follow at least five consecutive redirects; that where the server indicates the file is unavailable, in HTTP the four hundred range, the crawler may access any resources on the server; that where the file is unreachable due to server or network errors, in HTTP the five hundred range, the file is undefined and the crawler must assume complete disallow; that if the file remains undefined for a reasonably long period, given as an example of thirty days, crawlers may assume it unavailable or continue using a cached copy; that crawlers should not use a cached version for more than twenty-four hours unless the file is unreachable; that the parsing limit must be at least five hundred kibibytes; and, from the security considerations, that the protocol is not a substitute for valid content security measures, that listing paths in the file exposes them publicly and makes them discoverable, and that access should be controlled by a measure appropriate to the application layer, HTTP authentication being the example given. The formal ABNF syntax, the special character table, the percent-encoding examples and the worked longest-match examples in that document were read but are not reproduced here. The arithmetic panel combines one quoted period with a stated assumption and contains no measurement. Nothing here is legal advice.
If your booking calendar has more open weeks than you’d like, I’ll build you a free preview of your booking site before you pay a cent.
Get a free website previewThe machinery, in three stages, and what it refuses to promise
What are the three stages of Google Search?
Crawling, indexing and serving. Google describes Search as a fully automated engine using software known as web crawlers that explore the web regularly to find pages to add to its index, and notes that the vast majority of pages listed in results are not manually submitted but found and added automatically. Crawling is downloading text, images and videos from pages found on the internet. Indexing is analysing those files and storing the information in the index, described as a large database. Serving is returning information relevant to a query. The documentation is explicit that not all pages make it through each stage, which is the sentence most advice skips.
Is my page certain to appear in Google?
No. The documentation says Google offers no guarantee that it will crawl, index or serve a page, even where the page follows the Search Essentials, and it repeats the point at the indexing stage by saying there is no guarantee of indexing and that not every page Google processes will be indexed. That is a more honest position than most of the industry adopts on its behalf. A guide with eight pages should expect some of them to sit outside the index and treat that as normal. It also means any promise of inclusion is a promise about something the operator of the system declines to promise.
Can anybody pay for a faster crawl or a better rank?
No, and the wording leaves no room. Before explaining anything else the documentation states that Google does not accept payment to crawl a site more frequently, or rank it higher, and adds that if anyone tells you otherwise, they are wrong. It repeats the point at the serving stage, saying Google does not accept payment to rank pages higher and that ranking is done programmatically. What can legitimately be bought is advertising, which is a separate and clearly labelled part of the results page. What cannot be bought is the position of an ordinary result, at any price, through any relationship.
How does Google find a page in the first place?
Three routes, called URL discovery. The documentation notes there is no central registry of all web pages, so Google must constantly look for new and updated ones. Some pages are known because Google has already visited them. Others are discovered when Google extracts a link from a known page to a new page, the example given being a hub page such as a category page linking to a new blog post. Still others are discovered when you submit a sitemap. For a small guiding site the middle route matters most, because it means an unlinked page is an undiscovered page: a trip page nothing links to is, from the crawler's point of view, not there.
What can a broken robots.txt do?
Close your entire site to conforming crawlers. RFC 9309, the IETF proposed standard published in September 2022, distinguishes two failures. Where the server indicates the file is unavailable, meaning the four hundred range in HTTP, the crawler may access any resources on the server. Where the file is unreachable because of server or network errors, meaning the five hundred range, the file is undefined and the crawler must assume complete disallow. So a missing robots.txt opens the site and a broken one closes it. The specification adds that if it stays undefined for a reasonably long period, giving thirty days as an example, crawlers may treat it as unavailable or keep a cached copy.
Can I use robots.txt to hide a page?
No, and the specification says so twice. The protocol definition states directly that these rules are not a form of access authorization. The security considerations section goes further: the protocol is not a substitute for valid content security measures, and listing paths in the file exposes them publicly and thus makes the paths discoverable. Its own advice is to use a security measure appropriate to the application layer, giving HTTP authentication as the web example. For a guide the practical version is that a page you genuinely want private needs a password, not a line in a public text file that also signposts what you were hiding.
Why do my ten town pages not produce ten results?
Because of the canonical step. During indexing Google determines whether a page is a duplicate of another page on the internet or canonical, and it selects the canonical by first clustering pages with similar content and then choosing the one most representative of the group. The others become alternate versions that may be served in different contexts, such as a mobile device or a very specific query. For a guide with a page per town written from one template with the place name swapped, that clustering is the whole risk: ten near-identical pages produce one canonical and nine alternates rather than ten entries.
Sources & methods
- In-depth guide to how Google Search works, Google Search Central
- RFC 9309, Robots Exclusion Protocol (RFC Editor)
- RFC 9309 at the IETF Datatracker
Every figure here is traced to a named public source and checked against it. Licensing, tax, and fee rules change. Verify your state’s current rules with the agency directly before you count on any number here.
Evan Knox
I build booking websites and run the ads and search for owner-run fishing guides, one operation per stretch of water. My first guide client, Bowman Fly Fishing, grew its revenue 4x in a year from that work. Field Notes is where I put the straight numbers on the business of guiding.
More field notes
You run the water. I'll handle getting you found.
I'm Evan, and SEO is exactly what I do for guides: I build booking sites and run the search and ads for owner-run guide operations, one operation per stretch of water, so anglers find and book you first. If you would rather fish than fight Google, text me at (470) 777-9686 and I'll build you a free preview of your site before you pay a thing.