Methodology: How We Collect Guide Data

The federal government maintains written standards for how a statistical survey should be designed, conducted and published. They were issued as a notice of decision in September 2006, they apply to every agency subject to the Paperwork Reduction Act, and they replaced directives that had stood since 1974.

Almost nothing published about the fishing guide trade would satisfy them. That is not an accusation aimed at anybody in particular; it is a statement about what it costs to produce a defensible number and about how rarely anybody in a small trade pays it. This page sets out what those standards ask for, what this site does and does not do, and how to read anybody's figure including ours. No benchmark is asserted here. The rest of the data material is gathered at the guide industry data hub.

What are the federal standards?

A published framework for designing, conducting and disseminating surveys.

The Office of Management and Budget issued revised Standards and Guidelines for Statistical Surveys as a notice of decision published at 71 FR 55522 on 22 September 2006.

The notice states that the guidance applies to all federal agencies subject to the Paperwork Reduction Act of 1995, and that it is intended to ensure the results of federally sponsored statistical surveys are as reliable and useful as possible.

It records that the revised standards replace OMB Statistical Policy Directives numbers one and two, on standards for statistical surveys and standards for publishing statistics respectively, which had last been revised in 1974.

It cites the Director's statistical policy functions under the Paperwork Reduction Act as the authority, and notes that the federal statistical programme is decentralised among more than seventy agencies or organisational units.

The notice is on the Federal Register.

Why the underlying data is missing in the first place is set out by the tips piece.

Steelhead Water, CA

What does the framework actually care about?

That the method is documented and the results are presented honestly.

The notice describes the standards as intended to ensure that surveys are designed to produce reliable data as efficiently as possible, and that methods are documented and results presented in a manner making the data as accessible and useful as possible.

Two halves, and the second is the one that gets skipped everywhere: documentation of method is treated as part of the product rather than as an appendix.

Which is the point that transfers directly to a small trade, since a figure published without its method is not a smaller version of a good figure, it is a different kind of object.

The notice also frames the purpose in terms of confidence in reliability, which is a standard about how the number will be used rather than about how it was produced.

Numbers about a trade get quoted in negotiations, in pricing decisions and in business plans, and a number that cannot bear that weight should not be published in a form that invites it.

What happens when it is quoted anyway is set out by the marketing spend piece.

MJF, MO

What does this site actually do?

Reads primary sources, states assumptions, and refuses to fill gaps.

Every figure on this site falls into one of three categories, and each is labelled where it appears.

A quoted figure from a primary source, meaning a statute, a regulation or an official notice, cited with a link so that it can be checked and rejected.

An arithmetic illustration built from stated assumptions, which is not a measurement of anything and is labelled as an assumption in the panel where it appears.

And an operational judgment, which is somebody's opinion about how to run something and is labelled as judgment rather than as a sourced figure.

What does not appear is a fourth category, being an industry average, a typical rate or a benchmark, because no consulted source publishes one for this trade.

Where a page would ordinarily carry such a figure, it says so instead.

An example of that refusal is set out by the repeat rate benchmarks piece.

What are the three questions?

Who, how many, and when.

Who was asked determines what the figure describes, and it is almost never what the headline implies.

A figure drawn from members of one association, or from operators listed on one booking platform, describes that population rather than the trade.

How many answered determines whether the figure describes even that population, since a low response rate concentrates the result in whoever was motivated to reply.

And when determines whether it still holds, which matters enormously for anything about price in a trade where costs have moved sharply.

A figure that cannot answer all three is an impression that has been formatted as data, and formatting is the entire difference.

Applying those three to any number you encounter takes about a minute and disqualifies most of them.

How that plays out on rates specifically is set out by the state day rates piece.

Why does a low response rate matter so much?

Because the people who answer are not a random subset.

Anybody who responds to a survey about their own business has chosen to, and the choosing is correlated with the answer in ways that are difficult to correct.

Operators doing well are more likely to disclose figures than operators struggling, which biases any income or rate figure upward.

Operators with time in the off-season respond more than operators working year-round, which biases the sample towards a particular kind of business.

And operators who are on a mailing list at all differ from those who are not, which is a selection effect before anybody has been asked anything.

None of this is fixable by asking more people; it is fixable only by knowing who did not answer and how they differ, which is expensive.

Which is precisely why serious survey work devotes substantial effort to nonresponse and small surveys ignore it entirely.

The same problem in a client survey is set out by the client survey piece.

Is a small survey worthless then?

No, and it should be described accurately.

Eleven operators on one river telling you what they charge is genuinely useful information about that river, and it is not a national average.

Which is the whole distinction: the problem is never the sample size, it is the claim made on top of it.

A figure presented as what eleven guides on the upper beats told me in March is honest, specific and usable, and it is more useful than a national figure would be.

Presented as the average rate for trout guides, the same eleven answers become misleading without a single number changing.

Which means the discipline available to a small operation is entirely about description rather than about method.

Say who, say how many, say when, and the figure can stand.

Where local figures actually come from is set out by the channel share piece.

What about association figures?

Better than most, and still describing members.

A trade body surveying its own membership is doing the honest version of this, since the population is defined and the respondents are identifiable to whoever ran it.

What it cannot do is describe the trade, because membership is itself a selection: operators who join associations differ systematically from those who do not, usually by size, by permanence and by how established they are.

Which biases every figure in the same direction and is rarely noted, because the body producing the figure has no incentive to point out that its members are unrepresentative.

The correction is not to discard such figures but to read them as what they are, which is a description of a professionalised subset.

For a guide comparing themselves to it, that matters practically: an association rate figure is a ceiling rather than an average for most of the trade.

Which is useful information provided nobody mistakes it for the middle.

Where the same effect appears in platform data, the direction reverses, which is worth holding in mind when the two disagree.

Does anecdote count for anything?

Yes, when it is treated as what it is.

Guides frequently dismiss their own accumulated observation as merely anecdotal and then quote a national figure they cannot source, which inverts the reliability of the two.

A guide who has worked one river for twelve years knows things about that river that no survey would capture, and knows them with a confidence a survey could not justify.

The failure is not in trusting that knowledge; it is in generalising it, since the same guide knows almost nothing about the water two states away.

Which suggests a rule that runs opposite to the usual instinct: trust your own observation about your own water more than any published figure, and trust it about nowhere else at all.

Written down annually, that observation becomes something better than anecdote without ever becoming a survey.

Which is the practical position available to almost every operator and taken by very few.

The record that supports it is described by the debrief piece.

What about the numbers on this site's other pages?

Each one is labelled, and the labels are the point.

Where a page carries a panel of arithmetic, the panel names its assumptions in the panel itself rather than in a footnote, because a reader who takes the number away should take the assumption with it.

Where a page carries a headline figure, it names what kind of figure it is: a quoted provision, an illustration, or a judgment.

Where a page would conventionally carry an industry average, it says instead that none is asserted and explains why, which is the standard applied consistently across the whole cluster.

None of that makes the arithmetic correct for your operation, and it is not intended to: an illustration exists to show the shape of a calculation rather than to supply its inputs.

Substituting your own figures into any of them takes a few minutes and produces something worth more than the illustration ever was.

Which is the honest use of everything in this cluster.

What about figures from platforms?

Real data about their own users, and frequently presented as more.

A booking platform genuinely knows what was charged for trips booked through it, which is a large and precise data set about a specific population.

What it does not know is anything about trips booked directly, which for many operations is the majority of their business and frequently the higher-priced part.

Which means a platform's rate figure systematically describes the segment most likely to compete on price, and that is a real bias rather than a quibble.

The same applies to lead-time and seasonality figures, since platform bookings behave differently from repeat direct bookings by construction.

None of which makes the figures wrong; it makes them figures about a population that has to be named.

Where a platform names it, the data is valuable, and where it does not, the three questions do the work.

What direct booking changes is set out by the lead times piece.

What would a useful trade survey look like?

Narrow, local, repeated, and honest about who answered.

The version that would genuinely help is not a national study but a small one, run on one fishery, repeated annually, reporting the number asked and the number who answered.

Which is achievable by an association, an outfitter or a group of guides who trust each other, and costs an afternoon rather than a budget.

Repetition is what makes it valuable, since the level is less informative than the direction and the direction is visible after three years.

Publishing the response rate is the single practice that would separate such an effort from everything currently circulating.

Nobody does this, and any group that did would own the only defensible figures on their water.

Which is a genuine opportunity rather than a criticism.

The measurement discipline it needs is set out by the numbers piece.

How should the reader treat this site?

As sourced argument rather than as data.

What is offered here is a reading of primary material, arithmetic from stated assumptions, and opinions about how to run an operation, in that order of reliability.

Every regulatory claim carries a link to the text so that it can be checked, and several of those texts say something other than what is commonly believed.

Every arithmetic panel states its assumptions in the panel, because an unstated assumption is how an illustration becomes a benchmark by accident.

And every operational judgment is labelled as such, because the difference between a sourced fact and a strong opinion is exactly what this page is about.

Where a figure is genuinely unavailable, the page says so rather than estimating, which is the standard the whole cluster is built on.

The clearest example of that refusal is set out by the price index piece.

Where does trade data usually go wrong?

Six ways, and the unstated population is the first.

Publishing a figure without naming who was asked, so the reader assumes it describes the trade.

Omitting the response rate, which is the single most informative number in any survey and the one least often given.

Presenting a platform's own bookings as the market, when direct business behaves differently by construction.

Quoting a figure whose only source is another article quoting it.

Averaging across fisheries that have nothing in common, which produces a number describing nobody.

And publishing an old figure without a date, in a trade where costs have moved sharply.

How to compute the one figure that is genuinely yours is set out by the income model piece.

What is the working standard?

Name the population, the count and the date, or do not publish the number.

Ask the three questions of every figure you encounter, including any figure on this site.

Treat a small local survey as valuable and describe it accurately, since the problem is never the sample size but the claim placed on it.

Assume any platform figure describes that platform's users until somebody says otherwise.

Distrust any number whose trail terminates in another article, which is most of them.

Where you publish anything yourself, publish the response rate, because almost nobody does and it is the number that decides whether the rest means anything.

And where the figure does not exist, say so, because an honest absence is more useful than a confident invention.

The statistical policy authority is 44 U.S.C. 3504, and the notice is mirrored on govinfo.

The clearest worked example of the whole approach is the tips piece.