The End-of-Season Client Survey

There is a federal statute whose entire purpose is stopping agencies asking people questions they do not need answered. Before a government body may collect information, it must demonstrate that the collection is the least burdensome necessary, is not duplicative of information already accessible, and has practical utility.

Practical utility is the phrase worth carrying into a client survey, because almost no question on almost any client survey has any. A guide sending eleven questions in November is collecting information they will never act on, from people whose goodwill is the actual asset, and the Paperwork Reduction Act exists because somebody in 1980 noticed that pattern at scale. None of it binds a fishing guide. It is a standard, and a harsh one. Neighbouring material sits under the ops playbooks hub.

What is the standard?

Three tests, and the third is the one that kills most questions.

Section 1320.5(d)(1) of Title 5 provides that to obtain approval for a collection of information an agency must demonstrate it has taken every reasonable step to ensure the collection is the least burdensome necessary for the proper performance of its functions, is not duplicative of information otherwise accessible, and has practical utility.

The same paragraph adds that the agency shall seek to minimise its own costs of collecting, processing and using the information, but shall not do so by shifting disproportionate costs or burdens onto the public.

Which is a remarkably direct statement of the thing every badly designed survey does, being making the respondent do work so the sender does not have to think.

Practical utility is the operative test: not whether the answer would be interesting, but whether it would be used.

Part 1320 is carried on the eCFR.

What the season's own records already tell you is set out by the debrief piece.

Elizabeth Marie, MA

What else does the regulation refuse?

A list of specific impositions, and two of them transfer directly.

Section 1320.5(d)(2) provides that unless a specific need is demonstrated, approval will not be given to a collection requiring respondents to report more often than quarterly, or to prepare a written response in fewer than thirty days after receipt.

It also refuses, in connection with a statistical survey, a design that is not intended to produce valid and reliable results capable of being generalised to the universe of study.

And it refuses a pledge of confidentiality not supported by authority in statute or regulation, or not supported by disclosure and data security policies consistent with the pledge.

The quarterly limit and the thirty-day response window are both instructive for anybody who has sent a survey with a deadline in it.

The generalisability point is the one that quietly demolishes most small-business surveys, since eleven responses from your happiest clients do not describe your client base.

What that sample problem does to a number is examined by the numbers piece.

Althea K Sport, MA

Should a guide send one at all?

Rarely, and only with a decision attached.

The honest test is whether there is a decision you are about to make that the answers would change, and if there is not, the survey is a way of appearing thorough.

Where there is such a decision, being a rate change, a new trip type, a boat purchase or a change of water, the survey has practical utility by definition.

Where there is not, the information you want already exists in the trip records, the cancellation reasons and the rebooking pattern, and reading those costs nobody anything.

Which is the duplication test from the regulation, applied honestly: do not ask people for information otherwise accessible to you.

Most guides who run this test discover the survey they were about to send has one question in it worth asking.

Asking that one question, well, produces more than eleven questions produce.

What the existing records hold is described by the debrief piece.

What is the one question?

What nearly stopped you booking, and it is not asked at the end of the season.

The most valuable information a guide can obtain is what made a person hesitate before they became a client, because that hesitation is happening to everybody else who did not book.

It has to be asked while it is retrievable, which is within a week or two of the booking rather than in November when the memory has been overwritten by the trip itself.

Phrased directly, it works: was there anything that nearly put you off booking.

The answers are consistently specific and consistently actionable, being the price page, the lack of dates, an unanswered question or an unclear cancellation policy.

All four are things you can fix in an afternoon and none of them would ever appear in a satisfaction survey.

One question, asked at the right moment, with a decision waiting for the answer.

Where the fix usually lands is described by the booking page piece.

What about the second question?

What would you have paid for that you did not get.

Which is a pricing and product question disguised as a service one, and it produces answers nobody volunteers otherwise.

Typical answers are a longer day, an earlier start, photographs, transport, gear, or a second guide for a group, and every one of them is a thing you could sell.

It works because it gives the client permission to say something that would otherwise sound like a complaint.

Asked at the end of a good day it is answered honestly, and asked in a November survey it is answered vaguely if at all.

Which is the general principle underneath all of this: the answer quality depends far more on when you ask than on how you phrase it.

The pricing consequence of the answers is worked through by the pricing piece.

When is a real survey justified?

Before a significant change, and then it should be a conversation.

An operation about to add a second boat, change waters, or move a rate materially has a genuine decision and genuinely needs input.

What produces useful input in that situation is not a form but six phone calls to six clients who represent different parts of the business.

Twenty minutes each, two hours total, and the quality of what comes back is not comparable to anything a questionnaire produces.

People say things on the phone they will not type, and the follow-up question is where the actual information lives.

Which is also the reason the regulation's burden test matters: a phone call is a burden on you and a form is a burden on them.

Choosing to carry the burden yourself is both better manners and better research.

Who to call is answered by the segmenting piece.

What should never be asked?

Anything you already know, and anything scored out of ten.

How was your guide is a question whose answer you can predict for every client on your list, which means it has no practical utility whatever.

A rating out of ten produces a number that cannot be acted on, since the difference between an eight and a nine tells you nothing you could change.

How did you hear about us is the most-asked and least-reliable question in small business, because people genuinely do not remember and will name whatever they saw last.

Anything about satisfaction generally, which flatters the sender and produces nothing.

And anything requiring the client to rank a list, which is real cognitive work in exchange for data you will not use.

Each of those fails the practical utility test on its own terms.

How attribution should actually be measured is set out by the marketing report piece.

What about online reviews?

A different instrument entirely, and it is not research.

Guides conflate the survey and the review request because both involve asking a client for words, and the two have almost nothing in common.

A review is marketing produced by the client, is public, and is asked for because you want the artefact rather than the information.

A survey is research, is private, and is asked for because you intend to change something.

Which matters practically, because asking for both in the same message halves the chance of getting either and makes the request read as a transaction.

Separate them by weeks, ask for the review from clients who plainly had a good day, and ask the research question of everybody.

Where a client volunteers something in a review that belongs in the research, take it, but do not design the review request to elicit it.

The review mechanics are covered by the reviews piece.

How should the question be delivered?

In the body of a normal message, not as a link.

A single question typed into a text or an email is answered by replying, which is almost no work and produces a full-sentence answer.

A link to a form is a decision to open something, on a device, and the drop-off between the two is enormous for a one-question ask.

Which is the burden test again: the form is easier for you to process and harder for them to complete, and the regulation is explicit that the sender's convenience does not justify shifting the cost.

The answers also come back better, since a reply is written in the client's own voice and a form field is written to fill a box.

Twelve replies to process by hand is fifteen minutes, which is less than building the form took.

Above about forty responses the arithmetic changes and a form starts earning its place, which almost no single-boat operation reaches.

Where a form is genuinely needed, one question on it is still the right number.

Is anonymity worth offering?

No, and offering it costs you the useful part.

An anonymous answer cannot be followed up, cannot be weighted by who said it, and cannot be acted on for the person who raised it.

Which removes most of the value, since knowing that a nine-year client said something is a substantial part of what the something means.

The regulation's own hostility to unsupported confidentiality pledges points the same way: a promise about how information will be handled has to be one you can actually keep.

In a business where you know every respondent personally, an anonymity promise is close to meaningless anyway and clients know it.

What does work is making it easy to say something uncomfortable, which is a matter of how the question is phrased rather than of hiding who answered.

Asking what nearly stopped you rather than what was wrong gives somebody a way to raise a real problem without accusing anybody.

What if nobody answers?

That is information, and it is usually about the ask.

A one-question message to sixty past clients that produces two replies has told you something about the relationship rather than about the question.

Most commonly it means the message read as marketing, which happens when it arrives from a business address, contains a link, or sits underneath a paragraph of preamble.

The version that gets answered is short, personal, from a name, and asks the question in the first line rather than the fourth.

Where it still produces nothing, the honest conclusion is that these clients are transactional rather than relational, which is a real and useful finding.

It also predicts the rebooking rate better than any survey answer would have, which is the ironic version of practical utility.

Chasing non-responders is not worth doing for research, since the second ask converts almost nobody and costs goodwill.

Does the timing matter that much?

More than the questions do.

The booking hesitation is retrievable for about two weeks and then gone, overwritten by the experience of the trip itself.

What somebody would have paid more for is answerable at the take-out and vague by November.

Whether they will return is not knowable from asking at all, since people say yes and mean it and then do not, which is why the repeat rate is measured rather than surveyed.

Which sorts the whole category: ask about the past at the moment it happened, and never ask about the future.

An end-of-season survey is by construction asking about the past at the worst possible moment, which is why it produces such thin material.

Where an annual contact is wanted, that is a different thing with a different purpose and should not pretend to be research.

What that annual contact should be is set out by the winter email piece.

What about the client who had a bad day?

Call them, and do not send them a form.

The one situation where you genuinely need information is the trip that went wrong, and it is the one situation where a survey is worst.

A form invites a written complaint that becomes a fixed position, whereas a call invites an explanation that can be responded to.

It also demonstrates something a form cannot, which is that a person noticed and cared enough to pick up a phone.

The call is short and has one question in it: what would have made that day better.

Not an apology, not an explanation of the conditions, and no offer attached until they have finished answering.

Most of those conversations end better than they start, and the ones that do not were going to be lost anyway.

The wider handling of that situation is set out by the difficult clients piece.

What should be recorded?

The answer, verbatim, against the client, the same day.

Summarising an answer destroys most of its value, since the specific words are what let you notice that four people said nearly the same thing.

Which is the only reliable signal available at this sample size: not a percentage, but a repetition.

Three clients independently mentioning the same friction is worth more than any distribution of scores, and it is visible only if the words were kept.

Keeping them against the client rather than in a survey file is what lets you see who said it, which matters because a first-timer and a nine-year client are reporting different things.

Reviewing the whole set once, in the off-season, is when the pattern appears, and it takes twenty minutes.

Acting on one thing from it is the entire point, and acting on nothing is what happens when the answers were never kept.

Where they belong is described by the debrief piece.

Where do surveys usually go wrong?

Six ways, and having no decision waiting is the first.

Sending one without a decision attached, so the answers are collected and nothing happens.

Asking eleven questions when one would have done, which lowers the response rate and dilutes the answers you wanted.

Asking in November about things retrievable only in June.

Scoring things out of ten, producing numbers that cannot be acted on.

Treating twelve self-selected responses as though they described sixty clients.

And summarising the answers instead of keeping the words, so the repetitions that carry the signal are invisible.

What the season's own numbers say instead is set out by the repeat rate piece.

What is the working approach?

Two questions, asked at the right moments, and six phone calls a year.

Within two weeks of a booking, ask whether anything nearly put them off booking, and record the answer word for word.

At the take-out, ask what they would have paid for that they did not get.

Do not send an end-of-season survey, because it asks about the past at the worst possible moment and produces nothing you can act on.

Before any significant change, telephone six clients from different parts of the business rather than sending anybody a form.

Call anybody whose day went badly, ask one question, and offer nothing until they have finished answering.

Review everything once in the off-season, looking for repetitions rather than distributions, and act on one thing.

The statute behind the standard is 44 U.S.C. 3506, with the regulation mirrored on govinfo.

The record all of it lands in is described by the debrief piece.