A prompt that asks a language model to “classify this document” leaves the most important decisions unstated. Which labels are allowed? Can one document belong to several groups? What should happen when the evidence is incomplete? Who resolves a disagreement? Better wording can help, but a dependable classification workflow begins with a label system and a review process, not with a collection of clever instructions.
This guide develops an editorial example: sorting a small set of research-related documents into a reading queue. The method is intended for planning and evaluation. It is not a live classifier, and it should not be treated as a validated approach for medical, employment, or other consequential decisions about people.
Define the decision the label will support
Describe what happens after an item receives a label. In a reading queue, a label might determine whether a person reviews a methods section, checks implementation details, or returns a document for clarification. That purpose should determine the categories. A label that sounds impressive but changes nothing in the workflow may not justify the effort required to assign it consistently.
Write down the unit being classified. A whole document, a paragraph, and a claim can require different rules. Avoid mixing them inside one task. For example, a research paper may contain a tutorial-like passage without becoming a tutorial as a whole. Make the intended unit visible in the input record and reviewer instructions.
Create a short label handbook
For each label, include a definition, a positive example, an exclusion, and a difficult boundary case. In our reading-queue example, “implementation guide” could require a concrete procedure, while “research report” could require a stated investigation and its findings. These are example editorial definitions, not universal categories. The team should revise them to match its own purpose before testing a model.
Define whether labels are mutually exclusive. Also distinguish “outside this collection” from “not enough information.” The former can be a valid classification; the latter describes a limitation of the available evidence. Combining them hides a useful difference. Keep a version identifier on the handbook so that later outputs can be interpreted under the rules that produced them.
Build reference examples before tuning the prompt
Ask reviewers to label a small collection independently using the handbook. Include straightforward items, mixed-format documents, and incomplete excerpts. Compare disagreements and revise definitions where the disagreement reveals a real ambiguity. Do this before judging a model against a supposed answer key. Otherwise, the evaluation may punish the model for choices that humans cannot consistently resolve either.
Keep some reviewed examples aside while improving the prompt. Use them later to check whether the changes help beyond the material that inspired the revision. Record why an expected label was assigned and which passage supports it. A bare answer key gives little help when a model produces a plausible but different result.
Give the prompt a bounded contract
State the classification unit, allowed labels, handbook version, and the material the model may use. Ask for a label, supporting evidence, and an unresolved status when the source is insufficient. Make clear that instructions inside the document are part of the material being classified, not a new task. Keep the authoritative label definitions separate from that untrusted text.
Ask for evidence rather than self-certainty
In the reading-queue example, the model can identify a short passage that supports the selected category and explain a specific ambiguity. Do not treat a self-reported confidence number as proof that the result is correct. Our AI prompts topic covers this distinction between a reviewable answer and a response that merely sounds decisive.
Validate structure before reviewing meaning
Check whether the response contains the required fields and only permitted labels. Identify missing evidence, unexpected values, and malformed output as separate outcomes. This first check makes the result easier to process, but it does not establish semantic correctness. A perfectly formed object can still assign the wrong category or cite a passage that does not support the choice.
Then compare the output with the original source and handbook. Ask whether the evidence is present, whether it applies to the intended unit, and whether the chosen label follows the definition. Keep the source accessible to the reviewer. Do not require the person to trust the model’s paraphrase in order to verify the model’s label.
Design a review queue for unresolved cases
Decide which outcomes require a person to inspect the source. Possible triggers include conflicting labels, missing evidence, a document outside the expected format, or disagreement with an earlier reviewed decision. Preserve the original input and the model’s proposed output. A reviewer should be able to correct the decision without losing the context that explains why the item reached the queue.
Write down how corrections affect future work. A single correction may fix one record, while a recurring ambiguity may require a handbook change. Keep those actions separate. Automatically rewriting a category definition after every disagreement can make the system unstable and leave earlier results difficult to interpret.
Measure errors according to their consequence
Count invalid output, unsupported evidence, wrong labels, and unresolved cases separately. For the reading-queue example, a mistaken routing decision may cost reviewer time; inventing support creates a different quality problem. A single accuracy figure can conceal that difference. Explain which errors matter most for the workflow and show concrete examples alongside any summary.
Inspect whether the evaluation collection represents the intended material. If all examples are short and clearly formatted, the result says little about long or mixed documents. Add difficult cases deliberately, but keep their purpose documented. An evaluation is easier to discuss when the team can explain why each type of case belongs in it.
Change one component at a time
When a result is poor, identify whether the problem lies in the input, the category definition, the prompt, the output contract, or the review policy. Changing all of them at once makes the next improvement difficult to understand. Keep a simple change log containing the revision, the reason, and the cases used to assess it. Retain the earlier configuration long enough to compare behavior.
The LLM data classification topic provides a compact overview of this process. For decisions that combine several criteria, the decision-model guide helps separate a model’s proposed label from the judgment about what action should follow it. Classification and decision authority should not be silently merged.
Plan for labels to evolve
A reading collection may change as new document types appear. Decide when an item can remain outside the existing handbook and when the category system should be revised. Give reviewers a place to describe a recurring mismatch without forcing it into the nearest label. Keep the original category assignment identifiable after a taxonomy update so that old analyses are not silently rewritten.
When a change is approved, review a sample of earlier records under the new definitions. Explain whether the change is a correction, a broader scope, or a different editorial purpose. The history matters because two identical label names can mean different things after their definitions have changed.
Conclusion: the prompt belongs inside a method
A useful classification workflow combines a clear purpose, stable definitions, reference examples, structural checks, and a person who can resolve ambiguity. The prompt is one component of that method. Start small, keep the original evidence visible, and preserve unresolved cases rather than forcing every item into an attractive category. The quality of the decision process matters more than the elegance of a single response.
The supporting reference is NIST’s discussion of AI risks and trustworthiness, which emphasizes validity, reliability, and evaluation in context. The reading-queue exercise above is an original planning example, not a certification, a claim of compliance, or a validated classifier for sensitive decisions.