Question and scope
This review asks what quality sampling can show about outsourced candidate outreach. The unit is one outreach message and the candidate response that follows. The observation window runs from the first send to the next action or the close of the case. The scope includes first contact, follow-up, scheduling and rejection messages. That boundary keeps the review tied to messages the desk actually sends.
The review does not estimate a national outreach quality rate or claim that one poor message caused one candidate to disengage. It also excludes paid advertising, job board copy and messages sent by hiring managers directly. The narrow scope keeps the evidence tied to the outreach work an assistant can control. Add the excluded channels to the scope note so the reader can see the limit.
Data Sources and Methodology
The review draws on ten public sources covering employment records, labor turnover, occupational context, information security and consumer protection. Each source is used for a defined concept or record boundary rather than for a numeric estimate. The method compares the approved role brief, the sent message, the candidate reply and the assistant's next action. A case is counted as defective when the message conflicts with the brief or breaks a written outreach rule.
National context is separated from site-level inference, and no source is treated as proof that one message changed one candidate's decision. The review reports defect counts by type and by case rather than a single quality score. Where the sample is small, it describes the cases instead of computing a rate. The definitions stay fixed across the compared periods so a change in behavior is not confused with a change in measurement.
What sampling can and cannot show
Sampling can show whether a defined set of rules is being followed in the messages reviewed. It can reveal patterns such as missing role details, late follow-ups or wrong next steps. It can also show whether a template change reached the queue. Those are useful findings because they point to a specific fix. A sample that finds the same defect twice is already useful.
Sampling cannot prove that every message was good or that candidates were satisfied. A sample is a view of the cases selected, not a census. A clean sample may coexist with a rare but serious failure outside the selection. The review should describe its coverage and avoid claiming more than it checked. Write the coverage sentence in the report before the reader asks.
Sampling also cannot separate the message from the candidate's other experiences. A candidate may receive a good message and still withdraw after a poor interview. The review measures the message and the immediate response, not the whole candidate journey. That limit belongs beside the result. Keep the journey boundary in the limitations section.
Defining the defect
A defect is a message that breaks a written outreach rule or contradicts the approved role brief. The definition should be specific enough that two reviewers can agree on the same case. "Unclear message" is too broad, while "does not name the role" is observable. Start with a short list of defects that matter most. Write the rule and the brief reference beside each defect category.
Useful categories include missing role detail, wrong next step, incorrect schedule or pay information, late send, missing suppression and inappropriate tone. Add a category for a message that is technically correct but impossible to act on. The last category catches vague replies that a checklist alone would miss. Add one example of a vague reply that still followed the template.
Write one example for each defect and keep the examples beside the checklist. The examples anchor the reviewer's judgment and make coaching concrete. Review the examples when a new role or template launches. A definition that nobody can apply consistently is not a useful measure. Keep the examples in the same document as the checklist.
Sample size and coverage
The sample should cover each message type and each assistant who sends outreach. A sample drawn only from first contact will miss follow-up defects. A sample drawn from one assistant will miss a template that another assistant uses differently. Stratify the sample by type and by sender when the volume allows. Record the number of assistants included and the number left out.
A fixed number per week is easier to sustain than a percentage that changes with volume. Ten messages per assistant per week is a workable starting point for a moderate queue. Increase the number after a template change or a complaint. Decrease it after several clean periods, but keep a minimum. Write the reason for the sample size in the routine.
Record the sample size and the selection method beside the result. A rate without a denominator is not comparable. A sample chosen because the messages looked suspicious is not a random sample. The reader needs to know how the cases were selected before trusting the pattern. State whether the sample was random, stratified or chosen by case type.
Reviewer consistency
Two reviewers can read the same message and reach different conclusions. Test agreement before treating the scores as a trend. Ask both reviewers to score the same twenty messages without discussing them. Compare the scores and discuss the disagreements. Use a simple agreement count rather than a complex statistical test.
A simple agreement check is enough for a small desk. Count how often the reviewers choose the same defect category. If they disagree often, the definition or the examples need work. Do not add more categories in response to disagreement. Tighten the existing ones first. Write the disagreement example in the training note.
Rotate the reviewer occasionally so one person's preferences do not become the standard. Keep the checklist and the examples stable during the rotation. A new reviewer should score a known set before reviewing live cases. The calibration set protects the team from a quiet drift in standards. Record the date of the rotation so the calibration set stays current.
Timing of the review
Review the message close to the send date. A review weeks later loses the context of the queue and the candidate's reply. It also makes coaching less useful because the assistant may have moved to a different role. A weekly review keeps the feedback connected to the work. Read the message before the candidate reply so the sequence is clear.
Read the candidate's reply with the original message. A reply can show that the message was confusing even when it followed the template. It can also show that the candidate misunderstood a clear message. The pair gives the reviewer more evidence than the sent text alone. Save the pair in the review record with candidate details removed.
Check the follow-up behavior as part of the same case. A good first message followed by no second contact may still fail the candidate. A second contact sent too soon may feel like pressure. The review should cover the sequence, not only the first send. Mark the follow-up date beside the first send date.
Limits of the evidence
A quality sample measures the messages selected, not the assistant's overall performance. A small sample can miss a rare defect or overrepresent an unusual week. Report the sample size and the selection method so the reader can judge the coverage. Avoid ranking assistants on a handful of cases. Name the coverage limit in the same paragraph as the result.
The review also cannot establish causation. A candidate may disengage for reasons unrelated to the message. A defect may contribute to a decision without being the cause. Report the defect and the response as separate observations, and avoid claiming more than the records show. Record the candidate's stated reason when it is available.
Definitions can drift over time. A category that was clear at the start may become ambiguous after a process change. Review the definitions each quarter and mark any change. A trend that crosses a definition change is not a trend. Write the definition change date on the chart.
A one-desk protocol
Choose one month and one outreach stage. Define the defect categories and select a sample by message type and sender. Score each message against the role brief and the checklist. Read the candidate reply and the next action for each case. Keep the sample selection note beside the defect table.
Run a calibration set with two reviewers before the live review. Compare the scores and revise the examples where they disagree. Then review the live sample and record the defect counts. Choose one template or instruction change based on the most common defect. Ask the second reviewer to score the same set before the live review.
Repeat the review after four weeks with the same definitions. Compare the counts and the sample size. A stable method shows whether the change helped. A single dramatic audit is less useful than a routine that the team can repeat. Write the change date beside the comparison.
Reporting the result
Report the number of messages reviewed, the number with at least one defect and the count by defect type. Include the sample selection method and the reviewer agreement result. Show the candidate reply rate beside the defect count so the reader can see the response context. Put the definitions and the limits next to the table. Show the defect count and the sample size in the same row.
Note any change in template, role mix or sender group during the period. A new role with an unusual schedule can explain a cluster of missing details. A template launch can explain a temporary increase in wrong next steps. The context prevents a reader from treating a normal transition as a decline. Keep the context note short and place it above the table.
Employment record guidance from the EEOC recordkeeping requirements and the Department of Labor pay records guidance frames the records used in the review. Labor turnover context from the BLS Job Openings and Labor Turnover Survey and role detail from the BLS Occupational Outlook Handbook keep the comparison grounded. Compensation context from the BLS National Compensation Survey helps the reviewer check pay statements against a public reference. Information security guidance from the NIST Cybersecurity Framework and consumer protection guidance from the FTC data security guidance covers the handling of candidate messages. Hiring guidance from the Small Business Administration, occupational context from O*NET and population context from the U.S. Census Bureau help describe the outreach work accurately.