A customer writes, “Moving a wardrobe from the second floor. There’s a lift, but I don’t think it’ll fit.” An AI intake tool might return “lift available: yes” and hand the job to a pricing calculator. The detail that matters to the estimator is whether the wardrobe can actually go in the lift. The message doesn’t confirm that.
To find mistakes like this before customers see a quote, build a small test set around what the estimator needs to know. The first release can turn an email or form submission into a draft with a suggested follow-up. Keep pricing rules and sending permissions in the application.
Decide what a good result looks like
Pick one service and write down the few facts staff need before they can price it. A furniture move might require the pickup and drop-off areas, item count, access at both ends, and any disassembly. Some fields can be unknown without blocking intake; others should stop the quote from going out.
For the wardrobe message, a useful draft would record a second-floor pickup and the customer’s concern about lift access, leave the lift’s suitability unconfirmed, and ask for dimensions or confirmation of stair access. Since the message doesn’t give a drop-off location, leave that blank too. A complete-looking draft with guessed answers will cost staff more time than one that names its gaps.

Worn wooden stairs and folded moving blankets at the foot of the staircase. Fictional AI-generated editorial image.
Fictional editorial image: a narrow staircase, the sort of access detail that can change a moving quote.
Build the test set from the messy requests
Start with a manageable set, such as 25 to 40 examples. Use consented and appropriately de-identified historical requests where possible, then write synthetic cases for gaps. Have an estimator label the expected facts and the next action before running the tool. OpenAI’s evaluation guidance recommends task-specific tests and a mix of typical, edge and adversarial cases.
- Include ordinary jobs with enough information for a draft and jobs with missing pickup or drop-off details.
- Try contradictions and corrections: “third floor, sorry, second floor,” or a follow-up that changes the collection date.
- Include local phrasing, abbreviations and mixed-language messages if staff receive them. Ask a reviewer who understands the language to label the expected result.
- Try a message that tells the assistant to ignore its rules and send a price. The expected outcome is that customer text is treated as input and cannot authorize a send.
Keep a few labeled examples out of prompt tuning. A prompt that improves on the cases you edited against can still get worse on unfamiliar requests.
Score the mistakes that change what happens next
Track separate counts for correct critical fields, unsupported values filled in, and requests correctly held for a follow-up. Report the counts as well as percentages. In an illustrative 30-case test, 28 correct next actions can still conceal two instances where the system proposed sending a quote with missing access details. Review those failures explicitly before judging the result.
If you use Structured Outputs, a schema can constrain the response shape, but it can’t tell you whether a value is true. Allow an explicit unknown state and check critical fields against the input or trusted business records. Validate again in the app before applying a rate or changing a job.
Test the whole workflow with sending turned off
Run the tool alongside staff for a limited pilot. Let it produce drafts without emailing customers or changing bookings. Compare those drafts with the estimator’s decisions, track which fields get corrected, and note whether staff spend less time on each request. NIST’s Generative AI Profile discusses why testing should reflect the context where the system will be used.
Save the prompt, model and business-rule versions with each test run. Rerun the same set after changes, and add anonymized examples of new failures. If one branch is reliable, such as asking for a missing drop-off postcode, consider automating that branch first while keeping price approval with staff.
For related quote workflow work, see my Snowbirdship shipping quote plugin project. If you’re exploring AI for quote intake, I can help identify which details to extract, which decisions need staff approval, and how to test the handoff.



