What can go wrong when AI features interact with customers?

What is the main question?

How should companies evaluate risk before launching customer-facing AI features?

What else should teams answer?

• What risks appear when AI interacts with customers?
• What should product teams check before launching AI features?
• What controls reduce customer-facing AI risk?
• What evidence should vendors or internal teams provide?

Why customer-facing AI requires stronger review

Customer-facing AI requires stronger review because its mistakes are visible outside the company and can affect trust, regulatory exposure, brand risk, fraud risk, data protection, and customer harm. A feature that chats with customers, drafts advice, recommends actions, summarizes account information, or automates support can expose sensitive data, produce inaccurate or unsafe outputs, bypass policy, or fail to hand off to a human at the right time. Before launch, companies should define the feature's scope, the customers it serves, the data it may use, the claims it may make, the situations it must refuse, and the evidence needed to prove controls worked during testing and production.

The business review should cover both the customer experience and the control surface. Product teams need measurable acceptance tests, not only a general statement that the model performs well. Legal, risk, privacy, security, and customer operations should agree on launch criteria, escalation paths, monitoring, and what residual risk the business is accepting.

What can go wrong in customer interactions?

Customer-facing AI can reveal data to the wrong user, make unsupported promises, provide inappropriate advice, misstate policy, hallucinate commitments, mishandle complaints, or create a confusing experience when it cannot help. Customers may also try to manipulate prompts, request prohibited information, automate abuse, or use the feature for fraud. Even when no one is malicious, ordinary ambiguity can create risk if the system guesses instead of escalating.

• Sensitive data exposure through prompts, retrieved records, chat history, logs, or generated answers.
• Unsafe or inaccurate outputs that affect money, health, legal rights, account access, or customer obligations.
• Prompt manipulation that pushes the system outside approved policy or support boundaries.
• Abuse patterns such as scripted attacks, refund manipulation, identity misuse, or spam.
• Weak handoff when the customer needs a human, supervisor, fraud team, or regulated process.

Which workflows need the most control?

Higher-risk workflows need stronger controls and more formal approval. Examples include financial decisions, health information, legal guidance, regulated claims, identity verification, account changes, payments, refunds, complaint handling, contractual commitments, safety issues, and support interactions involving vulnerable customers. A low-risk product discovery assistant may need a lighter review than an account assistant that can see private records and recommend changes.

Use risk tiers so governance does not become a blocker. Low-risk features can move through lighter checks. Medium-risk features may need scoped data access, output review, abuse monitoring, and user messaging. High-risk features need formal launch criteria, red-team testing, human escalation, legal and privacy review, and clear evidence retention.

What should be defined before launch?

Define what the AI feature is allowed to do and what it must not do. Product owners should write scope boundaries, approved data sources, restricted topics, refusal behavior, escalation triggers, human review requirements, customer disclosures, retention expectations, logging needs, and incident paths. Acceptance tests should be measurable and tied to the actual customer journey.

• Approved customer intents and unsupported intents.
• Data sources the feature may use and data it must not reveal.
• Situations requiring human handoff or supervisor review.
• Policy language for refunds, account changes, support commitments, and regulated topics.
• Success, failure, and abuse metrics the team will monitor after launch.
• Owners for reviewing alerts, complaints, test results, and exceptions.

What controls may be needed

Controls may include scope limitation, input filtering, output filtering, retrieval controls, human escalation, abuse monitoring, audit logs, pre-launch testing, red teaming, response review, and operational playbooks. If the feature can access account data, retrieval and identity checks need to be strong. If it can trigger actions, approval gates and action logs matter. If it operates in a regulated workflow, product, legal, risk, and compliance owners should agree on review standards before release.

No single control fully solves customer-facing AI risk. Controls should be layered: design limits what the feature can do, data controls limit what it can use, testing finds failure modes, monitoring detects live issues, and escalation processes recover when the system needs help.

What evidence should buyers request?

Evidence should show that the feature has been tested against realistic customer scenarios and that the team can monitor it after launch. If the feature is bought from a vendor, ask for proof of the control surface, control outcomes, enterprise readiness, and known limits. If the feature is built internally, the same evidence should exist for product and risk approval.

• What customer workflows and high-risk scenarios were included in testing?
• What test cases cover sensitive data, unsafe outputs, abuse, and prompt manipulation?
• How are human handoffs triggered, tracked, and reviewed?
• What logs show prompts, outputs, policy decisions, escalations, and customer-impacting actions?
• How are false positives, customer complaints, and urgent incidents handled?
• What known limits and residual risks are documented for business approval?

Practical checklist

• Classify the feature by customer impact and data sensitivity.
• Define approved and prohibited customer intents.
• Map data sources, retention, logging, and human review points.
• Create measurable launch criteria and acceptance tests.
• Test sensitive data exposure, unsafe outputs, abuse, prompt manipulation, and handoff failures.
• Review high-risk workflows with legal, privacy, risk, security, and operations.
• Monitor live performance, alerts, complaints, and escalations after launch.
• Retest after model, policy, data source, or workflow changes.

FAQ