When AI Makes Commercial Decisions, Who Checks the Rules?

Software can help a business compare costs, forecast demand or suggest a price. Those uses raise different questions from a system that profiles an individual customer and sets a price specifically for them. The important first step is to describe what the system actually does, which data influences it and who can change the result.

Do not collapse recommendation, dynamic and personalised pricing

Reports about McDonald's described a tool that makes restaurant-specific price recommendations. McDonald's has said that people make the final pricing decisions and that the tool does not set prices for individual customers or change them in real time. The company disputes parts of the reporting about how its system is used. A difference between restaurant prices does not, by itself, prove that a customer was individually profiled or charged a real-time surge price.

That distinction is useful for any business using automated commercial advice. A dashboard may suggest a price based on location, costs or aggregate sales. A separate system might use information about an identifiable customer to decide which offer they see. Record which of these is happening, rather than using the label "AI pricing" as a substitute for understanding the data flow.

Check the decision, the data and the objective

Begin with the result the business wants: sustainable margin, a clear discount policy, better use of available stock or another specific outcome. Then ask whether the data is accurate, current and relevant. A model trained on old transactions may miss customers who stopped buying, changes in costs or local circumstances that staff understand. An apparent increase in revenue is not proof of improved profit or customer value.

Set limits on what the system may recommend and who may approve a change. Managers need enough information to question a recommendation, compare it with other evidence and record when they override it. Review results over time for unexpected patterns, not just the first pilot period. Keep a route for customer complaints and staff escalation; an explanation from a supplier should not be the only way to investigate a disputed outcome.

Understand when data-protection rules apply

If a pricing system processes personal data, the organisation remains responsible for assessing the applicable data-protection duties, explaining the processing and managing its supplier relationship. The ICO describes the automated-decision provisions as applying to decisions based solely on automated processing, including profiling, that have a legal or similarly significant effect on a person. A human review must be meaningful to be relevant; a person who cannot change or question the output is not useful oversight.

This does not mean every algorithm-assisted price calculation is prohibited or falls into that category. Whether a system is within scope depends on the information used, the decision made, the role of human involvement and the effect on people. The ICO is updating its guidance following legislative changes, so check the current rules and obtain appropriate legal advice for a material deployment rather than treating this article as a legal assessment.

Evidence to request from the provider

• A plain-language list of the information used, its source and whether it relates to identifiable people.
• The purpose the system is optimising, the limits on its recommendations and known limitations.
• Test results and monitoring for inaccurate inputs, unexpected outcomes and changes in performance.
• Records of human approvals, overrides, complaints and material changes to the model or rules.
• Contract terms covering data use, supplier access, retention, incident reporting and the end of the service.

Also ask whether the supplier can explain a particular recommendation using the information and rules that applied at the time. If the provider cannot reconstruct a result, the business may struggle to answer a customer or regulator who asks how it was reached.

Put a review around the system before launch

1. Map the inputs, outputs, people affected and business objective.
2. Decide which changes need approval and who has authority to reject them.
3. Test representative scenarios, including missing or inaccurate information.
4. Keep records of the version, data, result and any human intervention.
5. Set a review date and a way to pause the system if outcomes depart from expectations.

Technical governance and data management should be in place before automated recommendations become routine. If your software made a decision a customer challenged, could you explain how that decision was reached?

Sources and further reading

• Reporting on the questions about McDonald's pricing recommendations
• McDonald's: Separating fact from fiction about how it sets prices
• ICO: Automated decision-making, including profiling
• ICO: AI contracts and third parties
• Assessing the value and risks of an AI investment