💼 Scenario
InsureAI is an insurance technology company that uses machine learning to set auto insurance premiums for 5 million policyholders. The pricing model considers 150 variables including driving behavior (from telematics devices), vehicle characteristics, credit score, claim history, and geographic factors. The model has reduced underwriting losses by 25%, saving the company $200 million annually.
A state insurance regulator has initiated a review after consumer advocacy groups alleged that InsureAI's pricing algorithm produces systematically higher premiums for policyholders in predominantly minority zip codes, even after controlling for driving behavior and claim history. Internal analysis confirms that zip code, credit score, and education level (all legal rating factors) serve as proxies for race, producing a disparate impact where minority policyholders pay 18% more on average.
The company faces a dilemma: removing these proxy variables reduces the model's predictive accuracy by 8%, increasing underwriting losses by an estimated $60 million annually. The CEO argues that the variables are actuarially justified and legally permitted. The Chief Ethics Officer argues that legal permission does not equate to ethical acceptability.