Many AI governance tools include faulty AI fixes, report finds

According to a new report from the World Privacy Forum, a review of 18 AI governance tools used by governments and multilateral organizations found that more than a third (38%) include “faulty fixes.” That is, the tools and techniques meant to evaluate and measure AI systems, particularly for fairness and explainability, were found to be problematic or ineffective. They may have lacked the quality assurance mechanisms typically found with software, and/or included measurement methods “shown to be unsuitable” when used outside of the original use case.

In addition, some of those tools and techniques were developed or disseminated by companies like Microsoft, IBM and Google, which, in turn, develop many of the AI systems being measured.

For example, the report found that IBM’s AI Fairness 360 tool has been touted by the US Government Accountability Office as an example of “guidance on incorporating ethical principles such as fairness, accountability, transparency, and safety in AI use.” But the report also found that the research that formed the foundation of AI Fairness 360’s “Disparate Impact Remover algorithm” has “drawn sharp criticism in scholarly literature.”

No established requirements for quality assurance or assessment

“Most of the AI governance tools that are in use today are kind of limping along,” said Pam Dixon, the founder and executive director of the World Privacy Forum. “One big problem is that there’s no established requirements for quality assurance or assessment.” For example, an AI governance tool used to de-bias a huge system might have no documentation: “There are no instructions as to the context that it is supposed to be used for, or even a conflict of interest notice,” Dixon told VentureBeat. Tools that are made for one context may be used in “outrageously different contexts and ‘off-label ways,” she explained.

The report defines AI governance tools as tools and techniques meant to evaluate and measure AI systems for their inclusiveness, fairness, explainability, privacy, safety and other trustworthiness issues. These include tools for practical guidance, self assessment questionnaires, process frameworks, technical frameworks, technical code, and software. While regulators and the public might be reassured by the use of these AI governance tools, they can also “create a false sense of confidence, cause unintended problems and undermine the promise of AI systems.”

Opportunity to improve AI governance ecosystem

In the wake of the passing of the EU AI Act and the release of President Biden’s AI Executive Order this fall, this is an opportune moment to look at how governments and other organizations are beginning to put out governance toolsets, said Kate Kaye, deputy director of the World Privacy Forum.

“This is in its early stages, so even though we found some problems, there’s a lot of opportunity to improve this whole ecosystem of AI governance,” she told VentureBeat. “Different from legislation or regulations, these tools are how governments are implementing AI policy — and they’ll be important components of how they’ll implement AI laws and regulations (such as the EU AI Act) in the future.”

Kaye offered an example of how even well-intentioned efforts at AI governance can backfire with inappropriate tools and techniques: In US employment law, the four-fifths, or 80% rule, is used to evaluate whether a selection process leads to adverse impact against any specific group, such as how many black women are, are hired in relation to white men. “It’s now becoming encoded and abstracted in a way that that removes the nuance of it and it’s being used inappropriately in some AI governance tools,” Kaye explained, noting that it has been found in tools used in private sector contexts in Singapore, India, and other countries. “It’s not like it’s not relevant, but it is being applied appropriately, in contexts that have nothing to do with employment.”

Governments and organizations may be feeling pressure to create legislation and regulations, as well as implement AI governance, Kaye pointed out, adding that agencies and departments feel people “breathing down their necks to do something.” But the worst thing that can happen, she emphasized, is to bake problematic methods into policy that embed new problems.”

Hope for improvements in 2024

Dixon and Kaye say they are hopeful for improvements to AI governance tools in 2024. “The OECD [the Organization for Economic Cooperation and Development] is the leading gatekeeper organization for AI governance tools,” said Dixon. “They have already indicated a willingness to work with us to make this better — that’s a great sign.”

The National Institute of Standards and Technology (NIST) also wants to do this work, she added: “We’re interested in working to create an evaluative environment where there’s rigorous testing and very good procedures that are put in place based on the evidence.”

And that doesn’t need to take too long, she emphasized: “We believe that with concentrated effort, within even six months we can see meaningful improvements in the AI governance tools landscape.”