Updated 30 January 2023
It has been proposed that, “Privacy matters to the electorate, and smart business looks at how to use data to find out information while remaining in compliance with regulatory rules.” Since “smart business” also consists of “the electorate” as employees, at least one burning question is whether privacy or ethical violations in technologies like artificial intelligence (AI) will really matter sufficiently to employees who may be more concerned about putting food on the table than about raising concerns or performing whistleblowing, with potentially negative job consequences for them? And what happens if the country, region, or sector is too immature to have meaningful regulatory rules to comply with? Does it then become a case of almost anything goes? After all, no laws will be broken by the “smart business” in this case.
Furthermore, given that the law is silent on many privacy issues, is mere compliance with regulations – some of which may be significantly outdated and not aligned with technological advances – really a sufficient form of due diligence for “smart business?” Even more, local privacy law could be toothless against organizations based in other jurisdictions committing AI-based privacy law violations against local citizens.
And if privacy regulation and compliance were truly enough, then why do 98% of Americans still feel they should have more control over the sharing of their data, and why are 79% of India residents still uncomfortable with the sale of their data to third parties, and why do 74% of people around the world still express concern about their data? So, no, regulatory rules are most certainly not as effective as some may think they are, especially not for “smart business.” In other words, mere privacy compliance is a necessary but wholly insufficient condition for truly smart (and ethical) business.
Introducing Artificial Intelligence (AI) Privacy Protection Challenges
In can be argued that smart businesses look for ways for technology to help them achieve their strategic objectives, and building artificial intelligence systems, although still nascent, may suit some use cases. However, organizations have had little incentive “… to build privacy protections into their systems. Major privacy breaches in recent years have made for breathless headlines, but ultimately very little fallout for the companies responsible.”
Artificial intelligence has been no different when seen through a privacy by design lens as privacy has not been top-of-mind in the development of AI technologies. There is a high risk to individuals’ rights and freedoms in the AI processing of personal data, something quite different to the risk posed by data breaches, but also with very little “fallout” for the companies responsible. Some privacy challenges of AI include:
- Data persistence – data existing longer than the human subjects that created it, driven by low data storage costs
- Data repurposing – data being used beyond their originally imagined purpose
- Data spillovers – data collected on people who are not the target of data collection
Data collected by means of AI also raises privacy issues like informed consent freely given, being able to opt out, limiting data collection, describing the nature of AI processing, and even being able to delete data on request. However, how would human subjects of the collected data, perhaps given a spillover effect, even know data was collected about them to be able to make any inquiries of organizations with respect to their own data, or to request that it be deleted?
For European Union (EU) subjects, Article 22 of the General Data Protection Regulation, given some exceptions, requires that, “The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling… .” What does this mean for automated artificial intelligence algorithms in the context of EU subjects, noting that any human interventions in the AI process – perhaps to get around the solely automated processing requirement – cannot be fabricated and must have results that demonstrate a true influence?
AI and Threats to Privacy and Democracy
In 2017, the Economist found that half of the world’s countries scored lower for democracy than the previous year, mainly because of the erosion of confidence in government and public institutions. In alignment with this, according to the Director Journal, in 2017, the 28th Governor General of Canada articulated the growing and “disturbing” global pattern of mistrust in institutions, finding for the first time in the same year that less than half of Canadians trust their government, business, media, non-governmental organizations, and their leaders.
The role of the psychographic profiling of Facebook users, and the implications for privacy in artificial intelligence in the Cambridge Analytica scandal in the 2016 US presidential election, added much to this erosion of confidence, and threats to democracy continues to be fueled by AI manipulating democracy’s levers. As another example, that US company Clearview AI violated Canadian privacy laws by collecting photographs of Canadian adults and even children for mass surveillance and facial recognition without their consent, and for commercial sale, only serves to further reduce trust and confidence in AI businesses, as well as to reduce trust and confidence in entire countries’ ability to appropriately direct matters related to privacy and AI. There are incidentally separate, parallel investigations into Clearview AI in Australia and the United Kingdom.
A report into the above matter by the Office of the Privacy Commissioner (OPC) of Canada notes that the information scraping seems to violate the terms of service of, for example, Facebook, YouTube, Instagram, Twitter and Venmo (section 6.ii.). Furthermore, while Clearview AI claim the information is freely available on the internet so consent is not required, the OPC finds that express consent is indeed required in the case of especially sensitive biometric information, and/or when the collection, use or disclosure is “outside the reasonable expectations of the individual (sections 38 and 40).” While there is no doubt that facial information is one of the most sensitive types of personal data, would a reasonable person really consent to their most sensitive data, their biometric data, being used by a commercial organization where the data could be used for anything, and where that purpose could change into anything else infinitely into the future (digital is forever)? Most certainly not.
Given that “[data] is the lifeblood of AI,” some of the most sensitive data is personally identifiable information (PII) and protected health information (PHI). We therefore need to examine the extent to which AI uses PII and PHI, including biometrics, and whether due caution has been practiced to ensure, for example, that democracy’s levers are not manipulated.
Introducing the Proactive Identification of Potential Privacy Violations in AI Developments
So how do we begin to determine whether privacy is protected in artificial intelligence deployments, when a key challenge in AI concerns the auditability of its algorithms? Whether or not the AI algorithms continually adapt to new information, how can we really know that the AI system is really computing what we think it is computing, and that it is protecting privacy from both regulatory and ethical perspectives?
A primary concern with artificial intelligence is its potential to replicate, reinforce or amplify harmful biases. Such biases can proliferate depending on the nature of the data collection performed, a process that may also result in issues such as the spillover effects introduced in an earlier paragraph.
The challenge with existing audit methods used to identify privacy and other weaknesses is that risks are identified only after the system has been deployed, and when a negative impact has already been realized. So, whereas internal audits usually generally serve as a sufficient second line of control, in the case of AI, audits serve only as necessary but insufficient controls. Indeed, a broader system of checks and balances is required to complement AI audits. With a possible AI audit framework being SMACTR, the audit practitioner might also consider COBIT as a starting point, developing:
- “Managed Innovation” (APO04), specifically the management practice of “Monitor the Implementation and Use of Innovation” (APO04.06), where the innovation in this case is AI
- “Managed Business Process Controls” (DSS06), specifically the management practice to “Ensure Traceability and Accountability for Information Events” (DSS06.05), which speaks to the nature of data and information collection as well as their flows
Note not only that accountability is a governance construct, but also that DSS06.05 contains a privacy construct – albeit a limited one – concerning the determination and fulfilment of data and data outcome retention requirements. That’s because the retention of personally identifiable data and information becomes a privacy issue when the original purpose for having collected and processed that data expires.
Privacy in an AI Context
Privacy in the context of AI has different considerations to data privacy in general. One of the challenges of protecting privacy in artificial intelligence concerns how to create suitable regulations that protect privacy without stifling advances in AI technology. The data contexts at stake are both the scanning mechanisms that enable the AI tools to learn about their environments, as well as the nature of the data itself and how it is used to create the AI capability. Using the spillover effects raised in this blog post as an example, the traditional consent requirement for organizations looking to use personal data is weak, as there is no consent acquired in spillover data collections; the victims of spillover data have no say in the matter as they do not even know they’re involved.
Consent is also not as powerful a tool as one may be led to believe, even if the requirements for consent are that it is informed and freely given. The Clearview AI example shows that consent was not sought as much as it should have been according to the OPC. As a similar example, Microsoft removed its database of 10 million facial photographs – which were being used by organizations like IBM, Panasonic, Alibaba, military researchers and Chinese surveillance firms – as most of the people whose faces were in the dataset were not aware their image had been included.
In terms of the democracy, the two matters requiring deeper regulatory and/or policy attention concern not only the sources of data collected or accessed for the artificial intelligence tools, but also how the data is used to effect partisan outcomes. Indeed, partisan outcomes contradict at least one of the 7 pillars of Privacy by Design, where the goal is to achieve positive sum outcomes rather than one political party winning at the expense of another, the latter producing a zero-sum outcome rather than the desired positive sum outcome.
While most organizations may be looking to privacy compliance and ethics in the use of sensitive personal information, in artificial intelligence, the challenges can be significantly different in both content and in scale. It is the role of both the IT governance professional and of the privacy professional to ensure that these AI-based privacy challenges are the subject of appropriate oversight.
Editor’s note: For more privacy resources from ISACA, learn about the Certified Data Privacy Solutions Engineer (CDPSE) credential.