Last weekend I was talking to a friend who works for a medical tech company, about the development of AI in medical diagnosis. Apparently, AI medical diagnostic systems are not permitted for use in the UK because the ways in which the systems are in a constant state of self-development means that regulator or whoever would be unable to review a particular diagnosis after the fact; simple or more sophisticated flow charts are permitted, of course. Rather, tech companies are developing their systems in places like Rwanda where regulation is weaker; which also brings to mind the testing of the contraceptive pill in Costa Rica in the 1950s.
The general point about the ethics of the developers is relevant to the development of AI systems, whether medical research or automated facial recognition or other biometric systems and others, presumably. And, one fears, we are in danger of allowing tech companies who, let’s be honest, have little or no interest in this area, of running roughshod over people’s freedoms.
A recent article by Crawford and Paglen (2019), The Politics of Images in Machine Learning Training Sets (https://www.excavating.ai) examined the some of the most widely used datasets employed in the development of image and human behaviour recognition systems. The researchers spent several years looking at the logic behind how the images have been used to train the systems to recognise the world. Noting that images do not describe themselves, they need to be described and interpreted and these are theory-laden; but they found how training image sets, the bedrock of the systems, were full of prejudice and bias and, sometimes truly ugly assumptions.
I realise that I am fated to find the finer details of this topic more interesting than most but, the more one reads about the researchers’ investigation of the data sets, the grimmer one’s mood becomes. ImageNet, a widely used dataset, one learns, is at one point described thus:
‘At the image layer of the training set, like everywhere else, we find assumptions, politics, and worldviews. According to ImageNet, for example, Sigourney Weaver is a “hermaphrodite,” a young man wearing a straw hat is a “tosser,” and a young woman lying on a beach towel is a “kleptomaniac.” But the worldview of ImageNet isn’t limited to the bizarre or derogatory conjoining of pictures and labels. Of course, these sorts of assumptions have their own dark histories and attendant politics.’ (ibid)
And, on reading the article, the assumptions and outrageous descriptions and categorisations within the datasets gets worse, culminating in, one understands, IBM categorising race by skull shape. In response to concerns, in January 2019, a large part of the ImageNet dataset was removed from Stanford University’s servers. Following this, several other large datasets were removed from servers elsewhere, including 10m images from Microsoft MS Celeb. The latter, a victory for privacy, certainly, but these datasets have already been downloaded and used in the development of systems. The datasets with their crude assumptions and downright lousy academic underpinning, live on.
The use of AI is becoming increasingly part of everyday life. The Daily Telegraph reports that Unilever is to use AI to assess job candidates’ suitability by analysing their reactions to a set of questions (27 October 2019). Cries of foul will fall on deaf ears, fascination with new technology and its evident benefits means that it is likely to be adopted without reflection.
If we do not know the history of the datasets, how they were developed and evolved, what assumptions they were based on, then we cannot subject AI systems to independent test and scrutiny; to regulation, if you like. Amongst other things, we will not be able see what features and behaviour the systems consider to be acceptable and those which differ from the norm. But because decision-making has been automated, we risk favouring these systems over all others because, from the outside, they appear to be independent, fair and quick and efficient and free of human biases, which is precisely what they are not.
