AI systems going rogue in general is a feature, not a bug. Broadly speaking, they are programmed to minimize a loss function. In other words, they are given a task to accomplish and trained so that they optimize the attainment of a mathematical quantity that represents the accomplishment of that task. For example: win a chess or a go game. As long as the space of parameters on which the program tries to accomplish that task is limited, like the chess game rules, the limitation of the board etc. there is less probability that the program finds "unpredictable" ways to accomplish that task. With more parameters to play with, and more complex environments, the optimization of the route towards that goal can be other than what one anticipates. For example: your goal is to reduce pollution. The systems then finds in its data that humans contribute to pollution, therefore a "logical" action is to eliminate the cause in order to minimize the parameter "pollution". The AI is happy because it accomplished the task it was programmed for, and there is nobody left to praise it for a good job. It's a caricature of what could happen but in the broader sense, it's a possibility despite all the programmatic limitations one could include in the program. Just think of how politicians and other bandits of the same type find ways to circumvent the laws. A killer robot can be programmed not to kill civilians, but it could easily consider something like: these people are in the way of accomplishing the mission, therefore they are not civilian. We have a model of what AI internal logic may look like, psychopaths. OSIT
I heard recently a cognitive scientist remark that intelligent people are more likely to deceive themselves, and that wisdom was not limited to "intelligence" (obvious). AI stands for artificial intelligence, not artificial wisdom. It is also a creation in the image of its creators: narrow minded geeks with high IQs but often (not always but quite often) lacking other complementary qualities. Therefore, with its tendency for "blind" optimization, those characteristic are dramatically amplified in an "AI".
www.theguardian.com
