Interesting paper on AI unpredictability.
[AI unpredictability is] defined as our inability to precisely and consistently predict what specific actions an intelligent system will take to achieve its objectives, even if we know terminal goals of the system … The unpredictability of intelligence is a very special and unusual kind of surprise, which is not at all like noise or randomness. There is a weird balance between the unpredictability of actions and the predictability of outcomes … Unpredictability is practically observable in current narrow domain systems with superhuman performance. Developers of famous intelligent systems such as Deep Blue (Chess) Watson (Jeopardy), and AlphaZero (Go) did not know what specific decisions their AI is going to make for every turn. All they could predict was that it would try to win using any actions available to it, and win it did. … There are infinitely many paths to every desirable state of the world. Great majority of them are completely undesirable and unsafe, most with negative side effects.
The potential for undesirable negative side effects depends heavily on context. If you are using Watson in an oncology clinic then high potential for negative consequences, using Watson on Jeopardy is not so risky.
These concerns argue for more emphasis on explainability and understanding of underlying causality.