Explainable AI (XAI) is often treated as a minor curiosity in the landrush to implement deep learning. As the use of machine learning matures this will reverse and XAI will be the norm with black box systems becoming the exception.
Explainability is critical to winning real world adoption. Escalating consequences, cognitive biases and regulation are significant hurdles to applying black box systems. Explainability helps move interesting academic exercises into practical daily use.
Requiring explainability may make it more challenging to optimize a model for a specific test set. However, the right mindset is to think about the entire machine learning project end-to-end and with that perspective XAI is very attractive. Whether it is troubleshooting errors from data drift or generating insights into the underlying system you are modeling, explainable solutions have the advantage.