Josh Wills from Slack delivered a good presentation this week on “Visibility and Monitoring for Machine Learning Models”.
One take away from his talk that resonated:
By far the most important question to ask your team is: ‘How often do you want to deploy this machine learning model?’
‘Once’ is a bad answer. Good answers are ‘never’ or ‘over and over and over again’
If it is not important enough to keep working on it, than it is not important enough to put into production.
Part of the reason this question can get overlooked is the disconnect between the ‘lab’ mentality of heads down data scientist and the ‘factory’ mentality required to deliver ongoing value.
A team out of Google described the Hidden Technical Debt in Machine Learning Systems and conceptual visualized how much of a deployed system is everything else beyond the core model:
That little black box in the middle is what we are focused on when we are wearing our ‘lab’ hat, but when the ‘factory’ hat goes on we have to think about the whole picture.
One thought on “Key question before deploying ML”