The best machine learning projects are rife with potential feedback loops.
It is important to see machine learning projects as agile ongoing processes rather than one-off projects and to recognize that these feedback loops are critical to creating sustained business value.
If you focus too narrowly on model optimization than you might feel that these feedback loops are annoyance rather than assets.
Typical examples include:
What you learn during data exploration can cause you to go back and revisit your data preparation.
What you learn in modeling can cause you to revisit your data exploration and feature engineering
What you learn in exploration and modeling can give you new insights into the underlying real-world system you are trying to model.
Deploying your model into production can change the underlying real-world system you are modeling, which might then require updates in the model.
Changes in your data sources can invalidate your original assessment of your models success,
- And so on ….