Interesting study on reproducing the results of machine learning papers.
Three points stuck with me:
- 62% success rate was higher than I expected.
- “Finding 1: Having fewer equations per page makes a paper more reproducible.” made perfect sense.
- This once again reinforced the importance of hyperparameters. Sometimes I wonder how much “machine learning” is going on vs. how much hyperparameter trail and error is going on.