Validating more loop optimizations
Below I inserted a figure to illustrate the difference: The first row refers to “Scenario 1”, and the 3rd row describes a more “classic” approach where you further split your training data into a training subset and a validation set.Then, you train your model on the training subset and evaluate in on the validation set to optimize its hyperparameters, for example.This is sometimes called "loop hoisting." For some compiled programming languages such as FORTRAN and C/C , an optimizing compiler can sometimes determine that an expression is constant and move it outside of the loop for you. Rick Wicklin, Ph D, is a distinguished researcher in computational statistics at SAS and is a principal developer of PROC IML and SAS/IML Studio.
Train the model on the former, evaluate the model on the latter (by “evaluate” I mean calculating performance metrics such as the error, precision, recall, ROC auc, etc.) Split the dataset into a separate test and training set.Use techniques such as k-fold cross-validation on the training set to find the “optimal” set of hyperparameters for your model.If you are done with hyperparameter tuning, use the independent test set to get an unbiased estimate of its performance.In practice, the Infinity Optimization Process includes 69 distinct steps (with 105 sub-steps.) But showing every single step would be impractical and confusing—instead, the diagram of the process represents its core components.If you haven’t been seeing excellent results from your optimization efforts, it may be due to a lack in your process or team’s expertise.