Hyper-parameter Tuning for a machine learning model

Question

Why a hyper-parameter like regularization parameter (a real number) cannot be trained over training data along with model parameters? What will go wrong?

Answer 1

This is generally done to prevent overfitting. Model parameters are trained using the training set. Hyper-parameter tuning is done using a validation set that is (ideally) completely independent of the training data. The final performance should be evaluated on a test set. Typical splits are 80/10/10 or 60/20/20.

If you tune your hypermeters on the training set, you will very likely vastly overfit and suffer a performance hit on the test set.

Try it out, See the difference in performance on your test set when you do hyper-parameter tuning on the training set. vs on a separate validation set