What is the learning rate parameter in TensorflowJs

DanielJuly 5, 2023

Let's say we have a super simple neuronal network with just one weight. The initial weight value is 0.2, and with this weight, it will make a prediction with some error.

Hoping to minimize the prediction error, also known as the loss function, we will increase the weight value to 0.4. The discrepancy between a weight of 0.2 and 0.4 is called the learning rate.

Using this new weight of 0.4 the prediction error may get bigger or smaller. And based on this error our learning algorithm may change the direction of the weight update and maybe also the learning rate.

The aim is to find the weight that gives the minimum of the loss function so that the predictions of the neuronal network become more accurate.

Using a learning rate that is too big will miss the minimum of the loss function, while with a too small value, the training will become slow and expensive.

For example in TensorflowJs we can set the learning rate using the learningRate parameter in the constructor of some optimizers:

model.compile({
    optimizer: tf.train.sgd(0.01), // learning rate to 0.01
    /* ... */
});