Optimizer ( Adam, SGD )

Adam

Adam: Adaptive moment estimation

Adam = RMSprop + Momentum

Momentum :  gradient descent 시 최소점을 찾기 위해 모든 스텝을 밟는 것이 아닌 스텝을 건너 뛴다. 

Stochastic gradient descent(SGD)

 

 

Image Credit: CS231n

Adagrad

It makes big updates for infrequent parameters and small updates for frequent parameters. For this reason, it is well-suited for dealing with sparse data.

The main benefit of Adagrad is that we don’t need to tune the learning rate manually. Most implementations use a default value of 0.01 and leave it at that.

Disadvantage —

Its main weakness is that its learning rate is always Decreasing and decaying.

 

AdaDelta

It is an extension of AdaGrad which tends to remove the decaying learning Rate problem of it.

Another thing with AdaDelta is that we don’t even need to set a default learning rate.