[Transformer] Position-wise Feed-Forward Networks (2)

nlp.seas.harvard.edu/2018/04/01/attention.html#position-wise-feed-forward-networks

The Annotated Transformer

 

    FFN(x)=max( 0, xW1+b1 )W2 + b2

class PositionwiseFeedForward(nn.Module):
    "Implements FFN equation."
    def __init__(self, d_model, d_ff, dropout=0.1):
        super(PositionwiseFeedForward, self).__init__()
        # Torch linears have a `b` by default. 
        self.w_1 = nn.Linear(d_model, d_ff)
        self.w_2 = nn.Linear(d_ff, d_model)
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):
        return self.w_2(self.dropout(F.relu(self.w_1(x))))

 Transformer network 에서 encoder와 decorder는 각 각 feed forward network를 가지고 있다. 

 x에 linear transform을 적용 후 ReLu activation을 수행 후 다시 linear transform을 적용한다. 

[ReLu 적용 이유]

woogi-tech.tistory.com/entry/ML-%EC%84%A0%ED%98%95%EB%8C%80%EC%88%98-Linear-Transformation-in-Neural-Network

[ML / 선형대수] Linear Transformation in Neural Network