CNN 텍스트 유사도 분석(Feat. Quora pairs)

kaggle www.kaggle.com/c/quora-question-pairs/submissions

Quora 질문 답변 사이트에서
같은 질문에 대한 판별 문제 column
# ['id', 'qid1', 'qid2', 'question1', 'question2', 'is_duplicate']

Quora Question Pairs

CNN - 합성  신경망

In deep learning, a convolutional neural network (CNN, or ConvNet) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[1] They are also known as shift invariant or space invariant artificial neural networks (SIANN), based on their shared-weights architecture and translation invariance characteristics.[2][3] They have applications in image and video recognition, recommender systems,[4] image classification, Image segmentation, medical image analysis, natural language processing,[5] brain-computer interfaces,[6] and financial time series.[7]

Convolutional neural network - Wikipedia

 

Process

* Embedding

import tensorflow as tf
from tensorflow.keras import layers

class SentenceEmbedding(tf.keras.layers):
    def __init__(self,**kwargs):
        super(SentenceEmbedding,self).__init__()

        self.conv = layers.Conv1D(kargs['conv_num_filters'],kargs['conv_window_size'],
                                  activation=tf.keras.activations.relu,
                                  padding='same')

 

 

• filters: 가중치, (NLP-단어의 수,Image-이미지의 특징)
• kernel_size: filter 행렬의 크기
• activation: 목표 수치로 변환 해주는 함수
  gelu(...): Applies the Gaussian error linear unit (GELU) activation function.
  linear(...): Linear activation function (pass-through).
  relu(...): Applies the rectified linear unit activation function.
  sigmoid(...): Sigmoid activation function, sigmoid(x) = 1 / (1 + exp(-x)).
  softmax(...): Softmax converts a real vector to a vector of categorical probabilities.
  tanh(...): Hyperbolic tangent activation function.
• padding: 
  same: input size와 output size가 같다.
  valid: "패딩 없음"을 의미
  casual: input_size보다 큰(팽창한) output_size

		self.max_pool = layers.MaxPool1D(karg['max_pool_seq_len'],1)

 

• pool_size:(int) maxpool 사이즈
• strides: pool 이동 간격
• padding: 사이즈 변경 
  
  max_pool input의 특징 추출

 

 

 

		self.dense = layers.Dense(kargs['sent_embedding_dimension'],
        							activation=tf.keras.activation.relu)

 

units: output dimension

# activation으로 output의 크기를 정한다.

 

 

 

 

    def call(self,x):
        x = self.conv(x)
        x = self.maxpool(x)
        x = self.dense(x)
        return tf.squeeze(x)

# tf.squeeze(x) 불필요한 차원 제거

 

# SimilarityModel (본 모델)

class SentenceSimilarityModel(tf.keras.Model):
    def __init__(self,**kargs):
        super(SentenceSimilarityModel, self).__init__(name=kargs['model_name'])

        self.word_embedding = layers.Embedding(kargs['vocab_size'],kargs['word_embedding_dimension'])
        self.base_encoder = SentenceEmbedding(**kargs) # 문장1 임베딩
        self.hypo_encoder = SentenceEmbedding(**kargs) # 문장2 임베딩
        self.dense = layers.Dense(kargs['hidden_dimension'],
                                activation=tf.keras.activations.relu)
        self.logit = layers.Dense(1,activation=tf.keras.activations.sigmoid)
        self.dropout = layers.Dropout(kargs['dropout_rate'])
    def call(self,x):
        x1, x2 = x
        b_x = self.word_embedding(x1)
        h_x = self.word_embedding(x2)
        b_x = self.dropout(b_x)
        h_x = self.dropout(h_x)

        b_x = self.base_encoder(b_x)
        h_x = self.hypo_encoder(h_x)

        e_x = tf.concat([b_x,h_x],-1)
        e_x = self.dense(e_x)
        e_x = self.dropout(e_x)

        return self.logit(e_x) # 0~1로 변환

 

• tf.keras.layers.Dropout( 
  rate, 
  noise_shape=None, 
  seed=None, 
  **kwargs)
  
  rate: (Float) 0~1 Fraction of the input units to drop.
  
  Bayesian 신경망을 통해 해당 뉴런에 대한 학습이 무의미 할 경우 다른 뉴런에 학습을 중점으로 둔다.