728x90

BPE ๋‹จ๊ณ„

- ๋นˆ๋„์ˆ˜์™€ ํ•จ๊ป˜ ์ฃผ์–ด์ง„ ๋ฐ์ดํ„ฐ์…‹์—์„œ ๋‹จ์–ด ์ถ”์ถœ

- ์–ดํœ˜ ์‚ฌ์ „ ํฌ๊ธฐ ์ •์˜ (vocab_dict)

- ๋‹จ์–ด๋ฅผ ๋ฌธ์ž ์‹œํ€€์Šค๋กœ ๋ถ„ํ• 

- ๋ฌธ์ž ์‹œํ€€์Šค์˜ ๋ชจ๋“  ๊ณ ์œ  ๋ฌธ์ž๋ฅผ ์–ดํœ˜ ์‚ฌ์ „์— ์ถ”๊ฐ€

- ๋นˆ๋„๊ฐ€ ๋†’์€ ๊ธฐํ˜ธ ์Œ์„ ์„ ํƒํ•˜๊ณ  ๋ณ‘ํ•ฉ

- ์–ดํœ˜ ์‚ฌ์ „ ํฌ๊ธฐ์— ๋„๋‹ฌํ•  ๋•Œ๊นŒ์ง€ ์•ž ๋‹ค์„ฏ ๋‹จ๊ณ„ ๋ฐ˜๋ณต

 

๋ฐ˜์‘ํ˜•
์ €์ž‘์žํ‘œ์‹œ

'๐Ÿ—ฃ๏ธ Natural Language Processing' ์นดํ…Œ๊ณ ๋ฆฌ์˜ ๋‹ค๋ฅธ ๊ธ€

small scale text data classification  (0) 2023.05.16
[M1 Transformers] M1 Mac Transformers Install Error  (0) 2022.06.19
[BERT] ํ•˜์œ„ ๋‹จ์–ด ํ† ํฐํ™” + OOV (Out of Vocabulary)  (0) 2022.05.09
[BERT Dict] NSP(Next Senctenct Prediction) Task  (0) 2022.05.07
๋ฒ„ํ‚คํŒ…(bucketing)์„ ์ด์šฉํ•œ ํ•™์Šต ๋ณต์žก๋„ ํ•ด๊ฒฐ  (0) 2021.03.28
๋‹คํ–ˆ๋‹ค
์ƒ๋‹จ์œผ๋กœ

ํ‹ฐ์Šคํ† ๋ฆฌํˆด๋ฐ”