Keyword_Extraction_Frequancy_Final.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# KoBigBird_Keyword_Extraction_Frequancy_Weighted\n",
    "\n",
    "단어의 빈도수에 따른 가중치와 DBSCAN을 사용한 동의어제거를 적용한 최종 코드"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 본문 키워드 형태소 분석하여 후보 키워드 추출하기\n",
    "\n",
    "형태소 분석기 komoran을 사용하기 위해 konlpy 설치"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install konlpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 불용어 제거"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import stopwords as eng_stopwords\n",
    "\n",
    "eng_stopwords_set = set(eng_stopwords.words('english'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def remove_stopwords(nouns_list, stopword_set):\n",
    "    arr_removed = [n for n in nouns_list if n not in stopword_set]\n",
    "    return arr_removed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'고로', '딩동', '아니면', '？', '다시 말하자면', '‘', '나오', '어느 년도', '한 까닭에', '알 수 있다', '할만하다', '이럴정도로', '하곤하였다', '주저하지 않고', '로 인하여', '우리', '바로', '혹시', '뤼', '하자마자', '바꾸어서 한다면', '얼마간', '저기', '버', '대해 말하자면', '경우', '까악', '잘', '어떤', '의', '이 때문에', '그중에서', '마저도', '이래', '요만한걸', '줄은 몰랏다', '두', '좍좍', '하고있었다', '하마터면', '심지어', '하는것이 낫다', '주', '받', '여섯', '조차', '그리하여', '할지언정', '너', '즈음하여', '드', '야', '데', '연이서', '입각하여', '정도에 이르다', '실로', '생각한대로', '반드시', '이러이러하다', '이 되다', '오자마자', '텐', '하는 김에', '이것', '생각하', '이와같다면', '＃', '어떠한', '하기만 하면', '어찌됏어', '않기 위해서', '너희들', '어느때', '에 한하다', '아야', '소인', '앞에서', '＋', '때가 되어', '할수있다', '어때', '～', '허걱', '하하', '하', '무릎쓰고', '하지 않도록', '또', '아이', '에', '하든지', '그래', '해봐요', '매번', '설사', '영차', '오호', '하는 편이 낫다', '근거하여', '우리들', '위하여', '하더라도', '각각', '자마자', '시각', '그것', '거바', '어디', '아니라면', '만일', '대하면', '겨우', '힘입어', '그렇', '다른', '하면 할수록', '가지', '무렵', '툭', '뒤따라', '휴', '공동으로', '그럼에도 불구하고', '따라서', '때', '까지 미치다', '마음대로', '］', '지만', '두번째로', '인젠', '있', '총적으로', '(', '하지 않는다면', '등', '구토하다', '관한', '저것', '구체적으로', '그런데', '영', '하나', '결론을 낼 수 있다', '참', '이 외에', '요만큼', '각', '그치지 않다', '이렇게되면', '혹은', '”', '＆', '를', '같이', '위에서 서술한바와같이', '하기는한데', '일단', '로부터', '까지도', '남짓', '않', '앗', '조금', '꽈당', '알았어', '지말고', '안 그러면', '휘익', '어느것', '하려고하다', '편', '<', '.', '등등', '언제', '이 정도의', '헉', '할 생각이다', '도착하다', '매', '운운', '여보시오', '한항목', '제', '—', '불구하고', '지금', '할줄알다', '년', '로', '’', '령', '그러면', '9', '바꾸어서 말하면', '_', '관해서는', '앞', '답다', '하는것도', '말', '관계가 있다', '때문에', '이천육', '그렇지만', '>', '아니나다를가', '구', '동시에', '아하', '윙윙', '1', '우에 종합한것과같이', '결국', '그렇지', '마치', '아울러', '정도', '시간', '것', '살', '반대로', '하겠는가', '——', '마저', '이르기까지', '아이고', '해도된다', '라 해도', '않기 위하여', '어떻해', '여덟', '해서는 안된다', '물론', '이지만', '셋', '게다가', '의거하여', '안', '보는데서', '더욱더', '전', '허', '《', '으로 인하여', '또한', '예컨대', '든간에', '반대로 말하자면', '몇', '이때', '바꾸어말하면', '뿐', '한다면', '도달하다', '것과 같이', '놀라다', '의해서', '얼마큼', '육', '제외하고', '예를 들자면', '다섯', '거', '비교적', '의지하여', '가까스로', '마', '탕탕', '참나', '하게하다', '동안', '본대로', '개의치않고', '다음으로', '만 못하다', '고려하면', '그위에', '향해서', '그렇지않으면', '따라', '이천칠', '있다', '하도록하다', '。', '＞', '같다', '모르', '〈', '｜', '더구나', '관하여', '댕그', '얼마든지', '지든지', '여전히', '︿', '＜', '놓', '２', \"'\", '솨', '관련이 있다', '한마디', '〉', '거의', '이상', '한 이유는', '아이야', '잠깐', '응', '다음에', '자신', '쯤', '의해되다', '하는바', '［', '통하여', '보다더', '그럼', '어기여차', '…', '부류의 사람들', '당장', '일', '흐흐', '알', '세', '이외에도', '점', '말하자면', '끙끙', '어느', '이라면', '그렇지 않으면', '비길수 없다', '토하다', '&', '이로 인하여', '할망정', '`', '설령', '지', '왜', '한켠으로는', '흥', '싶', '끼익', '3', '갖고말하자면', '만약에', '아니었다면', '외에도', '하지만', '０', '상대적으로 말하자면', '향하여', '하게될것이다', '타다', '시초에', '중', '할 줄 안다', '오', '하면된다', '리', '에 달려 있다', '함께', '여기', '이렇', '》', '시키', '각종', '-', '어떤것들', '인 듯하다', '응당', '예를 들면', '하면서', '그런', '딱', '!', ')', '다음', '저', '어쩔수 없다', '된이상', '바', '주룩주룩', '아', '부터', '즉', '할뿐', '우선', '할수있어', '이었다', '이천팔', '임에 틀림없다', '비걱거리다', '그저', '본', '하기 때문에', '하지마라', '쳇', '만', '4', '씨', '걸', '하여금', '아무거나', '｝', '하고 있다', '5', '전후', '하기보다는', '）', '나머지는', '저것만큼', '콸콸', '헉헉', '메', '*', '륙', '만은 아니다', '에 대해', '그에 따르는', '대해서', '진짜로', '및', '요컨대', '틈타', '다소', '이 밖에', '어찌하여', '따지지 않다', '~', '에서', '너희', '이와 같다', '통하', '한적이있다', '이와 같은', '여부', '하물며', '으로', '좀', '，', '바꾸어말하자면', '어째서', '：', '과연', '보', '만큼', '줄은모른다', '어떻', '비로소', '보이', '하느니', '할때', '그러나', '이젠', '어떤것', '오직', '와아', '근거로', '언젠가', '?', '그런 까닭에', '더불어', '약간', '쪽으로', '2', '첫번째로', '=', '|', '뚝뚝', '둥둥', '봐', '기점으로', '아이구', '로써', '그래도', '팔', '헐떡헐떡', '급', '해도좋다', '각자', '했어요', '한때', '그런즉', '하구나', '시작하여', '이용하여', '그래서', '이런', '하기에', '이봐', '타인', '월', '위해서', '、', '버금', '된바에야', '％', '１', '비록', '따위', '이만큼', '건', '그렇지 않다면', '더라도', '여러분', '허허', '네', '한다면 몰라도', '들', '소생', '삐걱거리다', '기대여', '위하', '7', '한', '중의하나', '８', '그때', '어찌하든지', '둘', '곳', '와르르', '명', '에 가서', '오르다', '이렇게말하자면', '인', '￥', '뿐만아니라', '어느곳', '５', '제각기', '일반적으로', ',', '잇따라', '이쪽', '여차', '연관되다', '대로 하다', '이러한', '엉엉', '무슨', '오로지', '습니다', '사회', '...', '향하다', '펄렁', '내', '여', '옆사람', '따르', '＠', '！', '삼', '되', '자기집', '중에서', '많', '결과에 이르다', '모', '좋아', '이', '예하면', '개', '가', '오히려', '왜냐하면', '쉿', '얼마', '결', '자', '자기', '$', '；', '아이쿠', '당신', '에 있다', '가령', '어느해', '“', '까닭으로', '＄', '만약', '게우다', '관계없이', '불문하고', '그', '혼자', '일곱', '어쨋든', '과', '즉시', '양자', '쾅쾅', '비슷하다', '사', '기타', '대하여', '훨씬', '９', '어', '막론하고', '넷', '어찌됏든', '더욱이는', ';', '+', '습니까', '할 지경이다', '남들', '에게', '쿵', '못하', '퍽', '수', '좋', '노', '차라리', '때문', '이번', '얼마 안 되는 것', '이어서', '이천구', '와 같은 사람들', '７', '겸사겸사', '적', '6', '하도다', '총적으로 말하면', '총적으로 보면', '칠', '번', '아무도', '이곳', '만이 아니다', '무엇때문에', '어느쪽', '일지라도', '이렇게 많은 것', '0', '시키다', '문제', '그만이다', '３', '그들', '전자', '의해', '메쓰겁다', '%', '다수', '거니와', '을', '요만한 것', '형식으로 쓰여', '나', '까지', '으로써', '（', '한데', '그러', '점에서 보아', '보드득', '말하', '삐걱', '단지', '없', '곧', '퉤', '할 힘이 있다', '할 따름이다', '이렇구나', '속', '다', '그러므로', '대하', '같', '8', '팍', '바와같이', '무엇', '식', '원', '누구', '비하면', '이리하여', '견지에서', '논하지 않다', '어찌', '기준으로', '한 후', '밖에 안된다', '더', '\"', '저희', '바꿔 말하면', '그렇게 함으로써', '예', '더군다나', '다만', '모두', '잠시', '아니', '하지마', '｛', '저쪽', '전부', '하는것만 못하다', '이와 반대로', '말할것도 없고', '일때', '＊', '４', '만들', '뒤이어', '하도록시키다', '그러한즉', '봐라', '것들', '누가 알겠는가', '앞의것', '분', '집', '듯', '그리고', '뿐이다', '설마', '아홉', '우르르', '으로서', '입장에서', '크', '얼마만큼', '６', '해야한다', '할지라도', '그러니까', '다른 방면으로', '비추어 보아', '아래윗', '해요', '하여야', '일것이다', '^', '얼마나', '·', '그러니', '조차도', '와', '다시', '후', '어이', '다시말하면', '붕붕', '졸졸', '하기 위하여', '@', '\\\\', '뿐만 아니라', '이유만으로', '어떻게'}\n"
     ]
    }
   ],
   "source": [
    "import csv\n",
    "\n",
    "stopwords = []\n",
    "with open('./data/stopwords/stopwords.txt', 'r') as f:\n",
    "    for line in f:\n",
    "        stopwords.append(line.rstrip('\\n'))\n",
    "\n",
    "with open('./data/stopwords/stopwords-ko.txt', 'r') as f:\n",
    "    for line in f:\n",
    "        stopwords.append(line.rstrip('\\n'))\n",
    "        \n",
    "with open('./data/stopwords/stopwords-np.txt', 'r') as f:\n",
    "    tsv_file = csv.reader(f, delimiter=\"\\t\")\n",
    "    for line in tsv_file:\n",
    "        stopwords.append(line[0])\n",
    "\n",
    "stopwords_set = set(stopwords)\n",
    "print(stopwords_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 명사구 추출 함수 정의"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import konlpy\n",
    "import nltk\n",
    "from konlpy.tag import Okt\n",
    "\n",
    "def get_nouns_phrase(text):  \n",
    "    okt = Okt()\n",
    "    phrases = okt.phrases(text)\n",
    "    return phrases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from konlpy.tag import Mecab\n",
    "\n",
    "# mecab = Mecab()\n",
    "\n",
    "# def extract_noun_phrases(text):\n",
    "#     tagged_tokens = mecab.pos(text)\n",
    "#     noun_phrases = []\n",
    "#     current_noun_phrase = []\n",
    "#     tokens = [token for token, _ in tagged_tokens]\n",
    "\n",
    "#     for i, (word, pos) in enumerate(tagged_tokens):\n",
    "#         if pos.startswith('N') or pos.startswith('MM') or (pos.startswith('VA') and i < len(tagged_tokens) - 1 and tagged_tokens[i + 1][1].startswith('N')) or pos.startswith('XR'):\n",
    "#             current_noun_phrase.append((i, pos))\n",
    "#         else:\n",
    "#             if current_noun_phrase:\n",
    "#                 start_idx = current_noun_phrase[0][0]\n",
    "#                 end_idx = current_noun_phrase[-1][0]\n",
    "#                 noun_phrase = ' '.join(tokens[start_idx:end_idx + 1])\n",
    "\n",
    "#                 if len(noun_phrase.split()) > 1 or len(noun_phrase) > 1:\n",
    "#                     noun_phrases.append(noun_phrase)\n",
    "\n",
    "#                 current_noun_phrase = []\n",
    "\n",
    "#         if i < len(tagged_tokens) - 1 and pos.startswith('N') and not tagged_tokens[i + 1][1].startswith('N'):\n",
    "#             if current_noun_phrase:\n",
    "#                 start_idx = current_noun_phrase[0][0]\n",
    "#                 end_idx = current_noun_phrase[-1][0]\n",
    "#                 noun_phrase = ' '.join(tokens[start_idx:end_idx + 1])\n",
    "\n",
    "#                 if len(noun_phrase.split()) > 1 or len(noun_phrase) > 1:\n",
    "#                     noun_phrases.append(noun_phrase)\n",
    "\n",
    "#                 current_noun_phrase = []\n",
    "\n",
    "#     if current_noun_phrase:\n",
    "#         start_idx = current_noun_phrase[0][0]\n",
    "#         end_idx = current_noun_phrase[-1][0]\n",
    "#         noun_phrase = ' '.join(tokens[start_idx:end_idx + 1])\n",
    "\n",
    "#         if len(noun_phrase.split()) > 1 or len(noun_phrase) > 1:\n",
    "#             noun_phrases.append(noun_phrase)\n",
    "\n",
    "#     return noun_phrases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 각 형태소 분석기별 명사 추출 함수 정의\n",
    "\n",
    "형태소 분석을 통해 명사 추출, 불용어 제거, 영어 단어 추출하는 함수"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from konlpy.tag import Mecab \n",
    "from konlpy.tag import Komoran\n",
    "from konlpy.tag import Kkma\n",
    "from konlpy.tag import Okt\n",
    "from collections import Counter\n",
    "\n",
    "def get_nouns_mecab(text, use_phrase=False):\n",
    "    mecab = Mecab()\n",
    "    nouns = mecab.nouns(text)\n",
    "    if use_phrase:\n",
    "        phrases = get_nouns_phrase(text)\n",
    "        nouns.extend(phrases)\n",
    "    eng_nouns = get_nouns_eng(text)\n",
    "    eng_nouns = [ n for n in eng_nouns if len(n) != 1 ]\n",
    "    eng_nouns = remove_stopwords(eng_nouns, eng_stopwords_set)\n",
    "    nouns.extend(eng_nouns)\n",
    "    nouns = remove_stopwords(nouns, stopwords_set)\n",
    "    counts = Counter(nouns)\n",
    "    nouns = set(nouns)\n",
    "    nouns = list(nouns)\n",
    "    \n",
    "    return nouns, counts\n",
    "\n",
    "def get_nouns_komoran(text, use_phrase=False):\n",
    "    komoran = Komoran()\n",
    "    nouns = komoran.nouns(text)\n",
    "    if use_phrase:\n",
    "        phrases = get_nouns_phrase(text)\n",
    "        nouns.extend(phrases)\n",
    "    eng_nouns = get_nouns_eng(text)\n",
    "    eng_nouns = [ n for n in eng_nouns if len(n) != 1 ]\n",
    "    eng_nouns = remove_stopwords(eng_nouns, eng_stopwords_set)\n",
    "    nouns.extend(eng_nouns)\n",
    "    nouns = remove_stopwords(nouns, stopwords_set)\n",
    "    counts = Counter(nouns)\n",
    "    nouns = set(nouns)\n",
    "    nouns = list(nouns)\n",
    "    \n",
    "    return nouns, counts\n",
    "\n",
    "def get_nouns_kkma(text):\n",
    "    kkma = Kkma()\n",
    "    nouns = kkma.nouns(text)\n",
    "    nuons = remove_stopwords(nouns, stopwords_set)\n",
    "    nouns = set(nouns)\n",
    "    nouns = list(nouns)\n",
    "    eng_nouns = get_nouns_eng(text)\n",
    "    eng_nouns = [ n for n in eng_nouns if len(n) != 1 ]\n",
    "    nouns.extend(eng_nouns)\n",
    "    return nouns\n",
    "\n",
    "def get_nouns_okt(text):\n",
    "    okt = Okt()\n",
    "    nouns = okt.nouns(text)\n",
    "    nuons = remove_stopwords(nouns, stopwords_set)\n",
    "    nouns = set(nouns)\n",
    "    nouns = list(nouns)\n",
    "    eng_nouns = get_nouns_eng(text)\n",
    "    eng_nouns = [ n for n in eng_nouns if len(n) != 1 ]\n",
    "    nouns.extend(eng_nouns)\n",
    "    \n",
    "    return nouns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "\n",
    "from string import punctuation\n",
    "import re\n",
    "\n",
    "def get_nouns_eng(text):\n",
    "    # 명사 추출 시 url 제거\n",
    "    url_pattern = re.compile(r'http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\\(\\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+')\n",
    "    text = re.sub(url_pattern, '', text)\n",
    "    url_pattern = re.compile(r\"[-a-zA-Z0-9@:%._\\+~#=]{1,256}\\.[a-zA-Z0-9()]{2,6}\\b([-a-zA-Z0-9()@:%_\\+.~#?&//=]*)\")\n",
    "    text = re.sub(url_pattern, '', text)\n",
    "    \n",
    "    text = text.encode('utf-8').decode('ascii', 'ignore')\n",
    "\n",
    "    result = \"\"\n",
    "    for t in text:\n",
    "        if t not in punctuation:\n",
    "            result += t\n",
    "\n",
    "    text = result.strip()\n",
    "\n",
    "    word_tokens = nltk.word_tokenize(text)\n",
    "    tokens_pos = nltk.pos_tag(word_tokens)\n",
    "    NN_words = []\n",
    "    for word, pos in tokens_pos:\n",
    "        if 'NN' in pos or 'NNP' in pos:\n",
    "            NN_words.append(word)\n",
    "            \n",
    "    return NN_words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 데이터 불러오기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import jsonlines\n",
    "\n",
    "sample = []\n",
    "\n",
    "with jsonlines.open(\"./data/brunch_NLI_test.jsonl\") as f:\n",
    "    for line in f.iter():\n",
    "        if line['likes'] >= 180:\n",
    "            sample.append(line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "217"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 각 KoBigBird 모델 불러오기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/user03/anaconda3/envs/mecab/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n",
      "Some weights of the model checkpoint at monologg/kobigbird-bert-base were not used when initializing BigBirdModel: ['cls.predictions.decoder.weight', 'bert.pooler.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.weight', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'bert.pooler.bias']\n",
      "- This IS expected if you are initializing BigBirdModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BigBirdModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    }
   ],
   "source": [
    "from transformers import  AutoModel,AutoTokenizer\n",
    "\n",
    "model = AutoModel.from_pretrained(\"monologg/kobigbird-bert-base\", attention_type=\"original_full\", add_pooling_layer=False)\n",
    "#model = AutoModel.from_pretrained(\"./model/sts/saved_model_epoch_7.pt\", attention_type=\"original_full\", add_pooling_layer=False)\n",
    "#model = AutoModel.from_pretrained(\"./model/nli/saved_model_epoch_2.pt\", attention_type=\"original_full\", add_pooling_layer=False)\n",
    "#model = AutoModel.from_pretrained(\"./model/word_to_title/saved_model_epoch_3.pt\", attention_type=\"original_full\", add_pooling_layer=False)\n",
    "#model = AutoModel.from_pretrained(\"./model/title_to_text/saved_model_epoch_2.pt\", attention_type=\"original_full\", add_pooling_layer=False)\n",
    "#model = AutoModel.from_pretrained(\"./model/word_to_text_fm3/saved_model_epoch_3.pt\", attention_type=\"original_full\", add_pooling_layer=False)\n",
    "#model = AutoModel.from_pretrained(\"./model/word_to_text_sts_way/saved_model_epoch_2.pt\", attention_type=\"original_full\", add_pooling_layer=False)\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"monologg/kobigbird-bert-base\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BigBirdModel(\n",
       "  (embeddings): BigBirdEmbeddings(\n",
       "    (word_embeddings): Embedding(32500, 768, padding_idx=0)\n",
       "    (position_embeddings): Embedding(4096, 768)\n",
       "    (token_type_embeddings): Embedding(2, 768)\n",
       "    (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "    (dropout): Dropout(p=0.1, inplace=False)\n",
       "  )\n",
       "  (encoder): BigBirdEncoder(\n",
       "    (layer): ModuleList(\n",
       "      (0): BigBirdLayer(\n",
       "        (attention): BigBirdAttention(\n",
       "          (self): BigBirdSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BigBirdSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BigBirdIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          (intermediate_act_fn): NewGELUActivation()\n",
       "        )\n",
       "        (output): BigBirdOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (1): BigBirdLayer(\n",
       "        (attention): BigBirdAttention(\n",
       "          (self): BigBirdSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BigBirdSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BigBirdIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          (intermediate_act_fn): NewGELUActivation()\n",
       "        )\n",
       "        (output): BigBirdOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (2): BigBirdLayer(\n",
       "        (attention): BigBirdAttention(\n",
       "          (self): BigBirdSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BigBirdSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BigBirdIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          (intermediate_act_fn): NewGELUActivation()\n",
       "        )\n",
       "        (output): BigBirdOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (3): BigBirdLayer(\n",
       "        (attention): BigBirdAttention(\n",
       "          (self): BigBirdSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BigBirdSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BigBirdIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          (intermediate_act_fn): NewGELUActivation()\n",
       "        )\n",
       "        (output): BigBirdOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (4): BigBirdLayer(\n",
       "        (attention): BigBirdAttention(\n",
       "          (self): BigBirdSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BigBirdSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BigBirdIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          (intermediate_act_fn): NewGELUActivation()\n",
       "        )\n",
       "        (output): BigBirdOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (5): BigBirdLayer(\n",
       "        (attention): BigBirdAttention(\n",
       "          (self): BigBirdSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BigBirdSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BigBirdIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          (intermediate_act_fn): NewGELUActivation()\n",
       "        )\n",
       "        (output): BigBirdOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (6): BigBirdLayer(\n",
       "        (attention): BigBirdAttention(\n",
       "          (self): BigBirdSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BigBirdSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BigBirdIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          (intermediate_act_fn): NewGELUActivation()\n",
       "        )\n",
       "        (output): BigBirdOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (7): BigBirdLayer(\n",
       "        (attention): BigBirdAttention(\n",
       "          (self): BigBirdSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BigBirdSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BigBirdIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          (intermediate_act_fn): NewGELUActivation()\n",
       "        )\n",
       "        (output): BigBirdOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (8): BigBirdLayer(\n",
       "        (attention): BigBirdAttention(\n",
       "          (self): BigBirdSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BigBirdSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BigBirdIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          (intermediate_act_fn): NewGELUActivation()\n",
       "        )\n",
       "        (output): BigBirdOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (9): BigBirdLayer(\n",
       "        (attention): BigBirdAttention(\n",
       "          (self): BigBirdSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BigBirdSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BigBirdIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          (intermediate_act_fn): NewGELUActivation()\n",
       "        )\n",
       "        (output): BigBirdOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (10): BigBirdLayer(\n",
       "        (attention): BigBirdAttention(\n",
       "          (self): BigBirdSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BigBirdSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BigBirdIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          (intermediate_act_fn): NewGELUActivation()\n",
       "        )\n",
       "        (output): BigBirdOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (11): BigBirdLayer(\n",
       "        (attention): BigBirdAttention(\n",
       "          (self): BigBirdSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BigBirdSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BigBirdIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          (intermediate_act_fn): NewGELUActivation()\n",
       "        )\n",
       "        (output): BigBirdOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
    "\n",
    "model.to('cuda')\n",
    "model.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 최종 키워드 추출 함수(빈도수에 따른 가중치 반영)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_main_keywords(nouns, counts, title, text, model, tokenizer, use_freq_w=False):\n",
    "    \n",
    "    tokenized_nouns = {}\n",
    "    for keyword in nouns:\n",
    "        tokenized_nouns[keyword] = tokenizer(keyword, max_length=32, padding='max_length', return_tensors='pt')\n",
    "    \n",
    "    keyword_vectors = {}\n",
    "    for keyword, tokenized in tokenized_nouns.items():\n",
    "        output = model(**tokenized.to(device))[0][:,0,:]\n",
    "        keyword_vectors[keyword] = output\n",
    "        \n",
    "    tokenized_title = tokenizer(title, max_length=128, padding=\"max_length\", return_tensors='pt')\n",
    "    tokenized_text = tokenizer(text, max_length=4096, padding=\"max_length\", return_tensors='pt')\n",
    "\n",
    "    title_vec = model(**tokenized_title.to(device))[0][:,0,:]\n",
    "    body_text_vec = model(**tokenized_text.to(device))[0][:,0,:]\n",
    "    \n",
    "    \n",
    "    if use_freq_w:\n",
    "        total_words_counts = 0\n",
    "        for word, count in counts.items():\n",
    "            if count != 1:\n",
    "                total_words_counts += count\n",
    "        print(total_words_counts)\n",
    "        \n",
    "        freq_w = {}\n",
    "    \n",
    "        for keyword in nouns:\n",
    "            if counts[keyword] <= 1:\n",
    "                freq_w[keyword] = 0\n",
    "                continue\n",
    "            freq_w[keyword] = (counts[keyword]/total_words_counts)\n",
    "        \n",
    "        print(freq_w)\n",
    "        for keyword, vec in keyword_vectors.items():\n",
    "            body_text_vec = ((freq_w[keyword]* vec) + body_text_vec)\n",
    "        \n",
    "        body_text_vec = body_text_vec/total_words_counts\n",
    "        \n",
    "    cos = nn.CosineSimilarity(dim=1, eps=1e-6)\n",
    "    similarity = {}\n",
    "\n",
    "    for keyword, vector in keyword_vectors.items():\n",
    "        sim = cos(vector,  title_vec*(0.0) + body_text_vec*(1.0))\n",
    "        similarity[keyword] = sim\n",
    "        \n",
    "    sorted_dict = sorted(similarity.items(), key = lambda item: item[1], reverse = True)\n",
    "    \n",
    "    # print(keyword_vectors)\n",
    "    print(f\"제목과 본문의 유사도 {cos(title_vec, body_text_vec)}\\n\")\n",
    "    print(title+'\\n')\n",
    "    print(text+'\\n')\n",
    "    print(sorted_dict,'\\n')\n",
    "    #print(nouns)\n",
    "    \n",
    "    return keyword_vectors, similarity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 클러스터링 후 클러스터 내 1위 단어만 추출(동의어 제거)\n",
    "\n",
    "사이킷런의 DBSCAN 사용"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "  \n",
    "from sklearn.cluster import DBSCAN\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.preprocessing import normalize\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "\n",
    "def cluster_dbscan(eps, min_samples, vec_dict, sim_dict, max_words=10):\n",
    "    word_features_numpy = {}\n",
    "    for word, feat in vec_dict.items():\n",
    "        word_features_numpy[word] = feat.to('cpu')[0].numpy()\n",
    "        \n",
    "    features_df = pd.DataFrame(word_features_numpy)\n",
    "    features_df = features_df.transpose()\n",
    "    \n",
    "    clustering = DBSCAN(eps=eps, min_samples=min_samples).fit(features_df)\n",
    "    DBSCAN_dataset = features_df.copy()\n",
    "    DBSCAN_dataset.loc[:,'Cluster'] = clustering.labels_ \n",
    "    \n",
    "    print(DBSCAN_dataset.Cluster.value_counts().to_frame())\n",
    "    print(clustering.labels_ .max())\n",
    "    \n",
    "    sim_list = []\n",
    "    for word in DBSCAN_dataset.index:\n",
    "        sim_list.append(sim_dict[word].item())\n",
    "        \n",
    "    DBSCAN_dataset['sim'] = sim_list\n",
    "    \n",
    "    cluster_list = DBSCAN_dataset.Cluster.to_list()\n",
    "    cluseter_set = set(cluster_list)\n",
    "    cluster_list = list(cluseter_set)\n",
    "    \n",
    "    selected_words = []\n",
    "\n",
    "    for cluster in cluster_list:\n",
    "        if cluster == -1 or cluster == 0:\n",
    "            df = DBSCAN_dataset[DBSCAN_dataset['Cluster']==cluster]\n",
    "            #print(df.index.to_list())\n",
    "            selected_words.extend(df.index.to_list())\n",
    "        else:\n",
    "            df = DBSCAN_dataset[DBSCAN_dataset['Cluster']==cluster]\n",
    "            df = df.sort_values('sim', ascending=False)\n",
    "            top_word = df.iloc[0].name\n",
    "            selected_words.append(top_word)\n",
    "            \n",
    "    selected_words_dict = {}\n",
    "\n",
    "    for word in selected_words:\n",
    "        selected_words_dict[word] = sim_dict[word]\n",
    "    sorted_selected_words = sorted(selected_words_dict.items(), key = lambda item: item[1], reverse = True)\n",
    "    \n",
    "    return sorted_selected_words[:max_words]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['가족', '에세이', '호칭']\n",
      "제목과 본문의 유사도 tensor([0.9065], device='cuda:0')\n",
      "\n",
      "나는 가끔 아파를 아빠라 부른다.\n",
      "\n",
      " 내 글을 우연히 읽었다며 지인은 물었다.  매영씨, 매영씨는 왜 아버지를 아빠라 부르시나요? 아버지가 더 거리감 있어 보이는데요. 아빠라는 호칭은 너무 가깝지 않나요? 어색한 웃음이 흘러나왔다. 나는 왜 아빠를 아빠라 부르는 걸까. 마땅한 대답이 생각나지 않았다. 날이 좋다고, 오늘은 우리 맛있는 것을 먹자며 말을 돌렸다. 창 밖에는 비가 오고 있었다. 헤어지고 집에 오는 길에 아버지라는 말을 내내 중얼거렸다. 기름칠을 한 것처럼 입에서 미끄러져 흘렀다. 아버지라는 말을 어디서 처음 들었던가. 드라마였던 것 같다. 드라마 속 아버지들은 대부분 권위적이지만 능력이 있었다. 아들에게 좋은 스승이자 라이벌이었다. 아들과 아버지가 싸우는 이유는 뛰어넘기 위한 발버둥이거나 사랑 때문이었다. 우리에겐 없는 것이었다. 집에 도착해서는 아빠라는 말을 한참 동안 중얼거렸다. 아빠. 아파. 아파. 아빠 발음할수록 아빠는 아파가 되고 아파는 아빠가 되었다. 아직도 폭력의 여운이 사라지지 않았다는 것이겠지. 내게 아버지란 단어는 너무 멀고 무겁다.   어떻게 아빠를 아빠라고 부르지 않을 수 있을까. \n",
      "\n",
      "[('아빠', tensor([0.9181], device='cuda:0')), ('아버지', tensor([0.9043], device='cuda:0')), ('대답', tensor([0.9012], device='cuda:0')), ('사랑', tensor([0.8968], device='cuda:0')), ('스승', tensor([0.8966], device='cuda:0')), ('길', tensor([0.8926], device='cuda:0')), ('창', tensor([0.8917], device='cuda:0')), ('글', tensor([0.8904], device='cuda:0')), ('발버둥', tensor([0.8904], device='cuda:0')), ('기름', tensor([0.8896], device='cuda:0')), ('단어', tensor([0.8860], device='cuda:0')), ('여운', tensor([0.8854], device='cuda:0')), ('라이벌', tensor([0.8842], device='cuda:0')), ('폭력', tensor([0.8796], device='cuda:0')), ('아파', tensor([0.8778], device='cuda:0')), ('거리', tensor([0.8773], device='cuda:0')), ('날', tensor([0.8766], device='cuda:0')), ('비', tensor([0.8766], device='cuda:0')), ('웃음', tensor([0.8751], device='cuda:0')), ('호칭', tensor([0.8741], device='cuda:0')), ('입', tensor([0.8730], device='cuda:0')), ('발음', tensor([0.8699], device='cuda:0')), ('아들', tensor([0.8696], device='cuda:0')), ('이유', tensor([0.8689], device='cuda:0')), ('내게', tensor([0.8666], device='cuda:0')), ('능력', tensor([0.8665], device='cuda:0')), ('드라마', tensor([0.8662], device='cuda:0')), ('지인', tensor([0.8629], device='cuda:0')), ('걸까', tensor([0.8604], device='cuda:0')), ('위', tensor([0.8592], device='cuda:0')), ('도착', tensor([0.8563], device='cuda:0')), ('한참', tensor([0.8445], device='cuda:0')), ('대부분', tensor([0.8224], device='cuda:0'))] \n",
      "\n",
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n",
      "    Cluster\n",
      "-1       22\n",
      " 1        6\n",
      " 2        3\n",
      " 0        2\n",
      "2\n",
      "[('아빠', tensor([0.9181], device='cuda:0')), ('대답', tensor([0.9012], device='cuda:0')), ('창', tensor([0.8917], device='cuda:0')), ('발버둥', tensor([0.8904], device='cuda:0')), ('단어', tensor([0.8860], device='cuda:0')), ('라이벌', tensor([0.8842], device='cuda:0')), ('폭력', tensor([0.8796], device='cuda:0')), ('아파', tensor([0.8778], device='cuda:0')), ('거리', tensor([0.8773], device='cuda:0')), ('날', tensor([0.8766], device='cuda:0')), ('비', tensor([0.8766], device='cuda:0')), ('웃음', tensor([0.8751], device='cuda:0')), ('호칭', tensor([0.8741], device='cuda:0')), ('입', tensor([0.8730], device='cuda:0')), ('발음', tensor([0.8699], device='cuda:0')), ('아들', tensor([0.8696], device='cuda:0')), ('이유', tensor([0.8689], device='cuda:0')), ('내게', tensor([0.8666], device='cuda:0')), ('능력', tensor([0.8665], device='cuda:0')), ('드라마', tensor([0.8662], device='cuda:0'))]\n"
     ]
    }
   ],
   "source": [
    "sample_essay = sample[200]\n",
    "\n",
    "with torch.no_grad():\n",
    "    print(sample_essay['keywords'])\n",
    "    \n",
    "    title = sample_essay['title']\n",
    "    text = sample_essay['body_text']\n",
    "    text = text.replace('\\n','').replace('\\t','').replace('\\r','') # 코모란 에러 방지\n",
    "    nouns, counts = get_nouns_mecab(text, use_phrase=False)\n",
    "    vec_dict, sim_dict = extract_main_keywords(nouns, counts, title, text, model, tokenizer, use_freq_w=False)\n",
    "    keywords = cluster_dbscan(eps=4.9, min_samples=2, vec_dict=vec_dict, sim_dict=sim_dict, max_words=20)\n",
    "    print(keywords)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### kr-wordrank와 비교"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      아빠:\t5.2586\n",
      "      말을:\t4.1436\n",
      "      아버:\t2.9174\n",
      "      사랑:\t2.8845\n",
      "      집에:\t2.8730\n",
      "      위한:\t2.5356\n",
      "     대부분:\t2.5059\n",
      "    있었다.:\t2.3040\n",
      "      있어:\t2.1997\n",
      "     능력이:\t2.1779\n",
      "     웃음이:\t1.9623\n",
      "     아들과:\t1.9526\n",
      "      너무:\t1.9155\n",
      "    않았다는:\t1.9000\n",
      "      없는:\t1.8874\n",
      "    스승이자:\t1.8707\n",
      "      처음:\t1.8506\n",
      "  중얼거렸다.:\t1.8174\n",
      "      좋은:\t1.7735\n",
      "     지인은:\t1.7099\n"
     ]
    }
   ],
   "source": [
    "from krwordrank.word import KRWordRank\n",
    "\n",
    "min_count = 1   # 단어의 최소 출현 빈도수 (그래프 생성 시)\n",
    "max_length = 15 # 단어의 최대 길이\n",
    "wordrank_extractor = KRWordRank(min_count=min_count, max_length=max_length)\n",
    "\n",
    "beta = 0.85    # PageRank의 decaying factor beta\n",
    "max_iter = 20\n",
    "texts = [text]\n",
    "keywords, rank, graph = wordrank_extractor.extract(texts, beta, max_iter)\n",
    "\n",
    "for word, r in sorted(keywords.items(), key=lambda x:x[1], reverse=True)[:20]:\n",
    "        print('%8s:\\t%.4f' % (word, r))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mecab",
   "language": "python",
   "name": "mecab"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}