[국민대 AI빅데이터 분석 경진대회] 회고록

부트캠프를 다니면서 타이타닉을 해보며 대회의 재미를 보았다.

 

이번에 처음 데이콘에서 대회를 나가 참가를 해보았다.

 

 

부트캠프 사람들이 아직 친하지 않아 직접 팀을 구성해 1팀 ~ 5팀을 만들어 대회를 참가하게 했다.

팀을 만들다보니 본인은 혼자 참가하게 되었다 :(

 

 1주차 - Cosine Similarity를 처음 써보면서 익혀나갔다.

부스팅을 쓰는 것은 가지고 있는 data로 다른 사람들의 data에 대한 예측을 하는 것이라 생각했고 이는 같은 data이니 필요 없다고 생각했다. 이때부터 늪에 빠지기 시작했다.

 

 

 

 

이렇게 cosine 유사도 뿐만 아니라 negative sampling과 score를 사용하는 모델 등 다양하게 시도를 해봤다.

하지만 기본 베이스 점수 자체가 높아서 그런지 0.159를 넘기기가 쉽지가 않았고

 

추천 시스템이다 보니 data -  apply에 이미 feature들이 함유되어 있는거 같다. 

 

2주차, 3주차 까지 feature들을 뽑아 그것으로 사람들의 추천을 가져올 수 있을 거라 생각했지만 결국  apply로만의 cosine유사도 점수를 이기질 못했다.

 

마지막 주차에서 feature들은 조건들이다 생각, apply data들 중 제대로 된 조건들로 유사도를 뽑아 내야겠다는 생각을 했다. 

 

EDA를 한걸 보면

 

 

 

 

 

 

 

 

 

 

 

 

 

이런 식으로 수치를 보면서 걸러낼 것들을 걸러 내면서 진행했다.

feature를 뽑겠다 보다 내 생각에 맞춰 데이터를 뽑아서 진행했고 그로인해 내 주장의 타당한 이유를 뽑아내기 어려웠다.

 

 

 

 

 

감상평 - 모델을 구축하기도 쉽지 않았고 feature를 뽑아 낼 때마다 점수가 오르는 재미가 없고 내려가기만 하니 자괴감이 들었다. 다른 사람들의 코드를 보고 생각 하지 못한 부분이 있으면 배워서 다음 대회에도 쓰고자 한다.

첫 대회에 혼자 나가 고생을 많이 했지만 생각보다 체계적으로 하지 못했고 감으로 어림잡아 했던 것이 크다.

모델 또한 직접 구현하는 것으로 모델을 구축하는데, 아이디어를 떠오르는데 시간을 많이 썼고 EDA도 의미 있는걸 뽑아내지 못했다. 또한 public에 집중을해 결국 overfit이 났기에 타이타닉 때 처럼 적절한 선을 못잡았던거 같다.

 

최종 : Public : 47등, Private : 135등 <- overfit, 다른 Public 점수가 낮은게 Private에서 더 높은 게 있었다.

첫 대회는 뼈아팠지만 모델을 처음 부터 구축하고 EDA가 아쉽지만 시간 투자를 해 증명을 하면서 정리해야겠다.

 

 

끝으로 최종 제출 전전 모델을 공유한다.

import pandas as pd
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
import category_encoders as ce
from surprise import SVD
from surprise import Dataset
from surprise import accuracy
from surprise.model_selection import train_test_split
from surprise.dataset import DatasetAutoFolds
import os
import matplotlib.pyplot as plt
import seaborn as sns

def recall5(answer_df, submission_df):

    primary_col = answer_df.columns[0]
    secondary_col = answer_df.columns[1]

    # submission의 예측이 각각 5개인지 확인
    prediction_counts = submission_df.groupby(primary_col).size()
    if not all(prediction_counts == 5):
        raise ValueError(f"Each {primary_col} should have exactly 5 {secondary_col} predictions.")


    # submission의 예측된 값들에 null값이 있는지 확인
    if submission_df[secondary_col].isnull().any():
        raise ValueError(f"Predicted {secondary_col} contains NULL values.")

    # 예측값에 중복이 있는지 확인
    duplicated_preds = submission_df.groupby(primary_col).apply(lambda x: x[secondary_col].duplicated().any())
    if duplicated_preds.any():
        raise ValueError(f"Predicted {secondary_col} contains duplicates for some {primary_col}.")


    # Filter the submission dataframe based on the primary_col present in the answer dataframe
    # primary_col 즉 resume_seq가 양측에 있는지 확인 후 남김
    submission_df = submission_df[submission_df[primary_col].isin(answer_df[primary_col])]

    # For each primary_col, get the top 5 predicted secondary_col values
    #
    top_5_preds = submission_df.groupby(primary_col).apply(lambda x: x[secondary_col].head(5).tolist()).to_dict()

    # Convert the answer_df to a dictionary for easier lookup
    true_dict = answer_df.groupby(primary_col).apply(lambda x: x[secondary_col].tolist()).to_dict()


    individual_recalls = []
    for key, val in true_dict.items():
        if key in top_5_preds:
            correct_matches = len(set(true_dict[key]) & set(top_5_preds[key]))
            individual_recall = correct_matches / min(len(val), 5) # 공정한 평가를 가능하게 위하여 분모(k)를 'min(len(val), 5)' 로 설정함
            individual_recalls.append(individual_recall)


    recall = np.mean(individual_recalls)
    return recall

def minimum_condition(resume, company):
  if(resume['degree'] < company['education']):
    return False
  else:
    return True
   # if(resume['career_month']>company['career_end'] or resume['career_month']<company['career_start']): 다 0임
     # return False
   # else:

def check_salary(salary):
  if salary>3000:
    return 3
  elif salary>2000:
    return 2
  elif salary>1000:
    return 1
  else:
    return 0
  

def check_career_month(career_month):
  if career_month>120:
    return 3
  elif career_month>60:
    return 2
  elif career_month>0:
    return 1
  else:
    return 0
  
def check_univ_score(univ_score):
  if univ_score>=90:
    return 90
  elif univ_score>=80:
    return 80
  elif univ_score>=70:
    return 70
  elif univ_score>=60:
    return 60
  elif univ_score>=50:
    return 50
  elif univ_score>=40:
    return 40
  elif univ_score>=30:
    return 30
  elif univ_score>=20:
    return 20
  elif univ_score>=10:
    return 10
  else:
    return 0
  



apply_train_df = pd.read_csv('apply_train.csv')
resume_train_df = pd.read_csv('resume.csv')
certificate_df = pd.read_csv('resume_certificate.csv')
education_train_df = pd.read_csv('resume_education.csv')
language_train_df = pd.read_csv('resume_language.csv')
company_train_df = pd.read_csv('company.csv')
recruitment_train_df = pd.read_csv('recruitment.csv')

all_apply_matrix = apply_train_df.groupby(['resume_seq', 'recruitment_seq']).size().unstack(fill_value=0)

count_by_recruitment_seq = apply_train_df.groupby('recruitment_seq')['resume_seq'].count()
count_by_recruitment_seq = count_by_recruitment_seq.reset_index()
count_by_recruitment_seq.columns = ['recruitment_seq', 'rating']

total_user = resume_train_df.sort_values(by='resume_seq')
certificate_df = certificate_df.dropna(subset=['certificate_contents'])
tmp = certificate_df.groupby('resume_seq')['certificate_contents'].apply(';'.join).reset_index()
total_user = pd.merge(total_user, tmp, on='resume_seq', how='left')
total_user = pd.merge(total_user, education_train_df, on='resume_seq', how='left')
total_user = pd.merge(total_user, apply_train_df, on='resume_seq', how='left')

total_item = recruitment_train_df
total_item = pd.merge(total_item, company_train_df, on='recruitment_seq', how='left')
total_item = pd.merge(total_item, count_by_recruitment_seq, on='recruitment_seq', how='left')

total = pd.merge(total_user, total_item, on='recruitment_seq', how='left')
average_address_seq1 = total.groupby('resume_seq')['address_seq1'].mean().reset_index()
average_address_seq1.columns = ['resume_seq', 'location']
total = pd.merge(total, average_address_seq1, on='resume_seq', how='left')

total_train = total[(total['address_seq1'] - total['location']<10) & (total['address_seq1'] - total['location']>-10)]

total_train2 = total[total['education'] <= total['degree']]
total_train3 = total[total['education'] > total['degree']]
not_in_total_train2 = total_train3[~total_train3['resume_seq'].isin(total_train2['resume_seq'])]
merged_total = pd.concat([total_train2, not_in_total_train2], ignore_index=True)


apply_train_m = merged_total[['resume_seq', 'recruitment_seq']]
apply_pred_m = total[['resume_seq', 'recruitment_seq']]
apply_train_groupby = apply_train_m.groupby('resume_seq')['recruitment_seq'].apply(list)
apply_pred_groupby = apply_pred_m.groupby('resume_seq')['recruitment_seq'].apply(list)

#학습, 검증 분리
train, test = [], []
for uid, iids in zip(apply_train_groupby.index.tolist(), apply_train_groupby.values.tolist()):
    if(len(iids)>1):
      for iid in iids[:-1]:
        train.append([uid,iid])
      test.append([uid, iids[-1]])
    else:
        train.append([uid,iid])
        test.append([uid, iid])
apply_train = pd.DataFrame(train, columns=['resume_seq', 'recruitment_seq'])
apply_test = pd.DataFrame(test, columns=['resume_seq', 'recruitment_seq'])
apply_pred = apply_train_m.copy()


resume_d = total[['resume_seq', 'location']]
recruitment_d = total[['recruitment_seq', 'address_seq1']]

resume_d = resume_d.drop_duplicates()
recruitment_d = recruitment_d.drop_duplicates()

resume_d = resume_d.set_index('resume_seq')
recruitment_d = recruitment_d.set_index('recruitment_seq')

resume_d['location'] = resume_d['location'].astype(int)

distance_matrix_new = []
count = 0

tem_2 = [] # U00140
tem_3 = [] # U00004
tem_4 = [] # U00292
tem_5 = [] # U00424
tem_6 = [] # U01105
tem_7 = [] # U01461
tem_8 = [] # U01703
tem_11 = [] # U02309
tem_14 = [] # U02786


for recruitment_seq in range(len(recruitment_d)): #
  if abs(resume_d.loc['U00140']['location'] - recruitment_d.iloc[recruitment_seq]['address_seq1']) <= 3:
        a = True
  else:
        a = False
  tem_2.append(a)

for recruitment_seq in range(len(recruitment_d)): #
  if abs(resume_d.loc['U00004']['location'] - recruitment_d.iloc[recruitment_seq]['address_seq1']) <= 3:
        a = True
  else:
        a = False
  tem_3.append(a)

for recruitment_seq in range(len(recruitment_d)):
  if abs(resume_d.loc['U00292']['location'] - recruitment_d.iloc[recruitment_seq]['address_seq1']) <= 3:
        a = True
  else:
        a = False
  tem_4.append(a)

for recruitment_seq in range(len(recruitment_d)):
  if abs(resume_d.loc['U00424']['location'] - recruitment_d.iloc[recruitment_seq]['address_seq1']) <= 3:
        a = True
  else:
        a = False
  tem_5.append(a)

for recruitment_seq in range(len(recruitment_d)):
  if abs(resume_d.loc['U01105']['location'] - recruitment_d.iloc[recruitment_seq]['address_seq1']) <= 3:
        a = True
  else:
        a = False
  tem_6.append(a)

for recruitment_seq in range(len(recruitment_d)):
  if abs(resume_d.loc['U01461']['location'] - recruitment_d.iloc[recruitment_seq]['address_seq1']) <= 3:
        a = True
  else:
        a = False
  tem_7.append(a)

for recruitment_seq in range(len(recruitment_d)):
  if abs(resume_d.loc['U01703']['location'] - recruitment_d.iloc[recruitment_seq]['address_seq1']) <= 3:
        a = True
  else:
        a = False
  tem_8.append(a)

for recruitment_seq in range(len(recruitment_d)):
  if abs(resume_d.loc['U02309']['location'] - recruitment_d.iloc[recruitment_seq]['address_seq1']) <= 3:
        a = True
  else:
        a = False
  tem_11.append(a)

for recruitment_seq in range(len(recruitment_d)):
  if abs(resume_d.loc['U02786']['location'] - recruitment_d.iloc[recruitment_seq]['address_seq1']) <= 3:
        a = True
  else:
        a = False
  tem_14.append(a)



# apply_item_matrix의 인덱스(=resume_seq)와 컬럼(=recruitment_seq)을 뽑아냄
for resume_seq in range(len(resume_d)):
    if resume_d.iloc[resume_seq]['location'] == 2:
      distance_matrix_new.append(tem_2)
    elif resume_d.iloc[resume_seq]['location'] == 3:
      distance_matrix_new.append(tem_3)
    elif resume_d.iloc[resume_seq]['location'] == 4:
      distance_matrix_new.append(tem_4)
    elif resume_d.iloc[resume_seq]['location'] == 5:
      distance_matrix_new.append(tem_5)
    elif resume_d.iloc[resume_seq]['location'] == 6:
      distance_matrix_new.append(tem_6)
    elif resume_d.iloc[resume_seq]['location'] == 7:
      distance_matrix_new.append(tem_7)
    elif resume_d.iloc[resume_seq]['location'] == 8:
      distance_matrix_new.append(tem_8)
    elif resume_d.iloc[resume_seq]['location'] == 11:
      distance_matrix_new.append(tem_11)
    else:
      distance_matrix_new.append(tem_2)

# 새로운 DataFrame 생성
distance_matrix_new = pd.DataFrame(distance_matrix_new, index=all_apply_matrix.index, columns=all_apply_matrix.columns)


resume_c = resume_train_df.copy()
recruitment_c = recruitment_train_df.copy()
# 'resume_seq' 열을 기준으로 정렬
resume_c = resume_c.sort_values(by='resume_seq')
# 인덱스를 순차적으로 변경
resume_c['resume_seq'] = [f'U{i:05}' for i in range(1, len(resume_c) + 1)]
resume_c = resume_c.set_index('resume_seq')

# 'resume_seq' 열을 기준으로 정렬
recruitment_c = recruitment_c.sort_values(by='recruitment_seq')
# 인덱스를 순차적으로 변경
recruitment_c['recruitment_seq'] = [f'R{i:05}' for i in range(1, len(recruitment_c) + 1)]
recruitment_c = recruitment_c.set_index('recruitment_seq')

resume_c = resume_c[['degree']]
recruitment_c = recruitment_c[['education']]

apply_item_matrix_new = []
count = 0
# 초등학생들 ^^
tem_6 = [] # U05886이 6학년
tem_5 = [] # U08422이 5학년
tem_4 = [] # U04600이 4학년
tem_3 = [] # U03400이 3학년
tem_2 = [] # U04220이 2학년

for recruitment_seq in range(len(recruitment_c)): #
  if resume_c.loc['U05886']['degree'] < recruitment_c.iloc[recruitment_seq]['education']:
        a = False
  else:
        a = True
  tem_6.append(a)

for recruitment_seq in range(len(recruitment_c)):
  if resume_c.loc['U08422']['degree'] < recruitment_c.iloc[recruitment_seq]['education']:
        a = False
  else:
        a = True
  tem_5.append(a)

for recruitment_seq in range(len(recruitment_c)):
  if ((resume_c.loc['U04600']['degree'] < recruitment_c.iloc[recruitment_seq]['education']) and ((recruitment_c.iloc[recruitment_seq]['education'] - resume_c.loc['U04600']['degree'])< -1)):
        a = False
  else:
        a = True
  tem_4.append(a)

for recruitment_seq in range(len(recruitment_c)):
  if resume_c.loc['U03400']['degree'] < recruitment_c.iloc[recruitment_seq]['education']:
        a = False
  else:
        a = True
  tem_3.append(a)

for recruitment_seq in range(len(recruitment_c)):
  if resume_c.loc['U04220']['degree'] < recruitment_c.iloc[recruitment_seq]['education']:
        a = False
  else:
        a = True
  tem_2.append(a)

# apply_item_matrix의 인덱스(=resume_seq)와 컬럼(=recruitment_seq)을 뽑아냄
for resume_seq in range(len(resume_c)):
    if resume_c.iloc[resume_seq]['degree'] == 6:
      apply_item_matrix_new.append(tem_6)
    elif resume_c.iloc[resume_seq]['degree'] == 5:
      apply_item_matrix_new.append(tem_5)
    elif resume_c.iloc[resume_seq]['degree'] == 4:
      apply_item_matrix_new.append(tem_4)
    elif resume_c.iloc[resume_seq]['degree'] == 3:
      apply_item_matrix_new.append(tem_3)
    else:
      apply_item_matrix_new.append(tem_2)

# 새로운 DataFrame 생성
apply_item_matrix_new = pd.DataFrame(apply_item_matrix_new, index=all_apply_matrix.index, columns=all_apply_matrix.columns)

check_degree = total[['resume_seq', 'degree']]
check_degree = check_degree.drop_duplicates(subset='resume_seq')

apply_train = pd.merge(apply_train, check_degree, on = 'resume_seq', how = 'left')
apply_pred = pd.merge(apply_pred, check_degree, on = 'resume_seq', how = 'left')

apply_train_2 = apply_train[apply_train['degree']==2]
apply_train_3 = apply_train[apply_train['degree']==3]
apply_train_4 = apply_train[apply_train['degree']==4]
apply_train_5 = apply_train[(apply_train['degree']==5) | (apply_train['degree']==6)]

apply_pred_2 = apply_pred[apply_pred['degree']==2]
apply_pred_3 = apply_pred[apply_pred['degree']==3]
apply_pred_4 = apply_pred[apply_pred['degree']==4]
apply_pred_5 = apply_pred[(apply_pred['degree']==5) | (apply_pred['degree']==6)]

# 사용자-아이템 행렬 생성: 구직자가 해당 채용 공고에 지원했으면 1, 아니면 0으로 설정
apply_train_item_matrix_2 = apply_train_2.groupby(['resume_seq', 'recruitment_seq']).size().unstack(fill_value=0)
apply_train_item_matrix_3 = apply_train_3.groupby(['resume_seq', 'recruitment_seq']).size().unstack(fill_value=0)
apply_train_item_matrix_4 = apply_train_4.groupby(['resume_seq', 'recruitment_seq']).size().unstack(fill_value=0)
apply_train_item_matrix_5 = apply_train_5.groupby(['resume_seq', 'recruitment_seq']).size().unstack(fill_value=0)

apply_train_similarity_2 = cosine_similarity(apply_train_item_matrix_2) # 사용자 간의 유사성 계산
apply_train_similarity_3 = cosine_similarity(apply_train_item_matrix_3) # 사용자 간의 유사성 계산
apply_train_similarity_4 = cosine_similarity(apply_train_item_matrix_4) # 사용자 간의 유사성 계산
apply_train_similarity_5 = cosine_similarity(apply_train_item_matrix_5) # 사용자 간의 유사성 계산


apply_pred_item_matrix_2 = apply_pred_2.groupby(['resume_seq', 'recruitment_seq']).size().unstack(fill_value=0)
apply_pred_item_matrix_3 = apply_pred_3.groupby(['resume_seq', 'recruitment_seq']).size().unstack(fill_value=0)
apply_pred_item_matrix_4 = apply_pred_4.groupby(['resume_seq', 'recruitment_seq']).size().unstack(fill_value=0)
apply_pred_item_matrix_5 = apply_pred_5.groupby(['resume_seq', 'recruitment_seq']).size().unstack(fill_value=0)

apply_pred_similarity_2 = cosine_similarity(apply_pred_item_matrix_2) # 사용자 간의 유사성 계산
apply_pred_similarity_3 = cosine_similarity(apply_pred_item_matrix_3) # 사용자 간의 유사성 계산
apply_pred_similarity_4 = cosine_similarity(apply_pred_item_matrix_4) # 사용자 간의 유사성 계산
apply_pred_similarity_5 = cosine_similarity(apply_pred_item_matrix_5) # 사용자 간의 유사성 계산




# train_사용자 간의 유사성 계산
user_train_similarity_2 = apply_train_similarity_2
user_predicted_scores_2 = user_train_similarity_2.dot(apply_train_item_matrix_2) / np.array([np.abs(user_train_similarity_2).sum(axis=1)]).T

item_train_similarity_2 = cosine_similarity(apply_train_item_matrix_2.T)
item_train_predicted_scores_2 = apply_train_item_matrix_2.dot(item_train_similarity_2)

sort_user_list_2 = []
applied_jobs_2 = []


for idx, user in enumerate(apply_pred_item_matrix_2.index):
    # 해당 사용자가 지원한 채용 공고
    applied_job = apply_train[apply_train['resume_seq'] == user]['recruitment_seq'].values
    applied_jobs_2.append([user, applied_job])


# train_사용자 간의 유사성 계산
user_train_similarity_3 = apply_train_similarity_3
user_predicted_scores_3 = user_train_similarity_3.dot(apply_train_item_matrix_3) / np.array([np.abs(user_train_similarity_3).sum(axis=1)]).T

item_train_similarity_3 = cosine_similarity(apply_train_item_matrix_3.T)
item_train_predicted_scores_3 = apply_train_item_matrix_3.dot(item_train_similarity_3)

sort_user_list_3 = []
applied_jobs_3 = []


for idx, user in enumerate(apply_pred_item_matrix_3.index):
    # 해당 사용자가 지원한 채용 공고
    applied_job = apply_train[apply_train['resume_seq'] == user]['recruitment_seq'].values
    applied_jobs_3.append([user, applied_job])


# train_사용자 간의 유사성 계산
user_train_similarity_4 = apply_train_similarity_4
user_predicted_scores_4 = user_train_similarity_4.dot(apply_train_item_matrix_4) / np.array([np.abs(user_train_similarity_4).sum(axis=1)]).T

item_train_similarity_4 = cosine_similarity(apply_train_item_matrix_4.T)
item_train_predicted_scores_4 = apply_train_item_matrix_4.dot(item_train_similarity_4)

sort_user_list_4 = []
applied_jobs_4 = []


for idx, user in enumerate(apply_pred_item_matrix_4.index):
    # 해당 사용자가 지원한 채용 공고
    applied_job = apply_train[apply_train['resume_seq'] == user]['recruitment_seq'].values
    applied_jobs_4.append([user, applied_job])

# train_사용자 간의 유사성 계산
user_train_similarity_5 = apply_train_similarity_5
user_predicted_scores_5 = user_train_similarity_5.dot(apply_train_item_matrix_5) / np.array([np.abs(user_train_similarity_5).sum(axis=1)]).T

item_train_similarity_5 = cosine_similarity(apply_train_item_matrix_5.T)
item_train_predicted_scores_5 = apply_train_item_matrix_5.dot(item_train_similarity_5)

sort_user_list_5 = []
applied_jobs_5 = []


for idx, user in enumerate(apply_pred_item_matrix_5.index):
    # 해당 사용자가 지원한 채용 공고
    applied_job = apply_train[apply_train['resume_seq'] == user]['recruitment_seq'].values
    applied_jobs_5.append([user, applied_job])













max_alpha = 0
max_score = 0
alpha = 0.01

while(alpha < 1):
    # 이미 지원한 채용 공고 제외하고 추천
    recommendations_2 = []


    for idx, user in enumerate(apply_train_item_matrix_2.index):
        re_applied_jobs = []
        # 해당 사용자의 추천 점수 (높은 점수부터 정렬)
        sorted_job_indices = (user_predicted_scores_2[idx] + item_train_predicted_scores_2.loc[user]*alpha).argsort()[::-1]
        for idxx, j in enumerate(sorted_job_indices):
            if(len(re_applied_jobs) >= 5):
                break
            else:
                job = apply_train_item_matrix_2.columns[j]
                if (job not in applied_jobs_2[idx][1] and apply_item_matrix_new.loc[user].loc[job] and (job not in re_applied_jobs) and distance_matrix_new.loc[user].loc[job]):
                    re_applied_jobs.append(job)
                    recommendations_2.append([user, job])
        

    # 이미 지원한 채용 공고 제외하고 추천
    recommendations_3 = []


    for idx, user in enumerate(apply_train_item_matrix_3.index):
        re_applied_jobs = []
        # 해당 사용자의 추천 점수 (높은 점수부터 정렬)
        sorted_job_indices = (user_predicted_scores_3[idx] + item_train_predicted_scores_3.loc[user]*alpha).argsort()[::-1]
        for idxx, j in enumerate(sorted_job_indices):
            if(len(re_applied_jobs) >= 5):
                break
            else:
                job = apply_train_item_matrix_3.columns[j]
                if (job not in applied_jobs_3[idx][1] and apply_item_matrix_new.loc[user].loc[job] and (job not in re_applied_jobs) and distance_matrix_new.loc[user].loc[job]):
                    re_applied_jobs.append(job)
                    recommendations_3.append([user, job])


    # 이미 지원한 채용 공고 제외하고 추천
    recommendations_4 = []


    for idx, user in enumerate(apply_train_item_matrix_4.index):
        re_applied_jobs = []
        # 해당 사용자의 추천 점수 (높은 점수부터 정렬)
        sorted_job_indices = (user_predicted_scores_4[idx] + item_train_predicted_scores_4.loc[user]*alpha).argsort()[::-1]
        for idxx, j in enumerate(sorted_job_indices):
            if(len(re_applied_jobs) >= 5):
                break
            else:
                job = apply_train_item_matrix_4.columns[j]
                if (job not in applied_jobs_4[idx][1] and apply_item_matrix_new.loc[user].loc[job] and (job not in re_applied_jobs) and distance_matrix_new.loc[user].loc[job]):
                    re_applied_jobs.append(job)
                    recommendations_4.append([user, job])

    # 이미 지원한 채용 공고 제외하고 추천
    recommendations_5 = []


    for idx, user in enumerate(apply_train_item_matrix_5.index):
        re_applied_jobs = []
        # 해당 사용자의 추천 점수 (높은 점수부터 정렬)
        sorted_job_indices = (user_predicted_scores_5[idx] + item_train_predicted_scores_5.loc[user]*alpha).argsort()[::-1]
        for idxx, j in enumerate(sorted_job_indices):
            if(len(re_applied_jobs) >= 5):
                break
            else:
                job = apply_train_item_matrix_5.columns[j]
                if (job not in applied_jobs_5[idx][1] and apply_item_matrix_new.loc[user].loc[job] and (job not in re_applied_jobs) and distance_matrix_new.loc[user].loc[job]):
                    re_applied_jobs.append(job)
                    recommendations_5.append([user, job])

    val_prediction_2 = pd.DataFrame(recommendations_2, columns=['resume_seq', 'recruitment_seq'])
    val_prediction_3 = pd.DataFrame(recommendations_3, columns=['resume_seq', 'recruitment_seq'])
    val_prediction_4 = pd.DataFrame(recommendations_4, columns=['resume_seq', 'recruitment_seq'])
    val_prediction_5 = pd.DataFrame(recommendations_5, columns=['resume_seq', 'recruitment_seq'])
    val_prediction = pd.concat([val_prediction_2, val_prediction_3], ignore_index=True)
    val_prediction = pd.concat([val_prediction, val_prediction_4], ignore_index=True)
    val_prediction = pd.concat([val_prediction, val_prediction_5], ignore_index=True)
    val_prediction.sort_values(by='resume_seq', inplace=True)
    score = recall5(apply_test,val_prediction)
    alpha += 0.01
    print(alpha)
    if (score > max_score):
        max_score = score
        max_alpha = alpha


print(max_score)
print(max_alpha)