编程语言
首页 > 编程语言> > 自然语言处理(NLP)编程实践-1.1 使用逻辑回归实现情感分类

自然语言处理(NLP)编程实践-1.1 使用逻辑回归实现情感分类

作者:互联网

内容汇总:https://blog.csdn.net/weixin_43093481/article/details/114989382?spm=1001.2014.3001.5501
代码:https://github.com/Ogmx/Natural-Language-Processing-Specialization
——————————————————————————————————————————

作业 1: 逻辑回归(Logistic Regression)

学习目标:
  学习逻辑回归,你将会学习使用逻辑回归对推特进行情感分析。给出一个推特,你要判断其是正向情感还是负向情感。

具体而言,将会学习:

我们将使用一系列推特数据。在最后你的模型应该能得到99%的准确率。

导入函数和数据

# run this cell to import nltk
import nltk
from os import getcwd

导入函数

从该地址下载本实验需要的数据documentation for the twitter_samples dataset.

nltk.download('twitter_samples')
nltk.download('stopwords')

从 utils.py 导入帮助函数:

# add folder, tmp2, from our local workspace containing pre-downloaded corpora files to nltk's data path
# this enables importing of these files without downloading it again when we refresh our workspace

filePath = f"{getcwd()}/../tmp2/"
nltk.data.path.append(filePath)
import numpy as np
import pandas as pd
from nltk.corpus import twitter_samples 

from utils import process_tweet, build_freqs

准备数据

# select the set of positive and negative tweets
all_positive_tweets = twitter_samples.strings('positive_tweets.json')
all_negative_tweets = twitter_samples.strings('negative_tweets.json')
# split the data into two pieces, one for training and one for testing (validation set) 
test_pos = all_positive_tweets[4000:]
train_pos = all_positive_tweets[:4000]
test_neg = all_negative_tweets[4000:]
train_neg = all_negative_tweets[:4000]

train_x = train_pos + train_neg 
test_x = test_pos + test_neg
# combine positive and negative labels
train_y = np.append(np.ones((len(train_pos), 1)), np.zeros((len(train_neg), 1)), axis=0)
test_y = np.append(np.ones((len(test_pos), 1)), np.zeros((len(test_neg), 1)), axis=0)
# Print the shape train and test sets
print("train_y.shape = " + str(train_y.shape))
print("test_y.shape = " + str(test_y.shape))

train_y.shape = (8000, 1)
test_y.shape = (2000, 1)

    for y,tweet in zip(ys, tweets):
        for word in process_tweet(tweet):
            pair = (word, y)
            if pair in freqs:
                freqs[pair] += 1
            else:
                freqs[pair] = 1
# create frequency dictionary
freqs = build_freqs(train_x, train_y)

# check the output
print("type(freqs) = " + str(type(freqs)))
print("len(freqs) = " + str(len(freqs.keys())))

type(freqs) = <class ‘dict’>
len(freqs) = 11346

处理推特

使用 process_tweet() 函数对推特中的每个单词进行向量化,去停用词和词根化

# test the function below
print('This is an example of a positive tweet: \n', train_x[0])
print('\nThis is an example of the processed version of the tweet: \n', process_tweet(train_x[0]))

This is an example of a positive tweet:
#FollowFriday @France_Inte @PKuchly57 @Milipol_Paris for being top engaged members in my community this week

标签:NLP,1.1,tweet,print,freqs,np,test,theta,自然语言
来源: https://blog.csdn.net/weixin_43093481/article/details/116356879