说明

使用 snownlp 库，分析微博内容，得到情绪数值，结果是一个小数，越接近1，说明越偏向积极；越接近0，说明越偏向消极。然后保存到mysql数据库中

要求

• 将 mysql 驱动放入到spark的 jars 目录

• 需要在服务器安装 snownlp 库，执行下面命令：

  pip install snownlp
  

代码

'''
分析网络情绪
'''

import findspark

findspark.init()

from pyspark.sql import  SparkSession 

def handleRow(row):
    content = row["content"]
    from snownlp import SnowNLP
    s = SnowNLP(content)
    score = s.sentiments
    abstract = s.summary(1)

    return abstract[:60],score

def exe():
    spark = SparkSession.builder.\
    appName("mood").\
    master("local[*]").\
    enableHiveSupport().\
    getOrCreate()

    # spark.sparkContext.addPyFile("hdfs:///yuqing/packages/site_packages.zip")

    df = spark.sql("select content from yuqing_dwd.t_weibo")
    # df.show()

    resRDD = df.rdd.map( handleRow )
    resRDD.foreach(print)

    ndf = spark.createDataFrame(resRDD,["content","score"])

    # ndf.show()
    # ndf.printSchema()
    # ndf.createOrReplaceTempView("t_weibo_temp")

    # resdf = spark.sql("select * from t_weibo_temp")

    prop = {"user": "root","password": "root","driver": "com.mysql.jdbc.Driver"}
    ndf.write.jdbc("jdbc:mysql://hadoop1:3306/yuqing_dws?useSSL=false&characterEncoding=utf8", "t_mood_statistics","append", prop)

    spark.stop()

if __name__ == "__main__":
    exe()

原文出处：/show_1IX3HXNaVnum.html