【研发随感:情感分析的沟沟坎坎(catches)】

A sentiment analysis catch:冤枉:1 [human]-A 冤枉 [human]-B 2. [human] 冤枉

冤枉:1 [human]-A 冤枉 [human]-B 2. [human] 冤枉

in (1), human-A is adversary and human-B is victim; in (2), human is a victim
in sentiment analysis, if the adversary is the topic, it is always a negative mention.  The catch is about the victim: if the topic is an event, or unknown, then sentences mentioning victims or actions or states involving victims are negative in tone; but if the topic is the victim, it is difficult to classify.  By human nature, when unfortunate events happen to victims, the natural sentiment is sympathy.
Sympathy is not negative at least; and it cannot be classified as positive either. It does not seem to be neutral. So it is tricky in the two-way or three-way sentiment classification.And in

And in text, we have tons and tons of expressions involving victims or victim-events, making sentiment classification difficult to conform to a traditional standard. For another example, John was ill.

For another example, “John was ill”.  Most people would classify John as being mentioned negatively here, which sounds right, after all, unhealthiness is not a desirable state.  But on second thought, we would find it inappropriate to have such a sentiment analysis: John did not want to be ill; it was very unfortunate that he fell ill, and it was not his fault to be ill so how can we conclude him simply to be a negative mention?

情感分析在不幸事件和受害人角色方面,该怎样确定一个合适的标准呢?大拇指或中指显得不够了。请各位语义计算的老师指教。1. 指责的对象(中指);2 赞扬的对象(大拇指);3 (理应)同情的对象算什么呢?当然我们可以考虑增加一个标签。这个标签是 Victim 【受害者】】还是 Sympathy【同情】?前者是描述事实;后者带有情感。但是不是对于每个提及受害者的文句,里面都必然有同情意味呢?如果不是,Sympathy 的标签就有点强加的味道,不符合语言理解和情感分析的本义。

对于好人好事和坏人坏事的情感分析 并不对称。做好事的人是 benefactor 【做好事者】自然是正面的,这件好事惠及的人是 beneficiary 【受益者】,也无疑是正面的
相对应的做坏事的人 自然是负面的。但坏事伤及的对象 如上所述 却不能总是不分青红皂白划归负面。虽然很多情感系统的确这么做 但会出现与事理不合 让人难以接受的场景或啼笑皆非的舆情分析结论。

倒霉的确带有负面语气 但倒霉的人通常是无辜的弱者 应该引起同情 而同情心是人类正面的情感。就这么点事儿 要说也没那么复杂 但在情感系统落地为应用的时候 常常闹出笑话。

索性再分享一些情感分析的坑。这些个沟沟坎坎,搞 thumbs-up-down 分类的人通常无感,也无能区分,却是细线条真实情感分析和应用绕不过去的。表达需求的说法,如何做情感分析呢?“小姑娘需要一台 iPhone7”,正面还是负面?对于品牌 iPhone7,这是一个很正面的事儿。品牌被人需求,这说明它受欢迎。但是对于“小姑娘”,这是负面的情形:她有这个需求表明她的愿望还没有得到满足。如果我们把主语从【human】改成品牌,需求的 statement 常常是:

“iPhone7 需要加长续航时间。”

这是很负面的表述:iPhone7 有很大缺点,续航时间不足。这是两种最常见的需求表述。对于细线条情感分析意义重大,涉及非常有用的市场和客户情报的挖掘。

顺着这个思路往前走一步,可以从直接表达情感的语句,进入表面上的陈述句,但背后却隐藏了正面负面的情报。譬如 购买,驾驶,使用等表述。得,打住,今儿个就分享到这儿。

 

【相关】

舆情挖掘

中文处理

Parsing

【置顶:立委NLP博文一览】

《朝华午拾》总目录

【基于关键词的舆情分类面临挑战】

[Abstract] Five challenges to keyword-based sentiment classification: (1)  domain portability; (2)  micro-blogs: sentence/twit classification is a lot tougher than document classification; (3) when big data become small: big data load when sliced and diced based on the users’ needs quickly becomes mall, and  a precision-challenged classifier is bound to have trouble; (4) association of sentiments with object:  e.g. comparative expressions like “Google is a lot better than Yahoo”; (5) too coarse-grained: no actionable insights, this is fatal.

做自动舆情挖掘(sentiment mining)已经好几年了,做之前思考这个课题又有好多年(当年我给这个方向的项目起了个名字,叫 Value Tagging,代码 VTag,大约2002年吧,做了一些可行性研究,把研发的 proposal 提交给老板,当时因为管理层的意见不一和工程及产品经理的合作不佳,使得我的研发组对这个关键项目没能上马,保守地说,由此而来的技术损失伤害了公司的起飞),该是做一个简单的科普式小结的时候了。

首先对大数据的舆情挖掘是建立在对具体语言单位的舆情抽取(sentiment extraction)的基础之上。只有当语言海洋中千千万万的舆情表达被抽取存贮到某个数据库以后,我们才有条件针对具体的舆情问题(如某特定品牌的网络形象或某话题的舆情走势),搜索有代表性的舆情资料,并将搜索结果整合提炼,然后以某种方式(譬如《品牌舆情图》或《话题晴雨表》)表达给情报使用者。

舆情抽取的主流是利用机器学习基于关键词的分类(sentiment classification),通常的做法非常粗线条,就是把要处理的语言单位(通常是文章 document 或帖子 post)分类为正面(positive)和负面(negative),所谓 thumbs up and down classification。后来加入了中性(neutral),也有在中性之外加入一类 mixed (正反兼有)。这种做法非常流行快捷,在某个特定领域(譬如影评论坛 movie reviews),分类质量可以很高。18年前,我们让实习生做过这样的暑假项目,用的是简单的贝叶斯算法,在影评数据上精度也达到90%以上。这是因为在一个狭窄的领域里面,评论用语相当固定有限,正面负面的评价用词及其密度不同,界限清晰,识别自然不难。而且现在很多领域都不愁 labeled data,越来越多的用户评价系统在网络上运转,如 Amazon,Yelp,积累了大量的已经分类好的数据,给机器分类的广泛应用提供了有理的大数据条件。

但是,上述分类遇到了以下挑战。

首先,领域移植性不好,影评数据训练出来的分类器换到电子器件的客户评价分类上就不管用。

要对多个领域训练出多个分类器,很耗时,效果也不能得到保证。于是有人开始研究独立于领域(domain-independent)的舆情分类,其假设前提是舆情表达各个领域之间既有领域独有的表述方式,也有共通的表达(比如,好/good 在任何领域都是正面的形容词,坏/bad 在任何领域都是负面的),而且二者有相当的信息冗余度。这个假设在语言单位较大(譬如长的帖子或文本)时,是成立的。因此,这个挑战不算是致命的。但是很多应用领域,语言单位不大,譬如社会媒体中风行的微博/tweets,就很短,这一点构成下述第二个挑战。

第二个挑战来自移动时代压倒多数的短消息。语言单位的缩小使得分类所需要的词汇证据减少,分类难为无米之炊,精度自然大受影响。

从文件到帖子到段落再到短消息,语言单位每一步变小,舆情分类日益艰难。这就是为什么多数分类支持的舆情系统在微博主导的社会媒体应用时文本抽取质量低下的根本原因(一般精度不过50%-60%)。当然,文本抽取精度不好并不表明不可用,它可以用大数据来弥补(由于大数据信息天生的大冗余度,利用sampling、整合等方法,一个大数据源的整体精度可以远远高于具体文本抽取的精度),使得最终挖掘出来的舆情概貌还是靠谱的。然而,大数据即便在大数据时代也不是总是存在的,因为一个真实世界的应用系统需要提供各种数据切割(slicing and dicing)的功能,这就使得很多应用场景大数据变成了小数据,这是下面要谈的第三个问题。

第三是大数据切割的挑战。

本来我们利用机器来应对大数据时代的信息挑战,起因就是信息时代的数据量之大。如果数据量小,蛮可以利用传统方式雇佣分析员,用人的分析来提供所要的情报,很多年以来的客户调查就是如此。可是现在大数据了,别说社会媒体整体的爆炸性增长,就是一个大品牌的粉丝网页(fan pages)或一个企业的官方网页,每时每刻所产生的数据也相当惊人,总之无法依靠人工去捕捉、监测情报的变化,以便随时调整与客户的互动策略。这是机器挖掘(无论分类还是更细致的舆情分析)不可不行的时代召唤和现实基础。但是,观察具体应用和情报需求的现场就会发现,用户不会满足于一个静态的、概览似的情报结果,他们所需要的是这样一个工具,它可以随时对原始数据和抽取情报进行各种各样的动态切割(slice/dice 原是烹饪术语,用在情报现场,就是,
“to break a body of information down into smaller parts or to examine it from different viewpoints so that you can understand it better”,
摘自 http://whatis.techtarget.com/definition/slice-and-dice)。舆情切割有种种不同依据的需求,譬如根据舆情的类别,根据男女的性别,根据数据源,根据时间或地理位置,根据数据的点击率等。有的时候还有多次切割的需求,譬如要看看美国加州(地理)的妇女(性别)对于某个品牌在去年夏季(时间)的舆论反映。最典型的切割应用是以时间为维度的《动态晴雨表》,可以反映一个研究对象的舆情走势(trends)。譬如把一年的总数据,根据每月、每周、每日,甚至每小时予以切割,然后观察其分布走势,这对于监测和追踪新话题的舆情消长,对于新产品的发布,新广告的效用评估(譬如美式足球赛上的巨额品牌广告的客户效应)等,都有着至关重要的情报作用。总之,大数据很可能在具体应用时要被切割成小数据,一个分类精度不高(precision-challenged)的系统就会捉襟见肘,被大数据遮盖的缺陷凸显,被过滤净化的结果在小数据时会变得不再可信。

第四个挑战是找舆情对象的问题。

在几乎所有的舆情分析应用中,舆情与舆情的对象必须联系起来,而这一基本要求常常成为舆情分类系统的软肋。当然,在特定数据源和场景中,可能不存在这个问题,比如对 Amazon/Yelp 这类客户评价数据 (review data) 的舆情分析,可以预设舆情的对象是已知的(往往在标题上,或者其他 meta data 的固定位子),每一个review都是针对这个对象(虽然不尽然,reviews 中也可能提到其他的品牌或产品,但是总体上是没问题的,这是由 review data 的特性决定的)。然而在很多社会媒体的自发舆情表述中(譬如微博/脸书/论坛等),在舆情分类之后就有一个找对象的问题。这个问题在比较类语言表达中(比如,”谷歌比雅虎强老鼻子啦” 这样语句,正面评价“强”到底是指雅虎还是谷歌,这看似简单的问题,就难倒了一大帮机器学家,道理很简单,机器分类系统依靠的是 keywords,一般没有语言结构的支持,更谈不上理解)。与青春躁动期的小屁孩也差不多,满腔情绪却找不到合适的表达或发泄对象,这几乎成了所有褒贬分类系统的克星。在随兴自发的社会媒体中,这类语言现象并不鲜见,一边夸张三一边骂李四更是网络粉丝们常见的表达(譬如方韩粉丝的网络大战)。

第五个挑战是颗粒度的问题。

这是分类系统的致命伤,它们只知道数翘大拇指还是伸出中指的数量,这对舆情的总体概览有点意义,但是这远远不是最有价值的舆情情报,关于这一点,我在以前的科普随笔中论述过:

褒贬分析只提供舆情的一个概览,它本身并不是 actionable insights.  知道很多人喜欢或者不喜欢一个品牌,so what?企业还是不知道怎么办,最多是在广告宣传投资量的决策上有些参考价值,对于改进品牌产品,适应用户需求,褒贬舆情太过抽象,不能提供有价值的情报。这就要求舆情分析冲破两分、三分、五分的分类法,去发掘这些情绪的背后的动因(reasons/motivation),回答为什么网民喜欢(不喜欢)一个品牌的问题。譬如挖掘发现,原来喜欢麦当劳的主要原因是它发放优惠券,而不喜欢它的原因主要是嫌它热量太大,不利减肥。这样的舆情才是企业在了解自己品牌基本形象以后,最渴望得到的 actionable 情报,因为他们可以据此调整产品方向(如增加绿色品种和花样,水果、色拉等),改变广告策略(如强调其绿色的部分)。摘自【立委科普:舆情挖掘的背后

上面列举的机器舆情分类系统的挑战,并不是要否定机器学习在舆情领域的价值,而是要阐明以下的观点:粗线条的机器分类只是舆情自动分析的开始,万里长征的第一步。一个真正有价值的舆情挖掘系统还需要更多更细致的舆情自动抽取和挖掘的技术来支持。鉴于学界和业界 90% 以上自称做 sentiment 的系统,都是以机器分类作为基础支撑的,明白这一点尤其重要。那么什么是舆情抽取和挖掘系统所需要的完整的技术基础呢?且听下回分解吧(如果兴起的话)。

原载:【立委科普:基于关键词的舆情分类系统面临挑战

【相关】

《立委科普:关键词外传》

《立委科普:关键词革命》

《铿锵众人行,parsing 可以颠覆关键词吗?》 

【置顶:立委NLP博文一览】

《朝华午拾》总目录

【立委随笔:听喜马拉雅老罗侃人工智能】

关于AI的几点小感想:

1. AI这一波狂热,甚至连文科生罗胖都开始信服了。那天堵车听喜马拉雅,收听罗胖的跨年演讲,谈产品 高科技等,觉得这小子铁嘴钢牙,不全是嘴皮子功夫,有真货。譬如,论产品和服务,从空间移向时间,就很有高度。唯独他谈AI的时候,让我莫名。有几分道理,更多是谬误,参杂一起,言之凿凿。这才相信了AI的营销已经多么有成效,把一个完全的外行人也整成导师了。

罗胖说,很快(言之凿凿说的是五到十年)一多半人就要失业了,为人工智能取代。这不是新鲜的观点,开复老师等大咖也一直在说。老罗说人类历史都是人奴役、压榨、剥削人的历史,终于历史走到了尽头:大多数人连被奴役被剥削的价值也没有了,因为人工智能不怕被奴役和剥削,不抱怨、不造反、不暴动。干嘛要剥削人呢?

怎么办?也许就是给这些完全没有价值的多余人一个VR的眼镜,让他们一辈子在游戏中度过。如果不想落入这个群体,很少的几个可能是做设计家和创新家。因为对于现存的一切,人工智能无所不能。只有对不存在的东西,对于新设计和新创新,人类还有一点点赢面。

这些把人类与人工智能割裂成两个平等、独立或对立的竞争对手的说法,已经流传甚广,似是而非,但也很难证伪。先放在一边。

奇葩的是,他居然搞懂了深度神经。简单说就是深度神经就是一个怪兽,胃口特别大,只要给它喂大数据,他就无所不能。罗胖说,深度神经可以看历史上所有的医书,然后分析你所有的生命数据,然后给你建议吃什么药。你必须听他的,因为你穷尽一辈子也不可能理解人工智能的智能。但是他肯定比你和你在人类所能找到的任何专家高明。

总之人工智能行的是上帝的逻辑,我们人的逻辑无法理解 也无须理解。顺之者昌逆之者亡。这个逻辑的算法基础就是深度神经。

罗胖这些说法其实早已在现下媒体,以及早先的科幻中,为无数记者作家描述过。他不过是利用他所特长的语言艺术表达出来。

2. 看老友在朋友圈提ai+酒,就让我想起历史上魏晋的药+酒,都是性感、时髦、流行而且催情的。很浪漫,也很颓靡。

3. 老友接着谈他看好机器人情感,觉得前景无限。老友说的机器人的 AI 情感,这里的情感是说的什么呢?

(1) AI 对于人类表述的情感的捕捉:譬如舆情挖掘
(2) AI 机器人(譬如微软小冰)自己所表现出来的情感表述

(1)是已经和正在实现的事儿,毋庸置疑,但与常人所说的机器人情感大概不是一回事儿,虽然不少人有意无意混淆二者。

(2)是典型的“逢场作戏”:微软小冰细语款款地说爱我很多次了,我要不是做 NLU 的,可能早就被她迷惑了。

说的是(2)这种情感大有前途么?

可以想到的前途是: 虚拟恋人(安慰失恋的人);老人陪伴(宽慰孤独的老人)。不怀疑这种东西最终可以以假乱真。将来市场化时候唯一要着力做的是,消除心理障碍,要给客户洗脑,这个机器人,不是机器,而是人。

(绝不能泄露天机:这与人类情感,没有一毛钱的关系。她爱死你了,你也爱死她了,你们结婚,也绝不会有爱的结晶。)

 

【相关】

【泥沙龙笔记:强人工智能的伟哥测试】

强弱人工智能之辩

【置顶:立委NLP博文一览】

《朝华午拾》总目录

大数据淹没下的冰美人: 喜欢范冰冰的理由

最近两年“忽悠”大数据,常有网友问我:OK,我看到了,热点话题你是可以从大数据中挖掘出一些东东来,做成奇奇怪怪的词云来,蛮唬人的。可这些信息有什么价值?差不多都是我耳闻过的 rumors 。

Good question,说破大天去,挖掘出来的情报如果没有信息量(e.g.人所共知),或者只是进一步传谣,所谓大数据及其挖掘不过是浮云。大数据专家?饶了我吧。不如回家种红薯,或可产生些许价值。But wait,这个问题细究起来,绝对不是简单否定,就可解决的。至少可以从下面几个方面来看:

1. 对象和目的:回顾一下,我们的对象是海量增长的社会媒体大数据,我们的目的是透过这些 人一辈子也无法尽览的大数据,挖掘民意和舆情。这些舆情都是有统计意义的现实存在,而不是存在于个别人头脑里的天才思想,只要挖掘出来的情报具有代表性,这些情报是不是新颖,不影响我们为挖掘所设立的舆情目标。

2. 信息量问题:乍看,挖掘出来的情报都似有所闻,但其实挖掘还是增加了信息的价值。似有所闻并不奇怪,本来就是流传广泛的公共信息(open sources)。所增加的信息在于,作为个体,每个人的所闻都是零碎的,不可能具有大数据挖掘的全面。更重要的是,即便你是一个 人生即网网即人生 的无可救药的网虫,多数挖掘对你都不是新闻,你对这些信息也不可能具有量化的排序能力。就说你是冰美人死心塌地的粉丝,你也不可能给我们列出大众心目中的冰冰小姐随时间曲线的形象消长,也无法量化大家喜欢她或者不喜欢她的种种理由的分布。你也许可以蒙对几项,你绝不可能给出本文给出的结果。这些排序和分布就是附加的新信息。从另一个角度看这个问题,如果 90% 的挖掘结果不具有新闻价值,而只具有印证的意义(但也间接说明大数据挖掘是靠谱的),那也不坏。作为信息消费者,我们会自然把注意力集中到意想不到的情报上去,wow,原来还有这个啊。这些 surprise info 是你不借助大数据深度挖掘工具永远无法看到的信息,它被淹没在数据的海洋中。

3.  挖掘的情报价值:这个话题太大,不同的消费者,政府、企业或个人,可以看到和用到不同的价值,你觉得没有价值的结果,在有心者眼里可能具有决定的意义。一个企业愿意花百万美元去挖掘社煤大数据中的客户情报,他们必然是看到了一般人感觉不到的挖掘价值。手工民意调查的时代即将过去,因为大数据的挖掘会更加多快好省。如果你是范冰冰团队的策划人,下述挖掘对整个团队的市场化努力以及公关策略的调整,应该具有相当的参考价值。

回到冰美人的大数据挖掘来。前面的博文已经展示了舆情概貌,总之是冰小姐现在是如日中天,压倒性地被大众喜爱,甚至迷恋。那么喜欢她的理由呢?先看褒贬云图:

理由千千万,代表性的理由归纳列表如下:

Why like her Mentions
漂亮美艳 …… 6420
霸气 2493
火, 红 1208
范冰冰的胸 686
公益项目 684
时尚女王 446
武媚娘传奇 326
底妆白皙 298
年轻 232
时尚 229
敬业 133
范爷的皮肤 164
 

成功

130
愈挫愈勇 121
低调 121
大方 118
聪明 110
精湛 92
精致 87
合拍 84
成熟 75

我个人的理由也在其列呢:看来不少人与我一样,因为《武媚娘传奇》而喜欢她。我们择其几点,drill down  看看真实数据怎样表达的:

(1) 前两项压倒性的美丽和霸气,是喜欢的最大理由。

自动化所吴同…

范爷霸气

barbiewong

范冰冰靚到

sunnynoe

范冰冰最好看

奢享时光。

范爷好看

@janiceho96  范冰冰好美好美
lingwu01

还是范爷霸气威武

THE_MIST

范冰冰幾靚

奢享时光。F

范冰冰好漂亮

caotiantian9

还是喜欢范爷, 霸气

莫欺少年穷丶楠

范冰冰, 妖艳

(2) 公益项目:原来这条公益项目的新闻大体是同一个来源,被各大网站反复转载,至少说明冰小姐团队做得很成功。

这名字真的无语

12月11日, 吴亦凡参加《我们小区艺体能》“圣诞慈善篮球赛”,慈善赛的收入将在比赛结束后以冠军队的名义捐赠给福利机构, 用于帮助困难居民。 2014年6月, 吴亦凡加入范冰冰公益项目“爱里的心”捐赠队伍, 成为继李嘉诚后第二个捐助的人。

(3) 范冰冰的胸:女星被人评头论足,不足为怪。但范小姐的胸之所以跃升为好评前五,其实有两个原因。

一是冰美人的自身条件的优越(魔鬼身材)以及形象塑造的重点所致,性感女神,是很多人的由衷赞佩。

二是在传统社会与现代化过程的急剧变革时期,她的美胸展示不仅仅是粉丝喜爱的理由,也成为争议的焦点(事实上,后面还会提到,冰美人的胸不仅是喜爱的理由,也是不喜爱的理由,尺度太大,被某些人认为是伤风败俗)。有争议就会有卫护、有赞美:

节奏大师大手子

范冰冰的胸真的好美

野猪

谈谈法制晚报中青网新华社死磕毕福剑与不道德视频的播放 范冰冰的胸没有危害, 有危害的是你龌蹉的思想意识; 毕福剑的嘴没有危害, 有危害的是僵化教条、极端政治的思维方式。

其实,以范冰冰饰演的武媚娘为代表的唐代仕女式的袒胸露背,曾经被广电局严审,视为洪水猛兽,以致片子被勒令重新剪接,曾经引起社会媒体的轩然大波, 编剧李亚玲的痛心评论传遍全网:

烟入眼而落泪

此事件造成了颇为强烈的网络热议。 著名编剧李亚玲做出评论:“《武媚娘传奇》中的胸确实剪掉了! 真没想到21世纪的人会这么保守, 甚至不如千年前的唐人开放。

自由飞翔的鸟人

在停播的四天中, 片方解决了重大技术问题, 紧锣密鼓的将原有画面全部裁成了大头贴, 斥资千万的服装造型只剩下脸上的粉底…… 著名编剧李亚玲就此事做出评论:“《武媚娘传奇》中的胸确实剪掉了! 真没想到21世纪的人会这么保守, 甚至不如千年前的唐人开放。

Unknown

由于播出方湖南卫视在一官方微博中, 对于此次停播给出的“因技术原因”这一理由过于简单, 使网友在一头雾水的同时, 也对相关部门的管理水平提出质疑。 对此, 著名编剧李亚玲做出评论:“《武媚娘传奇》中的胸确实剪掉了! 真没想到21世纪的人会这么保守, 甚至不如千年前的唐人开放。

MJ浮生若梦

昨晚复播的三集中,“武媚娘”和其他众妃子宫女们的胸部镜头全部被剪, 画面仅呈现远景和肩膀以上的近景, 网友纷纷吐槽:“现在一个屏幕全是脸”、“满屏都是大头照”。 著名编剧李亚玲也表达了自己的看法:“《武媚娘传奇》中的胸确实剪掉了! 真没想到21世纪的人会这么保守, 甚至不如千年前的唐人开放。”

光明社区

我一个女性都看不下去了, 还我们原版。” 对此, 著名编剧李亚玲做出评论:“《武媚娘传奇》中的胸确实剪掉了! 真没想到21世纪的人会这么保守, 甚至不如千年前的唐人开放。

(3) 底妆白皙:地球人都明白,冰小姐代表了化妆艺术的巅峰。她的底妆衬托了她形象的明艳绝色。有老友说,我只爱化妆的冰冰。

Unknown

蕾哈娜 (Rihanna) 的阳光感卷发, 与健康的蜜糖棕色皮肤融为一体, 浓密卷翘的假睫毛塑造出迷人电眼。 鱼草网化妆猜你喜欢的文章: 范爷搪瓷底妆白皙胜雪 女星美妆花尽心思 泫雅红唇领衔4minute 秀冷傲别样风情 赵丽颖周冬雨戏内争宠 戏外淡妆拼清纯

503247463

【范冰冰搪瓷底妆白皙胜雪彩妆花尽心思】近日, 电影《一夜惊喜》在北京召开发布会。 主演范冰冰、蒋劲夫、李治廷、导演金依萌出席, 范冰冰当天搪瓷底妆展现如雪的肌肤, 纤长浓密的假睫毛是范爷明亮双眸的利器, 加上丰盈感十足的大波浪卷发, 俨然真实版芭比娃娃

(4)皮肤:光靠化妆是做不了影后的,赞美冰美人除了酥胸外,就是她凝脂般的雪肤了(她的脸相对比之下,则争议较多,有机会可以挖掘展示一下,总之是粉丝视若桃花,爱得要死,但也有横挑鼻子竖挑眼的嫌她面相不正):

博士一年级

范爷的皮肤真好

zahara蕾儿

这时皮肤吸收最好! 范爷的皮肤为什么这么好! 就是每日坚持一片面膜!

Unknown

难怪范爷的皮肤这么棒呢

幻雨不幻风

范爷的皮肤貌似一直都挺好的

ambrosia52

不过范爷的皮肤真心好啊·羡慕! 我都是懒, 嫌麻烦不敷面膜。

糖口香糖

范爷的皮肤真是没话说, 和林志玲有的一拼

关闭

范爷的皮肤真的好好哟, 哇,坚持15年每天用面膜呀, 向她看齐。

十世轮回阴阳间

范爷的皮肤一直都很好, 人家是纯天然的

Unknown

谢霆锋与范冰冰亲密合影 范爷的皮肤白皙细腻 在《锋味》中, 范冰冰真人出镜, 与谢霆锋暧昧互动。

康婕

我想想啊… 高圆圆, 范爷的皮肤不错, 范范也可以, 春哥, AG 虽然是整的· …. 不过他们的黑眼圈

总之,喜欢女星的理由,看来还是以貌取人为主,人之常情。其他品质(敬业、低调、聪明、成熟云云)也有提及,当无法与美胸美肤相比。那么不喜欢的理由呢?大约多与娱乐界的绯闻和流言蜚语有关,这方面挖掘下去近乎传谣了,不太好写,假语村言,点到为止,留待下篇博文拿捏分寸吧。

 

【相关】

【社煤挖掘:大数据女星排行榜】

【社煤挖掘:大数据男神排行榜】

【大数据淹没下的冰美人】的系列博文链接:

大数据淹没下的冰美人(之一)

大数据淹没下的冰美人(之二) 

大数据淹没下的冰美人(之四): 流言蜚语篇(慎入)

中文处理

【置顶:立委NLP博文一览】

《朝华午拾》总目录

 

【社煤挖掘:大数据女星排行榜】

Mirror mirror on the wall, who now is the fairest one of all?

自从毛委员开启新中国,半个多世纪以来影视娱乐界历经磨难起伏,如今迎来了百花齐放星光灿烂的黄金时代。被无数粉丝奉为当今中国影视女皇的范冰冰,在众多女星中究竟排名如何?我们让大数据说话,让社会媒体告诉你。

先比较一下据说当今PK范冰冰呼声最高的张馨予。

PK

张馨予天生丽质,比范冰冰更接地气,可惜她饰演的几位角色都是刁毒的坏女人,据说连带她也受牵累。下表是过去27个月社煤大数据挖掘的概览比较:

单看这概览便知,在大家心目中,张馨予还嫩了点,无法挑战范冰冰的影后地位。从人气上看,张小姐还不及范小姐的四分之一;褒贬指标 45% 虽然算是不错的媒体形象了,但也远远不及范冰冰的 62%。粉丝的狂热度也略逊一筹(70 pk 76)。下图是时间曲线。首先是热议度比较,可见出除了 2013 年十月前后,张馨予的热议度陡升外(为什么陡升,等空来细细挖掘看),过去两年多的其余时间均不及冰小姐。

反映媒体形象的褒贬指标(net-sentiment)的曲线表明,范冰冰差不多步步领先张馨予,而且评价度一直比较平稳。张馨予则起伏较大,只在四个时间点短暂超越范冰冰。

下面的情绪指标(passion intensity)也类似,不过两位更加接近(有三个时间点馨予短暂超越冰冰),说明各自都有一批狂热死忠的粉丝,争吵起来不惧激烈。

上述挖掘比较的结果,用我们创造的同时含有人气指标(品牌大小或泡泡的大小来表示),褒贬指标(net sentiment 横轴)和激情指标(passion intensity 纵轴)的多维品牌比较一览图 (Brand Passion Index,BPI) 独特图示如下:

张馨予不在话下,那么其他红星呢?请看:

哈,论人气,范冰冰稳居第一,比第二位的章子怡还要高出一倍。可论受欢迎的程度(褒贬度)这项硬指标,美丽大方接地气的高圆圆才是普罗百姓(也是笔者)心目中真正的无冕之后,高达 74 %,比范小姐高出 12 个点。

网民心目中的真正星后 高圆圆

(可惜的是,我所喜欢的徐静蕾风光不再,逐渐淡出女星圈,步入中年的她已经更多转型为导演了。)

高圆圆 pk 范冰冰是完胜。过去两年多除了几个短暂的时间点,圆圆是一路领先:

范冰冰只能说是话题女王或眼球女王,高圆圆才是网民心目中的真正皇后:

高圆圆为什么具有如此高的口碑成为网民的偶像呢?下面是圆圆的两张云图,可以提供一点insights:

最后把调查的六位当红女星的过去两年多的比较曲线一并提供如下,供大数据爱好者们研究参照:

 

【后记】

《圆冰之战花絮:泥沙龙笔记,核武器封喉篇》

我:
这个需要较真,为大数据正名。自摸、莫衷一是、粉丝之战都是有的,根子都是缺乏大数据的nlp,有了nlp,我们就坐在大数据之上,而不是被数据淹死,看不到真像,无论真像多么残酷。你冰的脑残粉多不多,当然不少,不独你一个,有图为证

大数据淹没下的冰美人(之三): 喜欢的理由

但是,世界上怕就怕大数据nlp和但是二词,与脑残粉一样巨多的是流言蜚语:大数据淹没下的冰美人(之四): 流言蜚语篇(慎入)。而事实上,流言蜚语对你冰的buzz的贡献巨大,你懂的,国人奏好这口。这些都是舆情,但只是舆情的一个片面,大象的耳朵或大腿而已。真正决定舆情的是大数据的褒贬比例(所谓 net sentiment),为此立委洋洋洒洒写了五大篇大数据调查的总结系列,对你冰算是仁至义尽了吧。

请看结论篇: 大数据淹没下的冰美人(之五): 星光灿烂谁为最?

比没文化更可怕的是没核武器。圆冰之战,该结束了吧

 

【大数据淹没下的冰美人】的系列博文链接:

大数据淹没下的冰美人(之一)
大数据淹没下的冰美人(之二)

大数据淹没下的冰美人(之三): 喜欢的理由

大数据淹没下的冰美人(之四): 流言蜚语篇

【置顶:立委博客NLP博文一览】

 

【社煤挖掘:大数据男神排行榜】

女弟子有花痴者,为心目中的男神排名争论不休。究竟大众舆论是怎样的呢?为女生谋福利,咱们大数据挖掘一把,也好增加她们的谈资。

以前为女星排名狠狠挖掘过一次(见篇末【相关博文】),结果甚合吾意,高圆圆是众望所归当之无愧的女皇,而范冰冰不过是炒作女神。

到了男星,就麻烦了,根本就记不住谁是谁,在我眼里,他们跟你我一样,都是一坨泥巴。于是专从女生中收集候选人,做了这次社会媒体的形象排名,叫做男神榜,供女生争论和消遣用。

无数女为你痴狂:男神榜首宁泽涛

初步排名结果如下。宁泽涛高居榜首。进一步的细节挖掘待后发布。

看看这张男星图,体育圈的宁泽涛出类拔萃,卓尔不群的样子。

男神对比图(Brand Passion Index)里三个维度,褒贬度(毁誉的比例),狂热度(就是脑残度),炒作度(背后往往有推手)。

炒作由圈的大小表示,决定于媒体中提及和眼球数的多寡。另外两个维度是 XY 坐标,X 是褒贬,Y 是热度。

这够一目了然了吧,这图是有专利的。

下表是中文社会媒体大数据过去两年半对这八位男星自动舆情挖掘排列出来的男神排行榜。

男神第二名:靳东

男神第三名:钟汉良

男神第四名:王凯

男神第五名:胡歌

男神第六名:林丹

男神第八名:霍建华

 

Download

impressions 是眼球数,与炒作度mentions基本一致。最重要的是 net-sentiment 褒贬指数,喜欢与不喜欢的比例,好话多于坏话就是零度以上,坏话多于好话就是零下。男神嘛,都在零上。50% 算是相当地受欢迎了。

竖线不是零度,竖线是被比较的几位的中值。

passion 是狂热度,情绪的烈度,数字越大,脑残越多。


Download

这是炒作度排名。黄晓明为最。估计是那场婚礼闹的,虽然他的受欢迎程度其实很一般,褒贬指数才 38。但以我男生的眼光,他确实有型,五官跟雕塑刀刻似的,而且也有胸肌。可你为啥着急娶了女神 Baby 呢。世纪婚礼究竟是给你涨了粉丝,还是碎了粉丝的心,下次大数据挖掘告诉你。

男神第七名黄晓明

 

【相关博文】

【泥沙龙笔记:弃暗投明,明在何方】

我:
just had a small talk with Tanya on US election, she was super angry and there was a big demonstration against Trump in her school too

行:
@wei
在我们这个群里,我们都见证了立委清晰的预测了川普对希拉里的领先优势。与传统媒体相比,这次社交网络所反映的民意更准确。也许更为重要的是分析整个选举过程中与时间相关的一些关键变量。
不过有一个问题和缺点,这个分析没有反映美国的选举人制度,事实上希拉里克林顿所取得的选票高于川普。如果能有回缩的地域分析,特别是,摇摆州的地域分析,比如说佛罗里达等的回溯

我:
是的。这次其实是千载难逢的机会,因为太多人关注,太多人 bet,应该认真当成一个项目去做,精心设计。

利:
不光是美国人关注,我们在国内也非常关注

行:
证明了新工具的力量。这也是这次川普当选的最正面的事件。

我:
我这种票友性质地玩,只是显示了大数据里面的确有名堂
但不是震撼性的。

利:
我跟美国的朋友们说:不管谁赢得了总统,都是大数据分析赢了

行:
等我有钱了,我来投你。

毛:
对,我也想过这个事,难点恐怕在于网上的信息恐怕难以分清出自何地?

我:
票友性质不是说的技术:技术是deep,靠谱和专业的,我从来都不小看自己;票友是说我对 domain (政治、大选)是票友 ,到现在对选举人制度还是模模糊糊,它到底怎么工作的

行:
lP地址不是相对能反映地域吗?

我:
推特是最大最动态的数据源,我们有推特的地理,应该大体足够从地理上区分了
我们也有种族,还有年龄和性别等信息。

行:
强烈建议回溯一下摇摆州。挖矿!非常值得进一步挖掘。

我:
没那个精力和兴趣了,公司缩水,也没有几个兵了,日常的琐务也要做
大数据不好赚钱。烧钱倒是哗哗的。

行:
需要设计出一个能赚钱的商业模式。技术是根本,但不是全部。

毛:
如果能把地理年龄这些结合进去,那你的系统大有前景。

Nick:
同意,伟哥可以写本书:
how is a presidential election won or stolen?把选举人票考虑进去

我:
有兵的时候,鸡毛蒜皮我不管,我爱怎么玩怎么玩, 到头来连兵都保不住,还玩个球啊。一个教训,不要把技术开发得过头。小公司的构建内,任何一个部门都不宜超前太多,超前了,就意味着末路的来临。

Nick:
@wei 早就叫你弃暗投明

我:
弃暗投明倒有个明啊 一厢情愿哪里行。

技术并不是越深入越先进越好,by nature 作为科学家,我们总是想越深越好
结果是产品来不及消化,技术总吃不饱,最后最先裁剪的就是技术 呵呵 反正也消化不了全部,你再优秀也没价值 其实是有前车之鉴的:《朝华午拾 – 水牛风云》
十几年再来一次,仿佛时光倒转。

一个机构作为一个整体,必须保证大体相称的发展水平,才可相谐。一个部门太出色,overperforming,其他部门无法消化,也就成了目标。譬如研发,要质量我给你最好的质量,超过“世界第一”,要广度我给你整出20个世界主要语言的深度分析 (deep parsing),cover 语言数据的 90+%,要领域化可以在两周内 deliver 一个 domain 所需的情报单位(一种关系,或一个事件),只要定义明确,产品的情报挖掘的瓶颈永远不在这个自然语言研发部门。结果呢,部门需要为部门的太好表现付出代价。这个世界就是这样诡异。

话说回来,一套技术在同一个公司挥洒了10年还没走人,对我这样害怕变动的人,公司也已经相当不易了。对得起我,我也对得起它了。当年没有我的技术,公司早死翘翘了。如今有了技术不能起飞,也怪不得我,公司从上到下,在这一点是共识:论技术和由此而来的数据质量,我们绝对领先对手。市场做不起来,打败不了对手,是技术以外的因由,我无能为力。另一方面也可以说,市场不成熟,技术变钱不是那么简单 market economy 决定的。

白:
NLP部门因为表现太好而不受欢迎,听起来是天方夜谭,如果不是伟哥亲历,谁信呀……

我:
反正我信。
我们吃不饱有日子了。一直都是我们催产品经理,而不是相反:求求你,给我们一个任务吧。产品经理说:就根据客户反馈小修小补吧。我们的数据质量已经行业领先很久了,一直是领先。

白:
用嘴投票还是用脚投票,这是一个问题

我:
新的 specs,或者出不来,或者出来了,我们 deliver 了,产品却实施不了。

严:
@wei 还是觉得公司产品方向太窄了,这么好的技术被局限在这么窄的应用范围。董事会老是要Focus。

邓:
听起来CEO应该负很大的责任啊

我:
据说是市场太小了,或饱和了。产品在一个 niche market,这个社会媒体大数据挖掘的market一度被疯狂追捧和夸大。几年下来发现,价值得到验证,市场也确实存在,但是就是不够大。拓展其他 market 需要有眼光的产品老总。对于“高新技术”,有眼光的产品老总比熊猫还稀少。高新技术比较适合做大公司的花瓶,其价值在于花瓶的股市效应。或者,适合一个巨大平台,帮助连接顾客和厂家: 这个可以产生真正的价值,譬如 Facebook。高新技术对于创业其实很难,第一缺乏资源(不能吃一辈子VC),第二缺乏平台(连大数据都要花大价钱购买,更甭提顾客与厂家的network了),第三缺乏熊猫。好不容易都凑齐了,最佳的出路也就是有幸被巨头看重收购了事。这个概率不到十分之一吧。也就是说,你哪怕有再牛的技术,你这辈子活过了三个人的寿命,有机会创业10次,你可能创业成功,如果成功是以被收购作为标准的话。如果成功是以上市成为独角兽作为标准,那么你需要的机会数是下一个量级,五年一个轮回,你大概需要活500岁才可撞上狗屎运。

老总的眼光各有自己的局限,譬如,原来一直做 b2b saas 的 就一直沿着以前的经验和熟悉的领域想技术的用场。超出经验领域之外 是很难的。产品创新不再是技术的创新,而是产品层面不断加 features,越加越多。为了讨好不同的客户。结果是 90% features 基本没人用,产品也因此变得让人眼花缭乱了。为什么 agency 喜欢这样的产品?因为他们是 power users, features 越多,他们越爽。其他客户面对众多 features,只会晕菜,反而起反作用。

NLP 的真正威力是把数据转为情报,如果一个产品只需要一种情报,譬如舆情,无法消化其他可能有用的情报,NLP 就处于语义落地吃不饱的地位。你吃不饱,你的价值就丧失。

我:
洪诗人有空可以为nlp写一首挽歌,为nlp超出产品一叹。

悟:
李氏唐朝西游记
维度无穷NLP录
立宪定法三权六
委身侍主天地合
@wei 我先抛砖引玉, 见笑了

我:
这砖抛的,狂赞。
【相关】

Final Update of Social Media Sentiment Statistics Before Election

Trump sucks in social media big data in Spanish

Did Trump’s Gettysburg speech enable the support rate to soar as claimed?

Pulse:tracking US election, live feed,real time!

【大数据跟踪美大选每日更新,希拉里成功反击,拉川普下水】

【社煤挖掘:大数据告诉我们,希拉里选情告急】

【社煤挖掘:川普的葛底斯堡演讲使支持率飙升了吗?】

【社煤挖掘:为什么要选ta而不是ta做总统?】

Big data mining shows clear social rating decline of Trump last month

Clinton, 5 years ago. How time flies …

【社媒挖掘:川大叔喜大妈谁长出了总统样?】

【川普和希拉里的幽默竞赛】

【大数据舆情挖掘:希拉里川普最近一个月的形象消长】

欧阳峰:论保守派该投票克林顿

【立委科普:自动民调】

【关于舆情挖掘】

《朝华午拾》总目录

【李白对话录之10:白老师的麻烦不是白老师的】

我:

突然想起一句话 怕忘了 写在这:

“白老师的麻烦是 他懂的 我不懂 我懂的 他懂。”

谁的麻烦?

乔姆斯基说 麻烦是白老师的

菲尔默说 麻烦是我的

后一种语义深度分析的结论是如何得出的?

语义要多茁壮 才能敌得过句法的标配啊。

而且这种语义的蛛丝马迹并非每个人都有捕捉的能力 它远远超出语言学 与一个人的背景知识和领悟力有关

遇到这种极深度的人工智慧 目前能想出来的形式化途径 还是词驱动比较靠谱 如果真想较真探索的话

“麻烦 问题 毛病” 这类词有两个与【human】有关的坑

一个是标配 表达的是所有关系 possessive

另一个是 about 要求填坑的是 【event】或【entity】 后者自然也包括 【human】

白:

“他的教训我一辈子忘不了”

谁被教训?

我: 哈。

回到前面, 近水楼台的 【human】 “白老师” 是标配。

另一条词驱动的可能路径自然休眠。因为词驱动 也就埋下来唤醒的种子。

上下文中遇到另一个 【human】 candidate “我”,加上其他一时也整不清楚但终究可能抓到的蛛丝马迹, 于是休眠唤醒 了。

白:

好像sentiment在休眠唤醒中起比较重要的作用

我:

此句是一例 本来是褒 可不唤醒就是贬了。

白:

标配的麻烦,把负面情感赋与那谁,等到后面说的都是正面,纠结了,另一个human就有空子钻了。

我:

对对对

这个 trick 我们做了n年 sentiment 摸索出来了就在用。典型案例是: “Thank you for misleading me”

Thank 里表达的抽象的褒 由于遭遇了 misleading 的较为具体的贬 而转化为讽刺。

还有:“你做的好事儿 great”。这里 great 的讽刺也是有迹可寻的。

白:

more specific expressions承载的sentiment优先

我:

遇到过两次记者采访,两次都被问到 你们教给机器 sentiment,机器可以理解正话反说 和 讽刺 吗?

我的回答是:这是一个挑战 但其中的一些常见的讽刺说法 是可以形式化 可以捕捉到的。举例就是上面。

白:

具体override抽象。

我:

yes yes yes

白:

如果二者纠结,具体承载的sentiment才是基调,抽象的反向sentiment不是抵消而是修辞手法的开关。

我:

我一直在强调,sentiment 的世界里面,主要是两类东西:一类是情绪的表达,一类是情绪背后的理由。

有些人只表达情绪,但有些人为了说服或影响别人,好恶表态的前后,会说一通理由:you make a point,then you need to support your point with arguments

所谓 sentiment analysis 很长一段时间 领域里面以为那是一个简单的分类问题:thumbs up thumbs down。这个浅陋而流行的观点只是针对的情绪,而面对情绪背后千变万化的理由 就有些抓瞎了。可是没有后者,那个sentiment就没啥特别的价值。

所谓讽刺,只是情绪的转向,正话反说。具体的理由是不能转向的,否则人类的交流就没有一个 protocol 而可以相互理解了。褒贬里面具体的东西 我们叫 pros and cons, 那个东西因为其具体,所以语义是恒定的,不会轻易改变。

情绪却不同。人是一个奇怪的动物,爱极而恨,恨极而爱,都有。甚至很多时候 爱恨交织 自己都搞不清楚。表达为语言,就更诡异善变。

英语口语中 sick 是强烈的褒义情绪,shit 和 crap 等词也不是贬义,bad ass is very positive too:

“The inside of a prius is bad ass no lie.” 是非常正面的褒奖。

人类在情绪表达中说反话,或者由于反话说常了 community 都理解成正话了,这种情形也屡见不鲜。

关键词的褒贬分类系统遇到这种东西不傻眼才怪:当然如果input很长,可以 assume 这类现象只是杂音,整个关键词分类还可以靠谱。但一旦是社会媒体的短消息,这种语言模型比丢硬币好不了多少。

汉语中 老婆太喜欢老公了 喜欢到不知道怎么好了 就说 杀千刀的。

再举一个今天遇到的 sentiment 实际案例:
@Monster47_eNd nah, you have no idea how bad I would kill to eat taco bell or any kind of shit like that.
瞧瞧里面的 sentiment triggers: bad;kill;shit 三个都是强烈的 negative triggers
谈论的 topic 是 Taco Bell,一家流行的墨西哥快餐连锁品牌。
这条短消息通篇没有褒义词出现,因此没有理解、缺乏结构的关键词系统只能得出贬义的结论。但这句话其实是对 Taco Bell 异乎寻常的褒奖 用的是完全草根普罗的用语。

谷歌的神经翻译遇到口语化的句子也基本抓瞎,训练的数据严重口语不足(那是因为双语语料质量过得去的来源大多是正规文档,组织人力去标注口语,做地道的口语翻译,是一个浩大的工程,巨头也无能为力吧):
@ Monster47_eNd nah,你不知道我會殺了多少吃塔可鐘或任何種類的狗屎。

尝试“人工”翻译一哈:
@ Monster47_eNd nah,你不知道为了能吃上Taco Bell 的东东,我會怎样不惜代价(哪怕让我杀人都行)。

简单的译法是:
想吃 Taco Bell 这样的垃圾,我他妈都想疯了。

谁要再说 sentiment 好做,我TM跟他急。这无疑是 NLP 中最艰涩的果子之一。
【相关】

《泥沙龙笔记:parsing 的休眠反悔机制》

【立委科普:基于关键词的舆情分类系统面临挑战】

【立委科普:舆情挖掘的背后】

【李白对话录之九:语义破格的出口】 

李白对话录之八:有语义落地直通车的parser才是核武器

【李白对话录之七:NLP 的 Components 及其关系】

【李白对话录之六:如何学习和处置“打了一拳”】

【李白对话录之五:你波你的波,我粒我的粒】

【李白对话录之四:RNN 与语言学算法】

【李白对话录之三:从“把手”谈起】

【李白隔空对话录之二:关于词类活用】

《李白对话录:关于纯语义系统》

中文处理

Parsing

【置顶:立委NLP博文一览】

《朝华午拾》总目录

Pulse:实时舆情追踪美国大选,live feed,real time!

http://www.netbase.com/presidential-elections2016/

Clinton has been mostly leading the social media sentiment :

Screenshots at 4:50pm 11/8/2016:

11082016a

110820160450b

110820160450c

110820160450d

110820160450e

Again go check our website live on Pulse:

http://www.netbase.com/presidential-elections2016/

 

[Related]

【社煤挖掘:川普的葛底斯堡演讲使支持率飙升了吗?】

【社煤挖掘:为什么要选ta而不是ta做总统?】

Big data mining shows clear social rating decline of Trump last month

Clinton, 5 years ago. How time flies …

【社媒挖掘:川大叔喜大妈谁长出了总统样?】

【川普和希拉里的幽默竞赛】

【大数据舆情挖掘:希拉里川普最近一个月的形象消长】

欧阳峰:论保守派该投票克林顿

【立委科普:自动民调】

【关于舆情挖掘】

《朝华午拾》总目录

 

Final Update of Social Media Sentiment Statistics Before Election

Final update before election:

brand-passion-index-1

timeline-comparison-2
Net sentiment last 24 hours: Trump +7 ; Clinton -9.  The last day analysis of social media.  Buzz:

timeline-comparison-3
So contrary to the popular belief, Trump actually is leading in social media just before the election day.

Compare the above with last month ups and downs to put it in larger context:

brand-passion-index-2
Last 3 month sentiment: Trump -11; Clinton -18.
Buzz for Trump never fails:

timeline-comparison-4

Trump’s Word Clouds:

sentiment-drivers-6

sentiment-drivers-7sentiment-drivers-8

 

 

 

 

 

 

Clinton’s Word Clouds:

sentiment-drivers-9

sentiment-drivers-10

sentiment-drivers-11
Trump 3-month summary:

trumpsummary3m

Clinton 3-month summary:

clintonsummary3m

Ethnicity:

ethinic

RW:
伟哥的东西,好是好,就是没有体现美国的选人制度
Xin:
主要是白人黑人和亚裔人数比例并没有代表实际的选民百分比。
RW:
理论上讲,只要有一方得到所有选票的23%, 他或她就可能当选

 

[Related]

【社煤挖掘:川普的葛底斯堡演讲使支持率飙升了吗?】

【社煤挖掘:为什么要选ta而不是ta做总统?】

Big data mining shows clear social rating decline of Trump last month

Clinton, 5 years ago. How time flies …

【社媒挖掘:川大叔喜大妈谁长出了总统样?】

【川普和希拉里的幽默竞赛】

【大数据舆情挖掘:希拉里川普最近一个月的形象消长】

欧阳峰:论保守派该投票克林顿

【立委科普:自动民调】

【关于舆情挖掘】

《朝华午拾》总目录

【大数据跟踪美大选每日更新,希拉里成功反击,拉川普下水】

昨天发布了【社煤挖掘:大数据告诉我们,希拉里选情告急】,鉴于大选的临近和选情的瞬息万变,我们决定用我们的社煤挖掘的核武器,每日跟踪大数据选情。

美国大选大数据一日一更新,11/1/2016 前24小时,看FBI事件发酵后的走势最新动态:

timeline-comparison-52

1101us

嗨 过去 24 小时,克林顿赶上来了也:两人打平,都是 -12%。热议度克林顿更甚,这也难怪,FBI 重启以后,议论焦点从老川转移到老喜身上。看看BPI这图,这一对真是冤家啊,纠缠在一起:

brand-passion-index-32

川大叔整个被喜大妈包住了,严严实实,比孙悟空的紧箍圈还厉害。Note:里面的圈是川普,外面的圈是希拉里,貌似希拉里气场如今大过老川了。照这个趋势,克林顿希望蛮好。

昨天晚上看新闻,说虽然 FBI 重启对克林顿选情影响很大,传统的新闻民调 CNN poll 还是希拉里领先五个百分点,其他的民调有曾一度只领先一个百分点的记录。虽然都比以前的领先幅度缩小,但仍然领先。川普阵营批判说这些个民调都是被操纵的,他们那边的民调是川普领先。这些个极小数据的民调极易偏差,公婆各有理,还是 put aside,咱们看真正的大数据:这是川普与希拉里最近24小时的 big data summary 对比

1101huanpu24

1101clinton24

回顾重温一下一周来(10/25-11/1)的走向,作为希拉里选情起伏的背景:

timeline-comparison-53

brand-passion-index-33

到现在为止的一周平均 net sentiment,Trump 是 2%,Clinton 是 -12%,可见希拉里的反击,主要不是把自己的 social rating 提升了(过去一天还是 -12),而是把对手拉下水了,让川普从周平均的 +2 拉到现在的冰点以下 -12。克林顿用的是什么伎俩赶上来的呢?

朋友说,大招来了:原来 拉川普下水是找到了川普与普京勾搭的新证据啊:

50740893092863278

A Veteran Spy Has Given the FBI Information Alleging a Russian Operation to Cultivate Donald Trump

Donald Trump Used Legally Dubious Method to Avoid Paying Taxes

约:
有点标题党,内容还算靠谱:

希拉里这次要坐牢?

施:
这次选举是测试大数据有效性的一个试金石,我感觉可能无效….
另:美帝国主义的人民群众也太不成熟了,一点自己的信念都没有?都受舆情影响,吃瓜群众表示不懂

南:
关键是很多选民都没有被社交媒体覆盖到吧

施:
情绪和投票时间的关系是什么样的?

Nick:
没错。伟哥说这么多没用,就一句话:谁能上。

张:
看样子是川普了,我很好奇这个家伙上来会是什么结果

我:
我这才是实事求是,动态跟踪,全方位大数据信息。“谁能上”那算个啥啊?
在胶着的选情下,那就是赌命,有没有大数据,都可以一赌,也都有不小的概率猜中,或猜不中,没有半点营养。如果是非胶着状态,大数据预测比其他预测更准。我坚信。要学那个AI大嘴巴,谁不会?他们根本连技术细节都没有,不过是制造了一个话题,顶了一个AI的帽子,利用普罗和媒体对AI的敬畏。我的选情追踪和分析,比那个高出不知几个数量级,这还真不是吹的。今天的选情趋势如果能够持续,大选日前没有新的定时炸弹被引爆,我预测克林顿当选的可能性可达80%

Nick:
@wei 是骡子是马,拉出来溜溜。就一句话:谁赢。

我:
这样吧,大选日前一天,我做个预测,根据一直到那一刻的综合大数据 analytics,现在不行,选情还在变化,并且显然有胶着的迹象。

Xi:
@wei , 别那么保守! 得老莫者, 得天下! 肯定是Hillary赢了。。。

Nick:
@wei 这算什么本事?

我:
尼克是星座骗女青年骗惯了,只知道短平快 如何得手,顾不了失手的后果了。
反正我有大数据 有平台 有深度parisng 我就这么每日追踪 不打无准备之仗。
以唐老师的说法,得老墨者得天下,那是克林顿无疑了,西班牙语舆情那是一面倒,克林顿高高在上,从来没有下来过

白:
伟哥这是要把谁能上做成红学的节奏。
最后,谁能上不重要了,为了谁能上而秀肌肉的人互撕。

我:
重在过程 不在结果。
这次大选好 富有戏剧性和悬念, 具有观赏性和互撕性, 跌宕起伏 精彩纷呈

阿:
我开了个盘口 目前二人押川普 四人押希太 欢迎加入
重在结果 不在过程

我:
问一句 为什么希拉里推特说的三点facts
第一条说 fbi 并未重启电邮门调查,只是提议重启。

Nick:
@wei 加入盘口,eat your own dog food

我:
第二个 fact 是 fbi director 自己并不清楚新发现的邮件有多少相关
据信很可能是已经审查过的邮件的另一个拷贝。
这个 director 涉嫌扰乱大选,对一个不知结果的新线索 可以按程序重启调查 但在大选前造成舆论 难逃干扰大选的怀疑,他可能也有违法乱纪的麻烦。

 

【相关】

【社煤挖掘:大数据告诉我们,希拉里选情告急】

CNBC‎: AI system finds Trump will win the White House and is more popular than Obama in 2008

Trump sucks in social media big data in Spanish

Did Trump’s Gettysburg speech enable the support rate to soar as claimed?

【社煤挖掘:川普的葛底斯堡演讲使支持率飙升了吗?】

【社煤挖掘:为什么要选ta而不是ta做总统?】

Big data mining shows clear social rating decline of Trump last month

Clinton, 5 years ago. How time flies …

【社媒挖掘:川大叔喜大妈谁长出了总统样?】

【川普和希拉里的幽默竞赛】

【大数据舆情挖掘:希拉里川普最近一个月的形象消长】

欧阳峰:论保守派该投票克林顿

【立委科普:自动民调】

【关于舆情挖掘】

《朝华午拾》总目录

【社煤挖掘:大数据告诉我们,希拉里选情告急】

这是最近最近一周的对比图:

brand-passion-index-15
的确显得不妙,川大叔领先了。是不是因为FBI重启调查造成的结果?
这是过去24小时的图:

brand-passion-index-17
这是一个月的涨跌对比:

timeline-comparison-25

至此局势基本清晰了:希拉里的确选情告急。MD 这大选真是瞬息万变啊,不久前还是喜妈领先或胶着,如今川大叔居然翻身了,选情的变化无常真是让人惊心动魄。

这是last week:

timeline-comparison-26

这一周喜婆,很被动很不利。过去24小时 一直在零下20上下,而老川在零上10左右,有30点的差距 NND:

timeline-comparison-27

看看更大的背景,过去三个月的选情对比:

timeline-comparison-28

原来是, 喜大妈好容易领先了,此前一直落后,直到九月底。九月底到十月中是喜妈的极盛期,是川普的麻烦期。

至于热议度,从来都没有变过,总是川普压倒:

timeline-comparison-31

眼球数也是一样:

timeline-comparison-32

一年来的狂热度(passion intensity)基本上也是川普领先,但喜婆也有不有不少强烈粉她或恨她的,所以曲线有交叉:

timeline-comparison-33

这个 passion intensity 与所谓 engagement 应该有强烈的正相关,因为你痴迷或痛恨一个 candidate 你就愿意尽一切所能去投入、鼓噪、撕逼。

最好是赶快把川大叔的最新丑闻抖出来。这家伙那么多年,难道就留不下比电话录音更猛、更铁的丑闻证据。常识告诉我们肯定有 skeleton in the closet,可是这家伙太狡猾,可能一辈子做商人太过精明,连染有液体的内裤也不曾留下过?是时候从 closet 拿出来了。反正这次大选已经 low 得不能再 low 了,索性 low 到底。不过如果要是有,不会等到今天,大选只剩下一周、先期投票已经开始。

这么看来,作为 data scientist,我不敢不尊重 data 一厢情愿宣传喜妈的赢面大了。赶巧我一周前调查的那个月是克林顿选情的黄金月,结果令人鼓舞。

我们的大数据平台有 27 种 filters,用我们的大数据工具可以对数据做不同的组合切割,要是在会玩的分析师手中,可以做出很漂亮的各种角度的分析报告和图表出来。地理、时间只是其中两项。

电邮门是摧毁性的。FBI 选在大选前一周重启,这个简直是不可思议。比川普的录音曝光的时间点厉害。那家印度所谓AI公司押宝可能押对了,虽然对于数据的分析能力和角度,远不如我们的平台的丰富灵活。他们基本只有一个 engagement 的度量,连最起码的 sentiment classification 都没有,更不用说 social media deep sentiments 了。无论怎么说,希拉里最近选情告急是显然的。至于这种告急多大程度上影响真正的选票,还需要研究。

朋友提醒所谓社会媒体,其实是 pull 和 push 两种信息的交融,其来源也包含了不少news等,这些自上而下的贴子反映的是两党宣传部门的调子,高音量,影响也大,但并非真正的普罗网虫自下而上的好恶和呼声,最好是尽可能剔除前者才能看清真正的民意。下面的一个月走势对比图,我们只留下 twitter,FB,blog 和 microblog 四种社会媒体,剔除了 news 和其他的社会媒体:

timeline-comparison-49

下面是推特 only,大同小异:

timeline-comparison-50

对比一下所有的社会媒体,包括 news 网站,似乎对于这次大选,pull 和 push的确是混杂的,而且并没有大的冲突和鸿沟:

timeline-comparison-51

希拉里为什么选情告急?看看近一个月的希拉里云图,开始红多绿少了:

sentiment-drivers-43

sentiment-drivers-44

对比一下川普的云图,是红绿相当,趋向是绿有变多的趋势,尤其是第二张情绪(emotion)性云图:

sentiment-drivers-45

sentiment-drivers-46

再看看近一周的云图对比, 舆论和选情的确在发生微妙的变化。这是川普最近一周的sentiment 云图:

sentiment-drivers-47

sentiment-drivers-48
对比喜婆婆的一周云图:

sentiment-drivers-49

sentiment-drivers-50

下面是网民的针对希拉里来的正负行为表述的云图:

sentiment-drivers-51

not vote 希拉里的呼声与 vote for her 的不相上下。对比一下川普最近一周的呼声:

sentiment-drivers-52
vote 的呼声超过 not vote for him

这是最近一周关于克林顿流传最广的posts:

clinton_trouble

FBI 重启调查显然被川普利用到了极致,影响深远。

Most popular posts last week by engagement:

clinton_trouble1

Most popular posts last week on Clinton by replies and comments:

clinton_trouble2

Some random sample posts:

clinton_tposts_random
negative comments are rampant on Clinton recently:

clinton_tposts

29367bc4bae054ee9a6262d9cccdfed6

如果这次希拉里输了,the FBI director Comey 居功至伟。因为自从录音丑闻以后,选情对希拉里极为有利,选情的大幅度下滑与FBI重启调查紧密相关。媒体的特点是打摆子,再热的话题随着时间也会冷却,被其他话题代替。这次的问题在,FBI 重启电邮门调查的话题还没等到冷却,大选就结束了,媒体和话题对选民的影响当下为重。而录音丑闻的话题显然已经度过了发酵和热议期,已经冷却,被 FBI 话题代替了。从爆料的角度,录音丑闻略微早了一些,可谁料到在这个节骨眼 FBI 突然来这么一招呢。

看看最近一周的#Hashtags,也可以了解一点社会媒体话题的热度:

word-cloud-23

与事件有关的有: #fbi #hillarysemails #hillarysemail #podestaemails19 #podestaemails20
Negative ones include: #wikileaks #neverhillary #crookedhillary #votetrump

Look at the buzz around Hillary below: the biggest is “FBI” in the brands cloud mentioned with her in the last week’s data:

word-cloud-24

The overall buzz last week:

word-cloud-26

这是最近一周有关希拉里话题的emoji图:

hullery1weekemoji

虽然说笑比哭还,希拉里及其阵营和粉丝却笑不起来,一周内用到这个话题的emoji总数高达 12,894,243 。这也是社会媒体的特点吧,用图画表达情绪。情绪的主调就是 哭。邮件门终于炸了。

现在的纠结是,【大数据告诉我们,希拉里选情告急】,到底发还是不发?为了党派利益和反川立场,不能发。长老川志气,灭吾党威风。为了 data scientist 的职业精神,应该发。一切从数据和事实出发,是信息时代之基。中和的办法是,先发一篇批驳那篇流传甚广的所谓印度AI公司预测川普要赢,因为那一篇的调查区间与我此前做的调查区间基本相同,那是希拉里选情最好的一个月,他们居然根据 engagement alone 大嘴巴预测川普的胜选,根本就没有深度数据的精神,就是赌一把而已。也许等批完了伪AI,宣扬了真NLU,然后再发这篇 【大数据告诉我们,希拉里选情告急】。

FBI director 说这次重启调查,需要很长时间才能厘清。现在只是有了新线索需要重启,不能说明希拉里有罪无罪。没有结论前,先弄得满城风雨,客观上就是给选情带来变数。虽然在 prove 有罪前,都应该假定无罪,但是只要有风声,人就不可能不受影响。所以说这个时间点是最关键的。如果这次重启调查另有黑箱,就更惊心动魄了。如果不是有背后的黑箱和势力,这个时间点的电邮门爆炸纯属与新线索的发现巧合,那就是希拉里的运气不佳,命无天子之福。一辈子强性格,卧薪尝胆,忍辱负重,功亏一篑,无功而返,保不准还有牢狱之灾。可以预测,大选失败就是她急剧衰老的开始。

一周前有个记者interview川普,川普一再说,希拉里这个犯罪的人,根本就不该被允许参加竞选。记者问,哪里犯罪了?川普说电邮门泄密,还有删除邮件隐瞒罪恶。当时这个重启调查还没有。记者问,这个案子不是有结论了吗,难到你不相信FBI的结论?川普说,他们弄错了,把罪犯轻易放了。这是一个腐烂的机构,blah blah。可是,同样这个组织,老川现在是赞誉有加。这就是一个无法无天满嘴跑火车的老狐狸。法律对他是儿戏,顺着他的就对,不顺着他心意的就是 corrupt,rigged,这种人怎么可以放心让他当总统?

中间选民的数量在这种拉锯战中至关重要,据说不少。中间选民如果决定投票,其趋向基本决定于大选前一周的舆论趋向。本来是无所谓是鸡是鸭的,如今满世界说一方不好,合理的推断就是去投另一方了。现在看来,这场竞赛的确是拉锯战,很胶着,不是一方远远超过另一方。一个月前,当录音丑闻爆料的时候,那个时间点,希拉里远远超过川普,毫无悬念。一个月不到,选情大变,就不好说了,迹象是,仍然胶着。

不过,反过来看,川普的 popularity 的确是民意的反映。不管这个人怎么让人厌恶,他所批判的问题的确长久存在。某种意义上,Sanders 这样的极端社会主义者今年能有不俗的表现,成为很多年轻一代的偶像,也是基于类似的对现状不满、对establishment的反叛的民意。而希拉里显然是体系内的老旧派,让人看不到变革的希望。人心思变的时候,一个体系外的怪物也可以被寄托希望。至少他敢于做不同事情,没有瓶瓶罐罐的牵扯。

上台就上台吧,看看他造出一个什么世界。

老闻100年前就说过:
这是一沟绝望的死水,清风吹不起半点漪沦。不如多扔些破铜烂铁,爽性泼你的剩菜残羹。
。。。。。。
这是一沟绝望的死水,这里断不是美的所在,不如让给丑恶来开垦,看它造出个什么世界。

 

【相关】

CNBC‎: AI system finds Trump will win the White House and is more popular than Obama in 2008

Trump sucks in social media big data in Spanish

Did Trump’s Gettysburg speech enable the support rate to soar as claimed?

【社煤挖掘:川普的葛底斯堡演讲使支持率飙升了吗?】

【社煤挖掘:为什么要选ta而不是ta做总统?】

Big data mining shows clear social rating decline of Trump last month

Clinton, 5 years ago. How time flies …

【社媒挖掘:川大叔喜大妈谁长出了总统样?】

【川普和希拉里的幽默竞赛】

【大数据舆情挖掘:希拉里川普最近一个月的形象消长】

欧阳峰:论保守派该投票克林顿

【立委科普:自动民调】

【关于舆情挖掘】

《朝华午拾》总目录

 

Trump sucks in social media big data in Spanish

As promised, let us get down to the business of big data mining of public opinions and sentiments from Spanish social media on the US election campaign.

We know that in the automated mining of public opinions and sentiments for Trump and Clinton we did before, Spanish-Americans are severely under-represented, with only 8% Hispanic posters in comparison with their 16% in population according to 2010 census (widely believed to be more than 16% today), perhaps because of language and/or cultural barriers.  So we decide to use our multilingual mining tools to do a similar automated survey from Spanish Social Media to complement our earlier studies.

This is Trump as represented in Spanish social media for the last 30 days (09/29-10/29), the key is his social rating as reflected by his net sentiment -33% (in comparison with his rating of -9% in English social media for the same period): way below the freezing point, it really sucks, as also illustrated by the concentration of negative Spanish expressions (red-font) in his word cloud visualization.

By the net sentiment -33%, it corresponds to 242,672 negative mentions vs. 121,584 positive mentions, as shown below. In other words, negative comments are about twice as much as positive comments on Trump in Spanish social media in the last 30 days.

This is the buzz in the last 30 days for Trump: mentions and potential impressions (eye balls): millions of data points and indeed a very hot topic in the social media.

This is the BPI (Brand Passion Index) graph for directly comparing Trump and Clinton for their social ratings in the Spanish social media in the last 30 days:

As seen, there is simply no comparison: to refresh our memory, let us contrast it with the BPI comparison in the English social media:

Earlier in one of my election campaign mining posts on Chinese data, I said, if Chinese only were to vote, Trump would fail horribly, as shown by the big margin in the leading position of Clinton over Trump:

This is even more true based on social media big data from Spanish.

This is the comparison trends of passion intensity between Trump and Clinton:

The visualization by weeks of the same passion intensity data, instead of by days, show even more clearly that people are very passionate about both candidates in the Spanish social media discussions, the intensity of sentiment expressed for Clinton are slightly higher than for Trump:

This is the trends graph for their respective net sentiment, showing their social images in Spanish-speaking communities:

We already know that there is simply no comparison: in this 30-day duration, even when Clinton dropped to its lowest point (close to zero) on Oct 9th, she was still way ahead of Trump whose net sentiment at the time was -40%. In any other time segments, we see an even bigger margin (as big as 40 to 80 points in gap) between the two. Clinton has consistently been leading.

In terms of buzz, Trump generates more noise (mentions) than Clinton consistently, although the gap is not as large as that in English social media:

This is the geo graph, so the social data come from mostly the US and Mexico, some from other Latin America countries and Spain:

Since only the Mexicans in the US may have the voting power, we should exclude media from outside the US to have a clearer picture of how the Spanish-speaking voters may have an impact on this election. Before we do that filtering, we note the fact that Trump sucks in the minds of Mexican people, which is no surprise at all given his irresponsible comments about the Mexican people.

Our social media tool is equipped with geo-filtering capabilities: you can add a geo-fence to a topic to retrieve all social media posts authored from within a fenced location. This allows you to analyze location-based content irrespective of post text. That is exactly what we need in order to do a study for Spanish-speaking communities in the US who are likely to be voters, excluding those media from Mexico or other Spanish-speaking countries. communities in the US who are likely to be voters, excluding those media from Mexico or other countries. This is also needed when we need to do study for those critical swing states to see the true pictures of the likelihood of the public sentiments and opinions in those states that will decide the destiny of the candidates and the future of the US (stay tuned, swing states social media mining will come shortly thanks to our fully automated mining system based on natural language deep parsing).

Now I have excluded Spanish data from outside America, it turned out that the social ratings are roughly the same as before: the reduction of the data does not change the general public opinions from Spanish communities, US or beyond US., US or beyond US. This is US only Spanish social media:

This is summary of Trump for Spanish data within US:

It is clear that Trump’s image truly sucks in the Spanish-speaking communities in the US, communities in the US, which is no surprise and so natural and evident that we simply just confirm and verify that with big data and high-tech now.

These are sentiment drivers (i.e. pros and cons as well as emotion expressions) of Trump :

We might need Google Translate to interpret them but the color coding remains universal: red is for negative comments and green is positive. More red than green means a poor image or social rating.

In contrast, the Clinton’s word clouds involve way more green than red: showing her support rate remains high in the Spanish-speaking communities of the US.

It looks like that the emotional sentiments for Clinton are not as good as Clinton’s sentiment drivers for her pros and cons.

Sources of this study:

Domains of this study:

[Related]

Did Trump’s Gettysburg speech enable the support rate to soar as claimed?

Big data mining shows clear social rating decline of Trump last month

Clinton, 5 years ago. How time flies …

Automated Suevey

Dr Li’s NLP Blog in English

Did Trump’s Gettysburg speech enable the support rate to soar as claimed?

Last few days have seen tons of reports on Trump’s Gettysburg speech and its impact on his support rate, which is claimed by some of his campaign media to soar due to this powerful speech.  We would love to verify this and uncover the true picture based on big data mining from the social media.

First, here is one link on his speech:

DONALD J. TRUMP DELIVERS GROUNDBREAKING CONTRACT FOR THE AMERICAN VOTER IN GETTYSBURG. (The most widely circulated related post in Chinese social media seems to be this: Trump’s heavyweight speech enables the soaring of the support rate and possible stock market crash).

Believed to be a historical speech in his last dash in the campaign, Trump basically said: I am willing to have a contract with the American people on reforming the politics and making America great again, with this plan outline of my administration in the time frame I promised when I am in office, I will make things happen, believe me.

Trump made the speech on the 22nd this month, in order to mine true public opinions of the speech impact, we can investigate the data around 22nd for the social media automated data analysis.  We believe that automated polling based on big data and language understanding technology is much more revealing and dependable than the traditional manual polls, with phone calls to something like 500 to 1,000 people.  The latter is laughably lacking sufficient data to be trustworthy.

timeline-comparison-14

What does the above trend graph tell us?

1  Trump in this time interval was indeed on the rise. The “soaring” claim this time does not entirely come out of nowhere, but, there is a big BUT.

2. BUT, a careful look at the public opinions represented by net sentiment (a measure reflecting the ratio of positive mentions over negative mentions in social media) shows that Trump has basically stayed below the freezing point (i.e. more negative than positive) in this time interval, with only a brief rise above the zero point near the 22nd speech, and soon went down underwater again.

3. The soaring claim cannot withstand scrutiny at all as soaring implies a sharp rise of support after the speech event in comparison with before, which is not the case.

4. The fact is, Uncle Trump’s social media image dropped to the bottom on the 18th (with net sentiment of -20%) of this month.  From 18th to 22nd when he delivered the speech, his net sentiment was steadily on rise from -20% to 0), but  from 22nd to 25th, it no longer went up, but fell back down, so there is no ground for the claim of support soaring as an effect of his speech, not at all.

5. Although not soaring, Uncle Trump’s speech did not lead to sharp drop either, in terms of the buzz generated, this speech can be said to be fairly well delivered in his performance. After the speech, the net sentiment of public opinions slightly dropped, basically maintaining the fundamentals close to zero.

6.  The above big data investigation shows that the media campaign can be very misleading against the objective evidence and real life data.  This is all propaganda, which cannot be trusted at its face value: from so-called “support rate soared” to “possible stock market crash”. Basically nonsense or noise of campaign, and it cannot be taken seriously.

The following figure is a summary of the surveyed interval:

trump1

As seen, the average public opinion net-sentiment for this interval is -9%, with positive rating consisting of 2.7 million mentions, and negative rating of 3.2 million mentions.

How do we interpret -9% as an indicator of public opinions and sentiments? According to our previous numerous automated surveys of political figures, this is certainly not a good public opinion rating, but not particularly bad either as we have seen worse.  Basically, -9% is under the average line among politicians reflecting the public image in people’s minds in the social media.  Nevertheless, compared with Trump’s own public ratings before, there is a recorded 13 points jump in this interval, which is pretty good for him and his campaign.  But the progress is clearly not the effect of his speech.

This is the social media statistics on the data sources of this investigation:

trump2

In terms of the ratio, Twitter ranks no 1, it is the most dynamic social media on politics for sure, with the largest amount of tweets generated every minute. Among a total of 34.5 million mentions on Trump, Twitter accounted for 23.9 million.  In comparison, Facebook has 1.7 million mentions.

Well, let’s zoom in on the last 30 days instead of only the days around the speech, to provide a bigger background for uncovering the overall trends of this political fight in the 2016 US presidential campaign between Trump and Clinton.

timeline-comparison-15

The 30 days range from 9/28-10/28, during which the two lines in the comparison trends chart show the contrast of Trump and Clinton in their respective daily ups and downs of net sentiment (reflecting their social rating trends).  The general impression is that the fight seems to be fairly tight.  Both are so scandal-ridden, both are tough and belligerent.  And both are fairly poor in social ratings.  The trends might look a bit clearer if we visualize the trends data by weeks instead of by day:

timeline-comparison-16

No matter how much I dislike Trump, and regardless of my dislike of Clinton whom I have decided to vote anyway in order to make sure the annoying Trump is out of the race,  as a data scientist, I have to rely on data which says that Hillary’s recent situation is not too optimistic: Trump actually at times went a little ahead of Clinton (a troubling fact to recognize and see).

timeline-comparison-17

The graph above shows a comparison of the mentions (buzz, so to speak).  In terms of buzz, Trump is a natural topic-king, having generated most noise and comments, good or bad.  Clinton is no comparison in this regard.

timeline-comparison-18

The above is a comparison of public opinion passion intensity: like/love or dislike/hate?  The passion intensity for Trump is really high, showing that he has some crazy fans and/or deep haters in the people.  Hillary Clinton has been controversial also and it is not rare that we come across people with very intensified sentiments towards her too.  But still, Trump is sort of political anomaly, and he is more likely to cause fanaticism or controversy than his opponent Hillary.

In his recent Gettysburg speech, Trump highlighted the so-called danger of the election being manipulated. He clearly exaggerated the procedure risks, more than past candidates in history using the same election protocol and mechanism.  By doing so, he paved the way for future non-recognition of the election results. He was even fooling the entire nation by saying publicly nonsense like he would totally accept the election results if he wins: this is not humor or sense of humor, it depicts a dangerous political figure with ambition unchecked.  A very troubling sign and fairly dirty political tricks or fire he is playing with now, to my mind.  Now the situation is, if Clinton has a substantial lead to beat him by a large margin, this old Uncle Trump would have no excuse or room for instigating incidents after the election.  But if it is closer to see-saw, which is not unlikely given the trends analysis we have shown above, then our country might be in some trouble: Uncle Trump and his die-hard fans most certainly will make some trouble.  Given the seriousness of this situation and pressing risks of political turmoil possibly to follow,  we now see quite some people, including some conservative minds, begin to call for the election of Hillary for the sake of preventing Trump from possible trouble making.  I am one with that mind-set too, given that I do not like Hillary either.  If not for Trump, in ordinary elections like this when I do not like candidates of both major parties, I would most likely vote for a third party, or abstain from voting, but this election is different, it is too dangerous as it stands.  It is like a time bomb hidden somewhere in the Trump’s house, totally unpredictable. In order to prevent him from spilling, it is safer to vote for Clinton.

In comparison with my earlier automated sentiment analysis blogged about a week ago (Big data mining shows clear social rating decline of Trump last month),this updated, more recent BPI brand comparison chart seems to be more see-saw: Clinton’s recent campaign seems to be stuck somewhere.

brand-passion-index-11

Over the last 30 days, Clinton’s net sentiment rating is -17%, while Trump’s is -19%.  Clinton is only slightly ahead of Trump.  Fortunately, Trump’s speech did not really reverse the gap between the two, which is seen fairly clearly from the following historical trends represented by three different circles in brand comparison (the darker circle represents more recent data): the general trends of Clinton are still there: it started lagging behind and went better and now is a bit stuck, but still leading.

 

brand-passion-index-12

Yes, Clinton’s most recent campaign activities are not making significant progress, despite more resources put to use as shown by bigger darker circle in the graph.  Among the three circles of Clinton, we can see that the smallest and lightest circle stands for the first 10 days of data in the past 30 days, starting obviously behind Trump.  The last two circles are data of the last 20 days, seemingly in situ, although the circle becomes larger, indicating more campaign input and more buzz generated.  But the benefits are not so obvious.  On the other side, Trump’s trends show a zigzag, with the overall trends actual declining in the past 30 days.  The middle ten days, there was a clear rise in his social rating, but the last ten days have been going down back.  Look at Trump’s 30-day social cloud of Word Cloud for pros and cons and Word Cloud for emotions:

Let us have a look at Trump’s 30-day social media sentiment word clouds, the first is more about commenting on his pros and cons, and the second is more direct and emotional expressions on him:sentiment-drivers-38

sentiment-drivers-37
One friend took a glance at the red font expression “fuck”, and asked: who are subjects and objects of “fuck” here?  In fact, the subject generally does not appear in the social posts, by default it is the poster himself, reflecting part of the general public, the object of “fuck” is, of course, Trump, for otherwise our deep linguistics based system will not count it as a negative mention of trump reflected in the graph.  Let us show some random samples side by side of the graph:

trumpfuck

trumpfuck2
My goodness, the “fuck” mentions account for 5% of the emotional data, the poor old Uncle Trump is fucked 40 million times in social media within one-month duration, showing how this guy is hated by some of the people whom he is supposed to represent and govern if he takes office.   See how they actually express their strong dislike of Trump:

fucking moron
fucking idiot
asshole
shithead

you name it, to the point even some Republicans also curse him like crazy:

Trump is a fucking idiot. Thank you for ruining the Republican Party you shithead.

Looking at the following figure of popular media, it seems that the most widely circulated political posts in social media involve quite some political video works:

trumpmedia

The domains figure below shows that the Tumblr posts on politics contribute more than Facebook:

domains-6

In terms of demographics background of social media posters, there is a fair balance between male and female: male 52% female 48% (in contrast to Chinese social media where only 25% females are posting political comments on US presidential campaign).  The figure below shows the ethnic background of the posters, with 70% Caucasians, 13% African Americans, 8% Hispanic and 6% Asians.  It looks like that the Hispanic Americans and Asian Americans are under-represented in the English social media in comparison with their due population ratios, as a result, this study may have missed some of their voice (but we have another similar study using Chinese social media, which shows a clear and big lead of Clinton over Trump; given time, we should do another automated survey using our multilingual engine for Spanish social media.  Another suggestion from friends is to do a similar study on swing states because after all these are the key states that will decide the outcome of this election, we can filter the data by locations where posts are from to simulate that study).  There might be a language or cultural reasons for this under-representation.

trumpethinics

This last table involves a bit of fun facts of the investigation.  In social media, people tend to talk most about the campaign, on the Wednesday and Sunday evenings, with 9 o’clock as the peak, for example, on the topic of Trump, nine o’clock on Sunday evening generated 1,357,766 messages within one hour.  No wonder there is no shortage of big data from social media on politics.  It is all about big data. In contrast, with the traditional  manual poll, no matter how sampling is done, the limitation in the number of data points is so challenging:
with typically 500 to 1000 phone calls, how can we trust that the poll represents the public opinions of 200 million voters?  They are laughably too sparse in data.  Of course, in the pre-big-data age, there were simply no alternatives to collect public opinion in a timely manner with limited budgets.  This is the beauty of Automatic Survey, which is bound to outperform the manual survey and become the mainstream of polls.

trumpdayhour

Authors with most followers are:

trumpmedia2

Most mentioned authors are listed below:

trumpauthors

Tell me when in history did we ever have this much data and info, with this powerful data mining capabilities of fully sutomated mining of public opinions and sentiments at scale?

trumppopularposts

 

[Related]

Big data mining shows clear social rating decline of Trump last month

Clinton, 5 years ago. How time flies …

Automated Suevey

Dr Li’s NLP Blog in English

 

 

【社煤挖掘:川普的葛底斯堡演讲使支持率飙升了吗?】

反正日夜颠倒了,那就较真一下,看看大数据大知识,对于川普的葛底斯堡演说的所谓舆情飙升到底是怎么回事。先给几个links:

DONALD J. TRUMP DELIVERS GROUNDBREAKING CONTRACT FOR THE AMERICAN VOTER IN GETTYSBURG

报道的是本月22日川大叔的历史性演说,旨在振奋人心,做竞选的最后冲刺,大意:
寡人与美国人民有个约定,看我的,believe me

中文舆论中,这篇似乎流传最广:【川普重磅演讲致支持率飙升 全球股市将暴跌?】。

因为川普演说是22日,为了看舆情的飙升对比,可以以22日为中心取前后几天的社会媒体大数据做分析,看个究竟。至少比传统民调打五百、一千个电话来调查,自动民调的大数据(millions 的数据点)还是靠谱一些吧。

timeline-comparison-14
这张趋势图怎么看?

1 川普在这个时间区间总体的确是上升。飙升之说,不完全是无中生有(准确地说,其实是捕风捉影,见下)。

2 但是,仔细看舆情(net sentiment)图可以发现,川普这段时间基本上还是一直没有摆脱负面舆情多于正面舆情的局面,舆情曲线除了22号当天短暂超越冰点,总体一直是零下。

3. 飙升之说经不起推敲,因为凡飙升,必须是事件后比事件前的舆情,有明显的飞跃,其实不然。

4. 事实是,川大叔近期舆情的谷底是本月18号(零下20+),从18号到22号 他 deliver speech 前,他的舆情已经有比较明显的提升(从 -20 到 0),而从 22 号 到 25 号,舆情不升反略降,飙升从何谈起?

5. 虽然没有飙升,但川大叔这次表演还是及格的。至少 speech 后,舆情没有大跌,基本保持了接近零度的基本面。

6 由此可见,媒体造势是多么地捕风捉影。以后各位看到这种明显是宣传(propaganda)的帖子,可以多一个心眼了:通常的宣传造势的帖子都在夸大其词(如果不公然颠倒黑白或歪曲事实的话),从所谓“舆情飙升”到预计“股市暴跌”,都是要显示川普演说的重量级。基本是无稽之言,不能当真的。

下图是这个调查区间的数据小结:

trump1

这个区间的平均舆情指数是 -9%,2.7 million 的正面评价,3.2 million 的负面评价。

-9% 是一个什么概念,根据我们以往对政治人物的多次舆情调查来看,这不是一个好的舆情,但也不是特别糟糕,属于平均线下。但是,与川普自己的总体舆情比较,这个区间表现良好,有 13 点的提升,但这个提升并非所谓演说飙升带来的。

这是社煤数据源的统计:

trump2

从比例看,推特永远是最 dynamic,量也最大,总热议度 34.5 million mentions,推特占了 23.9 million。不少社煤的分析 apps 干脆扔掉其他的数据源,只做推特,作为社会媒体的代表,也基本上可以了。但是,感觉上还是,只做推特,虽然大数据之量可以保证,但可能偏差会大一些,因为喜欢上推特跟踪政治人物和话题,吐槽或粉丝的人,只是社会阶层中的一部分,往往是比较狂热的一批。推特这个公共平台,本来就长于偶像和followers(粉丝或“黑”)互动。其他的社会媒体可能更平实一些,譬如 Facebook 上的发言基本是说给朋友圈的。Facebook 也有 1.7 million 的热议。

好,我们把区间放大,看 last 30 days 的趋势,作为这次演说前后趋势的一个背景。

timeline-comparison-15
这是 9/28-10/28 的川普与克林顿舆情趋势对比图,by days;仔细解读前,总体印象是够纠缠的。这两位老头老太也真是,剪不断理还乱,不是冤家不碰头,呵呵。两位都那么多丑闻缠身,性格都很tough倔强。看看一个月来 by weeks 的曲线也许更明朗:

timeline-comparison-16

不管我多么厌恶川普,也不管我为了厌恶川普而决定选举并不喜欢的克林顿,作为 data scientist,不得不说,希拉里最近的情势不是很乐观:川普居然开始有点儿领先克林顿的趋势了,NND。

timeline-comparison-17

上图是热议度(mentions)的对比。这个没的说,川普天生的话题大王,克林顿无论如何也赶不上。

timeline-comparison-18

这是舆情烈度的对比:喜欢或厌恶川普的还是更加狂热,虽然印象中希拉里克林顿比起其他政治人物所引起的情绪已经要更趋于激烈了。可是川普是个政治异数,还是更容易引起狂热或争议。

川普在演说中特别强调选举被操纵的危险,他显然在夸大这种危险,为将来的不承认选举结果做铺垫。挺恶心人的。现在的情况是,如果克林顿大幅度领先,川大叔再流氓也没辙。如果是拉锯接近,就麻烦了,老川和川粉几乎肯定要闹事。可现在的选情显得有些胶着拉锯,这也是为什么很多人包括保守派开始有倡议,说为了川普,请投票克林顿。本来我是要投第三党的,或者弃权不投,但是这次选举不同,危险太大,川老是个定时炸弹,而且不可预测。为了防止他撒泼,还是投给克林顿好。至少让他看看,马戏团的表演是上不了台面的,由不得他胡来。沐猴而冠变不成林肯。

对比我 一周前做的自动民调 Big data mining shows clear social rating decline of Trump last month,下面这个品牌对比图似乎更加拉锯,克林顿最近选情不是很佳。

brand-passion-index-11

最近30天,克林顿是 -17%,川普是 -19%,略领先于川普。所幸,川普的这次演讲并没有真正扭转两人的差距,从下面这张历史趋势品牌对比看,克林顿从开始的舆情落后,变为领先的趋势还在:

brand-passion-index-12
不过最近克林顿的选情是原地踏步,并没有明显进展。比较克林顿的三个圈可知,最淡的圈是过去30天的前10天,明显落后于川普,后两个圈是最近20天,基本原地,只是圈子变大了,说明竞选的投入和力度加大了,但效益并不明显。而从川普方面的三个圈圈看趋势,这老头儿实际的总体趋势是下跌,过去三十天,中间的十天舆情有改观,但最近的十天又倒回去了,虽然热议度有增长。(MD,这个分析没法细做,越做越惊心动魄,很难保持平和的心态,可咱是 data scientist 啊。朋友说,“就是要挖点惊心动魄的”,真心唯恐天下不乱啊。)看看川普的30天社煤的褒贬云图(Word Cloud for pros and cons)和情绪云图(Word Cloud for emotions)吧:

sentiment-drivers-38

sentiment-drivers-37
朋友一眼看中了那红红的 fuck 舆情,问:“fuck”的主语和宾语是谁?

主语一般不出现,默认是普罗网虫,fuck 的宾语当然是川普,否则上不来他的负面情绪云图:

trumpfuck

trumpfuck2
天,fuck mentions 占据了情绪数据的 5%,老川在一个月里被社煤普罗 fuck 了近40万次,可见这家伙如果上台会有多少与他不共戴天的子民。看上面怎么吐槽 fuck 的:

fucking moron
fucking idiot
asshole
shithead

you name it,甚至疑似共和党人也fuck他:
Trump is a fucking idiot. Thank you for ruining the Republican Party you shithead.

 

看 popular media,貌似流传最广的大多是视频:

trumpmedia

Tumblr 超越 Facebook 成为社煤老二?

domains-6

从来没用过 Tumblr 这名字也拗口 怎么这么 popular?

西方媒体吐槽的,男女比较均衡:male 52% female 48%,对比中文社媒,明显是女人少谈政治的:才占25%。这次调查的种族背景分布:

trumpethinics

还是白大哥占压倒多数。族裔信息占社煤帖子中的近一半,所以这个社煤族裔分布的情报应该是靠谱的。黑大哥第二,占 13%,亚裔才 6%。墨大哥 8%, 与其人口比例不相称吧(?):由于语言或文化障碍,under-represented here??

这个有点意思,喜欢到社煤吐槽的人,集中在周三和周日的晚上,晚九点达到高峰, 譬如 关于川普话题的社煤,在周日晚上九点高达 1,357,766, 一个小时就有一百三十五万帖啊,够大数据吧。

trumpdayhour

这还才是 sampling 的 data, 推特sampling占总量大约十分之一吧,如果是 data hose (要额外付钱的)一网打尽的话,数据量又要增加一个量级。不过,对于大数据情报挖掘,再增加一个量级已经没有什么意义了,不会实质上改变调查的结果的。说明一下,那个周日的统计量应该是过去一个月的调查中的周日的总和,一个月有四个周日,那个数据应该除以4,然后乘以10,才是川普数据周日九点的那是时间区间的真实量。总之是地地道道的大数据。相比之下,传统民调,不管怎么抽样,感觉都是儿戏,有点胡闹:
500 个电话,说是代表了两亿人的民意舆情,不是儿戏是什么。不过,前大数据时代,那是没办法的办法。自动民调是大势所趋

下图是影响最大 followers 最多的 authors:

trumpmedia2

Most mentioned authors below:

trumpauthors

什么时代有过如此丰富的信息与如此强大的数据挖掘能力?

RW:
@wei 你实际上可以好好搞一个大选预测引擎,利用你现在的methodology, finetune 一下,可以吸引很多眼球。效果好,下次就可以收费了。一炮而红,还有什么是更有效的marketing?

我:
我要是有微信数据的话,不打炮也会红。什么都不用变,就是现在的引擎,现在的app,只要有微信,什么情报专家也难比拟。为什么现在发布中文舆情挖掘不如英文挖掘那么有底气?不是我中文不行,而是数据源太 crappy 了。闹来闹去也就是新浪微博、天涯论坛、中文推特或脸书。至少全球华人大陆背景的,这个压倒多数,都在用微信,而数据够不着,得不到反映。

李:
@wei 我公司有团队做着类似的事情

我:
你能染指微信数据?

李:
微信个人数据只有腾讯有。

看看流传最广的社煤帖子都是什么?

trumppopularposts

从 total engagement 指标看,无疑是川普自己的推特账号,以及 Fox : 这大概是唯一的主流媒体中仅存的共和党的声音了。也不怪,老川在竞选造势中,不断指着鼻子骂主流媒体,甚至刻薄主持人的偏袒。历史上似乎还没有一个候选人与主流媒体如此对着干,也没有一个人被主流媒体如此地厌恶。

展示到这里,朋友转来一个最新的帖子,说是用人工智能预测美国大选,川普会赢:Trump will win the election and is more popular than Obama in 2008, AI system finds,quote:

But the entrepreneur admitted that there were limitations to the data in that sentiment around social media posts is difficult for the system to analyze. Just because somebody engages with a Trump tweet, it doesn’t mean that they support him. Also there are currently more people on social media than there were in the three previous presidential elections.

haha,同行是冤家,他的AI能比我自然语言deep parsing支持的 I 吗?从文中看,他着重 engagement,这玩意儿的本质就是话题性、热议度吧。早就说了,川普是话题大王,热议度绝对领先。(就跟冰冰一样,话题女王最后在舆情上还是败给了舆情青睐的圆圆,不是?)不是码农相轻,他这个很大程度上是博眼球,大家都说川普要输,我偏说他必赢。两周后即便错了,这个名已经传出去了。川普团队也会不遗余力帮助宣传转发这个。

Xi:
那个印度鬼子也有点瞎扯了。
知道ip地址跟知道ssl加密后的搜索的内容是两码事儿啊!
不知道是记者不懂呢,还是这小子就是在瞎胡弄了。

洪:
印度ai公司预测美国大选,有50%以上测准概率,中国ai公司也别放过这个机会

毛:
伟哥为什么认为川普必赢?不是说希拉莉的赢率是 95% 吗?

南山/邓保军: 不是wei说的

我:
这叫横插一杠子。川普要赢,我去跳河。。。

毛:
哦,伟哥是在转述。

我:
跳河是玩笑了,我移民回加拿大总是可以吧。

李:
韩国这个料就爆得好。希拉里在关键时刻,也有可能爆大料

我:
问题是谁爆谁的料。两人都到了最后的时刻,似乎能找到的爆料也都差不多用了。再不用就不赶趟了。很多地方的提早投票都已经开始了,有杀手锏最多再等两三天是极限了,要给媒体和普罗一个消化和咀嚼的时间。

毛:
@wei 但是老印的那个系统并非专为本届大选而开发,并且说是已经连续报准了三届呀?

我:
我的也不是专为大选开发的呀。而且上次奥巴马决定用我们,你看他就赢了,我们也助了一臂之力呢。

毛:
你们两家的配方不同?

我:
奥巴马团队拥抱新技术,用舆情挖掘帮助监测调整竞选策略,这个比预测牛一点点吧。预测是作为 outsider 来赌概率。我这个是 engage in the process、技术提供助力 呵呵。当时不允许说的。

李:
奥巴马有可能会去硅谷打工唉

毛:
是否在舆情之外还有什么因素?

李:
原来你那个奥巴马照片不是蜡像呀

我:
假做真时真亦假呀

002_510_image

 

【相关】

【社煤挖掘:为什么要选ta而不是ta做总统?】

Big data mining shows clear social rating decline of Trump last month

Clinton, 5 years ago. How time flies …

【社媒挖掘:川大叔喜大妈谁长出了总统样?】

【川普和希拉里的幽默竞赛】

【大数据舆情挖掘:希拉里川普最近一个月的形象消长】

欧阳峰:论保守派该投票克林顿

【立委科普:自动民调】

【关于舆情挖掘】

《朝华午拾》总目录

 

 

 

 

【社煤挖掘:为什么要选ta而不是ta做总统?】

中文社煤挖掘美国大选的华人舆情,接着练。

Why and why not Clinton/Trump?

Why 喜大妈?Why 川大叔?Why not Clinton? Why not Trump?这是大选的首要问题,也是我们舆情挖掘想要探究的重点。Why???

First, why Clinton and why not Clinton? 看看喜大妈在舆情中的优劣对比图(pros and cons)。

sentiment-drivers-33

why Clinton?剔除竞选表现优秀等等与总统辩论和 campaign 有关的好话(“领先”、“获胜”、“占上风”、“赢得”等)外,主要理由有:

1. 老练 强硬; 2. 乐观; 2. 清楚; 4 换发活力 谈笑风生; 5. 梦想共同市场

拿着放大镜,除了政治套话和谀辞外也没看到什么真正的亮点。舆情领先,只能说对手太差了吧。四年前与奥巴马竞争被甩出一条街去,那是遇到了真正的强手。

OK,why not Clinton?

1. 性侵 性骚扰 威胁(她丈夫做的好事,她来背黑锅,呵呵。照常理她是受害者,可以同情的,不料给同样管不住下半身的川普一抹黑,她倒成了性侵的帮凶,说是威胁被性侵的女性。最滑稽的是,川普自己的丑闻曝光,他却一本正经带了一帮前总统克林顿的绯闻女士开记者会,来抹黑自己的对手克林顿夫人。滑稽逆天了。)

2. 邮件门 曝光 泄密

3 竞选团队的不轨行为 操纵大选 作弊

4. 克林顿基金会的问题

5. 华尔街收费

6 健康问题

7 撒谎、可耻

8. 缺乏判断力

这些都不是新鲜事儿,大选以来已经炒了很久了,但比起她的长处(经验老练等少数几条),喜妈被抓住的辫子还真不少。再看网民的情绪性吐槽, 说好话都是相似的,坏话却各有不同:轻的是,“乏善可陈”、“不喜欢”、“不信任”; 重的是:“妖婆”,“婊子”、“灾难”、“无耻”、“邪恶”。

sentiment-drivers-34
作为对比,来看川大叔,why or why not Trump?

sentiment-drivers-35

pros:1. 减税;2. 承诺 崛起 (America great again);3. 真实;4. 擅长 business
cons:
1. 曝光的视频丑闻 性骚扰
2. 偷税漏税
3. 吹嘘
4 咄咄逼人 喜怒无常
5 粗鄙、威胁
6 撒谎

情绪性吐槽,轻的是 “不靠谱”、“出言不逊”,重的是 “恶心”、“愚蠢”、“卑劣”、“众叛亲离”。

sentiment-drivers-36
上篇中文社煤自动民调博文发了以后有朋友问,为什么不见大名鼎鼎的脸书。(微信不见可以理解,人家数据不对外开放,对隐私性特别敏感,比脸书严多了。不过,地球人都知道,反映我大唐舆情最及时精准的大数据宝库,非微信莫属)。查对了一下,上次做的中文舆情调查,不知何故 Facebook 不在 top 10,只占调查数据的 0.1%:

sources-9

记得以前的英语社煤调查,通常的比例是 70% twitter,20% Facebook, 其他所有论坛和社交媒体只占 10%。最近加了 instagram、Tumblr 等,格局似有变。但是中文在海外,除了推特,Facebook 本来应该有比重的,特别是我台湾同胞,用 Facebook 跟东土用微信一样普遍。

再看看这次调查的网民背景分类。

1.  职业是科技为主(大概不少是咱码农),其次才是新闻界和教育界。这些人喜欢到网上嚷嚷。

professions

这是他们的兴趣(interests),有意思的关联似乎是,喜欢谈政治的与喜欢谈宗教和美食的有相当大交集。

interests

这是年龄分组,分布比较均匀,但还是中青年为主。

age

性别不用说,男多女少。男人谈政治与女人谈shopping一样热心。

gender

最后看看地理分布,社煤的地理来源:
geo-regions

 

 

【相关】

【社媒挖掘:川大叔喜大妈谁长出了总统样?】

Big data mining shows clear social rating decline of Trump last month

【川普和希拉里的幽默竞赛】

【大数据舆情挖掘:希拉里川普最近一个月的形象消长】

论保守派该投票克林顿

【立委科普:自动民调】

【关于舆情挖掘】

《朝华午拾》总目录

【社媒挖掘:川大叔喜大妈谁长出了总统样?】

眼看决战时刻快到了,调查一下华人怎么看美国大选,最近一个月的舆情趋势。中文社会媒体对于美国总统候选人的自动调查。

aaa

先看喜大妈,是过去三十天的调查(时间区间:9/26-10/25)
summary-metrics-new-3
mentions 是热议度,net sentiment 是褒贬指数,反映的网民心目中的形象。

summary-metrics-6
很自然,二者并不总是吻合:譬如,在十月10日到11日的时候,希拉里被热议,而她的褒贬指数则跌入谷底。那天有喜大妈的什么丑闻吗?咱们把时间按周(by weeks)而不是按日来看 trends,粗线条看趋势也许更明显一些:

summary-metrics-7
Anyway,过去30天的总社煤形象分(net sentiment)是 11%,比起英语世界的冰点之下(-18%)好太多了,似乎华语世界远不如英语世界对老政客喜大妈的吐槽刻薄。

作为对比,我们看看川普(特朗普)在同一个时期的社会形象的消长趋势:川普过去30天的总社煤形象分(net sentiment)是 -12%,比希拉里的+11%成鲜明对比。

summary-metrics-8

看上面的趋势图(by weeks),川普的热议度一直居高不下,话题之王名副其实,但他的社会评价却一直在冰点之下,十月初更是跌入万丈深渊。同时期的希拉里,热议度与社会评价却时有交叉。趋势 by days:

summary-metrics-9

这样看来,虽然有所谓华人挺川的民间鼓噪,总体来看,川大叔在华人的网上口水战中,与喜大妈完全不是一个量级的对手。川普很臭,真地很臭。在英语社煤中,川普也很臭(-20%),但希拉里也不香,民间厌恶她诅咒她的说法随处可见,得分 -18%,略好于川普。譬如电邮门事件,很多老美对此深恶痛绝,不少华人(包括在下)心里难免觉得是小题大作。为什么华人世界对希拉里没有那么反感呢?居然给希拉里 +11% 的高评价。朋友说,希拉里更符合华人主流价值观吧。

这是我们的品牌对比图,三维直观地对比两位候选人在社煤的形象位置:

brand-passion-index-10

希拉里领先太多,虽然热议度略逊。

总有人质疑社煤挖掘的情报价值,说也许NLU不过关,挖掘有误呢。更多的质疑是,也许某党的人士更愿意搅浑水呢(譬如利用水军或机器人bots)。凡此总总,都给社会媒体舆情挖掘在多大程度上反映民意,提出了疑问和挑战。其实,对于传统的民调,不同的机构有不同的结果,加上手工民调的取样不可能大,error margin 也大。各机构结果也颇不同,所以大家也都是一肚子怀疑。不断有怀疑,还是不断有民调在进行。这是大选年的信息“刚需”吧。

所有的自动的或人工的民调,都可能有偏差,都只能做民意的参考。但是我要强调的是:

1. 现在的深度 NLU 支持的舆情挖掘,已经今非昔比,加上大数据信息冗余度的支撑,精准度在宏观上是可以保障的;

2. 全自动的社煤民调,其大数据的特性,是人工民调无法比的(时效以及costs也无法比,见【立委科普:自动民调】);

3. 虽然社煤上的口水、噪音以及不同党派或群体在其上的反映都可能有很大差异,但是社煤民调的消长趋势的情报以及不同候选人(或品牌)的对比情报,是相对可靠的。怎么讲?因为自动系统具有与生俱来的一视同仁性。

时间维度上的舆情消长,具有相对的比较价值,它基本不受噪音或其他因素的影响。也不大受系统数据质量的影响(当然,太臭的舆情系统也还是糊不上墙,跟抛硬币差不了太多的一袋子词这样的“主流”舆情分类,在短消息压倒多数的社会媒体,还是不要提了吧,见一切声称用机器学习做社会媒体舆情挖掘的系统,都值得怀疑)。

我们目前的系统,是 deep parsing 支持,本性是 precision 优于 recall(precision 不降低,recall 也可以慢慢爬上来,譬如我们的英语舆情系统就有相当好的recall,recall在符号逻辑路线里面,本质上就是开发时间的函数)。Given big data 这样的场景,recall 的某种缺失,其实并不影响舆情的相对意义,因为决定 recall 的是规则量,缺少的是一些长尾 pattern rules,而语言学的 rules 不会因为时间或候选人的不同,而有所不同。同理,因为系统的编制是独立于千变万化的候选人、品牌或话题,因此数据质量对于候选人之间的比较,是靠谱的。这样看,舆情趋势和候选人对比的情报挖掘,的确真实地反映了民意的消长和相对评价。下面是这次自动民调的 Top 10 数据来源(可惜没有“她”,我是说 wechat),还是最动态反映舆情的推特中文帖子占多数(其中 66% 简体,30% 繁体,4% 粤语)。

domains-5

看一下popular的帖子,居然小方的也在其列。倒也不怪,方在中文社煤还是有影响力的。

chuanpupopularposts

小方总结得不错啊,难得同意他:满嘴跑火车的川大叔是“谎言大王”。其实川普与其说是谎话连篇,不如说是他根本不care 或不屑去核对事实。就跟北京出租司机信口开河成为习惯一样,话说到这里,转一篇我的老友刚写的博文(论保守派该投票克林顿),quote:

川普说话不顾事实是众所周知的。只要他一开口,就忙坏了各种事实核查 fact check ……
更重要的是,川普不仅犯了大大小小众多的事实错误,而且对事实抱着强烈的轻蔑和鄙视。

总结一下这次民调的结果可以说,如果是华人投票,川普不仅是 lose 而是要死得很惨,很难看。(当然,不管华人与否,川普都没有啥胜算。)

timeline-comparison-12

这是 by days 的趋势对比,这种持续的舆情领先在大选前很难改变吧:

timeline-comparison-13

【更多美国大选舆情的自动调查还在进行整理中,stay tuned】

 

【相关】

【社煤挖掘:为什么要选ta而不是ta做总统?】

Big data mining shows clear social rating decline of Trump last month

【川普和希拉里的幽默竞赛】

【大数据舆情挖掘:希拉里川普最近一个月的形象消长】

论保守派该投票克林顿

【立委科普:自动民调】

【立委科普:舆情挖掘的背后】

【社媒挖掘:《品牌舆情图》的设计问题】

一切声称用机器学习做社会媒体舆情挖掘的系统,都值得怀疑

【关于舆情挖掘】

《朝华午拾》总目录

 

 

 

 

 

 

 

 

 

Big data mining shows clear social rating decline of Trump last month

Big data mining from last month’ social media shows clear decline of Trump in comparison with Clinton

aaa

Our automatic big data mining for public opinions and sentiments from social media speaks loud and clear: Tump’s social image sucks.

Look at last 30 days of social media on the Hillary and Trump’s social image and standing in our Brand Passion Index (BPI) comparison chart below:

brand-passion-index-8

Three points to note:
1 Trump has more than twice buzz than Hillary in terms of social media coverage (the size of the circles indicates the degree of mentions);
2. The intensity of sentiments from the general public of netters is more intense for Chump than for Clinton: the Y-axis shows the passion intensity
3. The social ratings and images of the two are both quite poor, but Trump is more criticized in social: the X-axis of Net Sentiment shows the index social sentiment ratings.  Both are under freezing point (meaning more negative comments than positive).

If we want to automatically investigate the trend of the past month and their social images’ ups and downs, we can have the data segmented into two or three segments.  Figure below shows the trends contrast of the first 15 days of social media data vs. the second 15 days of data in the 30-day period (up to 10/21/2016):

brand-passion-index-7

See, in the past month, with the presidential election debates and scandals getting attention, Trump’s media image significantly deteriorated, represented by the public opinion circles shifting from the right on the X-axis to the left side (for dislike or hate sentiments: the lighter circle represents data older than the darker circle).  His social rating was clearly better than Hillary to start with and ended up worse than that of Hillary.  At the same time, Hillary’s social media image has improved, the circle moves a bit from the left to right. Two candidates have always been below the freezing point, clearly shown in the figure, but just a month ago, Clinton was rated even lower than Trump in public opinions of the social media: it is not the people who like Trump that much, but the general public showed more dislike for Hillary for whatever reasons.

As seen, our BPI brand comparison chart attempts to visualize four-dimensional information:
1. net sentiment for social ratings on the X-axis;
2. the passion intensity of public sentiments on the Y-axis;
3. buzz circle size, representing mentions of soundbites;
4. The two circles of the same brands show the coarse-grained time dimension for general trends.

It is not very easy to represent 4 dimensions of analytics in a two-dimensional graph.  Hope the above attempt in our patented visualization efforts is insightful and not confusing.

If we are not happy with the divide-into-two strategy for one month of data to show the trends, how about cut them into three pieces?  Here is the Figure for .three circles in the time dimension.

brand-passion-index-6

We should have used different colors for the two political brands to make visualization a bit clearer.  Nevertheless, we see the trends for Clinton in her three circles of social media sentiments shifting from the lower left corner to the upper right in a zigzag path: getting better, then worse, and ended up with somewhere in between at this point (more exactly, up to the point of 10/21/2016). For the same 3 segments of data, Trump’s (brand) image started not bad, then went slightly better, and finally fell into the abyss.

The above is to use our own brand comparison chart (BPI) to decode the two US presidential candidates’ social images change and trends.  This analysis, entirely automated based on deep Natural Language Parsing technology, is supported by data points in a magnitude many times more than the traditional manual polls which are by nature severely restricted in data size and time response.

What are the sources of social media data for the above automated polling?  They are based on random social media sampling of big data, headed by the most dynamic source of Twitter, as shown below.

sources-5

sources-4

sources-3

This is a summary of the public opinions and sentiments:

%e5%b7%9d%e6%99%ae%e5%b8%8c%e6%8b%89%e9%87%8c

As seen, it is indeed BIG data: a month of random sampling of social media data involves the mentions of the candidates for nearly 200 million times, a total of up to 3,600+ billion impressions (potential eyeballs). Trump accounted for 70 percent of the buzz while Clinton only 30 percent.

The overall social rating during the period of 09/21/2016 through 10/21/2016, Trump’s net sentiment is minus 20%, and Clinton is minus 18%.  These measures show a rating much lower than that of most other VIP analysis we have done before using the same calculations.  Fairly nasty images, really.   And the big data trends show that Trump sucks most.

The following is some social media soundbites for Trump:

Bill Clinton disgraced the office with the very behavior you find appalling in…
In closing, yes, maybe Trump does suffer from a severe case of CWS.
Instead, in this alternate NY Times universe, Trump’s campaign was falling …
Russian media often praise Trump for his business acumen.
This letter is the reason why Trump is so popular
Trump won
I’m proud of Trump for taking a stand for what’s right.
Kudos to Trump for speaking THE TRUTH!
Trump won
I’m glad I’m too tired to write Trump/Putin fuckfic.
#trump won
Trump is the reason Trump will lose this election.
Trump is blamed for inciting violence.
Breaking that system was the reason people wanted Trump.
I hate Donald Trump for ruining my party.
>>32201754 Trump is literally blamed by Clinton supporters for being too friendly with Russia.
Another heated moment came when Trump delivered an aside in reponse to …
@dka_gannongal I think Donald Trump is a hoax created by the Chinese….
Skeptical_Inquirer The drawing makes Trump look too normal.
I’m proud of Donald Trump for answering that honestly!
Donald grossing me out with his mouth features @smerconish …
Controlling his sniffles seems to have left Trump extraordinarily exhausted
Trump all the way people trump trump trump
Trump wins
Think that posting crap on BB is making Trump look ridiculous.
I was proud of Trump for making America great again tonight.
MIL is FURIOUS at Trump for betraying her!
@realdonaldTrump Trump Cartel Trump Cartel America is already great, thanks to President Obama.
Kudos to Mr Trump for providing the jobs!!
The main reason to vote for Trump is JOBS!
Yes donal trump has angered many of us with his WORDS.
Trump pissed off a lot of Canadians with his wall comments.
Losing this election will make Trump the biggest loser the world has ever seen.
Billy Bush’s career is merely collateral damage caused by Trump’s wrenching ..
So blame Donald for opening that door.
The most important reason I am voting for Trump is Clinton is a crook.
Trump has been criticized for being overly complimentary of Putin.
Kudos to Trump for reaching out to Latinos with some Spanish.
Those statements make Trump’s latest moment even creepier.
I’m mad at FBN for parroting the anti-Trump talking points.
Kudos to Trump for ignoring Barack today @realDonaldTrump
Trump has been criticized for being overly complimentary of Putin.
OT How Donald Trump’s rhetoric has turned his precious brand toxic via …
It’s these kinds of remarks that make Trump supporters look like incredible …
Trump is blamed for inciting ethnic tensions.
Trump is the only reason the GOP is competitive in this race.
Its why Republicans are furious at Trump for saying the voting process is rigged.
Billy Bush’s career is merely collateral damage caused by Trump’s wrenching ..
Donald Trump is the dumbest, worst presidential candidate your country …
I am so disappointed in Colby Keller for supporting Trump.
Billy Bush’s career is merely collateral damage caused by Trump’s wrenching..
In swing states, Trump continues to struggle.
Trump wins
Co-host Jedediah Bila agreed, saying that the move makes Trump look desperate.
Trump wins
“Trump attacks Clinton for being bisexual!”
TRUMP win
Pence also praised Trump for apologizing following the tape’s disclosure.
In swing states, Trump continues to struggle.
the reason Trump is so dangerous to the establishment is he is unapologetical..

Here are some public social media soundbites for Clinton in the same period:

Hillary deserves worse than jail.
Congratulations to Hillary & her campaign staff for wining three Presidential ..
I HATE @chicanochamberofcommerce FOR INTRODUCING THAT HILLARY …
As it turns out, Hillary creeped out a number of people with her grin.
Hillary trumped Trump
Trump won!  Hillary lost
Hillary violated the Special Access Program (SAP) for disclosing about the …
I trust Flint water more than Hillary
Hillary continued to baffle us with her bovine feces.
NEUROLOGISTS HATE HILLARY FOR USING THIS TRADE SECRET DRUG!!!!…
CONGRATULATIONS TO HILLARY CLINTON FOR WINNING THE PRESIDENCY
Supreme Court: Hillary is our only choice for keeping LGBT rights.
kudos to hillary for remaining sane, I’d have killed him by now
How is he blaming Hillary for sexually assaulting women. He’s such a shithead
The only reason I’m voting for Hillary is that Donald is the only other choice
Hillary creeps me out with that weird smirk.
Hillary is annoying asf with all of her laughing
I credit Hillary for the Cubs waking up
When you listen to Hillary talk it is really stupid
On the other hand, Hillary Clinton has a thorough knowledge by virtue of …
Americans deserve better than Hillary
Certain family members are also upset with me for speaking out against …
Hillary is hated by all her security detail for being so abusive
Hillary beat trump
The only reason to vote for Hillary is she’s a woman.
Certain family members are also upset with me for speaking out against ….
I am glad you seem to be against Hillary as well Joe Pepe.
Hillary scares me with her acions.
Unfortunately Wikileaks is the monster created by Hillary & democrats.
I’m just glad you’re down with evil Hillary.
Hillary was not mad at Bill for what he did.  She was mad he got caught.  ……
These stories are falling apart like Hillary on 9/11
Iam so glad he is finally admitting this about Hillary Clinton.
Why hate a man for doing nothing like Hillary Clinton
Hillary molested me with a cigar while Bill watched.
You are upset with Hillary for doing the same as all her predecessors.
I feel like Hillary Clinton is God’s punishment on America for its sins.
Trumps beats Hillary
You seem so proud of Hillary for laughing at rape victims.
Of course Putin is going to hate Hillary for publicly announcing false …
Russia is pissed off at Hillary for blaming the for wikileaks!
Hillary will not win.  Good faith is stronger than evil.  Trump wins??
I am proud of Hillary for standing up for what is good in the USA.
Hillarys plans are worse than Obama
Hillary is the nightmare “the people” have created.
Funny how the Hillary supporters are trashing Trump for saying the same …
???????????? I am so proud of the USA for making Hillary Clinton president.
Hillary, you’re a hoax created by the Chinese
Trump trumps Hillary
During the debate, Trump praised Hillary for having the will to fight.
Trump is better person than Hillary
Donald TRUMPED Hillary
Kudos to Hillary for her accomplishments.
He also praised Hillary for handling the situation with dignity.
During the debate, Trump praised Hillary for having the will to fight.
People like Hillary in senate is the reason this country is going downhill.
Hillary did worse than expectations.
Trump will prosecute Hillary for her crimes, TRUMP will!
Have to praise Hillary for keeping her focus.
a landslide victory for Hillary will restore confidence in American democracy ..
I was so proud of Hillary tonight for acting like a tough, independent woman.
I dislike Hillary Clinton, as I think she is a corrupt, corporate shill.
Hillary did worse than Timmy Kaine
Im so glad he finally brought Benghazi against Hillary
Hillary, thank you for confirmation that the Wikileaks documents are authentic
Supreme Court justices is the only reason why I’d vote for Hillary.
Massive kudos to Hillary for keeping her cool with that beast behind her.
Congrats to Hillary for actually answering the questions. She’s spot on. #debate

 

[Related]

Social media mining: Did Trump’s Gettysburg speech enable the support rate to soar as claimed?

Big data mining shows clear social rating decline of Trump last month

Clinton, 5 years ago. How time flies …

Automated Suevey

【大数据舆情挖掘:希拉里川普最近一个月的形象消长】

aaa

大数据舆情挖掘,看图说话。
先看近一个月来在社会媒体上的希拉里和川普的品牌形象对比图:

brand-passion-index-8

看点三:
1 川普的 buzz 大过 希拉里一倍多,川普是话题中心(圈的大小表明热议度)
2. 普罗对川普比对希拉里,情绪更趋激烈:表现在 Y 轴的 passion intensity 上
3. 两人总体都不讨人喜欢,川普更加让人厌恶,表现在 x 轴上的 Net Sentiment(也就是褒贬对比的度量)。两人都在冰点之下,社会媒体的形象不佳。

如果我们要自动调查过去一个月时间的趋向和形象消长,可以考虑把数据分割为两段或三段来看此消彼长,先一分为二来看图:

brand-passion-index-7

看到了吧,过去一个月,随着总统大选辩论和丑闻的揭示和宣传,川普的媒体形象显著恶化,表现在舆情圈圈从右(x轴上的右是评价度高 love like,左边是评价度低 hate dislike)向左的位移。本来评价度clearly比希拉里要好,终于比希拉里差了。同时,希拉里的社会媒体形象有所改善,圈圈在从左向右位移。两个人始终都是冰点以下,吐槽多于赞美,但是就在一个月前,还是喜妈更不受待见:不是民众更喜欢老川,而是普罗更厌恶喜妈。

这个品牌对比图示表达了四维信息:
1. net sentiment 评价度 x 轴
2. passion intensity 舆情烈度 y 轴
3. buzz 圈圈的大小,是热议度
4. 一分为二的两个圈是时间的粗线条切割的维度

在二维的图纸上,要表达四维的信息,的确不是很容易。

要是嫌第四维时间太粗线条,咱们一分为三看看:

brand-passion-index-6

三个圈,浓度的深浅表达的是时间的远近。当短短的一个月的时间,被一分为三的时候,我们看到了什么趋向呢?请注意颜色的深浅,对应的是时间的远近。我们看到,喜妈的三个圈圈是左下角到右上(还是visualization设计不到家,不同品牌应该用不同的颜色区分才好)。原来喜妈的评价是先好,后坏,最后回到中间。而老川在同一个时间点,是先中,后略好,最后跌入深渊。

以上是利用我们自创的品牌对比图(有美国专利的)来看候选人的形象消长。

社会媒体数据的来源呢?Twitter 为主:

sources-5

sources-4

sources-3

这是一个月来的舆情总结:

%e5%b7%9d%e6%99%ae%e5%b8%8c%e6%8b%89%e9%87%8c

的确是大数据了,一个月的随机的社会媒体数据样本里面,两人的 mentions 就有近两亿,眼球数共计高达3万6千亿。川普占7成,喜妈才三成。川普跟冰冰类似,都是话题之王。

总体社会评价,川普零下20%,喜妈零下18%。

下面是有关川普的社煤数据选摘:

Bill Clinton disgraced the office with the very behavior you find appalling in Trump.
In closing, yes, maybe Trump does suffer from a severe case of CWS.
Instead, in this alternate NY Times universe, Trump’s campaign was falling apart.
Russian media often praise Trump for his business acumen.
This letter is the reason why Trump is so popular
Trump won
I’m proud of Trump for taking a stand for what’s right.
Kudos to Trump for speaking THE TRUTH!
Trump won
I’m glad I’m too tired to write Trump/Putin fuckfic.
#trump won
Trump is the reason Trump will lose this election.
Trump is blamed for inciting violence.
Breaking that system was the reason people wanted Trump.
I hate Donald Trump for ruining my party.
>>32201754 Trump is literally blamed by Clinton supporters for being too friendly with Russia.
Another heated moment came when Trump delivered an aside in reponse to a Clinton one-liner.
@dka_gannongal I think Donald Trump is a hoax created by the Chinese….
Skeptical_Inquirer The drawing makes Trump look too normal.
I’m proud of Donald Trump for answering that honestly!
Donald grossing me out with his mouth features @smerconish @realdonaldtrump
Controlling his sniffles seems to have left Trump extraordinarily exhausted
Trump all the way people trump trump trump
Trump wins
Think that posting crap on BB is making Trump look ridiculous.
I was proud of Trump for making America great again tonight.
MIL is FURIOUS at Trump for betraying her!
@realdonaldTrump Trump Cartel Trump Cartel America is already great, thanks to President Obama.
Kudos to Mr Trump for providing the jobs!!
The main reason to vote for Trump is JOBS!
Yes donal trump has angered many of us with his WORDS.
Trump pissed off a lot of Canadians with his wall comments.
Losing this election will make Trump the biggest loser the world has ever seen.
Billy Bush’s career is merely collateral damage caused by Trump’s wrenching migration.
So blame Donald for opening that door.
The most important reason I am voting for Trump is Clinton is a crook.
Trump has been criticized for being overly complimentary of Putin.
Kudos to Trump for reaching out to Latinos with some Spanish.
Those statements make Trump’s latest moment even creepier.
I’m mad at FBN for parroting the anti-Trump talking points.
Kudos to Trump for ignoring Barack today @realDonaldTrump
Trump has been criticized for being overly complimentary of Putin.
OT How Donald Trump’s rhetoric has turned his precious brand toxic via The Independent.
It’s these kinds of remarks that make Trump supporters look like incredible idiots.
Trump is blamed for inciting ethnic tensions.
Trump is the only reason the GOP is competitive in this race.
Its why Republicans are furious at Trump for saying the voting process is rigged.
Billy Bush’s career is merely collateral damage caused by Trump’s wrenching migration.
Donald Trump is the dumbest, worst presidential candidate your country has EVER produced.
I am so disappointed in Colby Keller for supporting Trump.
Billy Bush’s career is merely collateral damage caused by Trump’s wrenching migration.
In swing states, Trump continues to struggle.
Trump wins
Co-host Jedediah Bila agreed, saying that the move makes Trump look desperate.
Trump wins
“Trump attacks Clinton for being bisexual!”
TRUMP win
Pence also praised Trump for apologizing following the tape’s disclosure.
In swing states, Trump continues to struggle.
the reason Trump is so dangerous to the establishment is he is unapologetically alpha.

关于希拉里的社会媒体样本数据摘选:

Hillary deserves worse than jail.
Congratulations to Hillary & her campaign staff for wining three Presidential debates.
I HATE @chicanochamberofcommerce FOR INTRODUCING THAT HILLARY GIF INTO MY LIFE
As it turns out, Hillary creeped out a number of people with her grin.
Hillary trumped Trump
Trump won!  Hillary lost
Hillary violated the Special Access Program (SAP) for disclosing about the nuclear weapons!!
I trust Flint water more than Hillary
Hillary continued to baffle us with her bovine feces.
NEUROLOGISTS HATE HILLARY FOR USING THIS TRADE SECRET DRUG!!!!…
CONGRATULATIONS TO HILLARY CLINTON FOR WINNING THE PRESIDENCY
Supreme Court: Hillary is our only choice for keeping LGBT rights.
kudos to hillary for remaining sane, I’d have killed him by now
How is he blaming Hillary for sexually assaulting women. He’s such a shithead
The only reason I’m voting for Hillary is that Donald is the only other choice
Hillary creeps me out with that weird smirk.
Hillary is annoying asf with all of her laughing
I credit Hillary for the Cubs waking up
When you listen to Hillary talk it is really stupid
On the other hand, Hillary Clinton has a thorough knowledge by virtue of her tenure as Secretary of State.
Americans deserve better than Hillary
Certain family members are also upset with me for speaking out against Hillary.
Hillary is hated by all her security detail for being so abusive
Hillary beat trump
The only reason to vote for Hillary is she’s a woman.
Certain family members are also upset with me for speaking out against Hillary.
I am glad you seem to be against Hillary as well Joe Pepe.
Hillary scares me with her acions.
Unfortunately Wikileaks is the monster created by Hillary & democrats.
I’m just glad you’re down with evil Hillary.
Hillary was not mad at Bill for what he did.  She was mad he got caught.  Just like she is not ashamed of what she did she is angry she got caught.
These stories are falling apart like Hillary on 9/11
Iam so glad he is finally admitting this about Hillary Clinton.
Why hate a man for doing nothing like Hillary Clinton
Hillary molested me with a cigar while Bill watched.
You are upset with Hillary for doing the same as all her predecessors.
I feel like Hillary Clinton is God’s punishment on America for its sins.
Trumps beats Hillary
You seem so proud of Hillary for laughing at rape victims.
Of course Putin is going to hate Hillary for publicly announcing false accusations.
Russia is pissed off at Hillary for blaming the for wikileaks!
Hillary will not win.  Good faith is stronger than evil.  Trump wins??
I am proud of Hillary for standing up for what is good in the USA.
Hillarys plans are worse than Obama
Hillary is the nightmare “the people” have created.
Funny how the Hillary supporters are trashing Trump for saying the same thing.
???????????? I am so proud of the USA for making Hillary Clinton president.
Hillary, you’re a hoax created by the Chinese
Trump trumps Hillary
During the debate, Trump praised Hillary for having the will to fight.
Trump is better person than Hillary
Donald TRUMPED Hillary
Kudos to Hillary for her accomplishments.
He also praised Hillary for handling the situation with dignity.
During the debate, Trump praised Hillary for having the will to fight.
People like Hillary in senate is the reason this country is going downhill.
Hillary did worse than expectations.
Trump will prosecute Hillary for her crimes, TRUMP will!
Have to praise Hillary for keeping her focus.
a landslide victory for Hillary will restore confidence in American democracy vindicated
I was so proud of Hillary tonight for acting like a tough, independent woman.
I dislike Hillary Clinton, as I think she is a corrupt, corporate shill.
Hillary did worse than Timmy Kaine
Im so glad he finally brought Benghazi against Hillary
Hillary, thank you for confirmation that the Wikileaks documents are authentic and you did that tonight when you accused the Russians of hacking your servers!  We the people deserve better than you!
Supreme Court justices is the only reason why I’d vote for Hillary.
Massive kudos to Hillary for keeping her cool with that beast behind her.
Congrats to Hillary for actually answering the questions. She’s spot on. #debate

 

【相关】

【关于舆情挖掘】

《朝华午拾》总目录

Who we are. Not an ad, but a snapshot.

NetBase

WHO WE ARE

n1

EMPOWERING GLOBAL BUSINESSES WITH SOCIAL INSIGHTS

We are uniquely positioned to help global businesses create real business value from the unprecedented level of growth opportunities presented each day by social media. We have the industry’s fastest and most accurate social analytics platform, strong partnerships with companies like Twitter, DataSift, and Tumblr, and award-winning patented language technology.

We empower brands and agencies to make the smartest business decisions grounded on the deepest and most reliable consumer insights from social. We’ve grown 300 percent year-over-year and excited to see revenue grow by 4,000% since the second quarter of 2012.

RECENT ACCOLADES

We were recently named a top rated social media management platform by software users on TrustRadius and a market leader by G2 Crowd.

n2

“NetBase is one of the strongest global social listening and analytics tools in the market. Their new interface makes customized dashboard creation a breeze.”

– Omri Duek, Coca-Cola

“Data reporting is both broad and detailed, with the ability to drill down from annual data to hourly data. NetBase allows us to have a pulse on the marketplace in just a few minutes.”

– Susie Thomas, VP, Palisades Media Group

“We started with a gen one solution, but then found that we needed to move to a tool with a better accuracy that could support digital strategy and insights research. NetBase satisfied all our needs.”

– Jared Degnan, Director of Digital Strategy

“As one of the first brands to test NetBase Audience 3D for our Mobile App launch, we’ve found that we could engage with our consumers on a deeper, more human level that further drives them to be brand champions.”

– Mihir Minawala, Manager of Social, Industry & Competitive Intelligence, Taco Bell

OUR CUSTOMERS

We work with executives from forward-looking agencies and leading brands across all verticals in over 99 countries. Our customers use NetBase for real-time consumer insights across the organization, from brand and digital marketing, public relations, product management to customer care.

KEY MILESTONES

  • March 2003
    Founded by Michael Osofsky at MIT. Later joined by Wei Li, Chief NetBase Scientist
  • July 2009
    P&G, Coca-Cola and Kraft signed as first customers of NetBase
  • January 2014
    Named Best-in-Class By Consumer Goods Technology
  • April 2014
    Launched Brand Live Pulse, the first real-time view of brands’ social movements
  • May 2014
    Celebrated 10 years with 500% customer growth in 3 years
  • January 2015
    AdAge Names 5 NetBase Customers to the Agency A-List
  • March 2015
    Introduced Audience 3D, the first ever 3D view of audiences
  • April 2015
    Raised $33 MM in Series E Round
  • November 2015
    Named Market Leader by G2 Crowd. Earned Top Ratings by Trust Radius

n3

What inspired you to join NetBase?

It was exciting to build the technology that could quickly surface meaningful customer insights at scale. For example, what used to take a day to run a simple analysis now takes just a second. Our platform now analyzes data in “Google time”, yet the depth and breadth of our analysis is exponentially deeper and larger than what you’ll ever get from a Google search.

What are you most proud of at NetBase?

I’m especially proud that we have the industry’s most accurate, deepest, fastest, and more granular text analysis technology. This enables us to gives our customers very actionable insights, unlike other platforms that offer broad sentiment analysis and general trending topics. Plus, NetBase reads 42 languages. Other platforms don’t even come close. We are customer-centric. Our platform truly helps customers quickly identify their priorities and next steps. This is what sets us apart.

What is the next frontier for NetBase?

With the exploding growth of social and mobile data and new social networks emerging, we’ll be working on connecting all these data points to help our customers get even more out of social data. As Chief Scientist, I’m more excited than ever to develop a “recipe” that can work with the world’s languages and further expand our language offerings.

WE’RE GLOBAL: 42 LANGUAGES, 99+ COUNTRIES, 8 OFFICES

NetBase Solutions, Inc  © 2016

Overview of Natural Language Processing

Dr. Wei Li’s English Blog on NLP

【离皇冠上的明珠只有一步之遥的感觉】

1471802218_457583

parsing 是最好的游戏,而且实用。

据说好玩的游戏都没用,有实用价值的东西做不成游戏。但是,对于AI人员,parsing 却是这么一个最好玩但也最有用的游戏。纵情于此,乐得其所,死得其所也。

禹:
李老师parser有没有觉得太烧脑呢?
做parser少了个做字。感觉上先是一个比较优雅的规则集,然后发现规则之外又那么多例外,然后开始调规则,解决冲突,然后’整理规则的事情还得亲力亲为,做好几年感觉会不会很烦?

我:
不烦 特别好玩。能玩AI公认的世界级人类难题且登顶在望,何烦之有?
烦的是好做的语言 做着做着 没啥可做了 那才叫烦。英语就有点做烦了。做中文不烦 还有不少土地没有归顺 夺取一个城池或山头 就如将军打仗赢了一个战役似的 特别有满足感。

梁:
收复领地?

我:

【打过长江去,解放全中国!】

parsing 是最好的游戏。先撒一个default的网,尽量楼。其实不能算“优雅的规则集”,土八路的战略,谈不上优雅。倒有点像原始积累期的跑马,搂到越多越好。然后才开始 lexicalist 的精度攻坚,这才是愚公移山。在 default 与 lexicalist 的策略之间,建立动态通信管道,一盘棋就下活了。
譬如说吧,汉语离合词,就是一大战役。量词搭配,是中小战役。ABAB、AABB等重叠式是阵地战。定语从句界限不好缠,算是大战役。远距离填坑,反而不算大战役。因为远距离填坑在句法基本到位之后,已经不再是远距离了,而且填的逻辑SVO的坑,大多要语义相谐,变得很琐碎,但其实难度不大。(这就是白老师说的,要让大数据训练自动代替人工的语义中间件的琐碎工作。而且这个大数据是不需要标注的。白老师的RNN宏图不知道啥时开工,或已经开工?)

parsing 是最好的游戏。一方面它其实不是愚公面对的似乎永无尽头的大山,虽然这个 monster 看上去还是挺吓人的。但大面上看,结构是可以见底的,细节可以永远纠缠下去。另一方面,它又是公认的世界级人类难题。不少人说,自然语言理解(NLU)是人工智能(AI)的终极难题,而 deep parsing 是公认的通向NLU的必由之路,其重要性可比陈景润为攀登哥德巴赫猜想之巅所做出的1+1=2.  我们这代人不会忘记30多年前迎来“科学的春天”时除迟先生的如花妙笔:“自然科学的皇后是数学。数学的皇冠是数论。哥德巴赫猜想,则是皇冠上的明珠。…… 现在,离开皇冠上的明珠,只有一步之遥了。”(作为毛时代最后的知青,笔者是坐着拖拉机在颠簸的山路回县城的路上读到徐迟的长篇报告文学作品【哥德巴赫猜想】的,一口气读完,头晕眼花却兴奋不已。)

不世出的林彪都会悲观主义,问红旗到底要打到多久。但做 deep parsing,现在就可以明确地说,红旗登顶在望,短则一年,长则三五年而已。登顶可以定义为 open domain 正规文体达到 95% 左右的精度广度(f-score, near-human performance)。换句话说,就是结构分析的水平已经超过一般人,仅稍逊色于语言学家。譬如,英语我们五六年前就登顶了

最有意义的还是因为 parsing 的确有用,说它是自然语言应用核武器毫不为过。有它没它,做起事来就大不一样。shallow parsing 可以以一当十,到了 deep parsing,就是以一当百+了。换句话说,这是一个已经成熟(90+精度可以认为是成熟了)、潜力几乎无限的技术。

刘:
@wei 对parsing的执着令人钦佩

我:
多谢鼓励。parsing 最终落地,不在技术的三五个百分点的差距,而在有没有一个好的产品经理,既懂市场和客户,也欣赏和理解技术的潜力。

刘:
任何技术都是这样的

我:
量变引起质变。90以后,四五个百分点的差别,也许对产品和客户没有太大的影响。但是10多个百分点就大不一样了。譬如,社会媒体 open domain 舆情分析的精度,我们利用 deep parsing support 比对手利用机器学习去做,要高出近20个百分点。结果就天差地别。虽然做出来的报表可以一样花哨,但是真要试图利用舆情做具体分析并支持决策,这样的差距是糊弄不过去的。大数据的统计性过滤可以容忍一定的错误,但不能容忍才六七十精度的系统。

当然也有客户本来就是做报表赶时髦,而不是利用 insights 帮助调整 marketing 的策略或作为决策的依据,对这类客户,精度和质量不如产品好用、fancy、便宜更能打动他们。而且这类客户目前还不在少数。这时候单单有过硬的技术,也还是使不上劲儿。这实际上也是市场还不够成熟的一个表现。拥抱大数据成为潮流后,市场的消化、识别和运用能力还没跟上来。从这个角度看市场,北美的市场成熟度比较东土,明显成熟多了。

 

【相关】

泥沙龙笔记:parsing 是引擎的核武器,再论NLP与搜索

泥沙龙笔记:从 sparse data 再论parsing乃是NLP应用的核武器

It is untrue that Google SyntaxNet is the “world’s most accurate parser”

【立委科普:NLP核武器的奥秘】

徐迟:【哥德巴赫猜想】

《朝华点滴:插队的日子(一)》

关于 parsing

【关于中文NLP】

【置顶:立委NLP博文一览】

《朝华午拾》总目录

The mainstream sentiment approach simply breaks in front of social media

I have articulated this point in various previous posts or blogs before, but the world is so dominated by the mainstream that it does not seem to carry.  So let me make it simple to be understood:

The sentiment classification approach based on bag of words (BOW) model, so far the dominant approach in the mainstream for sentiment analysis, simply breaks in front of social media.  The major reason is simple: the social media posts are full of short messages which do not have the “keyword density” required by a classifier to make the proper sentiment decision.   Larger training sets cannot help this fundamental defect of the methodology.  The precision ceiling for this line of work in real-life social media is found to be 60%, far behind the widely acknowledged precision minimum 80% for a usable extraction system.  Trusting a machine learning classifier to perform social media sentiment mining is not much better than flipping a coin.

So let us get straight.  From now on, whoever claims the use of machine learning for social media mining of public opinions and sentiments is likely to be a trap (unless it is verified to have involved parsing of linguistic structures or patterns, which so far has never been heard of in practical systems based on machine learning).  Fancy visualizations may make the results of the mainstream approach look real and attractive but they are just not trustable at all.

Related Posts:

Why deep parsing rules instead of deep learning model for sentiment analysis?
Pros and Cons of Two Approaches: Machine Learning and Grammar Engineering
Coarse-grained vs. fine-grained sentiment analysis
一切声称用机器学习做社会媒体舆情挖掘的系统,都值得怀疑
【立委科普:基于关键词的舆情分类系统面临挑战】

【语义计算沙龙:sentiment 中的讽刺和正话反说】

w:
看见前面的“降温”,这降温是何极性?
“本来就防寒措施准备不足,这不现在又要降温了。”
“他这几天高烧不退,还好刚才开始降温了。”
还有“降级”
“他这学期实在太差,学校对他实行了降级处理。”
“由防恐工作己经卓有成效,上面终于把防恐等级降级了。”
看这“随风倒”的词,在做情感分析,只看词的话也不大可靠。
我:
大小 多少 高低 长短 增减 之类 自身没有褒贬 根据它所修饰者而定
w:
这些中性词一看很清楚。降级这个偏向性应该很明显。
我:
那是。
降 中性。
w:

我:
级 级别 职称 等级 这些属于隐性正面词 在语言中被激发 被 increase、start or decrease、suspend 这类概念的词所激发。
w:
李老师研究是颇有心得啊!那存不存在褒义词贬义用,贬义词褒义用。
我:
当然存在 譬如讽刺 挖苦 自嘲 和 讥笑
w:
是的
我:
譬如 “谢谢你的完美误导”:“谢谢” 与 “完美” 的褒义 在“误导”前 转为讽刺 因此是负面
w:
呵呵! 那机器拿捏起来,就比较困难了。
我:
再如
“居然拿狐狸肉冒充牛肉上架 沃尔玛 你真行”
你真行 从褒奖转为讽刺贬义。
w:
例子,李老师也是随手拈来啊,这个”真行” -都是用坏了。这些应该是属于超一般文本的自然语言处理。觉得,在NLP 特别棘手这种讽刺,隐喻,借代,拟人之类的
我:
我过去五年就是陷在 sentiment 泥淖呢。
w:
好不容易搭配个语义模板出来,总那么多非法的都是合法的而且是高水平的修辞手法
这让机器莫名其妙,莫衷一是
我:
上面这样的正好反说有迹可循,有套路,小心一点,还是可以抓住的。做过 sentiment 后 事实抽取为基础的知识图谱就是小菜了。
白:
sentiment也是有结构的,仅仅正负面标签远远不够。
w:
同意白老师。独立标签
独立标签也许就如李老师之前所说的关键词
白:
“坑挖深了”有歧义,和不同预期有关。
原来挖浅了,现在按照要求深挖,可以交差了。这是一个意思。
没预期挖那么深,一不留神用力过猛,挖深了,是“过深”的意思。
【相关】

【关于舆情挖掘】

《朝华午拾》总目录

【关于舆情挖掘】

【喋喋不休论大数据(立委博文汇总)】

【新智元笔记:再谈舆情】

舆情挖掘系统独立验证的意义

【社煤挖掘:雷同学之死】

《利用大数据高科技,实时监测美国总统大选舆情变化》

世人皆错nlp不错,民调错大数据也不会错

社媒大数据的困境:微信的风行导致舆情的碎片化

从微信的用户体验谈大数据挖掘的客户情报

社媒挖掘:社会媒体疯传柴静调查,毁誉参半,争议趋于情绪化

奥巴马赢了昨晚辩论吗?舆情自动检测告诉你

全球社交媒体热议阿里巴巴上市

到底社媒曲线与股市曲线有没有、有多少相关度?

再谈舆情与股市的相关性

【『科学』预测:A-股 看好】

舆情挖掘用于股市房市预测靠谱么?

大数据帮助决策实例:《走进“大数据”——洗衣机寻购记》

【社媒挖掘:外来快餐店风光不再】

【社媒挖掘:中国手机市场仍处于战国争雄的阶段】

世界杯是全世界的热点,纵不懂也有义务挖掘一哈

【大数据挖掘:方崔大战一年回顾】(更正版)

【大数据挖掘:转基因一年回顾】

【大数据挖掘:“苦逼”小崔2013年5-7月为什么跌入谷底?】

【大数据挖掘:转基因中文网络的自动民调,东风压倒西风?】

【大数据挖掘:转基因英文网络的自动民调和分析】

只认数据不认人:IRT 的鼓噪左右美国民情了么?

继续转基因的大数据挖掘:谁在说话?发自何处?能代表美国人民么

关于转基因及其社会媒体大数据挖掘的种种问题

【美国网民怎么看转基因:英文社交媒体大数据调查告诉你】

【社媒挖掘:必胜客是七夕节情侣聚餐的首选之地?】

【社媒挖掘:大数据时代的危机管理】

测试粤语舆情挖掘:拿娱乐界名人阿娇和陈冠希开刀

【社媒挖掘:不朽邓丽君】

【社媒挖掘:社会媒体眼中的李开复老师】

【社媒挖掘:糟糕透顶的方韩社会形象】

社媒挖掘:关于狗肉的争议

社媒挖掘:央视的老毕

社媒挖掘:老毕私下辱毛事件再挖掘

大数据淹没下的冰美人(之一)

大数据淹没下的冰美人(之二)

大数据淹没下的冰美人(之三): 喜欢的理由

大数据淹没下的冰美人(之四): 流言蜚语篇(慎入)

大数据淹没下的冰美人(之五): 星光灿烂谁为最?

【社媒挖掘:成都暴打事件中的男司机和女司机】

【社媒挖掘:社会媒体眼中的陳水扁】

【社媒挖掘:社会媒体眼中的李登輝】

【社媒挖掘:馬英九施政一年來輿情晴雨表】

【社媒挖掘:臺灣政壇輿情圖】

【社媒挖掘:社会媒体眼中的臺灣綠營大佬】

舆情挖掘:九合一國民黨慘敗 馬英九時代行將結束?

社会媒体舆情自动分析:马英九 vs 陈水扁

社媒挖掘:争议人物方博士被逐,提升了其网路形象

方韩大战高频情绪性词的词频分析

方韩大战的舆情自动分析:小方的评价比韩少差太多了

社媒挖掘:苹果CEO库克公开承认同志身份,媒体反应相当正面

苹果智能手表会是可穿戴设备的革命么?

全球社交媒体热议苹果推出 iPhone 6

互联网盛世英雄马云的媒体形象

革命革到自身头上,给咱“科学网”也挖掘一下形象

两年来中国红十字会的社会媒体形象调查

自动民调Walmart,挖掘发现跨国公司在中国的日子不好过

【社媒挖掘:“剩女”问题】

【舆情挖掘:2013央视春晚播后】

【舆情挖掘:年三十挖一挖央视春晚】

新浪微博下周要大跌?舆情指数不看好,负面评价太多(疑似虚惊)

【大数据挖掘:微信(WeChat)】

【大数据解读:方崔大战对转基因形象的影响】

【微博自动民调:薄熙来、薛蛮子和李天一】

【社媒挖掘:第一夫人光彩夺目赞誉有加】

Chinese First Lady in Social Media

Social media mining on credit industry in China

Sina Weibo IPO and its automatic real time monitoring

Social media mining: Teens and Issues

立委元宵节大数据科技访谈土豆视频上网

【大数据挖掘:中国红十字会的社会媒体形象】

【社媒挖掘:社会媒体眼中的财政悬崖】

【社媒挖掘:美国的枪支管制任重道远】

【舆情挖掘:房市总体看好】

【社媒挖掘:社会媒体眼中的米拉先生】

【社会媒体:现代婚姻推背图】

【社会媒体:现代爱情推背图】

【科学技术之云】

新鲜出炉:2012 热点话题五大盘点之五【小方vs韩2】

【凡事不决问 social:切糕是神马?】

Social media mining: 2013 vs. 2012

社会媒体测试知名品牌百度,有惊人发现

尝试揭秘百度的“哪里有小姐”: 小姐年年讲、月月讲、天天讲?

舆情自动分析表明,谷歌的社会评价度高出百度一倍

圣诞社媒印象: 简体世界狂欢,繁體世界分享

WordClouds: Season’s sentiments, pros & cons of Xmas

新鲜出炉:2012 热点话题五大盘点之一【吊丝】

新鲜出炉:2012 热点的社会媒体五大盘点之二【林书豪】

新鲜出炉:2012 热点话题五大盘点之三【舌尖上的中国】

新鲜出炉:2012 热点话题五大盘点之四【三星vs苹果】

社会媒体比烂,但国骂隐含舆情

肮脏语言研究:英语篇

肮脏语言研究:汉语篇(18岁以下勿入)

新年新打算:【社媒挖掘】专栏开张大吉