Ted Talks
环境：Python2.7 Anaconda Jupyter Notebook
数据集: https://www.kaggle.com/rounakbanik/ted-talks
导入相应的库
%matplotlib inline
import pandas as pd
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
import seaborn as sns #matplotlib的默认作图风格就会被覆盖成seaborn的格式
import json
from pandas.io.json import json_normalize
from wordcloud import WordCloud, STOPWORDS #词云
df = pd.read_csv('ted_main.csv')
df.colums #数据集的首行表头
Index([u'comments', u'description', u'duration', u'event', u'film_date',
u'languages', u'main_speaker', u'name', u'num_speaker',
u'published_date', u'ratings', u'related_talks', u'speaker_occupation',
u'tags', u'title', u'url', u'views'],
dtype='object')
#调整表头顺序
df = df[['name', 'title', 'description', 'main_speaker', 'speaker_occupation', 'num_speaker', 'duration', 'event', 'film_date', 'published_date', 'comments', 'tags', 'languages', 'ratings', 'related_talks', 'url', 'views']]
Features Available
- name: The official name of the TED Talk. Includes the title and the speaker.
- title: The title of the talk
- description: A blurb of what the talk is about.
- main_speaker: The first named speaker of the talk.
- speaker_occupation: The occupation of the main speaker.
- num_speaker: The number of speakers in the talk.
- duration: The duration of the talk in seconds.
- event: The TED/TEDx event where the talk took place.
- film_date: The Unix timestamp of the filming.
- published_date: The Unix timestamp for the publication of the talk on TED.com
- comments: The number of first level comments made on the talk.
- tags: The themes associated with the talk.
- languages: The number of languages in which the talk is available.
- ratings: A stringified dictionary of the various ratings given to the talk (inspiring, fascinating, jaw dropping, etc.)
- related_talks: A list of dictionaries of recommended talks to watch next.
- url: The URL of the talk.
- views: The number of views on the talk.
image.png
convert the Unix timestamps into a human readable format
import datetime
df['film_date'] = df['film_date'].apply(lambda x: datetime.datetime.fromtimestamp( int(x)).strftime('%d-%m-%Y'))
df['published_date'] = df['published_date'].apply(lambda x: datetime.datetime.fromtimestamp( int(x)).strftime('%d-%m-%Y'))
image.png
image.png
#根据views量排序前15行数据
pop_talks = df[['title','main_speaker','views','film_date']].sort_values('views',ascending=False)[:15]
pop_talks
image.png
<<Do Schools Kill Creativity>> 点击量最高4722w的观看记录
#切分main_speaker的前三个字母,新增一列abbr数据
pop_talks["abbr"] = pop_talks['main_speaker'].apply(lambda x: x[:3])
pop_talks.head()
image.png
sns.set_style("whitegrid")
plt.figure(figsize=(10,6))
sns.barplot(x='abbr',y='views',data=pop_talks)
image.png
ken 位居views点击量榜首
sns.distplot(df['views'])
image.png
sns.distplot(df[df['views'] < 0.4e7]['views'])
image.png
sns.distplot(df[(df['views'] > 0.5e4)&(df['views'] < 0.4e7)]['views']) #多条件布尔索引
image.png
df['views'].describe()
image.png
TED Talks的views平均数为160万。中位数是112万。这表明TED Talks的普及程度非常高。我们也注意到大部分Talks的views不到400万。我们将在后面的章节中将这个框图作为框图的切点。
sns.distplot(df['comments'])
image.png
sns.distplot(df[df['comments'] < 500]['comments'])
image.png
sns.jointplot(x='views', y='comments', data=df)
image.png
df['comments'].describe()
image.png
平均每次Talks都有191.5条评论。假设评论是建设性的批评，我们可以得出结论，TED社区高度参与讨论循环谈判。与评论相关的标准偏差很大。事实上，它甚至比意味着这些措施可能对异常值敏感的意思更大。我们将绘制这个图来检查分布的性质。谈话的最小评论数是2，最大数是6404.该范围是6402 ..尽管如此，最少的数字可能是最近发表的谈话的结果。
df[['views', 'comments']].corr() #相关系数函数
image.png
如散点图和相关矩阵所示，相关系数略大于0.5。这表明两个数量之间的中等到强相关性。如上所述，这个结果是相当期待的。现在让我们来检查一下有史以来最受关注的10个会谈的观点和评论数量。
df[['title', 'main_speaker','views', 'comments']].sort_values('comments', ascending=False).head(10)
image.png
df['dis_quo'] = df['comments']/df['views'] #新增一列‘dis_quo’
#评论数/点击量之比 前10行
df[['title', 'main_speaker','views', 'comments', 'dis_quo', 'film_date']].sort_values('dis_quo', ascending=False).head(10)
image.png
month_order = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']
day_order = ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun']
df['month'] = df['film_date'].apply(lambda x: month_order[int(x.split('-')[1]) - 1])
df['month'].head()
month_df = pd.DataFrame(df['month'].value_counts()).reset_index()
month_df.columns = ['month', 'talks']
sns.barplot(x='month',y='talks',data=month_df,order=month_order)
image.png
二月显然是最受欢迎的会议，而八月和一月是最不受欢迎的。二月份的人气很大程度上是由于二月份举行的官方会议。让我们只检查TEDX talks的分配。
#使用datetime的strtime方法获取一个日期是周几
def getday(x):
day, month, year = (int(i) for i in x.split('-'))
answer = datetime.date(year, month, day).strftime("%A")
return answer[:3]
#使用datetime的weekday方法获取一个日期是一周里的第几天，用这个当索引在day_order里取相应的value值
def getday2(x):
day, month, year = (int(i) for i in x.split('-'))
answer = datetime.date(year, month, day).weekday()
return day_order[answer]
image.png
df['day'] = df['film_date'].apply(getday) #新增一列day列
day_df = pd.DataFrame(df['day'].value_counts()).reset_index()
day_df.columns = ['day', 'talks']
sns.barplot(x='day', y='talks', data=day_df, order=day_order)
image.png
day的分布几乎是一个钟声曲线，星期三和星期四是最受欢迎的日子，周日是最不受欢迎的。这是非常有趣的，因为我认为大部分的会议都会在周末的某个时候发生。
df['year'] = df['film_date'].apply(lambda x: x.split('-')[2])
year_df = pd.DataFrame(df['year'].value_counts().reset_index())
year_df.columns = ['year', 'talks']
plt.figure(figsize=(18,5))
sns.pointplot(x='year', y='talks', data=year_df)
image.png
如预期的那样，自1984年成立以来，TED Talks的数量逐年增加。如果在2009年进行，数量会急剧增加。了解2009年成为TED Talks数量增加两倍以上的转折点的原因可能是有趣的。自2009年以来，TED Talks数量几乎不变。
months = {'Jan': 1, 'Feb': 2, 'Mar': 3, 'Apr': 4, 'May': 5, 'Jun': 6, 'Jul': 7, 'Aug': 8, 'Sep': 9, 'Oct': 10, 'Nov': 11, 'Dec': 12}
hmap_df = df.copy()
hmap_df['film_date'] = hmap_df['film_date'].apply(lambda x: month_order[int(x.split('-')[1]) - 1] + " " + str(x.split('-')[2]))
hmap_df = pd.pivot_table(hmap_df[['film_date', 'title']], index='film_date', aggfunc='count').reset_index()
hmap_df['month_num'] = hmap_df['film_date'].apply(lambda x: months[x.split()[0]])
hmap_df['year'] = hmap_df['film_date'].apply(lambda x: x.split()[1])
hmap_df = hmap_df.sort_values(['year', 'month_num'])
hmap_df = hmap_df[['month_num', 'year', 'title']]
hmap_df = hmap_df.pivot('month_num', 'year', 'title')
hmap_df = hmap_df.fillna(0)
image.png
