我的书签
添加书签
移除书签14.2 MovieLens 1M数据集
GroupLens Research（http://www.grouplens.org/node/73 ）采集了一组从20世纪90年末到21世纪初由MovieLens用户提供的电影评分数据。这些数据中包括电影评分、电影元数据（风格类型和年代）以及关于用户的人口统计学数据（年龄、邮编、性别和职业等）。基于机器学习算法的推荐系统一般都会对此类数据感兴趣。虽然我不会在本书中详细介绍机器学习技术，但我会告诉你如何对这种数据进行切片切块以满足实际需求。
MovieLens 1M数据集含有来自6000名用户对4000部电影的100万条评分数据。它分为三个表：评分、用户信息和电影信息。将该数据从zip文件中解压出来之后，可以通过pandas.read_table将各个表分别读到一个pandas DataFrame对象中：
import pandas as pd# Make display smallerpd.options.display.max_rows = 10unames = ['user_id', 'gender', 'age', 'occupation', 'zip']users = pd.read_table('datasets/movielens/users.dat', sep='::',header=None, names=unames)rnames = ['user_id', 'movie_id', 'rating', 'timestamp']ratings = pd.read_table('datasets/movielens/ratings.dat', sep='::',header=None, names=rnames)mnames = ['movie_id', 'title', 'genres']movies = pd.read_table('datasets/movielens/movies.dat', sep='::',header=None, names=mnames)
利用Python的切片语法，通过查看每个DataFrame的前几行即可验证数据加载工作是否一切顺利：
In [69]: users[:5]Out[69]:user_id gender age occupation zip0 1 F 1 10 480671 2 M 56 16 700722 3 M 25 15 551173 4 M 45 7 024604 5 M 25 20 55455In [70]: ratings[:5]Out[70]:user_id movie_id rating timestamp0 1 1193 5 9783007601 1 661 3 9783021092 1 914 3 9783019683 1 3408 4 9783002754 1 2355 5 978824291In [71]: movies[:5]Out[71]:movie_id title genres0 1 Toy Story (1995) Animation|Children's|Comedy1 2 Jumanji (1995) Adventure|Children's|Fantasy2 3 Grumpier Old Men (1995) Comedy|Romance3 4 Waiting to Exhale (1995) Comedy|Drama4 5 Father of the Bride Part II (1995) ComedyIn [72]: ratingsOut[72]:user_id movie_id rating timestamp0 1 1193 5 9783007601 1 661 3 9783021092 1 914 3 9783019683 1 3408 4 9783002754 1 2355 5 978824291... ... ... ... ...1000204 6040 1091 1 9567165411000205 6040 1094 5 9567048871000206 6040 562 5 9567047461000207 6040 1096 4 9567156481000208 6040 1097 4 956715569[1000209 rows x 4 columns]
注意，其中的年龄和职业是以编码形式给出的，它们的具体含义请参考该数据集的README文件。分析散布在三个表中的数据可不是一件轻松的事情。假设我们想要根据性别和年龄计算某部电影的平均得分，如果将所有数据都合并到一个表中的话问题就简单多了。我们先用pandas的merge函数将ratings跟users合并到一起，然后再将movies也合并进去。pandas会根据列名的重叠情况推断出哪些列是合并（或连接）键：
In [73]: data = pd.merge(pd.merge(ratings, users), movies)In [74]: dataOut[74]:user_id movie_id rating timestamp gender age occupation zip \0 1 1193 5 978300760 F 1 10 480671 2 1193 5 978298413 M 56 16 700722 12 1193 4 978220179 M 25 12 327933 15 1193 4 978199279 M 25 7 229034 17 1193 5 978158471 M 50 1 95350... ... ... ... ... ... ... ... ...1000204 5949 2198 5 958846401 M 18 17 479011000205 5675 2703 3 976029116 M 35 14 300301000206 5780 2845 1 958153068 M 18 17 928861000207 5851 3607 5 957756608 F 18 20 554101000208 5938 2909 4 957273353 M 25 1 35401title genres0 One Flew Over the Cuckoo's Nest (1975) Drama1 One Flew Over the Cuckoo's Nest (1975) Drama2 One Flew Over the Cuckoo's Nest (1975) Drama3 One Flew Over the Cuckoo's Nest (1975) Drama4 One Flew Over the Cuckoo's Nest (1975) Drama... ... ...1000204 Modulations (1998) Documentary1000205 Broken Vessels (1998) Drama1000206 White Boys (1999) Drama1000207 One Little Indian (1973) Comedy|Drama|Western1000208 Five Wives, Three Secretaries and Me (1998) Documentary[1000209 rows x 10 columns]In [75]: data.iloc[0]Out[75]:user_id 1movie_id 1193rating 5timestamp 978300760gender Fage 1occupation 10zip 48067title One Flew Over the Cuckoo's Nest (1975)genres DramaName: 0, dtype: object
为了按性别计算每部电影的平均得分，我们可以使用pivot_table方法：
In [76]: mean_ratings = data.pivot_table('rating', index='title',....: columns='gender', aggfunc='mean')In [77]: mean_ratings[:5]Out[77]:gender F Mtitle$1,000,000 Duck (1971) 3.375000 2.761905'Night Mother (1986) 3.388889 3.352941'Til There Was You (1997) 2.675676 2.733333'burbs, The (1989) 2.793478 2.962085...And Justice for All (1979) 3.828571 3.689024
该操作产生了另一个DataFrame，其内容为电影平均得分，行标为电影名称（索引），列标为性别。现在，我打算过滤掉评分数据不够250条的电影（随便选的一个数字）。为了达到这个目的，我先对title进行分组，然后利用size()得到一个含有各电影分组大小的Series对象：
In [78]: ratings_by_title = data.groupby('title').size()In [79]: ratings_by_title[:10]Out[79]:title$1,000,000 Duck (1971) 37'Night Mother (1986) 70'Til There Was You (1997) 52'burbs, The (1989) 303...And Justice for All (1979) 1991-900 (1994) 210 Things I Hate About You (1999) 700101 Dalmatians (1961) 565101 Dalmatians (1996) 36412 Angry Men (1957) 616dtype: int64In [80]: active_titles = ratings_by_title.index[ratings_by_title >= 250]In [81]: active_titlesOut[81]:Index([''burbs, The (1989)', '10 Things I Hate About You (1999)','101 Dalmatians (1961)', '101 Dalmatians (1996)', '12 Angry Men (1957)','13th Warrior, The (1999)', '2 Days in the Valley (1996)','20,000 Leagues Under the Sea (1954)', '2001: A Space Odyssey (1968)','2010 (1984)',...'X-Men (2000)', 'Year of Living Dangerously (1982)','Yellow Submarine (1968)', 'You've Got Mail (1998)','Young Frankenstein (1974)', 'Young Guns (1988)','Young Guns II (1990)', 'Young Sherlock Holmes (1985)','Zero Effect (1998)', 'eXistenZ (1999)'],dtype='object', name='title', length=1216)
标题索引中含有评分数据大于250条的电影名称，然后我们就可以据此从前面的mean_ratings中选取所需的行了：
# Select rows on the indexIn [82]: mean_ratings = mean_ratings.loc[active_titles]In [83]: mean_ratingsOut[83]:gender F Mtitle'burbs, The (1989) 2.793478 2.96208510 Things I Hate About You (1999) 3.646552 3.311966101 Dalmatians (1961) 3.791444 3.500000101 Dalmatians (1996) 3.240000 2.91121512 Angry Men (1957) 4.184397 4.328421... ... ...Young Guns (1988) 3.371795 3.425620Young Guns II (1990) 2.934783 2.904025Young Sherlock Holmes (1985) 3.514706 3.363344Zero Effect (1998) 3.864407 3.723140eXistenZ (1999) 3.098592 3.289086[1216 rows x 2 columns]
为了了解女性观众最喜欢的电影，我们可以对F列降序排列：
In [85]: top_female_ratings = mean_ratings.sort_values(by='F', ascending=False)In [86]: top_female_ratings[:10]Out[86]:gender F MtitleClose Shave, A (1995) 4.644444 4.473795Wrong Trousers, The (1993) 4.588235 4.478261Sunset Blvd. (a.k.a. Sunset Boulevard) (1950) 4.572650 4.464589Wallace & Gromit: The Best of Aardman Animation... 4.563107 4.385075Schindler's List (1993) 4.562602 4.491415Shawshank Redemption, The (1994) 4.539075 4.560625Grand Day Out, A (1992) 4.537879 4.293255To Kill a Mockingbird (1962) 4.536667 4.372611Creature Comforts (1990) 4.513889 4.272277Usual Suspects, The (1995) 4.513317 4.518248
计算评分分歧
假设我们想要找出男性和女性观众分歧最大的电影。一个办法是给mean_ratings加上一个用于存放平均得分之差的列，并对其进行排序：
In [87]: mean_ratings['diff'] = mean_ratings['M'] - mean_ratings['F']
按”diff”排序即可得到分歧最大且女性观众更喜欢的电影：
In [88]: sorted_by_diff = mean_ratings.sort_values(by='diff')In [89]: sorted_by_diff[:10]Out[89]:gender F M difftitleDirty Dancing (1987) 3.790378 2.959596 -0.830782Jumpin' Jack Flash (1986) 3.254717 2.578358 -0.676359Grease (1978) 3.975265 3.367041 -0.608224Little Women (1994) 3.870588 3.321739 -0.548849Steel Magnolias (1989) 3.901734 3.365957 -0.535777Anastasia (1997) 3.800000 3.281609 -0.518391Rocky Horror Picture Show, The (1975) 3.673016 3.160131 -0.512885Color Purple, The (1985) 4.158192 3.659341 -0.498851Age of Innocence, The (1993) 3.827068 3.339506 -0.487561Free Willy (1993) 2.921348 2.438776 -0.482573
对排序结果反序并取出前10行，得到的则是男性观众更喜欢的电影：
# Reverse order of rows, take first 10 rowsIn [90]: sorted_by_diff[::-1][:10]Out[90]:gender F M difftitleGood, The Bad and The Ugly, The (1966) 3.494949 4.221300 0.726351Kentucky Fried Movie, The (1977) 2.878788 3.555147 0.676359Dumb & Dumber (1994) 2.697987 3.336595 0.638608Longest Day, The (1962) 3.411765 4.031447 0.619682Cable Guy, The (1996) 2.250000 2.863787 0.613787Evil Dead II (Dead By Dawn) (1987) 3.297297 3.909283 0.611985Hidden, The (1987) 3.137931 3.745098 0.607167Rocky III (1982) 2.361702 2.943503 0.581801Caddyshack (1980) 3.396135 3.969737 0.573602For a Few Dollars More (1965) 3.409091 3.953795 0.544704
如果只是想要找出分歧最大的电影（不考虑性别因素），则可以计算得分数据的方差或标准差：
# Standard deviation of rating grouped by titleIn [91]: rating_std_by_title = data.groupby('title')['rating'].std()# Filter down to active_titlesIn [92]: rating_std_by_title = rating_std_by_title.loc[active_titles]# Order Series by value in descending orderIn [93]: rating_std_by_title.sort_values(ascending=False)[:10]Out[93]:titleDumb & Dumber (1994) 1.321333Blair Witch Project, The (1999) 1.316368Natural Born Killers (1994) 1.307198Tank Girl (1995) 1.277695Rocky Horror Picture Show, The (1975) 1.260177Eyes Wide Shut (1999) 1.259624Evita (1996) 1.253631Billy Madison (1995) 1.249970Fear and Loathing in Las Vegas (1998) 1.246408Bicentennial Man (1999) 1.245533Name: rating, dtype: float64
可能你已经注意到了，电影分类是以竖线（|）分隔的字符串形式给出的。如果想对电影分类进行分析的话，就需要先将其转换成更有用的形式才行。
