1
00:00:00,350 --> 00:00:01,280
Hey, welcome back.

2
00:00:01,280 --> 00:00:07,370
We will continue developing our movie recommendation system, doing popularity based filtering, which

3
00:00:07,370 --> 00:00:12,410
means we want to show a bunch of popular movies to our users.

4
00:00:13,100 --> 00:00:17,810
So in such a project, your project will have two parts, right?

5
00:00:17,840 --> 00:00:26,450
One part is the developing of the model, which is done typically in a Jupyter notebook such as this

6
00:00:26,450 --> 00:00:27,050
one here.

7
00:00:27,050 --> 00:00:34,280
And then you have the web development part where you develop your websites, where you show the movie

8
00:00:34,310 --> 00:00:36,890
thumbnails and the titles and so on.

9
00:00:36,890 --> 00:00:44,810
So that website, that part of the project, that website will take the model from the Jupyter Notebook.

10
00:00:44,810 --> 00:00:51,350
Basically you develop the model in Jupyter Notebook and then you just paste it in your web app files

11
00:00:51,350 --> 00:00:55,970
and integrate it with your HTML and CSS.

12
00:00:55,970 --> 00:01:00,420
So we're doing the part of of building the model.

13
00:01:00,420 --> 00:01:05,640
And so let's build the model for showing the most popular movies.

14
00:01:07,820 --> 00:01:12,380
To do that, we're going to use a formula from IMDb.

15
00:01:12,620 --> 00:01:21,410
IMDb is a movie database, so it's a website where you can see the ratings of movies and other information

16
00:01:21,410 --> 00:01:22,670
about movies.

17
00:01:23,670 --> 00:01:28,680
So let's create a new section here and maybe a heading three.

18
00:01:28,710 --> 00:01:37,530
So this is calculate a weighted rating, right?

19
00:01:37,800 --> 00:01:43,750
This one here was H three as well.

20
00:01:43,770 --> 00:01:46,860
This was load the data.

21
00:01:46,860 --> 00:01:47,430
Right?

22
00:01:47,430 --> 00:01:53,970
So previously we loaded the data and now we want to calculate the average rating in a text.

23
00:01:54,300 --> 00:01:58,330
I'm just going to paste here the formula we will use.

24
00:01:58,350 --> 00:02:01,320
So that's the weighted rating.

25
00:02:01,320 --> 00:02:05,310
So we will calculate a weighted rating for each movie.

26
00:02:05,340 --> 00:02:08,280
This is a formula for from IMDb.

27
00:02:08,460 --> 00:02:10,400
So you would ask why?

28
00:02:10,410 --> 00:02:16,270
Why a weighted rating and what are all these these letters, the symbols?

29
00:02:16,290 --> 00:02:24,400
Well, a weighted rating will calculate the weighted rating because currently we have a vote average

30
00:02:24,550 --> 00:02:26,710
A rating basically for every movie.

31
00:02:26,710 --> 00:02:34,150
For example, for Avatar, we have 7.2, and that's calculated out of 1800 votes.

32
00:02:34,150 --> 00:02:40,240
So 1800 people voted and the average of those votes was 7.2.

33
00:02:40,240 --> 00:02:48,400
But what happens if there is a movie which is not very popular and three people voted for that movie

34
00:02:48,400 --> 00:02:52,030
and they all voted ten in a scale 1 to 10.

35
00:02:52,060 --> 00:02:52,900
They voted ten.

36
00:02:52,900 --> 00:02:55,630
So the average for that movie is ten.

37
00:02:55,630 --> 00:03:01,780
So that's not a good representation of the popularity of that movie because those three people perhaps

38
00:03:01,780 --> 00:03:05,170
were the cuisines of the directors of the movie.

39
00:03:05,170 --> 00:03:05,410
Right.

40
00:03:05,410 --> 00:03:08,230
So we want more data.

41
00:03:08,260 --> 00:03:12,040
That's why we want to calculate a weighted rating.

42
00:03:12,040 --> 00:03:20,440
So a number, an average, which is a combination of these two factors, Right.

43
00:03:20,680 --> 00:03:25,090
So that's why we use this formula.

44
00:03:25,810 --> 00:03:30,210
So let me explain you here what each of these letters are.

45
00:03:30,220 --> 00:03:36,310
So we have V that is the number of votes for a movie.

46
00:03:37,270 --> 00:03:41,020
So we have V already a number of votes.

47
00:03:41,050 --> 00:03:43,750
It's under the vote count, right?

48
00:03:44,700 --> 00:03:46,770
11 800.

49
00:03:47,760 --> 00:03:49,950
4500, and so on.

50
00:03:52,040 --> 00:04:05,420
And then we have m m is the minimum number of votes required for a movie to be considered in the list

51
00:04:05,420 --> 00:04:06,990
of popular movies.

52
00:04:07,010 --> 00:04:07,760
Right.

53
00:04:08,150 --> 00:04:11,360
Because we don't want to work on all the data.

54
00:04:11,510 --> 00:04:14,980
So we want to put a threshold there.

55
00:04:14,990 --> 00:04:16,790
And then what else we have?

56
00:04:16,820 --> 00:04:23,540
We have we have R that is the average rating of the movie.

57
00:04:24,200 --> 00:04:27,890
So that's under movies.

58
00:04:27,890 --> 00:04:30,260
Data frame under vote average.

59
00:04:30,260 --> 00:04:36,290
So 7.2 in this case would be R And then lastly, we have C.

60
00:04:36,320 --> 00:04:41,240
C is the average rating across all movies.

61
00:04:41,240 --> 00:04:45,050
So we'll have to calculate some of these numbers, right?

62
00:04:45,910 --> 00:04:53,230
We have V already, so we have V for each row, each movie, and we have R also.

63
00:04:53,230 --> 00:04:54,970
So let me make this bold.

64
00:04:55,000 --> 00:05:01,180
Now let's calculate M and C, So I'll add a code cell here.

65
00:05:01,300 --> 00:05:05,620
So M will be equal to movies.

66
00:05:05,650 --> 00:05:09,430
Movies is the name of my movies data frame.

67
00:05:09,430 --> 00:05:11,800
So that's a variable holding that.

68
00:05:11,800 --> 00:05:14,050
So this is the number of votes, right?

69
00:05:14,050 --> 00:05:16,540
So we want to access.

70
00:05:17,170 --> 00:05:18,610
Let me see.

71
00:05:18,640 --> 00:05:26,740
That's a vote underscore count votes underscore count.

72
00:05:27,190 --> 00:05:33,670
And I want to apply a quantile of 0.9.

73
00:05:33,850 --> 00:05:36,310
You can experiment with different values.

74
00:05:36,310 --> 00:05:38,200
So what is 0.9?

75
00:05:38,410 --> 00:05:41,380
0.9 means 90%.

76
00:05:41,380 --> 00:05:50,630
And if I print M now here, I execute this cell and I'll get 1838.

77
00:05:50,630 --> 00:06:04,460
So that means 90% of the movies have a vote count, less than 1000 and 838 and 10% of the movies in

78
00:06:04,460 --> 00:06:08,600
the data frame have a vote count of higher than this number.

79
00:06:08,960 --> 00:06:14,630
So we will consider only those movies who have a vote count higher than this.

80
00:06:14,660 --> 00:06:16,250
That's the idea here.

81
00:06:17,720 --> 00:06:24,860
That's the value for M, Let's calculate c, C is simply movies.

82
00:06:24,950 --> 00:06:28,070
So we're talking about average rating here.

83
00:06:29,020 --> 00:06:29,950
Vote.

84
00:06:30,660 --> 00:06:33,750
Rating, not vote count.

85
00:06:33,750 --> 00:06:34,890
Vote rating.

86
00:06:34,920 --> 00:06:37,500
Is it rating or is it ratings?

87
00:06:37,860 --> 00:06:38,610
No, sorry.

88
00:06:38,610 --> 00:06:39,870
It's vote average.

89
00:06:39,900 --> 00:06:41,040
Vote average.

90
00:06:44,160 --> 00:06:45,840
Dot mean.

91
00:06:46,080 --> 00:06:54,510
So that's a method of a data frame column or a data frame series.

92
00:06:54,510 --> 00:07:01,860
In other words, in more technical terms, that is a series press enter once see there, execute with

93
00:07:01,860 --> 00:07:02,700
Ctrl enter.

94
00:07:02,700 --> 00:07:05,880
And that's the average rating of all movies.

95
00:07:06,630 --> 00:07:18,150
So now we have M as well and we have C, so we have all these and let's calculate the W, R, so the

96
00:07:18,150 --> 00:07:23,100
weighted rating for each row in the movies database.

97
00:07:27,340 --> 00:07:30,550
So to do that, we would do something like this.

98
00:07:31,660 --> 00:07:32,770
Movies.

99
00:07:32,980 --> 00:07:39,790
And we want to add a new weighted rating column.

100
00:07:39,790 --> 00:07:44,740
So a new column will be created in the movies data frame.

101
00:07:48,970 --> 00:07:51,420
So we don't have that column yet.

102
00:07:51,430 --> 00:07:51,790
Right.

103
00:07:51,790 --> 00:07:53,230
It will be a new column.

104
00:07:53,290 --> 00:08:02,260
Add it here, append it after the existing columns, and that will be equal to movies that apply.

105
00:08:03,550 --> 00:08:14,200
And here now, we want to write the name of a function, a custom function, which we need.

106
00:08:14,230 --> 00:08:16,960
We will have to create, right.

107
00:08:17,470 --> 00:08:21,160
And we want to apply this on the axis one.

108
00:08:21,730 --> 00:08:28,450
So the columns and before I execute this, I mean you can execute it, but you will get a name error

109
00:08:28,450 --> 00:08:31,120
because this name is not defined.

110
00:08:31,270 --> 00:08:36,760
So before I execute this, I will add a code block above this code block.

111
00:08:38,220 --> 00:08:41,580
And in here I'll define a weight.

112
00:08:42,240 --> 00:08:42,470
Sorry.

113
00:08:42,570 --> 00:08:43,289
Or weight.

114
00:08:43,380 --> 00:08:44,510
Wherever you prefer.

115
00:08:44,520 --> 00:08:44,850
Weight.

116
00:08:44,850 --> 00:08:46,200
It's a rating.

117
00:08:48,610 --> 00:08:49,690
Function.

118
00:08:51,150 --> 00:08:54,870
Which we'll get as a first argument.

119
00:08:54,870 --> 00:08:59,010
It will get an X, or you can also write.

120
00:08:59,040 --> 00:09:01,830
DF So it's just an argument name.

121
00:09:02,310 --> 00:09:14,400
So it will get a data frame, it will get also an M and a C, So the average rating across all movies

122
00:09:14,400 --> 00:09:17,190
and the minimum votes required.

123
00:09:17,190 --> 00:09:25,350
So M which is equal to M So it's equal to the value in here which we calculated already.

124
00:09:25,350 --> 00:09:33,420
So this parameter has a default value and also C is equal to C rates.

125
00:09:33,840 --> 00:09:40,620
And then we will write here, we will apply this formula in Python.

126
00:09:41,640 --> 00:09:46,320
So let me copy this and have it somewhere in here.

127
00:09:48,970 --> 00:09:50,890
Just so that we can see it.

128
00:09:51,460 --> 00:10:04,240
So w r will be equal to V divided by v plus m, and all that is multiplied by r.

129
00:10:05,770 --> 00:10:07,630
So what is R?

130
00:10:07,660 --> 00:10:08,950
Well, r is.

131
00:10:08,980 --> 00:10:10,780
Let's do r here.

132
00:10:10,810 --> 00:10:12,880
R is df.

133
00:10:12,880 --> 00:10:14,830
So this parameter.

134
00:10:17,170 --> 00:10:18,120
Volts.

135
00:10:18,340 --> 00:10:21,070
So that was the average rating.

136
00:10:22,230 --> 00:10:25,140
So that is volts average.

137
00:10:26,670 --> 00:10:27,300
Right.

138
00:10:28,870 --> 00:10:36,490
So basically when we execute this line later on, this line will call the weighted rating function.

139
00:10:36,610 --> 00:10:43,380
And as as TF, the movies data frame will be passed to DF.

140
00:10:43,390 --> 00:10:52,530
So basically in here we will have r equal to the column volt average, right?

141
00:10:52,570 --> 00:11:05,590
Because DF is actually the movie's data frame, so we'll get a access to that column and then we multiply

142
00:11:05,590 --> 00:11:08,960
this to each row of that column.

143
00:11:08,980 --> 00:11:10,810
That's the idea.

144
00:11:11,470 --> 00:11:20,980
Plus, let me put this entire expression in parentheses and then plus in parentheses again, this will

145
00:11:20,980 --> 00:11:28,330
be M divided by V plus M.

146
00:11:29,860 --> 00:11:31,500
And that's multiply, see?

147
00:11:31,690 --> 00:11:32,760
So there we go.

148
00:11:32,770 --> 00:11:33,450
O v l.

149
00:11:33,460 --> 00:11:40,120
So v also needs to come from df volts.

150
00:11:41,690 --> 00:11:42,500
Count.

151
00:11:43,400 --> 00:11:44,150
So yeah.

152
00:11:44,180 --> 00:11:47,420
V was the number of votes for a movie.

153
00:11:47,960 --> 00:11:52,700
So with that you can then return w r.

154
00:11:54,790 --> 00:11:56,410
And executed.

155
00:11:57,130 --> 00:12:00,740
So the execution of the function definition was successful.

156
00:12:00,760 --> 00:12:03,790
But now we need to call that function.

157
00:12:03,790 --> 00:12:09,130
Actually, the apply method will call that function on the background.

158
00:12:09,160 --> 00:12:12,940
So you don't need to place parentheses here, right?

159
00:12:12,970 --> 00:12:15,250
Just the name of the function.

160
00:12:16,680 --> 00:12:18,300
So execute that.

161
00:12:20,410 --> 00:12:21,850
It will take a while.

162
00:12:22,810 --> 00:12:23,320
Right.

163
00:12:23,320 --> 00:12:25,240
So it seems we got an error.

164
00:12:25,270 --> 00:12:26,680
Zero division error.

165
00:12:26,750 --> 00:12:32,110
Now, Deepnote will not show you the entire error, so you need to click on show error details.

166
00:12:32,110 --> 00:12:37,900
That's very important so that you know what what exactly happens here?

167
00:12:40,610 --> 00:12:48,020
So you will see highlighted the expression where the zero division error happened.

168
00:12:48,290 --> 00:12:59,990
So there was an error when V was trying to divide V plus M, so obviously v plus M was equal to zero

169
00:12:59,990 --> 00:13:01,130
at some point.

170
00:13:01,130 --> 00:13:06,710
And that is where the interpreter breaks and gives you this error.

171
00:13:07,040 --> 00:13:13,010
So V and m v is the vote count, right?

172
00:13:13,040 --> 00:13:14,510
M is not zero.

173
00:13:14,510 --> 00:13:16,400
We know M is not zero.

174
00:13:19,850 --> 00:13:21,740
So V and M.

175
00:13:22,760 --> 00:13:31,870
So I think we have some vote counts as zero values in the vote count.

176
00:13:31,880 --> 00:13:36,860
Basically if you display the entire data frame in this cell.

177
00:13:36,860 --> 00:13:44,240
So instead of movies that had you want to just say movies and then scroll to.

178
00:13:46,400 --> 00:13:47,900
Vote count.

179
00:13:47,930 --> 00:13:54,300
You will see here that this ranges from 0 to 1003.

180
00:13:54,480 --> 00:13:56,690
Sorry, 13,000 something.

181
00:13:57,600 --> 00:13:58,250
Right.

182
00:14:03,180 --> 00:14:06,570
Now if your machine shuts down due to inactivity.

183
00:14:06,570 --> 00:14:17,010
So it means the server you're using on Deepnote, then you can just click on that run notebook and that

184
00:14:17,010 --> 00:14:19,110
will start up the notebook again.

185
00:14:19,110 --> 00:14:24,630
So that happens if you left the notebook not running for a while.

186
00:14:26,970 --> 00:14:36,810
So now this will run all the cells and yeah, we can go back to vote count in here to sort it by ascending

187
00:14:36,810 --> 00:14:41,160
order and you'll see that some movies, they have a zero value.

188
00:14:41,780 --> 00:14:54,240
And to fix that we can go down here and before we calculate the weighted rating for our movie and even

189
00:14:54,240 --> 00:14:57,330
before defining the function here.

190
00:14:57,330 --> 00:15:05,700
So in this cell somewhere here, I want to add a code cell and I want to create a new movies filtered

191
00:15:06,750 --> 00:15:12,660
dataframe, which is equal to the movies existing dataframe dot copy.

192
00:15:12,660 --> 00:15:23,460
So this will create a copy of the dataframe and using this filtering so loc square brackets movies.

193
00:15:25,470 --> 00:15:31,770
Vote count greater or equal to M.

194
00:15:31,920 --> 00:15:35,390
So M is the vote threshold.

195
00:15:35,400 --> 00:15:37,830
We calculate it up here.

196
00:15:38,010 --> 00:15:42,210
So 1838in my case.

197
00:15:43,650 --> 00:15:57,450
So this filtered move is now if I print it out in here, I execute this and we're going to get movies

198
00:15:57,450 --> 00:16:00,690
with a vote count from.

199
00:16:00,690 --> 00:16:06,840
You see a 1842 13,752.

200
00:16:06,840 --> 00:16:14,220
So not from zero, but from 1840, which is close to the M threshold here.

201
00:16:14,490 --> 00:16:18,840
So and then we go down here.

202
00:16:18,990 --> 00:16:27,880
So in here now we use movies filtered this new data frame, the filtered data frame, which contains

203
00:16:27,880 --> 00:16:29,380
less records.

204
00:16:30,280 --> 00:16:31,720
And here as well.

205
00:16:32,530 --> 00:16:40,480
And go to the next line before you execute this, you may want to just print it out.

206
00:16:40,930 --> 00:16:42,550
So press enter.

207
00:16:42,910 --> 00:16:43,990
So there we go.

208
00:16:43,990 --> 00:16:50,020
And we should have a weighted rating column now, this one in here.

209
00:16:50,260 --> 00:16:56,230
So that's the column we have just created with these values.

210
00:16:57,140 --> 00:17:07,579
And then in the next line we add a code cell and we say movies filtered dot sort values.

211
00:17:09,770 --> 00:17:14,480
Sort by weighted average.

212
00:17:14,869 --> 00:17:18,050
Let me confirm if that was the actual name.

213
00:17:18,680 --> 00:17:19,000
Sorry.

214
00:17:19,010 --> 00:17:20,089
Weighted rating.

215
00:17:20,119 --> 00:17:22,460
Weighted rating.

216
00:17:23,780 --> 00:17:30,590
So the new column we have just calculated ascending will be equal to false.

217
00:17:31,160 --> 00:17:35,420
So that should give us let's get the heads.

218
00:17:35,420 --> 00:17:40,040
And this time we set ten and print out.

219
00:17:40,760 --> 00:17:47,870
So we got the first ten records of the data frame sorting by this column.

220
00:17:48,610 --> 00:17:49,990
So that means.

221
00:17:52,850 --> 00:17:59,360
These movies are now the most popular ones, as you see the titles here.

222
00:18:00,400 --> 00:18:05,980
You can also, just before you apply the heads here, you can also do.

223
00:18:06,710 --> 00:18:10,040
Square brackets and then again square brackets.

224
00:18:10,040 --> 00:18:11,870
And you provide a list of.

225
00:18:13,930 --> 00:18:20,260
Columns you want to see here, like title voter rating.

226
00:18:21,280 --> 00:18:22,390
Press Enter.

227
00:18:23,470 --> 00:18:24,040
Sorry.

228
00:18:27,600 --> 00:18:30,450
This was weighted rating.

229
00:18:32,600 --> 00:18:39,920
And so now you see a cleaner table with just the titles and the weighted rating, which was the column

230
00:18:39,920 --> 00:18:41,360
we calculated.

231
00:18:41,720 --> 00:18:46,940
I'll actually hide this sidebar to show more room here, more, more code for you.

232
00:18:47,630 --> 00:18:52,400
So that was a popularity based recommendation system.

233
00:18:52,400 --> 00:18:53,840
So this was the model.

234
00:18:53,840 --> 00:18:59,690
And you can then build a website, you can get this code, put it on the website, and so have a function

235
00:18:59,690 --> 00:19:09,260
there which queries which executes this code and it returns basically these values, which of course

236
00:19:09,260 --> 00:19:19,520
you can also convert it into a dictionary so you can apply at the end here dot dict or to dict a method,

237
00:19:20,840 --> 00:19:24,170
execute that and you'll get this dictionary.

238
00:19:25,130 --> 00:19:32,820
So if you want to serve that dictionary to the web app, it depends on how things are set up on the

239
00:19:32,820 --> 00:19:33,630
web app.

240
00:19:34,200 --> 00:19:37,140
And so that's again our data frame.

241
00:19:37,140 --> 00:19:38,750
And yeah, thanks for following this.

242
00:19:38,760 --> 00:19:44,850
We will do some more advanced filtering, more advanced recommendation systems in the next videos.

243
00:19:44,850 --> 00:19:48,360
So content based and collaborative based filtering.

244
00:19:48,390 --> 00:19:48,990
See you there.