1
00:00:00,060 --> 00:00:09,750
Hi, welcome back in this video, we will write an algorithm which gets a sentence, which usually is

2
00:00:09,750 --> 00:00:12,720
a question, for example, what are vegetables?

3
00:00:13,140 --> 00:00:21,390
And then the algorithm also has access to a variable which has texts, and in that text there are several

4
00:00:21,390 --> 00:00:22,020
sentences.

5
00:00:22,320 --> 00:00:28,380
So the algorithm should find the answer to that question in that text.

6
00:00:28,740 --> 00:00:36,660
So this is a common case for chatbots when you have a chat bot, which asks her questions in the next

7
00:00:36,660 --> 00:00:36,930
video.

8
00:00:36,930 --> 00:00:43,620
So we will build a chatbots, which answers Wikipedia questions about a certain topic, and the chat

9
00:00:43,620 --> 00:00:50,880
bot will read a Wikipedia article and give you the answer so that Shadbolt is based on these algorithms

10
00:00:50,880 --> 00:00:53,070
that we will write in this video.

11
00:00:53,310 --> 00:01:00,480
So in this algorithm, this algorithm will get this question, and then it will return the sentence

12
00:01:00,480 --> 00:01:07,710
among the sentences of a sentence that is the most similar to this question.

13
00:01:07,860 --> 00:01:09,060
So what are vegetables?

14
00:01:09,300 --> 00:01:13,920
We expect to get vegetables or old plants from this algorithm?

15
00:01:15,710 --> 00:01:21,590
For this or will he use any of the key, the library we have been using, so we will reuse some of the

16
00:01:21,590 --> 00:01:26,300
code of that and ldk lovely the code that we wrote previously.

17
00:01:26,470 --> 00:01:35,180
And then we will use site kids learn, which is a library that allows us to do this matching of the

18
00:01:35,180 --> 00:01:38,420
best sentence, given another sentence.

19
00:01:38,720 --> 00:01:42,980
So let's first see what's codes we need from the analytic library.

20
00:01:43,190 --> 00:01:50,480
We actually need this lemon function in here, which just to remind you what this does is it gets one

21
00:01:50,480 --> 00:01:56,930
sentence as inputs and it's tokenize is that sentence into words.

22
00:01:56,930 --> 00:02:00,530
So it's it generates a list of words.

23
00:02:00,830 --> 00:02:03,440
Each word is an item war vet list.

24
00:02:03,710 --> 00:02:04,850
So one list here.

25
00:02:04,850 --> 00:02:11,060
And then for each item we we find out the tags that which is the type of words.

26
00:02:11,060 --> 00:02:15,670
So for each word, we know what type of word that is is a verb or a noun, or et cetera.

27
00:02:16,430 --> 00:02:23,420
Then we convert each words into a limo, so we find the limo for each words.

28
00:02:25,130 --> 00:02:33,920
And if we get that function, for example, if we put this census into that function, we would get

29
00:02:33,920 --> 00:02:43,070
as output from that function a list which would look something like what would be the limit of R and

30
00:02:43,910 --> 00:02:46,280
vegetable, the limit of vegetables.

31
00:02:46,550 --> 00:02:48,530
So that is the output of that function.

32
00:02:49,160 --> 00:02:50,180
This kind of list.

33
00:02:50,630 --> 00:02:56,180
So we need to get that and copy it in here.

34
00:02:56,660 --> 00:02:58,370
So we do have that functional.

35
00:02:59,030 --> 00:03:03,720
Then what else do we need a K?

36
00:03:04,180 --> 00:03:04,880
Learn.

37
00:03:06,380 --> 00:03:09,380
Does feature extraction.

38
00:03:11,200 --> 00:03:26,260
The Texas imports T capital t f IDF vector riser, that is a class which will do a statistical analysis

39
00:03:26,260 --> 00:03:35,560
which will find out the importance of words in a curfew's a corpus is basically a piece of text.

40
00:03:36,250 --> 00:03:44,770
So this will help us build a matrix where we have the importance of each word, and that will help us

41
00:03:44,770 --> 00:03:48,520
to compare the similarity between sentences of the words.

42
00:03:48,640 --> 00:03:49,840
You'll see how that works.

43
00:03:50,260 --> 00:03:57,400
So execu, that's a then we need to create an instance of that class.

44
00:04:02,410 --> 00:04:08,260
Now, I said that this class is working on finding the importance of words.

45
00:04:08,650 --> 00:04:15,130
So therefore, this needs to have some sort of least as input a list of words.

46
00:04:15,640 --> 00:04:23,350
But instead of it putting the list of words here, this class is designed to get as inputs, a tokenize

47
00:04:23,350 --> 00:04:29,620
or a tokenizing which tokenize this sentences.

48
00:04:29,830 --> 00:04:33,010
So each convert them into words and we do have that function.

49
00:04:33,430 --> 00:04:35,110
So that is the LEMI function.

50
00:04:35,860 --> 00:04:42,700
So this function should get a sentence and should return a list, object type, mate of words.

51
00:04:43,210 --> 00:04:44,350
That is what we do there.

52
00:04:45,160 --> 00:04:45,940
Execute that.

53
00:04:46,690 --> 00:04:53,200
If you see TV, there's nothing to see right now because this is just a representation of the class

54
00:04:53,200 --> 00:04:54,220
without any value.

55
00:04:54,220 --> 00:04:54,640
See it.

56
00:04:55,090 --> 00:05:04,690
Now we need to do that finding of the words importance using another variable here to f for this and

57
00:05:04,690 --> 00:05:12,640
we need to use that object type and points of the fit transform method, which is an algorithm that

58
00:05:12,640 --> 00:05:16,240
will give the sentence tokens as inputs.

59
00:05:17,410 --> 00:05:18,940
Now what is the sentence tokens?

60
00:05:18,970 --> 00:05:22,030
Well, this is a variable that we haven't defined yet, actually.

61
00:05:22,510 --> 00:05:28,360
So we need to define this perhaps under the function here in a new cell.

62
00:05:28,990 --> 00:05:31,840
So we need to calculate the sentence tokens.

63
00:05:35,460 --> 00:05:40,440
This was the Nulty cable that sent tokenize.

64
00:05:41,250 --> 00:05:42,980
How do we get Texas input?

65
00:05:43,440 --> 00:05:47,490
So we are working on this text in here.

66
00:05:48,120 --> 00:05:50,190
And if you see what we get from this.

67
00:05:55,650 --> 00:06:03,840
Is a lease of three items each item represents, or is a sentence of that text?

68
00:06:04,050 --> 00:06:08,680
So originally, vegetables were collected from the wild by hunter-gatherers.

69
00:06:08,700 --> 00:06:09,930
That is the first sentence.

70
00:06:10,420 --> 00:06:15,620
Then we have vegetables or plants in here and so on.

71
00:06:15,630 --> 00:06:17,460
We have three items in that list.

72
00:06:17,970 --> 00:06:25,170
If you've got an error saying that, you need to download some nasty key data, so then just execute.

73
00:06:25,170 --> 00:06:28,500
That's code that is shown in there to download the dataset.

74
00:06:29,190 --> 00:06:30,360
Then I can execute that.

75
00:06:30,390 --> 00:06:35,010
Since I have this variable not defined in here, I can go ahead and execute it.

76
00:06:37,550 --> 00:06:42,850
Now this ask me to download, that's so I'm going to create a causal link here.

77
00:06:43,230 --> 00:06:45,470
Download it and then execute again.

78
00:06:47,120 --> 00:06:49,670
We've got an error here, a limited size.

79
00:06:49,670 --> 00:06:51,110
There is no defined.

80
00:06:55,520 --> 00:06:57,410
So it's talking about this line.

81
00:07:00,210 --> 00:07:06,210
That is the limit either instance.

82
00:07:08,360 --> 00:07:15,560
This one in here, so I'm going to get that coats and just place it in here.

83
00:07:17,420 --> 00:07:18,000
Execute.

84
00:07:18,920 --> 00:07:21,280
So that's the limo we function as a limo, Ticer.

85
00:07:21,800 --> 00:07:23,660
Now we go down.

86
00:07:25,620 --> 00:07:31,170
And execute that again, we need the Nets Library as well.

87
00:07:31,290 --> 00:07:35,790
So I'm going to cover that in here, so I'll execute.

88
00:07:35,790 --> 00:07:36,720
That's in here as well.

89
00:07:37,590 --> 00:07:38,550
Execute that again.

90
00:07:43,100 --> 00:07:46,460
If you want to see what S.F. is, you can just call it.

91
00:07:49,170 --> 00:07:53,520
So it's a mad streaks three by eight.

92
00:07:53,820 --> 00:07:59,910
No, I know you are completely lost at this point, so I want to show you what this looks like, actually.

93
00:08:00,180 --> 00:08:01,830
So this is an optional step.

94
00:08:02,640 --> 00:08:12,810
This code sale here to see this matrix you can do to dots to array that is a method that will display

95
00:08:12,810 --> 00:08:14,160
the actual matrix.

96
00:08:16,070 --> 00:08:19,760
This looks intimidating, but it's only explaining what it means.

97
00:08:20,250 --> 00:08:25,230
No, so this is a matrix made of lists.

98
00:08:25,250 --> 00:08:29,430
We have one this year, so it starts in here and it ends in here.

99
00:08:29,460 --> 00:08:30,560
That is the first list.

100
00:08:30,800 --> 00:08:32,020
Then we have the second list.

101
00:08:32,030 --> 00:08:35,960
It starts in here, it ends in here and then we have the third list.

102
00:08:36,620 --> 00:08:48,290
So we have one list for each sentence because you see that this ETF was built from the sentence tokens

103
00:08:48,650 --> 00:08:50,740
variable, which is this one in here.

104
00:08:50,750 --> 00:08:52,970
So it's a list of three items.

105
00:08:53,570 --> 00:09:03,260
Now let me copy that list and just put it in here temporarily just to see what's happening here.

106
00:09:03,980 --> 00:09:06,440
So one sentence one of this.

107
00:09:06,650 --> 00:09:17,420
And then if you look inside each list, you'll see that we have one two three four five six seven eight

108
00:09:17,450 --> 00:09:20,630
items, eight numbers.

109
00:09:21,320 --> 00:09:24,920
The second lease also has eight numbers, if you count.

110
00:09:25,310 --> 00:09:26,390
And the third one as well.

111
00:09:26,420 --> 00:09:29,450
So each of the lists have eight items.

112
00:09:29,990 --> 00:09:30,780
Why eight?

113
00:09:30,800 --> 00:09:36,870
Well, that is because the longest sentence in our texts has eight words.

114
00:09:37,190 --> 00:09:42,020
We can count one two three four five.

115
00:09:42,350 --> 00:09:45,620
This is not counted these articles.

116
00:09:46,250 --> 00:09:57,170
So six seven eight eight words, and each number represents the weights of each words in the entire

117
00:09:57,410 --> 00:09:57,950
text.

118
00:09:58,670 --> 00:10:07,730
So, for example, zero point three means the weight of originally in the text is that number, and

119
00:10:07,730 --> 00:10:10,430
then we have vegetables 0.52.

120
00:10:10,970 --> 00:10:18,740
It says a bigger weights because it's being used here as well and here to get a better representation

121
00:10:18,740 --> 00:10:26,660
of this matrix, you could use pandas, imports pandas and then creates a dataframe from pandas data

122
00:10:26,660 --> 00:10:27,110
frame.

123
00:10:27,650 --> 00:10:30,800
So again, this cell here and that's one are optional.

124
00:10:31,430 --> 00:10:40,550
They are just to show you what's going on here and then you say to feed that to rate as a source of

125
00:10:40,550 --> 00:10:45,760
the data frame, the columns will be TV.

126
00:10:45,770 --> 00:10:47,780
So the vector rise or.

127
00:10:53,890 --> 00:11:02,800
Which we defined in year that get feature names and methods, so call that method and then print out

128
00:11:03,010 --> 00:11:04,060
Jeff in here.

129
00:11:07,110 --> 00:11:08,910
So that is a better representation.

130
00:11:10,500 --> 00:11:22,620
You see the weight of the words be a 0.3 and the weight of collect is zero two in the first sentence

131
00:11:23,940 --> 00:11:27,240
and then the weight of those words in the second sentence and so on.

132
00:11:27,960 --> 00:11:34,500
For example, the weight of originally in the first sentence is 0.52.

133
00:11:34,650 --> 00:11:42,270
Both the weight of that towards in the second and third sentences is zero because there's no such words

134
00:11:42,270 --> 00:11:43,800
in those sentences.

135
00:11:45,760 --> 00:11:51,320
That is why we have all these zeros in here and also here, so it's the same matrix.

136
00:11:52,760 --> 00:11:53,120
All right.

137
00:11:53,120 --> 00:12:02,390
So this is how it works, but we need now to add another sentence to the sentence tokens.

138
00:12:02,720 --> 00:12:07,460
So when we created the sentence tokens, we use the text variable.

139
00:12:07,940 --> 00:12:15,110
We need to also append the question variable in that list of sentences.

140
00:12:15,530 --> 00:12:18,650
The question variable is this one which has this question.

141
00:12:19,460 --> 00:12:20,480
So execu that's.

142
00:12:22,430 --> 00:12:23,660
Now you see the sentence.

143
00:12:26,440 --> 00:12:27,130
Tokens.

144
00:12:29,180 --> 00:12:40,790
Is this list then you execute all the other cells to redefine them with that question, and it's all

145
00:12:40,790 --> 00:12:41,240
of them.

146
00:12:42,530 --> 00:12:51,020
You see, the array now has four lists instead of just three.

147
00:12:51,320 --> 00:12:52,790
And we delete that.

148
00:12:54,230 --> 00:12:57,920
I also see the dataframe has one more.

149
00:13:00,480 --> 00:13:04,740
There's no end, no to find the similarity.

150
00:13:04,770 --> 00:13:05,400
Finally.

151
00:13:07,860 --> 00:13:19,320
Let's say values are equal to cosine similarity, which we need to import from as key learn.

152
00:13:21,470 --> 00:13:23,000
That's metrics.

153
00:13:23,210 --> 00:13:27,170
Those pairwise imports, of course, sign.

154
00:13:28,880 --> 00:13:37,040
Similarity, so cosine similarity now finds a similarity between two mattresses, the first one being

155
00:13:37,280 --> 00:13:40,970
t f minus one.

156
00:13:41,420 --> 00:13:51,110
So t f, you know, t f is this and t f minus one minus one is the lost item of that matrix.

157
00:13:51,290 --> 00:13:54,050
So the lease that represents the question.

158
00:13:54,350 --> 00:13:58,160
So we want to compare to that question to the entire matrix.

159
00:13:58,580 --> 00:14:01,490
T f t f minus one with t f.

160
00:14:05,920 --> 00:14:06,680
Execute that.

161
00:14:07,300 --> 00:14:15,550
And this is what we get, we get some early and this early now has four values.

162
00:14:16,090 --> 00:14:25,060
As you see, each value represents the similarity between the question and each of the sentences in

163
00:14:25,060 --> 00:14:26,020
the texts.

164
00:14:26,650 --> 00:14:35,320
So that means how similar is our question what are vegetables with the first sentence?

165
00:14:36,670 --> 00:14:46,210
So the first sentence is, let me get this just to keep it here for explaining you what's going on.

166
00:14:46,840 --> 00:14:54,430
So what are vegetables is similar with the first sentence at this coefficients zero point thirty nine?

167
00:14:54,640 --> 00:15:01,420
And then it's similar to the second sentence, which is vegetables or plants.

168
00:15:01,870 --> 00:15:04,330
It's 0.59.

169
00:15:04,900 --> 00:15:10,420
The similarity coefficients for that, 0.46 for the other one.

170
00:15:11,920 --> 00:15:13,360
And then we have this one.

171
00:15:13,750 --> 00:15:21,880
One is the similarity between the question what our vegetables and the same question, what our vegetables,

172
00:15:21,880 --> 00:15:26,680
because you know, we appended that question to the sentence tokens.

173
00:15:27,940 --> 00:15:32,560
So the similarity between two identical sentences is one.

174
00:15:33,040 --> 00:15:34,000
It's hundred percent.

175
00:15:34,700 --> 00:15:41,830
Now we want to find out which sentence is the one that has the maximum value in this list.

176
00:15:42,520 --> 00:15:43,480
So let's find that out.

177
00:15:45,460 --> 00:15:48,460
We need to exit from the index of that census.

178
00:15:49,060 --> 00:15:57,580
We do that using values that are sort you covered methods, and if you see what's index is.

179
00:15:58,960 --> 00:16:06,820
Now you'll get an array, but you only want the first list because we have nested lists here, so we

180
00:16:06,820 --> 00:16:08,590
get the first list we execute again.

181
00:16:10,120 --> 00:16:14,730
You see, no, there's no two lists, but it's only one inside event.

182
00:16:14,750 --> 00:16:20,590
Then out of that first list, we get the minus two index.

183
00:16:22,360 --> 00:16:29,830
So that will give us the index of the maximum value, actually, it's not the highest value, it's the

184
00:16:29,830 --> 00:16:34,810
second greatest value because the first one is one.

185
00:16:34,960 --> 00:16:41,860
So if we apply, it's minus one here to get the light of the lost value out of that sort of values,

186
00:16:42,220 --> 00:16:43,600
we get an index of three.

187
00:16:43,690 --> 00:16:45,400
So that's the index for that.

188
00:16:45,670 --> 00:16:48,000
But with minus two, we get one.

189
00:16:48,010 --> 00:16:53,590
So Index zero, index one, we get the correct index for, that's maximum value.

190
00:16:54,160 --> 00:16:57,640
So that's a variable we need now in just a bit.

191
00:16:57,970 --> 00:17:00,720
However, first we want two points of values.

192
00:17:02,080 --> 00:17:06,550
That's so we want to see values that's flat and.

193
00:17:09,069 --> 00:17:09,910
And we get this.

194
00:17:11,800 --> 00:17:18,640
So it's the same values, basically, but we don't have these two nested lists, we have only one list

195
00:17:18,730 --> 00:17:19,240
in here.

196
00:17:23,030 --> 00:17:30,140
Lustily, now we can extract the coefficients, which is equal to the values flats.

197
00:17:31,490 --> 00:17:34,910
Well, it's flat matrix.

198
00:17:37,660 --> 00:17:43,900
Minus two, so we want to get always that before the last item.

199
00:17:49,790 --> 00:17:57,590
So now we have the coefficients and we could say something like, finally, if coefficient is greater

200
00:17:57,590 --> 00:17:58,140
than.

201
00:17:58,160 --> 00:18:05,210
Well, let's put a threshold of zero point three, then we'll return or we print out whatever we want

202
00:18:05,210 --> 00:18:05,690
to do.

203
00:18:05,990 --> 00:18:11,270
We printouts from sentence tokens we print out.

204
00:18:14,860 --> 00:18:23,170
The item with index one, if you execute that, you get the answer of that question.

205
00:18:23,410 --> 00:18:26,080
So the question was what are vegetables?

206
00:18:26,500 --> 00:18:29,380
The answer is the vegetables are all plants.

207
00:18:30,610 --> 00:18:37,960
So that is how you can get the most similar as or given a question from a piece of text.

208
00:18:38,500 --> 00:18:42,730
Using these quite advanced algorithm, that's useful in here.

209
00:18:45,070 --> 00:18:47,770
So thank you for following this, and I'll talk to you in the next few days.