1
00:00:00,060 --> 00:00:03,540
Previously, you learned how to limit his words.

2
00:00:05,310 --> 00:00:11,370
Using the limited ICE method in this video, you learn how to limit ice.

3
00:00:11,700 --> 00:00:13,290
Complete sentences.

4
00:00:14,040 --> 00:00:21,630
So which means that you will find the limb of each word in a sentence and then be able to compare two

5
00:00:21,630 --> 00:00:23,760
sentences if they are the same.

6
00:00:24,420 --> 00:00:30,780
So the lemur is the basic form of a word, as you learned in the previous video.

7
00:00:30,810 --> 00:00:38,040
Now, of course, so far we have simply done a simple comparison, such as if this lemur is equal to

8
00:00:38,040 --> 00:00:45,840
that lemur, but later on, we will do more complex similarity comparisons, which will give us a certain

9
00:00:45,840 --> 00:00:53,520
value, a degree of similarity between two sentences because sentences cannot always be exactly the

10
00:00:53,520 --> 00:01:01,370
same, but they could be similar to a certain degree, which tells us if this tax is related to that

11
00:01:01,380 --> 00:01:02,190
text or not.

12
00:01:03,000 --> 00:01:10,350
But before we do the comparisons, of course, we have to limit these sentences in order to get better

13
00:01:10,350 --> 00:01:11,010
results.

14
00:01:11,190 --> 00:01:17,370
Of course, you can also compare two sentences using the similarity algorithms that all teach you in

15
00:01:17,370 --> 00:01:23,880
the next videos without limitation, but with limits ization, you get some more accurate results.

16
00:01:24,510 --> 00:01:28,620
So let's focus on limits ization of sentences in this video.

17
00:01:30,090 --> 00:01:33,060
Now you cannot simply use that code.

18
00:01:35,780 --> 00:01:42,650
With a sentence, let's say vegetables are types of glands.

19
00:01:44,220 --> 00:01:46,200
This will not work.

20
00:01:50,180 --> 00:01:57,170
Let me also dangle the word in that dictionary first before I do the lemons ization.

21
00:01:59,520 --> 00:02:05,610
Ports and analytics, so you have to do that any time you restart the notebook, at least when using

22
00:02:05,610 --> 00:02:09,990
deep notes, if you're using a local Jupiter notebook them, that shouldn't be the case.

23
00:02:10,800 --> 00:02:12,810
You don't have to run that every time.

24
00:02:14,430 --> 00:02:15,330
So execute again.

25
00:02:19,370 --> 00:02:22,190
And let's check what the limo one is.

26
00:02:23,560 --> 00:02:33,190
So vegetables or types of plants, and nothing happens because this method is designed to get single

27
00:02:33,190 --> 00:02:37,360
words as inputs, not sentences.

28
00:02:37,960 --> 00:02:42,460
So we have to do something before we provides the words here.

29
00:02:42,670 --> 00:02:49,660
We have to first divide so the sentence into words, also known as tokens.

30
00:02:49,910 --> 00:02:53,080
And this process is known as tokenization.

31
00:02:53,770 --> 00:03:04,110
So let me convert this using shift command or control m I have converted into a text.

32
00:03:04,130 --> 00:03:08,380
So, so token tokenizing sentences.

33
00:03:11,330 --> 00:03:19,880
To took a nice sentence is you can use the N L T K Dot words tokenize methods.

34
00:03:20,840 --> 00:03:23,240
This expects a string as inputs.

35
00:03:23,540 --> 00:03:25,670
So that's a b this sentence.

36
00:03:25,850 --> 00:03:31,730
So I'm going to save that sentence here in a variable sentence.

37
00:03:32,270 --> 00:03:33,140
A sequel to that.

38
00:03:33,590 --> 00:03:40,430
Let me remove that part and I have to execute this cell now to record this variable either namespace.

39
00:03:41,540 --> 00:03:45,500
And this will not work for now because it's empty, so remove it.

40
00:03:46,860 --> 00:03:51,750
Tokenizing Census says this will get a census as an input.

41
00:03:52,710 --> 00:04:01,020
If you want to save the result of this worth tokenize methods in a variable, then let's to sentence

42
00:04:02,460 --> 00:04:03,090
tokens.

43
00:04:03,300 --> 00:04:05,340
So we're getting the tokens of a sentence.

44
00:04:05,950 --> 00:04:07,770
And let me bring that up for the moment here.

45
00:04:13,190 --> 00:04:19,910
Now for this, we need to download the Poom Dictionary, so let's execute that as well.

46
00:04:24,890 --> 00:04:25,460
Executes.

47
00:04:27,200 --> 00:04:28,700
Download successful.

48
00:04:29,940 --> 00:04:31,590
And so this is the result.

49
00:04:31,800 --> 00:04:35,980
So it's a powerful list made of different items.

50
00:04:36,000 --> 00:04:40,500
Each item is a string and it's contains the parts of the sentence.

51
00:04:40,910 --> 00:04:49,440
And so you could also do this splitting using the Python string method called Split.

52
00:04:49,770 --> 00:04:58,230
But this method is more appropriate because it also takes into account different punctuations of of

53
00:04:58,380 --> 00:05:01,050
language of the English language in this case.

54
00:05:01,920 --> 00:05:07,710
So it will be more accurate in giving you a set of words for a sentence.

55
00:05:08,190 --> 00:05:17,310
However, you should use the lower version of the sentence because that will give you all this tokens

56
00:05:17,370 --> 00:05:18,720
in lowercase.

57
00:05:18,930 --> 00:05:27,000
So of the lowercase version, and this is needed because now we will pass these inputs, these words

58
00:05:27,000 --> 00:05:34,080
into the of ties, methods and the clematis method to works on lowercase words only.

59
00:05:34,500 --> 00:05:38,130
So let's clematis each of these words for that.

60
00:05:38,130 --> 00:05:40,090
That's I think we need a for loop.

61
00:05:40,650 --> 00:05:43,230
So we need to iterate over each token.

62
00:05:43,240 --> 00:05:46,980
So for token in sentence tokens.

63
00:05:48,770 --> 00:05:59,390
And then points to the limits either instance in here and called the limits his methods, which expects

64
00:05:59,390 --> 00:06:03,710
two arguments of the first one, is the world, so the token in this case.

65
00:06:04,980 --> 00:06:12,510
That's and then that's in the next iteration and so on sort of token and the tank, which could be a

66
00:06:12,510 --> 00:06:14,190
verb, could be a noun.

67
00:06:14,640 --> 00:06:21,270
The problem is we don't know whether each token will be a noun or a verb or an adjective.

68
00:06:21,480 --> 00:06:27,320
So let's try first with an as the second argument and just print.

69
00:06:28,410 --> 00:06:32,220
Lemme just to see what's going on here.

70
00:06:32,250 --> 00:06:35,490
So Vegetable was successfully lemma ties.

71
00:06:35,490 --> 00:06:41,420
You see from vegetables, we got vegetable, but all is not limitless.

72
00:06:41,730 --> 00:06:46,230
The lemov or should be B, but it's not be here.

73
00:06:47,070 --> 00:06:55,500
So if we change to it v, then we get before the verb, but not the lemma, the correct lemma for vegetables.

74
00:06:55,890 --> 00:07:03,330
So we need to provide this in a dynamic fashion for that we need in this upper sale.

75
00:07:03,330 --> 00:07:07,590
Here, I'm going to create tags, post tags.

76
00:07:09,640 --> 00:07:15,520
For each of these tokens, Paul's means parts of speech.

77
00:07:15,850 --> 00:07:23,310
So part of speech tags, you'll see what these or you need to use analytic thought.

78
00:07:23,860 --> 00:07:31,580
Paul's tag, that's methods which expects a least so sentence tokens.

79
00:07:31,630 --> 00:07:32,920
In this case, that's one.

80
00:07:33,010 --> 00:07:33,790
That's Alice.

81
00:07:34,390 --> 00:07:40,720
And then I'm just going to print it all this out, and this is the output.

82
00:07:41,950 --> 00:07:47,380
You might get an error saying that you need to execute analytic downloads.

83
00:07:47,530 --> 00:07:54,010
Some think a data datasets will just exec events and then execute this code again if you've got that

84
00:07:54,010 --> 00:07:55,330
error instead of this list.

85
00:07:55,810 --> 00:07:57,830
So in my case, I had downloaded before.

86
00:07:58,600 --> 00:07:59,350
So no errors.

87
00:07:59,380 --> 00:08:00,190
This is the ultimate.

88
00:08:00,610 --> 00:08:06,370
So this is a list of 2+.

89
00:08:07,520 --> 00:08:10,490
Each table has two items.

90
00:08:10,790 --> 00:08:19,850
The first item is the token of input the list, so vegetables or types of plants and the periods.

91
00:08:20,660 --> 00:08:23,240
The second item is the tag.

92
00:08:24,050 --> 00:08:33,500
Now we're intercedes about the first letter of the tag, and for the first item is end, which means

93
00:08:33,500 --> 00:08:33,830
it's a.

94
00:08:34,370 --> 00:08:37,700
Vegetables is a known or is a verb.

95
00:08:38,090 --> 00:08:42,289
Types as a noun or as a preposition.

96
00:08:42,289 --> 00:08:45,110
E II is the codes for the prepositions.

97
00:08:45,440 --> 00:08:46,920
Plants is a noun.

98
00:08:46,940 --> 00:08:55,490
Also, if we had a adjectives here, we would get a four adjectives and are four adverbs.

99
00:08:55,820 --> 00:09:04,970
Now let's make use of that least of tipples in our for loop here to extract from that first letter from

100
00:09:04,970 --> 00:09:05,840
each table.

101
00:09:05,960 --> 00:09:16,910
For that, we need to iterate over both of the sentence tokens lease and also the post tax lists, so

102
00:09:16,910 --> 00:09:18,890
they have the same number of items.

103
00:09:19,670 --> 00:09:25,820
One two three four five six one two three four five six.

104
00:09:26,030 --> 00:09:32,110
They always will have the same number of items because we're talking about the same sentence tokens.

105
00:09:32,610 --> 00:09:37,310
Sue to iterate over two sequences of the same time.

106
00:09:37,310 --> 00:09:42,050
You need to wrap them into parentheses and use ZIP.

107
00:09:42,680 --> 00:09:50,870
And then you need two variables for token force tag in zip census tokens, post tags, and then we do

108
00:09:50,870 --> 00:09:51,440
the rest.

109
00:09:52,490 --> 00:10:00,710
So in the first iteration, token will be equal to vegetables and post tag will be equal to vegetable.

110
00:10:01,160 --> 00:10:06,800
So then we can put vegetables here and then in here.

111
00:10:07,520 --> 00:10:12,350
Instead of that, we put that dynamic POS tag.

112
00:10:12,590 --> 00:10:18,710
But be careful here because post tag is the tip of the entire table.

113
00:10:19,520 --> 00:10:27,650
What we want is the second item of the table, so we see one there the index one, which gives us that

114
00:10:27,650 --> 00:10:32,720
string and then we want the first item of that string.

115
00:10:32,990 --> 00:10:39,110
So we only want to be and therefore we say zero here and we want the end to be lowercase.

116
00:10:39,110 --> 00:10:40,640
So we say that's a lower.

117
00:10:41,880 --> 00:10:49,650
That is equal to lower case, and in the first iteration, in the second iteration, it would be a lower

118
00:10:49,650 --> 00:10:51,840
case V and so on.

119
00:10:52,620 --> 00:11:00,570
And then we print the limo, so I execute that and I get this output vegetable.

120
00:11:01,140 --> 00:11:10,440
So vegetables was converted into vegetable, vegetables to vegetable or to be so far, so good types

121
00:11:11,070 --> 00:11:12,120
to type.

122
00:11:12,960 --> 00:11:16,740
And then off is not converted at all.

123
00:11:16,770 --> 00:11:18,420
We get this error key error.

124
00:11:19,300 --> 00:11:22,110
And if you see the error, the entire error.

125
00:11:24,990 --> 00:11:30,840
You find, though, that the reason for this is or doesn't have a limo.

126
00:11:31,230 --> 00:11:35,820
So this methods here will not work for prepositions.

127
00:11:36,240 --> 00:11:44,970
Therefore, we need to make this more intelligence and check if course tags.

128
00:11:46,320 --> 00:11:47,430
So let me copy that.

129
00:11:49,370 --> 00:11:59,270
Events in a lease of an and V and a four adjectives and all four adverbs.

130
00:11:59,750 --> 00:12:08,720
So if the proposed tax is one of those because only these have limos, not I, for example.

131
00:12:10,190 --> 00:12:13,010
Only in that case, we want to find out the limo.

132
00:12:15,790 --> 00:12:20,500
And prince of the limo else we just print out.

133
00:12:22,330 --> 00:12:28,060
The token, as it is, if I execute that now, we will get this.

134
00:12:28,420 --> 00:12:39,850
So vegetable b type of plant and the periods, of course, if you don't want this prepositions and the

135
00:12:39,850 --> 00:12:46,660
period, which I think you should then include them in comparisons because they don't have any real

136
00:12:46,660 --> 00:12:47,320
meanings.

137
00:12:47,320 --> 00:12:54,700
So usually when you work with natural language processing, you want to work with verbs and nouns and

138
00:12:54,700 --> 00:12:56,620
adjectives and adverbs.

139
00:12:57,130 --> 00:13:06,160
So I don't think I would say that you can just leave out those and just work with the words that can

140
00:13:06,160 --> 00:13:06,770
be limited.

141
00:13:07,870 --> 00:13:09,760
So vegetable b type of plant.

142
00:13:10,300 --> 00:13:12,400
Now, of course, these are printed out here.

143
00:13:12,850 --> 00:13:21,550
So if you want to save them in a lease, then you want to create here a list such a sentence limbless

144
00:13:21,970 --> 00:13:24,820
equal to an empty list event instead of printing that out.

145
00:13:25,060 --> 00:13:26,650
You can say sentence.

146
00:13:28,440 --> 00:13:35,610
Limos, lots of pens and the pen, the limo there and then exits all the loops and.

147
00:13:37,350 --> 00:13:40,540
Cole's sentence, Lamb was very bold.

148
00:13:41,550 --> 00:13:50,700
Then you get the results in a sentence form, so we can then modify this code to compare two sentences

149
00:13:51,060 --> 00:13:53,820
or even better puts it in a function.

150
00:13:54,660 --> 00:13:56,070
So let me create a function.

151
00:13:57,740 --> 00:14:02,760
Define Lemo me.

152
00:14:03,620 --> 00:14:08,030
Let's say like that's this will get us inputs a sense of variable.

153
00:14:08,240 --> 00:14:14,780
We choose a sentence and then we need to tokenized that sentence in that function.

154
00:14:15,080 --> 00:14:16,280
So I'll get that.

155
00:14:17,640 --> 00:14:24,150
And cockpits in here, so we get sense you sent the stock as a sequel to that.

156
00:14:24,390 --> 00:14:31,680
So this should be sentinel the function will gets a sentence, wants to shrink as inputs, it will convert

157
00:14:31,680 --> 00:14:36,750
it into sensors, tokens, then I'll indent that on over.

158
00:14:36,760 --> 00:14:41,100
Function sends us lemons for token POSTECH and supercenters tokens.

159
00:14:41,430 --> 00:14:44,940
So that's one that's a local variable is here.

160
00:14:45,390 --> 00:14:49,890
We do that sentence limits of return of that.

161
00:14:52,290 --> 00:15:01,290
Sentence lemons and executes the function definition, and let's call the function lemon me.

162
00:15:03,600 --> 00:15:08,790
And so what was this sentence, let's try that sentence.

163
00:15:15,680 --> 00:15:24,380
Executes and so we get the tokens of that sentence, if we try to change that or save, that's into

164
00:15:24,380 --> 00:15:27,320
a variable L1, right?

165
00:15:27,560 --> 00:15:39,650
L1 is that and use the function again L2 with another input string Lemma me, is the name of the function.

166
00:15:40,610 --> 00:15:43,610
And this time, let's say a.

167
00:15:46,050 --> 00:15:49,560
Vegetable is a type of plant.

168
00:15:51,930 --> 00:16:00,720
Out to here, it seems we didn't get the correct results because we have this variable here, which

169
00:16:00,720 --> 00:16:08,340
is also a global variables, so things are getting messed up in here and this is a case sometimes with

170
00:16:08,340 --> 00:16:09,270
Jupyter notebooks.

171
00:16:09,450 --> 00:16:12,450
So I'm going to refresh this page.

172
00:16:12,630 --> 00:16:13,140
You can.

173
00:16:13,350 --> 00:16:16,080
If you are locally, you can restart your notebook.

174
00:16:16,980 --> 00:16:24,840
And then what I'll do is I'll just execute this function only, of course, for import analytic.

175
00:16:25,320 --> 00:16:32,250
So I'll execute only this so that I don't get that sense as tokens variable in the namespace.

176
00:16:32,820 --> 00:16:37,260
Then execute that and want is vegetable B type plants.

177
00:16:37,620 --> 00:16:43,380
L2 is are we got this results, a vegetable type.

178
00:16:43,590 --> 00:16:50,670
So it seems it's not working because of post tanks or what creates.

179
00:16:50,670 --> 00:16:54,360
It's in here for this tokens.

180
00:16:54,540 --> 00:16:59,670
So we want to also get that post tanks and puts it.

181
00:17:01,370 --> 00:17:10,790
On the census tokens, so we create census tokens in here and then we create using those census tokens,

182
00:17:10,790 --> 00:17:15,859
we create post tags and then use those post tags in here.

183
00:17:16,790 --> 00:17:25,400
So before so it didn't work because post-tax was being used with a previous census since it was defined

184
00:17:25,400 --> 00:17:25,940
globally.

185
00:17:26,119 --> 00:17:31,700
So with this census now, it should work, but you need to execute the function again.

186
00:17:34,470 --> 00:17:37,770
And then execute that and then exec events.

187
00:17:39,030 --> 00:17:43,900
Now we see that the tokens are the same, so we can compare this to.

188
00:17:46,570 --> 00:17:50,530
Variables, if they are the same, of course, and we get true.

189
00:17:52,650 --> 00:17:59,970
So that is how we compare to sentences using natural language processing.

190
00:18:00,450 --> 00:18:05,820
Of course, this is an exact comparison, as I mentioned in the beginning of the video.

191
00:18:06,390 --> 00:18:12,310
This doesn't give us a percentage and a coefficient of similarity for that.

192
00:18:12,330 --> 00:18:20,310
We're going to compare two sentences in the next video and gets a coefficient of similarity.

193
00:18:20,700 --> 00:18:22,590
And then we can put a threshold.

194
00:18:22,590 --> 00:18:28,950
We can see if this coefficient is greater than this than this sentences are the same and so on.

195
00:18:29,220 --> 00:18:30,720
So I'll talk to you in the next video.