0 1 00:00:00,600 --> 00:00:00,960 All right. 1 2 00:00:00,960 --> 00:00:05,750 So, now that we've gotten our UIImagePickerController set up and ready to go, 2 3 00:00:05,910 --> 00:00:10,460 and we've also incorporated our Inception v3 machine learning model, 3 4 00:00:10,560 --> 00:00:14,550 we're ready to start getting some interpretations using that model. 4 5 00:00:14,610 --> 00:00:19,890 So the first thing I'm going to do is I'm going to go just below where we set the image views image 5 6 00:00:19,890 --> 00:00:22,590 property to the image that the user picked, 6 7 00:00:22,620 --> 00:00:32,100 we're going to convert this UIImage into a CIImage which stands for Core Image image and that's a special 7 8 00:00:32,100 --> 00:00:39,090 type of image that allow us to use the Vision framework and the CoreML framework in order to get 8 9 00:00:39,120 --> 00:00:45,650 an interpretation from it. To do that, all we need to write is let ciimage, 9 10 00:00:45,660 --> 00:00:47,190 so we'll just call it ciimage, 10 11 00:00:47,220 --> 00:00:49,180 you can call it whichever you like, 11 12 00:00:49,460 --> 00:01:00,090 equal Core Image image, open bracket, and the image that we're going to be using to process is that 12 13 00:01:00,090 --> 00:01:02,360 userPickedImage that we got from earlier on. 13 14 00:01:02,430 --> 00:01:09,240 So the code as it is is fine and it will work, but I'm going to add a little security feature in order 14 15 00:01:09,240 --> 00:01:12,120 to make it a little bit safer. 15 16 00:01:12,150 --> 00:01:19,440 So what I'm going to do is I'm going to add a "guard" statement in front of the "let," and then I'm going 16 17 00:01:19,440 --> 00:01:29,670 to write an "else" statement that will trip if this part fails, i.e., if it's unable to convert our UIImage 17 18 00:01:29,730 --> 00:01:33,070 that the user picked into a CIImage. 18 19 00:01:33,090 --> 00:01:38,480 Now, that shouldn't really happen, but if it does, we want to trigger a fatalError 19 20 00:01:38,670 --> 00:01:47,550 and we also want to write a message saying something like, "Could not convert to CIImage," so that we 20 21 00:01:47,550 --> 00:01:49,650 know exactly what's happening. 21 22 00:01:49,650 --> 00:01:55,170 All right. So, now that we've converted our UIImage into a CIImage, we're going to create a method 22 23 00:01:55,410 --> 00:02:01,900 that will process that CIImage and get an interpretation or classification out of it. 23 24 00:02:01,920 --> 00:02:06,450 So I'm going to write this just above my cameraTapped IBAction. 24 25 00:02:06,510 --> 00:02:12,990 I'm going to create a new method and the method is going to be called detect and it's going to take 25 26 00:02:13,080 --> 00:02:20,910 only one parameter and the parameter is going to be called image and the data type will be a CIImage. 26 27 00:02:21,720 --> 00:02:28,800 And what I'm going to do inside this method is I'm going to use the Inception v3 model. 27 28 00:02:28,890 --> 00:02:34,920 So all you have to do is to create a new object, going to be called model, and we're going to use the 28 29 00:02:35,280 --> 00:02:44,140 and VNCoreMLModel as a container for our mlmodel and our model is called Inceptionv3. 29 30 00:02:44,370 --> 00:02:49,920 So in this part where you've got a placeholder for mlmodel, you're going to write the model that was 30 31 00:02:49,920 --> 00:02:54,870 generated when we dragged and dropped in this Inceptionv3 mlmodel. 31 32 00:02:54,900 --> 00:03:00,270 So you can see it's already autosuggesting, but you can also find out if you just click on this little 32 33 00:03:00,330 --> 00:03:07,650 arrow here and you can see that our class that was generated is called Inceptionv3 and it's got a property 33 34 00:03:07,650 --> 00:03:09,320 called model. 34 35 00:03:09,390 --> 00:03:18,240 So, now we've got Inceptionv3 creating a new object and we're tapping into its model property. So, now 35 36 00:03:18,420 --> 00:03:25,740 we have created a object called model using the VNCoreMLModel container and creating a new 36 37 00:03:25,770 --> 00:03:31,770 object of Inceptionv3 and getting its model property loaded up. 37 38 00:03:31,770 --> 00:03:39,120 So, basically, this model is now what we're going to be using to classify our image. 38 39 00:03:39,120 --> 00:03:44,970 Now, you'll see that if you write this line, you're going to get an error, and it's saying that this line 39 40 00:03:44,970 --> 00:03:49,890 of code can throw an error and we're not handling it at all. 40 41 00:03:49,890 --> 00:03:52,340 So let's go ahead and handle that by typing 41 42 00:03:52,360 --> 00:03:55,230 "try?" in front of this. 42 43 00:03:55,260 --> 00:04:03,540 So when we add "try?" it's going to attempt to perform this operation that might throw an error. If this 43 44 00:04:03,540 --> 00:04:08,190 operation succeeds, then the result is going to be wrapped as a optional. 44 45 00:04:08,220 --> 00:04:15,770 But if it fails, i.e., if there was an error that was thrown, then the result of this line will be nil. 45 46 00:04:15,810 --> 00:04:23,280 So to guard against those situations where this fails and we end up getting a model that is nil, we don't 46 47 00:04:23,280 --> 00:04:27,420 want to do any more image processing with something that with a nil model. 47 48 00:04:27,420 --> 00:04:34,470 So if our app does crash under those conditions, we want to know exactly why it did, so we want to send 48 49 00:04:34,470 --> 00:04:37,280 an error message to our debug console. 49 50 00:04:37,440 --> 00:04:41,780 So to do this, I'm going to wrap everything inside a "guard" statement. 50 51 00:04:41,790 --> 00:04:50,820 So, now instead of "if let," I've got "guard let." And this means that if my model is nil, then I'm going to 51 52 00:04:50,820 --> 00:05:00,720 trigger an "else" statement that is going to throw a fatalError, giving me the message that, maybe, "Loading 52 53 00:05:00,930 --> 00:05:10,070 CoreML Model failed." That way I'll get it printed in my deep console and I'll know exactly which part 53 54 00:05:10,070 --> 00:05:11,390 of my code failed. 54 55 00:05:11,510 --> 00:05:17,390 So, it's basically a way of allowing your app to exit, giving you plenty of information about what caused 55 56 00:05:17,390 --> 00:05:19,390 it to exit or to crash. 56 57 00:05:19,460 --> 00:05:25,310 And in fact, if you think about it, in this case, we could probably also benefit from wrapping it inside 57 58 00:05:25,370 --> 00:05:32,960 a "guard" statement because in the cases where we can actually convert our UIImage into a CIImage successfully, 58 59 00:05:33,260 --> 00:05:36,640 then we could end up crashing our app without knowing what caused it. 59 60 00:05:36,680 --> 00:05:38,000 So let's do the same. 60 61 00:05:38,000 --> 00:05:44,190 So if you remember how to add the "guard" statement and how it worked, then pause the video and add it now. 61 62 00:05:44,210 --> 00:05:51,050 So we want to throw a fatalError telling us in the debug console that we couldn't convert the UIImage 62 63 00:05:51,080 --> 00:05:52,640 into a CIImage. 63 64 00:05:53,670 --> 00:05:55,290 Now, if you don't remember how to do that, 64 65 00:05:55,290 --> 00:05:56,910 I'm going to show it to you now. 65 66 00:05:57,000 --> 00:06:01,810 So instead of simply "let," we're going to call it "guard let," 66 67 00:06:01,980 --> 00:06:09,660 and it's going to evaluate this statement. If it's true, then it's simply going to do nothing. But if it's false, 67 68 00:06:09,720 --> 00:06:13,930 then we're going to trigger this "else" statement. 68 69 00:06:14,040 --> 00:06:15,330 So instead of an is-- 69 70 00:06:15,470 --> 00:06:22,140 So instead of an "if else," it's a "guard else. And in here, we're going to throw a fatalError 70 71 00:06:22,410 --> 00:06:32,580 that tells us that we could not convert UIImage into a CIImage. 71 72 00:06:33,080 --> 00:06:41,390 So, now if our app crashes because this failed, we'll know exactly what happens because we have that message 72 73 00:06:41,510 --> 00:06:42,040 in there. 73 74 00:06:42,170 --> 00:06:46,730 So you might be wondering where this VNCoreMLModel comes from. 74 75 00:06:46,730 --> 00:06:52,790 Well, it actually comes from the vision framework. It allows us to perform an image analysis requests 75 76 00:06:53,120 --> 00:06:57,260 that uses our CoreMLModel to process images. 76 77 00:06:57,290 --> 00:07:03,700 So the next thing that we're going to do is we're actually going to use this model that we're certain 77 78 00:07:03,710 --> 00:07:08,110 exists because this part didn't trip and our app didn't exit. 78 79 00:07:08,150 --> 00:07:12,550 So the next part will only happen if this was not nil. 79 80 00:07:12,770 --> 00:07:16,480 So here, we're going to create a Vision CoreMLRequest. 80 81 00:07:16,500 --> 00:07:23,450 So we're going to call it request and it's going to equal VNCoreMLRequest 81 82 00:07:23,480 --> 00:07:26,320 and we're going to use the one that has a completion handler. 82 83 00:07:26,400 --> 00:07:31,460 We're going to hit into the model that we're going to be using is, of course, this model that we created 83 84 00:07:31,520 --> 00:07:32,250 earlier on. 84 85 00:07:32,480 --> 00:07:39,290 And if you now just click on this VNRequestCompletionHandler? placeholder and hit enter, 85 86 00:07:39,320 --> 00:07:42,480 it'll insert the code automatically for you. 86 87 00:07:42,530 --> 00:07:46,250 So, now as a part of our completion handler, we're going to get two things back. 87 88 00:07:46,250 --> 00:07:48,010 One is the request. 88 89 00:07:48,290 --> 00:07:50,690 So we're going to call it simply request. 89 90 00:07:50,690 --> 00:07:51,980 The next thing is an error, 90 91 00:07:51,980 --> 00:07:58,090 so we're going to call this error. And the code that we want to happen inside this completion block, i.e., 91 92 00:07:58,100 --> 00:08:05,220 when that request has completed is to process the results of that request. 92 93 00:08:05,270 --> 00:08:10,280 So to do that, we're going to create an object called results, 93 94 00:08:10,280 --> 00:08:14,270 so let results = request. 94 95 00:08:14,510 --> 00:08:18,570 So that comes from the completion handler callback .results. 95 96 00:08:18,590 --> 00:08:24,920 And as you can see at the moment, its data type is an array of any objects. 96 97 00:08:24,920 --> 00:08:27,080 So we're going to make that little bit more specific 97 98 00:08:27,080 --> 00:08:36,770 by, again, using the "as?" and instead of being an array of any data type, we're going to make 98 99 00:08:36,770 --> 00:08:40,480 it an array of VNClassificationObservations. 99 100 00:08:40,640 --> 00:08:46,360 And this is a class that holds classification observations after our models been processed. 100 101 00:08:46,430 --> 00:08:53,860 So, again, we want to use a guard to make sure that if this fails, our code exits, and we know why it failed. 101 102 00:08:53,990 --> 00:08:59,530 So go ahead and wrap this line inside a "guard" statement as we did before. 102 103 00:08:59,660 --> 00:09:02,370 Pause the video and try it out. 103 104 00:09:02,390 --> 00:09:02,720 All right. 104 105 00:09:02,720 --> 00:09:07,910 So as we did before, all we need to do is add the "guard" in front of the "let." 105 106 00:09:08,050 --> 00:09:13,660 I'm going to add an "else" statement that addresses what should happen when that fails. 106 107 00:09:14,180 --> 00:09:22,340 So, again, we're going to trigger a fatalError and the message is going to say, maybe, "Model failed to 107 108 00:09:22,370 --> 00:09:23,290 process 108 109 00:09:23,360 --> 00:09:24,670 image." 109 110 00:09:24,920 --> 00:09:30,770 But if it succeeds, then we should have this object called results which we should be up to print out 110 111 00:09:30,770 --> 00:09:33,680 into the console in order to see what results we got back. 111 112 00:09:34,970 --> 00:09:39,790 The last thing that we need to do is to actually perform this request. 112 113 00:09:39,800 --> 00:09:46,520 So you might notice that this request has a model associated with it, but it doesn't actually know which 113 114 00:09:46,580 --> 00:09:51,440 image to perform that classification request on. 114 115 00:09:51,440 --> 00:09:53,240 So this is what we're going to address now. 115 116 00:09:53,240 --> 00:09:57,640 We're going to create a handler that specifies the image we want to classify. 116 117 00:09:57,800 --> 00:10:07,400 So I'm going to write let händler = VNImageRequestHandler and it's going to take a ciimage, 117 118 00:10:08,180 --> 00:10:16,060 and the image that we're going to input in here is the one that was passed into this method as a parameter, 118 119 00:10:16,100 --> 00:10:21,950 so the CIImage that we're going to pass in, and that image is going to come from, of course, over here. 119 120 00:10:22,130 --> 00:10:25,380 So we're going to call that method in here a little bit later on. 120 121 00:10:25,400 --> 00:10:31,670 So the image that the user picked gets converted into ciimage, then gets passed into this method 121 122 00:10:32,060 --> 00:10:39,740 as a CIImage, and then gets put into this handler to specify that it's the one that we want to classify 122 123 00:10:40,040 --> 00:10:42,290 using our machine learning model. 123 124 00:10:42,320 --> 00:10:49,330 So I'm going to use the parameter name, of course, which is image, and there is my image handler, 124 125 00:10:49,480 --> 00:10:59,600 and now all I need to do is just to write try using my handler to perform the request which is, of course, 125 126 00:10:59,690 --> 00:11:02,590 this request that we created early on. 126 127 00:11:02,630 --> 00:11:08,780 So this line of code will work as long as you don't get any errors here. 127 128 00:11:09,050 --> 00:11:12,890 Now, as you can see, we've got the exclamation mark after the try 128 129 00:11:12,920 --> 00:11:15,760 which means that we're forcing it to execute this line. 129 130 00:11:15,890 --> 00:11:22,910 But instead, the safer way to do this would be to wrap this execution inside a "do catch" block. 130 131 00:11:23,120 --> 00:11:25,170 So if you don't remember from previously, 131 132 00:11:25,190 --> 00:11:32,480 all you need to do is write the keyword "do," and then enclose whatever you want to try 132 133 00:11:32,690 --> 00:11:39,740 inside the do block. And then you have another keyword called "catch" which catches any errors that are thrown. 133 134 00:11:39,950 --> 00:11:45,380 And if there are any errors, then we want to print them into the console. And that's all you need to do 134 135 00:11:45,470 --> 00:11:48,950 in order to use a machine learning model to classify your image. 135 136 00:11:48,950 --> 00:11:54,770 So the last thing we need to do before we can run our app and test it out is that we actually need to 136 137 00:11:54,770 --> 00:11:59,600 call this beautiful method that we've just created. At the moment, 137 138 00:11:59,600 --> 00:12:00,970 it's not called anywhere, 138 139 00:12:01,040 --> 00:12:04,410 and we won't get any results back if we run it now. 139 140 00:12:04,610 --> 00:12:12,200 And in fact, you can see that this warning here is sort of giving you a hint as to the fact that maybe 140 141 00:12:12,530 --> 00:12:20,510 you have a bug in your code, because it's saying that this CIImage that you created was never used 141 142 00:12:20,600 --> 00:12:22,850 anywhere. It says, here's a fix, 142 143 00:12:22,850 --> 00:12:27,260 instead of giving it a name, just make it a Boolean test. 143 144 00:12:27,260 --> 00:12:28,850 But that's actually not what we want, 144 145 00:12:28,850 --> 00:12:31,200 it's onto the right problem, 145 146 00:12:31,370 --> 00:12:38,420 we haven't used this anywhere in our code, but it suggested fix is not always the right one to go for. 146 147 00:12:38,690 --> 00:12:46,760 Instead, what we want to do is we want to pass the ciimage into this method detect so that the image 147 148 00:12:46,760 --> 00:12:50,080 can be used to be classified by our model. 148 149 00:12:50,270 --> 00:12:57,050 So very simply, all we need to do is detect and the image that we want to detect is of type ciimage 149 150 00:12:57,140 --> 00:13:02,890 and that, of course, is this image that we converted from our userPickedImage. 150 151 00:13:02,990 --> 00:13:06,850 So, now if we run our app, see what happens. 151 152 00:13:06,860 --> 00:13:07,120 All right. 152 153 00:13:07,130 --> 00:13:10,320 So here, I've got the app and I'm going to tap on the camera button. 153 154 00:13:10,400 --> 00:13:15,720 I'm just going to take a picture of my keyboard and I'm going to click use photo. 154 155 00:13:15,830 --> 00:13:17,960 So this is the data that we got back. 155 156 00:13:18,050 --> 00:13:21,600 And as you can see, it looks like a pretty big mess. 156 157 00:13:21,620 --> 00:13:26,810 But if you dig into it in detail, you can see that we've got a whole bunch of these 157 158 00:13:26,870 --> 00:13:28,350 VNClassificationObservationObjects. 158 159 00:13:28,700 --> 00:13:35,540 And if we go from the top, you can see that the first one which was rated with 82 percent confidence 159 160 00:13:36,620 --> 00:13:39,300 is computer keyboard or keypad 160 161 00:13:39,350 --> 00:13:41,380 which is exactly what it is. 161 162 00:13:41,450 --> 00:13:48,410 Now the next one with low percentage confidence is maybe a typewriter keyboard which is also I can see 162 163 00:13:48,410 --> 00:13:49,420 where it's coming from. 163 164 00:13:49,490 --> 00:13:54,080 And then you've got this one with temps and confidence which is a notebook or notebook computer. 164 165 00:13:54,260 --> 00:14:01,040 So as you can see this Inceptionv3 model is incredibly accurate in predicting what that image shows. 165 166 00:14:01,040 --> 00:14:04,100 Now, the only problem is that this is a tangle of data. 166 167 00:14:04,100 --> 00:14:06,220 So we have to make some sense out of it. 167 168 00:14:06,290 --> 00:14:09,520 So, in the next lesson, that's exactly what we're going to be doing. 168 169 00:14:09,530 --> 00:14:14,630 We're going to be focusing on the task that we've set for ourselves in the beginning which is being 169 170 00:14:14,630 --> 00:14:21,140 able to take an image of anything and classify it as whether if it's a hot dog or whether if it's not 170 171 00:14:21,260 --> 00:14:22,100 a hot dog. 171 172 00:14:22,340 --> 00:14:24,400 So we're going to do that in the next lesson.