0 1 00:00:01,020 --> 00:00:01,320 All right. 1 2 00:00:01,320 --> 00:00:03,130 So here's the solution to the challenge. 2 3 00:00:03,240 --> 00:00:09,320 So the first thing that we need to do is we need to import some frameworks and we're going to be using CoreML, 3 4 00:00:09,360 --> 00:00:11,630 as well as Vision. 4 5 00:00:11,670 --> 00:00:13,330 Apple's machine learning framework. 5 6 00:00:13,350 --> 00:00:19,980 And the next thing we're going to do is to create a method called detect that takes a single parameter 6 7 00:00:20,070 --> 00:00:21,610 called image. 7 8 00:00:22,020 --> 00:00:25,810 And this is going to be in the format of a CIImage, 8 9 00:00:25,830 --> 00:00:28,070 so a Core Image Image if you remember. 9 10 00:00:28,080 --> 00:00:34,260 So in order to pass our userPickedImage into this method, we're going to have to convert our 10 11 00:00:34,350 --> 00:00:34,780 userPickedImage. 11 12 00:00:34,800 --> 00:00:40,500 So let's create something called ciImage and use it to store the results of our CIImage overload 12 13 00:00:40,590 --> 00:00:43,770 that turns a UIImage into CIImage. 13 14 00:00:43,890 --> 00:00:49,230 And as you can see, Xcode is very unhappy with this because, at the moment, if you have a look at this 14 15 00:00:49,230 --> 00:00:52,880 userPickedImage, its data type is of optional Any? 15 16 00:00:53,100 --> 00:00:59,530 So we have to convert that into the data type that is expected in this method which is a UIImage. 16 17 00:00:59,670 --> 00:01:02,800 When we try to display this image inside the imageView, 17 18 00:01:02,880 --> 00:01:09,530 we downcasted it into a UIImage, so we can actually do this at a higher level by moving this up here. 18 19 00:01:09,690 --> 00:01:14,970 So we're saying that let the userPickedImage be of data type UIImage, and we can wrap this whole thing 19 20 00:01:15,000 --> 00:01:18,180 inside an "if let" just to be ultra safe. 20 21 00:01:18,180 --> 00:01:25,190 All right. So, now we're saying if the user did indeed picked an image and it can be converted into a 21 22 00:01:25,200 --> 00:01:30,960 UIImage, then we're going to call that userPickedImage, and then we're going to convert it into 22 23 00:01:30,960 --> 00:01:31,840 a CIIMAGE. 23 24 00:01:32,160 --> 00:01:36,690 Now, I'm going to use a guard statement to protect the CIImage. 24 25 00:01:36,690 --> 00:01:41,860 So guard let this statement to try and convert our UIImage into a CIImage. 25 26 00:01:41,970 --> 00:01:45,670 But if it fails, then I'm going to throw a fatalError. 26 27 00:01:46,410 --> 00:01:53,580 And the message is going to say, "Cannot convert to CIImage," so that we'll know what's going on. 27 28 00:01:53,610 --> 00:02:00,360 But if it did succeed, then we should be able to call this method detect passing in this newly created 28 29 00:02:00,360 --> 00:02:01,260 CIImage. 29 30 00:02:01,290 --> 00:02:02,370 So let's do that. 30 31 00:02:06,580 --> 00:02:11,010 And just check to make sure that your casing is the same as this one. 31 32 00:02:11,200 --> 00:02:18,890 Alternatively, you could just call it something like converted CIImage, and that will make it a little 32 33 00:02:18,890 --> 00:02:21,540 bit more explicit when you're selecting it here. 33 34 00:02:21,890 --> 00:02:22,490 All right. Cool. 34 35 00:02:22,490 --> 00:02:25,400 So we've let the user pick an image. 35 36 00:02:25,400 --> 00:02:32,280 We've converted it into a CIImage, and then we are passing it into this method called detect. 36 37 00:02:32,540 --> 00:02:37,550 So at this stage, our image is right here and we're going to do some funky stuff with it in order to 37 38 00:02:37,560 --> 00:02:40,460 detect what is the flower that's in the image. 38 39 00:02:40,460 --> 00:02:46,370 So the first thing that we need to do is to use VN CoreML model to create a vision container for 39 40 00:02:46,370 --> 00:02:47,890 our mlmodel. 40 41 00:02:47,900 --> 00:02:50,110 So what is our model called? 41 42 00:02:50,130 --> 00:02:55,560 So let's have a look at this class and you can see it's called FlowerClassifier and it has this variable 42 43 00:02:55,590 --> 00:03:00,200 called model, and that's what we're going to tap into and we're going to create a vision container for 43 44 00:03:00,200 --> 00:03:00,590 it. 44 45 00:03:00,590 --> 00:03:11,350 So let the model = VNCoreMLModel for model, which is called 45 46 00:03:11,480 --> 00:03:15,240 FlowerClassifier().model. 46 47 00:03:15,260 --> 00:03:18,600 And this, of course, is something that can throw an error. 47 48 00:03:18,830 --> 00:03:26,900 So let's market with a "try" and use a "guard" statement to protect the cases where it might fail, in which 48 49 00:03:26,900 --> 00:03:32,410 case, we'll throw a fatalError and we'll say, "Cannot import model." 49 50 00:03:32,540 --> 00:03:34,850 So we've created a vision container for our model. 50 51 00:03:34,850 --> 00:03:41,270 The next thing we need to do is to create a request, so we'll call it request, and it's going to be 51 52 00:03:41,270 --> 00:03:44,350 done using VNCoreMLRequest 52 53 00:03:49,010 --> 00:03:57,710 and it is going to take a model which is our newly created model up there, and it's also going to have a 53 54 00:03:57,710 --> 00:04:03,650 completion handler. So we're going to call the request that comes back, request, and the error, we're going 54 55 00:04:03,650 --> 00:04:08,260 to call error. And we're just going to delete that code placeholder for now. 55 56 00:04:08,540 --> 00:04:10,880 Okay, so we've got a model. We've got our request. 56 57 00:04:10,910 --> 00:04:14,290 We just need to create the handler to process that request. 57 58 00:04:14,330 --> 00:04:16,650 So let's call that handler, handler. 58 59 00:04:17,150 --> 00:04:23,140 And we're going to use VNImageRequestHandler to create it. 59 60 00:04:23,270 --> 00:04:31,520 And it's simply going to take a CIImage as the input which is, of course, this image that got passed 60 61 00:04:31,520 --> 00:04:32,280 into the method. 61 62 00:04:32,300 --> 00:04:33,950 So it's just called image. 62 63 00:04:33,960 --> 00:04:34,490 All right. 63 64 00:04:34,700 --> 00:04:43,820 And then we're going to say händler.perform request and the request is in the format of a array 64 65 00:04:44,060 --> 00:04:51,050 of VNCoreMLRequest, and it is going to equal this request that we created up here. 65 66 00:04:51,200 --> 00:04:53,510 So this line of code can also throw an error. 66 67 00:04:53,690 --> 00:05:00,680 So let's market with a try and enclose it in a "do catch" block, so that we can actually log the error 67 68 00:05:00,770 --> 00:05:06,580 if there was one. 68 69 00:05:06,580 --> 00:05:07,440 All right. 69 70 00:05:07,570 --> 00:05:16,780 Final step is once this request has completed, then our completion handler is going to activate and we're 70 71 00:05:16,780 --> 00:05:20,460 going to get back a request or an error if there was a problem. 71 72 00:05:20,620 --> 00:05:26,130 So we're going to do something with that request, namely, we're going to process it and look for where 72 73 00:05:26,130 --> 00:05:28,450 that image was classified as. 73 74 00:05:28,460 --> 00:05:37,360 So I'm going to call it classification and I'm going to set it to equal the request.results?. first, 74 75 00:05:37,780 --> 00:05:45,370 and this is going to be cast as? VNClassificationObservation. 75 76 00:05:45,400 --> 00:05:50,070 Now, the classification has a property called identifier. 76 77 00:05:50,140 --> 00:05:54,820 And this is a string that describes what the classification was. 77 78 00:05:54,820 --> 00:06:01,480 So if it was a flower, it might be rose, or it might be Barberton daisy, or whatever it may be. But this is 78 79 00:06:01,480 --> 00:06:05,350 what we want to put at the top of our navigation controller. 79 80 00:06:05,590 --> 00:06:08,570 So let's tap into our navigation bar. 80 81 00:06:08,740 --> 00:06:19,570 So self.navigationItem.title and that is going to be set to equal the classification identifier. 81 82 00:06:19,570 --> 00:06:21,750 So let's put that over here. 82 83 00:06:23,340 --> 00:06:29,280 And that's all the code that we need to write in order to use our FlowerClassifer model to detect what 83 84 00:06:29,280 --> 00:06:32,610 flower might be shown on the image that the user took. 84 85 00:06:32,610 --> 00:06:34,020 So let's give it a spin. 85 86 00:06:34,030 --> 00:06:35,110 All right, so here's our app. 86 87 00:06:35,130 --> 00:06:41,850 I'm going to tap the camera button and I'm going to take a photo of our flower and I'm going to hit 87 88 00:06:41,910 --> 00:06:43,110 Use Photo. 88 89 00:06:43,230 --> 00:06:47,100 Now, it's going to process that image and give us the result. 89 90 00:06:47,100 --> 00:06:50,340 So it's telling us that this is a Barberton daisy 90 91 00:06:50,490 --> 00:06:54,000 and, although, I have to say my knowledge of flowers is pretty limited, 91 92 00:06:54,090 --> 00:06:56,020 so this app is really useful for me. 92 93 00:06:56,130 --> 00:07:01,590 But I've Googled what Barberton daisy looks like and I think this is the right one. 93 94 00:07:01,680 --> 00:07:04,620 So there's a couple of UI things that are a little bit niggling. 94 95 00:07:04,620 --> 00:07:08,090 Firstly, why does our image look so odd on screen? 95 96 00:07:08,310 --> 00:07:14,020 Well, remember that we allowed the user to edit the image and that cropped it into a square crop, 96 97 00:07:14,070 --> 00:07:16,440 so a 1x1 scale factor image. 97 98 00:07:16,440 --> 00:07:18,890 Now, our screen is actually a rectangle. 98 99 00:07:18,900 --> 00:07:25,620 So in order for this to look a little bit more decent, you can change the image views content mode to 99 100 00:07:25,680 --> 00:07:28,980 Aspect Fit or Aspect Fill whichever you prefer. 100 101 00:07:29,300 --> 00:07:36,420 And the other thing that is kind of odd is that why is it the Barberton Dezi classification is all in 101 102 00:07:36,450 --> 00:07:37,300 lowercase. 102 103 00:07:37,410 --> 00:07:41,770 Ideally, I would want it to say capital "B" and capital "D" for Barberton Daisy, 103 104 00:07:41,820 --> 00:07:44,110 especially when it's displayed on the nav bar. 104 105 00:07:44,190 --> 00:07:50,070 So the reason is because if you have a look in our flower labels all of these are actually all lowercase. 105 106 00:07:50,220 --> 00:07:52,790 But we have a neat trick in Swift that we can change that. 106 107 00:07:52,830 --> 00:07:58,140 So here where we're setting the navigation title to the classification.identifier string. 107 108 00:07:58,140 --> 00:08:01,050 We can add another dot and we can say "capitalized" 108 109 00:08:01,050 --> 00:08:04,870 and it turns it into a capitalized representation of the string. 109 110 00:08:05,280 --> 00:08:15,630 And now, if you run the app again, you can see that it looks much better. 110 111 00:08:15,650 --> 00:08:15,940 All right. 111 112 00:08:15,950 --> 00:08:17,330 So looks a little bit nicer. 112 113 00:08:17,330 --> 00:08:22,640 And also you can see that the result is now capitallized which looks a lot nicer. 113 114 00:08:22,640 --> 00:08:23,020 All right. 114 115 00:08:23,030 --> 00:08:23,480 Awesome. 115 116 00:08:23,480 --> 00:08:29,210 So we've now managed to make our app use our custom image classification model that was converted from 116 117 00:08:29,300 --> 00:08:34,930 a open-source Caffe model, and we're able to start classifying all the flowers that are around us. 117 118 00:08:34,940 --> 00:08:36,320 So if you haven't coded along, 118 119 00:08:36,330 --> 00:08:39,640 give it a spin and check out this awesome app. 119 120 00:08:39,770 --> 00:08:44,570 But in the next lesson, we're going to make this app even more awesome because we're going to cross-match 120 121 00:08:44,750 --> 00:08:51,020 the result that we got back from our image classification against Wikipedia to bring up a description 121 122 00:08:51,140 --> 00:08:56,690 and a useful snippet of information about the flower, as well as an image of the flower on Wikipedia. 122 123 00:08:56,690 --> 00:08:58,570 So you've got all of that to look forward to. 123 124 00:08:58,640 --> 00:09:00,410 And I'll see you on the next lesson.