0
1
00:00:00,320 --> 00:00:00,680
All right.
1

2
00:00:00,720 --> 00:00:05,400
So we've been talking about all the fantastic things that you can do with machine learning,
2

3
00:00:05,520 --> 00:00:08,930
but now it's time to get hands on with CoreML.
3

4
00:00:08,940 --> 00:00:13,920
So what exactly is CoreML and why is it so exciting?
4

5
00:00:13,920 --> 00:00:19,500
So CoreML essentially allows you to do two things that makes it easy to integrate machine learning
5

6
00:00:19,830 --> 00:00:21,980
into your IOs project.
6

7
00:00:21,990 --> 00:00:28,830
So the first thing is that it alows you to load a pre-trained machine learning model and they include
7

8
00:00:28,830 --> 00:00:35,220
an easy way of converting your pre-trained model, whether if it was trained on Caffe or Keras, or wherever
8

9
00:00:35,220 --> 00:00:41,090
it may be. You can take that model and convert it into a class that can be used within Xcode.
9

10
00:00:41,190 --> 00:00:46,680
For those of you who are already familiar with machine learning, your pre-trained model gets converted into
10

11
00:00:46,680 --> 00:00:54,270
a .mlmodel file which is a open-file format, and includes all of your input-output layers, as well
11

12
00:00:54,270 --> 00:00:55,800
as all your training weights.
12

13
00:00:55,800 --> 00:01:01,520
Now, the second thing that CoreML allows you to do is to make predictions to have the model loaded
13

14
00:01:01,560 --> 00:01:03,440
up locally on the device
14

15
00:01:03,450 --> 00:01:09,650
once the user download your app and your app is able to use it to make the predictions, be it
15

16
00:01:09,690 --> 00:01:15,260
image recognition, speech recognition, or whatever else it is that you trained your model to do.
16

17
00:01:15,270 --> 00:01:21,120
Now, if you go to developer.apple.com/machine-learning, you can see that they compiled a
17

18
00:01:21,120 --> 00:01:27,290
website that includes an overview on how you can use machine learning in your iOS apps.
18

19
00:01:27,420 --> 00:01:32,870
But the most interesting part of this website are the ready-to-use models down here.
19

20
00:01:32,910 --> 00:01:37,860
So these are essentially plug-and-play models which already been pre-trained. But the one that we're
20

21
00:01:37,860 --> 00:01:44,040
going to be using in our project is this one from Google which is Inception v3 and it's been trained
21

22
00:01:44,100 --> 00:01:51,150
on over a thousand categories on loads and loads of images, and it's learned to recognize these images
22

23
00:01:51,420 --> 00:01:54,150
and be able to classify them.
23

24
00:01:54,240 --> 00:02:01,110
So we're going to be using this to identify what our user takes a photo of in the app and we're going
24

25
00:02:01,110 --> 00:02:05,920
to check to see if the classification was a Hotdog or if it was not a Hotdog.
25

26
00:02:05,940 --> 00:02:12,990
Now, in WWDC, they announced all of the things that CoreML is able to do, but essentially, it boils
26

27
00:02:12,990 --> 00:02:15,030
down to just two things.
27

28
00:02:15,030 --> 00:02:18,240
It's able to do classification or regression.
28

29
00:02:18,260 --> 00:02:23,550
And at the moment, it's pretty much limited to just those two types of tasks, but I'm sure in the future
29

30
00:02:23,550 --> 00:02:24,840
this will be extended.
30

31
00:02:24,900 --> 00:02:29,970
And besides in most cases, when you're implementing machine learning, you'll only be using classification
31

32
00:02:29,970 --> 00:02:31,350
or regression anyways.
32

33
00:02:31,350 --> 00:02:36,780
Now, we've spoken about some of the things that CoreML can do, but one of the most important things that
33

34
00:02:36,780 --> 00:02:43,620
it can do is that you're not able to use any of your app data or user generated data while they're using
34

35
00:02:43,620 --> 00:02:46,490
your app to train the model further.
35

36
00:02:46,500 --> 00:02:52,650
So the model is loaded up as a pre-trained model and it stays as it is. So static.
36

37
00:02:52,710 --> 00:02:58,140
And although there are workarounds in most cases, your model will just stay as it is and you'll only
37

38
00:02:58,140 --> 00:03:01,500
be able to update it when you push a new update to your app.
38

39
00:03:01,500 --> 00:03:07,290
So for the moment, anyways, you won't be able to use the user-generated data in the app to train the model
39

40
00:03:07,350 --> 00:03:08,910
on the fly.
40

41
00:03:08,910 --> 00:03:12,080
Now, the third thing is that it's not encrypted.
41

42
00:03:12,090 --> 00:03:19,350
So if you're using a proprietary model or if anything within your package contains sensitive data, then
42

43
00:03:19,350 --> 00:03:25,290
you have to be aware that none of this is encrypted. And in fact, when you convert your pre-trained model
43

44
00:03:25,590 --> 00:03:31,770
into a .mlmodel file, if you have a look within the file structure, you'll actually see, in most
44

45
00:03:31,770 --> 00:03:37,110
cases, it's split into a whole bunch of JSONS that contain the training weights and the inputs and the
45

46
00:03:37,110 --> 00:03:38,210
outputs, et cetera.
46

47
00:03:38,250 --> 00:03:44,690
So be aware that if you have, say, a financial model that has a proprietary way of predicting the stock
47

48
00:03:44,690 --> 00:03:51,060
market, for example, you might not want to use it with CoreML just because you might be giving your
48

49
00:03:51,060 --> 00:03:54,840
secret source away for anyone who cares to look into the source code of your app.
49

50
00:03:54,840 --> 00:03:57,480
So those are some of the limitations to CoreML.
50

51
00:03:57,510 --> 00:04:03,950
But remember that the most exciting thing about CoreML is that it just works out of the package
51

52
00:04:04,050 --> 00:04:05,780
like many other Apple products.
52

53
00:04:05,880 --> 00:04:11,220
And even if you don't know a whole lot about machine learning, you can use it straight away without a
53

54
00:04:11,220 --> 00:04:13,700
lot of code and without a lot of complexity.
54

55
00:04:13,710 --> 00:04:16,920
And that's exactly what we're going to show you in this tutorial.
55

56
00:04:16,920 --> 00:04:24,390
So you'll be able to take something like a pre-trained model on recognizing plants and build it into
56

57
00:04:24,390 --> 00:04:30,840
your app and start feeding photos to it and they'll be able to send you outputs that you can then display
57

58
00:04:30,840 --> 00:04:36,150
to the user all without you having to understand anything that's happening in the background.
58

59
00:04:36,150 --> 00:04:41,280
So, for example, one of my favorite apps is called Garden Answers and it's really cool because you can
59

60
00:04:41,280 --> 00:04:48,450
take a photo of a plant and it's able to classify that plant based on the way that the flower or the
60

61
00:04:48,450 --> 00:04:49,710
leaf looks.
61

62
00:04:49,710 --> 00:04:54,520
Now, my one gripe with it though is that it requires an internet connection.
62

63
00:04:54,690 --> 00:05:00,720
So very often when you're wanting to identify plants, you're either abroad or you're somewhere without
63

64
00:05:00,750 --> 00:05:02,130
internet connection,
64

65
00:05:02,220 --> 00:05:07,080
you know, really in the sticks. And at the moment, what they're doing is that they're querying an API and
65

66
00:05:07,080 --> 00:05:11,910
their model exists somewhere in the cloud which they send the image data to,
66

67
00:05:12,060 --> 00:05:15,600
and then they can get an answer back as an interpretation.
67

68
00:05:15,600 --> 00:05:22,770
But with CoreML, you can actually put that entire classifier within your app on your device, so your
68

69
00:05:22,770 --> 00:05:25,180
users won't even need an internet connection
69

70
00:05:25,380 --> 00:05:28,680
and it will be able to do it on the local device.
70

71
00:05:28,710 --> 00:05:34,350
So if you're not using one of the ready-to-use models that Apple has provided on their website, then
71

72
00:05:34,350 --> 00:05:41,850
you'll need to convert your pre-trained model into a .mlmodel file, and Apple provides a relatively
72

73
00:05:41,850 --> 00:05:49,260
easy way of doing that for all of the major frameworks, including Caffe, including Keras scikit-learn, et cetera.
73

74
00:05:49,440 --> 00:05:55,200
One of the most notable ones are missing from this, however, is TensorFlow which is something I really
74

75
00:05:55,200 --> 00:05:59,910
like. But there are open-source converters out there and it's actually not very difficult to convert
75

76
00:05:59,910 --> 00:06:03,000
your TensorFlow into a mlmodel file either.
76

77
00:06:03,000 --> 00:06:09,230
So, essentially, what I'm trying to say is that there's two routes for you to go down using CoreML.
77

78
00:06:09,240 --> 00:06:14,760
One is if you're a complete beginner to machine learning, then you can just use one of the plug-and-play
78

79
00:06:14,760 --> 00:06:20,100
models and you can start building it into your app with relative ease.
79

80
00:06:20,130 --> 00:06:24,600
Now, if you're somebody who is very serious about machine learning and you've already got your model
80

81
00:06:24,600 --> 00:06:31,620
train on Keras, or train on Caffe, wherever it may be, you can then convert it into an mlmodel file being
81

82
00:06:31,620 --> 00:06:38,650
aware of the limitations of CoreML and implementing it in your app relatively easily as well.
82

83
00:06:38,760 --> 00:06:43,590
So enough talking. We've spoken a lot about CoreML and machine learning.
83

84
00:06:43,680 --> 00:06:50,700
It's really time to get stuck in and start creating our very own CoreML app. So I'll see you on the next
84

85
00:06:50,700 --> 00:06:51,050
lesson.