0
1
00:00:00,950 --> 00:00:01,290
All right.
1

2
00:00:01,320 --> 00:00:07,790
So in the last lesson, we managed to get our 3D dice to get rendered and displayed in augmented reality.
2

3
00:00:07,800 --> 00:00:13,110
Now, the only problem is that it looks a bit weird because it's kind of just free floating in midair
3

4
00:00:13,170 --> 00:00:16,350
which is not very realistic for a solid object.
4

5
00:00:16,680 --> 00:00:20,210
So in this lesson, we're going to focus on plane detection.
5

6
00:00:20,370 --> 00:00:27,360
So we're going to be using a structure within ARKit that allows us to detect horizontal planes in
6

7
00:00:27,360 --> 00:00:29,200
the camera image that's captured.
7

8
00:00:29,280 --> 00:00:35,610
So at the moment, we can only detect horizontal planes because, as you can see, the options only include
8

9
00:00:35,610 --> 00:00:36,560
horizontal.
9

10
00:00:36,570 --> 00:00:42,420
Now, we think that in the near future, vertical plane detection is going to be added in here as an option.
10

11
00:00:42,420 --> 00:00:46,990
And I expect that the Apple engineers are probably just working on it as we speak.
11

12
00:00:47,040 --> 00:00:52,090
And that was certainly the indication that we got at WWDC.
12

13
00:00:52,200 --> 00:00:56,220
So talking to some of the engineers, they did seem to hint that this was something that was going to
13

14
00:00:56,220 --> 00:01:01,940
be enabled within a short period of time, and not something that we would have to wait for, say, iOS
14

15
00:01:02,040 --> 00:01:03,570
12 to appear.
15

16
00:01:03,570 --> 00:01:05,210
So that's really encouraging.
16

17
00:01:05,220 --> 00:01:11,190
But it does mean that, in the meantime, we won't be able to do any vertical plane detection.
17

18
00:01:11,280 --> 00:01:17,730
So all of you guys who have ideas of, you know, hanging paintings onto the wall and seeing how it looks
18

19
00:01:17,730 --> 00:01:22,890
in AR, you're going to have to hold your horses for now until they update the plane detection options
19

20
00:01:22,980 --> 00:01:29,820
to include vertical. But horizontal is good enough for our dice app because what we want to do is we
20

21
00:01:29,820 --> 00:01:36,390
want to be able to place our dice on the horizontal surface that we detect in augmented reality. So, say,
21

22
00:01:36,420 --> 00:01:38,590
if we're looking at a table or the floor,
22

23
00:01:38,670 --> 00:01:45,480
I want to be able to place my dice onto that horizontal surface and give the illusion that it is a part
23

24
00:01:45,540 --> 00:01:46,840
of the real world.
24

25
00:01:46,860 --> 00:01:50,880
So it's like when you play Pokémon GO  and all the Pokemon appear on the ground.
25

26
00:01:50,880 --> 00:01:53,370
This is what we're going to be doing in this lesson.
26

27
00:01:53,520 --> 00:01:53,780
All right.
27

28
00:01:53,790 --> 00:02:00,540
So the first thing we need to do is I'm going to comment out the part where we created our dice.
28

29
00:02:00,540 --> 00:02:05,640
I'm going to come back to that later, but for now I just want to focus on detecting the horizontal plane.
29

30
00:02:05,670 --> 00:02:10,950
So if you go into your viewWillAppear where you created your session configuration,
30

31
00:02:10,950 --> 00:02:17,820
I'm going to add another property to that configuration which is, as we saw earlier on, the plane detection
31

32
00:02:17,820 --> 00:02:18,510
property.
32

33
00:02:18,630 --> 00:02:21,560
And I'm going to set that to equal .horizontal.
33

34
00:02:21,600 --> 00:02:23,870
So as you can see, this is a enum.
34

35
00:02:23,910 --> 00:02:29,790
And if you don't remember what enums are, then have a look back at the intermediate Swift lessons where
35

36
00:02:29,790 --> 00:02:31,890
we went into that in more detail.
36

37
00:02:31,890 --> 00:02:38,700
But, again, having the .horizontal as an enum, rather than having something like configuration.horizontal
37

38
00:02:38,700 --> 00:02:45,640
planeDetection = true is another good indication, code wise that vertical is coming.
38

39
00:02:45,660 --> 00:02:46,110
All right, cool.
39

40
00:02:46,140 --> 00:02:52,140
So we've done that. We've said that let our configuration be our ARWWorldTrackingSessionConfiguration
40

41
00:02:52,380 --> 00:02:57,440
and enable horizontal plane detection on that configuration, and then run it.
41

42
00:02:57,480 --> 00:03:05,100
So the next step is we need to use one of the delegate methods from the ARSCNViewDelegate. And the
42

43
00:03:05,100 --> 00:03:08,790
one that we're going to be using is a method called didAdd.
43

44
00:03:08,820 --> 00:03:16,500
So if you just start typing didAdd with the ARSCNViewDelegate adopted, then you will see this delegate
44

45
00:03:16,500 --> 00:03:22,430
method show up. And as it says, it tells the delegate, which is this current view controller, that a SceneKit
45

46
00:03:22,440 --> 00:03:27,200
node corresponding to a new AR anchor has been added to the scene.
46

47
00:03:27,210 --> 00:03:27,440
Okay.
47

48
00:03:27,480 --> 00:03:28,660
So a lot of new words here.
48

49
00:03:28,680 --> 00:03:34,830
But it basically means that it's detected a horizontal surface and it's given that detected surface
49

50
00:03:34,920 --> 00:03:37,530
a width and a height which is an AR anchor
50

51
00:03:37,650 --> 00:03:41,750
so that we'll be able to use it to place things or to visualize it.
51

52
00:03:41,910 --> 00:03:49,320
And I'm going to hit enter to insert that piece of code. And inside this didAdd method is where we're going
52

53
00:03:49,320 --> 00:03:51,090
to setup our horizontal plane.
53

54
00:03:51,150 --> 00:03:57,330
So the most important thing that we get from this method is that when it detects a new horizontal plane,
54

55
00:03:57,390 --> 00:04:02,460
it's going to, firstly, call this method and trigger the code that's inside.
55

56
00:04:02,460 --> 00:04:09,900
But also we're going to receive this object called anchor and the anchor is of data type ARAnchor,
56

57
00:04:09,900 --> 00:04:15,680
so it's a real world position orientation that can be used for placing objects in an ARScene.
57

58
00:04:15,690 --> 00:04:22,710
So, this, you can think of as almost like a tile on the floor or on your horizontal plane, and you can
58

59
00:04:22,710 --> 00:04:29,730
use that tile to place objects. It has a dimension. It has a position. It has a rotation. It has a whole bunch
59

60
00:04:29,730 --> 00:04:36,420
of coordinates. And we're going to use those real world coordinates of that plane in order to place our
60

61
00:04:36,450 --> 00:04:37,980
object onto it.
61

62
00:04:37,980 --> 00:04:40,260
So this anchor is where we're going to be working with.
62

63
00:04:40,440 --> 00:04:45,520
Now, ARAnchor is a broad category of anchors.
63

64
00:04:45,600 --> 00:04:53,790
We want to specifically check to see if the anchor was of the type ARPlaneAnchor, i.e., that that anchor
64

65
00:04:53,790 --> 00:04:58,500
corresponds to a plane, rather than some other sort of 3D object in the scene.
65

66
00:04:58,800 --> 00:05:08,580
So to do that, we can say if anchor is ARPlaneAnchor, and this line of code basically checks to see
66

67
00:05:08,700 --> 00:05:16,780
if this new anchor that was added is of the type ARPlaneAnchor, i.e., it is from our plane detection.
67

68
00:05:16,770 --> 00:05:28,200
Now, if that was true, then we're going to print "plane detected," and if it's false or else, we're simply
68

69
00:05:28,200 --> 00:05:33,300
going to return which means that we're just going to exit from this method call.
69

70
00:05:33,390 --> 00:05:36,870
So let's give it a go and see what happens when we try to detect some planes.
70

71
00:05:39,750 --> 00:05:41,270
Now, with ARKit,
71

72
00:05:41,280 --> 00:05:46,230
you notice that there's usually a bit of delay before it detects a plane.
72

73
00:05:46,230 --> 00:05:52,710
So we can enable one of the debug options which will show us as it's trying to look for points in the
73

74
00:05:52,710 --> 00:05:55,230
view that are lining up to form a plane.
74

75
00:05:55,230 --> 00:06:01,470
So to do that, we're going to go right to the top of our viewDidLoad and we're gonna write
75

76
00:06:01,640 --> 00:06:09,180
self.sceneView.debugOptions, and this, again, is an array, but we're only gonna have one item in it ,and
76

77
00:06:09,180 --> 00:06:16,740
we're gonna use ARSCNDebugOptions.showFeaturedPoints.
77

78
00:06:16,740 --> 00:06:28,300
All right. So let's run the app again and you can see the difference.
78

79
00:06:28,410 --> 00:06:28,650
All right.
79

80
00:06:28,680 --> 00:06:33,600
So that made it a little bit easier to see what was actually happening behind the scenes in order to
80

81
00:06:33,600 --> 00:06:34,920
enable plane detection.
81

82
00:06:34,920 --> 00:06:37,160
So it was looking for all of these feature points.
82

83
00:06:37,560 --> 00:06:43,710
And you'll notice that feature points don't get generated very easily when you have a shiny metallic
83

84
00:06:43,710 --> 00:06:44,220
object.
84

85
00:06:44,220 --> 00:06:49,530
So if you go to a mirror and you try to use the app to detect feature points, you won't actually get
85

86
00:06:49,650 --> 00:06:50,730
all that far.
86

87
00:06:50,880 --> 00:06:57,060
But if you have a soft fabric that has maybe some creases in it that casts a little bit of shadow and
87

88
00:06:57,060 --> 00:07:02,760
gives a little bit texture to that plane, then your plane detection will work so much quicker and so
88

89
00:07:02,760 --> 00:07:03,570
much easier.
89

90
00:07:03,600 --> 00:07:08,490
But before the feature points actually appear on screen, your plane detection isn't taking place.
90

91
00:07:08,580 --> 00:07:12,630
So it's just a good way of being able to debug what's going on.
91

92
00:07:12,630 --> 00:07:12,930
All right.
92

93
00:07:12,960 --> 00:07:19,950
So, now instead of printing that the plane is detected, I'm going to downcast our anchor into the type
93

94
00:07:20,040 --> 00:07:21,180
ARPlaneAnchor.
94

95
00:07:21,210 --> 00:07:31,520
So I will say let planeAnchor = anchor as ARPlaneAnchor.
95

96
00:07:31,710 --> 00:07:40,530
So our code basically checks to see if this anchor that was added or detected is of type ARPlaneAnchor.
96

97
00:07:40,740 --> 00:07:47,430
And if it is, then we are going to change its type from ARAnchor to ARPlaneAnchor.
97

98
00:07:47,430 --> 00:07:54,000
Now, you'll notice that this is pretty wordy and there's certainly ways of making this more succinct
98

99
00:07:54,420 --> 00:07:57,200
and much, much shorter on the number of lines of code.
99

100
00:07:57,270 --> 00:08:01,950
But because there's going to be quite a lot of things going on in this block of code, some of which is
100

101
00:08:01,950 --> 00:08:05,080
quite abstract and quite difficult to understand,
101

102
00:08:05,190 --> 00:08:11,550
I'm going to try and keep my code as expressive as possible for now, and then later on, we can refactor
102

103
00:08:11,550 --> 00:08:16,510
it and make it shorter and simpler.
103

104
00:08:16,790 --> 00:08:23,360
So, now we have this constant called planeAnchor that is equal to the anchor that we got back from the
104

105
00:08:23,360 --> 00:08:28,990
delegate method which detected a new area that corresponds to a horizontal plane,
105

106
00:08:29,090 --> 00:08:33,190
and we have changed its data type to ARPlaneAnchor.
106

107
00:08:33,370 --> 00:08:33,710
All right.
107

108
00:08:33,710 --> 00:08:39,020
So the next thing that we need to do, we're going to convert the dimensions of our anchor into what's
108

109
00:08:39,020 --> 00:08:40,550
called a ScenePlane.
109

110
00:08:40,580 --> 00:08:46,910
So similar to how we created spheres and boxes using SceneBox or SceneSphere,
110

111
00:08:46,910 --> 00:08:52,190
there's something called SCNPlane that allows us to create a plane in SceneKit. And we're going to
111

112
00:08:52,190 --> 00:09:00,440
call ours plane and it's going to be created using ScenePlane and SCNPlane only requires two things:
112

113
00:09:00,620 --> 00:09:01,890
a width and height.
113

114
00:09:01,910 --> 00:09:09,620
So luckily, the anchor that we've got, which is of type ARPlaneAnchor has a property called extent. And
114

115
00:09:09,620 --> 00:09:15,860
the extent comprises of a width and a height. So you can think of an anchor as simply just like a tile
115

116
00:09:15,860 --> 00:09:20,090
on the ground, and it has a width, it has a height,
116

117
00:09:20,090 --> 00:09:25,130
and we're going to use that tile that's been detected and convert it into something that we can use with
117

118
00:09:25,130 --> 00:09:29,150
SceneKit and that we can render onto our sceneView.
118

119
00:09:29,150 --> 00:09:38,750
So the width is going to be planeAnchor.extent. x. And the height is going to be 
119

120
00:09:39,770 --> 00:09:42,590
planeAnchor.extent.z.
120

121
00:09:42,620 --> 00:09:48,980
So, a really important thing is that this is x and this is z, a really common error is that people will
121

122
00:09:48,980 --> 00:09:50,190
write y here.
122

123
00:09:50,210 --> 00:09:55,850
But if you actually read the documentation by option clicking on extent, you can see that this is the
123

124
00:09:55,910 --> 00:09:58,700
estimated width and length of the detected plane.
124

125
00:09:58,850 --> 00:10:05,060
And most importantly, although, what type of this property is vector_float3, i.e., a three-dimensional
125

126
00:10:05,060 --> 00:10:13,940
object, a planeAnchor is always two-dimensional, and is always position size in only the x and z directions.
126

127
00:10:14,000 --> 00:10:19,040
So make sure that you don't put y in here because, otherwise, your code is not going to work
127

128
00:10:19,040 --> 00:10:25,050
going forth. Now, the next thing you might notice is that Xcode is giving us this error here, and it's
128

129
00:10:25,070 --> 00:10:32,120
telling us that it can't convert the type 'Float,' which this extent.x is, into the expected type which
129

130
00:10:32,120 --> 00:10:36,080
is 'CGFloat' which this function requires of its parameters.
130

131
00:10:36,080 --> 00:10:42,860
So we can go ahead and cast it into a CGFloat by just clicking Fix. And we fixed the x,
131

132
00:10:42,860 --> 00:10:45,800
so now we need to fix the extent.z.
132

133
00:10:45,860 --> 00:10:52,580
So the next step is to create a planeNode. So you remember when we created spheres? We first created
133

134
00:10:52,580 --> 00:11:02,450
the geometry using SCNSphere and then we created a node to attach that geometry, too. let planeNode equals
134

135
00:11:02,540 --> 00:11:04,110
SCNNode.
135

136
00:11:05,240 --> 00:11:14,270
And then I want to set the planeNodes position property, planeNode.position =
136

137
00:11:14,270 --> 00:11:14,720
SCNVector3.
137

138
00:11:17,940 --> 00:11:24,490
So the x is going to be planeAnchor.center.x
138

139
00:11:24,510 --> 00:11:27,750
and y is going to be zero.
139

140
00:11:28,960 --> 00:11:35,150
The y position is going to be zero because, remember, it's a flat horizontal plane,
140

141
00:11:35,410 --> 00:11:37,230
so there is no y element.
141

142
00:11:37,240 --> 00:11:44,750
We don't want it to be above the horizontal plane that was detected. And z is going to be 
142

143
00:11:45,840 --> 00:11:48,860
planeAnchor.center.z.
143

144
00:11:49,260 --> 00:11:49,510
All right.
144

145
00:11:49,530 --> 00:11:55,180
So up till now, this is all pretty similar to what we did previously with the boxes and the spheres.
145

146
00:11:55,200 --> 00:11:59,580
Now, the next thing that we have to understand is a bit of a tricky concept.
146

147
00:11:59,910 --> 00:12:06,510
Now, the idea here is that we're creating this thing called a SCNPlane which is a rectangular one-sided
147

148
00:12:06,510 --> 00:12:10,970
plain geometry of a specified width and height.
148

149
00:12:11,070 --> 00:12:18,110
So we specified its width and height using the anchor that was detected.
149

150
00:12:18,450 --> 00:12:24,930
And then we might think that we're done but, actually, there's a slight problem because SCNPlanes are
150

151
00:12:24,930 --> 00:12:28,370
created as a vertical plane.
151

152
00:12:28,410 --> 00:12:37,260
So if you have a look at this graphic here and the 3D key, the red is the x, the green is the y, and the
152

153
00:12:37,260 --> 00:12:41,700
z component is non-existent because this is a 2D object.
153

154
00:12:41,700 --> 00:12:49,500
Now, as you might remember, in ARKit on our sceneView, everything is 3D, and also our plane is horizontal.
154

155
00:12:49,500 --> 00:12:51,890
So we need to convert this vertical plane
155

156
00:12:51,930 --> 00:12:59,760
that's got a component in the x and y into a flat plane that's got a component in the x and the
156

157
00:12:59,760 --> 00:13:00,780
z.
157

158
00:13:00,780 --> 00:13:07,410
So to do that, we're going to have to transform our planeNode and we're going to have to rotate it by
158

159
00:13:07,410 --> 00:13:14,840
90 degrees in order to make it horizontal. So planeNode.transform.
159

160
00:13:15,150 --> 00:13:22,110
And we're going to specify the amount that we want to transform it using something called a 
160

161
00:13:22,200 --> 00:13:25,510
SCNMatrix4MakeRotation.
161

162
00:13:25,740 --> 00:13:28,020
And this takes four parameters,
162

163
00:13:28,020 --> 00:13:34,050
first of which, is the angle that you want to rotate it by, and then the next three parameters, you specify
163

164
00:13:34,170 --> 00:13:37,410
along which axis you want to rotate your object.
164

165
00:13:37,410 --> 00:13:41,250
So the angle that we want to rotate our plane by is 90 degrees.
165

166
00:13:41,280 --> 00:13:44,060
We want to change it from vertical to horizontal.
166

167
00:13:44,070 --> 00:13:47,190
Now, this is in radians.
167

168
00:13:47,190 --> 00:13:50,320
So just a bit of recap on high school math,
168

169
00:13:50,370 --> 00:13:54,420
1 pi radian is equivalent to 180 degrees.
169

170
00:13:54,480 --> 00:13:59,400
So in order to rotate this by 90 degrees, we need half pi radians.
170

171
00:13:59,430 --> 00:14:08,880
So I'm going to grab pi by writing float.pi, and this is something that's available in UIKit because
171

172
00:14:08,970 --> 00:14:14,730
if you're working with spheres, if you're working with angles, and you tend to need to use pi, divide it by
172

173
00:14:14,730 --> 00:14:18,480
2, and that's specifying that we're rotating by 90 degrees.
173

174
00:14:18,480 --> 00:14:25,290
However, if you have a look at this function, you can see that the angle is the amount of rotation, in
174

175
00:14:25,290 --> 00:14:25,950
radians.
175

176
00:14:25,950 --> 00:14:30,680
We got all of that right. Measured counterclockwise around the rotation axis.
176

177
00:14:31,170 --> 00:14:38,730
So we actually want it to be rotated clockwise. So because when it's positive, it's counterclockwise.
177

178
00:14:38,790 --> 00:14:42,810
So to cancel that, we just need to add a minus in front of it.
178

179
00:14:42,840 --> 00:14:51,030
So two minuses cancel out and this is now rotating our vertical plane 90 degrees clockwise, and then
179

180
00:14:51,090 --> 00:14:58,320
we are going to specify the axis that we want to rotate around which is, of course, the x axis. Snd for
180

181
00:14:58,320 --> 00:15:00,300
the other two, we're going to leave it alone.
181

182
00:15:00,600 --> 00:15:04,690
So our rotation only has an x component.
182

183
00:15:04,710 --> 00:15:05,010
All right.
183

184
00:15:05,030 --> 00:15:09,880
So that is our plane node created and transformed.
184

185
00:15:09,900 --> 00:15:15,840
Now, for our human eyes to be able to see it and evaluate whether if our plane was created in the correct
185

186
00:15:15,840 --> 00:15:18,610
place, we're going to need to give it a material.
186

187
00:15:18,630 --> 00:15:26,400
So in this lesson, there is a download for a grid.png file. And this is the asset that Apple provided
187

188
00:15:26,460 --> 00:15:30,750
in their example project for us to be able to visualize these planes.
188

189
00:15:30,750 --> 00:15:32,780
So we're going to use the same image.
189

190
00:15:32,970 --> 00:15:39,990
And the important thing about this image that you might notice is that it's a .png file type.
190

191
00:15:40,560 --> 00:15:44,390
And you may or may not know, but PNG  file types have transparencies,
191

192
00:15:44,430 --> 00:15:48,850
so we'll be able to see through our grid and see the material underneath.
192

193
00:15:48,900 --> 00:15:51,450
And you're going to see really soon what that looks like.
193

194
00:15:51,810 --> 00:15:52,050
All right.
194

195
00:15:52,080 --> 00:15:58,560
So I'm going to drag my grid and, once again, into the art.scnassets folder, and then I'm going
195

196
00:15:58,560 --> 00:16:01,250
to create something called gridMaterial.
196

197
00:16:01,560 --> 00:16:05,880
And, again, it's going to be created using SCNMaterial.
197

198
00:16:05,940 --> 00:16:14,940
So we've done this before and we're going to set the gridMaterial's diffuse.contents to a UIImage
198

199
00:16:14,970 --> 00:16:17,720
which is the one that we dragged in just now.
199

200
00:16:18,720 --> 00:16:20,460
So we're going to have to specify the folder.
200

201
00:16:20,550 --> 00:16:25,770
So art.scnassets/grid.png.
201

202
00:16:26,100 --> 00:16:26,930
Okay. Cool.
202

203
00:16:26,940 --> 00:16:32,640
And we're going to assign this material to our planeGeometry that we created earlier on.
203

204
00:16:32,730 --> 00:16:38,430
So plane. materials = gridMaterial.
204

205
00:16:38,610 --> 00:16:45,500
And finally, we're going to set the geometry of our planeNode to our plane.
205

206
00:16:45,880 --> 00:16:46,360
All right.
206

207
00:16:46,390 --> 00:16:52,870
So the only last thing we need to do as we did before with our spheres and our boxes is to add our
207

208
00:16:52,870 --> 00:16:59,070
childNode into the rootNode. So you can see that as a part of this delegate method. We also have this parameter
208

209
00:16:59,080 --> 00:17:02,480
called node which gives us a blank node to work with.
209

210
00:17:02,530 --> 00:17:07,660
So you can either, as we did before, tap into the sceneView.SCNRootNode and childNode,
210

211
00:17:08,260 --> 00:17:14,560
or we can just simply use this node that got created when the didAdd delegate method got called.
211

212
00:17:14,740 --> 00:17:20,860
node.addChildNode planeNode onto the scene.
212

213
00:17:20,860 --> 00:17:32,880
So let's give it a run and see what it looks like.
213

214
00:17:33,070 --> 00:17:33,350
All right.
214

215
00:17:33,380 --> 00:17:34,360
So there you have it.
215

216
00:17:34,380 --> 00:17:39,660
Just with a few lines of code and the assistance of ARKit, we've been able to implement an immensely
216

217
00:17:39,660 --> 00:17:45,930
complicated computational task which is detecting a horizontal plane in the real world based on the
217

218
00:17:45,930 --> 00:17:51,120
image that's coming in through the camera. So you can pat yourself on the back and we can thank ARKit
218

219
00:17:51,120 --> 00:17:53,070
for making our lives easier.
219

220
00:17:53,100 --> 00:17:58,770
So in the next lesson, we're going to be looking at how we can detect touch in the scene because, ultimately,
220

221
00:17:58,770 --> 00:18:05,220
we want to be able to touch a plane that we've detected, say, a table or the floor and be able to place
221

222
00:18:05,310 --> 00:18:10,230
our dice at the position that we touched in the 3D world and scaled,
222

223
00:18:10,230 --> 00:18:15,570
so it looks like it's a realistic dice that's been placed in the 3D world.
223

224
00:18:15,570 --> 00:18:17,820
So we're going to do that in the next lesson.
224

225
00:18:17,850 --> 00:18:18,690
So I'll see you there.