3d Graphics On Iphone

3D Graphics on iPhone Douglass Turner [email protected] @dugla #iphonemeetup3dtalkacton

A talk in three acts ...

• • •

Intro to Computer Graphics Computer Graphics in OpenGL OpenGL and CocoaTouch Playing Nice Together

Intro to Computer Graphics (a trip down the graphics pipeline)

Build and Tesselate Model Copyright Steve Jubinville

Pose Model Copyright Steve Jubinville

Aim Camera

Design Appearance. Light Scene. Copyright © 2006 Holger Schömann

Design Appearance. Light Scene. Copyright © 2006 Florian Wild

Design Appearance. Light Scene. Copyright © 2006 Jack Qiao

Render Scene

Cull & clip model triangles to viewing frustrum.

Rasterize From models to pixels. Scanline interpolation.

• Depth Comparison • Apply Appearance - Vertex level. Pixel level. • Composite Elements

Present Image

Update Model State

Rinse, repeat.

Computer Graphics in OpenGL

OpenGL

Cocoa

OpenGL is old school

You are flipping levers on a state machine.

float foo[]

glPushMatrix() and glPopMatrix()

Data Structures are for wimps.

OpenGL is a dragster. Not a Prius. Just buckle up ...

... and enjoy the ride, baby!

iPhone 3GS by the Numbers (via AnandTech http://bit.ly/1pQaJO)



~7M triangles / second





~230,000 realtime triangles (30fps)





iPhone display: 320 * 480 = 153,600 pixels





230,000 / 153,600 = ~1.5 realtime-triangles / pixel

;-)

OpenGL websites are a bit different than Cocoa websites ...

Isn’t that sweet ...

Um... WTF?

Can’t you just feel the love?

Dude. I think my eyes are bleeding.

No worries. safari.oreilly.com is your friend

Hierarchical Modeling. Transformation Stacks. Coordinate Frames.

Pose Model

Pose Model

You are already familiar with hierarchical modeling and coordinate frames.

[self.parentView addSubview:childView];

or ...

for (UIView *v in self.containerView.subviews) { CGRect bounds = [v convertRect:v.bounds toView:scrollView]; if (CGRectIntersectsRect(bounds, scrollView.bounds)) { NSLog(@"View %d is visible", v.tag); } else { NSLog(@"View %d is hidden", v.tag); } } // for (self.containerView.subviews)

You are already familiar with hierarchical modeling and coordinate frames.

- (void)setTransform:(CGAffineTransform)newValue { // Scale along x-axis only CGAffineTransform constrainedTransform = CGAffineTransformScale(CGAffineTransformIdentity, newValue.a, 1.0); }

newValue.a = constrainedTransform.a; newValue.b = constrainedTransform.b; newValue.c = constrainedTransform.c; newValue.d = constrainedTransform.d; [super setTransform:newValue];

You are already already familiar with transformations.

HelloTeapot Demo

Source: http://github.com/turner/HelloTeapot

HelloTeapot Code Walkthrough

http://github.com/turner/HelloTeapot/blob/master/Classes/GLViewController.[hm]





// Position Camera glMatrixMode(GL_MODELVIEW); glLoadIdentity(); glLoadMatrixf(_openGLCameraInverseTransform);





// Position Light glPushMatrix(); glMultMatrixf(_cameraTransform);

glPopMatrix();



// Position Model glPushMatrix();



JLMMatrix3DSetZRotationUsingDegrees(rotation, -45.0f); JLMMatrix3DSetScaling(scale, sx, sy, sz); JLMMatrix3DMultiply(rotation, scale, concatenation); glMultMatrixf(concatenation);



// Render for(int i = 0; i < num_teapot_indices; i += new_teapot_indicies[i] + 1) {





} // for (num_teapot_indices)

glPopMatrix();

glEnable(GL_LIGHT3); glLightfv(GL_LIGHT3, GL_DIFFUSE, spotLight);

glDrawElements(GL_TRIANGLE_STRIP, indices[i], GL_UNSIGNED_SHORT, &indices[i+1]);

http://github.com/turner/HelloTeapot/blob/master/Classes/GLViewController.[hm]





// Position Camera glMatrixMode(GL_MODELVIEW); glLoadIdentity(); glLoadMatrixf(_openGLCameraInverseTransform);





// Position Light glPushMatrix(); glMultMatrixf(_cameraTransform);

glPopMatrix();



// Position Model glPushMatrix();



JLMMatrix3DSetZRotationUsingDegrees(rotation, -45.0f); JLMMatrix3DSetScaling(scale, sx, sy, sz); JLMMatrix3DMultiply(rotation, scale, concatenation); glMultMatrixf(concatenation);



// Render for(int i = 0; i < num_teapot_indices; i += new_teapot_indicies[i] + 1) {





} // for (num_teapot_indices)

glPopMatrix();

Pose Model

glEnable(GL_LIGHT3); glLightfv(GL_LIGHT3, GL_DIFFUSE, spotLight);

glDrawElements(GL_TRIANGLE_STRIP, indices[i], GL_UNSIGNED_SHORT, &indices[i+1]);

http://github.com/turner/HelloTeapot/blob/master/Classes/GLViewController.[hm]





// Position Camera glMatrixMode(GL_MODELVIEW); glLoadIdentity(); glLoadMatrixf(_openGLCameraInverseTransform);





// Position Light glPushMatrix(); glMultMatrixf(_cameraTransform);

glPopMatrix();



// Position Model glPushMatrix();



JLMMatrix3DSetZRotationUsingDegrees(rotation, -45.0f); JLMMatrix3DSetScaling(scale, sx, sy, sz); JLMMatrix3DMultiply(rotation, scale, concatenation); glMultMatrixf(concatenation);



// Render for(int i = 0; i < num_teapot_indices; i += new_teapot_indicies[i] + 1) {





} // for (num_teapot_indices)

glPopMatrix();

Light Scene

glEnable(GL_LIGHT3); glLightfv(GL_LIGHT3, GL_DIFFUSE, spotLight);

glDrawElements(GL_TRIANGLE_STRIP, indices[i], GL_UNSIGNED_SHORT, &indices[i+1]);

http://github.com/turner/HelloTeapot/blob/master/Classes/GLViewController.[hm]





// Position Camera glMatrixMode(GL_MODELVIEW); glLoadIdentity(); glLoadMatrixf(_openGLCameraInverseTransform);





// Position Light glPushMatrix(); glMultMatrixf(_cameraTransform);

glPopMatrix();



// Position Model glPushMatrix();



JLMMatrix3DSetZRotationUsingDegrees(rotation, -45.0f); JLMMatrix3DSetScaling(scale, sx, sy, sz); JLMMatrix3DMultiply(rotation, scale, concatenation); glMultMatrixf(concatenation);



// Render for(int i = 0; i < num_teapot_indices; i += new_teapot_indicies[i] + 1) {





} // for (num_teapot_indices)

glPopMatrix();

glEnable(GL_LIGHT3); glLightfv(GL_LIGHT3, GL_DIFFUSE, spotLight);

Aim Camera ... Dude, where is the camera ...?

glDrawElements(GL_TRIANGLE_STRIP, indices[i], GL_UNSIGNED_SHORT, &indices[i+1]);

http://github.com/turner/HelloTeapot/blob/master/Classes/GLViewController.[hm]

camera

teapot

wTc

wTt

world

cT t

camera

teapot

?

glMatrixMode(GL_MODELVIEW); glLoadMatrixf(_openGLCameraInverseTransform);

cT t

camera

(wTc)

teapot

-1

wTt

world

cT t

=

cT w

* wTt

cT t

=

cT w

* wTt





// Position Camera glMatrixMode(GL_MODELVIEW); glLoadIdentity(); glLoadMatrixf(_openGLCameraInverseTransform);





// Position Light glPushMatrix(); glMultMatrixf(_cameraTransform);

glPopMatrix();



// Position Model glPushMatrix();



JLMMatrix3DSetZRotationUsingDegrees(rotation, -45.0f); JLMMatrix3DSetScaling(scale, sx, sy, sz); JLMMatrix3DMultiply(rotation, scale, concatenation); glMultMatrixf(concatenation);



// Render for(int i = 0; i < num_teapot_indices; i += new_teapot_indicies[i] + 1) {





} // for (num_teapot_indices)

glPopMatrix();

cT t

=

cT w

* wTt

glEnable(GL_LIGHT3); glLightfv(GL_LIGHT3, GL_DIFFUSE, spotLight);

glDrawElements(GL_TRIANGLE_STRIP, indices[i], GL_UNSIGNED_SHORT, &indices[i+1]);

http://github.com/turner/HelloTeapot/blob/master/Classes/GLViewController.[hm]

- (void)placeCameraAtLocation:(M3DVector3f)location target:(M3DVector3f)target up:(M3DVector3f)up;

http://github.com/turner/HelloTeapot/blob/master/Classes/GLViewController.[hm]

Render float teapot_vertices [] = { 0.0663, 0.1178, 0.0, 0.0672, 0.1152, 0.0, 0.0639, 0.1178, 0.0178043, ... }; float teapot_normals[] = {















};

-0.987635, -0.156768, 0, -0.902861, -0.429933, 0, -0.953562, -0.156989, -0.257047, ...

short indicies[] = { // howmany vertices in vertex strip 26, // vertex strip indices 1122, 1243, 1272, 1242, ... ,1283, 1199, ... };

http://github.com/turner/HelloTeapot/blob/master/teapot.h

Render

glEnableClientState(GL_VERTEX_ARRAY); glVertexPointer(3 ,GL_FLOAT, 0, teapot_vertices); glEnableClientState(GL_NORMAL_ARRAY); glNormalPointer(GL_FLOAT, 0, teapot_normals); glEnable(GL_NORMALIZE);

http://github.com/turner/HelloTeapot/blob/master/Classes/GLViewController.[hm]





// Position Camera glMatrixMode(GL_MODELVIEW); glLoadIdentity(); glLoadMatrixf(_openGLCameraInverseTransform);





// Position Light glPushMatrix(); glMultMatrixf(_cameraTransform);

glPopMatrix();



// Position Model glPushMatrix();



JLMMatrix3DSetZRotationUsingDegrees(rotation, -45.0f); JLMMatrix3DSetScaling(scale, sx, sy, sz); JLMMatrix3DMultiply(rotation, scale, concatenation); glMultMatrixf(concatenation);



// Render for(int i = 0; i < num_teapot_indices; i += indicies[i] + 1) {





} // for (num_teapot_indices)

glPopMatrix();

Render

glEnable(GL_LIGHT3); glLightfv(GL_LIGHT3, GL_DIFFUSE, spotLight);

glDrawElements(GL_TRIANGLE_STRIP, indices[i], GL_UNSIGNED_SHORT, &indices[i+1]);

http://github.com/turner/HelloTeapot/blob/master/Classes/GLViewController.[hm]

http://github.com/turner/HelloTeapot/blob/master/Classes/GLViewController.[hm]

invert( camera_transform ) red_light green_light blue_light push() camera_transform white_spotlight pop() push() teapot_transform draw() pop()

You’ll want to understand this stuff if you intend to do AR* apps like all the cool kids.

*Augmented

Reality - See http://layar.com

Design Appearance

Mipmapped textures are your friend. They will become your workhorse for visual complexity.

Texture Everything Copyright © Douglass Turner







glGenTextures(1, &_name); glBindTexture(GL_TEXTURE_2D, _name); glTexEnvf(GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_REPLACE); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_REPEAT); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_REPEAT); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR); glTexImage2D(GL_TEXTURE_2D, 0, glColor, _w, _h, 0, glColor, glFormat, data); glGenerateMipmapOES(GL_TEXTURE_2D);

Method initWithTextureFile:mipmap:

Class github.com/turner/HelloTexture/blob/master/Classes/TEITexture.[h,m]

HelloTexture Demo

Source: http://github.com/turner/HelloTexture

Two Words Texture. Atlas.

t

s

Texture Space

t

s

Texture Space

HelloParticleSystem Demo texture atlas hacks

OpenGL ES 2.0 == GLSL

Shaders Take it to Another Level

Copyright © Douglass Turner

Shaders Take it to Another Level

Copyright © Douglass Turner

Shaders Take it to Another Level

ZBrush Modeling Daniel Lieske

Shaders Take it to Another Level

ZBrush Modeling Daniel Lieske Material Design Ralf Stumpf

Shaders Take it to Another Level

ZBrush Modeling Daniel Lieske Material Design Ralf Stumpf

Shaders Take it to Another Level

Copyright: Stephen Molyneaux

GLSL is cool. Sadly, OpenGL ES 2.0 (iPhone 3Gs) does not support hardware anti-aliasing. Stick with mipmapped textures until you really know what you are doing. Or don’t care about aliasing artifacts ...

1 sample per pixel

4 jittered/filtered samples per pixel

Remember the iPhone Imperitive Your app will be beautiful.

OpenGL and Cocoa Touch Playing Nice Together

MVC OpenGL is M. Keep V dumb. Lots of chatter between M and V. • GLGravity | http://bit.ly/2dS8YI • Jeff LaMarche OpenGL ES Template | http://bit.ly/2IvZtV

Touch Sequence & Accelerometer

HelloParticleSystem Demo Particle system. Touch. Acceleration

Touch Sequence

Initialize model state touchesBegan:withEvent:

Evolve model state touchesMoved:withEvent:

Clean up model state touchesEnded:withEvent: touchesCancelled:withEvent:

Accelerometer

- (void)accelerometer:(UIAccelerometer*)accelerometer didAccelerate:(UIAcceleration*)acceleration { // Compute "G". Use low-pass filter to attenuate instantaneous acceleration. self.accelerationValueX = acceleration.x * kFilteringFactor + self.accelerationValueX * (1.0 - kFilteringFactor); self.accelerationValueY = acceleration.y * kFilteringFactor + self.accelerationValueY * (1.0 - kFilteringFactor); self.accelerationValueZ = acceleration.z * kFilteringFactor + self.accelerationValueZ * (1.0 - kFilteringFactor); // ParticleSystem particles live in flatland. Use x and y compoments of "G". [ParticleSystem setGravity:CGPointMake(self.accelerationValueX, self.accelerationValueY)]; }

UIAccelerometerDelegate

+ (void)setGravity:(CGPoint)gravityVector { // Normalize 2D “G” vector. Only care about direction. float length = sqrtf(gravityVector.x * gravityVector.x + gravityVector.y * gravityVector.y); ParticleSystemGravity.x = gravityVector.x / length; ParticleSystemGravity.y = gravityVector.y / length; // Flip y-component of gravity vector to be consistent with screen space coordinate // system used in ParticleSystem. ParticleSystemGravity.y = -(ParticleSystemGravity.y); }

ParticleSystem - Class State





// dv = dv/dt * dt float dv_x = (ParticleSystemGravity.x * gravityScaleFactor * dt); float dv_y = (ParticleSystemGravity.y * gravityScaleFactor * dt); particle.velocity = CGPointMake(particle.velocity.x + dv_x, particle.velocity.y + dv_y); // dx = dx/dt * dt float dx = particle.velocity.x * dt; float dy = particle.velocity.y * dt; // x = x_old * dx particle.location = CGPointMake(particle.location.x + dx, particle.location.y + dy);

ParticleSystem - Instance State



CoreAnimation: [myView setNeedsDisplay] & [myView drawRect]

OpenGL: - (void)startAnimation { animationTimer = [NSTimer scheduledTimerWithTimeInterval:animationInterval target:self selector:@selector(drawView) userInfo:nil repeats:YES]; }

iPhone is an Expression Platform You Must See: Stanford cs 193p Lecture 17 Video | On iTunes: http://bit.ly/PUrGj Creating New Expressive Social Mediums on the iPhone Ge Wang | CCO & CTO | Smule

iPhone is a general purpose sensing platform • Touch • Sound • Proximity • Cloud • Camera • INU (almost) • Geography

iPhone encourages plausible, hyper-real, extensions of human experience. Be interesting. Be brilliant.

Resources • @dugla - search on #iphonemeetup3dtalkacton • http://theelasticimage.posterous.com • This presentation on SlideShare: http://bit.ly/4wC4Mu

Douglass Turner @dugla #iphonemeetup3dtalkacton [email protected] Copyright © Douglass Turner