The goal of this assignment is to coarsely align the 3D head model with a 2D image. To do this you will click on points in the image and solve for a transformation that aligns 3D landmark points on the model with the 2D points in the image. The first part is to come up with a coarse initialization and then refine this in the second part.
Everything you need is in /course/cs296-4/Heads/
You can use the Matlab function defined in the course directory:
[X, ind]=getImageLmarks(im)This will let you click on the relevant points. Refer to figure Landmarks.jpg in that directory for the definitions of the landmark locations.
Loads the indices that say which vertex corresponds to which landmark.
load heads-male20.mat
fid=fopen('theadlndindex.txt','r');
lmarkVerts=fscanf(fid,'%e',[1 inf]);
fclose(fid);The order of the landmarks in this data structure is given in the definition of lmarkNames in getImageLmarks.m.
If you choose an image where the eyes and mouth are visible, then the following indices can be used
leyelind=21;
leyerind=22;
reyelind=23;
reyerind=24;
mouthcentind=10;I'd average the vertex values for the left/right sides of each eye to get the model coordinates of the eye center.
The calibration data is in /course/cs296-4/Calib711/.
This directory includes the a subdirectory code with the calibration information (camera.mat) and the routines for converting back and forth from Rodrigues vectors to rotation matrices; look at code/readme for more information. There is also a directory "undistorted" with the 711 frames that you should use.
I suggest using frame 45 (frontal face) and 103 (profile) as test cases.
Initialization. The goal is to recover an initial guess of the rotation matrix R and translation vector t that align the mean 3D head model with the image. You can use any method you like but three suggestions are
1. Manual positioning using some sort of GUI which allows you to rotate the model and move it in 3D.
2. Click three points on the image for which you have correspondence landmarks on the model. The distances between these points on the model are known. Solve for the 3D location of the model points in the image using one of the methods described in the paper by Haralick, Lee, Ottenberg and Nolle, IJCV 13(3), pp 331-356. This is in the course directory /course/cs296-4.08/Papers/haralick.pdf. This will give 4 solutions of which you can select one. From the matched points you can now easily compute the desired rotation and translation as described in Appendix I.
3. Use the HeightGui shown in class as a starting point. You could essentially put the mean head on the end of the height pole and move this around in the image (maybe too slow). You could slide this along to the right floor position and up to the right height. Then you could just rotate the head from there. This will work well in the more upright poses and maybe less well in others. The gui is in /course/cs296-4/results/711/HeightGui2.m.
You are free to do something completely different as well. Come to class on Feb 13 prepared to do a "show and tell". You don't have to have the world's best solution but I want us to discuss the solutions people found. The "best" solution can then be used by everyonen for Task 3.
Display your head model in the image in the initialized pose. Display only the visible verticies. Try making the surface partially transparent (you can play with the properties 'SpecularStrength', 'DiffuseStrength', and 'AmbientStrength' in trisurf.
Optimization. Click several points u_i in the image for which you know corresponding landmarks on the 3D head model (the more the better). Starting from your initialization from Task 1, perform a non-linear optimization in Matlab (eg lsqnonlin or something else). The thing we want to minimize is
sum_{i=1}^3(P(R*x_i +t)- u_i)^2
Where P(.) is the perspective projection operation x_i are the model points and u_i are the corresponding image points. You want R to be a rotation matrix (st RR'=I). One way to enforce this is to use Matt's code for converting back and forth from Rodrigues' vectors to rotation matrices. You do the optimization above using the Rodrigues representation. These functions are in /course/cs296-4/Calib711/code/rotation_mat2rod.m and rotation_rod2mat.m
Given your estimated pose of the shape, find where in the image each visible vertex lands. Use bi-linear interpolation to get the color of the vertex from the four nearest image pixels. Texture the head model with these image values and display the head in Matlab at several orientations.