written in front
- In order to ensure that the entire sample project is more intuitive and easy to understand, the numpy version will be used to display the source code of some functions, but the numpy version of the library is not used in the sample program. Before the original code of the Cython version and the numpy version, there are differences There will be labels, I hope readers pay attention.
- The jupyter version of the pipeline example program is here https://download.csdn.net/download/qq_45912037/85031147
completely free, welcome to download
text
Without further ado, let's start directly
0. Quote the referenced library
import os, sys
import numpy as np
import scipy.io as sio
from skimage import io
from time import time
import matplotlib.pyplot as plt
sys.path.append('..')
import face3d
from face3d import mesh
1. Load grid data
Mesh data includes: vertices, triangular mesh data, color (optional), texture (optional)
color is used here to represent facial texture
C = sio.loadmat('Data/example1.mat')
vertices = C['vertices']; colors = C['colors']; triangles = C['triangles']
colors = colors/np.max(colors)
The grid data here is a .mat file, and the vertices, colors, and triangles data in it are respectively obtained,
and the colors data are normalized.
The data specifications are as follows:
2. Change the vertex position
Change the position of the mesh object in world coordinates
s = 180/(np.max(vertices[:,1]) - np.min(vertices[:,1]))
R = mesh.transform.angle2matrix([0, 30, 0])
t = [0, 0, 0]
transformed_vertices = mesh.transform.similarity_transform(vertices, s, R, t)
The source code of mesh.transform.angle2matrix is as follows
def angle2matrix(angles):
''' get rotation matrix from three rotation angles(degree). right-handed.
Args:
angles: [3,]. x, y, z angles
x: pitch. positive for looking down.
y: yaw. positive for looking left.
z: roll. positive for tilting head right.
Returns:
R: [3, 3]. rotation matrix.
'''
x, y, z = np.deg2rad(angles[0]), np.deg2rad(angles[1]), np.deg2rad(angles[2])
# x
Rx=np.array([[1, 0, 0],
[0, cos(x), -sin(x)],
[0, sin(x), cos(x)]])
# y
Ry=np.array([[ cos(y), 0, sin(y)],
[ 0, 1, 0],
[-sin(y), 0, cos(y)]])
# z
Rz=np.array([[cos(z), -sin(z), 0],
[sin(z), cos(z), 0],
[ 0, 0, 1]])
R=Rz.dot(Ry.dot(Rx))
return R.astype(np.float32)
It's easy to understand, the corresponding rotation matrix is generated according to the input angle value
Another mesh.transform.similarity_transform source code is as follows
def similarity_transform(vertices, s, R, t3d):
''' similarity transform. dof = 7.
3D: s*R.dot(X) + t
Homo: M = [[sR, t],[0^T, 1]]. M.dot(X)
Args:(float32)
vertices: [nver, 3].
s: [1,]. scale factor.
R: [3,3]. rotation matrix.
t3d: [3,]. 3d translation vector.
Returns:
transformed vertices: [nver, 3]
'''
t3d = np.squeeze(np.array(t3d, dtype = np.float32))
transformed_vertices = s * vertices.dot(R.T) + t3d[np.newaxis, :]
return transformed_vertices
The input is the grid vertices vertices, scaling s, rotation matrix R and translation vector t3d
where s = 180/(np.max(vertices[:,1]) - np.min(vertices[:,1]))
is to Scale to a vertical distance of 180
and perform the spatial coordinate transformation s*R.dot(X) + t to output the transformed vertex position
3. Modify colors/textures (add light)
Add a point light source, and the light position is defined in the world coordinate system.
light_positions = np.array([[-128, -128, 300]])
light_intensities = np.array([[1, 1, 1]])
lit_colors = mesh.light.add_light(transformed_vertices, triangles, colors, light_positions, light_intensities)
The source code of mesh.light.add_light is as follows
def add_light(vertices, triangles, colors, light_positions = 0, light_intensities = 0):
''' Gouraud shading. add point lights.
In 3d face, usually assume:
1. The surface of face is Lambertian(reflect only the low frequencies of lighting)
2. Lighting can be an arbitrary combination of point sources
3. No specular (unless skin is oil, 23333)
Ref: https://cs184.eecs.berkeley.edu/lecture/pipeline
Args:
vertices: [nver, 3]
triangles: [ntri, 3]
light_positions: [nlight, 3]
light_intensities: [nlight, 3]
Returns:
lit_colors: [nver, 3]
'''
nver = vertices.shape[0]
normals = get_normal(vertices, triangles) # [nver, 3]
# ambient
# La = ka*Ia
# diffuse
# Ld = kd*(I/r^2)max(0, nxl)
direction_to_lights = vertices[np.newaxis, :, :] - light_positions[:, np.newaxis, :] # [nlight, nver, 3]
direction_to_lights_n = np.sqrt(np.sum(direction_to_lights**2, axis = 2)) # [nlight, nver]
direction_to_lights = direction_to_lights/direction_to_lights_n[:, :, np.newaxis]
normals_dot_lights = normals[np.newaxis, :, :]*direction_to_lights # [nlight, nver, 3]
normals_dot_lights = np.sum(normals_dot_lights, axis = 2) # [nlight, nver]
diffuse_output = colors[np.newaxis, :, :]*normals_dot_lights[:, :, np.newaxis]*light_intensities[:, np.newaxis, :]
diffuse_output = np.sum(diffuse_output, axis = 0) # [nver, 3]
# specular
# h = (v + l)/(|v + l|) bisector
# Ls = ks*(I/r^2)max(0, nxh)^p
# increasing p narrows the reflectionlob
lit_colors = diffuse_output # only diffuse part here.
lit_colors = np.minimum(np.maximum(lit_colors, 0), 1)
return lit_colors
The get_normal function is otherwise defined in the source code, so I won't repeat it here.
In 3d face, it is usually assumed that:
1. The surface of the human face is a Lambertian (Lambertian surface), which only reflects low-frequency light
2. Lighting can be any combination of point light sources
3. No specular reflection
These refer to https:// cs184.eecs.berkeley.edu/lecture/pipeline
but this site seems to be down
The input parameters include vertex coordinates, triangular mesh data, light source position, and light intensity.
After calculation, output the color data added to the light source.
(I don't know much about this part of the calculation, and I may specifically explain the light in the future)
4. Modify the vertex position
Transforms an object from world space coordinates to camera space, which is the viewer's perspective.
(ignore this step if using a standard camera)
camera_vertices = mesh.transform.lookat_camera(transformed_vertices, eye = [0, 0, 200], at = np.array([0, 0, 0]), up = None)
# -- project object from 3d world space into 2d image plane. orthographic or perspective projection
projected_vertices = mesh.transform.orthographic_project(camera_vertices)
The source code of mesh.transform.lookat_camera is as follows:
def normalize(x):
epsilon = 1e-12
norm = np.sqrt(np.sum(x**2, axis = 0))
norm = np.maximum(norm, epsilon)
return x/norm
def lookat_camera(vertices, eye, at = None, up = None):
""" 'look at' transformation: from world space to camera space
standard camera space:
camera located at the origin.
looking down negative z-axis.
vertical vector is y-axis.
Xcam = R(X - C)
Homo: [[R, -RC], [0, 1]]
Args:
vertices: [nver, 3]
eye: [3,] the XYZ world space position of the camera.5
at: [3,] a position along the center of the camera's gaze.
up: [3,] up direction
Returns:
transformed_vertices: [nver, 3]
"""
if at is None:
at = np.array([0, 0, 0], np.float32)
if up is None:
up = np.array([0, 1, 0], np.float32)
eye = np.array(eye).astype(np.float32)
at = np.array(at).astype(np.float32)
z_aixs = -normalize(at - eye) # look forward
x_aixs = normalize(np.cross(up, z_aixs)) # look right
y_axis = np.cross(z_aixs, x_aixs) # look up
R = np.stack((x_aixs, y_axis, z_aixs))#, axis = 0) # 3 x 3
transformed_vertices = vertices - eye # translation
transformed_vertices = transformed_vertices.dot(R.T) # rotation
return transformed_vertices
Standard camera space is:
the camera is at the origin; looking down is the negative z-axis; the vertical vector is the y-axis.
The input is the vertex coordinates, the XYZ world space position of the camera, the position along the center of the camera's line of sight [the default is (0, 0, 0)], and the upward direction [the default is (0, 1, 0)] to calculate the rotation according to the
input Matrix R and calculate the new vertex position by Xcam=R(XC).
5. Convert to 2D image
Set the length and width of the image to 256
h = w = 256
# change to image coords for rendering
image_vertices = mesh.transform.to_image(projected_vertices, h, w)
# render
rendering = mesh.render.render_colors(image_vertices, triangles, lit_colors, h, w)
The source code of the mesh.transform.to_image part is as follows:
def to_image(vertices, h, w, is_perspective = False):
''' change vertices to image coord system
3d system: XYZ, center(0, 0, 0)
2d image: x(u), y(v). center(w/2, h/2), flip y-axis.
Args:
vertices: [nver, 3]
h: height of the rendering
w : width of the rendering
Returns:
projected_vertices: [nver, 3]
'''
image_vertices = vertices.copy()
if is_perspective:
# if perspective, the projected vertices are normalized to [-1, 1]. so change it to image size first.
image_vertices[:,0] = image_vertices[:,0]*w/2
image_vertices[:,1] = image_vertices[:,1]*h/2
# move to center of image
image_vertices[:,0] = image_vertices[:,0] + w/2
image_vertices[:,1] = image_vertices[:,1] + h/2
# flip vertices along y-axis.
image_vertices[:,1] = h - image_vertices[:,1] - 1
return image_vertices
The input is the vertex coordinates, the length and width of the picture, and the perspective option is_perspective (the default is FALSE)
to obtain the two-dimensional vertex coordinates through calculation.
The source code of mesh.render.render_colors is as follows:
Note, this is the numpy version
def render_colors(vertices, triangles, colors, h, w, c = 3):
''' render mesh with colors
Args:
vertices: [nver, 3]
triangles: [ntri, 3]
colors: [nver, 3]
h: height
w: width
Returns:
image: [h, w, c].
'''
assert vertices.shape[0] == colors.shape[0]
# initial
image = np.zeros((h, w, c))
depth_buffer = np.zeros([h, w]) - 999999.
for i in range(triangles.shape[0]):
tri = triangles[i, :] # 3 vertex indices
# the inner bounding box
umin = max(int(np.ceil(np.min(vertices[tri, 0]))), 0)
umax = min(int(np.floor(np.max(vertices[tri, 0]))), w-1)
vmin = max(int(np.ceil(np.min(vertices[tri, 1]))), 0)
vmax = min(int(np.floor(np.max(vertices[tri, 1]))), h-1)
if umax<umin or vmax<vmin:
continue
for u in range(umin, umax+1):
for v in range(vmin, vmax+1):
if not isPointInTri([u,v], vertices[tri, :2]):
continue
w0, w1, w2 = get_point_weight([u, v], vertices[tri, :2])
point_depth = w0*vertices[tri[0], 2] + w1*vertices[tri[1], 2] + w2*vertices[tri[2], 2]
if point_depth > depth_buffer[v, u]:
depth_buffer[v, u] = point_depth
image[v, u, :] = w0*colors[tri[0], :] + w1*colors[tri[1], :] + w2*colors[tri[2], :]
return image
The input is vertex coordinates, triangle mesh data, mesh color data, and the length and width of the target image.
The output is the textured 2D image data of the target.
6. Save pictures
save_folder = 'results/pipeline'
if not os.path.exists(save_folder):
os.mkdir(save_folder)
io.imsave('{}/rendering.jpg'.format(save_folder), rendering)
Lossy conversion from float32 to uint8. Range [0, 1]. Convert image to uint8 prior to saving to suppress this warning.
7. Results display
Generated 2D image display
plt.imshow(rendering)
plt.show()
The model of the camera space can also be displayed with the library function inside
# ---- show mesh
mesh.vis.plot_mesh(camera_vertices, triangles)
plt.show()
