Hello, Ask a question again. There is no mosh(SMPL theta and beta) in STB、RHD、FreiHand dataset etc. How to translate 3D keypoints to mesh(SMPL theta beta)? Hope your reply, thanks.

In config.py, I set OFFICIAL_MANO_PATH to the right hand model and Run python prepare_mano.py.Then I can get the converted MANO model about right hand. But when I use the converted MANO model about right hand, the result is so bad. Where are the wrongs? Thus, I want to known how to use the MANO model about right hand OR how to convert the MANO model about right hand? looking forward to your reply! Thanks a lot.

Thank you for great project，I have a few questions about training IKNet

1. When changing the original 16 rotations of MANO into 21 rotations, do W, T0, I0, M0, R0, and L0 share the rotation of W in the original MANO?
2. I found the joints_xyz calculated by using MANO ref_pose and the transformed 21 rotation parameters using the method in hand_mesh.py is not equal to the 'J_transformed' saved in the MANO pkl file , the order of joints has been adjusted according to kinematics.py. When using the MANO dataset to train IKNet, how did you get the ground truth 3D joint annotation in Lxyz? Is the calculation method of FK (Q) the same as the calculation method of joint_xyz in hand_mesh.py

Paper say that: DetNet is trained on 3 datasets: theCMU Panoptic Dataset (CMU) , the Rendered Hand-pose Dataset (RHD) and the GANerated Hands Dataset(GAN).

Since the images of three datasets are different from each other, can u please tell me how to preprocess the image?

You have done really a great work!

When I read your paper about, I am a little confused about how to find the best beta in IKNet by minimizing E(beta). Is beta directly got by solving the function? Or using ML methods like Newton down-hill method?

Thank you Best wishes

你好，在wrappers.py文件166行，使用delta = delta * length 作为ikmodel的输入之一 方便问一下这么做的原因吗？ delta是关节归一化向量，乘以关节长度是为什么呢？

我理解输入到后面ik模型的参数 需要有手部mesh模板参数，手部姿态参数sita（delta），skinning weights，手部坐标参数（xyz）

所以就比较困惑只输入delta就可以了，为什么要相乘一下length。。。。

Hi,I got confused about another problem.

In your paper ,u said "As previous work, we perform a global alignment to better measure the local hand pose. " How do u implement the "global alignment"? Is it just to transalate the root joint to same location of label (Is the lable here is also root-relative and normalized using reference bone?) I got AUC of 0.1 only using DetNet retrained in RHD.

Could u provide the "prevous work" that do a global alignment like u? And it would bebetter if their code has been public available. Thanks!

Thank you for your great work! I'm trying to use the model output to animate a virtual hand in unity, I tried to set the quaternion into unity's localrotation but it did not work. Could you share some insight about how I can achieve that?

Hello, First, I would like to congrats you on the amazing paper. Moreover, I have a question regarding IK architecture. I would like to know if there is any comparison between the IK architecture that you proposed here and the other algorithm that you previously proposed based on Levenberg-Marquadr on Mano's hand. Additionally, could you guide me on applying the IK architecture without running the entire code as I have some ground truth 3d coordinates, and I want to obtain the IK parameters? Thanks a lot.

Greetings and many thanks for the great work.

I wanted to utilize your code to extract MoCAP data given a first person RGB video dataset that has a clear view of both hands during a task. Given that your model is restricted to predicting from a single hand I wonder whether it will consistently show preference for the left hand if presented with videos that display both? If that's the case I suppose I could parse the dataset twice, flipping it the second time to obtain both hands' coordinates, right?

I'm curious as it seems to be one of the better datasets publicly available, not only does it include really accurate 3D poses, but they are all on real images include challenging poses and object interactions. Along with all of this, it includes MANO hand shape ground truths. I would love to see how this model performs.

It also allows for seeing how this performs without needing alignment since both camera intrinsics and scale are included for each image

I'm also curious if this would be a good alternative for training IKNet instead of the MoCap data since it includes the hand shape ground truths. I'm not sure if I should open a separate issue for that to make it easier for others to find

I am trying to use IKNet separately, starting from hand keypoints that have been extracted with MediaPipe. In order for this to work, I need to make sure that the Mediapipe hand coordinates are preprocessed in order to match the expected input format of IKNet (origin, scale, possibly rotation as well??).

I ran into two questions here:

1. I can see from your code that the keypoints have te be shifted to make 'M1' the origin. Bust what is the assumed scale? In the code you use IK_UNIT_LENGTH when rescaling from Mano reference keypoints, but it is not clear what this relates to or where it comes from. Also, is there an assumption on rotation of the hand (e.g. palm orientation)?

2. I was assuming that the 'mpii_ref' keypoint set you pass as input to the IKNet would be some kind of "relaxed" reference hand (this is converted from the mano code base). When I plot it, however, the projection onto the xz plane matches this assumption, but the y coordinates look very strange, so I am assuming I am doing something wrong in interpreting this. Or maybe this incorporates some assumptions about the IKNet model input that I need to convert also to xyz keypoints input - since this seems to be passed as a reference hand? Could you clarify?

Examples: (1) mpii_ref hand in front view (looking fine) (2) mpii_ref hand in rotated xyz view, showing unnaturally curved fingers and very long wrist-to-thumb connection (3) For comparison: mediapipe hand in front view (4) For comparison: mediapipe hand in same xyz view as above

Hi, In minimal-hand, inappropriate dependency versioning constraints can cause risks.

Below are the dependencies and version constraints that the project is using

pygame==1.9.4
open3d==0.9
tensorflow_gpu==1.14.0
transforms3d==0.3.1
keyboard==0.13.4
opencv_python==3.4.3.18
numpy==1.18.1

The version constraint == will introduce the risk of dependency conflicts because the scope of dependencies is too strict. The version constraint No Upper Bound and * will introduce the risk of the missing API Error because the latest version of the dependencies may remove some APIs.

After further analysis, in this project, The version constraint of dependency keyboard can be changed to >=0.9.3,<=0.13.5. The version constraint of dependency numpy can be changed to >=1.8.0,<=1.23.0rc3.

The above modification suggestions can reduce the dependency conflicts as much as possible, and introduce the latest version as much as possible without calling Error in the projects.

The invocation of the current project includes all the following methods.

keyboard.is_pressed

numpy.linalg.norm

pygame.init
open3d.visualization.Visualizer.update_renderer
tensorflow.pad
pickle.load
open3d.visualization.Visualizer.update_geometry
tensorflow.layers.dense
zero_padding
detnet
open3d.geometry.TriangleMesh
wrappers.ModelPipeline
self.ik_model.process
tf_hmap_to_uv
pygame.display.set_mode.blit
open3d.visualization.Visualizer.create_window
tensorflow.nn.relu
cv2.VideoCapture
pickle.load.toarray
load_pkl
lmaps.append
numpy.maximum
tensorflow.ConfigProto
dmaps.append
tensorflow.norm
tensorflow.reshape
tensorflow.contrib.layers.xavier_initializer
self.cap.read
matplotlib.pyplot.show
tensorflow.cast
numpy.matmul
dense
open3d.geometry.TriangleMesh.compute_triangle_normals
tensorflow.layers.batch_normalization
numpy.sum
viewer.get_view_control.set_constant_z_far
capture.read
transforms3d.quaternions.quat2mat
pygame.time.Clock
viewer.get_view_control.convert_to_pinhole_camera_parameters
numpy.expand_dims
tensorflow.contrib.layers.l2_regularizer
tensorflow.concat.get_shape
str
utils.OneEuroFilter
xyz_to_delta
self.compute_alpha
self.dx_filter.process
open3d.geometry.TriangleMesh.compute_vertex_normals
matplotlib.pyplot.plot
pickle.dump
conv_bn
pygame.time.Clock.tick
tensorflow.expand_dims
features.get_shape.as_list
tensorflow.nn.max_pool2d
keyboard.is_pressed
tensorflow.name_scope
frame_large.np.flip.copy
data.items
numpy.abs
net_2d
open3d.visualization.Visualizer
len
utils.OneEuroFilter.process
dense_bn
matplotlib.pyplot.legend
tensorflow.gather_nd
tensorflow.argmax
LowPassFilter
tensorflow.train.Saver
pygame.surfarray.make_surface
tensorflow.stack
numpy.linalg.norm
MANOHandJoints.labels.index
viewer.get_view_control.convert_from_pinhole_camera_parameters
tensorflow.nn.sigmoid
matplotlib.pyplot.xlabel
plot_pck
inputs.get_shape
calculate_auc
numpy.linspace.reshape
pygame.display.update
tensorflow.train.Saver.restore
tensorflow.concat
open3d.utility.Vector3dVector
bottleneck
open3d.visualization.Visualizer.poll_events
self.det_model.process
tensorflow.initializers.truncated_normal
open3d.visualization.Visualizer.get_view_control
numpy.transpose
int
xyz.get_shape.as_list
hand_mesh.HandMesh.set_abs_quat
numpy.tile
cam_params.intrinsic.set_intrinsics
self.ref_T.append
open3d.utility.Vector3iVector
numpy.array
viewer.get_render_option.load_from_json
self.graph.as_default
open3d.visualization.Visualizer.get_render_option
tensorflow.shape
get_pose_tile
mano_to_mpii
xyz.get_shape
tensorflow.tile
ModelIK
tensorflow.layers.conv2d
numpy.stack
tensorflow.transpose
tensorflow.Session
frame.np.flip.copy
pygame.display.set_mode
MANOHandJoints.mesh_mapping.items
cv2.resize
open3d.geometry.TriangleMesh.paint_uniform_color
transforms3d.axangles.axangle2mat
hmaps.append
net_3d
range
pygame.display.set_caption
hand_mesh.HandMesh
MPIIHandJoints.labels.index
self.verts.copy
self.x_filter.process
tensorflow.Graph
ModelDet
numpy.linspace
wrappers.ModelPipeline.process
matplotlib.pyplot.grid
tensorflow.variable_scope
numpy.concatenate
tensorflow.constant
tensorflow.maximum
self.ref_pose.append
conv_bn_relu
capture.OpenCVCapture
live_application
matplotlib.pyplot.ylabel
matplotlib.pyplot.tight_layout
open3d.visualization.Visualizer.add_geometry
kinematics.mpii_to_mano
utils.imresize
tensorflow.placeholder
cam_params.extrinsic.copy
numpy.stack.append
self.sess.run
resnet50
open
numpy.flip
tensorflow.where
prepare_mano
numpy.finfo
network_fn
numpy.zeros
inputs.get_shape.as_list

@developer Could please help me check this issue? May I pull a request to fix it? Thank you very much.

您好，非常感谢您能分享您的成果。 我在阅读您的代码和尝试的时候有几个问题想请教下 1.在代码里没有评估beta参数，所以生成的手部并不能保持原来手的形状对吗，比如手指长短，比例，厚度 2.生成的手部模型的大小是固定的，不会因为图片中手的大小而变化，是这样吗 3.代码的输入是视频如果只输入一张图片是否会让结果变差呢