Evaluation Script

[Davidyao99]

Great work! I was wondering if you will release evaluation scripts used for human and camera poses in your paper.

[hongsukchoi]

The evaluation code and dataloader code are quite messy right now. I think it's gonna take some more time before I release the evaluation code for reproduction. If you are just interested in evaluation metric code, I can clean some bit I provie it you in a shorter time.

[Davidyao99]

Yup, I am mainly just interested in the evaluation metric code, wondering which library / algorithms you used to compute those metrics! Thanks so much for the prompt response!

[hongsukchoi]

Hi! You can refer to this.
https://drive.google.com/drive/folders/1_DVYPIRPbG-Iu00RyzTyu4ElzM6Hyak7?usp=sharing

Disclaimer
This is not the official metric code and might be deprecated. It will take some more time to clean up the code, so I'm providing you with the code that might be useful.

[orkrukx]

Hello @hongsukchoi, I'm currently trying to reproduce the papers' results on EgoHumans. As I understand that you can't share the evaluation code right now, would it be possible to know the overall methodology when running the pipeline on EgoHumans?

When running the evaluation on videos/image sequences, do you run the full pipeline on each frame individually (Grounded-SAM2 + Dust3r + HMR2.0 + ViTPose + Optimization), or do you run Dust3r and HMR2.0 on the first frame, then optimize the following frames starting from the previous one to cut down processing time?

In particular, HSfM takes roughly 10s on a A100 GPU for optimizing a single frame using 4 views (using the code from align_world_env_and_smpl_hsfm_optim.py). Do you have some information about how long it took to run HSfM on the full EgoHumans dataset?

Thank you

[hongsukchoi]

When running the evaluation on videos/image sequences, do you run the full pipeline on each frame individually (Grounded-SAM2 + Dust3r + HMR2.0 + ViTPose + Optimization), or do you run Dust3r and HMR2.0 on the first frame, then optimize the following frames starting from the previous one to cut down processing time?

I do for all frames.

In particular, HSfM takes roughly 10s on a A100 GPU for optimizing a single frame using 4 views (using the code from align_world_env_and_smpl_hsfm_optim.py). Do you have some information about how long it took to run HSfM on the full EgoHumans dataset?

I don't recall excatly... I think I used slurm (2080tis) for SAM2,HMR2.0, and ViTPose which take less than a day. Overall, including dust3r and hsfm, I think it was less than a day.

[moonsliu]

Great work! I would like to know whether you evaluated every frame of egohuman or only specific frames. If only a subset of frames has been evaluated, could you provide the directory of the specific frames?