neural-rendering
Papers with tag neural-rendering
2022
- ARAH: Animatable Volume Rendering of Articulated Human SDFsShaofei Wang, Katja Schwarz, Andreas Geiger, and Siyu TangIn 2022
Combining human body models with differentiable rendering has recentlyenabled animatable avatars of clothed humans from sparse sets of multi-view RGBvideos. While state-of-the-art approaches achieve realistic appearance withneural radiance fields (NeRF), the inferred geometry often lacks detail due tomissing geometric constraints. Further, animating avatars inout-of-distribution poses is not yet possible because the mapping fromobservation space to canonical space does not generalize faithfully to unseenposes. In this work, we address these shortcomings and propose a model tocreate animatable clothed human avatars with detailed geometry that generalizewell to out-of-distribution poses. To achieve detailed geometry, we combine anarticulated implicit surface representation with volume rendering. Forgeneralization, we propose a novel joint root-finding algorithm forsimultaneous ray-surface intersection search and correspondence search. Ouralgorithm enables efficient point sampling and accurate point canonicalizationwhile generalizing well to unseen poses. We demonstrate that our proposedpipeline can generate clothed avatars with high-quality pose-dependent geometryand appearance from a sparse set of multi-view RGB videos. Our method achievesstate-of-the-art performance on geometry and appearance reconstruction whilecreating animatable avatars that generalize well to out-of-distribution posesbeyond the small number of training poses.
clothed implicit neural-rendering@inproceedings{ARAH, title = {ARAH: Animatable Volume Rendering of Articulated Human SDFs}, author = {Wang, Shaofei and Schwarz, Katja and Geiger, Andreas and Tang, Siyu}, year = {2022}, tags = {clothed, implicit, neural-rendering}, } - Self-Supervised 3D Human Pose Estimation in Static Video Via Neural RenderingLuca Schmidtke, Benjamin Hou, Athanasios Vlontzos, and Bernhard KainzIn 2022
Inferring 3D human pose from 2D images is a challenging and long-standingproblem in the field of computer vision with many applications including motioncapture, virtual reality, surveillance or gait analysis for sports andmedicine. We present preliminary results for a method to estimate 3D pose from2D video containing a single person and a static background without the needfor any manual landmark annotations. We achieve this by formulating a simpleyet effective self-supervision task: our model is required to reconstruct arandom frame of a video given a frame from another timepoint and a renderedimage of a transformed human shape template. Crucially for optimisation, ourray casting based rendering pipeline is fully differentiable, enabling end toend training solely based on the reconstruction task.
pose-estimation neural-rendering self-supervised@inproceedings{2210.04514, title = {Self-Supervised 3D Human Pose Estimation in Static Video Via Neural Rendering}, author = {Schmidtke, Luca and Hou, Benjamin and Vlontzos, Athanasios and Kainz, Bernhard}, year = {2022}, tags = {pose-estimation, neural-rendering, self-supervised}, } - SelfNeRF: Fast Training NeRF for Human from Monocular Self-rotating VideoBo Peng, Jun Hu, Jingtao Zhou, and Juyong ZhangIn 2022
In this paper, we propose SelfNeRF, an efficient neural radiance field basednovel view synthesis method for human performance. Given monocularself-rotating videos of human performers, SelfNeRF can train from scratch andachieve high-fidelity results in about twenty minutes. Some recent works haveutilized the neural radiance field for dynamic human reconstruction. However,most of these methods need multi-view inputs and require hours of training,making it still difficult for practical use. To address this challengingproblem, we introduce a surface-relative representation based onmulti-resolution hash encoding that can greatly improve the training speed andaggregate inter-frame information. Extensive experimental results on severaldifferent datasets demonstrate the effectiveness and efficiency of SelfNeRF tochallenging monocular videos.
monocular neural-rendering@inproceedings{selfnerf, title = {SelfNeRF: Fast Training NeRF for Human from Monocular Self-rotating Video}, author = {Peng, Bo and Hu, Jun and Zhou, Jingtao and Zhang, Juyong}, year = {2022}, tags = {monocular, neural-rendering}, } - MonoNHR: Monocular Neural Human RendererHongsuk Choi, Gyeongsik Moon, Matthieu Armando, Vincent Leroy, Kyoung Mu Lee, and Gregory RogezIn 2022
Existing neural human rendering methods struggle with a single image inputdue to the lack of information in invisible areas and the depth ambiguity ofpixels in visible areas. In this regard, we propose Monocular Neural HumanRenderer (MonoNHR), a novel approach that renders robust free-viewpoint imagesof an arbitrary human given only a single image. MonoNHR is the first methodthat (i) renders human subjects never seen during training in a monocularsetup, and (ii) is trained in a weakly-supervised manner without geometrysupervision. First, we propose to disentangle 3D geometry and texture featuresand to condition the texture inference on the 3D geometry features. Second, weintroduce a Mesh Inpainter module that inpaints the occluded parts exploitinghuman structural priors such as symmetry. Experiments on ZJU-MoCap, AIST, andHUMBI datasets show that our approach significantly outperforms the recentmethods adapted to the monocular case.
monocular neural-rendering@inproceedings{MonoNHR, title = {MonoNHR: Monocular Neural Human Renderer}, author = {Choi, Hongsuk and Moon, Gyeongsik and Armando, Matthieu and Leroy, Vincent and Lee, Kyoung Mu and Rogez, Gregory}, year = {2022}, tags = {monocular, neural-rendering}, }