results

This report describes the interim progress on the studies within the scope of 3D User Interfaces and Direct Manipulation Solutions. The report is organized as separate chapters, each of which includes the related state of the art, the project, and the progress made so far.

The report starts with the direct manipulation techniques. In sections 1 and 2 several camera-based and sensor-based direct manipulation solutions are described. Section 3 describes how to define gestures using the input devices and the meanings of these gestures. In sections 4 and 5, 3D UI and rendering related solutions are presented.

Download document

Present document has been prepared in fulfillment of Deliverable 3.1.1, required as an intermediate result of Work Package 3 of the EU Seventh Framework Program ALL 3D IMAGING PHONE Project. Specification of 3D display HW module means the description of the 3D display related hardware, low level software, mechanical and optical components which will be developed and integrated in the project.

Download Document

Capin, T.; Pulli, K.; Akenine-Moller,” The State of the Art in Mobile Graphics Research“, T. Computer Graphics and Applications, IEEE Volume 28, Issue 4, July-Aug. 2008 Page(s):74 - 84 Digital Object Identifier 10.1109/MCG.2008.83

Abstract: High-quality computer graphics let mobile-device users access more compelling content. Still, the devices’ limitations and requirements differ substantially from those of a PC. This survey of mobile graphics research describes current solutions in terms of specialized hardware (including 3D displays), rendering and transmission, visualization, and user interfaces.

T. Balogh, P. T. Kovács, Z. Megyesi, A. Barsi. “HoloVizio - True 3D Display System“. 2008 NEM Summit, Saint Malo, France, October 2008.

Abstract: This paper presents HoloVizio true 3D display technology, and its real life implementations, the HoloVizio display models from Holografika. Our patented technology uses a specially arranged array of optical modules and a holographic screen. Each point of the holographic screen emits light beams of different color and intensity to various directions. The light beams generated in the optical modules hit the screen points in various angles and the holographic screen makes the necessary optical transformation to compose these beams into a perfectly continuous 3D view. With proper software control, light beams leaving the pixels propagate in multiple directions, as if they were emitted from the points of 3D objects at fixed spatial locations. We describe the 10 Mpixel desktop display and the 50Mpixel large-scale system. We cover the real-time control issues at high pixel-count systems with the HoloVizio software environment, give solutions for displaying various kinds of content, including 3D scenes from existing computer applications, 3D CAD models and natural content. We also describe concrete 3D applications developed in the frame of European projects.

K. Müller, A. Smolic, K. Dix, P. Merkle, P. Kauff, and T. Wiegand, “View Synthesis for Advanced 3D Video Systems”, EURASIP Journal on Image and Video Processing, 2009.

Abstract: The paper is about 3D video applications and systems based on multiview autostereoscopic displays. These are expected to play an important role in home user environments, since they support multi-user 3D sensation and motion parallax impression. The tremendous data rate cannot be handled efficiently by representation and coding formats such as MVC or MPEG-C Part 3. Multiview video plus depth is a new format that efficiently supports such advanced 3DV systems, but this requires high quality intermediate view synthesis. For this, a new approach is presented that separates unreliable image regions along depth discontinuities from reliable image regions, which are treated separately and fused to the final interpolated view. In contrast to previous layered approaches, our algorithm uses two boundary layers and one reliable layer, performs image-based 3D warping only and was generically implemented, i.e. doesn’t necessarily rely on 3D graphics support. Furthermore, different hole-filling and filtering methods are added to provide high-quality intermediate views. As a result, high quality intermediate views for an existing 9-view auto-stereoscopic display as well as other stereo- and multiscopic displays are presented, which prove the suitability of our approach for advanced 3DV systems.

P. Merkle, Y. Morvan, A. Smolic, D. Farin, K. Mueller, P.H.N. de With, and T. Wiegand, “The Effects of Multiview Depth Video Compression on Multiview Rendering”, Signal Processing: Image Communication, 2009.

Abstract: This research investigates the interaction between different techniques for depth compression and view synthesis rendering with multiview video plus scene depth data. Two different approaches for depth coding are compared, namely H.264/MVC, using temporal and inter-view reference images for efficient prediction, and the novel platelet-based coding algorithm, characterized by being adapted to the special characteristics of depth-images. Since depth-images are a 2D representation of the 3D scene geometry, depth-image errors lead to geometry distortions. Therefore, the influence of geometry distortions resulting from coding artifacts is evaluated for both coding approaches in two different ways. First, the variation of 3D surface meshes is analyzed using the Hausdorff distance and second, the distortion is evaluated for 2D view synthesis rendering, where color and depth information are used together to render virtual intermediate camera views of the scene. The results show that – although its R-D performance is worse – platelet-based depth coding outperforms H.264, due to improved sharp edge preservation. Therefore depth coding needs to be evaluated with respect to geometry distortions.

K. Müller, A. Smolic, K. Dix, P. Merkle, P. Kauff, and T. Wiegand,”Reliability-based Generation and View Synthesis in Layered Depth Video“, Proc. MMSP 2008, IEEE International Workshop on Multimedia Signal Processing, Cairns, Australia, October 2008.

Abstract: In this paper, a system for video rendering on multiscopic 3D displays is considered where the data is represented as layered depth video (LDV). This representation consists of one full or central video with associated per-pixel depth and additional residual layers. Thus, only one full view with additional residual data needs to be transmitted. The LDV data is used at the receiver to generate all intermediate views for the display. The paper presents the LDV layer extraction as well as the view synthesis, using a scene reliability-driven approach. Here, unreliable image regions are detected and in contrast to previous approaches the residual data is enlarged to reduce artifacts in unreliable areas during rendering. To provide maximum data coverage, the residual data remains at its original positions and will not be projected towards the central view. The view synthesis process also uses this reliability analysis to provide higher quality intermediate views than previous approaches. As a final result, high quality intermediate views for an existing 9-view auto-stereoscopic display are presented, which prove the suitability of the LDV approach for advanced 3D video (3DV) systems.

A. Smolic, K. Müller, K. Dix, P. Merkle, P. Kauff, and T. Wiegand, ” Intermediate View Interpolation Based on Multiview Video Plus Depth for Advanced 3D Video Systems “, Proc. ICIP 2008, IEEE International Conference on Image Processing, San Diego, CA, USA, October 2008.

Abstract: A system for video on multiscopic 3D displays is considered where the data representation consists of multiview video plus scene depth. At most, 3 multiview video signals are being transmitted and used together with the depth data to generate intermediate views at the receiver. The paper presents an approach to such an intermediate view interpolation that separates unreliable image regions along depth discontinuities from reliable image regions. These image regions are processed with different algorithms and previous layered approaches, two boundary layers and one reliable layer is used. Moreover, the presented technique does not rely on 3D graphics support but uses image-based 3D warping instead. For enhanced quality intermediate view generation, hole-filling and filtering methods are described. As a result, high quality intermediate views for an existing 9-view auto-stereoscopic display are presented, which prove the suitability of the approach for advanced 3D video (3DV) systems.

This deliverable makes a user analysis including different user classes, user profiles and use case scenarios after presenting some important design principles of user interface and user interaction. Most important scenarios are chosen as the use case scenarios in this part. Lastly, the requirements for mobile users are categorized such as software, hardware, UI, performance, etc. and labeled as ‘must’, ‘should’ and ‘may’ according to their level of importance.

Download Document

Specification of 3D camera HW module means the description of the 3D camera related hardware, low level software, mechanical and optical components which will be developed and integrated in the project.

Download Document