Samsung Science & Technology Center

Computer vision algorithms powered by deep neural networks recently exhibit impressive performance in a lot of tasks. Inspired by this trend, practitioners now expect to deploy state-of-the-art algorithms in practical systems. However, there are accuracy-sensitive and/or cost-sensitive computer vision problems in the real world, which include autonomous driving, medical diagnosis, machine inspection, embedded systems, and so on, and these applications often suffer from unexpectedly tough challenges. Therefore, computer vision algorithms should be able to handle the challenges effectively to be deployed in the wild condition by improving their robustness and reliability.

This talk initiates the discussion about visual understanding problems in more challenging scenarios. I first introduce several examples of computer vision problems that should operate in extremely wild environments, and discuss how computer vision algorithms can overcome existing limitations. Furthermore, I propose a proper research direction for computer vision and machine learning to make them more valuable to our society in technical aspects.

Robust visual recognition under extreme conditions such as adverse weather and low-light conditions is of great importance in real-world applications. In this context, our project aims to develop recognition models robust against adverse conditions in extreme environments. The first half of this talk will briefly review our efforts to learn robust recognition models such as recognition-aware image enhancement and learning features invariant to adverse conditions, and then discuss their limitations and ways of improving them with emphasis on how to represent the effect of adverse conditions on neural features and how to utilize the representations for learning robust models.

Another goal of our project is the development of datasets for learning robust recognition models. Most existing datasets for visual recognition do not take account of input degradations that can be caused in extreme environments, which leads to severe performance degradation of existing models in extreme conditions. In the second half of this talk, we will introduce our attempts to resolve this problem including the acquisition of real-world extreme condition datasets and the synthetic generation of realistic extreme condition datasets. We will also discuss about remaining challenges and share our ideas on possible future directions.

Robust visual recognition under extreme conditions such as adverse weather and low-light conditions is of great importance in real-world applications. In this context, our project aims to develop recognition models robust against adverse conditions in extreme environments. The first half of this talk will briefly review our efforts to learn robust recognition models such as recognition-aware image enhancement and learning features invariant to adverse conditions, and then discuss their limitations and ways of improving them with emphasis on how to represent the effect of adverse conditions on neural features and how to utilize the representations for learning robust models.

Another goal of our project is the development of datasets for learning robust recognition models. Most existing datasets for visual recognition do not take account of input degradations that can be caused in extreme environments, which leads to severe performance degradation of existing models in extreme conditions. In the second half of this talk, we will introduce our attempts to resolve this problem including the acquisition of real-world extreme condition datasets and the synthetic generation of realistic extreme condition datasets. We will also discuss about remaining challenges and share our ideas on possible future directions.

While AI systems powered by deep neural networks have made tremendous success in solving various real-world applications domains such as computer vision and natural language understanding, such good performances are often obtained by training and evaluating on curated benchmark datasets, that contain clean, fully labeled data instances wit predefined set of tasks. However, in real-world scenarios, models may need to be trained with insufficient amount of data, partly-labeled data, and data available only on local devices/institutions, with increasing set of classes or tasks.

Moreover, architectures with hyperparameters that have been tuned on benchmark datasets may not be optimal for real-world data, tasks, and hardware constraints, and extensive search for optimal model architectures and hyperparameters may be necessary to find the optimal models for a given combination of datasets and devices. In this talk, I will introduce models and algorithms to tackle such new challenges that arise when deploying deeplearning models to real-world AI applications.

Acquiring high quality images is a long-term goal of image processing and computational photography. In this talk, I will discuss various aspects of high quality image acquisition, such as definition of high quality, image restoration and enhancement, and novel acquisition setup and process. First, I will propose the term of crispy pixels as my own definition of a high quality image that considers perception and aesthetics as well as domainspecific properties. Then, the core ideas and potential future directions of conventional and novel approaches for restoring and enhancing captured images to obtain crispy pixels will be introduced, including image priors, optimization, deep learning, multi-frame fusion, and multi-cameras.

I will also discuss computational imaging approaches, such as deep optics, where the acquisition setup is optimized together with the image processing pipeline to achieve extreme image quality. Finally, I will present my personal views on the future of imaging that may provide novel experiences and applications of images and videos probably with 3D information.

Underwater imaging is a challenging task to provide visual information for underwater scenes due to extreme underwater environments. We often suffer from degraded quality and low visibility of underwater images such as low contrast, haze, blur, and color distortion, since the light is absorbed and scattered by turbid water and/or floating particles in the water. In order to overcome the visibility degradation of underwater images, conventional methods designed simplified underwater image formation models by assuming ideal environments or adopted deep learning techniques using synthetic training datasets.

However, the model based methods yield limited performance of underwater image restoration often failing to work under harsh environments such as muddy water and green algae, and the data-driven methods suffer from the lack of paired datasets including ground truth for supervised learning. In this presentation, I will introduce our recent research works for underwater image restoration and enhancement. A model based underwater image restoration method is first introduced based on a complete and unified underwater imaging model. Moreover, we consider the underwater image enhancement as an image-to-image translation problem among different domains and propose a data-driven method for underwater image enhancement based on deep learning framework. Last but not least, we discuss the issues and challenges of underwater image restoration and enhancement.

Non-Line-of-Sight (NLOS) imaging allows us to see hidden objects or scenes not in the line of sight. State-of-the-art NLOS imaging technology has showcased the great potential of NLOS imaging as the next-generation imaging system. However, conventional NLOS systems capture large amounts of input data through point-of-point focal scanning in general and reconstruct albedo properties of hidden scenes in a very low spatial resolution through a long computational process.

In addition, solving an inverse rendering problem relies on various simplifications and constraints in a controlled lab environment. While NLOS imaging has great potential for applications in automotive, medical, surveillance, military, etc., many fundamental problems must be solved to put NLOS imaging into practical use in everyday life. As the ultimate goal for making NLOS imaging practical, we focus on two different aspects of NLOS imaging challenges. First, we introduce a novel differentiable transient rendering framework to help bring the potential of differentiable approaches into the transient regime.

Non-Line-of-Sight (NLOS) imaging allows us to see hidden objects or scenes not in the line of sight. State-of-the-art NLOS imaging technology has showcased the great potential of NLOS imaging as the next-generation imaging system. However, conventional NLOS systems capture large amounts of input data through point-of-point focal scanning in general and reconstruct albedo properties of hidden scenes in a very low spatial resolution through a long computational process.

In addition, solving an inverse rendering problem relies on various simplifications and constraints in a controlled lab environment. While NLOS imaging has great potential for applications in automotive, medical, surveillance, military, etc., many fundamental problems must be solved to put NLOS imaging into practical use in everyday life. As the ultimate goal for making NLOS imaging practical, we focus on two different aspects of NLOS imaging challenges. First, we introduce a novel differentiable transient rendering framework to help bring the potential of differentiable approaches into the transient regime.

To differentiate the transient path integral, we need to take into account that scattering events at path vertices are no longer independent; instead, tracking the time of flight of light requires treating such scattering events at path vertices jointly as a multidimensional, evolving manifold. We thus turn to the generalized transport theorem and introduce a novel correlated importance term, which links the time-integrated contribution of a path to its light throughput, and allows us to handle discontinuities in the light and sensor functions.

We have presented results in several challenging scenarios where the time of flight of light plays an important role, such as optimizing indices of refraction, NLOS tracking with nonplanar relay walls, and NLOS tracking around two corners. Second, we propose a novel neural network-based solution, which is built upon the considerations of several distinct features of the existing NLOS system. Our network adopts the Transformer architecture to decode the outputs from the globally correlated inputs.

On top of this, we introduce the hierarchical Transformer decoder, which can effectively deal with the severe ill-posedness of NLOS imaging by decoding the outputs in a coarse-to-fine manner. We also redesign the backbone to be more specialized to the NLOS imaging configuration, by providing the geometry information of the measurements and aggregating the inputs into the scan-wise features.

We anticipate that the proposed contributions help the state-of-the-art NLOS imaging technology become more mature and that this study is moving toward making NLOS imaging more practical, in order to realize the true impact of this new imaging technology.

3D perception from video cameras is essential for robotics and AR/VR systems. We have developed omnidirectional dense depth estimation (OmniMVS) and visual SLAM (OmniSLAM) which uses a multi-camera system with ultra-wide FOV (>220 degrees) fisheye lenses. Our original camera configuration is effective in capturing the images of scenes with sufficient multi-view stereo overlaps, which enables pure image-to-image matching for depth estimation without considering the semantics. Our OmniMVS algorithm employs a hybrid 2D/3D deep neural network for accurate 360-degree depth measurements. For robustness and accuracy, the OmniMVS network is trained with numerous synthetic images with ground-truth depth values generated in various virtual indoor and outdoor environments. For robust motion estimation, our robust visual-inertial odometer algorithm (ROVINS) computes the camera trajectory using both 360 image features and IMU measurements. For robust and accurate environment map building, OmniSLAM utilizes the OmniMVS depthmaps and the visual features in all directions. By combining the dense depth measurements by OmniMVS and accurate trajectory by OmniSLAM, dense 3D mesh modeling of the environment became possible. We are currently improving the quality of 3D modeling by investigating recent advances in deep-learning based 3D modeling including various NeRF algorithms, and finding synergetic combination of our depth-sensing algorithms and neural network-based modeling approaches. In this talk, I will introduce the core ideas of our algorithms and the applications such as automated 3D mesh generation and a visual positioning system.