This article is part of our exclusive IEEE Journal Watch series in partnership with IEEE Xplore.

In spatial augmented reality, light is projected onto surrounding objects to manipulate their appearance. The idea isn’t new, but researchers are now finding more and more sophisticated ways of taking the technique to the next level. Most recently, one group in Japan has developed an immersive augmented reality system that reduces blur, nearly eliminates shadows, and overall boosts users’ perception that the projections are realistic.

They describe their system in a study published in the May issue of IEEE Transactions on Visualization and Computer Graphics.

Projection mapping is a form of spatial augmented reality whereby projectors overlay digital images directly onto physical surfaces or objects, allowing users to see an altered reality without the need to wear a head-mounted display. While this kind of technology is great for entertainment purposes, it is also being explored for applications such as remote collaboration, healthcare, industrial design, urban planning, art creation, and office work.

“In many cases, the goal is not simply to display an image, but to change how a physical object appears—for example, its color, texture, or material properties,” explains Daisuke Iwai, a professor at the University of Osaka’s Graduate School of Engineering Science, who was involved in the study.

However, a major challenge in projection mapping has been to eliminate the shadows that appear as users move around the augmented reality space, for example as they pass in front of a projector or touch the augmented object. To reduce shadows, some research groups have developed projection mapping systems with many projectors from different angles. If a hand blocks light from some projectors, light from the remaining projectors still reaches the surface.

While multiple projectors can help reduce shadows, this approach gives rise to another issue. “Each projector illuminates the surface from a different direction, so the projected pixels have slightly different shapes on the surface,” explains Iwai. “When many such images are overlaid, the result can look blurred.”

He and his colleagues sought to address this issue by first measuring how a projected image is actually blurred on the target surface, and then pre-processing the input image so that the final overlaid projection appears sharper.

Avoiding image blurring

Importantly, the system estimates the blur of the combined projection and creates a single pre-compensated content image that is shared by all projectors after geometric alignment, Iwai says. “This keeps the method simple and scalable while preserving the sharpness of the projected result.”

In experiments on a flat tabletop, the new projection mapping system resulted in no clearly defined observable shadows when a hand was placed near the surface. The display’s brightness was only slightly reduced when a fingertip touched the surface. When the system was tested on a non-flat surface—a mannequin head—shadows were evident as users interacted with the object, but only faintly.

While the system offered image quality comparable to a conventional projection mapping system also evaluated in the study, the new blur compensation technique was able to achieve much faster computations speeds, at about 100 seconds, compared to the conventional system, at 2,500 seconds.

The researchers also assessed their projection mapping system through user studies involving 17 participants. In one scenario, the system projected text onto the table next to physically printed text. Participants were significantly slower and less accurate at identifying projected text, suggesting that the projected content became harder to distinguish from real printed content.

Iwai notes that the system currently requires individual calibration of each projector, and so in future work the team is interested in developing distributed rendering systems where each projector is controlled by a small edge computer.

“Additionally, we would like to test the system in real application scenarios, such as industrial design support, where accurately changing the perceived material properties of a physical object could be very valuable,” he says.