Naples/Darmstadt: Hardly anything remains of the colorful surface decoration of ancient Greek sculptures, and the velvet fabric covers on furniture from later centuries have gradually faded with time. Works of art in museums today are often kept in glass display cabinets or cordoned off to prevent them being knocked against or interfered with by members of the public and handled or prodded by inquisitive children. However, they can hardly be protected from natural and artificial light which adversely affects the original colors over the years. As a scientific partner in the PERCEIVE research project, the Fraunhofer Institute for Computer Graphics Research IGD is developing AI software that not only reconstitutes the original colors of the work of art but also predicts how they will change in the future. In addition to sculptures, the project is also focused on paintings, textiles, historical photos and digital art.

The starting point for the Fraunhofer IGD research team is material reconstruction, i.e. obtaining information about the surface. Team leader Pavel Rojtberg explains: “AI allows us to develop reflectance models, even in a virtual environment.” Until now, these models were created with a complex real-world setup of cameras and lighting, a procedure that is no longer necessary with the AI-based solution. “This saves time and money,” adds Rojtberg. On the sole basis of a few images taken with a smartphone or other device, it is possible to estimate the reflection and shadows cast on the respective materials such as velvet or silk.

Within a matter of seconds, the pre-trained AI software then depicts how the work of art and its colors appear under different light conditions, for example the gown worn by Britain’s Queen Victoria or a stone statue dating back to the Roman Empire. In addition, the technology will be able to automatically segment geometric data and thereby determine which part of a chair is made of wood and which is made of fabric.

The basis for the subsequent estimation of the color change over time includes historical photographs and recordings from multispectral cameras as well as chemical-physical properties of colors, which serve as a reference value. In this way, the software can also predict what the work of art will look like in a further fifty or one hundred years, depending on exposure to light. The solutions obtained will be applicable not only for digital remodeling, but also for actual physical restoration. This will allow curators to predict when such work will be necessary due to the progressive fading of colors.

The Fraunhofer team will develop an alpha version of the AI technology by the end of 2024, and it should be available for use by museums and researchers within the next three years. Holger Graf, head of the Virtual and Augmented Reality department: “When perfected, these tools will function in a purely web- and service-based manner, allowing them to be used not only on-site in museums via augmented reality, but also for virtual exhibitions.”