Google Arts & Culture has teamed up with the Centre Pompidou, a cultural complex in Paris, to pay tribute to Vassily Kandinsky with a virtual exhibition of the artist’s works and other documents. You can view some of Kandinsky’s pieces in an augmented reality gallery. At the heart of the exhibit is a machine learning experiment that tries to replicate synesthesia, a condition the abstract art pioneer had.
In a nutshell, synesthesia turns information that stimulates one of your senses into a multi-sensory experience. For some people (including Kandinsky, Billie Eilish and Pharrell Williams), the condition deepens the association between colors and sounds or moods.
“When Kandinsky painted, two senses worked systematically together: hearing and sight,” Serge Lasvignes, president of the Centre Pompidou, wrote in a blog post. “Colors and shapes translated into sounds, harmonies, and vibrations made up lines and patterns.” He heard red as a violin and yellow as a trumpet, for instance.
For the “Play a Kandinsky” experiment, Google and the center recruited experimental musicians Antoine Bertin and NSDOS in an attempt to convey what Kandinsky may have heard while creating his art. They analyzed notes from Kandinsky about the synesthetic effects he felt and tried to replicate the sounds that the artist may have encountered while painting his 1925 piece “Yellow Red Blue.”
The team trained Google’s Transformer neural network using music from the time Kandinsky was around. The system then created new scores to try to determine what the painting might sound like from a current perspective.
You can click around and zoom into parts of the painting to experience sounds that the team (and perhaps Kandinsky) associated with the shapes and colors. You have the option to make your own sound mix and share it as well.
Those of us who don’t have synesthesia may never fully understand what those multi-sensory experiences are like, and folks who do have it might hear entirely different sounds when they see the same colors. Still, the experiment is a fascinating one, as it appears to give at least some idea of what it’s like to have multiple senses activated by the same stimuli.
Source: Google tries to replicate synesthesia with its latest experiment
