Second "Science as Art"-Exhibition at the AI+X Summit '23

structurally colored hydroxypropylcellulose nanohelices on 3D print, 10 x 10 x 3cm

Alexander Jessernig is an experienced "artist" in the context of the "Science as Artworks" series, as he already participated in the first exhibition at the Kunsthaus Zürich in 2022. Back then he worked on a vase in the shape of a light bulb with his structural colors. Now he has developed his work further with a clearer focus on the subject, which receives his nanohelices blessing.

He presents us here with a Structural Colors Beetle. It is a 3D print of a stag beetle, which Jessernig has wrapped in his familiar iridescent hydroxypropylcellulose garb and in turn - also to immortalize the work - cast in epoxy resin. Live stag beetles in the wild are black or brown, but never this blue-green iridescent. It is the so-called jewel beetles, which are blue-green iridescent. The iridescence common to these beetles is not due to pigments in the exoskeleton, but instead is caused by structural coloration, in which microscopic texture in their cuticle selectively reflects specific frequencies of light in particular directions, like in Jessernig's work.

Belgian artist Jan Fabre worked with these magnificent beetles and, among other works, attached 1.6 million of them to the ceiling and chandeliers of the Hall of Mirrors in the Royal Palace of Brussels for his work Heaven of Delight. Jessernig takes a more intimate and Earthly approach, bringing splendor and jewels not to already magnificent places, but adding some glamour and glitter to the rather terrifying and frightening stag beetle male and therefore creating some lovely kind of evolutionary aberration.

Poster, digital print, 119 x 168 cm

The digital print How AI Sees Art is a map to explore how AI groups images of artworks by some seminal artists. The AI, in this case, OpenAI's CLIP, groups the art pieces according to its own embedding space.

Trigger for this analysis was the recent legal dispute between The Andy Warhol Foundation for the Visual Arts and photographer Lynn Goldsmith. Goldsmith took a photograph of pop musician Prince that Andy Warhol included in his famous silkscreen series. The Andy Warhol Foundation lost the dispute. The US supreme court ruled that Warhol's work did not fall under "fair use." For some advanced definition of "distance", Warhol's adaptation is indeed a close relative of Goldsmith's original photograph. However, any viable definition of "distance" may be tricky and attackable.

In How AI Sees Art the AI groups art according to the motifs (horses, portraits), in some cases according to era or color scheme. Grötschla and Wattenhofer then project the 512-dimensional embedding space of CLIP to a human-understandable 2D canvas. Some positions on the map are actually multilayered with images in the background that are not visible. What comes out is a glimpse into the mind of a modern multimodal machine learning model that creates its own island archipelago, with a horse island in the north and a special Frida Kahlo island in the south. The flower island in the west, has, if we consider the images in the back layers, actually a volcano. In the center of the map there is a valley of naïve abstract and art brut drawings – a lion smiling there - leading north-west into the plateau of naïve figurative and anatomic drawings. The peninsula of Christian painting seems to have its highlights in the cliffs of red-dressed Madonnas with a blue veil and nude baby Jesus.

The topography of How AI Sees Art can create interesting curatorial narratives almost as travel logs, if it also generates new art, historical or even conservatory insights will depend a lot on the data set, of course.
 

“It was the summer of 2020, […] and the world around us was spiralling. The dissonance was impossible to ignore. Inside our confined walls, we were surrounded by the vast and fast growing world of AI. Billions of dollars worth of datasets and TPU years, as well thousands of human years captured in research papers. All in the name of an implicit or explicit promise for a better future in science, health, agriculture, energy management, transportation, information consumption, human machine interaction, etc. Yet, it all seemed so disconnected from the reality of the world outside.

Rosy AI, is a Python program that applies a human-engineered algorithm to solve a mosaic puzzle, overlaid with an AI model that tries to “understand” the emerging image using state-of-the-art object detection technology. The mosaic is composed of 32K tiles of flower images that were originally compiled as a dataset for the purpose of training AI models in classifying flowers. It seemed to adequately represent the rosey image of AI, as reflected by the walls of its confined universe.

Starting from a random distribution or an initial image, tiles are replaced either by some best best fit algorithm, or are randomly selected for a potential switch, if the switch represents a Pareto-improvement in favor of the final image. The human viewer is a witness to AI’s attempts at recognizing the shapes emerging in the mosaic […].

The final art piece is […] a computer program running live in an infinite loop that renders a live video on the display. Each iteration of the loop is about 10 minutes long, and it captures the formation of the photo mosaic. When it completes the mosaic, it starts all over again.»

“AI Art: Rosy AI” by Yariv Adan, published March 30th, 2021 in towardsdatascience.com

Aluminium Dibond Digital Print, 60 x 60 cm

The three prints Guilty Pleasure, Growing Inequality and Timber Wall are excerpts selected by Konrad Schindler from the “Global Canopy Height Map for 2020", produced in 2021 and made accessible online in 2022. The map shows the global canopy top height for the year 2020 at 10 m ground sampling distance — i.e., with a 10 m resolution. A probabilistic deep learning model has been developed to retrieve canopy top height from Sentinel-2 satellite images anywhere on Earth. This model, an ensemble of convolutional neural networks, was trained with sparse supervision from canopy top height measurements derived from NASA’s Global Ecosystem Dynamics Investigation (GEDI).

The first section Schindler chose here is the West African Gulf of Guinea; Ghana, Ivory Coast and Liberia. In Ghana (bottom right of the image) one can see several protected forest reserves, standing out like remnants of gold plating. They are, as the title suggests, mainly surrounded by cocoa plantations that produce the beans for our Swiss (and other) chocolate. Since 2022, ETH runs a Mechatronic Engineering Masters program at Ashesi University north of the capital city Accra (on the far right). Further to the left are the large, clearly contoured orange patch of Taï National Park in Côte d'Ivoire and the seemingly pristine forest of Liberia, extending all the way to the coast.

The dark, largely treeless area at the top of the image are the savanna of Burkina Faso and Mali, stretching towards the Sahara Desert.
The second section is from the US state of Oregon, south of the city of Eugene. The checkerboard pattern we see are traces of the "O&C" lands. In 1866, every other square mile of land was granted to the Oregon and California Railroad. These lands now belong to the federal government and are managed by the Bureau of Land Management (BLM). They form the light squares of the checkerboard, which are populated by tall forests, while the dark squares are privately owned and used for timber extraction.

The third section shows the Hyrcanian Forests north of Tehran, Iran, covering the slopes of the Alborz Mountains along the Caspian Sea. The clear line of the timber wall is a natural formation of tall mountain forests nurtured by water evaporating from the Caspian Sea in the otherwise largely dry and barren region. Parts of it have recently been declared Unesco World Heritage.

The works are reminiscent of landscape photography, an artistic medium that Konrad Schindler also enjoys himself. In terms of art history, they also remind of land art projects of the 1960s and 1970s, like Robert Smithson’s Spiral Jetty at the Great Salt Lake. Compared to them, the natural and man-made landforms depicted in the prints are monumentally larger.

Special thanks to Dr Nico Lang from the EcoVision Lab, Photogrammetry and Remote Sensing, ETH Zurich and the Department of Computer Science, University of Copenhagen.
 

Aluminium Dibond Digital Print, triptych, 40 x 60cm

Gold Stars in a Crystal Forest is a tryptich of scanning electron microscope (SEM) images taken at different horizontal field widths (HFW 829, 59.2 and 10.4 µm). Scientifically the goal behind the images is to visualize gold nanoparticles (20 nm) that were conjugated with specific antibodies. The antibodies should be attached on the surface of the nanoparticles and recognize specific proteins. Thanks to the gold coloring of nanoparticles one can detect the proteins in specific applications and locations. The gold nanoparticles are kept in a sodium citrate solution. In the picture we see crystals of the sodium citrate as the "forest", while the small bright "stars" are the gold nanoparticles.

Gold Stars in a Crystal Forest remind us of bioluminescent fireflies and glowworms sparkling in the forest at night. Or, as is often the case in microscopic images, it reminds us of the macrocosm of the astronomical night sky.

In explaining their work, Dosnon and Cipolato are also very enthusiastic about the way the SEM works, in which the electrons are accelerated to very high speed to leverage their quantum nature and their particle/wave duality to overcome the limits of optical microscopy and access information at the nanoscale. This quantum duality is also alluded to in the surrealistic composition in which the cosmos of stars and galaxies is scattered in algae, ferns and snowflakes.
 

Interactive Installation

Segmenting humans in 3D indoor scenes has become increasingly important with the rise of human-centered robotics and AR/VR applications. To this end, Engelmann and his collaborators propose the task of joint 3D human semantic segmentation, instance segmentation and multi-human body-part segmentation. Their 3D segmentation identifies the head, right and left, upper and lower arm, both hands, the torso, the hips, the right and left, upper and lower legs as well as both feet.

The segmentation is reminiscent of Kazimir Malevich's suprematist work "Sportsmen" or "Sensation of Danger" aka "The Running Man" from 1931 and other figures from that time. With the interactive work, in which we as visitors can immerse ourselves, Malevich's work is in a way activated and we can all wear the futuristic costumes of that modern age and become athletes running after a sensation of danger.

To some extent, the research behind this work also reflects the progressive drive of the time, as the creators propose not only a framework for generating training data of synthetic humans interacting with real 3D scenes, but also a novel transformer-based model, which they call Human3D, which is the first end-to-end model for segmenting multiple human instances and their body-parts in a unified manner. In the constructivist period of Malevich, human segmentation was also a means to find new forms of expression, for example in dance. In the (de)constructivist segmentation we experience here, we ourselves synthetically become Human3D personae and automatically start to dance, to put the technology itself to the test - perhaps soon also with robots and in AR/VR applications.
 

Poster, 42 x 59 cm

While attempting to create a fast variant of dreambooth, a technique for fine-tuning the stable diffusion image generator to generate pictures of a specific new person, Nava created Failures by accident: When the hyperparameters are wrong, the generation process breaks down and no intelligible pictures can be produced anymore. However, when the process goes wrong but not completely, some weird and eery pictures can still be generated. These pictures often depict the fine-tuning subject Elvis Nava, but also can not. Also, they no longer bear any resemblance to the prompt that was used to generate them, which for some of them was a variant of «A centered professional realistic high key head shot of [Subject] man wearing a tailored blue suit, diffused studio lightning, light gray background, high focus, relaxed pose, symmetrical, natural face, serious eyes 40mm lens, HDR, UHD».

Besides, failures, accidents, chance and other non-intentional techniques in art, of course self-portraits have a long tradition in art history. Albrecht Dürers «Self-portrait» from 1500 might be one of the first and most famous. Also there, it seems to have a prompt saying “Albertus Durerus Noricus / ipsum me propriis sic effin / gebam coloribus aetatis / anno XXVIII” – “So I, Albrecht Dürer from Nuremberg, painted myself in lifelike colors at the age of 28”. In Dürer’s time these kind of epigraphs were close to amulets that bore wishes, spells and charms.

The prompt that Nava used seems to have created some kind of spell, that generated a multitude of bizarre portraits and other motifs and glitches. So, today the epigraph of the generative fan poster of self-portraits might be: “So I, Elvis Nava from Bergamo, generated myself, some houses and cities at lakes, some glitches and structures and a lot of other things in lifelike and non-lifelike colors at the age of 27”. While humans compared Dürer to Jesus Christ, the machine seems to compare Nava to skylines of picturesque towns.