Animation in X-Reality

   

Summer School in Vilnius

 
 
   
Projection mapping test on a replica of the Vilnius city hall to test the final projection. Video by Maxim Diehl.

The Animation in X-Reality summer school explored the expanded field of animation, encompassing motion capture, projection mapping, sound production, and creative artificial intelligence. Participants engaged in practical workshops led by collaborating universities, alongside enriching field trips and cultural activities.

This project was a collaborative effort among four universities: the Department of Graphical Systems at Vilnius Gediminas Technical University (the host), KISD (Köln International School of Design), Tampere University of Applied Sciences, and Academy of Media Arts Cologne.

KISD’s Contribution: AI for projection mapping

KISD’s workshop focused on collaborating with and appropriating artificial intelligence for filmmaking. A novel text-to-image (TTI) approach was applied in conjunction with various other neural architecture-based techniques, such as upscaling and frame interpolation. This involved combining image synthesis processes using text-to-image with traditional creative techniques like animation, sketching, and photography, fostering an experimental human-machine co-creation.

A core component of the workshop involved utilizing PyTTI-Tools, a powerful suite for generating CLIP-guided images. PyTTI’s origins trace back to 2021, when practitioners, researchers, and artists pioneered these text-to-image approaches. Specifically, Katherine Crowson1 published a foundational notebook, which sparked numerous variations and feature additions by other innovators. Henry Rachootin (@sportsracer48) then assembled key elements from these notebooks to create PyTTI. Later, David Marx (@DigThatData) significantly expanded upon PyTTI, transforming it into the comprehensive PyTTI-Tools that has become a go-to for advanced image generation.

A unique aspect of the KISD workshop was the application of image-to-image diffusion on actual architectural photos and drawings of the Vilnius City Hall. This technique allowed the initial weights for the diffusion guidance to be informed by the existing architectural features, which were then intertwined with creative prompts stemming from engaging with Vilnius’ rich cultural history.

The workshop provided hands-on experience with Python commands, Jupyter Notebooks, and command-line interfaces, familiarizing students with the JupyterLab and Anaconda ecosystems. Crucially, it also included a reflection on the training data used by the models and discussions on their potential societal impacts. The aim was to convey a basic understanding of command-line interfaces and to build intuition about how large-scale machine learning networks can contribute to the workflow of designers and artists. The workshop was conducted by Lisa Marleen Mantel and Laura Wagner, and organized in collaboration with the research project KITeGG on design and AI.

Other Workshops and Cultural Activities

The host university, Vilnius Gediminas Technical University, offered a workshop on music production with composer Marius Salynas, and another showcasing different motion capturing techniques, specifically optical motion capture. Tania de León Yong from the Academy of Media Arts Cologne led a workshop on projection mapping practice and the software Resolume.

Cultural activities included a trip to the Palace of Grand Dukes of Lithuania, a visit to a contemporary dance performance, a trip to the historic stronghold of Trakai, and a city tour in Vilnius with students from the host university.

Project Outcome and Follow-up

The project culminated in a projection mapping presentation on a model of the Vilnius City Hall building. As a follow-up, participants are now invited to refine their artwork to qualify for projection on the actual City Hall building during the light festival and the city’s 700-year anniversary in January 2022.

Process Images

A few glimpses into the creative process during the summer school workshops.

Students

  • Maxim Diehl
  • Alex Koch
  • Johannes Growe
  • Joshua Probst
  • Bessie Normand
  • Samia Rachid
  • Bea Timmermann

KISD Faculty

Duration & Location

May 23 - June 4, 2022 Vilnius Gediminas Technical University, Vilnius, Lithuania

Project Partners

  • Vilnius Gediminas Technical University
  • Tampere University of Applied Sciences
  • Academy of Media Arts Cologne
  • BMBF-Project KITeGG: Making AI Tangible And Comprehensible: Connecting Technology And Society Through Design

For more details, you can read the original article on the KISD (Köln International School of Design) website: Animation in X-Reality – Summer School in Vilnius

  1. Katherine Crowson et al., “VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance,” arXiv, last revised September 4, 2022, arXiv:2204.08583, https://doi.org/10.48550/arXiv.2204.08583. ↩