Neural City

Neural City

Neural City is my first attempt to create a video using a machine learning algorithm. Using NVIDIA's pix2pixHD machine learning script, I took a simple animation and completely transformed it. The dataset used to train this model is the Cityscapes dataset, hence the German looking features. Find links to the resources below:

http://chung.work/

https://www.cityscapes-dataset.com/

https://github.com/NVIDIA/pix2pixHD


Process

All of the images in the video were generated by outputting color block images of digitally modeled cityscapes. I first tested this workflow by mocking up images using stock vector in illustrator.

An image created using illustrator and vector silhouettes. The colors used correspond with the pre-trained algorithm.

An image created using illustrator and vector silhouettes. The colors used correspond with the pre-trained algorithm.

The resulting image after applying the pix2pix algorithm.

The resulting image after applying the pix2pix algorithm.

Applying the pix2pix algorithm produced this output:

While the fidelity of the image is low, it was successful as a proof of concept. I continued to test various outputs against the algorithm to see how much variance it could handle.

A challenging image for the pix2pix algorithm’s training data.

A challenging image for the pix2pix algorithm’s training data.

This image shows that when the input image diverges too much from the existing training data, it produces a poor result.

This image shows that when the input image diverges too much from the existing training data, it produces a poor result.

I learned that its tolerances were quite restrictive. Essentially, because most of the data had been collected from a dashboard camera, all of the images should look like those taken from a dashboard camera, and anything that did not look like it was taken in that manner would result in a strange looking image as seen above.

The 3D model I worked with was designed to resemble the Cityscapes dataset as much as possible while still being unique

The 3D model I worked with was designed to resemble the Cityscapes dataset as much as possible while still being unique

These results are marginally better, but still look strange when animated.

These results are marginally better, but still look strange when animated.

Additional images from the video below:

cornershot0358.jpg
cornershot0358_synthesized_image.jpg
streetcrane0206.jpg
streetcrane0206_synthesized_image.jpg
Thranger-2000

Thranger-2000

Chair for a Little One

Chair for a Little One