This bizarre image-translation tool turns drawings into photographs

A new machine learning-based image generator, called the Image-to-Image Demo, is getting attention for the bizarre creations it produces.

The interactive image-translation demo uses inputted outlines, or “edges,” to produce a complete photographic image, based on a body of images on which the machine learning algorithm used has been trained.

Sometimes this works out as you would expect. If you draw an outline of a shoe into a version of the tool that’s been trained on shoe images, you’ll likely get what looks like a shoe.

But input an outline of a cat in the cat-image generator, and you might notice the head and eyes ending up somewhere unexpected.

You can also erase the lines pre-supplied in the online demo and draw your own. You can, for example, draw a triangle and fill it with a blend of cat imagery.

“Sometimes the lines you draw are going to be really different from what it has seen before,” Mountain View-based engineer Christopher Hesse, who created the demo, told Recode. “You get some pretty bizarre results.”

Hesse created the tool to practice his skills in machine learning, a discipline he is studying in his spare time. The browser tool is based on an algorithm Hesse credited to UC Berkeley computer science postdoctoral student Phillip Isola. (Isola has co-authored a paper related to the algorithm. He declined to comment for this story, saying that the paper was still under review.)

According to Hesse, the algorithm was originally implemented in Torch, a computing framework that supports machine learning algorithms. Hesse ported it over to Google’s open-source machine learning framework, Tensorflow, which he is trying to learn. After translating the algorithm over to Tensorflow, he built the browser demo.

Hesse has trained the software on image databases of building facades, shoes, handbags and, of course, cats. He has two kinds of images from each data set — complete images and extracted outlines of those images.

But, really, it’s all about the crazy cats that it creates.

Hesse explains why the cat images stand out as particularly weird: “Some of the pictures look especially creepy, I think because it's easier to notice when an animal looks wrong, especially around the eyes. The auto-detected edges are not very good and in many cases didn't detect the cat's eyes, making it a bit worse for training the image translation model.”

This article originally appeared on Recode.net.

We have a request

Vox's journalism is free because we believe that everyone deserves to understand the world that they live in. That kind of knowledge helps create better citizens, neighbors, friends, parents, consumers and stewards of this planet. In short, understanding benefits everyone. You can join in on this mission by making a financial gift to Vox today. Reader support helps keep our work free, for everyone. Will you join us?

One-Time Monthly Annual

$95/year

$120/year

$250/year

$350/year

Other

$

Yes, I'll give $120/year

Yes, I'll give $120/year

We accept credit card, Apple Pay, and Google Pay. You can also contribute via