This bizarre image-translation tool turns drawings into photographs

A new machine learning-based image generator, called the Image-to-Image Demo, is getting attention for the bizarre creations it produces.

The interactive image-translation demo uses inputted outlines, or “edges,” to produce a complete photographic image, based on a body of images on which the machine learning algorithm used has been trained.

Sometimes this works out as you would expect. If you draw an outline of a shoe into a version of the tool that’s been trained on shoe images, you’ll likely get what looks like a shoe.

But input an outline of a cat in the cat-image generator, and you might notice the head and eyes ending up somewhere unexpected.

You can also erase the lines pre-supplied in the online demo and draw your own. You can, for example, draw a triangle and fill it with a blend of cat imagery.

“Sometimes the lines you draw are going to be really different from what it has seen before,” Mountain View-based engineer Christopher Hesse, who created the demo, told Recode. “You get some pretty bizarre results.”

Hesse created the tool to practice his skills in machine learning, a discipline he is studying in his spare time. The browser tool is based on an algorithm Hesse credited to UC Berkeley computer science postdoctoral student Phillip Isola. (Isola has co-authored a paper related to the algorithm. He declined to comment for this story, saying that the paper was still under review.)

According to Hesse, the algorithm was originally implemented in Torch, a computing framework that supports machine learning algorithms. Hesse ported it over to Google’s open-source machine learning framework, Tensorflow, which he is trying to learn. After translating the algorithm over to Tensorflow, he built the browser demo.

Hesse has trained the software on image databases of building facades, shoes, handbags and, of course, cats. He has two kinds of images from each data set — complete images and extracted outlines of those images.

But, really, it’s all about the crazy cats that it creates.

Hesse explains why the cat images stand out as particularly weird: “Some of the pictures look especially creepy, I think because it's easier to notice when an animal looks wrong, especially around the eyes. The auto-detected edges are not very good and in many cases didn't detect the cat's eyes, making it a bit worse for training the image translation model.”

This article originally appeared on Recode.net.

Will you join us?

Our biggest supporters are our readers — and we’re so grateful to everyone who has made a contribution during our September campaign. We’re less than 1,000 contributions away from reaching our goal for the month, which in turn will allow us to say yes more often when our incredible journalists come to us with questions they want to answer and projects they want to pursue. Will you make a contribution before the month ends and support our policy coverage through 2024 and beyond?

One-Time Monthly Annual

$5/month

$10/month

$25/month

$50/month

Other

$

Yes, I'll give $5/month

Yes, I'll give $5/month

We accept credit card, Apple Pay, and Google Pay. You can also contribute via