clock menu more-arrow no yes mobile

Filed under:

A guy trained a machine to "watch" Blade Runner. Then things got seriously sci-fi.

Aja Romano writes about pop culture, media, and ethics. Before joining Vox in 2016, they were a staff reporter at the Daily Dot. A 2019 fellow of the National Critics Institute, they’re considered an authority on fandom, the internet, and the culture wars.

Last week, Warner Bros. issued a DMCA takedown notice to the video streaming website Vimeo. The notice concerned a pretty standard list of illegally uploaded files from media properties Warner owns the copyright to — including episodes of Friends and Pretty Little Liars, as well as two uploads featuring footage from the Ridley Scott movie Blade Runner.

Just a routine example of copyright infringement, right? Not exactly. Warner Bros. had just made a fascinating mistake. Some of the Blade Runner footage — which Warner has since reinstated — wasn't actually Blade Runner footage. Or, rather, it was, but not in any form the world had ever seen.

Instead, it was part of a unique machine-learned encoding project, one that had attempted to reconstruct the classic Philip K. Dick android fable from a pile of disassembled data.

Sample reconstruction from the opening scene of Blade Runner.

In other words: Warner had just DMCA'd an artificial reconstruction of a film about artificial intelligence being indistinguishable from humans, because it couldn't distinguish between the simulation and the real thing.

Deconstructing Blade Runner using artificial intelligence

Terence Broad is a researcher living in London and working on a master's degree in creative computing. His dissertation, "Autoencoding Video Frames," sounds straightforwardly boring, until you realize that it's the key to the weird tangle of remix culture, internet copyright issues, and artificial intelligence that led Warner Bros. to file its takedown notice in the first place.

Broad's goal was to apply "deep learning" — a fundamental piece of artificial intelligence that uses algorithmic machine learning — to video; he wanted to discover what kinds of creations a rudimentary form of AI might be able to generate when it was "taught" to understand real video data.

As a medium, video contains a huge amount of visual information. When you watch a video on a computer, all that information has usually been encoded/compressed and then decoded/decompressed to allow a computer to read files that would otherwise be too big to store on its hard drive.

Normally, video encoding happens through an automated electronic process using a compression standard developed by humans who decide what the parameters should be — how much data should be compressed into what format, and how to package and reduce different kinds of data like aspect ratio, sound, metadata, and so forth.

Broad wanted to teach an artificial neural network how to achieve this video encoding process on its own, without relying on the human factor. An artificial neural network is a machine-built simulacrum of the functions carried out by the brain and the central nervous system. It's essentially a mechanical form of artificial intelligence that works to accomplish complex tasks by doing what a regular central nervous system does — using its various parts to gather information and communicate that information to the system as a whole.

Broad hoped that if he was successful, this new way of encoding might become "a new technique in the production of experimental image and video." But before that could happen, he had to teach the neural network how to watch a movie — not like a person would, but like a machine.

Do encoders dream of electric sheep? (Or, how do you "teach" an AI to watch a film?)

Broad decided to use a type of neural network called a convolutional autoencoder. First, he set up what's called a "learned similarity metric" to help the encoder identify Blade Runner data. The metric had the encoder read data from selected frames of the film, as well as "false" data, or data that's not part of the film. By comparing the data from the film to the "outside" data, the encoder "learned" to recognize the similarities among the pieces of data that were actually from Blade Runner. In other words, it now knew what the film "looked" like.

Once it had taught itself to recognize the Blade Runner data, the encoder reduced each frame of the film to a 200-digit representation of itself and reconstructed those 200 digits into a new frame intended to match the original. (Broad chose a small file size, which contributes to the blurriness of the reconstruction in the images and videos I've included in this story.) Finally, Broad had the encoder resequence the reconstructed frames to match the order of the original film.

In addition to Blade Runner, Broad also "taught" his autoencoder to "watch" the rotoscope-animated film A Scanner Darkly. Both films are adaptations of famed Philip K. Dick sci-fi novels, and Broad felt they would be especially fitting for the project (more on that below).

Broad repeated the "learning" process a total of six times for both films, each time tweaking the algorithm he used to help the machine get smarter about deciding how to read the assembled data. Here's what selected frames from Blade Runner looked like to the encoder after the sixth training. Below we see two columns of before/after shots. On the left is the original frame; on the right is the encoder's interpretation of the frame:

Real and generated samples from the first half of Blade Runner in steps of 4,000 frames, alternating real and constructed images.
Autoencoding Video Frames

During the six training rounds, Broad only used select frames from the two films. Once he finished the sixth round of training and fine-tuning, Broad instructed the neural network to reconstruct the entirety of both films, based on what it had "learned." Here's a glimpse at how A Scanner Darkly turned out:

Broad told Vox in an email that the neural network's version of the film was entirely unique, created based on what it "sees" in the original footage. "In essence, you are seeing the film through the neural network. So [the reconstruction] is the system's interpretation of the film (and the other films I put through the models), based on its limited representational 'understanding.'"

Why Philip K. Dick's work is perfect for this project

Dick was a legendary science fiction writer whose work frequently combined a focus on social issues with explorations in metaphysics and the reality of our universe. The many screen adaptations his works have inspired include Minority Report, Total Recall, The Adjustment Bureau, and the Amazon TV series The Man in the High Castle.

And then there's his famous novel Do Androids Dream of Electric Sheep?, which formed the basis of Blade Runner, a dystopian sci-fi masterpiece and one of the greatest films ever made. In the film, Harrison Ford's character Rick Deckard has a job that involves hunting down and killing "replicants" — an advanced group of androids that pass for humans in nearly every way. The film's antagonist, Roy Batty, is one of these replicants, famously played by a world-weary Rutger Hauer. Batty struggles with his humanity while fighting to extend his life and defeat Deckard before Deckard "retires him."

Dick was deeply concerned with the gap between the "only apparently real" and the "really real." In his dissertation, Broad said that he felt using two of Dick's works for his simulation project was only fitting:

[T]here could not be a more apt film to explore these themes [of subjective rationality] with than Blade Runner (1982)... which was one of the first novels to explore the themes of arial subjectivity, and which repeatedly depicts eyes, photographs and other symbols alluding to perception.

The other film chosen to model for this project is A Scanner Darkly (2006), another adaption of a Philip K. Dick novel (2011 [1977]). This story also explores themes of the nature of reality, and is particularly interesting for being reconstructed with a neural network as every frame of the film has already been reconstructed (hand traced over the original film) by an animator.

In other words, using Blade Runner had a deeply symbolic meaning relative to a project involving artificial recreation. "I felt like the first ever film remade by a neural network had to be Blade Runner," Broad told Vox.

A copyright conundrum

These complexities and nuances of sci-fi culture and artificial learning were quite possibly lost on whoever decided to file the takedown claim for Warner Bros. Perhaps that's why, after Vox contacted Warner Bros., the company conducted an investigation and reinstated the two videos it had initially taken down.

Still, Broad noted to Vox that the way he used Blade Runner in his AI research doesn't exactly constitute a cut-and-dried legal case: "No one has ever made a video like this before, so I guess there is no precedent for this and no legal definition of whether these reconstructed videos are an infringement of copyright."

But whether or not his videos continue to rise above copyright claims, Broad's experiments won't just stop with Blade Runner. On Medium, where he detailed the project, he wrote that he "was astonished at how well the model performed as soon as I started training it on Blade Runner," and that he would "certainly be doing more experiments training these models on more films in future to see what they produce."

The potential for machines to accurately and easily "read" and recreate video footage opens up exciting possibilities both for artificial intelligence and video creation. Obviously there's still a long way to go before Broad's neural network generates earth-shattering video technology, but we can safely say already — we've seen things you people wouldn't believe.

Sign up for the newsletter Today, Explained

Understand the world with a daily explainer plus the most compelling stories of the day.