A Brief History of Lighting in VFX

Post by Pete Wailes

Light. From even before the moment a new-born baby opens its eyes, we're surrounded by it. From the way the sun shines on a clear or cloudy day, to the difference between harsh and soft lights in the home, our brains have developed an uncanny ability to understand the way light falls on a scene in front of us, and understand what that says about the sources of light present.

It's no wonder then that the world of lighting in VFX is as much an art as it is a science. Striking the right balance between sources to create the right look, while balancing the need for accuracy against the desire to create the right aesthetic requires a mix of incredible creativity and an eye for detail, combined with some of the most intelligent software in the world.

Learning From Old Masters

No study of lighting can be complete without beginning by looking at the people who first mastered the art of using light effectively. Giotto, the first of the Italian Renaissance artists from the 14th century, Piero della Francesca in the 15th, Raphael and Michelangelo the 16th and so on all contributed to the steady body of work that by the 17th century produced some truly outstanding works of lighting in art.

The first truly great depictions of light in art come from the Dutch masters in what's now referred to as the Dutch Golden Age, with the move to realism in landscapes and interiors, as well as portraiture. These were amongst the first to attempt to accurately depict lighting in visual imagery, with artists like de Hooch, van Ruisdael, Cuyp and of course Rembrandt typifying the attempts of the age to create what we would consider realistic lighting, known as chiaroscuro. Chiaroscuro (literally light-dark in English) is the practice of using contrasting lightness and shade in art to produce the effect of depth and volume in flat media, such as paintings.

Of particular note in this cannon is their contemporary Johannes Vermeer, who dispite being lightly regarded during his life and almost completely forgotten after his death, is now considered as one of the true masters of the period, with his use of lighting in particular being second to none.

Starting with these figures and advancing through other greats, we've steadily developed an understanding of how to capture and express lighting in an accurate way. With those principles understood, the challenge for VFX has been to pull the same tricks to fool our minds into seeing things as real that can't possibly exist.

The Beginnings of Lighting in Film

When we now see Transformers or Avatar or this year's Big Hero 6, the lighting that we see used there in the digitally created world still draws from the real world. Even in completely CG worlds, references from the real world are used constantly to make sure that the end effect has the right "feel" to it. But what exactly is that feeling?

The same techniques that we use today to create a realistic feel of lighting in the mostly two dimensional world of film and TV were those the artists of old used to create depth and impact in their work. Chiaroscuro is as much a part of modern visual artistry as ever it has been. One only has to look at the AFI's list of greatest films to see this in evidence. Citizen Kane, Casablanca, Star Wars, 2001: A Space Odyssey, Gone with the Wind, Schindler's List, It's a Wonderful Life, E.T. the Extra-Terrestrial, Raging Bull... The list is a study of great lighting in film.

The irony of course with this is that great lighting in film, which looks natural and yet expressive, is lighting that never actually occurs in real life. To accurately reflect depth and create the right feel for the scene, the DOP (director of photography) will choose lenses, filters, and direct lighting choices. These things all artificially create excess contrast, which is necessary to produce the result of a natural looking shot, as without these, the worlds we see would appear washed out and bland.

It's therefore no surprise that when back in 2003, a survey of the International Cinematographers Guild produced a list of the ten most influential cinematographers, the list of films they'd worked on was studded with the same films as the AFI list.

The Digital Transition

The real magic in VFX in lighting doesn't really emerge until Toy Story. This is because, for the most part (even today), most lighting for VFX work is done through the use of practical effects mixed with CGI replacement. For example, the burning Skyfall Lodge at the end of Skyfall was lit through the use of a huge bank of lights to create the fire, with the lights removed and burning lodge adding in using VFX after the fact.

The best CGI lighting really happened in the animated world long before it turned up in film to any serious degree. As a result, one can't talk about computer graphics lighting without talking about Pixar, and mentioning Sharon Calahan. In 2014, Calahan was inducted into the American Society of Cinematographers, as was the first ASC member whose entire body of work is CGI1. Having joined Pixar as a lighting director of photography for Toy Story, she continues to work there now, and is considered one of the greatest cinematographers in her chosen medium. So let's have a look at the difference between the old and new systems of lighting from masters of the animated film, and break down how the lighting effects improve over time.

The Early Years

Back in the early 90's when Toy Story was being produced, Pixar hadn't yet decided that there should be a DOP. That only happened in A Bug's Life, their next film three years later. While producing Toy Story, the feeling was that the processes of creating an animated film through CGI were more like cell animation than live action film making. However, much of the methodology of lighting remained the same until relatively recently.

Scenes were, until Monsters University, lit using a process called direct lighting, in which lights are placed into a scene and directed toward a certain location. It's a very computationally simple way of creating lighting in a scene. By specifying where you want each light, and how you want it to act in the scene, you minimise the level of calculation involved. This has two consequences: firstly, it seriously limits the volume of lights than can be placed into a shot, as each one needs to be placed and directed. It also means that every shot has to be simple. If every shadow was specifically placed, every light manually set up, it forces animators into a position where the complexity of the models they create can only be pushed so far.

The Modern Solution

To get around this, Pixar switched to a system called Global Illumination for the creation of Monsters University2. The basic difference between direct lighting and global illumination is that global illumination calculates where light should travel based off of the surfaces it hits, their colour and texture and material, creating a system where lights can be placed and all the effects of those lights being updated correctly. Additionally, every reflective surface then becomes its own light source, casting light correctly to the rest of the scene. Imagine shining a torch in a dark room - light hitting every surface casts a light everywhere else. Rendering this kind of interaction is where global illumination comes into its own.

The system Pixar created still allows for the artists to dial up or down the realism knob as it were, pushing shadows and lighting over where they could ever be in film to create a more cartoon-esque look, but the fundamental physics involved still stays true to the real world. In fact, almost the entire film was created using this technique, because of its success. However, some particular parts were still rendered using direct lighting, such as the character Dean Hardscrabble, due to her particular aesthetic being designed to cast very specific shadows. Unfortunately, due to the way the GI systems work, it makes it incredibly difficult for lighting artists to predict how shadows and reflections will look without running the entire scene. Given that that can take several hours, it's not a simple ask. As a result, tool chains which include real-time rendering functionality to create simplified versions of scenes are now being worked on to reduce the time required to tweak shots.

Measure Twice, Light Once

An example of modern lighting taking to an extreme is in Christopher Nolan's Inception. Due to the particularly bizarre nature of the physics depicted in the film, the lighting had to be completely believable, in its unreality. For example, the famous scene where the city of Paris folds over onto itself, the team had to move to using Pixar's RenderMan to create the effect of the buildings being lit from above on the two different versions of the city. By matching the result images against photography shot on location, the team were able to tweak the end result to produce an effects sequence that perfectly matched the lighting present in the real world on both layers of the city.

The challenges of creating a photo-realistic scene that couldn't possibly exist didn't end there though. The team also had to create things like famous exploding reality shot, where a Paris street scene blows up around Cobb and Ariadne. To create that shot, the team filmed the explosions first using super-high speed cameras, then shot the actors. By overlaying one on top of the other digitally, they were able to put the characters into the explosion itself, and manipulate physics around them.

Through the mixing of purely CGI environments, mixed real and digital world shots and inventive film making, Nolan's team were able to batter physics into submission over and over again so successfully that the won the Best Visual Effects awards at the Academy Awards, BAFTAs, and Saturn Awards, as well as the same award from a host of critics awards as well.

The State of the Art

More recently though, advances in the size and power of render farms has allowed film makers to go even further. Again, we turn to Christopher Nolan, only this time with the film Interstellar. To make sure the black hole central to the film was rendered accurately, the team relied on the expertise of Kip Thorne, a theoretical astrophysicist, late of Caltech and executive producer on the film. Using the equations that govern how space and time bend around a black hole, and those that describe how light behaves, the team constructed a simulation of a black hole3.

To create the accretion disk, the disk of light and matter that spins around a black hole's axis, the team then created a flat, multi-coloured ring around the black hole, to simulate light falling into it. Unexpectedly, the equations produced a warping of the accretion disk itself, turning it into a halo around the hole. Light gets bent upwards from the disc into a massive arc, and under it as well, forming an upside down ripple, almost mirroring the top half. The end result was so compelling, that no manipulation was needed to make it more impressive or understandable. The physics of the universe itself was incredible enough.

The end result of this has been that two papers will be published from what the VFX time have learned in working on this, one for the physics community and another for the CGI crowd. It's a wonderful example of art imitating life, and creating something that's both beautiful, and true. And in the end, that's the wonder of modern lighting simulation. Even when it's doing something that's at the edge of physics, or even beyond it, it's still revealing something about the nature of physics, about the wonder of mathematics, and using those two most base of tools to delight us.