Lucasfilm’s Industrial Light & Magic, which won Oscars for the visual effects in Steven Spielberg’s Indiana Jones films Raiders of the Lost Ark and Temple of Doom, returned to use every trick in the book on the whopping 2,350 VFX shots in the fifth installment of the franchise.
In the opening action sequence of Indiana Jones and the Dial of Destiny, a young Harrison Ford appears in a 1944-set flashback highlighted by an action scene atop of moving train. Then we meet the elder Indy in 1969 for his next adventure, which includes a tuk-tuk chase in Morocco and — using the titular dial to time travel — a climax set during the epic siege of Syracuse.
The Hollywood Reporter spoke with production VFX supervisor Andrew Whitehurst and ILM VFX supervisor Robert Weaver about their work on the movie — the first in the franchise to be helmed by James Mangold (Logan, Ford v Ferrari). “The conversation never stops,” Whitehurst says of working with Mangold. “He is so engaged, and he just wants everything to be as good as it can possibly be.”
It goes without saying, a lot of attention has been placed on the young Harrison Ford, who appears during the movie’s opening scene. How’d you do it?
ANDREW WHITEHURST It’s called ILM Face Swap; it’s using an enormous number of techniques.
ROBERT WEAVER Face swap essentially is replacing the face with another face, whether it’s a younger version or somebody entirely different. In this case, it was the younger version. And as Andrew was saying, we utilized every trick in the book as far as what it would take to get each individual shot to the level that it needed to be. It employed using machine learning; it employed building a full CG asset to highly critical detail. This work doesn’t lend itself well to having a very consistent recipe; it’s completely dynamic to the individual shot. So there were times that we were leaning more on the CG asset, and there were times that we would be getting a bit more out of the machine learning passes.
WHITEHURST The one continuous element throughout all of this is having really great artists with really great eyes making those choices with Robert and me. And we had an enormous amount of reference material from earlier Indy films, which we got scanned, and we could use that and we could frame through that and understand what exactly the likeness was that we were trying to hit. And it’s building it up. We would initially do a low-resolution pass that we could give to the edit. So they were always cutting with an age-appropriate Indiana Jones, even if it was not a final quality, so that they could judge the performance in the cut and understand how that was working. And that meant we then got better notes back.
It’s building it up, and it’s using a huge variety of techniques, but it’s all down to artists looking at stuff, talking to us, really getting their heads around what the character is, looking at the performance that Harrison gave that’s driving all of this underneath, and then working and polishing that to get to the finished result.
WEAVER There were times we would rely on key frame [hand] animation.
WHITEHURST [In other instances,] we can shoot with multiple cameras at a performance, we can extract the 3D performance from that, and then we can use that to be mapped onto a different 3D face but maintaining that same performance.
So that was based on techniques used on The Irishman?
WEAVER That one component, yes, but the technology was better and the artists have learned more [since The Irishman was made]. All of these other components are equally as useful, or in some shots, more useful.
And then you mentioned machine learning. Obviously between the Indiana Jones movies and the Star Wars movies and everything he’s done, there’s a huge body of Harrison’s work on film. How much of that material did you use to teach the computer some of those more minute mannerisms?
WEAVER It was incredibly instrumental for the machine learning process to have that repository of imagery, for sure. But as Andrew also alluded to earlier, the artists are really the key component here. In addition to the machine learning, we also had a program that we call Face Finder that would, frame by frame, find a similar pose from that repository that artists could then reflect back on and see what in actuality it looked like in the past, and then draw comparisons and know in a very instructive way where we needed to do a bit more work to get what we needed.
WHITEHURST That’s one of the key things for me, when I was looking at the work coming in, is that ability to have a frame-by-frame reference to compare and go, “OK, well actually, you know, when he does this smirk around his cheek, there’s this little crease that always comes in. Maybe we just need to enhance that a little bit.” Getting a performance is a very holistic thing. It’s not just a physical, sculptural quality. It’s a lot of everything.
How did machine learning contribute?
WEAVER There are absolutely components to it that we rely on. Definitely. And that’s part of the evolution of us doing more successful digital human performance. [To teach the computer,] we primarily relied on the older Indy films and that target age. But we also captured present day.
WHITEHURST Yes, exactly. We shot a lot of reference material, as well as the actual shots from the film we were making. We went and deliberately set up the cameras and got Harrison to perform as he is now. We learn, the computer learns, but I think it’s also worth saying that even within feeding the footage, you need to be careful and select your footage, because some footage might skew results in a way that actually isn’t helpful for a particular shot. So there’s a lot of kind of skill and judgment in terms of selecting what you are going to use as reference.
WEAVER Exactly, that curation process that Andrew was just talking about was incredibly important. Artists would make determinations as to what is helpful.
For his body in this flashback, what was a new Harrison Ford performance? A body double? What was the combination of these things?
WHITEHURST Whenever possible, it’s Harrison. There were times where, for either shoot timing issues or because it was a more of a stunt type scenario, we had to use a stunt performer. So the body there would be the stunt performer. And there’s a couple of times where it’s a digi body, but it’s mostly newly shot Harrison.
Tell us about filming that train sequence.
WHITEHURST Everything involving actors was shot on a stage. We did go to the Austrian Alps and shot a load of material that we were able to use for some of the backgrounds, and we did a lot of scanning of that kind of that environment. We were then able to do a fully CG environment, because the nature of the narrative for that sequence meant that we had very specific requirements for the environment in different places. An awful lot of it ended up being fully CG-generated because there is no place that was going to give us what we needed.
[When filming the actors onstage,] we used bluescreen [instead of virtual production] mostly just for speed of getting through shots, and also we needed the flexibility in post to be able to change the background because we were still slightly adjusting exactly what the backgrounds were going to look like when we were shooting. We knew what the overall background would be, but in terms of, well, “We need more trees here, or fewer trees here,” or the sky, you know, dawn is, is happening over the whole sequence. And that timing is extremely difficult to do upfront. We start in pitch black, we end up post-dawn on a cloudy day.
How did Mads Mikkelson’s villain survive his fall from the train?
WHITEHURST This was a long conversation, with a lot of iterations. The thing that he gets hit by is a very lightweight hose that they use for putting water into steam trains, which itself can rotate. So he’s being hit by as soft a thing as you can be hit by that’s going to be hanging by a railway line in 1944. But, yes, it should be a little bit of a surprise when he shows up again later.
WEAVER It boils down to him being a very tough bugger. (Chuckles.)
WHITEHURST The one thing I will say is that when Mads does get knocked off the train, he did that. We put some crash mats there, and Mads Mikkelson knows how to take a punch and how to look like he’s been hit by something heavy. We had to make him travel back further because the train was moving at speed, and obviously we weren’t doing that on the stage. But in terms of the physicality of the performance, he did that.
Tell us about creating the tuk-tuk chase.
WHITEHURST We had a whole amazing second unit crew directed by Dan Bradley who went out to Fez [Morocco], which was where we were actually filming. This was a sequence that we had prevised and boarded. Because there’s a lot of dialogue that happens in the scene, we knew there was going to be process work back at Pinewood with our principals. And we needed to know how we were going to stitch that into location-based stunt material, also how we were going to acquire the materials to put in the backgrounds of the process work. So the previs and boarding was essential. That whole process was based on visits to the location and understanding that environment and what could be done there.
They just went and filmed some really amazing stunts, and we were able to shoot multicamera backgrounds, plus scanning everywhere. And so we were then able to build either fully digital versions of backgrounds or projected photography onto digital assets to create the backgrounds for the process work that we shot back at Pinewood.
There were a couple of stunts that we did at Pinewood just because there was no way of being able to get them on location for various reasons. Usually safety, or just the streets — they are super cramped — so you are limited into where you can put a camera, what kind of camera you can put in there.
Would you describe how you did the underwater dive sequence?
WHITEHURST We used every technique, including actually filming things in the Mediterranean. We shot in tanks at Pinewood. We did some dry-for-wet work, particularly close-ups, because it’s so performance critical. And then there’s a lot of full CG work and CG enhancement work on top of that. We were able to storyboard and previs the whole sequence upfront, which meant we could take a look at each shot and go, “Well, this shot should be full CG. This shot we think we can get on location, this shot we can get in a tank.” And we were able to divide and conquer using that approach.
There are a couple of fully CG shots in there, particularly wide establishing shots where we just couldn’t get the location to look right or that matched our requirements and the tank at Pinewood is just not big enough for doing that kind of work. The art department was able to build us some sections of a wreck, but because of the safety requirements for working underwater, it means that people have to be able to get in and out very quickly and safely. So there was a lot of enhancement work that we needed to do to really make it feel like an enclosed and threatening environment. And then of course, there’s the eels on top of that.
The close-ups were all dry-for-wet, and we were using stunt performers in the Pinewood tank.
Tell us about re-creating the siege of Syracuse.
WHITEHURST Because we had the scene earlier on of 1969 New York, we sort of set our world up as being a real place. And then we extended that and we needed ancient Syracuse to be the 213-214 B.C. equivalent. It had to feel like a real, lived in city. And that’s a huge amount of work.
WEAVER It was a massive build — building ancient Syracuse and being accurate to the time period. You could have taken the [virtual] camera at any point and flown down into the city and through the streets. All the props sitting on wagons would all be appropriate for 200 B.C. And that of course involved building the various boats and the ocean and the buildings that would have to get destroyed at various points in time and building all the tools that Archimedes employed to fight the Romans back in the day. It was quite an endeavor, but it was very rewarding.
Interview edited for length and clarity.
