Into the Frames: The Polar Express

“The Polar Express” is an American computer-animated Christmas musical directed by Robert Zemeckis based off the 1985 children’s book with the same name by Chris Van Allsburg. The movie is set on a Christmas Eve in the 1950s, following the adventures of a young boy who is invited by a conductor of a mysterious train that stopped in front of the young boy’s window. Together with other children, they embark on a journey in the train bound for the North Pole to visit Santa Claus.

The film was released on November 10, 2004, which was the second animated film to open in theaters within November of 2004 after “The Incredibles” that was released on November 5, 2004. The film was not able to compare to “The Incredibles,” only having total gross earnings of $314.1 million compared to “The Incredibles.” Note that “The Incredibles” is the fourth highest-grossing film of 2004 because of its total gross earnings of $631.6 million. However, besides “The Polar Express” having the achievement of the first film completely in motion capture, the film also had the achievement of having the highest IMAX gross of any film – a total gross of $71 million – until 2009.

Now that the story and success of “The Polar Express” has been covered, it’s time to delve into the development of the film through the eyes of the film’s animation supervisor, David Schaub. Schaub has written a four-part diary to VFXWorld to document his account of the film’s production from his perspective. The first part of the diary is about how their team tested different animation styles to encapsulate the book’s painterly illustrations and retain Tom Hanks’s performance without reinterpretation. Their director, Bob Zemeckis, worked with Ken Ralston, five-time Oscar winner and senior visual effects supervisor at Imageworks. They worked together since Zemeckis directed “Who Framed Roger Rabbit” (1998). Ralston worked with VFX supervisor, Sheena Duggal, to create their early explorations for applying the 2D approach. They first used a “combination of image processing tools and customized brushes that allowed an entire image to be treated as a pastel painting.” Another strategy would be utilizing “optical flow software to create motion maps for each pixel in a series of live-action images.” This would be used as a base to create time-coherent painterly effects over live-action sources that artists would paint directly onto selected single frames. Then the paintings would be used as keyframes for the optical flow software to apply motion onto said paintings over time. For the area of the images that did not work with optical flow, they were manipulated in the composite through a combination of 2D painting, tracking, and warping.

The second approach is taking a 3D approach using motion capture (mo-cap) for recording human performances, as suggested by CG supervisor Alberto Menache. The 3D approach could achieve the desired painterly aesthetic with great control and flexibility, but the approach was met with skepticism. Menache is experienced with mo-cap as he wrote a book on the subject and developed a mo-cap pipeline for crowd systems on “Spider-Man” at Imageworks. Ralston agreed to a test of the two methods for eight shots, with Tom Hanks playing as both the child and the conductor for the mo-cap method. Menache focused on software development, Schaub supervised the animation work, and Scott Stokdyk was the VFX supervisor for the test.

On July 10, 2002, the stage was set for the test. Ten days later, Schaube provided insight into the progress of their mo-cap variant for the scenes. The MoCap vendor tracks the body of the actor for motion capture, then handled by their integration supervisor, Albert Hastings. Hastings used Kaydaras FilmBox to apply the data to their skeletons to retarget adult Hanks to the physical proportions of an eight-year-old child for the shots that called for it. All the shots are then prepared for animation by upgrading the FilmBox rigs to the animation control rigs in Maya. The animation control rig had two control skeletons: one that has the motion capture data, and one that carried the animation. Animators also had blend control to combine the two skeletons to the proportions they please, such as 100% of one skeleton over the other. An animator could blend out of mo-cap and keyframe a section of the performance, and vice versa. Schaub thought this animation rig would resolve most problems that he could have anticipated with three animators to support the test: David Earl Smith, Keith Smith and Alex Tysowski.

At the start, with their current motion capture technology, the data they received was “very noisy and needs additional filtering.” However, adding a noise filter removes the subtle nuances in the acting that would need to be animated back into each of the affected shots. One instance was that the MoCap solver did not capture enough movement in the upper torso, thus the spine needed to be animated to create a flexible organic quality. Another issue was that the characters also did not align with the physical environment and props as intended. Days later, Menache adds a new version of PFS (Performance Facial System) for a muscle-driven face rig to develop higher-level controls that will combine and save poses from the actor’s facial muscles. These poses can be created by either animation leads, or from the actor’s performance and modified by animators to match the video reference.

With the new PFS, they were able to get more organic movement recorded from the actor’s cheeks to apply to the child character’s proportions. But there is missing data around the lips because it was not feasible to place markers inside the actor’s lips. The workaround for them was to animate all the dialog shapes by hand using the muscle controls in PFS. Furthermore, the markers on Hanks’s eyelids are hindered by the fat pad above the lids when his eyes are open, resulting in noisy and unpredictable data around the eyes. The resolution for this issue is turning off the mo-cap between the brow and the cheekbone, and instead animate the muscles in that region by hand. Schaube knew that this was only a temporary solution for the test, as trying to animate every single part that couldn’t be captured with mo-cap would be infeasible for an entire film. So, he envisioned an ideal toolset that would be of better use of the mo-cap data.

By September 19, 2002, Schaube’s team succeeded in capturing Hanks’s performances with such detail that Zemeckis abandons the 2D tests for complete dedication to the mo-cap approach. Zemeckis and his team revised the character’s model to fit the character’s age in the book, thus required the animation and mo-cap data to be retargeted to fit the new model. The crew adjusted the model with a new muscle-layout for the new model’s face, and applied facial animation, when necessary. By October 8, 2002, the crew successfully created the scenes for the test that satisfied the Warner Bros executives. From what took them three months to produce eight shots, they were now greenlit for an entire film within 18 months.

This was but a start to Schaube’s point-of-view in their production to create the first completely mo-cap film, one that stood the test of time as the top grossing film in IMAX theaters for 5 years! A highly innovative procedure that brought to life the painterly art style of its paperback predecessor while keeping the expressions of the actors that performed under the mo-cap spotlight. Be the next pioneer in the landscape that is the animation industry! Learn more in depth about the animation process through Schaube’s eyes in his four-part diary for VFXWorld.

Sources:

https://en.wikipedia.org/wiki/The_Polar_Express_(film)

https://www.awn.com/vfxworld/polar-express-diary-part-1-testing-and-prepping

https://www.movieinsider.com/movies/november/2004