Seeing is Believing

“Everything we focus on is story,” Dr. Corey Carbonara told the group. Carbonara is a Master Teacher, FDM professor and director of the digital communication technologies project. He and Dr. Michael Korpi, a fellow Baylor FDM professor, have consulted with Sony on various technologies and initiatives over the past 25 years. Both are members of the Academy of Digital Television Pioneers and also serve as Senior Research Fellows for IC2 Institute at the University of Texas. 

Demonstrations at the December 2016 meeting included Oculus Rift and Sony VR—virtual reality headsets—along with 360-degree videos of Baylor events and the augmented reality headset Microsoft HoloLens.

 

 

Virtual Reality

Virtual reality (VR) is the technology getting the most buzz in the tech industry. The user is able look around freely and explore virtual spaces rendered in three dimensions (3-D). Currently, the high-end VR technology is more of a niche hobby item for video game enthusiasts. On the Sony VR, Resident Evil 7—the first big-budget, full-length video game to use virtual reality—hit the market in mid-January. VR also is making the rounds at many of this year’s film festivals.

“All the VR stuff at Sundance is interesting as a curiosity,” Korpi says. “We’re not quite sure what it is yet.”

It harkens back to the early days of film, when there wasn’t a sense of true storytelling but rather more curiosities rarely longer than a minute or two that people marveled over as technical innovations. Few became engrossing and involved storytelling mediums. That hasn’t stopped the demand. VR was the most sought-after topic at January’s Consumer Electronics Show (CES) in Las Vegas.

“It’s happening on two levels,” Korpi says. “The high-end VR requires a lot of graphics horsepower and rendering, a very fast response time, as low latency as possible and as many pixels as possible. And that requires a really powerful PC.”

Korpi says PC (personal computer) gamers already have that kind of technology or at least something similar. He says it makes sense to ask people who already have most of the hardware to initially adopt the new technology.

“Getting people to buy a $2,000 PC plus the $800 head-mounted display, that’s not a real consumer item,” Korpi says. “On the other end, you have head-mounted display holders for your phone that are like 30 bucks or given away for free at a film festival. Then, the phone is the processor.”

Modern smartphones have more processing power than anything used to render 3-D in the 1980s, Korpi notes.

“Compared to a gaming PC, it’s limited in capability,” he says. “But it makes it try-able. It’s easy to set up and do. At both ends—at the cellphone end and the gaming PC end—the technology is good enough to do convincing environments that you can look around in. Ten years ago, you couldn’t have done that with that level of equipment.”

While the barrier for entry for the high-quality VR is an issue, another problem is VR sickness. Baylor FDM has conducted cybersickness studies, comparing the effects of watching a movie in a theater versus watching a movie in a VR theater. Roughly 40 percent of students playing a VR game got sick within 20 minutes and could not continue. Comparatively, 90 percent of the students made it through an entire VR movie.

“Netflix has this in mind,” Korpi says. “The VR headset gives you a better living room with a bigger screen, and you just sit back in your chair and watch the thing.”

The software FDM has been using for this virtual theater experience is Riftmax, which allows people to enter a virtual theater where there is a two-dimensional (2-D) video shown on the screen like there would be at a theater. The user can look around the virtual theater like at a physical theater. Users also can create their own avatars—a virtual representation of themselves—and sit next to each other in the theater.

This can lead to some unusual experiences, such as the one Carbonara witnessed while demonstrating last Halloween to Paramount Pictures how Baylor FDM uses Riftmax.

“When you’re in a Riftmax environment, you actually can produce a public theater event, which means that any user can decide which theater they want to attend,” Carbonara says. “I didn’t realize I had left the demonstration event live and open to the public.”

During the demonstration, Carbonara had the Paramount executives create their own avatars and pick a seat. Suddenly, an avatar that was not a part of the Paramount group joined them in the theater.

“In walks this guy [avatar] with a hatchet in the middle of his head,” Carbonara says. “He sits right next to one of the executives and says, ‘Hey, what are you watching? What, this is bogus!’ And the guy leaves.”

The Paramount executive thought the guest interaction was something staged and asked how Carbonara got the timing so perfect. Carbonara confessed, “I left it public by accident, and that was some random guy seeing what was playing in theater six.” 

This interaction also demonstrates that VR can be a place where people can get together. Carbonara says it is a very social event.

“What’s fascinating about it, even the few times we were setting this up for the study, is that if you leave an event open, people then talk—it’s like the theater before any of the previews show—and you get to talk to whomever you’re going to the theater with,” he says. The added perk is that a user can mute a virtual moviegoer who ends up being too disruptive.

Logan Trent and Marcos Luna are two FDM graduate students working with Baylor’s theater department and VirtuOSO, a university a cappella group, to develop a VR music video. The issue is that, with VR, there aren’t a lot of rules or guidelines for how to go about making these environmental experiences. As they venture out into uncharted territory, they’ve found workflow to be one of the most daunting challenges.

Trent says VR experiences give the user a feeling that film cannot, and Luna says they are “moving from a temporal medium to a spatial medium.”

“No one is working on the level we are,” Luna says, regarding other schools. “The other people doing this are big companies.”

 

 

360-Degree Video

Most people are more familiar with 360-degree video, a form of VR that takes place in a fixed perspective but allows the user to look freely in all directions. For instance, the hit Broadway musical Hamilton shot a 360-degree video of one of the rehearsals with all the cast gathered around in a circle. Another popular example was YouTube’s 2016 April Fools’ joke SnoopaVision - a 360-degree video of a theater in which viewers could watch videos with Snoop Dogg.

Baylor FDM is working with other programs to develop 360-degree videos on Baylor’s campus.

“We’ve been involved with Undergraduate Admissions to shoot Homecoming, to do 360 production for what it’s like to be at a football game and Pigskin Review,” Carbonara says. “Our students and our alumni have come back because of the attractiveness of Baylor being at the cutting edge [of this technology].”

Sony sees this as one of the big ways they can work alongside educators. Chris Ohven, senior manager of strategic business, says that while the technology is wowing everyone, “the story is the missing key component” and that he looks “to tell the story of the school.” Ohven says that Sony has already made the technology; it’s now a matter of figuring out the applications of that technology and connecting it to telling stories that affect people.

Sony account manager Ray Masterson works with several school and business accounts. He sees the mobile campus tours as the big need for schools.

“The campus visit is the key thing,” Masterson says. He believes that a strong mobile tour makes potential students more likely to visit a campus.

Ohven is excited to see multiple Baylor departments showing interest in using these technologies and developing a singular vision of how to use them.

“Baylor is unique in this initiative,” Ohven says.

Baylor’s work with these technologies is nearing the level of major corporations, and FDM’s focus on story is attracting people. Baylor is also collaborating with Google to develop 360-degree videos using GoPro’s 

Odyssey, an array of 16 cameras that shoot video in 360 degrees. Thanks to Google’s Jump program, FDM will have access to an entire array of software that will process and polish the 360-degree video after it has been captured. Once processed and published, these 360-degree videos can be viewed on a phone via YouTube. All it takes is a viewing case like Google Cardboard or the Samsung Gear.

The problem Korpi sees is processing power and battery. These types of video and VR experiences on phones are limited by how fast the processor can run without overheating the phone or depleting the battery life. Nonetheless, Carbonara sees 360-degree video as a way to expand storytelling, to go beyond narrative.

"From an exploration standpoint, I'm exited about trying things, and we may find we fail at 90 percent of the things we try, but we may discover something and I think the role of the academic institution is to push that envelope." 

“There was always the notion that the camera pointed in a certain direction is gatekeeping you from the other side of the scene. You’re pointing where things are going to be seen,” Carbonara says. “But [in 360,] if it’s news coverage, then you get to see around and behind where that camera vantage point was. From an emotional standpoint, it opens a whole new way of storytelling. There is no way to understand how this new mode of storytelling is going to be imitated the way storytelling has always been imitated for a 2-D screen or even a 3-D screen. From an exploration standpoint, I’m excited about trying things, and we may find we fail at 90 percent of the things we try, but we may discover something and I think the role of the academic institution is to push that envelope.”

Meanwhile, Korpi holds a view that will require much more advanced, complex planning in storytelling.

“I’m not so sure you can tell a story like that because you can’t move the camera around, and you can’t cut from one shot to the other,” he says. “That’s how you build the story [in film/TV].”

Carbonara says that one of the ways to direct viewers in a 360-degree video is spatial audio. He believes people often forget audio’s impact.

“It’s more than half the picture,” Carbonara says. “Being able to provide spatial referencing so that you have cues. By having a voice start 45 degrees behind the back of your right ear. You can dictate and provide the choreography with two things: visually—what’s happening with the physical movement—but you also have this ability to have the audio become a cue to turn you in the direction and to move you into that. Our students are getting a chance to learn about spatial audio.”

One project Korpi and Carbonara are exploring is the idea of a 360-degree video of the Normandy landing. They have talked with Col. John Antal, a military advisor for Gearbox Studios in Dallas who worked on the World War II video game series Brothers in Arms.

Korpi points to the photographic and film footage of the beach landings during the Allied Invasion of Normandy in Operation Overlord during World War II.

“The idea would be to take a 360-degree view of that beach now from all the points of view of all those newsreel camera positions,” he says. “At any point during your viewing of this footage, you can pause and enter the 3-D view and stand where that cameraman is standing. It’s a simple enhancement, but I’d be fascinated to do that.” 

 

 

Forefront of the Industry

Carbonara says virtual reality and augmented reality (AR) are becoming more rapidly understood by mainstream entertainment companies. Baylor FDM is at the forefront of the industry.

“Three years ago, we gave our first presentation ad hoc to the Society of Motion Picture and Television Engineers,” Carbonara says. “That was the first time they had experienced an Oculus. The next year, there was more interest. It was the idea of a couple of industries to shake hands with each other—the video game industry with the motion picture and television industry—and get them to understand a little bit more of what the possibility is. It didn’t take long for it to explode.”

Baylor FDM was invited to be a founding member of a virtual reality industry forum—the only purely academic institution selected.

“We’re honored to be a part of this initiative,” Carbonara says. “It’s reminiscent of what happened to us back in the ’80s when Michael had the foresight to provide a more level playing level field for us to compete with other universities as we were trying to grow the program. We got involved with The Society of Motion Picture and Television Engineers’ ground effort for high definition.”

Korpi saw that high definition was going to be an industry changer and that Baylor could play a role, even without any of the necessary equipment. Baylor and Massachusetts Institute of Technology (MIT) were the only schools involved in the early days of high definition.

"AR is bigger than VR by an order of magnitude because it taps into all the educational trainings so directly, and it's going to be our interface to the world."

“This is as big as HD,” Korpi says. “AR is bigger than VR by an order of magnitude because it taps into all the educational trainings so directly, and it’s going to be our interface to the world. We’re going to expect multiple layers of information to be available in that location, and we quickly call up or delete or hide the ones we want or don’t want.”

AR has a few key advantages over virtual reality. Much less is rendered, and there’s not the sense of motion sickness that often accompanies VR experiences. Instead of rendering an entire virtual world, AR projects virtual objects onto the real world. Korpi says a good example of this is Pokemon Go, which allowed people to use their camera to catch a Pokemon in a bush or under the bed.

“And there may be story applications for [AR],” Korpi says. “I think we see just a hint of that with Pokemon Go. It’s essentially a game. You’re building a story of how you collected these Pokemon.”

The Microsoft HoloLens is an AR headset that attempts to allow for augmented reality by projecting holograms into real-life spaces. The problem with the HoloLens is that the field of view is limited, meaning users must stand at exactly the right spot and look through the glasses at exactly the right angle for it to work. Also, the holograms aren’t nearly as graphically impressive as their VR counterparts and are easy to see as computer generated because they’re transparent.

Occipital’s Bridge, which uses a 3-D sensor called the Structure, is another technology Baylor FDM is exploring. It uses a smartphone to take live video of a person’s surroundings and patch it to the phone. The user enters a room and is greeted by a robot, who looks up at the user. Toss a ball, and the robot fetches it.

“Even with just the little demo, we see it’s pretty impressive,” Korpi says. “AR is going to sneak up on people. It’s going to happen gradually. For most people, their interface to the Internet and information today is primarily their smartphone. Ten years ago, it was their computer. When did the transition happen? Did you notice? I don’t even think I noticed when it tipped. I use mobile devices more now, and I have computers everywhere.”