Autor: Lars Drawert

Volumetric shooting

Volumetry and how it extends the range of narrative forms.

The interview was conducted by Lars Drawert and Torsten Schimmer (volucap).

LD: What is volumetry? What is volumetric shooting?

TS: Volumetry is an extension of the 360° videos. Not only do I look at the surroundings from a point of view determined by a camera, I can also move around in virtual space. This enables an environment to be experienced even more intensively.

We have any number of cameras, with us there are 32 that capture an object from different angles. Intelligent software behind it now digitally reconstructs the filmed object based on points in space. The individual spatial points initially result in a kind of "point cloud". These are then connected to each other in such a way that first polygons and then a so-called “mesh” is created. This “mesh” then has a certain shape. The recordings of the individual cameras are then digitally projected or placed on this “mesh”. You then created a 3D object.

So in principle, volumetric shooting is that you record an object with a certain number of cameras and then create a 3D object in a two-phase process, onto which you project the recorded video images. This creates the 3D character.

The difference to "classic" motion capture is that we actually capture an object and put it on a 3D model. In "classic" motion capture, a figure is captured in a suit, with tracking points on it. A digital skeleton is first calculated from this. This is then animated. The animated movements in particular, but also reflections and surfaces, appear more artificial and computer-generated.

With volumetry, a digital skeleton is only required in individual cases because real images are placed on the surfaces there. In the example with the basketball player, the figure would have looked straight ahead. We programmed a skeleton there so that the figure keeps eye contact with the viewer. The movements of the cell phone, i.e. the viewer, are tracked and coupled with the movements of the digital figure. This naturally leads to an intense, immersive experience. In addition, the figure is also placed in a 360° environment or a virtual scenery.
With the volumetry, even if it is still not technically perfect, the figures appear realistic - movements, facial expressions, gestures, etc. are transferred to the app in a photo-realistic manner.

LD: How long has the idea of volumetry been around and when did it become applicable?

TS: I recently heard that the first volumetric prototype existed in a US university about 20 years ago. I can't really confirm whether this is actually the case, but the basic idea of taking pictures of an object from different perspectives has been around for a while. This is called photogrammetry.

And this has certainly been made possible by the enormous computing power that has now become available and affordable.

LD: Which application examples are there?

TS: We implemented one of our first volumetric projects together with Telekom and Anotherworld. In the "Meeting Josh Mayo" app it is possible to learn cool dribbles and precise throws from the former basketball star of Telekom Baskets Bonn. We can move freely on the scanned basketball court while Josh shows us a few tricks.

Another application example are companies or manufacturers who virtually lead through one of their fabrics. The company founder, played by an actor and digitally reproduced, can greet the visitors and take a virtual tour.

There are holo-lenses in which (similar to AR- [Augmented Realty] applications) additional information and graphics can be shown. Waymarks through the factory, for example. Machines can also be explained virtually in this way. And that is not what a student does, but a digital image of the company founder or chief engineer.

Our customers also include many advertising agencies or the fashion industry with virtual fashion shows.

Such things can of course also be used very well for museums. You put on your glasses and get up close to the scenarios or interviews. This is exactly what volumetry does. For example, a scene could be in a historical exhibition where I sit in a barrack with someone who was in a concentration camp and the person tells me about the time there. And while you're being told that, I can also get up, look around and move. You are not limited to the one point of view, the one perspective.

Or the e-learning area. For example, instruction on an expensive or difficult-to-transport device can be realized much more cost-effectively via volumetry, because you no longer need to send the machine and top engineer around the world, but only the holo glasses with the studio recordings. This will not replace learning from person to person, but it can help to simplify processes or reduce the effort.

So when it comes to volumetry, I'm talking about a “new camera”. We have already indicated the various areas of application. Wherever there are use cases for classic 2D or 3D recordings and where an intensive immersive experience is desired, I also see use cases for volumetry. I can make a documentary, an advertisement, a film, I can convey teaching content, support in the treatment of trauma or design digital tours in museums.

Another application is second screen, so you can bring an AR Spiderman into your living room, for example. It is also possible to record actors in the volumetric studio and then use them in the corresponding scene, which may have to be digitally reproduced anyway. If you even use this for a pre-visualization of a complex and complex scene, you can save a lot of time and effort because the camera operators, lighting technicians or scene designers can define technical and design parameters in advance.

Or background movement of extras. The effort to conduct maybe 300 people in the background is no longer necessary.

LD: What technical parameters do you have?

TS: We have 32 cameras in the studio, each with 5K resolution, 25 or 30 frames per second and in RAW format. As for the amount of data: about 1 TB of raw data is generated per minute, which accumulates on our servers. After about 90 minutes, our hard drives are full and need to be replaced. But so much material has never come together in one day.

LD: How is the workflow?

TS: First, the recordings are planned and discussed. The servers always run when turning. We are, so to speak, live. We mark the best take and transfer it to the software. Precisely because the amount of data is so enormous, we have to document very precisely which recordings, which takes have been taken and should be used. We then transcode the preselected takes as 2D proxies, which can be played as MP4 on any normal computer, and hand them over to the director and producer. This gives you the opportunity to check the recordings again. Once the decision has been made, we generate the 3D model and deliver about 4 weeks later.

LD: How does the lighting design work with this type of production?

TS: You can create the lighting situation in the studio as it will later be found in the planned scenery. But that is not absolutely necessary and in fact it is rather the case that the surrounding scene is not yet 100% certain when the shoot is in the volumetric studio. For this reason, the lighting is initially shadow-free and the lighting mood including light color, highlights and shadows is added in post-production.

For example, the concrete lighting situation can be recorded with a 360 ° camera, let's say a street scene at night in the city with the typical street lamps. These recordings are then the template for the generation of the lighting effects in the volumetric recording. Even if the scene lighting in the studio is not 100% realistic, for example it is softer and more like a kind of ambient light, textures and reflections are generated and recorded better and more appropriately in the studio.

LD: How do you record the sound when the perspective can be changed freely in the end?

TS: We can use any type of microphone, currently our customers prefer to record the sound with a clip-on microphone. The software later generates changes in sound according to the chosen perspective. In a further step, additional reverb effects can be taken into account according to the selected viewpoint in the scenery. Objects in a room can be textured and thus change the character of the tone.

Related Articles: