ReactVR: Audio Spacialization and Click Events with the Sound Component

Adding sound to a ReactVR app is almost as easy as setting up the app itself. This tutorial is a crash course in audio spacialization and click events with the Sound component.

You can see (and hear!) the deployed app here: http://jarednielsen.com/ReactVR-MicDrop/

If you want to follow along, clone or download this repo: https://github.com/nielsenjared/ReactVR-MicDrop

Getting Started with ReactVR

Install react-vr-cli:

npm install -g react-vr-cli

Create a new ReactVR app: react-vr init ReactVR-MicDrop

Uppercase is the naming convention as react-vr-cli will generate a class component using the name you provide.

Change into the newly created directory and start the app:

cd ReactVR-MicDrop
npm start

Open http://localhost:8081/vr/

Well, hello there…

Equirectangular Images

ReactVR uses equirectangular images to create 3D scenes. Flickr is a great place to search for them. Find one, download it, and move it to the static_assets directory of your VR app.

On line 14 of index.vr.js, replace chess-world.jpg with your equirectangular image.

Save and refresh your app.

Importing Models

In a nutshell, a digital 3D model is a file outlining coordinates (vertices) in space. You can create and download models using Google Blocks. Blendswap is an excellent repository of models created using Blender.

Find and download a model and move the .obj and .mtl files to static_assets.

Again, if you want to build the demo, clone or download this repo and use the included models.

In index.vr.js, add Model to your react-vr import statement.

import {
AppRegistry,
asset,
Pano,
Text,
View,
Model,
} from 'react-vr';

Under your Pano component, add a new Model component:

<Model
source={{
obj: asset('speaker.obj'),
mtl: asset('speaker.mtl')
}}
style={{
color: "#666",
transform: [
{translate: [-4, 0, 0]},
{scale: 0.25}
]
}}
/>

Adding Light

From react-vr, import AmbientLight and PointLight:

import {
AppRegistry,
asset,
Pano,
Text,
View,
Model,
AmbientLight,
PointLight
} from 'react-vr';

Under the Pano component, add AmbientLight and PointLight components:

<AmbientLight intensity={0.5} />
<PointLight
style={{
color: 'white',
transform: [
{translate: [0, 0, 0]}
]
}}
/>

To the Model component add a lit property:

<Model
lit
source={{
obj: asset('speaker.obj'),
mtl: asset('speaker.mtl')
}}
style={{
color: "#666",
transform: [
{translate: [-4, 0, 0]},
{scale: 0.25}
]
}}
/>

Reload your scene to verify that your model imported and is lit.

Adding Sound

To add sound to a VR scene we need to import the Sound component from react-vr.

import {
AppRegistry,
asset,
Pano,
Text,
View,
Model,
AmbientLight,
PointLight,
Sound
} from 'react-vr';

Because we are working with 3D sound, we need to put Sound components inside parent components so that the sound has a point of origin, like so:

<Model
lit
source={{
obj: asset('speaker.obj'),
mtl: asset('speaker.mtl')
}}
style={{
color: "#666",
transform: [
{translate: [-4, 0, 0]},
{scale: 0.25}
]
}}
>
<Sound
source={{ wav: asset('drums.wav') }}
loop={true}
/>
</Model>

The drums loop should now be emanating from the speaker. The channel balance will change as you move your point of view in the scene.

Animating Models

Import from react-vr the Animated component and from react-native the Easing component:

import {
AppRegistry,
asset,
Pano,
Text,
View,
Model,
AmbientLight,
PointLight,
Sound,
Animated
} from 'react-vr';
import {
Easing
} from 'react-native';

Declare a component to animate using the Animated component: const AnimatedModel = Animated.createAnimatedComponent(Model);

In order to animate a model, we need to use state. At the top of the ReactVR_Blender component, add:

state = {
rotation: new Animated.Value(0)
}
componentDidMount() {
this.rotate();
}

Create a rotate method:

rotate = () => {
this.state.rotation.setValue(0);
Animated.timing(
this.state.rotation,
{
toValue: 360,
duration: 10000,
easing: Easing.linear,
}
).start(this.rotate);
}

Create a new AnimatedModel component:

<AnimatedModel
lit
source={{
obj: asset('mic.obj')
}}
style={{
color: "#666",
transform: [
{translate: [0, 0, -4]},
{rotateX: this.state.rotation},
{rotateY: this.state.rotation},
{rotateZ: this.state.rotation}
]
}}
/>

Buttons

There’s one more component we need to import, VrButton.

import {
AppRegistry,
asset,
Pano,
Text,
View,
Model,
Animated,
AmbientLight,
PointLight,
Sound,
VrButton
} from 'react-vr';

We wrap VrButton around any component we want to make ‘clickable’. Because triggering sound effects on interaction is very common, we have a built in method onClickSound that we can pass to our VrButton, like so:

<VrButton
onClickSound={{ wav: asset('mic.wav') }}
>
<AnimatedModel
lit
source={{
obj: asset('mic.obj'),
mtl: asset('mic.mtl')
}}
style={{
color: "#666",
transform: [
{translate: [0, 0, -4]},
{scale: 0.5},
{rotateX: this.state.rotation},
{rotateY: this.state.rotation},
{rotateZ: this.state.rotation}
]
}}
/>
</VrButton>

Mic drop.

Sources and Resources

A little about myself.