Mel Slater is Professor of Virtual Environments at the Department of Computer Science of the University College London as well as ICREA Research Professor at the Universitat Politècnica de Catalunya in Barcelona, Spain. His major research interest is the question of what makes virtual reality work, how is it possible to build virtual environments such that people respond realistically to events within them? What scientific explanations are there of this phenomenon?
What is Presence?
It is difficult to give a definition of Presence. I can tell you more about our approach and then I guess there is an implicit definition in that. When you put on a good Head Mounted Display with a wide field of view and good resolution, or when you step in something like a Cave, it is a qualitatively different experience from watching TV, or a movie, or talking on a mobile phone. The sensory data that is coming to you, or a lot of it, is computer generated, so your visual and auditory systems are being bombarded by sensory data generated from a computer and organized in such a way that it transports your idea of where you are. For instance, you might know that in reality you are standing in a lab in a computer science building, or wherever you were, but your senses are telling you something different, maybe that you are in a forest.
So Presence consists in cheating the brain?
Well, it is not really like cheating the brain. The brain is operating out all the sensory data it gets, and the only way it can make sense of that sensory data is transporting your sense of location. That has consequences: for example, if my sensory data tells me that I am in a forest, and then I start hearing growls, my heart is going to start racing because possibly there is some danger out there in the forest...
Even though you know you are not really in a forest...
Even though you realise it is not real, at some basic level the brain does not know that. It just operates out of the sensory data it gets and it makes you respond accordingly. So you hear the growl and your heart starts racing, you start sweating, you start looking around... And, at the same time, another part of your mind is saying: 'Hold on! I know this is not real!' But nevertheless, you started to have responses as if it were real. So, it is this sense of displacement and responding to events you receive virtually that we think of as Presence. And, as I said before, it is a qualitatively different experience to have that, when it is something surrounding you in stereo and in high resolution, than watching it on TV.
It has a lot to do with consciousness.
It is consciousness, but constrained to the virtual environment that is being displayed to you. It is your consciousness in the domain of the virtual environment. But what is different from physical reality is that you always have this split knowledge: one part is telling you that you are in this different place and another part of you knows this is not real, which might have some kind of dampening effect on your interactions.
Sometimes, when reading a good novel, we can also have a powerful experience of displacement and, eventually, get involved in the characters’ feelings. Is not that also Presence?
It is related. But if you are on a good immersive virtual environment and some danger happens, then your whole body is reacting, not just your heart rate, but you have a physical impulse to move, to get out of there. Imagine you are in a virtual environment and a fire breaks out. You might want to get out of there. But if you are reading a book, or playing a computer game, you might start having some reactions that are appropriate to that situation, but you are not going to run out of your bedroom. The difference between what I call Strong Presence and Weak Presence -where 'Strong' and 'Weak' don't mean good and bad, they are just labels, we could call them Presence X and Presence Y- is that in Strong Presence your whole body is involved in it. So you are there in the virtual environment, it surrounds you, it's personal; if somebody talks to you, you respond as if somebody was really talking. While Strong Presence involves the engagement of the whole body...
...Weak Presence does not.
Imagine you are playing a computer game. If your avatar is in danger you personally will not try to get out of the way with your real body, but you will try to manipulate your avatar out of the way. So this is Presence one stage removed; second level, if you like.
Imagination can also make you act like that, specially when you are a child. Is Presence somehow substituting the imagination with powerful stimulus?
Yes, there is a whole interesting area of research where the brain itself produces a virtual reality. People who have schizophrenia are essentially acting out in a virtual reality which is being produced in their own head, and this kind of research is very relevant for Presence. But we have to define a field that we know specifically what we are talking about, that we are interested in. It is not the general field of how the brain makes a sense reality but how this is related to the reality generated by a computer. If we don't have this domain, our study of Presence is the study of everything.
There is a need for focusing.
Yes. What is it that makes this head mounted display produce a sense of presence and this other one does not? What is the difference? Is it the field of view? The resolution? As a computer scientist, I want to know these very practical problems. Neuroscientists do also help because they explain the way the brain works. And then, if you, from an engineering point of view, present the data in this way it is more likely to succeed than if you present it in that other way. Therefore, it is a combination of neuroscience, psychology, perception research and, at the end of the day, engineering. Presence is the place where all these various areas meet together.
The results of your virtual version of the Milgram experiment suggest that the brain takes virtual reality as if it was real. Does this mean that we will have to "upgrade" our brain or that we rather have to change the definition of reality?
It is very difficult to get into definitions of reality. I think it is very simple: the brain gets sensory data and it has to interpret it. The Milgram experiment is a good example because if you look at the videos of the original experiment, you will see that the participants got very stressed. They started arguing with the experimenter, sweating, having nervous laughter and many different characteristics of anxiety.
When we ran the virtual version of the experiment, we also got all those things but everything was of lesser intensity. In any case this is why we are allowed to do it in virtual reality and not to do it in reality. It is because all the time you know it is not real. So, you are getting all these messages: “stop this”, “this is really unpleasant”, “I really don't like doing this”, but they still do it. Many people said they had to keep reminding themselves that this was not real and therefore they could carry on.
So, it is not really redefining reality. There is this split level. On the one side the perceptual system is taking in this data and things are real, because the sensory data is there: the virtual woman is screaming “let me out of here”, there are responses to your own actions... But, on the other hand, there is this cognitive knowledge: “that is not real” and therefore your response gets damped down a bit.
So, Presence research will not change the definition of what is real.
I don't think so. You can create entire realities in your head by taking a drug.
And these realities would be also real?
Yes. I am not going to try to get into what is real, but we operate out of the combination of how our own perceptual system works based on our knowledge, based on our history and based on our experience, and everyone is different. This combination of bottom up (what comes from the sensory data from the external world) and top down (which comes from your own internal processing) provides you with your personal reality. And if you take drugs, or if you are schizophrenic, almost everything is coming from the top down and nothing much from the external reality.
Another kind of reality, but reality at all...
The thing that counts is how you act on it. It is very difficult to work scientifically out of the definition of what is real because you can't get inside people heads and know what their experiences are. So the approach we operate from is looking at the consequences, and the consequences are “if this is real you should behave as if it is real”. Therefore we look at what we call the operational definition of Presence which is “you respond at many different levels as if it is real”.
What is the difference between Virtual Reality and Presence?
I never thought it in those terms. Virtual Reality is used in many different ways. One way just uses the description of the technology. Another way is used as the environment that the technology delivers. Presence is the human response to that.
What are the technology bottlenecks in Presence today?
There are many. The very biggest one is haptics, the sense of touch and force feedback. Although there are some devices that allow you to do this kind of thing - instruments to lift up objects and feel their weight and devices to let you feel some degree of touch but- there is nothing I know with you can walk around in a virtual environment and, if your real knee happens to hit a virtual wall, then you are going to feel it. It is perfectly feasible to have a visual system that, wherever you turn your head the data you get is virtual and it is perfectly possible to hear sounds coming from any direction which are entirely virtually generated. But the haptic sense, the sense of touch, this does not exist in a generalized sense.
How important is this limitation?
Very important. Research has shown something that is just in a sense obvious. For example, something that always happens to me: I am in a virtual environment looking at some experiment and I am holding the glasses you wear in the cave, and there is some virtual table. When I finish, I take off the glasses, and every time I try to put them on that virtual table! This immediately breaks the illusion.
Any other illusion breakers?
The second bigger bottleneck is to have very light weight, high field of view, high resolution display systems. A cave is fine but it costs half a million euros and it takes a lot of space. And the very wide field of view head mounted displays are too expensive, or have low resolution, or they just don't work. So, this area of displays still needs a lot of developing. However, these technologies are improving all the time, so I think we will experience a relatively fast change in this.
Is there any experiment you would you like to do but haven't been able to?
Not that I haven't been able to, is just they take a really long time. This is really worth knowing by people who talk a lot about Presence but never really do experiments. They probably don't realize that making an experiment is somewhat like making a movie. The one we are doing know it's taken a whole year to get to the point now, where we are just about ready to start running it. It can be very frustrating, because you have these ideas, but the latency in having the idea and having it realized through an experiment can be two years.
Wind tunnels are used in engineering to observe the behaviour of scale parts without needing to produce the real ones. Can Virtual Reality be used in the same way to predict the behaviour of people in particular situations?
Yes, I think that is a good analogy. Imagine you have some social situation you wanted to test and it is impossible to produce this situation in reality without causing a lot of trouble. One could use virtual reality as a simulator, just like the wind tunnel.
Can it, for example, help people who are afraid to speak in public?
I think this is fantastic application. We started working on it in 1999 and we did quite a lot of experiments with the psychologists working with this. Our programs are also being used for the study of paranoia, because we found that people started giving to virtual characters motives, intentions and thoughts that were entirely in their own heads. A psychiatry professor suggested that maybe people who have a tendency to paranoia would also be paranoid in the virtual reality, so we tried this out and it does work. There is a big area called Virtual Reality Therapy, which I think is one of the most interesting applications and where Presence is absolutely vital: if the person doesn't respond in the virtual situation something like he would respond in reality, then it cannot really be used for therapy.
Can you name other applications where Presence could help?
For example, simulating medical operations just like pilots do simulated flying, an area where haptics is very important. In design, rather than build your car with clay models, build your car with virtual reality, not only to look at the visual aspects of it, but also to test the ergonomics. It has many applications. Anything, really, that has to do with simulation.
However, it sounds like we are quite far from all of this. Are we really that far?
No, I don't think so. Most of my application work is in the area of psychology and social psychology but I know others who do a lot of work in the industry. For the car industry, for example, it is completely normal to make and experience their models in the virtual reality rather than building physical models. There is a huge amount of work out there going on with the use of virtual reality, which is common place these days.
Which are in your opinion the ethical issues associated to Presence, if any?
Any technology can be misused by people with evil intentions. As well as using virtual reality to train someone to be a therapist or to help people to overcome their fears you can use it to induce fears. You could give people a fear of something by the right manipulations within virtual reality. So, leaving that aside, because that is obviously a whole issue with virtual reality as with any technology at all, basically when one has research in virtual reality, one has to adhere to all the standard ethical guidelines. It is good to have these ethical guidelines because you always need external people looking at what you propose to do from that point of view.
What are the three fundamental questions in Presence?
One of them is “Why does it happen?” Let me give an example. If you are going to a virtual reality which not only you know cognitively is not real, but also if you just look around you see is not real -like there is no shadows, the illumination is wrong, the virtual characters look like cartoons...- you still get this strong sense of having your sense of place transported in a way you take events so seriously that you respond to them. Why? The content of what is in the virtual reality seems to be less important than the structure. By the structure I mean things like the field of view, the resolution, if is it stereo, if it is head tracking... All of these things seem to be paramount while the content, the visual appearance of things, seem to be less important. Take again the Milgram experiment; no one could look at the virtual character and say “well, this is a real person”, but everybody still responds as if it was real.
This is the issue we call “minimal cues”. There is some certain minimum level after which the brain starts to interpret data as if it was real. What is that minimum level? Why does it work? And a very interesting question that is emerging: we are now able to generate extremely realistic environments and characters. What difference is that it going to make? Is it going to push it into a new domain?
So, the first fundamental question would be a “Why?”
Yes, “Why does it work even though virtual reality is so poor compared to reality?”
And then, the second one...
“How is it going to work even when we make it really good?” These first two questions are practical ones. The third one is the scientific question: “How does Presence work in terms of the brain”. “What is the neuroscientist answer to what is Presence work at all?” This is something we are trying to explore in the Presenccia project.
What brought you to Presence research? How did you get involved in this?
There is a very particular moment which explains that. In 1989, my area of research was 3D graphics and interaction. I attended this conference called Siggraph and I was invited to experience virtual reality by the one company that was doing it at that time. So I went to the stand and I put on the head mounted display.
What happened?
At the beginning I was quite disappointed because I opened my eyes and all I saw were very big pixels of different colours. But then they said 'Move your head!'. And, as moved it, suddenly I was in another place! I was in a room and there was some music playing. I moved myself to look out the window, and down below was the sea and there was a boat, and I could tell the music was coming from the boat. So, I floated down to the boat and went into it looking for the source of the music.
And then?
Well, then I heard “Your time is up!” So they took off the head mounted display and I suddenly was back in the Siggraph convention!
Game over...
This was such a profound experience for me that I just changed the direction of my research. Of course, still using the 3D graphics I knew and my knowledge about human-computer interaction but now moving towards “why did this happen?” “How can we use this?” “Why does this work?” Within the first months of that experience we ran our very first Presence experiment in London.
Is still that “Why did this happen?” the main question you are trying to answer with your work?
Exactly. And “how can we make it better?”
Has your research in Presence changed the way you see the world?
No, I don't think so. I have always had a pretty crazy view of reality. It is the other way around: it is not that Presence changed my view of reality, it that my view of reality made me very interested in Presence. I have been always interested in alternative realities like dreams and out the body experiences and this kind of thing. And then suddenly this was brought into my own domain of scientific researcher.
Do you consider dreams also reality?
Yes, a kind of reality. You have a different sort of consciousness in dreams because you don't have your critical facilities in the sense you don't have a past. One of the things that is actually very interesting to contrast is that, when you are in a dream you don't ask yourself “how did I get here?”. If you dream you are in a beach you are just there and things start unfolding, you just kind of have these automatic responses. When you are in a virtual reality, one of the things that make you know it is virtual is that you always remember how you got there, the process by you got there. And therefore you know it is not real. If one morning you woke up and some secret organization had stuck you in a very very very high powered virtual reality that you didn't know that you are in, this would be your reality. You would not have that memory of having gone into it, you just would be there. Maybe this is how our life is actually. Maybe we are on somebody else's virtual reality...
