vrapp: The Next Generation

This is a first draft of a web page I'm planning to use to inform all the Gang people about proposals for the future of vrapp.

vrapp Change log

My lower level to do list for vrapp available locally at Brown:
/u/lsh/vrapp/TODO (a copy is available in the vrapp distribution mentioned in the installation instructions)

Email me at lsh@cs.brown.edu if you have any comments.

Installation instructions

Adam Lake has a snapshot of vrapp in action from the final rehearsal

We at Brown have little experience in trying to duplicate reality in a VR setting (geometry extraction, full body avatars, etc.), so I think that it is advantageous to concentrate on other aspects of VR. We should think about having a virtual environment in which we can do things that we can't do in real life but also have the ability to do things we would expect to be able to do in the real world.

The following are requests and possible research problems for vrapp. The ideas either came from conversations with people here at Brown, requests from users, or offhand comments about the future of vrapp.

Table of Contents in descending order of importance for September demo
The following includes suggestions received so far, ordered by me, subject to change.

1. Pointing
2. Directing Attention
3. Non-verbal communication (visual cues, etc)
4. Annotating the environment
5. Shared whiteboard
6. Sketch interface
7. Object locking
8. Non-Verbal Audio feedback for changes in the environment
9. Cartoon effects

Requests and Possible research problems

• pointing
  One of the biggest requests during the testing period preceding the demo was being able to point at objects in the scene. The participants had no way to call attention to various parts of the model under discussion. A way to call attention to parts of the model at a course level (eyeball, mirrors, optics) should be provided, but it may be necessary to point out parts at a finer level (parts of the optics).

  Since the September demo will have immersive users (Brown and UNC) and desktop users (Utah), pointing should be provided for both types of environments. A few different possible pointing techniques that come to mind follow.

  • Forsberg/Hong interactions techniques (paper)
    The easiest technique to try would be the "laser pointer" technique, which probably would be sufficient for our purposes. Basically, the technique is similiar to the 2D laser pointer technique, but the ray would originate from a 6-DOF device.
  • One glove
    Using one glove would allow more control over the environment than using a 6-DOF device, but it is much more complex to mirror the glove's position and orientation to other users. Forsberg/Hong are already working on this technique, so the benefits of including this technique in vrapp should be balanced against the duplicated effort needed to implement this technique. If this is a goal for vrapp, it is a longer term goal.
  • "talking hands"
    The user would wear two gloves which could be used to manipulate the scene and give feedback about what the user is trying to communicate. A balance would have to be struck between the hands as interface to the world and the hands as communication medium.

    Just mirroring the hands without allowing interaction with the scene includes the problems of mirroring to other users. Using the gloves to interface with the environment adds even more complexity to the problem. This is definitely a longer term goal, and the one glove technique should be implemented first as proof of concept.

  The 2D interaction technique should be tested first and then a simple 3D interaction technique should be implemented. The "talking hands" technique enters the little traveled territory of using two hands, but should be considered a longer term goal.

• Directing attention
  In the real world it is fairly easy to direct people's attention somewhere. This is often done by speaking up and motioning to a certain location. The conventional ways of directing attention are difficult in a virtual enviroment, but they can be replaced with methods that would not be possible in the real world and take advantage of benefits of the virtual environment. Care should be taken not to confuse the user, as discussed in Randy Pausch's SIGGRAPH '95 paper. Two possible ways of directing attention in a virtual environment follow.
  • Camera manipulation
    One possible way of directing attention would be to change other user's viewpoints so that they are slaved to a "master" user's viewpoint. Other users would see the same thing as the master, so the master can call attention to parts of a model and the master will know that the users are seeing the same thing. In order to avoid confusing the user, the viewspoint should not jump to the master's viewpoint but should be "flown" from the old viewpoint to the new one. If all the users are slaved to the master, then there isn't a problem of how overlapping avatars from all the users appear because there are no other users to see the overlap. If not all the users are slaved to the master, then there is the problem of finding a good way to represent the shared viewpoint to other users.
    
  • Object of interest oriented relative to user's view
    Another possible way of directing attention is to have all users see an object displayed with the same orientation relative to their view, so the object would have a different orientation for each viewer relative to the world. The object would face the same way towards each user, which would mean that the environment would not be the same for each user. To remind users that the object is located relative to each users' view instead of relative to the world probably will require a cue to remind users that the object is not normal. There is also the problem of creating an easy interface to this technique.

• Annotating the environment
  In order to make discussion about a model easier it would be helpful to be able to annotate a model. This could be as simple as using circles and arrows to draw attention to certain parts of a model.

• Shared white board functionality
  Someone during the demo mentioned shared white board functionality, which I envision as a conventional shared white board incorporated fully into a 3D environment. Basically this would provide a way to describe ideas within the environment by drawing on whatever surfaces have whiteboard functionality. An interesting problem would be controlling this in an immersive environment.

  But writing on a shared whiteboard surface in 3D is potentially awkward, with problems like hand-eye coordinatiion problems because of inaccuracy of input devices, perspective camera distortion, etc. With these problems it may be easier to use a 2D interface to the whiteboard in the form of a popup window or entirely divorce the whiteboard functionality from the 3D environment by using a conventional 2D shared whiteboard application in conjunction with vrapp.

• Non-verbal Communication
  In the real world, we communicate through non-verbal as well as verbal means. Without a person saying anything, we can tell if they are bored, happy, excited, tired, angry, etc. It would be beneficial to have this mode of communication in vrapp. The talking hands and other pointing techniques will help with this, but we should see how well texture mapped video helps in this realm.

  On the other hand, if audio isn't synchronized with the non-verbal communication it could produce results worse than audio alone. If audio that isn't synchronized with pointing it could easily produce misunderstandings. For example, if you point at one thing and then another and say the first is "bad" and the second one is "good". It is possible that the user on the receiving end will think the second thing is "bad" because of audio delay.

• Object locking
  Right now when a user wants to manipulate an object they have to wait to acquire a lock on that object. The question is whether locking is needed at all, and if so what can be done to make manipulating objects easier in an environment that uses locks. Some ideas that have been brought up in regard to object locking follow.
  • Test without any locking
    The planned use for vrapp is to help with collaborative design. In this environment it makes sense for users to manipulate objects as needed, without locking getting in their way. We should test vrapp without any object locking.
  • Manipulate object immediately
    Preliminary implementation completed, published as Providing A Low Latency User Experience In A High Latency Application
  • Show what object is locked
    One request that was received was to have some kind of feedback about what objects are already selected (locked) by others.

• Sketch interface
  Another person mentioned during the demo a sketch interface. One possible first step would be a sketch Open Inventor node that would operate as a client for a sketch program used elsewhere. This would be similiar to the current implemenation of the Alpha_1 node which allows only one person to make changes to an Alpha_1 model which are mirrored in all the Alpha_1 clients.

• Non-Verbal Audio feedback for changes in the environment
  This would be sound effects to notify the user of changes in the environment, even if the changes weren't in view. For example this could be a "whooshing" noise as an object is moved.

• Cartoon effects
  Preliminary implementation has been completed for motion blur and motion lines, published as Providing A Low Latency User Experience In A High Latency Application. Some of the possible future work follows, some of which would work better in a non-design enviroment.
  • Multi-segment motion blur and motion lines (use past deltas, not just current one)
  • Smoother creation and deletion of objects in the scene (fade in/fade out, enlarge/shrink, explode, etc, etc).
  • Instead of techniques that happen for one frame (current motion blur and lines), implement techniques that occur after motion (take one frame techniques and fade away, or do something like a puff of smoke when an object moves fast)