building virtual environments: process documentation
Building Virtual Environments (BVE)
a class taught by Professor Alan Price
at ACCAD (Advanced Computing Center for the Arts and Design)
at OSU (The Ohio State University), Winter 2011
To start, last quarter (Autumn 2010), I began learning Maya 2011, which is a 3D rendering software. I modeled and animated a silicon dioxide molecule and an hourglass. Thanks to the very fantastic instructor, Vita Berezina-Blackburn, I'm now comfortable mocking-up simple 3D forms. With rendering software, the sky's the limit.
To satisfy the requirements of Professor Price's class, our built virtual environment must:
- Be related to our thesis research interests
- Consider optional input/output methods and how they might be utilized
- Produce a functioning prototype: programming, visual design, & must have a physical component of engagement
___________________________________________________________________________________________
To keep things slightly organized:
Course content and learning will be in regular text
Supporting, but possibly tangential ideas, in blue.
___________________________________________________________________________________________
week 1:
As a class, we were shown some examples of past immersive projects done by students and professionals. The big change from this year and past years is that we are no longer learning and using the software called VirTools. The software that we've been learning is called Unity, a "game development tool" and there is a free version online that is very powerful.
We were shown game examples that use cinematic rendering...I think the point is that games and virtual environments can be beautiful and visually inspiring, especially when we borrow techniques from art, film and other related disciplines.
We were introduced to some alternative and gesture-based user interface - Joysticks / MIDI / Audio / Video / Motion Capture: data glove, Kinect, touchscreen, face recognition, etc.
And some alternative display methods - Immersive and Augmented display systems, for example: large-scale wall projections, touch tabletop, glasses that help one see stereoscopic images, etc.
I was excited at the range of tools with which we were able to experience and experiment. One might be tempted to make an environment so immersive that it takes away reality altogether. I was thinking that it might be good to do try to build an immersive environment that allows one to communicate with others in reality, as well as virtually.
I was immediately mesmerized by the Kinect. Why? Because it's using infrared light to detect people's gesture. I have been a photographer for about 12 years now, and I do remember using infrared film and lens covers and being fascinated by it. I knew that I wanted to experiment using the Kinect, but I wasn't sure how or where to start. I was also curious about the Kinect's abilities to recognize gesture and possible strengths and limitations. How sensitive is it / How sensitive can it get?
But..How immersed do we really want to get?
As I watched this episode of Futurama, (Volume 2: Episode 9), it became clear to me that immersive virtual environments can be fun for the person that's immersed but can potentially isolate one from reality (and people) in a potentially negative way. Is it possible to build virtual environment's that are designed to balance real and fantasy worlds?
Unity: We looked at some examples of Unity-built games.
We looked at the various resources on the Unity Website, read the basics page on Learning the Interface and watched all of the tutorial videos on our own time and during class.
When these examples were shown in class they looked really easy to do. Click that box, drag this over there, press play...voila! It was like watching a genius solve two rubix cubes in five minutes.
___________________________________________________________________________________________
week 2:
Unity: We continued to learn the interface, (one can navigate in x,y,z, in a similar way as in Maya. Unity seemingly borrowed existing conventions and made them even more intuitive, nice work Unity folks.)
We learned about game objects, components, prefabs, and instantiating. We were given a scripting tutorial to complete. This tutorial was essential to learn how scripts can "talk to each other" in Unity. This was my first experience doing javascript, I could follow along, but I honestly didn't understand how it may directly apply to my project until week 9.
Physical buttons vs. virtual buttonsI've been noticing a lot of buttons that do things. Some buttons
are "real" and some are "fake" buttons on a touchscreen. Are we
undecided if touchscreens should be everywhere and takeover? Some
like physical buttons more, I hear. I couldn't be happier with how
awesome my iPhone is. Right now in 2011, it seems people are okay
living in both worlds. Perhaps even enjoying the opportunity to
choose.
Have you seen the smart phones that have both buttons and have a
touchscreen? What a wacky hybrid. Why not? I guess, people like
them. I guess I don't like seeing or having two keyboards. That
logic makes me think the designers expect one of them to break or
something.
Maybe we're just beginning to experiment with different modes of
interaction and figuring a best possible application to meet met/unmet
needs/wants/dreams. How do we figure a best possible applicaton &
find unmet needs? Maybe design research?
___________________________________________________________________________________________
week 3:
We were introduced to the idea of flowcharts and/or pseudocode to plan and develop your scripting. "This process will not only help guide you towards solutions, but will aid in communicating your ideas to the instructor so that suggestions can be provided." – From the syllabus
Professor Price showed examples that explored concepts within Unity:
Colliders, Raycasting, (and Physics) Scripting examples: Selecting objects / Collision detection / Proximity detection / Sending messages
This part of the class is important to experience because we as designers have to understand how Unity is able to: select and detect objects, what Unity is capable of, and also be aware of what limitations may exist due to software and/or hardware constraints.
Emotiv EPOC headset:
Warning: *this is probably my most tangential idea*
The Emotiv EPOC headset works by measuring one's elevated post-exercise oxygen consumption. It is apparently able to do lots of cool things. The part that really sparks my interest is its supposed ability to detect emotion. Only five emotions so far: excitement, boredom, engagement, frustration, and meditation. What if there were pre-fabricated animated behaviors that represent the five detectable emotional states? The user would select an object, the Emotiv EPOC headset would calibrate the emotional state of the person at the time of the object selection, and then the object would inherit an appropriate animated behavior that reflects the person's emotion.
This is a massive undertaking. Animating just five emotions so that they universally represent what they're supposed to is enough work for a separate thesis project, according to Professor Price.
I was inspired from watching the National Geographic film about stress. What if this headset could be used with the goal of calming oneself' down to a meditative state? It could work like a biofeedback system with visualizations. It could be used with a large-scale projection in public spaces, like an airport. If someone is stressed-out due to a delayed flight, they could engage with their own health and be a part of public art.
I see a lot of potential using this headset along with other inputs; with the intent of making a virtual environment as accessible as possible.
I sent an email to the Emotiv people in regard to my ideas, and wanting to experiment with their headset. I have not heard back from them.
___________________________________________________________________________________________
week 4:
Proposal presentations
After my proposal presentation for Professor Price's class, I was challenged to think about what is "voluntary / involuntary", in regards to the user. And how can I design complicated ideas with "simplicity"?
Also, what if the Emotiv EPOC headset could be worn by participants no matter the mode (physical or virtual) generative design research method session? Professor Price brought light to this idea. Imagine: Emotional data being quantitative. Whoa.
Touch:
___________________________________________________________________________________________
week 5:
Planning and prototyping
An "ah-hah" moment I had that improved on ideas I had for the proposal presentation is that there is no good reason (I don't think) for the user having control over the camera view. I came to this epiphany while watching people really enjoy playing Super Smash Brothers for Wii. The camera moves dynamically and the people playing the game are absorbed in the action of their character. At this point, I really started to think more deeply about what functions the user needs to be able to control, and the other stuff should be automatically handled by the environment.
Another change from the proposal: I had been thinking that a Head's Up Display (HUD) would be necessary with different "modes", like play mode, select object mode, background mode, etc. I'm not sure why, but I dropped that completely, and took on the challenge of designing gestural commands with no HUD. Like the commercials say, "You are the remote", well, let's see if we can do that.
___________________________________________________________________________________________
week 6:
Professor Price showed us more about the perks of Unity: Real time Rendering
We learned more about: Materials and Shaders / UV mapping tutorials / Normal Maps / Reflection Maps
The confusing part about Unity is that it doesn't support Maya shaders and textures in a real time rendering sort of way. My impression after watching what Unity is capable of, it seems best to apply materials and textures in Unity, not in Maya. There is something about "baking" stuff in Maya, I think this is referring to a material/texture and/or animation within Maya that is set and then brought into Unity. I decided at this point that I'd try to use Unity for coloring models because the Maya baking thing is confusing to me.
Exciting e-mail exchange!
February 6th, 2011
me: "...What if the user can teach an object a desired behavior via the user's enactment? Kinect? User could build a scenario like in sandquery...then select pre-designed emotional animations for an object, or teach an object a desired motion that helps to better tell their story..."
Prof. Price: "As from our discussion last week, the emphasis on motion, both responsive to and recorded by the participant, could be something that identifies unique characteristics of a virtual environment application for 'sandquery' types of enactment when comparing with your other physical examples.
With the kinect now able to mirror and record full body gesture in the interface lab, I think you could experiment with something that not only allows the participant to manipulate objects with natural gestures, but also can record their movements like motion capture and then apply those movements to your objects in the scene. So the idea of the user "teaching" a behavior to an object is possible, and/or in some combination of providing motions that the user can select from...."
This was good news...My virtual environment could be partly about training objects via motion capture in real time.
___________________________________________________________________________________________
week 7:
Production and progress reports
Dynamic lighting vs. Pre-lighting: Real time shadows
Lightmapping with Beast in Unity / Lightmapping with Mental Ray in Maya / Lightmapping with Maya texture baking
Post-process VFX, is this Buffy stuff? I would really like to spend time learning how to do this someday. Film + effects...
There are cool image effects scripts in Unity Pro.
What I gathered from these lighting examples conducted in class, is that Beast in Unity is very powerful. At this point in my project development I'm not yet applying specific textures, materials and concerned with appropriate lighting. What Unity does with light and shadow is impressive, and I look forward to letting Unity calculating these visual elements (with a few tweaks of the default settings, of course).
How does this virtual environment's class fit in with my thesis interests? At this point in the quarter, I was excited to get feedback about my virtual environment plans from the three very smart people on my Thesis Research Committee.
In the Department of Design at OSU, all MFA candidates must have a 5th quarter review. This means writing about your research is due to one's three person committee, and a presentation about one's research and proposed future research is evaluated. Below are a few images from this presentation. For my thesis research I will conduct generative design research sessions with people, and then analyze and compare the data of three different methods. The first method is paper collage, the second is Sandquery, and the third is a virtual environment that I am now in the process of building. The important constraints that came from our group discussion after my 5th quarter review presentation are:
(1) The virtual environment must support three people in a real time collaborative way.
(2) The objects and materials selected must be consistent across the three methods: 2D, 3D (physical), and virtual.
(3) Design and build an environment that promotes play, body-storming, storytelling and enactment.
I had a lot of ideas. I mapped them out and asked Alan what I would be able to accomplish with the time left in the quarter.
___________________________________________________________________________________________
week 8:
Production and progress reports
The default Kinect block guy is a little dull. Alan suggested I make a new avatar, so I did. I made a rabbit, in honor of the year of the rabbit. For the building, I was inspired by Preston Blair's classic, Cartoon Animation. There are many ways to model 3D objects using Maya. My preferred method is using NURBS curves, lofting the curves, and then converting NURBS surfaces to polygons. The important part about making a new avatar that is compatible for the Kinect is to use the existing joint system of the default Kinect block guy. Then you can build new polygons on the existing joint system, make sure the joint rotations don't deform funny in Maya because they'll do the same in Unity. Pay attention to where all the pivot points are, as well. It's best to combine all of your Maya polygons into a single skin when it's ready to import to Unity. This keeps the file tidy.
*Tangental* Observational notes
One physical place: two games. One is digital + touchscreen, one is a physical hand game
Shuffleboard: physical hand game
Object of game: glide weighted disks on a long wooden board. Scoring is determined by how close the disks are to horizontal segments close to the edge of each side.
Learning notes: sometimes group deliberation occurs so that both sides can come to an agreement about the score. This game is highly competitive. If you have to take a break from playing, you risk losing the table to another group.
Hunks & Babes: digital + touchscreen game
Object of game: compare two similar, but not identical images. Touch where the images are different to get a higher score.
Learning notes: a maximum of two people can touch the screen accurately. People in the back and sides give verbal commands to the people in front. Time is decremented for incorrect guesses or "touch". There is no pause button. If you pause from playing you lose and you need to put more money in to play again. This game is addictive and expensive. And it poses a social problem. How can you have a quality social experience with friends when you can't have a conversation, or give hugs goodbye? I've seen this game in bars since 2005 and it hasn't changed at all.
___________________________________________________________________________________________
week 9:
Production and progress reports
___________________________________________________________________________________________
week 10:
Production and progress reports
At this point, the big bunny was training cubes. Professor Price suggested, "How about making a baby bunny?" So I did.
I have been inspired by lots of things in regard to gesture ideation. Tai Chi, Yoga, American Sign Language, and observed gesture by watching people interact with others, animals, things and telling stories with gesture. For instance, I tried "Right hand on right knee, left hand on left knee" as a "play" command. This is inspired from watching people play with their dogs. Unfortunately, this gesture specifically was not consistent using the Kinect. So, changes were made. The gestural commands have definitely been a trial and error process. The biggest learning point for me has been trying to come up with gestures that people probably WON'T do spontaneously, so the functions won't be triggered while the creator is training the object with their body movements via enactment.
I recorded sound events as user feedback for each function. This to me, for now, is more logical than expecting people to read about what's happening. The sounds become pretty intuitive the more you play with the virtual environment. I used my iPhone to record the sounds. I made sounds from toys, voice, and banging around things in daily life.
___________________________________________________________________________________________
exam week:
I did not do this project alone. I had a bunch of help from the friendly folks at ACCAD.
Scripting credits:
Main script: Me plus help from Alan Price, and students KC, J. Eisenmann and Benjamin Schroeder.
Grabbing script: Alan Price.
Mimicing and motion capture script: Alan Price.
This week has consisted of lots of user-testing (getting different sized people trying out the gestural commands). To simplify things, I made a total of three gestures toggle between different functions. I made some small changes to the sound events. There were some code bugs, meaning there were kinks in the code that needed to be sorted-out. Such as, the record and stop recording functions would happen too close in sequence sometimes. So, now the main script has some timers within the functions so that the virtual environment is more usable. Another problem happened after the first baby bunny was trained and playing back, and if one was trying to playback the second bunny, the sound for "stop playback" would be heard. This was a pickle, but was fixed in the code by making more variables about each baby bunny.
Next steps: I want to use raycasting for object selection instead of collision detection. This will open-up more options for placing objects within a larger scene.
What about more Kinects? With 3 Kinects in a circle with the user in the middle, I wonder if the depth-maps will be more accurate? It'd be great to get this technology to the point where it is able to detect hand gestures.
My intuitive hypothesis at this point, I think there is a strong connection between motion and emotion. And I'm excited to continue to develop this application - but with less bunnies.
