Final Project Journal
This blog will document the process of my Interactive Design MA final project.
Click on the collapsible list to expand.
1. Project Background The Problem: The deaf has difficulties communicating with the ordinary. Deaf individuals communicate through visual, auditory, or tactile modes. To communicate with the hearing individuals, the deaf can only use lip-reading or using residual hearing and spoken English received with a hearing aid. But communication is not easy: only 30% of spoken English can be accurately lip read. The hearing aid devices do not enable deaf individuals to hear at “normal” hearing levels. The most comfortable way for the deaf to communicate is to use sign language, but there are only a few hearing individuals who can understand it. 2. Project Vision 1) Design Question How might we help the deaf communicate more effectively with ordinary people? 2) Project Goal This project aims to help the deaf community communicate more effectively with the ordinary, so their lives can be easier, which will eventually increase their self-confidence. 3) Opportunities
As shown in the above figure, opportunities lie in the connection of each community’s ways to deliver or receive messages. Currently, the two communication systems are distinct from each other, this project aims to build a bridge between them. The problem will be solved when connections are fixed. 4) Industry Trend There are tools that help the deaf community deliver their messages to the hearing community, but there is no product that can help with both delivering and receiving messages at the same time. This creates a space for the proposed project as well as provides tools that can be used directly. 5) Professional Development This project includes designing a mobile device system and physical components. All the skills needed were taught through graduate studies. For the physical prototype, modeling for 3D printing needs to be self-taught. After graduation, I want to become a lead UX designer in a top technology company, with an emphasis on inclusive design and accessibility. This project can demonstrate my professional skills through learning and practice, as well as show my concentration on accessibility. 3. Project Scope For 10 weeks, quantitative and qualitative data will be collected from the deaf community. The system will be designed based on user data. Innovative concepts will be tested and prototypes will be created. After validating concepts with the deaf community, the chosen concept will be developed. The final delivery will be a working system prototype that has both digital and physical parts, which can improve communication efficiency between the deaf and the hearing will be delivered. Promotional materials such as mockups, flyers, and videos may be delivered as well. 4. Process and Methods This project will follow Design Council’s Double Diamond Design Process, which will help tackle the complex social problems and promote innovation. 1) Discover Secondary Research User Interviews Contextual Inquiry Survey Affinitization 2) Define Persona Problem Statement User Journey User Flow 3) Develop Low-Fidelity Prototype Idea Validation User Testing Iteration 4) Deliver Branding Visual Design High-Fidelity Prototype Physical Prototype
Week 1 Research
1. There is a lot of information about the deaf community on the national deaf center website. The most useful one is Communicating With Deaf Individuals: https://www.nationaldeafcenter.org/sites/default/files/Communicating%20with%20Deaf%20Individuals.pdf
2. Another one to understand how the deaf communicate:
3. A potential resource to contact the deaf community for user research:
4. Youtube has a lot of videos that represent the deaf individuals' thoughts, the most useful one showcases the misunderstandings between the hearing and the deaf:
5. This discussion helps me understand the deaf's communication obstacles better from a unique perspective:
We can not take it for granted that the deaf can read captions well and assume that sign language is just a translation of speaking language. Rather, "many Deaf are, unfortunately, not fluent in the majority Hearing language, and may not understand the captions, or misunderstand the content due to linguistic conflicts and comprehension issues."
Outline and Schedule
As mentioned in the project proposal, this project will follow Design Council’s Double Diamond Design Process. In the Discover phase, the deliveries should be data collected from primary and secondary research. In the Define phase, the insights will be translated into a persona, journey map, storyboard, and user flow. In the Develop phase, designs will be carried out through sketches and prototypes. In the Deliver phase, the visual portion will be improved and after testing and iteration, the final product will bedelivered with promotional materials.
If unexpected things happen and tasks can not be done on time, the fidelity of final deliveries can be reduced while design concepts can be expressed clearly. Currently, more time is needed for the Discover phase.
No enough time
Current status: seeing a trend of not finishing tasks at the expected level of quality due to lack of time.
Cause: the SCADPro course is occupying a lot of time, causing many short sleep nights.
If one task is occupying too much time, find the reason and take action to have the overload duties distributed to the team.
If unnecessary thinking about stressful tasks is occupying too much time, do meditation or seek help from friends and school.
Current status: relating with the last risk, seeing a trend of stress.
Cause: stress from lack of time, and concern about work.
Take care of myself: find time for rest, do more exercise, balance between school and life.
Refuse necessary work by speaking out, communicating with the professors and the group.
Prioritize health before everything else.
Getting things done, instead of focusing on being perfect.
Limit technical capability
Current status: already decided the area of technology
Cause: not familiar with AR and MR technology and never design within these areas.
Address questions one by one: How to prototype the MR program? How complex it can be? If development is needed, how can I finish to a certain level before the quarter ends?
After finding the answers, think about how much can be done new things need to be learned at the same time.
Update timeline and adjust outcome fidelity of the project.
Lack of research to support design
Current status: data is collected.
Cause: I am not a member of the deaf community, and have no experience connecting with them. It is possible failing to find the users for testing.
Find an alternate way to test: test with general users does not have to be the deaf.
Send prototype online and collect feedback.
Contact companies or charity organizations for possible contacts.
Week 2 Research
Key takeaways from NDC "Communicating With Deaf Individuals":
There is no "one-size-fits-all" approach to communication, nor is there a "typical" deaf person.
Broadly defined, communication for deaf individuals occurs through visual, auditory, or tactile modes (for individuals who are deafblind). Common visual communication modes include American Sign Language, cued speech, speech reading (lip reading), and gestures. Auditory communication includes using residual hearing and spoken English received through the ear, often augmented with a hearing aid or cochlear implant to enhance the ability to interpret sound. Tactile communication translates visual and auditory communication into the hand and other parts of the body.
When first meeting a deaf person, do not make assumptions about the individual’s communication. Rather, inquire directly about the individual’s communication needs.
Get the attention of the deaf individual before speaking. If the individual does not respond to the spoken name, a tap on shoulder or another visual signal is appropriate.
Only about 30% of English speech sounds are visible on the mouth under the best of conditions. Factors that can affect speechreading include residual hearing, body language and facial expressions, distance from the speaker, and awareness of the topic under discussion. Communication or conversations may be easier one-on-one in a quiet setting but more difficult in a group or in a noisy environment.
Key takeaways from Quora discussions:
Most of the time English is not the first language of deaf people in America. It is usually ASL or some form of visual communication.
The education of deaf people in America has usually provided about a fourth-grade reading level (that’s usually the statistic). Every Deaf person’s English-language background/education/exposure is different.
Firstly, subtitles and closed captions are two completely different things. With closed captions, you are told exactly what is happening in the background, with *car door slams* and *eerie breathing in the background* rather than just dialogue that is spoken.
Through American Sign Language (or another Sign Language system) deaf people are able to understand the movie on a similar scale as a hearing person. American Sign Language does a phenomenal job expressing feeling, emotion, suspense, and storytelling. Possibly even on a better scale than spoken English.
At this point in time, CC for movies is I think the best solution. I’ve been an interpreter in a movie theater and find it not a good experience. There’s no perfect solution other than making a true deaf film such as Peter Wolf’s “Think Me Nothing”, “Deafula” etc.
Varies of preferred communication: visual, auditory, or tactile.
How to address this variety through one solution; or focus on the majority?
A cue to raise their attention before speaking.
Add Visual or tactile cues to start communication.
The best way to interpret movies/tv is directly translated them to ASL, which will require an ASL interpreter.
For deaf/HH individuals, sign language is their first language, and English is their second language, there is translation involved. And the English education level for the community is different.
Is it possible to use ASL as the medium of communication through technology rather than English? Current trending technology mostly uses English as the medium.
Key takeaways from "Deaf and Hard-of-hearing Individuals’ Preferences for Wearable and Mobile Sound Awareness Technologies"
Findlater, Leah & Chinh, Bonnie & Jain, Dhruv & Froehlich, Jon & Kushalnagar, Raja & Lin, Angela. (2019). Deaf and Hard-of-hearing Individuals' Preferences for Wearable and Mobile Sound Awareness Technologies. CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1-13. 10.1145/3290605.3300276.
A primary outcome of this work is a set of recommendations for the design of mobile and wearable sound awareness technologies.
Form factor and feedback modalities. As a single device, the smartwatch is a promising form factor. However, there is a strong desire for both visual and haptic feedback (92% of participants) and even for this feedback to be provided on two separate devices (75% of participants); the most promising combinations are haptic feedback on a smartwatch and visual feedback on an HMD(head-mounted-display) or smartphone. For oral conversation support, HMDs offer the most promise.
Sound types. Urgent, safety-related sounds and voices directed at the user are of highest priority, followed by non-urgent alerts, people’s presence, and nature background noises. Outdoor background noises, voices not directed at the user, and indoor mechanical noises are of lower interest and could be excluded or at least automatically filtered out. This recommendation reflects past work [3, 27, 39] but with a more detailed prioritization of sound categories.
Sound characteristics. For information about the characteristics of sound, the source and location are of high interest (also reflecting past work ).
Captions. When comparing full vs. summary captions, full captions were of the highest interest, though keyword captions may be useful to some users who prefer oral communication. Captions should be provided on an HMD or smartphone.
Notification and filtering. Notifications should typically be shown immediately, but with the option to have lower priority sounds displayed only on demand. Users may want to specify which sounds should be filtered/shown and this may differ based on context (e.g., at home, at the store).
“Airplane” mode. Notifications should be able to be turned off easily, to reduce distraction and to accommodate the user’s desire to act appropriately in varying social contexts.
Cultural and Social Considerations Social context affects perceived usefulness and comfort with using a mobile or wearable sound awareness device, likely reflecting a variety of factors such as social norms around polite/rude technology use, changing social perceptions of wearable technology , and the stigma that can be associated with assistive technologies . Several participants reported feeling the need to explain what the technology does so that other people are accepting of it, which reflects past work showing that wearable devices are more likely to be socially acceptable if they are perceived as being used for assistive purposes . An important tension to address in future work is that users may be less socially comfortable using these devices around strangers, but that is also the context in which need or utility may be highest. An additional critical aspect is how a device like this may be used (or not) in a Deaf cultural context. The significant differences between individuals preferring sign language vs. oral communication vs. both likely reflect this consideration. Deaf people often get frustrated with research that uses visual/haptic devices to support communication or notification, but that does not capture usable information at all (e.g., hearing or signing gloves), have too much friction for usable communication, or are too cognitively taxing.
Key takeaways from “The Deaf Community and How You Can Get Involved”
Key takeaways from “Understanding Assistive Technology for Deaf and Hard of Hearing”
According to the National Association of the Deaf (NAD)
Deaf: Those who identify as Deaf (with a capital D) communicate with sign language. These are often those who have been deaf for most of their lives.
deaf: The lowercased is for those who do not identify as part of the Deaf culture. These can include those who became deaf later in life.
Hard of hearing (HoH): This describes those who have some hearing loss, but not complete hearing loss.
Assistive technology for Deaf/Hard of Hearing users
Alerting devices: An alerting device converts an audio alert (e.g., doorbell, fire alarm, alarm clock) into a visual or physical alert that the person can perceive.
Telecommunications: Many different options are available for those who are d/Deaf or HoH, including amplified telephones, TTY / TDD (software and hardware), real-time text (RTT), captioned telephones, Text-to-911, video chat, and text and video relay services.
Enhanced/Assistive listening: Systems can be used to overcome background noise and provide a more direct audio feed for someone who uses assistive listening devices.
In a classroom, a teacher could wear a small microphone that uses an FM radio system to transmit audio to a student’s hearing aid.
In a theater, an infrared or audio induction loop system can be used so that audience members with hearing impairments can hear the play through their hearing aids or cochlear implants.
At work, an employee can couple their cochlear implant or hearing aids with their computer via Bluetooth and hear their computer’s audio without needing headphones.
Accessibility for deaf and Hard of Hearing users
Here are some accessibility issues that restrict access to people who are d/Deaf/HoH:
Captions that are not synchronized properly
Phone-only customer support
What an Accessible Site Looks Like
Accurate Captions: Captions allow viewers who are deaf or hard of hearing to follow the dialogue and the action of a program simultaneously.
It’s critical that captions are accurate and also include non-speech elements, otherwise the content may be incorrect, incomprehensible, or incomplete
Transcripts: Providing a transcript is another great way for deaf or hard of hearing users to follow along and have another means of consuming the content. (However, transcripts should not be used as a replacement for captions!)
Multiple methods of contact and communication: Deaf and hard of hearing visitors may have a difficult time communicating over the phone. Providing an email address, or alternative means of contact will help.
High-quality, clear audio with minimal background noise: Quality audio will make it easier to ensure accurate captions. Poor audio quality makes it harder for transcribers to capture all the words spoken, leading to transcripts with many [inaudible] or flagged spots.
Clear and high-quality audio will also be easier for hard of hearing users to understand more clearly
Use of clear and simple language: American Sign Language (ASL) is a different language than English, and it has its own grammar structure. Individuals who use ASL as their primary language may not be fluent in English, so making written content clear and simple to understand is important, and can be done in the following ways:
Avoid slang and confusing jargon
Use headings and subheadings to properly structure your content
Include bulleted lists
Employ an active, rather than passive, voice
Provide definitions in simple terms
Use consistent language throughout the content
"People don't want to be rude so they make us invisible."
Cochlear Implant: with it, many D/HHs can directly communicate with the hearing.
Cochlear implant only works for certain groups of people (Wearing hearing aids first is a necessary step in the evaluation process for a cochlear implant.)
Expensive: cost between $20,000 and $40,000
These can serve as great guidelines when designing the product. Also, it is important to consider how one can approach the deaf community. Besides considering the functional aspects of the product, social and cultural considerations are also very important.
How to make the product functional while avoiding strengthening the D/HHs' disability identity?
Contacts for resources and potential primary research community:
Don Grushkin Deaf, Ph.D. In Language, Reading & Culture On Quora.
Chatham County Association of the Deaf on Facebook.
Sginall - SignAll Chat enables spontaneous communication between the Deaf and hearing.
Maybe it's better to only conduct surveys but not in-person interviews? (Communication difficulties - exact what we need to design for).
Based on what I learned from secondary research, I decided to design a survey and spread it to the deaf community.
At the same time, I need to start the ideation phase and justify them with research insights. Later the concept will be further adjusted according to primary research findings.
Competitor - Intelligent Gloves
Below is a pair of intelligent gloves that can turn language into audible language. It's very inspiring. But from last week's research, these still lack some consideration about the deaf culture:
how a device like this may be used (or not) in a Deaf cultural context. The significant differences between individuals preferring sign language vs. oral communication vs. both likely reflect this consideration. Deaf people often get frustrated with research that uses visual/haptic devices to support communication or notification, but that does not capture usable information at all (e.g., hearing or signing gloves), have too much friction for usable communication, or are too cognitively taxing.
Learning from this project, the advantage is the convenience for the deaf community.
Below are the things it inspired me to think about when designing:
It did not include other usable information other than the sign language itself.
When designing, the difference between the preferred ways of communication should be considered.
How to reduce the cognitive load?
Competitor or Tech Supplier - Signall
The goal behind this product is very similar to my project, but the actual system is not the same as my concept. Also, the goal is to help the deaf and the hearing communicates more easily and not be limited to a certain space. The company also has technology that translates sign language to English, which supports the possibility of my proposal.
Based on week 2 research and available technology, I proposed my concepts below.
Based on the key insights, this concept has advantages compared to other projects:
Easy to carry along
Two direction communication
Both the hearing and the deaf find it easier
Applicable to most situations
Adjustable for different communication preference
The other projects:
Single direction communication
The hearing finds it easier but not the deaf
Applicable to limited situations
Only work for the deaf that knows English
Before starting the design, I need to understand the constraints.
Smart glasses vs AR glasses vs MR glasses
The Difference Between Smartglasses & AR Glasses, and Why Everyone is Confused
Microsoft HoloLens 2 https://www.microsoft.com/en-us/hololens
Physical devices communication technology
HMD to mobile devices
Accessary to mobile devices
Which device works as the processor
Voice to text
English to ASL
ASL to English
HMD design guidelines
Optical/Visual Guidelines and Recommendations The following optical/visual parameters and issues are addressed:
Ocularity (monocular, biocular or binocular)
Pupil-forming versus non-pupil-forming optics
Exit pupil and eye relief
Luminance and contrast
See-through versus non-see-through considerations
Considerations for helmet-mounted sensors
Mobile device interface guidelines
Which device should the system use? Is it regular transparent screen glasses or AR, MR glasses?
I would say for now the MR glasses is the best option: the user needs to see the physical world; the area for the app on smart glasses is limited, which will distract the user's attention: the user eye's focal point needs to switch between the side and the person talking to him/her.
If using AR/MR glasses, Microsoft provided very detailed guidance of design. And the study of it should start now as this is a whole new area for me.
Seeing the bright future!!!
One of the concerns I had with Microsoft Holo Lens 2 is that the device is very expensive and bulky. But seeing this commercial-level product on the market makes the future of my proposed system very promising. This device only already has the ability to provide the service that my system is offering. And designing an app on the current mobile phone system is much easier.
I am excited to see that nreal comes with hand tracking. Which makes my proposed system possible.
Which platform should I use?
Microsoft provided very detailed design guidelines, with tools and kits. And the system can be used standalone on its MR glasses. Whereas, nreal is based on the current Android or IOS system, which means I need to design the system as an Android or IOS system. This is an advantage for regular users, but nreal's design guideline is not very detailed, I do not know how to start designing for their system with developer knowledge.
I need to learn more about how it works or dig into the guideline Microsoft provided to decide which platform I should use. I also need to consider the time remaining for me to complete the design.
While learning how to start designing AR apps, I finished the user flow.
Using AR/MR for this project. The justification.
User Interface Option
AR environment can use either GUI or NUI depending on the devices.
GUI when using AR apps through phones or tablets.
NUI when using some AR glasses, below is a demonstration of nreal interface.
User interfaces that you interact with using modalities such as touch, gestures, or voice are often referred to as Natural User Interfaces (NUI).
NUI is very intuitive but GUI also works for this project.
Information can be docked or anchored in physical space.
This is a new way of organizing information by associating it with a space in the physical world to correspond with relevance, etc.
The interpreter's translation can be docked with the hearing.
Organize cognitive load by spatial prioritization.
The concept of Situated Analytics (SA) is the use of data representations organized in relation to relevant objects, places, and persons in the physical world for the purpose of understanding, sensemaking, and decision-making.
Collaborate with an interpreter or other relevant actors separated by distance.
So not only a virtual interpreter but also a real interpreter can be present for translation.
Information can be collectively shared and looked at organically.
The above images show the possible ways of providing translation through an AR/ MR environment. The image on the left is an example of telepresence. The image on right is an extended version of video chat, that has a better demonstration of the person online.
For my project, I think telepresence is one of the best solutions for the deaf that an interpreter can be present whenever he/she needs it. The other solution is a virtual interpreter, and it's good enough for daily situations.
Seeing the need for interpreter participation, I updated my user flow, to give users the option of a real interpreter session for important occasions.
I was so scared when heard HoloLens can only be used indoors. But after investigating HoloLens2 and the use cases, I found that it's usable outside, standalone. So the original plan is still feasible.
Look into the design of Mixed Reality
Fundamental understanding of the design guidelines for Mixed Reality
Mixed reality blends both physical and digital worlds. These two realities mark the polar ends of a spectrum known as the virtuality continuum. We refer to this spectrum of realities as the mixed reality spectrum. On one end of the spectrum, we have the physical reality that we humans exist in. On the other end of the spectrum, we have the corresponding digital reality.
A Taxonomy of mixed reality Visual Displays
the application of mixed reality has gone beyond displays to include:
Environmental understanding: spatial mapping and anchors.
Human understanding: hand-tracking, eye-tracking, and speech input.
Locations and positioning in both physical and virtual spaces.
Collaboration on 3D assets in mixed reality spaces.
Human input can now include keyboards, mice, touch, ink, voice, and Kinect skeletal tracking.
API names in Windows that reveal environmental information are called the perception APIs. Environmental inputs can capture:
a person's body position in the physical world (head tracking)
objects, surfaces, and boundaries (spatial mapping and scene understanding)
ambient lighting and sound
A combination of three essential elements sets the stage for creating true mixed reality experiences:
Computer processing powered by the cloud
Advanced input methods
The experiences that overlay graphics, video streams, or holograms in the physical world are called augmented reality. The experiences that occlude your view to present a fully immersive digital experience are virtual reality. In the future, new devices with a more expansive range are expected: holographic devices will be more immersive, and immersive devices will be more holographic.
Towards the left (near physical reality). Users remain present in their physical reality and aren't made to believe they have left that reality.
In the middle (fully mixed reality). These experiences blend the real world and the digital world. For example, in the movie Jumanji, the physical structure of the house where the story took place was blended with a jungle environment.
Towards the right (near digital reality). Users experience a digital reality and are unaware of the physical reality around them.
Whether a device is tethered to a separate PC (via USB cable or Wi-Fi) or untethered doesn't reflect whether a device is holographic or immersive. Features that improve mobility often provide better experiences. Holographic and immersive devices can be either tethered or untethered.
To conclude, this project is on the left to middle spectrum: AR and MR work best as users need to remain present in their physical reality with part of the digital world components.
What is a hologram?
Holograms can respond to your gaze, gestures, and voice commands. They can even interact with real-world surfaces around you. Holograms are digital objects that are part of your world.
A hologram is made of light and sound
Holograms add light to your world, which means that you see both the light from the display and the light from your surrounding world. Since HoloLens uses an additive display that adds light, the black color will be rendered transparent.
Holograms can have different appearances and behaviors. Some are realistic and solid, and others are cartoonish and ethereal. You can use holograms to highlight features in your environment or use them as elements in your app's user interface.
Holograms can have different appearances and behaviors. Some are realistic and solid, and others are cartoonish and ethereal. You can use holograms to highlight features in your environment or use them as elements in your app's user interface.
A hologram can be placed in the world or tag along with you
When you have a fixed location for a hologram, you can place it precisely at that point in the world. As you walk around, the hologram appears stationary based on the world around you, just like a physical object. If you use a spatial anchor to pin the object, the system can even remember where you left it when you come back later.
Some holograms follow the user instead. They position themselves based on the user. You can choose to bring a hologram with you, and then place it on the wall once you get to another room.
Some scenarios demand that holograms remain easily discoverable or visible throughout the experience. There are two high-level approaches to this kind of positioning. Let's call them display-locked and body-locked.
Display-locked content is locked to the display device. This type of content is tricky for several reasons, including an unnatural feeling of "clingyness" that makes many users frustrated and wanting to "shake it off." In general, designers have found it better to avoid display-locking content.
Body-locked content can be far more forgiving. Body-locking is when you tether a hologram to the user's body or gaze vector in 3D space. Many experiences have adopted a body-locking behavior where the hologram follows the user's gaze, which allows the user to rotate their body and move through space without losing the hologram. Incorporating a delay helps the hologram movements to feel more natural. For example, some core UI of the Windows Holographic OS uses a variation on body-locking that follows the user's gaze with a gentle, elastic-like delay while the user turns their head.
Place the hologram at a comfortable viewing distance typically about 1-2 meters away from the head.
Allow elements to drift if they must be continually in the holographic frame, or consider moving your content to one side of the display when the user changes their point of view. For more information, see the billboarding and tag-along article.
Place holograms in the optimal zone--between 1.25 m and 5 m
Two meters is the optimal viewing distance. The experience will start to degrade as you get closer than one meter. At distances less than one meter, holograms that regularly move in depth are more likely to be problematic than stationary holograms. Consider gracefully clipping or fading out your content when it gets too close so you don't jar the user into an unpleasant viewing experience.
A hologram interacts with you and your world
Holograms enable personal interactions that aren't possible elsewhere. Because the HoloLens knows where it is in the world, a holographic character can look at you directly in the eyes and start a conversation with you.
A hologram can also interact with your surroundings. For example, you can place a holographic bouncing ball above a table. Then, with an air tap, watch the ball bounce, and make sound as it hits the table.
Holograms can also be occluded by real-world objects. For example, a holographic character might walk through a door and behind a wall, out of your sight.
Tips for integrating holograms and the real world
Aligning to gravitational rules makes holograms easier to relate to and more believable. For example: Place a holographic dog on the ground & a vase on the table rather than have them floating in space.
Many designers have found that they can integrate more believable holograms by creating a "negative shadow" on the surface that the hologram is sitting on. They do this by creating a soft glow on the ground around the hologram and then subtracting the "shadow" from the glow. The soft glow integrates with the light from the real world. The shadow is used to ground the hologram in the environment.
Types of mixed reality apps
Enhanced environment apps (HoloLens only)
Blended environment apps - for this project
Immersive environment apps
Designing for Content
let users adjust to the experience.
Large Objects: objects that can't normally fit within the holographic frame should be shrunk to fit when they're first introduced (either at a smaller scale or at a distance). The key is to let users see the full size of the object before the scale overwhelms the frame.
Many objects: experiences with many objects or components should consider using the full space around the user to avoid cluttering the holographic frame directly in front of the user. Let users understand the content layout in the experience.
One technique to achieve this is to provide persistent points (also known as landmarks) in the experience that anchors content to the real world.
Objects can also be placed in the periphery of the holographic frame to encourage user to look toward key content.
For this project, I'm building a world-scale experience (users wander beyond 5 meters)
The user walks around to different places
The scenarios can be either indoor or outdoor
The interface is docked relative to user head position but the virtual interpreter is docked around the person that's speaking to the user
Avoid head-locked content
We strongly discourage rendering head-locked content, which stays at a fixed spot in the display (such as a HUD). In general, head-locked content is uncomfortable for users and doesn't feel like a natural part of their world.
Head-locked content should usually be replaced with holograms that are attached to the user or placed in the world itself. For example, cursors should generally be pushed out into the world, scaling naturally to reflect the position and distance of the object under the user's gaze.
For maximum comfort, the optimal zone for hologram placement is between 1.25 m and 5 m. In every case, designers should attempt to structure content scenes to encourage users to interact 1 m or farther away from the content (for example, adjust content size and default placement parameters).
Although content may occasionally need to be displayed closer than 1 m, we recommend against ever presenting holograms closer than 40 cm. Thus, we recommend starting to fade out content at 40 cm and placing a rendering clipping plane at 30 cm to avoid any nearer objects.
Objects that move in depth are more likely than stationary objects to produce discomfort because of the vergence-accommodation conflict. Similarly, requiring users to rapidly switch between near-focus and far-focus (for example, because of a pop-up hologram requiring direct interaction) can cause visual discomfort and fatigue. Extra care should be taken to minimize how often users are: viewing content that is moving in depth; or rapidly switching focus between near and far holograms.
When designing content for direct (near) interaction in HoloLens 2, or in any applications where content must be placed closer than 1 m, extra care should be taken to ensure user comfort.
We recommend creating a “depth budget” for apps based on the amount of time a user is expected to view content that is near (less than 1.0 m) and moving in depth. An example is to avoid placing the user in those situations more than 25% of the time. If the depth budget is exceeded, we recommend careful user testing to ensure it remains a comfortable experience.
To avoid gaps in the user interaction experience, it's best to follow the guidance for a single model from beginning to end.
For this project, the interaction system that will be used is hands-free.
Because the user needs to use both hands for sign languages, most of the time, their hands for occupied. And to decrease the learning load, I will only use one interaction mode for now. But when the user is familiar with the MR environment, it is possible to include the Direct manipulation with hands model when the user is not communicating.
Because not all the deaf can speak, so more specifically, the interaction model is hands-free with Gaze and dwell.
This will also decrease social awkwardness when using large hand gestures, especially when the deaf community really care about it.
For similar reason above, using eye-gaze and dwell instead of head-gaze and dwell.
The choice of dwell time can be tricky. Novice users are ok with longer dwell times, while expert users want to quickly and efficiently navigate through their experiences. This leads to the challenge of how to adjust the dwell time to the specific needs of a user. If the dwell time is too short: The user may feel overwhelmed by having holograms react to their eye-gaze all the time. If the dwell time is too long: The experience may feel too slow and interruptive as the user has to keep looking at targets for a long time.
User experience elements
Rendering on holographic devices
Holographic devices have additive displays – Holograms are created by adding light to the light from the real world – white will appear brightly, while black will appear transparent.
Colors impact varies with the user’s environment – There are many diverse lighting conditions in a user’s room. Create content with appropriate levels of contrast to help with clarity.
Avoid dynamic lighting – Holograms that are uniformly lit in holographic experiences are the most efficient. Using advanced, dynamic lighting will likely exceed the capabilities of mobile devices. When dynamic lighting is required, it's recommended to use the Mixed Reality Toolkit Standard shader
Rendering light colors - White appears bright and should be used sparingly. For most cases, consider a white value around R 235 G 235 B 235. Large bright areas may cause user discomfort. For the UI window's backplate, it's recommended to use dark colors.
Rendering dark colors - Because of the nature of additive displays, dark colors appear transparent. A solid black object will appear no different from the real world. See Alpha channel below. To give the appearance of “black”, try a very dark grey RGB value such as 16,16,16.
Color uniformity - Typically holograms are rendered brightly enough so that they maintain color uniformity, whatever the background. Large areas may become blotchy. Avoid large regions of bright, solid color.
Gamut - HoloLens benefits from a "wide gamut" of color, conceptually similar to Adobe RGB. As a result, some colors can show different qualities and representation in the device.
Gamma - The brightness and contrast of the rendered image will vary between immersive and holographic devices. These device differences often appear to make dark areas of color and shadows, more or less bright.
Color separation - Also called "color breakup" or "color fringing", color separation most commonly occurs with moving holograms (including cursor) when a user tracks objects with their eyes.
Storytelling with light and color
Vignetting - A 'vignette' effect to darken materials can help focus the user's attention on the center of the field of view. This effect darkens the hologram's material at some radius from the user's gaze vector. This is also effective when the user views holograms from an oblique or glancing angle.
Emphasis - Draw attention to objects or points of interaction by contrasting colors, brightness, and lighting. For a more detailed look at lighting methods in storytelling, see Pixel Cinematography - A Lighting Approach for Computer Graphics.
The scale of an object is one of the most important visual cues because it gives viewer a sense of the objects size and cues to its location. Further, viewing objects at real scale is one of the key experience differentiators for mixed reality in general – something that hasn’t been possible on previous screen-based viewing.
Create clear hierarchy
Avoid using more than two different font families in a single context. Too many fonts will break the harmony and consistency of your experience and make it harder to consume information.
Avoid thin font weights
Avoid using light or semilight font weights for type sizes under 42 pt because thin vertical strokes will vibrate and degrade legibility. Modern fonts with enough stroke thickness work well. For example, Helvetica and Arial are legible in HoloLens using regular or bold weights.
In HoloLens, since the holograms are constructed with an additive light system, white text is highly legible. Even though white text works well without a back plate on HoloLens, a complex physical background could make the type difficult to read. We recommend using white text on a dark or colored back plate to improve the user's focus and minimize the distraction from a physical background.
To use dark text, you should use a bright back plate to make it readable. In additive color systems, black is displayed as transparent. This means you won't see the black text without a colored back plate.
Recommended font size
As you can expect, type sizes that we use on a PC or a tablet device (typically between 12–32pt) look small at a distance of 2 meters. It depends on the characteristics of each font, but in general the recommended minimum viewing angle and the font height for legibility are around 0.35°-0.4°/12.21-13.97 mm based on our user research studies. It's about 35-40 pt with the scaling factor introduced in Text in Unity page.
Interaction sound not applicable to this project.
User can choose the representative sound when the system translate and speak to the hearing.
Based on the above design considerations, I did sketches for the primary flow.
I did an inception sheet to figure out the direction of the style.
I created a moodboard to ideate style around the keywords from the inception sheet.
From the Inception Sheet, the color range I am considering is blue or purple with a splash of yellow. And this perfectly aligned with my moldboard. There is a trend that starts from blue, and with more and more red adding to it, then ended up with yellow even peach.
Blue sparks a feeling of professionalism and also when used with dark light, inspires a high-tech feeling. Yellow gives people a feeling of warmth and happiness.
Therefore, I plan to use blue as the primary color with an accent color of yellow. To give users a feeling of trustworthiness, professionalism yet still welcoming and helpful.
Most of the time when I see HOH persons using sign language, they always have a big smile on their faces. I hope this app can help bring them more smiles when communicating with everyone, even with those that do not know sign language.
Design for Hi-fidelity
Figma Interactive Prototype
Prototype in Unity for HoloLens2
Encountered a lot of problems with Visual Studio. No matter if I connect HoloLens2 with PC through USB or Wifi, there were always errors when I tried to deploy.
I tried several solutions online but none of them work, so I have to pause the development in Unity.
But I managed to simulate the MR environment: the near menu can move with "head", and "user" can interact with hands.
The downside is that the hand manipulation simulator is not very smooth and often times points in the wrong direction. So it's better to try the prototype on HoloLens.
I have to source external help with the issue. But due to the timeframe, the development in Unity will be continued after this quarter ends.
I need to finish the Unity prototype and integrate SignAll SDK to have the prototype really work and to better user testings.
Preferably the testing can be with HoloLens2 so that real data can be collected.
Even though currently HoloLens2 is not super smooth, doing this project, I am confident that I am on the right path for the future solution for more efficient communication between the deaf and the hearing.