IoT

The video below is a video presentation for a recent ACM conference paper. It illustrates using everyday objects to initiate IoT devices, for example, shaking a cup puts the kettle on.

Some interesting reflections:

• Voice activations, they can be frustrating, especially when people are misunderstood. They can be amazingly useful for people who have lost mobility, e.g. Motor Neurons Disease. Conversations with Things: UX Design for Chat and Voice, "Welcome to the future, where you can talk with the digital things around you: voice assistants, chatbots, and more. But these interactions can be unhelpful and frustrating—sometimes even offensive or biased. Conversations with Things teaches you how to design conversations that are useful, ethical, and human-centered—because everyone deserves to be understood, especially you."
• How could machine learning be integrated with day-to-day routines? Could machines learn our patterns? Perhaps even to learn about our frustrations and moods?
• What if there was a mistake in the trigger, for example, gesturing to increase the volume signalled the kettle to be boiled?
• Related area of study into Human-Objects Interaction, it's not really about moving a chair to switch on a light, rather it is about our routine actions that trigger smart objects. Is this falling to the realm of reading our minds....?
• The potential of the study shows how unconventional triggers can be used to launch preprogrammed chains of events. The future seeing how humans can communicate with smart objects through mediums other than phones or IoT hubs.
• Issues arising with regards to data privacy and security. Could a false trigger endanger a user?
• How to deal with multiple people in the same space? Mix of wants and needs? Public spaces? Commercial or industrial setting?
• Applications more in the business environment, for example, semi-automated restaurant kitchen, cashier-less shops. IFTTT shifted from serving consumers to effectively being a B2B SaSS company.
• What kind of future do we really want to create? Is it really necessary to shake my cup to turn on the kettle? How much abstraction do we want from the actual task? What problem are we trying to solve, and is it a good problem to solve?
• Would it be possible to get some kind of gesture feedback??
• Programming the object gestures would be troublesome and time consuming. And then there would be the issue of remembering what has been programmed. What opens the curtains again?
• Perhaps it would be better if things could read my mind, i'd feel like I have the power of psychokinesis!
• I do see benefits however, for example helping people with mobility issues lead more independent lives would be wonderful!

Mixed Reality

Mixed reality means integrating digital content into the physical world. Unity allows you to develop for Virtual Reality, Mixed Reality, and Augmented Reality.

For Mixed reality you can use MRTK, or Mixed Reality Tool Kit, which is Microsoft software to assist developers to get started with Mixed Reality dev. MRTK integrates with Unity. Hololense

Links:

• Learn MRTK
• Mixed Reality for Cultural Heritage
• Hololens

Mixed Reality Continuum

The end points of the Mixed Reality continuum are actual, everyday, physical reality, and then complete virtual reality, a matrix type world.

Augmented Reality is a little to the side of physical reality, where there is some digital environment which is overlaid onto physical reality.

Ubiquitous Computing, or IoT, is closer to physical reality, digital elements embedded in physical environment.

Virtual Reality as we know it today with the headsets are not perfect virtual reality, but are getting closer.

Augmented Virtuality is where the real world is mapped into the virtual world, for example a video window of the real world is displayed within the virtual world. Or perhaps the virtual world in some way maps onto the real world in real time. Could perhaps argue that Google Earth is a type of augmented virtuality.

An important idea is that mixed reality is a continuum, try not to see it that it is "AR -or- VR", there are elements of each within each other.

Augmented Reality

AR complements what you are experiencing in the real world. VR on the other hand replaces the real world - visual, auditory and depending on the experience, kinaesthetic stimuli are overwritten by the virtual inputs.

AR example applications could be restaurant and cafe displaying details to passers by. Navigation devices guiding walkers to their destinations. Language translation, historical buildings and landmark information.

Ethical Challenges of VR and AR

Link: When creating, deploying and using AR technology, developers, researchers, and marketers face a number of ethical challenges. These include:

• Facial Recognition & Anonymity AR has forward facing cameras and can therefore be used in facial recognition, identifying people in the streets, correlating location and time stamp. Amazon's Ring recently appeared in the news as having passed doorbell footage to law enforcement agencies.
• Mental & Social Side Effects Some critics argue that tech is contributing to the depersonalisation and social isolation of human beings. VR and AR however can also be a force for good, helping people for example with social anxiety and PTSD.
• Unrealistic Expectations Heavy usage of VR and AR can lead people to have unrealistic expectations of real life. An example could be in the adult entertainment industry, over-exaggerated fantasies could become too immersive and lead to real life problems.
• Reality Distortion AR could distort actual reality in a misleading way. For example, property advertisement marketeers could overlay reality so that the property looks better than it actually is! Children, people with special needs, and those living with ongoing mental health issues may need additional protections as they may struggle with the distinction between real and virtual. Safeguards need to be in place to avoid issues of gross manipulation!
• Manipulation & Crime Currently AR is unregulated. Dark patterns could be deployed to manipulate viewers into behaving in ways they would not ordinarily do, e.g. buying a certain product.
• Safety & Distraction For example, walking on the high-street reading restaurant and cafe advertisements could put the user in danger because they are no longer paying particular attention to their surroundings, like obstacles and traffic! Stories abound of teenagers walking out into a road while watching their mobile phones.

There is a need to create a regulatory infrastructure and safeguarding measures, both for the development and deployment of AR and VR. Like all technology this is challenging due to the speed and pervasiveness of development.

UbiComp & Wearables

Ubiquitous computing (UbiComp), a sub-category of IoT, is about embedding computing power in everyday objects. This extends the applicability of HCI and UXD into areas which are traditionally the remit of industrial design and HFE. We must think about how to design and build intelligent solutions that lighten the cognitive load of users.

Robotics

Robotics sits at the intersection of HCI, AI, machine learning, and Philosophy. It raises questions about human work and unemployment, taxation, and the end of humanity as we know it!

There is an emerging field of HRI - Human Robot Interaction. This is currently classified as a sub-field of HCI.

Context-Sensitive (Aware) Computing

Context is fundamental to human interaction, just ask any politician! There is personal context, social context, cultural context, historical context, etc. It is impossible to separate a person from their context, after all no person is an island. If this is the case, then why would this not be a subject of HCI? Would it not make sense that technology adapts to differing contexts?

And indeed it does! An example of context-aware computing is an iPad switching screen orientation between portrait and landscape modes in response to how it is being held. Or perhaps a mobile phone switching to "do-not-disturb" mode when it detects you are in a moving car. Web developers query for all kinds of user context, for example slow internet speed and accessibility needs, to deliver an optimal web experience.

Gesture-based Interaction (GBI)

Sign language is an entire language built on gestures. It is said that body language (a form of gesturing) in a face-to-face conversation accounts for 55% of communication. Currently my desktop computer accepts my keyboard and mouse inputs, as well as voice capabilities (although I rarely use those).

How would it be if our devices could also understand our gestures? This is the domain of GBI or Gesture Based Interaction. Advances have been made in the gaming industry (Xbox, Wii). Other possible applications are doctors being able to perform medical procedures communicating their actions through hand and finger gestures (to be honest a little frightening if something were to go wrong!).

Computer-Supported Cooperative Work (CSCW)

COVID-19 marked an explosion of people working together online. Yes, this is part of CSCW. A good way to think about this is to distinguish between time and place. When designing CSCW, we will either:

• be designing for users at the same time in the same place,
• at the same time in different places,
• or at different times in different places. CCSCW mediates cooperation across the boundaries of geography and time.

Other Areas to Consider

• Pen and touch-based interactions – how touch-based interfaces are taking us back to direct manipulation.
• Information visualisation – how powerful computers and a wealth of available data allow us to visualise complex information;
• Social computing – how computers affect the way we socialise.

References

‘APA Dictionary of Psychology’. 2022. [online]. Available at: https://dictionary.apa.org/ [accessed 1 Oct 2022].