Innovation Design Engineering (MA/MSc)
Deepak Mallya
Deepak Mallya is an Interaction Designer committed to humanising technology. He has a broad range of experience creating impactful physical and digital products, with a keen interest in designing meaningful supporting roles for the intelligent devices in our lives. His work is inspired by the ways in which we interact with each other and the cultures that surround them. He enjoys working with people across disciplines, allowing him to learn and apply interdisciplinary approaches to tackling complex challenges.
His work has been featured in global media outlets including BBC, Sky News, Reuters, The Evening Standard, Designboom and Lampoon Magazine.
Selected Awards
Fast Company World Changing Ideas (2020)
Imperial Venture Catalyst Challenge Moonshot Prize (2020)
Core77 Design award (2020, 2019, 2018)
RCA CERN Grand Challenge Finalist (2019)
Lexus Design Award (2018)
IxDA Award (2015)
The Associated Chambers of Commerce and Industry of India,
Innovation Excellence Award (2014)
Selected Exhibitions
Imperial Tech Foresight 2040: Moral Machines (2020)
Imperial Lates: Future Commuter (2020)
Royal Commission of 1851, Exhibition Road Festival (2020)
BA2119, Flight of the Future, Saatchi Gallery (2019)
Consumer Electronics Show (2014, 2015)
Experience
Co-founder and Lead designer, Treemouse Research and Design, New Delhi (2016–18)
Graduate Teaching Assistant, Dyson School of Design Engineering, Imperial College (2019–20)
Visiting Tutor, New media Design, National Institute of Design, Gandhinagar (2017)
Senior Interaction Designer, Honeywell Automation Control Systems Bangalore (2015–2016)
User Experience Designer, Design and New Applications SAP Labs, Bangalore (2013–2015)
I came to IDE with three goals in mind.
Design with AI
Neural Networks are some of the most consequential technologies shaping our everyday and I wanted to understand what it means to design with them. At IDE, I spent time tinkering with machine learning to demystify it for myself. Through my projects, I learned first hand to acknowledge their drawbacks and issues. I was also able to see the positive impact AI can have by complementing human agency. AI for me is becoming a medium of craft and a material to shape.
Collaborate across disciplines
As an interaction designer, I needed to understand how designers can think and work laterally to involve people from the sciences in the creative process. As critical stakeholders with a deep understanding of their areas of expertise, I wanted to understand their motivations and insights and create buy-in from them when taking on complex challenges. I learned how to navigate these channels, wear different hats and facilitate conversations to design a compelling narrative in my work.
Take innovation outside the lab
Design must convince users and stakeholders in the extensive process to create impact. I wanted to develop the skills to convince stakeholders and investors in the industry to see both the economic and creative value in the solution. Participating in business development programmes helped me identify customer needs, communicate market opportunities and build relationships with investors. It is not just ideas that get investment; it is also the abilities of the people behind them.
These two years have given me invaluable experiences in working hands-on with technology, building and managing relationships with people across the different stages of innovation, and have helped me understand what it takes to become a holistic design leader.
Language learning apps and education technologies have made languages accessible to millions of people around the world. Language app downloads have surged even during the pandemic, with people wanting to learn a new skill at home. Insights from interviewing self-learners showed that apps provide a strong foundation for the language, but learners lacked the confidence to apply the language they had learned and did not have a convenient way to practice.
Poli was built by talking to teachers, understanding the needs of learners and studying the habits of native speakers. Learning a new language requires practice and motivation. Poli is that flexible, convenient learning companion that makes learning feel natural and gives learners a glimpse into the cultures and linguistic habits of the people that surround those languages.
Languages have evolved over centuries—layered and blended by our interactions with each other. Grasping these cultural nuances would allow voice assistants to understand our motivations and intentions better so we can think beyond trivial task-oriented dynamics and cater to the higher motivations and expectations we have from them.
Poli is designed around Topics of conversation — Topics can be about informal everyday conversation like talking about the weather, or formal dialogues which cover more traditional social situations and norms, like how to address different people or work related topics.
People speak differently in different social situations, and Poli understands those subtleties. In the case of Hindi, mixed speech is the most natural way to converse everyday, and is a strong part of the identity of its speakers. Poli helps you understand these cultural idiosyncrasies, through special Informal Topics in Hinglish—the unofficial Hindi-English blend spoken across the country.
Don’t hesitate if you have cannot recall a word or don’t know how to say a sentence correctly. Poli is able to identify switches, suggest the right words or offer help with parts of speech.
Following conversations in a new language can be tough; so you can adjust Poli’s pace so it responds as fast or as slowly as you want allowing you to follow at a comfortable speed.
Hardware — Poli uses fairly simple hardware, including a hacked PS3 eye Camera for its mic array, a Raspberry Pi for the computation, and a speaker for its output
Software — Poli is built using a combination of Neural Networks working in tandem to deconstruct, analyse and synthesise voice interactions a large part of Poli’s technologies run on-device.
Button Details — Power, volume and mute buttons
Christening Poli with a Wake Word Listener Neural Net — 1. Data Gathering – Recordings of the words “Ok Poli” were gathered from different people. Recordings of rhyming and similar sounding words, random other words and ambient noises were also included to reduce false activations. 2. Training – Data was broken into training and test sets and trained multiple times as more data was gathered. 3. Testing – Once training was done, the wake word was tested to see if it could trigger only when the right words we uttered against noises, and distance etc. as more data was gathered and trained the precisions of the model increased. 4. Result – After about 4-5 rounds of improvements the final model was integrated with the other modules
While speaking a new language, learners have difficulty recalling words and the right grammar. Which means that Poli would need to extract meaning from users phrases by identifying which language the switched words belonged to, what parts-of-speech they could be, and match them with the right words, or grammar. I tinkered with some NLP technologies to extract information like which language words in a single utterance belonged to, what parts of speech they were, and names of people and places. These experiments were by no means comprehensive to training a robust code-switching model and would require more substantial quantities of data and understanding. However, they allowed me to think about the possibilities of how Poli could help with learning. For example, in Hindi, nouns have gender, which affects other parts of speech and grammar. It takes a while to develop the innate knowledge of how to address objects with the right gender, and this could help people learn those structures.
The working prototype can understand Hindi-English mixed speech utterances. The utterances are converted to text, and depending on the topic that Poli provides, it can see if the words and phrases match the right variations provided within the topic. If there are mismatches, then it gives corrections and encourages the learner to repeat their lines until they match correctly.
This project has been an attempt at humanising technology in more than one way. Through my research and tinkering, I realised the importance of culture in shaping our interactions with Intelligent machines. I have come to believe that if we are to make more inclusive AI, we must understand the cultures from which we come.
Interviews with self-learners — They felt apps and aides were great to get a basic grasp of the language, but a lot more practice was required to talk fluently. They didn’t feel confident and hesitated to speak and keep up with the pace of native speakers. They then start to forget what they had learned or lost interest. They also lacked the cultural and conversational context to apply what they have learnt.
What if self-learners had a language buddy that could help them apply their knowledge of the language they are learning? What if that buddy could provide them with cultural contexts around its usage? Poli’s goal is make you do the talking.
Creating content and interactions through role playing — Held role-playing exercises online, to craft the details of the interactions. I used the working prototype as a probe to spark ideas for content and imagine how Poli could respond to different scenarios.
This project is an investigation into how we change the way we speak when addressing different audiences, how we intuitively blend languages when we talk to each other, and don’t necessarily treat them as discrete entities. In linguistics, this is called code-switching. Code-switching is used in informal everyday conversation and happens so effortlessly that it is often not even noticed by speakers.
People code-switch to fit in amongst a particular community—to show that we can speak their language and share their identity and culture. It is used heavily by advertisers, comedians and politicians to connect with their audiences. Code-switching is used as a tool for language acquisition. We do it when we cannot recall a particular word, phrase or sentence, and replace it with a word from the language we know. Code-switching is often used by teachers in their classrooms to build better relationships with students, translate new words and explain complicated ideas. What about people trying to learn languages on their own?
Code-switching is entrenched in people's identities. It can be an effective strategy to help learn the new language through conversation, and can also provide new cultural contexts for learning it; bringing a level of familiarity to our interactions with voice assistants.