Engineering for Good: Software Engineer Joanna Zhang Discusses Technology to Help the Mobility Impaired

For the past few years, Bloomberg software engineer Joanna Zhang has been pursuing her passion project and developing Pallette, the world’s first open source tongue-computer interface. It enables people who are mobility impaired to use their tongue to interact wirelessly with their computers, tablets, and smartphones. Joanna’s effort is demonstrative of Bloomberg’s culture of philanthropy, in which employees give back where they live and work and use their expertise to create a lasting impact.

Joanna is part of the team which started the Pallette project during graduate school – she earned her Master of Science in Applied Information Science (Technion) and Master of Science Information Systems (Cornell) with a Concentration in Connective Media in 2016. While studying as part of the inaugural class at Cornell Tech in New York City, her professor tasked groups in a Human-Computer Interaction (HCI) class with developing new ways for people to interact with technology. Joanna’s group considered how people with disabilities might use technology. They observed that many assistive technologies were bulky around the body, and wanted to design a ‘hidden’ control interface that wouldn’t draw attention. They determined that one feasible way was to make use of people’s tongues. Leveraging this idea, they began designing and developing Pallette. Since graduating, Joanna has continued to work on this project with fellow classmate Daniel Levine, a biomechatronics researcher who recently graduated with a Master of Science in Media Arts and Sciences from the Tangible Media Group at MIT Media Lab.

On Thursday, October 3, 2019, Joanna will present her work on Pallette at the 2019 Grace Hopper Celebration of Women in Computing in Orlando, Florida. Our conversation with Joanna about engineering for good was edited for length and clarity.

How did you design the prototype of Pallette? Why did you continue working on this project after graduation?

For our first prototype, we retrofitted a red IBM TrackPoint pointing stick to a mouthguard and put the parts inside a waterproof plastic casing. It was surprisingly functional, but too bulky and impractical (wires were hanging out of our mouth), so we decided to design and build our own hardware. I have a background in electrical engineering and designed the circuit board. My classmates, who were also engineers, designed the outside casing and software. We had to learn a lot about product design.

We worked on this for more than two years during grad school. Since we had made significant progress and received great feedback from our professors and mentors, we didn’t want to abandon the project. We had also interviewed individuals with quadriplegia who were paralyzed from the neck down. Through a series of interview and feedback cycles, we understood that our technology actually had potential for practical use in paralyzed people’s lives. For example, they could conceivably use it to operate motorized doors, make phone calls, and serve as a text entry device for computers. Since quadriplegics are already burdened with medical expenses, we wanted to publish our technology as an open source project for everyone to use. That’s how our long-term effort began to develop a compact and versatile tongue-computer interface that anyone could make and use.

Daniel and I continue our work on releasing this technology as an open source project. We have incorporated user feedback and developed a new generation of prototypes and our website contains step-by-step instructions to help people manufacture and assemble the device. To increase awareness of the project, we entered the 2016 Hackaday Prize for Assistive Technology, where Pallette made it to the semifinals. Adafruit also wrote an article about our entry (I’m a big fan of Ladyada!).

Each device is personalized for the user and can be assembled with limited technical knowledge. The total cost of all the components is about $200, including the fitted mouthguard. This is relatively inexpensive compared to other assistive technologies. We are still looking for user feedback, and I’m presenting at Grace Hopper this year to promote the project in order to encourage others to get involved.

What were some of the challenges in building the device?

There were many challenges. We experimented with different touch technologies and embedding the electronics inside a retainer. We had to learn how to make the retainers and design the smallest possible circuit boards that could fit inside the retainer, while also providing good performance – there was a lot of trial and error and expense. We initially tried to publish our idea, but realized people couldn’t make their own retainers, and attaching the electronics to a mouthguard created significant mechanical challenges.

We also had to figure out how to detect tongue touches and movements. When the electronics were sealed inside a waterproof casing, the user’s tongue had to touch the board very hard for the tongue touches to register. Our next idea was to use infrared sensors. While this had surprisingly good performance, triangulating tongue touches required different configurations for the sensors’ positions on the board to precisely track tongue movements. Basically, every prototype had problems that required a technical solution.

The latest version of Pallette utilizes infrared sensors to track tongue movements and a microphone that registers the sound of gentle tongue taps. We’re still innovating and working on an easy way to replace the battery. We’d also like to use rechargeable batteries and a more energy-efficient circuit board, and the current prototype can still be made slimmer.

What interested you in this project? Why did you choose this?

I saw the social good in giving someone with a disability their independence by developing an assistive technology that’s private, discreet, and accessible. I realized the project’s impact when I met the users. They hadn’t provided feedback in the past about something designed for them to use and were quite eager to be part of the design process. We learned that people with disabilities are often disconnected from the technology that they use because they’re not included in its development.

Why should engineers think about doing philanthropic work when they can? What are the best ways to get involved in projects outside of work?

Why not? Daniel and I are creating a technology to help people live more independently, and there’s so much more we can achieve with our technical skills. I believe engineers have the abilities and responsibility to do something good. One reason I enjoy working at Bloomberg is that our company promotes philanthropy among its employees and has groups and socially-oriented projects that employees can volunteer to participate in.

Communities sometimes organize tech talks, meetups and hackathons, all of which are ways to learn about potential projects and their users. It’s important to feel connected with a project, so you are motivated to stay with it longer. This personal connection can be with the technical challenges, but it is more often with the people. As more people learn about different projects that they can connect with, I believe more people will become more likely to contribute. How much time one spends on a project depends on your goal, the timeline, and the community around it. When I’m excited about something, I can put in at least eight hours a week.

How has working on Pallette helped you with your work at Bloomberg?

When I’ve talked about this project with my team, my senior managers made suggestions about the product and who to reach out to for advice. Their encouragement was very inspiring, and that helped motivate me to do my job well too.

We build financial software at Bloomberg. Once we meet with clients and learn about how they use our software, we can better understand their pain points so we can do our jobs and improve their user experience. This inclusive problem-solving process is very similar to how we designed Palette – understanding our users and how they will use our technology helps us improve what we’re building for them.