See an online demo of the SignBook project at this link! The demo works best in Google Chrome, only a subset of features will work in other browsers.
In case you missed it – read the story behind SignBook here from a previous post.
Last October 31st I sent in my application for an internship at the company I longed to work for – an application I thought and prayed about for months. It was my dream internship. I meant to apply earlier, but never felt like I was ready. And I still didn’t feel ready as I applied that day…my hands trembled as I clicked submit.
Last May 21st I boarded a plane headed to India. It was a place I had never been, on the opposite side of the world – a journey I had thought and prayed about for months. It would be my longest time ever abroad, 9 weeks. I had made many preparations, but never felt like I was ready. And I still didn’t feel ready as I boarded that plane…my hands trembled as I said goodbye to the place that was my home.
Back in October, three days after I had applied for my dream internship, I talked with the recruiter. I had done the first interview task quite well and really in fact, had enjoyed it. However, I told the recruiter that another opportunity had come up, an opportunity I couldn’t pass up.
Nine weeks after I boarded that plane, I was coming back home. As I saw the bustle of international faces I could think of nothing but the faces of the children, teachers, and close friends I was departing from. Instead of my dream internship, I had come to know and love a community in a place I had never heard of, and been able to leave a mark in that place through technology and my skills, but even more so through new connections of the heart.
I had said no to my dream internship to instead develop technology for the deaf and hard of hearing and blind in India. I thought I was crazy to do so. And that feeling sometimes persisted even while I was there. But I tell you now, it was worth it. It wasn’t easy – at any point – but it was worth it.
And as I write now – jet-lagged, sleep-deprived, with a cold and newly-diagnosed ear infection – I wish to share the story of this experience with you. It is not a simple story, nor is it short. But I hope it will be worth the read.
I will continue to introduce this story even further back in time to when I was born. I was born with 80-90 dB hearing loss in both ears, diagnosed as severe to profoundly deaf. Because infant hearing screenings were not conducted at birth back then, it was not discovered until I was 12 months old.
My mother, who had studied child and family development in college, knew how much this could set me back, and was worried about my development with this critical delay. She prayed that I would one day be able to be independent and support myself with a job. This was her dream for me in this place of worry and fear.
At one year old, I was fitted with two hearing aids and brought into a Birth-to-3 program at Milwaukee’s Center for Communication, Hearing, and Deafness (CCHD), the only program in the state with such assistance for my needs – which happened to be within 10 minutes of my house. There I had early intervention with speech therapy and being taught American Sign Language (ASL) to communicate. So I was raised with fluency in both ASL and spoken English through the excellent and loving people at CCHD as well as my family learning sign and using it with me at home, in addition to exposure to spoken English through my hearing aids.
I was “mainstreamed” into public school with the help of an FM microphone system that helped me understand what the teacher said, and more personally, a sign language interpreter in every class with me, to ensure I could pick up everything the teacher said when my hearing technology (often) wasn’t enough.
I continued speech therapy through my early years (much to my displeasure – it was one of the most boring and frustrating things I have ever had to do) and many people commented that they couldn’t tell I was hard of hearing.
When I was 16 years old, a family friend asked my mother if I had considered a cochlear implant. In Wisconsin, it is required for insurance to cover cochlear implant costs for children under 18 who would benefit from it. After some hearing tests, I was told there would be a benefit and went for it. A year after the surgery, the tests had shown that I gained a 25% increase in single-word comprehension (one of the toughest tests, since there is no context for the word like I would have in ordinary situations).
Many (if not all) of these accommodations throughout my life were made possible through the Americans with Disabilities Act. It was passed into law in 1990. I was born in 1993. Lucky.
Lucky. Lucky. I am nothing but fortunate to be where I am and grateful to my parent’s prayers and God in heaven to be where I am today.
Today I am at Carnegie Mellon University studying in both one of the leading Electrical and Computer Engineering programs in the nation as well as perhaps the most well-known Human-Computer Interaction program in the world. This not by my own merit, but I truly believe by the support of so many along the way.
Now I am able to see my disability as a blessing, for all the ways it has brought me into a more compassionate understanding of the world and how my struggles with being hard of hearing have cultivated a heart of gratitude towards God and other people.
And today, my mother’s dream has become reality and even more so than she ever thought possible.
This is the beginning of my story.
From the telling of my past, I have but one story of deafness. This summer, across the globe, I was able to meet and get to know deaf children in India and understand part of their story and their situation.
In India, (according to the World Health Organization) 6% of the population has “disabling hearing loss”, with most hearing loss and deafness in India coming from causes that would be preventable with proper treatment.
At the Mathru Center for the Deaf and Differently-Abled, this very likely applies, since most children come from rural family environments. That means many of the children have deafness caused by a high fever that was left untreated, or another preventable cause. It breaks my heart that many students have hearing loss that could have been prevented and likely also had little proper support and care after its onset. (And that there are many more unknown children out there who aren’t getting any care.)
Through the generosity of the Mathru Educational Trust and those who support it, the children at the Free School for the Hearing-Impaired (part of the Mathru Center for the Differently-Abled) are able to receive education, housing, food, and care for no cost to the family. It was already an inspiring place to be able to contribute with our team’s work.
The first thing I noticed at the Center was that my ASL was not understood, outside of fingerspelling English words. We asked about what sign language was used and were given a dictionary of Indian Sign Language, developed in the late 1970s. This was not quite what they used either and eventually we realized their sign language was a very local version, which we called “Kannada Sign Language” after the area’s locally spoken language. Basically, we had found no available reference or dictionary for the signs we were seeing, with barely a mention even online.
Where this became especially noticed was with the new teacher at the Center when we first arrived. She had to pick up the local sign language only from other teachers or asking the students for signs. This all while she was adjusting to teaching at a new school with the added difficulty of teaching children who could not hear, and thus who she could not communicate effectively with or verbally discipline as is necessary in the early grade levels of Kindergarten through 4th offered at the school.
What’s more, in our second week two other teachers left the Center, leaving two vacancies. There are only four teachers for the deaf and hard of hearing at the Center. This then left only one teacher who had been at the school for more than two weeks.
Observing further in the classroom, we noticed teachers explaining objects or concepts frequently through pictures. Because the teachers had a limited sign vocabulary to work with, they often showed concepts through hand-drawn diagrams and scenarios on the chalkboard or through many hand-made or store-bought flashcards or posters. In class, teachers went back and forth between showing a picture, a written word, and the sign for that word with the students.
By our fourth week there, there were three new teachers, excited to work with the children, but daunted at the amount of effort it takes with all the adjustment, learning sign language, and preparing these visually-based lesson plans.
Beyond just the atmosphere of the Mathru Center, there were many cultural forces against us and our work to help deaf children – including many cultural misconceptions about deafness. Through our time there and talking with different people in the school and outside, we discovered a conception of deafness in the general populace of India roughly similar to the following:
However, I present also here the corrected and informed views of deafness (even misunderstood in America by many!):
So while we were learning to understand Indian culture, we were also coming to understand the misconceptions around deafness in India. Using my knowledge as a hard of hearing individual and past education in ASL and Deaf Culture in the US, I was able to discover these misconceptions and speak with teachers and a reporter about a better understanding of deafness. These misconceptions were hard to hear but were important to know and inspired and fueled our work even more as people committed to helping this misunderstood people group.
From all these insights both about the Center and Indian cultural perceptions of deafness, we thought about how we could help with technology. We first knew that there was a very clear need for a way for new teachers to be quickly trained in learning the local sign language.
Our first step was to film and edit two videos to help new teachers: one of the Kannada alphabet in sign language (with 51 letters!) and another of commonly used phrases. We also posted the videos on YouTube, so that these basic resources would be freely available online about Kannada Sign Language.
But how to make a better form of a dictionary or reference for them? We certainly didn’t have time in 9 weeks to document an entire language. The solution then seemed to fall in the hands of the teachers to self-document their sign language through some kind of video-taking and saving reference tool.
So around our third and fourth week in India, we began looking at all the requirements and brainstorming ideas from our observations. We wanted a tool that would allow teachers to record videos of signs, categorize them into lessons and units, and teach and learn the signs from each other. With the visually-based teaching methodology, we also wanted abilities to take or upload pictures in the tool. This way the teachers could integrate this dictionary as a tool for each other to learn signs as well as a classroom tool featuring a simultaneous juxtaposition of signs, videos, and English words.
Alongside the cultural conceptions of deafness, there was also a greater hope in this project that it could combat misconceptions of deafness by showing the rich vocabulary of sign language and by providing access for sign language learning to more people. Through the spread of this tool, many others could potentially understand deafness better by learning the language of the deaf and being able to communicate with them, the first step to understanding.
So we had the basic ideas and vision. The next critical problem: how?
In choosing the technology for this sign dictionary creator, we thought initially tablets would be great – very portable and easy to use. However, the teachers were unfamiliar with tablets and preferred computers or laptops. And all were more likely to gain better technology literacy from using a keyboard and mouse interface that could transfer to other tasks with computers in the future.
Thus began the huge task of creating this application from scratch, an application that would feature video and picture capturing, picture uploading, and creation, editing, updating, deleting, and categorizing of dictionary entries. In addition to this work was my part in all the projects our team was doing at the blind school including fixing and updating the braille tutors there and training teachers and staff on useful technology and web tools.
Around the sixth of nine weeks in India, I was overwhelmed. We had seen a great opportunity and need for this technology, but would we be able to finish it in time?
I started to code everywhere – at both schools, at home, in the car – and slowly the features came around. But for every feature that was added – several others broke or created more programming bugs. It was three steps forward, two steps back.
Along the way, we showed the experienced teacher at the Center the video capture tool (as one of the first features that was developed) and she became familiar with it and could use it well. Later we showed how to create words and topics and organize the dictionary. Slowly the teachers and us all became more excited at the potential – but it was not yet fully usable, filled with bugs, potential data erasures, and frustrating glitches.
In the beginning of the summer, I had pictured a few weeks at the end of summer to test the application, see how it was used in class, and iterate on it with the suggestions of the teachers. Now I was simply asking myself – would I even be able to leave anything usable behind?
At the beginning of our final week in India, the ninth week, the application was finally coming together but still not quite there. Tensions and stress ran high on the team as we tackled many different objectives and final things to do in this week – four presentations, a 30-page report, wrapping up three projects including documentation, help guides, and training teachers for each.
And I constantly felt I was the most behind – my application was still on the edge of working, almost there, but brought down by critical bugs. Most of all, on the new laptops we had gotten donated for the Center, the video recording was far too laggy to use and made the teachers frustrated. I had three days left to leave whatever I would leave behind, could leave behind.
At this point, I thought of the children. I thought of the opportunities that I had growing up. I thought of all the knowledge I had been given through assistive technology and all the ways technology had benefitted me. I thought of this special opportunity to come to India. I thought of the leaders and donors of TechBridgeWorld who had made this trip and program possible for my teammates and I. I thought of the gracious hosts at Mathru, including Ms. Muktha who had given so much to welcome us and make us feel at home. I thought of the hard-working teachers we had gotten to know who give so much with the little resources they have to give these children an education. I thought of the children. By statistics alone I should be one of them – a deaf child born in a developing country, without access to all that could help them.
But I was here. I had been given all these things throughout my life, and was now potential access for them to some of what could help them. And as far as I was able, God-willing, I would not leave without leaving something helpful to them. Not for my own sake, but for those I had dearly in mind, who were the reason I was here.
With this, I put my headphones in, chewed gum, whatever I was able to do to stay up in the night and code through the remaining difficulties. I changed the video format to allow non-laggy recording. I found all the remaining errors in the code I could find. And with a little perseverance and a little prayer, I was able to bring the teachers something helpful the last two days at the Center. Or, that I hoped was helpful. How would the teachers respond to the final technology? This was the final test…
I brought the finished product, now named SignBook, to the Center on our final Wednesday with the teachers. I spent as much time as I was able showing the teachers SignBook, training them, and them letting them use it on their own to test their independence with the tool and add topics and words of their own!
“This is very good because I can learn on my own and be fine teaching to the children. If I ask [the other teacher] she will teach me after 4:00 PM, but now I can ask and she will enter in this. I will see again, repeated, how to do the sign – it is the easy way! I can easily see this and then teach to the children.”
After an hour of training one of the new teachers, I quickly jotted down this quote of hers. SignBook had captured her excitement and imagination.
“This is a most wonderful thing you have done. Now anyone can document their sign language, and even give a copy to others to learn! This has so much potential. Potential to provide access to a family, a community, to communicate with the deaf. I am proud of you. I am proud of your team.”
This quote, paraphrased from a visitor to the school who saw our team’s projects, voiced the early hope I had, the ambitious and far-reaching hope that this could provide greater access to the deaf, their language, and change perceptions of deafness in India.
My heart leapt at these words and the interactions with teachers and staff the last three days at Mathru. They had picked up using SignBook more quickly than I had thought possible and each voiced their excitement at its potential. There was even a moment when 20 children gathered around as I trained a teacher, all engaged with SignBook as I used it to test their knowledge of words the teachers had input. Their faces lit up as they burst with excitement to proudly show their sign language knowledge and see more of what was on SignBook. I had done something good. I had left something usable, and useful! I could breathe easy.
Make no mistake – this is merely one part of our team’s work, merely one aspect of our story. iSTEP 2015 made incredible impact with our short time in India. And I could not have done anything without Minnar, Maya, Amal. Though I may have written the code for this one project, there are innumerable ways this project and the total impact of our summer was critically made possible by the efforts of my teammates.
At the Mathru Center there are now two laptops, both with a copy of SignBook, which exists as a simple folder containing all the dictionary data and an application interface runnable with a double click. This folder can also be simply copied to another computer – able to be spread very easily on thumb drives, which leads to the feasibility of widely spreading knowledge of sign language among anyone with computer access. Since TechBridgeWorld continues a partnership with the Mathru schools there is also potential for future development and iteration upon this first prototype of SignBook.
To close this story, I am so grateful for this opportunity and for everything that led me here – not the least including my deafness. If there is anything I hope to leave you with, it is these words:
Sometimes we must say no to our dream in order to say yes to the dreams of those you can make possible.
We each have a unique and powerful role to play on this earth. With noticing eyes for those in need and love for those less fortunate than us, let us not neglect the impact we can make upon the lives of others through our God-given gifts. You may discover things even greater than your dreams can foretell.
We’ve gotten to create two vlogs of our experience so far, both showing tours through the beautiful Mathru schools and talking about our projects. In case you missed them on Facebook/YouTube – take a watch below!
A weekend ago we headed south to see temples, palaces, and more around the city of Mysore. Below are some moments from the trip, along with interesting tidbits of history and culture that we learned about! (All factual information is paraphrased from Wikipedia.)
Hope you enjoyed hearing of the historical and cultural adventure of the iSTEP team in Mysore!
Here’s an update (and demo!) of what I’ve been working on the past week or two. Prepare for GIF madness!
After a few visits at the Mathru Center for the Differently-Abled, we realized that their sign language is very unique and there is no reference or dictionary for the language. This makes it hard for new teachers to learn the sign language, which is especially important since there is a shortage of teachers trained in teaching the deaf and hard-of-hearing. Many teachers must pick up sign language on the job! This can be a very demanding and difficult task, which we’ve witnessed this summer with many new teachers just starting out at the Center.
This led to the project I have been mostly working on, which is creating a way for teachers and students to document their sign language to form a community dictionary reference. This reference is to be used at the school on laptops or in the computer lab through a locally hosted website (so they won’t need internet and can store the video files on their computers).
We envision several possible use cases (and shall see how the school chooses to use it):
Check out the demo using the link below! (Works only on Google Chrome and Firefox browsers.)
Here is the basic user experience to create a gif video (will be changed more specifically for the context of the sign language application):
I tested the video capture application a lot with my teammates, which has led to many bug fixes as well as a hilarious set of GIFs saved on my computer. Here’s one from when the power went out while we were testing.
We also tested it with a teacher, and so far it seems easy to use! Next week there will be an added interface for sorting words by category and browsing different signs.
Hope you could check out the demo and make some GIFs of your own! We’ll keep you posted on how the project develops.
Last Sunday, we traveled to Bangalore – here are some pics and a story.
I also had a fascinating conversation at the temple! Here is the story:
It started when Minnar saw a cookbook titled Beyond Vegetarian Cooking at the temple gift shop. Upon reaching for the book, a store attendant, who was also a temple pandit (a Hindu priest/scholar) came up and said, “Do you want to learn how to have a karma-free diet?” Curious about what that meant, she started flipping through the book. The man then asked, “Can I ask you a question?” Minnar looked at Maya and the guy seemed charismatic and friendly, so she said, “Sure!” The man then asked, “What is the goal of your life?”
“Erik…” Minnar then looked over to me, in a different part of the shop. “What was that?” I asked. The pandit repeated for me, “What is the goal of your life?” I paused nervously – we were still in the main large worship room of the temple. Then I said, “Well, I am a Christian so…” The pandit said, “You don’t have to worry, that’s okay here.” So I gave my Christian answer: “To love God, love other people, through Jesus Christ, and tell others about him so they can also be forgiven from sins and enter the kingdom of God.”
He then asked, “Do you think that Krishna and Christ could be the same?” I said I didn’t know about Krishna. Then he went through the difference between matter and energy and said my idea of soul was his idea of energy.
From here, I went all religious nerd and asked curious questions I had about Hindusim:
“When do you believe someone stops reincarnating?” He says when you come to love and serve God fully and you stop desiring the things of the world.
“How do you know when you love God enough?” He responded, “How do you know when you are full after eating? It is the same – you will certainly know.”
“Do you believe that there are people on this earth, who have stopped reincarnating, who are perfect?” He said yes. I talked about the Christian doctrine of sin and we all make mistakes – only Jesus was a perfect human. He asked “Do you believe Jesus is God’s son?” I said yes. He then asked, “Do you also believe he is God?” I said, yes I also believe that. It made me realize even more that there are things about any religion that take time and thought to understand!
Anyhow, that was super interesting to me – the chance to meet a devout Hindu pandit who was open to curious questions about Hinduism! I bought a copy of the Bhagavad Gita, which I have started reading to understand more about Hinduism. Very interesting to be immersed in a religiously different culture and learn about it at the same time!
That’s the glimpse of last weekend! This weekend was more laid back. We have found that the culture here is to take a “serious weekend” and really relax. So we have done the same (at least for this weekend) – a lot of reading, writing, cooking, talking. Hope you have a restful weekend as well!
A lot has happened in the first two days.
First off, we arrived safely! It was a long but overall pleasant journey with several airplane movies and many on-and-off naps.
After we got off the plane to Bangalore, we got our luggage, went through customs, met Ms. Muktha (the school founder and director), drove to Mathru School and arrived at our accommodations around 3 AM Sunday, and then quickly went to sleep.
Sunday morning I went to a local church via a motorcycle ride with a drive from the school through the streets of Yelahanka (the suburb we are in) – quickly discovering the traffic differences of India such as driving on the right, honks around every turn, and there being very few traffic rules except being careful.
The rest of Sunday was mostly resting and receiving delicious food from our wonderfully hospitable hosts at the Mathru School. Just around dusk I walked with Amal around the nearby neighborhood, through the park, and seeing what local shops there are.
We haven’t yet gotten internet connection set up at our residence, or even all phone service yet, but I’m actually glad for that. Sometimes in a new place, too much connectivity causes a lot of time spent looking back, missing the chances to look around at what’s right in front of you. And so far it’s been captivating.
From the more laid-back culture, to the respected cows roaming the streets, to the delicious food that will definitely develop my spice tolerance by the end of the summer – it’s been different, but good.
Our first day at the school working as a team went very well. We got a tour of the School for the Blind, talked to Ms. Muktha about our projects and plans, and met with some of the teachers to explain the braille tutor comparative study we will conduct.
Ms. Muktha and the teachers at the school are all very friendly and helpful. Their excitement about the braille tutor projects and others have served to affirm our interest and goals for a cooperative experience – us both learning from each other and helping each other.
It’s been a great start here in Bangalore! Stay tuned for more updates to come.