IBM Develops Virtual Deaf Interpreter

You can read the full news from AP, here.

In a few words, pun not intended, IBM is using a computer to translate human speech, such as spoken English... into sign language (in this case, the British Sign Language, since this system was designed in England).

The system uses speech recognition to translate speech into words. (something IBM has pioneered for years, and if you've been working with disabled persons, maybe you've been aware of their speech recognition software).

From there, the system translates words into British Sign Language (I imagine that other sign languages, like ASL or at home, LSQ (Langue Signée du Québec) could be devised easily). The system displays an avatar and of course, hands, signings those words that have been heard by the computer.

The system is called SiSi (Say It, Sign It). You can watch a rather short demo on YouTube.

They say the system can be used when a sign interpreter isn't available, or when some sensitive issues are being discussed.

It is an interesting news in many many ways, since I am learning Sign Language, and as a computer programmer, I can imagine the hurdles a computer has to go through. Finally, during my years at the university, I was interested by the use of computers as tools, to compensate for our inability to do various things. When you're deaf, communication with hearing persons can be a major hurdle.

Translation. Translating from one language to another. Such a formidable dream and this is what is all about.

In the 60s, in the middle of the Cold War, some high rank officials in the US had wished of having on their desk... the Pravda... "automagically" translated in English.

Computers are known to do magical stuff... So IBM and others have worked on the idea and have continued to do so today.

Technology made it possible to have powerful computers in your hands. Earlier this year, there was this press release about IBM's portable MASTOR system. (MASTOR, Multilingual Automatic Speech TranslatOR).

So it is a logical step forward to attempt to translate a spoken language (such as English) to a visual language, such as Sign Language.

Technology has improved tremendously for the past 40 years, however, we're at a stand still, software-wise.

The problem is that computers are dumb devices. Even with the best translation software, they still don't understand the phrases they translate. Basically, it's the dictionary approach, translating one word at a time.

For instance, the word "lead". Is it the verb... to lead? Or the noun, lead? Should I say "leed"? or "led" ? A computer wouldn't know. This can ahem... lead to some interesting results. :)

Speech to text (and vice-versa) have been also major hurdles to computers. A computer is a digital device. It can handle a bunch of 1s and 0s. However we live in an analog world. That's why computers speak funny and also why computers are quite deaf. The richness of the human speech can not be brought down easily in a comprehensible form to a computer, in terms of 1s and 0s.

Finally, like any language, there isn't a direct 1-on-1 translation between English and... ASL, for instance. ASL is a *visual* language, therefore we're dealing with ideas and concepts... and not with words.

An example.: The verb "to be" isn't used in ASL. It doesn't mean that the function of the verb isn't carried out, it's just that in a visual world, you don't need to sign a verb about being.

"I am fine" is signed as "I, FINE", or sometimes as "I FINE, I". (In spoken French in Québec, we sometimes repeat the pronoun. "Tu m'aimes-tu" comes to my mind. If you know ASL, you aren't surprised of the French influence. End of digression. :) ).

Remember that famous quote: "I think, therefore I AM" ? Same idea. Since I am signing and it's about *me*, the function of the verb "to be" is implicitly being carried out while I'm signing.

I'm signing, therefore I AM ! As simple as that. Voilà! :)

___

So what this news tells me? First, I would say that human interpreters will still have a job, and the day you'll sign on a video relay service (such as Sorenson VRS) and being greeted by an avatar won't happen anytime soon!

However, I think this technology would enable more hearing people to communicate with deaf people.

A system like SiSi, on a portable device (such as the MASTOR), I'm thinking of people in authority (policemen, firemen, hospital staff, etc) when there's an URGENT need to communicate, it might help.

However it's usefulness... you know, despite all that science, all that technology, sometimes it's difficult to beat... the humble pen and a piece of paper. :)

Cheers!

-E