论文标题
使用自然语言和视觉语言的聋人多语言交流系统
Multilingual Communication System with Deaf Individuals Utilizing Natural and Visual Languages
论文作者
论文摘要
根据世界聋人联合会的说法,存在200多种符号语言。因此,了解聋人,甚至熟练的手语使用者,从而在聋人社区与社会其他成员之间遇到障碍,这是一项挑战。为了弥合这种语言障碍,我们提出了一个新颖的多语言通信系统,即Mugcat,以提高手语使用者的沟通效率。通过将公认的特定手势转换为表现力的图片,这是普遍的用法和语言独立性,我们的Mugcat系统大大帮助聋人传达了他们的思想。为了克服手语使用的局限性,这几乎不可能将其转化为普通人的完整句子,我们建议从不完整的手语语言翻译中重建有意义的句子。我们还测量了具有零散公认的手势的生成句子的语义相似性,以保持原始含义。实验结果表明,提出的系统可以实时工作,并从一些手语的手势中综合出精美的令人惊叹的插图和有意义的句子。这证明了我们的Mugcat有望协助聋人交流的潜力。
According to the World Federation of the Deaf, more than two hundred sign languages exist. Therefore, it is challenging to understand deaf individuals, even proficient sign language users, resulting in a barrier between the deaf community and the rest of society. To bridge this language barrier, we propose a novel multilingual communication system, namely MUGCAT, to improve the communication efficiency of sign language users. By converting recognized specific hand gestures into expressive pictures, which is universal usage and language independence, our MUGCAT system significantly helps deaf people convey their thoughts. To overcome the limitation of sign language usage, which is mostly impossible to translate into complete sentences for ordinary people, we propose to reconstruct meaningful sentences from the incomplete translation of sign language. We also measure the semantic similarity of generated sentences with fragmented recognized hand gestures to keep the original meaning. Experimental results show that the proposed system can work in a real-time manner and synthesize exquisite stunning illustrations and meaningful sentences from a few hand gestures of sign language. This proves that our MUGCAT has promising potential in assisting deaf communication.