Speech to Sign mode

Posted by Vishwa Prabhathiya on Thursday, March 26, 2020 | No comments

According to our project there are main two objectives. There are:

• Sign to Text mode
• Speech to Sign mode

If we consider about Speech to Sign mode, process is; ordinary user should execute this part and then he can speak in Sinhala to his microphone. When he speaks, system captures his speech and convert it into Sinhala text and separate words of that speech. This part was done by my team member. Finally I get that converted Sinhala text and map that Sinhala text with pre-animated animations. As an example; if ordinary user says “ මම යනවා ”, then the system separated its words as “ මම “ , “ යනවා “. Next I map these Sinhala words with particular pre-animated sign. Then we can see it. 

If we go into deep, here I use ‘ blender 2.79 ’ for the whole process. By using ‘blender’, I can animate the character according to Sinhala sign gestures. After creating some animations for Sinhala sign gestures, then I take these animations’ start and end key frame values and also the animation name to a text file. By doing this I can easily map this text file content with separated words which I mention above. These processes are done by ‘Python’. Because ‘blender’ not only use for animating purposes but also use for scripting purposes.

Scripting is the major part of this objective because without doing scripting we can only do the animation part of the character. To map the separated Sinhala words with pre-animated gestures, get the start and end key frame values and animation name to a text file, execute the program with user keyboard inputs, I used ‘Python’. Most important thing is play animations according to user’s speech is done by scripting in blender.
Below video shows a working time of me. Here I create an animation and then show how the final output will look like. 

Note: video playing speed increased.