(Reprinted please indicate the source)

Use SDK: Kinect for Windows SDK v2.0 public preview

This time we discuss speech recognition. Before that, I wrote two parts in a section. This time, on the contrary, I divided them into two parts.

Speech recognition is one of the reasons for using the official SDK. Otherwise, OpenNI is used. After all, Microsoft's achievements on SR (Speech Recognition) are good.

First, you need to download voice recognition SDK,Runtime library And what you want to support Runtime language SR stands for speech recognition in the runtime language.

TTS stands for text to voice, depending on your choice. I chose to download American English and Mainland Chinese. Of course, pit dad's Microsoft is also dedicated to Kinect

Get ready Kinect Runtime Language There are fewer supporting languages, but even Japanese does not support Chinese.

Maybe it's just Japanese pronunciation.

If you use VS Express as I do, please download it again. WDK7.1 This voice library needs a little bit of ATL

Keeping (apparently COM smart pointer), Hangdad's Microsoft doesn't have ATL Library in Express. To get legal ATL library, you need to download this.

WDK 7.1, which has a legitimate ALT library. I install it directly to C disk. Please include the directories of these three libraries in the project.

Voice Platform, WDK, Kinect

If you have your own needs, please include other libraries. It's better to put the voice platform in the first place.

At present, speech recognition should only recognize PCM coding, and the processed data is floating-point coding. We can not directly use IStream provided by SDK by default.

Yes, based on the commonality of COM components, we only need to inherit IStream to implement a Stream class to complete audio stream processing.

This class mainly implements Read method. Seek method is important in S R, but Kinect does not support it. This method can return S_OK directly.

There are two ways to implement this class. One is based on the audio frames mentioned in the previous section. Use a larger buffer (preferably circular in design)

Get the audio frame and write the data in it. Write the data in Read, which is more troublesome. Another is the example provided by SDK.

Puppet tactics.

Get the default IStream provided by SDK, Read the puppet when you read, and convert the data after you get it. The code is as follows:

1. //Implementation of IStream Read Method  
2. STDMETHODIMP KinectAudioStreamWrapper::Read(void *pBuffer, ULONG cbBuffer, ULONG *pcbRead){  
3.     //Parametric checking  
4.     if (!pBuffer || !pcbRead) return E_INVALIDARG;  
5.     //Return S_OK without using m_SpeechActive before reading  
6.     if (!m_SpeechActive){  
7.         *pcbRead = cbBuffer;  
8.         return S_OK;  
9.     }  
10.     HRESULT hr = S_OK;  
11.     //The goal is to convert floating-point coding into 16-bit PCM coding  
12.     INT16* const p16Buffer = reinterpret_cast<INT16*>(pBuffer);  
13.     //Length multiples  
14.     const int multiple = sizeof(float) / sizeof(INT16);  
15.     //Check that the buffer is released sufficiently  
16.     auto float_buffer_size = cbBuffer / multiple;  
17.     if (float_buffer_size > m_uFloatBuferSize){  
18.         //Re-apply for memory if not enough  
19.         m_uFloatBuferSize = float_buffer_size;  
20.         if (m_pFloatBuffer) delete[]m_pFloatBuffer;  
21.         m_pFloatBuffer = new float[m_uFloatBuferSize];  
22.     }  
23.     //Buffer write progress in bytes  
24.     BYTE* pWriteProgress = reinterpret_cast<BYTE*>(m_pFloatBuffer);  
25.     //Current readout  
26.     ULONG bytesRead = 0;  
27.     //Need to read  
28.     ULONG bytesNeed = cbBuffer * multiple;  
29.     //Circular reading  
30.     while (true){  
31.         //Where voice is no longer required  
32.         if (!m_SpeechActive){  
33.             *pcbRead = cbBuffer;  
34.             hr = S_OK;  
35.             break;  
36.         }  
37.         //Getting data from packaging objects  
38.         hr = m_p32BitAudio->Read(pWriteProgress, bytesNeed, &bytesRead);  
39.         bytesNeed -= bytesRead;  
40.         pWriteProgress += bytesRead;  
41.         //Check for adequacy  
42.         if (!bytesNeed){  
43.             *pcbRead = cbBuffer;  
44.             break;  
45.         }  
46.         //Otherwise, sleep for a period of time.  
47.         Sleep(20);  
48.     }  
49.     //Data processing float-> 16bit PCM  
50.     if (!bytesNeed){  
51.         for (UINT i = 0; i < cbBuffer / multiple; i++) {  
52.             float sample = m_pFloatBuffer[i];  
53.             //Interval guarantees  
54.             //sample = max(min(sample, 1.f), -1.f);  
55.             if (sample > 1.f) sample = 1.f;  
56.             if (sample < -1.f) sample = -1.f;  
57.             //Data conversion  
58.             float sampleScaled = sample * (float)SHRT_MAX;  
59.             p16Buffer[i] = (sampleScaled > 0.f) ? (INT16)(sampleScaled + 0.5f) : (INT16)(sampleScaled - 0.5f);  
60.         }  
61.     }  
62.     return hr;  
63. }  

Note that when the speech recognition engine is initialized, it needs to acquire certain audio data to complete. It may take several seconds or even more to initialize, which is very painful.

I don't know what Microsoft thinks. To this end, we add a variable to indicate whether the SR is active or not, and return false data to deceive the SR engine when it is not activated.

This eliminates the need to wait for initialization.

The way we use speech recognition here is to load static. SRGS Grammar files, of course, can be loaded into dynamic grammar, but please refer to the official documents for details.

As for the SRGS grammar, you can look at the W3C documents, you can also see Microsoft Documentation.

Okay, let's write a SRGS grammar file. SRGS is based on xml, so use XML for suffix names. I use Chinese here.

After all, Chinese has a great advantage in word processing. A few words can represent a sentence. Through my test, Chinese can recognize English, but the recognition rate

1. <?xml version="1.0" encoding="UTF-8" ?>  
2. <grammar version="1.0" xml:lang="zh-CN" mode="voice" root="Root speech" xmlns="http://www.w3.org/2001/06/grammar" tag-format="semantics/1.0">  
3.     <rule id="Root speech" scope="public">  
4.         <one-of>  
5.             <item>Forward</item>  
6.         </one-of>  
7.         <one-of>  
8.             <item>Back off</item>  
9.         </one-of>  
10.     </rule>  
11. </grammar>  

This is almost the simplest SRGS file, the important ones are:

At the beginning, the standard of xml is utf8.

The grammar tag is the main tag, in which the xml:lang attribute chooses your language, and the mainland Chinese is zh-CN. And root, the name of the root tag.

The SR engine recognizes the basic phrases from here, and the rest can be captured.

Each rule is labeled with rule tag, one-of tag is used for multiple choices, item is used for basic phrase, rule ref is used for rule reference, and string or tag tag is used for tag tag.

js-like scripts are marked with the data they want, as well as token tags, which allow you to view documents for understanding.

Our goal is to achieve the following phrases:

1. I found ZZ AA BB in XX YY

For example, I found two (ZZ) high explosive armour-piercing bombs (BB) on my foot (XX).

XX means location. This information is very important.

YY denotes relative orientation. This information is not important.

ZZ denotes quantity. This information is important.

AA denotes quantifiers, almost useless

BB stands for items. This information is very important.

1. <rule id="Find things">  
2.     <example> I found two armour-piercing bullets on my feet </example>  
3.   
4.     <item>I am here</item>  
5.     <ruleref uri="#Location "/>  
6.     <ruleref uri="#Relative position/>  
7.     <item>find</item>  
8.     <item repeat="0-1">了</item>  
9.     <ruleref uri="#Quantity "/>  
10.     <ruleref uri="#Quantifier "/>  
11.     <ruleref uri="#Target object"/>  
12. </rule>  
13.   
14. <rule id="place">  
15.     <example> foot </example>  
16.     <example> House </example>  
17.   
18.     <one-of>  
19.         <item>foot</item>  
20.         <item>House</item>  
21.         <item>ship</item>  
22.         <item>head</item>  
23.     </one-of>  
24. </rule>  
25.   
26. <rule id="relative position">  
27.     <example> upper </example>  
28.   
29.     <one-of>  
30.         <item>upper</item>  
31.         <item>Above</item>  
32.         <item>inside</item>  
33.         <item>Side</item>  
34.         <item>nearby</item>  
35.     </one-of>  
36. </rule>  
37.   
38. <rule id="Number">  
39.     <example> Two </example>  
40.   
41.     <one-of>  
42.         <item>One</item>  
43.         <item>Two</item>  
44.         <item>Two</item>  
45.         <item>Three</item>  
46.         <item>Four</item>  
47.         <item>Five</item>  
48.         <item>Six</item>  
49.         <item>Seven</item>  
50.         <item>Eight</item>  
51.         <item>Nine</item>  
52.         <item>Ten</item>  
53.     </one-of>  
54. </rule>  
55.   
56. <rule id="Classifier">  
57.     <example> Mei </example>  
58.   
59.     <one-of>  
60.         <item>Mei</item>  
61.         <item>individual</item>  
62.         <item>block</item>  
63.         <item>slice</item>  
64.         <item>Vehicle</item>  
65.         <item>frame</item>  
66.         <item>second</item>  
67.         <item>ministry</item>  
68.         <item>platform</item>  
69.         <item>hold</item>  
70.     </one-of>  
71. </rule>  
72.   
73. <rule id="Target object">  
74.     <example> Tank </example>  
75.   
76.     <one-of>  
77.         <item>High Explosive Armor Piercing Bomb</item>  
78.         <item>Armor piercing shell</item>  
79.         <item>Tank</item>  
80.         <item>pencil</item>  
81.         <item>Computer</item>  
82.         <item>Apple</item>  
83.         <item>Hammer</item>  
84.         <item>Mobile phone</item>  
85.         <item>Armstrong Cyclotron Accelerated Jet Armstrong Gun</item>  
86.     </one-of>  
87. </rule>  

This part is almost complete, and then updated in the "Root Language":

1. <rule id="Root speech" scope="public">  
2.     <one-of>  
3.         <item>  
4.             <ruleref uri="#Find something./>  
5.         </item>  
6.     </one-of>  
7. </rule>  

If we add another one, "war situation"

1. <rule id="Root speech" scope="public">  
2.     <one-of>  
3.         <item>  
4.             <ruleref uri="#Find something./>  
5.         </item>  
6.         <item>  
7.             <ruleref uri="#War situation "/>  
8.         </item>  
9.     </one-of>  
10. </rule>  

1. </pre><p></p><pre>  

That's it.

In practical use, we will encounter many synonyms, or choose many branches. If comparing recognized strings is laborious, we can use tag tags.

So in "quantity" you can write: out = 10; you can export 10,

1. <rule id="Number">  
2.         <example> Two </example>  
3.   
4.         <one-of>  
5.             <item>One<tag>out=1;</tag></item>  
6.             <item>Two<tag>out=2;</tag></item>  
7.             <item>Two<tag>out=2;</tag></item>  
8.             <item>Three<tag>out=3;</tag></item>  
9.             <item>Four<tag>out=4;</tag></item>  
10.             <item>Five<tag>out=5;</tag></item>  
11.             <item>Six<tag>out=6;</tag></item>  
12.             <item>Seven<tag>out=7;</tag></item>  
13.             <item>Eight<tag>out=8;</tag></item>  
14.             <item>Nine<tag>out=9;</tag></item>  
15.             <item>Ten<tag>out=10;</tag></item>  
16.         </one-of>  
17.     </rule>  

Strings can also be used. But data processing is digital, isn't it?

Similarly, we update Discovery to:

1. <rule id="Find things">  
2.     <example> I found two armour-piercing bullets on my feet </example>  
3.   
4.     <item>I am here</item>  
5.     <ruleref uri="#Location "/>  
6.     <tag> out.place = rules.place; </tag>  
7.     <ruleref uri="#Relative position/>  
8.     <item>find</item>  
9.     <item repeat="0-1">了</item>  
10.     <ruleref uri="#Quantity "/>  
11.     <tag> out.Number = rules.Number; </tag>  
12.     <ruleref uri="#Quantifier "/>  
13.     <ruleref uri="#Target object"/>  
14.     <tag> out.object = rules.Target object; </tag>  
15. </rule>  

After all, it's a script that you can read. out.A = rules.B, AB may not be the same, but B must be the same as the id quoted earlier.

Now let's add a phrase: "war situation" such as

We destroyed enemy toilets

It's also very simple. I'll put the code right here.

1. <?xml version="1.0" encoding="UTF-8" ?>  
2. <grammar version="1.0" xml:lang="zh-CN" mode="voice" root="Root speech" xmlns="http://www.w3.org/2001/06/grammar" tag-format="semantics/1.0">  
3.     <rule id="Root speech" scope="public">  
4.         <one-of>  
5.             <item>  
6.                 <ruleref uri="#Find something./>  
7.                 <tag> out.Find things = rules.Find things; </tag>  
8.             </item>  
9.             <item>  
10.                 <ruleref uri="#War situation "/>  
11.                 <tag> out.War situation = rules.War situation; </tag>  
12.             </item>  
13.         </one-of>  
14.     </rule>  
15.   
16.     <rule id="War situation">  
17.         <example> We destroyed enemy toilets </example>  
18.         <example> They pierced our armor. </example>  
19.           
20.         <ruleref uri="#Character Object/>  
21.         <tag> out.subject = rules.Character object; </tag>  
22.         <ruleref uri="#The war verb "/>  
23.         <tag> out.Predicate = rules.War verb; </tag>  
24.         <item repeat="0-1">了</item>  
25.         <ruleref uri="#Character Object/>  
26.         <tag> out.object = rules.Character object; </tag>  
27.         <item repeat="0-1">Of</item>  
28.         <ruleref uri="#The term "war situation"/>  
29.         <tag> out.object = rules.War condition NOUN; </tag>  
30.     </rule>  
31.       
32.     <rule id="Character object">  
33.         <example> We </example>  
34.         <example> they </example>  
35.   
36.         <one-of>  
37.             <item>We<tag>out=0;</tag></item>  
38.             <item>they<tag>out=1;</tag></item>  
39.             <item>We<tag>out=0;</tag></item>  
40.             <item>enemy<tag>out=1;</tag></item>  
41.         </one-of>  
42.     </rule>  
43.       
44.     <rule id="War verb">  
45.         <example> Destroy </example>  
46.         <example> beat </example>  
47.   
48.         <one-of>  
49.             <item>Destroy<tag>out=0;</tag></item>  
50.             <item>beat<tag>out=1;</tag></item>  
51.             <item>breakdown<tag>out=2;</tag></item>  
52.         </one-of>  
53.     </rule>  
54.       
55.     <rule id="War condition NOUN">  
56.         <example> armor </example>  
57.         <example> Computer </example>  
58.   
59.         <one-of>  
60.             <item>armor<tag>out=0;</tag></item>  
61.             <item>Toilet<tag>out=1;</tag></item>  
62.             <item>Computer<tag>out=2;</tag></item>  
63.             <item>Computer<tag>out=2;</tag></item>  
64.             <item>Nuclear Silo<tag>out=3;</tag></item>  
65.         </one-of>  
66.     </rule>  
67.   
68.     <rule id="Find things">  
69.         <example> I found two armour-piercing bullets on my feet </example>  
70.   
71.         <item>I am here</item>  
72.         <ruleref uri="#Location "/>  
73.         <tag> out.place = rules.place; </tag>  
74.         <ruleref uri="#Relative position/>  
75.         <item>find</item>  
76.         <item repeat="0-1">了</item>  
77.         <ruleref uri="#Quantity "/>  
78.         <tag> out.Number = rules.Number; </tag>  
79.         <ruleref uri="#Quantifier "/>  
80.         <ruleref uri="#Target object"/>  
81.         <tag> out.object = rules.Target object; </tag>  
82.     </rule>  
83.   
84.     <rule id="place">  
85.         <example> foot </example>  
86.         <example> House </example>  
87.   
88.         <one-of>  
89.             <item>foot<tag>out=0;</tag></item>  
90.             <item>House<tag>out=1;</tag></item>  
91.             <item>ship<tag>out=2;</tag></item>  
92.             <item>head<tag>out=3;</tag></item>  
93.         </one-of>  
94.     </rule>  
95.   
96.     <rule id="relative position">  
97.         <example> upper </example>  
98.   
99.         <one-of>  
100.             <item>upper</item>  
101.             <item>Above</item>  
102.             <item>inside</item>  
103.             <item>Side</item>  
104.             <item>nearby</item>  
105.         </one-of>  
106.     </rule>  
107.       
108.     <rule id="Number">  
109.         <example> Two </example>  
110.   
111.         <one-of>  
112.             <item>One<tag>out=1;</tag></item>  
113.             <item>Two<tag>out=2;</tag></item>  
114.             <item>Two<tag>out=2;</tag></item>  
115.             <item>Three<tag>out=3;</tag></item>  
116.             <item>Four<tag>out=4;</tag></item>  
117.             <item>Five<tag>out=5;</tag></item>  
118.             <item>Six<tag>out=6;</tag></item>  
119.             <item>Seven<tag>out=7;</tag></item>  
120.             <item>Eight<tag>out=8;</tag></item>  
121.             <item>Nine<tag>out=9;</tag></item>  
122.             <item>Ten<tag>out=10;</tag></item>  
123.         </one-of>  
124.     </rule>  
125.   
126.     <rule id="Classifier">  
127.         <example> Mei </example>  
128.   
129.         <one-of>  
130.             <item>Mei</item>  
131.             <item>individual</item>  
132.             <item>block</item>  
133.             <item>slice</item>  
134.             <item>Vehicle</item>  
135.             <item>frame</item>  
136.             <item>second</item>  
137.             <item>ministry</item>  
138.             <item>platform</item>  
139.             <item>hold</item>  
140.         </one-of>  
141.     </rule>  
142.       
143.     <rule id="Target object">  
144.         <example> Tank </example>  
145.   
146.         <one-of>  
147.             <item>High Explosive Armor Piercing Bomb<tag>out=0;</tag></item>  
148.             <item>Armor piercing shell<tag>out=1;</tag></item>  
149.             <item>Tank<tag>out=2;</tag></item>  
150.             <item>pencil<tag>out=3;</tag></item>  
151.             <item>Computer<tag>out=4;</tag></item>  
152.             <item>Apple<tag>out=5;</tag></item>  
153.             <item>Hammer<tag>out=6;</tag></item>  
154.             <item>Mobile phone<tag>out=7;</tag></item>  
155.             <item>Armstrong Cyclotron Accelerated Jet Armstrong Gun<tag>out=8;</tag></item>  
156.         </one-of>  
157.     </rule>  
158. </grammar>  

That's all for this time. This SRGS example is expected to help you. Let's write some C++ code in the next section.