Tracker quality - whitepaper

When transforming speech to text there are a lot of different factors that may affect the quality of the outcome of this transformation.  


Going into detail on this, is not the objective of this article, but to give you a general understanding we have listed some of the factors below, as well as a brief comment about each. 



The list below is neither exhaustive nor in any specific order but gives you insight into some of the external and internal factors. 


• Recording quality 

  Just like playing Mozart on a mobile phone makes it difficult to hear the nuances of the music, compared to listening on a proper set of hi-fi speakers, speech recognition software will struggle to hear the nuances and distinguish between words if the recording quality is poor.   

  In practice you can often identify poor quality when it sounds like your agents are speaking into a tin can, there are “fall outs” and words are cut off or muffled.   

  

• Cross talk

  This is only a problem for recordings made in mono, which is unfortunately still the majority. In mono both speakers share the same audio track, and thus words get mixed acoustically when both speakers talk at the same time. 
It is not a problem on stereo recordings, as each speaker is in separate tracks, and thus never “overlap”.    

  

• Noise 

  This could be all sorts of noise, like background talk, music, wind, traffic, heavy breathing etc. In an effort to mitigate the effect of the noise on the speech, we have an industrial research collaboration with Aalborg University where we are working on removing noise pollution on recordings.   

  

• Pronunciation 

  Given that Capturi’s main method for detecting a word is how that word “sounds”, the pronunciation is obviously highly important for the quality of the output. 


  • Incorrect pronunciation

    Often a problem with longer and more complex words, product names, words from foreign languages etc. A classic example is elderly people who have problems pronouncing modern product names.  

    

  • Dialects

    The same word can often be pronounced in various ways depending on the dialect. You probably know it from experience, that it may be difficult to understand people from other regions of your country. 

    While you may struggle to interpret what your uncle from the other side of the country is saying, Capturi can make up for this variation by adding multiple phonetic transcriptions of the same word. If you find that Capturi does not catch these variations, please collect examples and contact support. Then we will make sure to add these to the pronunciation dictionary. 

    

  • Mumbling

    Let’s just be honest - a lot of us mumble, speak too fast, too low, cut off words and sentences etc. All which adds to the difficulty of detecting what is actually being said.

    


• Mic placement

  We often experience that the sound from agents is quite low or fluctuate across different calls. This is often because they are not careful to place the boom correctly. The microphone should ideally be facing the mouth and placed around 2.5 cm (1 inch) away from the mouth. This will help reduce noise resulting from the microphone touching the face or breath noise.

What you can do to ensure optimal conditions for the speech analytics software:

• Instruct your agents to speak clearly. Not only is this highly important for your customers to understand what they are saying, it will also improve the tracker quality in Capturi.
  

• Get agents to repeat what customers are saying, if they can hear that the pronunciation is off. It helps alignment in the conversation as well as the output in Capturi.
  

• Reduce the noise level in the call center to minimise background noise. No customers like to be disturbed by background music or chatter from your colleagues anyway.  

  

• Make sure the agents use headsets with a boom, that the boom is placed correctly and that the headset settings are correctly configured. Some headsets even have settings that is optimised for speech analytics. 

  

• If possible, make sure to check that the recording setting on your phone system is set to “high”. Don’t compromise quality because the phone system wishes to compress the file as much as possible to reduce storage space. Get maximum quality and manage storage by reducing the time you keep the recordings instead. Capturi has no extra cost on storage, and it is much easier to access the recordings there anyway.

Speech recognizers utilize statistical language models to know which sequences of words are likely to occur and which are not – and should be trained on texts as close to the target domain as possible.

For instance, in a sales conversation, it is far more likely to encounter the words "the price is ..." compared to "the rice is ...".

For that reason, we’ve created our own proprietary models.

 Our existing language models are trained on three different corpora: 


1. A small corpus of highly curated text (KorpusDK from the Society for Danish Language and Literature)

2. A small corpus of conversation-like text (subtitles)

3. A huge corpus of non-curated text (Common Crawl).

We create a combined language model by interpolating the three corpora, using a development corpus of transcribed speech to determine the weights.

Every time a new customer joins Capturi, we do two preliminary things to ensure, that the foundation for delivering a good experience (quality) for that specific customer, is in place. 


1. We strengthen the Capturi language model by adding text from scraping the company’s website and potential other websites in the same sector. This improves the statistical language model’s ability to know which sequences of words are likely to occur and which are not, by adding material from the customers own domain. 


2. Upgrading the pronunciation dictionary. Add domain and customer specific words that may be unknown to the Capturi dictionary (which contains more than 75k active words already)
This is a manual process and requires linguistic knowledge to ensure that each word has the correct phoneme sequence. 

We can use precision and recall as a basis for discussing how you perceive the quality of your trackers, depending on the use case you are working on. Sometimes recall is the most important part, and other times you want to favour precision. 


Let’s take a few examples: 


1. Quality assurance 

   Let’s say you have a standard phrase that should always be mentioned in a given context, and you want to monitor how well the agents are at remembering this. In this case you will/should filter your data set to match the given context, and thus expect to get a hit on all the conversations you find. So, you will want to favour recall over precision, as you have narrowed down the data set significantly and expect a high hit percentage.  

   

2. Finding examples for review, training and giving feedback  

   In this case you often want high precision, and don’t necessarily care too much about recall. You just want to go straight to the examples and not waste time listening through false hits. 

   

3. Monitoring “behaviour” 

   This may be twofold:

   1. Monitoring expected good behaviour, for example the correct introduction phrase; “welcome to Capturi”. Here you want a high recall to catch as many examples as possible, as you expect a high hit percentage and thus the amount of false hits to be limited. 


   2. Monitoring unexpected and undesirable behaviour, for example inappropriate language. This must be expected to happen rarely, and you want to make sure that the hits you get are correct. Thus, you should favour precision over recall. 

So, based on these examples the common rule of thumb should be: 


1. If you expect a high volume of hits, you can accept lower precision and higher recall, as the number of false hits will be relatively limited both percentage and volume vice.
   


2. If you expect a low volume of hits, you want high precision, as favouring recall is more likely to result in a relatively large percentage (and possibly volume) of false hits.

When working with your own trackers you are, at some point, likely to experience four generic challenges:  


1. Too many hits - Recall is good but the precision is off - you get most of the right hits but also a lot of false hits.  


2. Too few hits - Precision if good but recall is too low - the hits are correct, but the system does not find enough of the possible hits. 


3. Correct hits - wrong context - Capturi finds the right words, but the context is wrong. 


4. The wrong hits - both precision and recall are off 


To mitigate these challenges there are several things to try. Some of which you can do yourself and others which the Capturi team can do.

Below is a list of the possibilities you/we have: