As of today natural language processing has taken various shapes and forms as a frequently used buzz word!
Before going into all the advantages or the 'hype' associated with it, let's talk about something 'core' - the framework or the main pieces involved here and how these work together.
The items that I am using for discussion here are mostly from leading players in conversational realm(like google, amazon, apple, df, microsoft & so on) - some pieces might have a different names but the idea is the same.
What is natural language processing?
Basically in simple words - an interface to which you feed in input in any given language which gets interpreted and processed to give back an intent or set of intents.
The language in natural form is called as 'utterance' and after processing, where most of the magic happens get transformed to intent or purpose.
There can be some processing applied to an 'utterance' before it is being submitted into the 'magic box' like spell checking if initiated from a chatbot or checking for commonly misused words which can lead to errors, early error detection.
Once the input is fed into the 'magic box' the result is compared with a given 'confidence factor' which can shift based on the maturity of the associated realm of context, if the result is within the confidence factor - well - the 'magic' worked and an intent is resolved. If the 'confidence factor' was too low then a 'fallback' intent is triggered.
The key here is to
- A) understand how the 'magic box' works to interpret the utterances to intents.
- B) how can fallbacks be shaped into known intents during the course of time.
The part A) - is what is called as 'machine learning' - which can have any known learning engine to process the data to apply the algorithm or set of algorithms to get the output.
Some of these input factors work backwards like setting up 'entities' which can have synonyms or know inputs - which can give exact language parse mapping, but the core piece here is does the system posses intelligence?
If I feed the system with C & D today and tell it the formula to compute E, if there is a variation is the formula - will the system adjust and recognise the variation to process E correctly, that's the key - for the learning factor over a period of time.
How to do this? - unsupervised, supervised or re-inforcement - there are a number of ways.
part B) - is what is important in terms of identifying if what I am asking for is it
- i) too complex? ii) in a different format? iii) or something with doesn't make sense.
most of the times it's i) or ii) but the process of getting there involves supervised learning and setting up the right labelling so that the system can recognise over a period of time how this works to make sense.
What tools to use? there are many - why not start with basic analytic tools and start working backwards.
This is just a very basic core framework - more specialised forms may include 'intent forecasting' - 'behaviour forecasting' and 'threat forecasting' using the core framework.
(For more details please refer to - google, microsoft or amazon conversational flow documentation)
In next post.. we'll try to cover another important topic ... 'data labelling'.
Before going into all the advantages or the 'hype' associated with it, let's talk about something 'core' - the framework or the main pieces involved here and how these work together.
The items that I am using for discussion here are mostly from leading players in conversational realm(like google, amazon, apple, df, microsoft & so on) - some pieces might have a different names but the idea is the same.
What is natural language processing?
Basically in simple words - an interface to which you feed in input in any given language which gets interpreted and processed to give back an intent or set of intents.
The language in natural form is called as 'utterance' and after processing, where most of the magic happens get transformed to intent or purpose.
There can be some processing applied to an 'utterance' before it is being submitted into the 'magic box' like spell checking if initiated from a chatbot or checking for commonly misused words which can lead to errors, early error detection.
Once the input is fed into the 'magic box' the result is compared with a given 'confidence factor' which can shift based on the maturity of the associated realm of context, if the result is within the confidence factor - well - the 'magic' worked and an intent is resolved. If the 'confidence factor' was too low then a 'fallback' intent is triggered.
The key here is to
- A) understand how the 'magic box' works to interpret the utterances to intents.
- B) how can fallbacks be shaped into known intents during the course of time.
The part A) - is what is called as 'machine learning' - which can have any known learning engine to process the data to apply the algorithm or set of algorithms to get the output.
Some of these input factors work backwards like setting up 'entities' which can have synonyms or know inputs - which can give exact language parse mapping, but the core piece here is does the system posses intelligence?
If I feed the system with C & D today and tell it the formula to compute E, if there is a variation is the formula - will the system adjust and recognise the variation to process E correctly, that's the key - for the learning factor over a period of time.
How to do this? - unsupervised, supervised or re-inforcement - there are a number of ways.
part B) - is what is important in terms of identifying if what I am asking for is it
- i) too complex? ii) in a different format? iii) or something with doesn't make sense.
most of the times it's i) or ii) but the process of getting there involves supervised learning and setting up the right labelling so that the system can recognise over a period of time how this works to make sense.
What tools to use? there are many - why not start with basic analytic tools and start working backwards.
This is just a very basic core framework - more specialised forms may include 'intent forecasting' - 'behaviour forecasting' and 'threat forecasting' using the core framework.
(For more details please refer to - google, microsoft or amazon conversational flow documentation)
In next post.. we'll try to cover another important topic ... 'data labelling'.
No comments:
Post a Comment