Microsoft Cognitive Services (powered by Microsoft Research) make you free from the difficulties of intelligence algorithms, and everyone who is not the scientist can easily use this pre-built (ready-to-go) APIs for your application intelligence.

, but we focus on the language understanding (the linguistic computation including tagging, parsing, etc) in this post, and help you understand the clear point of each APIs.

The natural language processing itself covers speech recognition, translation, sentiment analysis, summarization (which is also possible in Microsoft Word), language generation, etc… Microsoft Cognitive Services is having a lot of APIs (over 20) and corresponding APIs.
But we focus on the language understanding (the linguistic computation including tagging, parsing, etc) in this post, and focus on the only 2 APIs, which are Linguistic Analysis API and Language Understanding Intelligent Service (LUIS).
I hope this helps you understand the clear point of language understanding in Microsoft Cognitive Services.

Notice : This post doesn’t explain about “how to use”, and please refer the official document (Linguistic Analysis API, LUIS) for the api usage.

Linguistic Analysis API

If you need the fundamental language parsing capabilities, you must use this API. The Linguistic Analysis API returns the parsing result as json from the original sentences.

Today you can use the following 3 types of linguistic analyzers.

Token Analyzer (PennTreebank3 – regexes)

This is the most simple analyzer which just separates the sentences and tokenizes.
The following is the tokenized result of the sample sentence “I want a sweet-smelling flower with a red flowerbot.”

This result is not including so much useful information, but you can use this analyzer with other analyzers (you can specify multiple analyzers in one API call), and you can get the additional tokenization information with other analyzer’s result.

{"analyzerId": "08ea174b-bfdb-4e64-987e-602f85da7f72","result": [
    {"Len": 52,"Offset": 0,"Tokens": [
        {"Len": 1,"NormalizedToken": "I","Offset": 0,"RawToken": "I"
        },
        {"Len": 4,"NormalizedToken": "want","Offset": 2,"RawToken": "want"
        },
        {"Len": 1,"NormalizedToken": "a","Offset": 7,"RawToken": "a"
        },
        {"Len": 14,"NormalizedToken": "sweet-smelling","Offset": 9,"RawToken": "sweet-smelling"
        },
        {"Len": 6,"NormalizedToken": "flower","Offset": 24,"RawToken": "flower"
        },
        {"Len": 4,"NormalizedToken": "with","Offset": 31,"RawToken": "with"
        },
        {"Len": 1,"NormalizedToken": "a","Offset": 36,"RawToken": "a"
        },
        {"Len": 3,"NormalizedToken": "red","Offset": 38,"RawToken": "red"
        },
        {"Len": 9,"NormalizedToken": "flowerbot","Offset": 42,"RawToken": "flowerbot"
        },
        {"Len": 1,"NormalizedToken": ".","Offset": 51,"RawToken": "."
        }
      ]
    }
  ]
}

The sentence is not only separated by white space or punctuation marks, but is tokenized by the context in the sentence. For example, it can correctly tokenize like “what’s your name ?” (which is equivalent to “what is your name ?”), “Mr. Williams” (which doesn’t mean the punctuation mark), and …

Part-of-speech (POS) Tagging Analyzer (PennTreebank3 – cmm)

If you want to retrieve some keywords and analyze the sentence, the identification (noun, verb, etc) is often needed. For example, if you want the emotional keywords and measure the emotion, the adjective should be the key word.
This POS tagging analyzer identifies these tags.

The following is the result of this analyzer, using the input sentence “I want a sweet-smelling flower with a red flowerbot.”

{"analyzerId": "4fa79af1-f22c-408d-98bb-b7d7aeef7f04","result": [
    ["PRP","VBP","DT","JJ","NN","IN","DT","JJ","NN","."
    ]
  ]
}

(See “Penn Treebank P.O.S. Tags – Penn (University of Pennsylvania)“)

The tag is not also distinguished only by words, but also the context is concerned.
For example, “dog” is usually the noun in the sentence, but it is used as “verb” in the following sentence. (I referred this example in “Wikipedia : Part-of-speech tagging“.)

“The sailor dogs the hatch.”

Constituency Forest Analyzer (PennTreebank3 – SplitMerge)

Assuming that you’re a shop owner of selling flowers in the internet and providing the intelligent search engine. Customers might input as follows.

“I want a red and sweet-smelling flower.”
“I want a sweet-smelling flower except for red flowers.”
“I want a sweet-smelling flower with a red flowerbot.”

You see that the each sentence is indicating the absolutely different items. But if you use POS tagging analyzer (previous one), the results doesn’t meet the needs.
In such a case, you must use the constituency forest analyzer.

The following is the result of this analyzer in the case of the sentence “I want a sweet-smelling flower with a red flowerbot.”

{"analyzerId": "22a6b758-420f-4745-8a3c-46835a67c0d2","result": ["(TOP (S (NP (PRP I)) (VP (VBP want) (NP (NP (DT a) (JJ sweet-smelling) (NN flower)) (PP (IN with) (NP (DT a) (JJ red) (NN flowerbot))))) (. .)))"
  ]
}

(See “Penn Treebank II Tags – MIT“)

As you see, the result expresses the tree of structure. For example, it means that “a red flowerbot” is subordinating noun of “a sweet-smelling flower”.

Language Understanding Intelligent Service (LUIS)

The constituency forest analyzer (previous one) returns so much useful information, but it’s still difficult to understand this tree structure in your program code.

Language Understanding Intelligent Service (called “LUIS”) is not for just the parsing and analyzing like Linguistic Analysis API, but it provides the direct answer to some application’s scenarios for language understanding tasks.
For example, assuming that there’s an application for booking airline ticket. You might imagine that an application is having the form and field area of “departure location”, “destination location”, and “date/time”.
Using LUIS, you can retrieve these input values from natural language sentences (like, “I want a flight from Orlando to Seattle on 23rd Jul”). LUIS is very straightforward for language understanding tasks.

For example, you can directly retrieve the following values (departure location, destination location, date/time) enclosed by the branckets.

Book a flight from {Orlando} to {Seattle} on {29/10/2016}.
Book me a flight to {Seattle} on {29th Oct}.
I want a flight from {Orlando} to {Seattle} {next Saturday}.
. . .

Notice : You see, the 2nd example lacks the input value “departure location”. But, in such a case, you can distinguish which parameters is needed (fullfilled) for action using LUIS,  and you can ask the user to fill the required parameter.

When you use LUIS, first you register the scenarios (called “intent”) like “Book Flight”, “Get Weather”, etc… For each intent, you register the sample sentences (called “utterances”) and teach the result. When you push “train” button, LUIS learns the pattern.
Now you can use LUIS calling rest endpoint. The rest endpoint returns the result as json, which matches the registered intent.

Notice : LUIS also provides the active learning which you can correct the sentences from existing unlabeled sentences.

LUIS is not only understanding words, but the context in sentence. For example, if you put “Book me a flight to etcetera etcetera on 29th Oct”, the word “etcetera etcetera” might be extracted as destination location.

Note that LUIS requires the previously registered intent, then it’s not used for the ad-hoc resolving, like natural language search, ad-hoc question answering. etc…
Moreover, LUIS only retrieves the target key phase, and doesn’t analyze the phase itself. Let’s consider the next example.

“I want a flowerbot which matches the red flowers for my mother.”

LUIS might retrieve the phase “a flowerbot which matches the red flowers” as target, but it doesn’t mean “a flowerbot and red flowers.” If you want to analyze this phase for acknowledging the exact customer needings, you must use the previous Linguistic Analysis API.

Notice : The name of Text Analytic API in Microsoft Cognitive Services reminds you the language parsing, but this API just analyzes the sentiment (human feeling of satisfied or not satisfied). Note that this doesn’t analyze the human emotion (glad, sad, angry, etc) like Emotion API (also, in Microsoft Cognitive Services), and the result is the scalar value (just the degree of sentiment). You can also retrieve the key phase which impacts to the sentiment. For example, if you collect the customer voice and you want to know what the customer is not satisfied with your services, you can also understand the possible reasons by retrieving this key phases (“facility”, “staff”, etc).