Natural Language Processing Class 10 Notes CBSE

---

Natural Language Processing unit 7 Class 10 Aritificial Intelligence CBSE conveys the connction between human langauges and machine processing.

Here we discuss about Applications of NLP, Chatbots, Text Classificaiton, NLP processes such as Text Normalization and Bag of words algorithm

---

NLP PDF Notes Download

Open NLP Notes Class 10

What is NLP?

Till now we have covered two domains of AI that is Data Science & Computer vision and NLP is Third Domain of AI

1. Data Science: It is all about applying mathematical and statistical principles to data or in simple words, Data Science is the study of Data, This data can be of 3 types – Audio, Visual and Textual.
2. Computer Vision: In simple words is identifying the symbols from the given object (pictures) and learning the pattern and alert or predicting the future object using the camera.

Applications of Natural Language Processing

Some of the applications of Natural Language Processing that are used in the real-life scenario:

Automatic Summarization

• Summarizing the meaning of documents and information
• Extract the key emotional information from the text to understand there actions (Social Media)

Sentiment Analysis

• Definition: Identify sentiment among several posts or even in the same post where emotion is not always explicitly expressed.
• Companies use it to identify opinions and sentiments to understand what customers think about their products and services.
• Can be Positive, Negative or Neutral

Text classification

Virtual Assistants

Natural Language Processing: Getting Started

Revisiting the AI Project Cycle

Related ArticleAI Project Cycle

ChatBots

One of the most common applications of Natural Language Processing is a chatbot. Let us try some of the chatbots and see how they work.

^{Image and Brand Rights belong to the respective owners*}

Types of ChatBots

With the help of this experience, we can understand that there are 2 types of chatbots around us: Script-bot and Smart-bot. Let us understand what each of

Human Language VS Computer Language

Human Language

Computer Language

Arrangement of the words and meaning

There are rules in human language. There are nouns, verbs, adverbs, and adjectives. A word can be a noun at one time and an adjective some other time. There are rules to provide structure to a language.

Besides the matter of arrangement, there’s also meaning behind the language we use. Human communication is complex. There are multiple characteristics of the human language that might be easy for a human to understand but extremely difficult for a computer to understand.

Analog with a programming language:

Let’s understand Semantics and Syntax with some examples:

1. Different syntax, same semantics: 2+3 = 3+2
   • Here the way these statements are written is different, but their meanings are the same that is 5.
2. Different semantics, same syntax: 2/3 (Python 2.7) ≠ 2/3 (Python 3)
   • Here the statements written have the same syntax but their meanings are different. In Python 2.7, this statement would result in 1 while in Python 3, it would give an output of 1.5.

Multiple Meanings of a word

To understand let us have an example of the following three sentences:

1. His face turned red after he found out that he had taken the wrong bag
   • What does this mean? Is he feeling ashamed because he took another person’s bag instead of his? Is he feeling angry because he did not manage to steal the bag that he has been targeting?
2. The red car zoomed past his nose
   • Probably talking about the color of the car, that traveled close to him in a flash.
3. His face turns red after consuming the medicine
   • Is he having an allergic reaction? Or is he not able to bear the taste of that medicine?

Perfect Syntax, no Meaning

Sometimes, when the statement is perfectly correct syntax but there is no meaning to it.

Example: Chickens feed extravagantly while the moon drinks tea

This statement is correct grammatically but does this make any sense? In Human language, a perfect
balance of syntax and semantics is important for better understanding.

Data Processing

As we have already gone through some of the complications in human languages above, now it is time to see how Natural Language Processing makes it possible for machines to understand and speak Natural Languages just like humans.

Since we all know that the language of computers is Numerical, the very first step that comes to our mind is to convert our language to numbers. This conversion takes a few steps to happen. The first step to it is Text Normalisation.

Text Normalisation helps in cleaning up the textual data in such a way that it comes down to a level where its complexity is lower than the actual data.

1. Text Normalisation

In-Text Normalization, we undergo several steps to normalize the text to a lower level. That is, we will be working on text from multiple documents and the term used for the whole textual data from all the documents altogether is known as corpus.

2. Sentence Segmentation

Under sentence segmentation, the whole corpus is divided into sentences. Each sentence is taken as a different data so now the whole corpus gets reduced to sentences.

Example:

3. Tokenisation

After segmenting the sentences, each sentence is then further divided into tokens. A “Token” is a term used for any word or number or special character occurring in a sentence.

Under Tokenisation, every word, number, and special character is considered separately and each of them is now a separate token.

Example:

1. You want to see the dreams with close eyes and achieve them?

You

want

to

the

dreams

with

close

eyes

and

achieve

them

?

4. Removal of Stopwords

In this step, the tokens which are not necessary are removed from the token list. To make it easier for the computer to focus on meaningful terms, these words are removed.

Along with these words, a lot of times our corpus might have special characters and/or numbers.

Example

1. You want to see the dreams with close eyes and achieve them?
   • the removed words would be
   • to, the, and, ?
2. The outcome would be:
   • You want see dreams with close eyes achieve them

5. Converting text to a common case

As the name suggests, we e convert the whole text into a similar case, preferably lower case. This ensures that the case sensitivity of the machine does not consider the same words as different just because of different cases.

6. Stemming

Example:

Words	Affixes	Stem
healing	ing	heal
dreams	s	dream
studies	es	studi

7. Lemmatization

Example:

Words	Affixes	lemma
healing	ing	heal
dreams	s	dream
studies	es	study

Difference between stemming and lemmatization

Bag of word Algorithm

Bag of Words is a Natural Language Processing model which helps in extracting features out of the text which can be helpful in machine learning algorithms. In a bag of words, we get the occurrences of each word and construct the vocabulary for the corpus.

Bag of Words just creates a set of vectors containing the count of word occurrences in the document (reviews). Bag of Words vectors is easy to interpret.

The text on the left in this image is the normalized corpus that we have got after going through all the steps of text processing. Now, as we put this text into the bag of words algorithm, the algorithm returns to us the unique words out of the corpus and their occurrences in it. As you can see at the right, it shows us a list of words appearing in the corpus, and the numbers corresponding to it show how many times the word has occurred in the text body.

The bag of words gives us two things:

• A vocabulary of words for the corpus
• The frequency of these words (number of times it has occurred in the whole corpus).

Steps of the bag of words algorithm

1. Text Normalisation: Collecting data and pre-processing it
2. Create Dictionary: Making a list of all the unique words occurring in the corpus. (Vocabulary)
3. Create document vectors: For each document in the corpus, find out how many times the word from the unique list of words has occurred.
4. Create document vectors for all the documents.

Example:

Step 1: Collecting data and pre-processing it.

Step 2: Create Dictionary

Dictionary:

aman	and	anil	are	stressed	went
download	health	chatbot	therapist	a	to

Step 3: Create a document vector

How to make a document vector table?

In the document, vector vocabulary is written in the top row. Now, for each word in the document, if it matches the vocabulary, put a 1 under it. If the same word appears again, increment the previous value by 1. And if the word does not occur in that document, put a 0 under it.

aman	and	anil	are	stressed	went	to	a	therapist	download	health	chatbot
1	1	1	1	1	0	0	0	0	0	0	0

step 4: Creating a document vector table for all documents

aman	and	anil	are	stressed	went	to	a	therapist	download	health	chatbot
1	1	1	1	1	0	0	0	0	0	0	0
1	0	0	0	0	1	1	1	1	0	0	0
0	0	1	0	0	1	1	1	0	1	1	1

In this table, the header row contains the vocabulary of the corpus and three rows correspond to three different documents. Take a look at this table and analyze the positioning of 0s and 1s in it.

Finally, this gives us the document vector table for our corpus. But the tokens have still not converted to numbers. This leads us to the final steps of our algorithm: TFIDF.

TFIDF

TFIDF stands for Term Frequency & Inverse Document Frequency.

Term Frequency

Example of Term Frequency:

aman	and	anil	are	stressed	went	to	a	therapist	download	health	chatbot
1	1	1	1	1	0	0	0	0	0	0	0
1	0	0	0	0	1	1	1	1	0	0	0
0	0	1	0	0	1	1	1	0	1	1	1

Here, as we can see that the frequency of each word for each document has been recorded in the table. These numbers are nothing but the Term Frequencies!

Inverse Document Frequency

To understand IDF (Inverse Document Frequency) we should understand DF (Document Frequency) first.

DF (Document Frequency)

Example of Document Frequency:

aman	and	anil	are	stressed	went	to	a	therapist	download	health	chatbot
2	1	2	1	1	2	2	2	1	1	1	1

We can observe from the table is:

• Document frequency of ‘aman’, ‘anil’, ‘went’, ‘to’ and ‘a’ is 2 as they have occurred in two documents.
• Rest of them occurred in just one document hence the document frequency for them is one.

IDF (Inverse Document Frequency)

Example of Inverse Document Frequency:

aman	and	anil	are	stressed	went	to	a	therapist	download	health	chatbot
3/2	3/1	3/2	3/1	3/1	3/2	3/2	3/2	3/1	3/1	3/1	3/1

Formula of TFIDF

The formula of TFIDF for any word W becomes:

TFIDF(W) = TF(W) * log( IDF(W) )

Example of TFIDF:

aman	and	anil	are	stressed	went	to	a	therapist	download	health	chatbot
1*log(3/2)	1*log(3)	1*log(3/2)	1*log(3)	1*log(3)	0*log(3/2)	0*log(3/2)	0*log(3/2)	0*log(3)	0*log(3)	0*log(3)	0*log(3)
1*log(3/2)	0*log(3)	0*log(3/2)	0*log(3)	0*log(3)	1*log(3/2)	1*log(3/2)	1*log(3/2)	1*log(3)	0*log(3)	0*log(3)	0*log(3)
0*log(3/2)	0*log(3)	1*log(3/2)	0*log(3)	0*log(3)	1*log(3/2)	1*log(3/2)	1*log(3/2)	0*log(3)	1*log(3)	1*log(3)	1*log(3)

Here, we can see that the IDF values for Aman in each row are the same and a similar pattern is followed for all the words in the vocabulary. After calculating all the values, we get:

aman	and	anil	are	stressed	went	to	a	therapist	download	health	chatbot
0.176	0.477	0.176	0.477	0.477	0	0	0	0	0	0	0
0.176	0	0	0	0	0.176	0.176	0.176	0.477	0	0	0
0	0	0.176	0	0	0.176	0.176	0.176	0	0.477	0.477	0.477

Finally, the words have been converted to numbers. These numbers are the values of each document. Here, we can see that since we have less amount of data, words like ‘are’ and ‘and’ also have a high value. But as the IDF value increases, the value of that word decreases. That is, for example:

• Total Number of documents: 10
• Number of documents in which ‘and’ occurs: 10

Therefore, IDF(and) = 10/10 = 1

Which means: log(1) = 0. Hence, the value of ‘and’ becomes 0.

On the other hand, the number of documents in which ‘pollution’ occurs: 3

IDF(pollution) = 10/3 = 3.3333…

This means log(3.3333) = 0.522; which shows that the word ‘pollution’ has considerable value in the corpus.

Applications of TFIDF

TFIDF is commonly used in the Natural Language Processing domain. Some of its applications are:

1. Document Classification – Helps in classifying the type and genre of a document.
2. Topic Modelling – It helps in predicting the topic for a corpus.
3. Information Retrieval System – To extract the important information out of a corpus.
4. Stop word filtering – Helps in removing the unnecessary words from a text body.

Related ArticleImportant Questions on NLP

Frequently Asked Questions (FAQ’s)