Natural Language
Natural language refers to the way humans, communicate with each other in form of Speech, Text, signs, etc.
Need for NLP
Now a days businesses use massive quantities of unstructured, text-heavy data and need a way to efficiently process it. A lot of the information created online and stored in databases is natural human language. This is where NLP comes in to play. With Natural Language Processing programs, users/business can quickly process, analyse, and infer from their massive data sets to help them enhance their productivity.
What is natural language processing?
Natural language processing (NLP) is the process where a computer program processes human language as it is spoken and written [referred to as natural language] and responds back in a way we humans can understand. It is a Branch of artificial intelligence.
NLP combines computational linguistics: rule-based modelling of human language—with statistical, machine learning, and deep learning models.
NLP drives computer programs that convert text from one language to another, respond to speech, and summarize large volumes of text data rapidly—even in real time.
NLP tasks
Human language is filled with variations that make it highly difficult to write a program that accurately determines the intended meaning of text or speech.
There are many NLP tasks which break down human text and voice data in ways that help the computer make sense of what it's ingesting. Some of the tasks are listed below:
· Speech recognition, also known as speech-to-text, is a process of converting speech data into text data. Speech recognition is required for any application that follows voice commands or answers to verbal questions.
· Word sense disambiguation It is a task where a particular meaning of a word (the word having multiple meanings)is selected through a process of semantic analysis, which makes the most sense in the given context
o For example, word sense disambiguation helps distinguish the meaning of the verb 'make' in ‘make the grade’ (achieve)vs. ‘make a bet’ (place).
· Part of speech tagging[grammatical tagging], is the task of determining the part of speech of a particular word or a part of text based on the use and context.
This task helps in properly deciding what type if input is being provided and what should be the closest to perfect output.
o For example, in the sentence ‘I can make a paper plane,’, it identifies ‘make’ as a verb, in the sentence ‘What make of car do you own?’ make is identified as a noun.
· Co-reference resolution it is the process where a program identifies if and when two words refer to the same entity. The most common example is determining the person or object to which a certain pronoun refers.
o e.g., In the sentence: Mary went to the beach. She said it was an amazing experience.
· She refers to Mary. And it refers to the concert.
· Sentiment analysis This task attempts to extract and process subjective qualities such as attitudes, emotions, sarcasm, confusion, suspicion from textual data.
· Named entity recognition, or NEM, identifies words or phrases as useful entities. NEM identifies ‘Kentucky ’as a location or ‘Fred’ as a man's name.
How does natural language processing work?
NLP enables computer programs to understand and decode natural language same as humans do. Whether the language is in speech or text format, natural language processing uses artificial intelligence to take real-world inputs, process it, and transform it in a way o computer can understand and process.
There are two main phases to natural language processing:
· Data pre processing
· Algorithm Development
Data pre processing This step involves preparing and "cleaning" stage of text data for computers to be able to understand and analyze it. After this process the data is transformed into workable form, and all the textual features are highlighted, to be processed by an algorithm
Different Data Pre-Processing techniques:
· Tokenization: In this technique the text data is broken down in smaller chunks/tokens to work with.
· Stop word removal: In this technique the common words are removed from text, so that the remaining unique words offer the most information about the text.
· Lemmatization and stemming: In this technique the words are reduced to their root/original forms to process.
· Part-of-speech tagging: In this technique the words are marked based on the part-of speech they are -- such as nouns, verbs, and adjectives.
Algorithm Development.
Once the data out of the pre processing stage, an algorithm is developed for the machine to process it.
There are many different natural language processing algorithms, but two main types are commonly used:
• Rules-based system: This is one of the approaches which were being used in the early phases of Machine learning development. This approach leverages carefully designed Linguistics rules. Rule based Systems are still in use today.
• Machine learning-based system: This approach leverages many aspects of AI such as combination of Machine learning, neural networks and deep learning to develop algorithms and process the data. In this system statistical methods are used. These algorithms learn to perform based on the input training data sets.
Natural Language Processing algorithms finetune their own rules as their processing time passes and through constant processing and learning.
Techniques and methods of natural language processing
Syntax and semantic analysis are two main techniques used for natural language processing.
Syntax: This refers to the arrangement of words in a sentence to make grammatical sense. NLP uses syntax to infer meanings from a language using grammatical rules.
This Technique includes:
· Parsing
· Word segmentation
· Sentence breaking
· Morphological segmentation
· Stemming
Semantics: Natural language processing applies algorithms to understand the meaning, use of word and structure of sentences.
This Technique Includes:
· Word Sense Disambiguation: In this process the meaning of words is derived based on the context.
· Named Entity Recognition: This process picks up words that can be categorized into groups.
· Natural language generation: In this process a database is used to determine semantics behind words and generate new text data.
Three tools used commonly for natural language processing include
• Natural Language Toolkit (NLTK): NLTK is an open-source Python module with datasets and tutorials
• Gensim: Gensim is a Python library for topic modelling and document indexing
• Intel natural language processing Architect: Intel NLP Architect is another Python library for deep learning topologies and techniques.
There are some other python packages which can be used to Implement NLP such as Pycaret.
Some Examples for NLP implementation in the industries:
• Enhancing Medical Care: NLP enables the recognition and prediction of diseases based on electronic health records and patient’s own speech. This capability is being explored in health conditions that go from cardiovascular diseases to depression and even schizophrenia.
• For example, Amazon Comprehend Medical is a service that uses NLP to extract disease conditions, medications and treatment outcomes from patient notes, clinical trial reports and other electronic health records.
• Sentiment analysis: Organizations can determine what customers are saying about a service or product by identifying and extracting information insources like social media. This sentiment analysis can provide a lot of information about customers choices and their decision drivers.
• Spam Protection: Companies like Yahoo and Google filter and classify your emails with NLP by analyzing text in emails that flow through their servers and stopping spam before they even enter your inbox.
• Virtual Assistants: Amazon’s Alexa and Apple’s Siri are examples of intelligent voice driven interfaces that use NLP to respond to vocal prompts and do everything like find a particular shop, tell us the weather forecast, suggest the best route to the office or turn on the lights at home.
Natural Language Processing in Power BI
Power Bi is a tool which has lots of great features, one of them being integration of different Python and R packages to implement multiple models like NLP, Topic modelling, anomaly detection, ML and AI features.
Other than using Python or R code to implement Natural language Processing, Power BI Also provides Built – in Feature to leverage NLP feature, the Q&A Visual
• Power BI Q&A visual allows users to ask the question in natural language and get answers/outputs in form of report visuals.
• Q&A visual can also be integrated with other report visuals and works seamlessly.
• The Q&A visual interacts with other report visuals, and slicers like normal Power BI Dashboards.
By leveraging natural language processing (NLP), Power BI’s Q&A feature offers new ways to explore data and find hidden insights.
Power BIQ&A lets business users explore their datasets by asking specific questions and responding with the best and most relevant answers with proper visualizations without a need for strong technical background.
Power BIQ&A also displays suggestions to help users form their question, and each visualization changes accordingly as questions are modified.
users can even set how Q&A displays answers; for example, asking to ‘show monthly performance as a graph’; this lets users drill down into the data.
Power BI recently has enabled the in-built Q&A feature to be trained like a machine learning model, this will help the Power BI reports/Dashboards to company-specific language and phrases to provide end users perfect results.
There is anew section added, where the Developers can review every question asked by users to finetune the Power BI NLP for future
Power BIQ&A for Mobile Users.
Power BI offers on-the-fly versions of reporting services via mobile applications for Android, iOS and Windows 10 smartphones and tablets - even Apple Watch. On the mobile app, users can connect to and analyze predefined dashboards and reports created in Power BI Desktop and on-premises reports on Power BI Report Server easily.
However, for power BI iOS users there is a bonus i.e., exclusive access to two nifty conversational AI capabilities built into the app.
For iOS Users
· Users can add Siri shortcuts for frequent and favourite dashboards and access using voice commands whenever needed
· Power BI iOS users can also ask specific questions about their data and get new insights and analysis using the Q&A virtual analyst.
· The unique chat feature is in-built and tailor-made for mobile.
· The natural language querying is accessed via the action menu in each dashboard, where you can either type your question in the chat or use iOS native speech recognition (voice) to ask questions.
Q&A Visual in Power BI Desktop:
