What is natural language processing?

What is natural language processing?

Natural Language Processing (NLP) is a set of techniques that help a computer system understand human speech.

NLP is a subset of artificial intelligence. This is one of the most difficult tasks of AI, which has not been fully solved so far.

When did NLP appear??

The roots of natural language processing go back to the 1950s, when the famous English scientist Alan Turing published an article “Computing Machines and Mind”by offering the so-called “Turing test”. One of its criteria is the ability of a machine to automatically interpret and generate human speech.

On January 7, 1954, scientists at Georgetown University demonstrated the power of machine translation. The engineers were able to translate more than 60 sentences from Russian into English in a fully automatic manner. This event had a positive impact on the development of machine translation and went down in history as Georgetown Experiment.

In 1966, an American computer scientist of German origin, Joseph Weizenbaum, developed the world’s first Eliza chatbot within the walls of the Massachusetts Institute of Technology. The program parodied a dialogue with a psychotherapist using the technique of active listening.

By and large, the system rephrased the user’s messages to give the appearance of understanding what was said. However, in fact, the program did not delve into the essence of the dialogue. When she couldn’t find an answer, she usually answered “I see” (“Understood”) and turned the conversation in a different direction.

In the same year, the Automatic Language Processing Advisory Committee (ALPAC) released report and concluded that decades of research fell short of expectations. As a result, funding for machine translation has been drastically reduced.

Over the next decades, there were no breakthroughs in the field of NLP, until the emergence of the first machine learning algorithms in the 1980s. It was around this time that statistical machine translation systems emerged, resurrecting research.

The boom in language processing came in 2010, when deep learning algorithms began to develop. During this period, many developments appeared that we still use today, such as chat bots, auto-correctors, voice assistants, and others. Most often, recurrent neural networks began to be used to solve these problems.

Another revolution in NLP systems occurred in 2019, when OpenAI presented the language model Generative Pre-Trained Transformer 2, or GPT-2. Unlike existing generators, this neural network was able to create long lines of related text, answer questions, compose poetry and compose new recipes.

A year later, OpenAI showed a new version GPT-3and large technology companies, one after another, began to demonstrate their own developments in the field of large language models.

How do NLP systems work?

To answer this question, it is necessary to pay attention to how we humans use natural language.

When we hear or read any phrase, several processes occur simultaneously in our subconscious:

• perception;
• understanding of meaning;
• response.

Perception is the process of translating a sensory signal into symbolic form. For example, we can hear a particular word or see it written in different fonts. Any of these types of obtaining information must be converted into a single: written words.

Understanding the meaning is the most difficult task, which even people with their natural intelligence cannot always cope with. Due to ignorance of the context and incorrect interpretation of the phrase, various embarrassments and sometimes serious conflicts can arise.

For example, in 1956, at the height of the Cold War between the USSR and the United States, the head of the Soviet state, Nikita Khrushchev, delivered a speech in which the phrase sounded “We will bury you.” The Americans took what was said too literally and regarded it as a threat of a nuclear attack. Although in fact, Khrushchev only meant that socialism will outlive capitalism, and the phrase itself is an interpretation of the thesis of Karl Marx.

The incident quickly escalated into an international scandal, for which Soviet diplomats and the Secretary General of the Communist Party had to apologize.

That is why it is very important to correctly understand the meaning of speech, the context of what was said or written, in order to prevent such situations that affect people’s lives.

A reaction is the result of a decision. This is a fairly simple task that requires the formation of a set of possible answers based on the meaning of the perceived phrase, context, and possibly some internal experiences.

Natural language processing algorithms work on exactly the same principle.

Perception is the process of translating incoming information into a machine-understandable set of characters. If this is text from a chatbot, then such an incoming set will be direct. If this is an audio file or handwritten text, then first you need to translate it into a convenient form. Modern neural networks successfully cope with this.

The text response problem was also successfully solved by weighing the alternatives and comparing the results with each other. For a chatbot, this can be a text response from its knowledge base, and for a voice assistant, it can be an action with some smart home object, for example, turning on a light bulb.

With understanding, things are somewhat different and this issue should be considered separately.

How do AI systems understand speech?

Today, the following types of analysis are common in solving problems of understanding the language:

• statistical;
• formal grammatical;
• neural network.

Statistical widely used in machine translation services, automatic reviewers and some chatbots. The essence of the method lies in “feeding” the model of a huge amount of an array of texts in which statistical patterns have been established. Then such models are used to translate texts or generate new ones, sometimes with understanding of the context.

Formal-grammatical the approach is a mathematical apparatus that allows you to accurately and unambiguously determine the meaning of a phrase in a natural language as much as possible for a machine. However, this is not always possible, since the meaning of some phrases is not clear even to people.

For developed languages ​​like Russian or English, an accurate and detailed description of speech in mathematical terms is an extremely difficult problem. Therefore, the formal-grammatical approach is more often used for the syntactic analysis of artificial languages, from which ambiguities are specifically removed during design.

V neural network approach, deep learning neural networks are used to recognize the meaning of the input phrase and generate the reaction of the AI ​​system. They are trained on stimulus-response pairs, where the stimulus is a phrase in natural language, and the response is the response of the AI ​​system in it or any actions of the AI ​​system.

This is a very promising approach, but it has all the negative qualities of neural networks.

What are NLP systems used for?

Natural language processing systems are used to solve many problems, from creating chatbots to analyzing huge text documents.

The main tasks of NLP include:

• text analysis;
• speech recognition;
• text generation;
• text-to-speech transformation.

Text analysis is the intellectual processing of large amounts of information, the purpose of which is to identify patterns and similarities. It includes data mining, search, utterance analysis, question-answer systems, and sentiment assessment.

Speech recognition is the process of converting text files or voice into digital information. A simple example: when you access Siri, the algorithm recognizes speech in real time and converts it into text.

Text generation is the process of creating texts using computer algorithms.

Text-to-speech is the reverse process of speech recognition. An example is reading information from the Internet with voice assistants.

Where are natural language processing systems used?

There are many ways to use NLP technologies in everyday life:

• email services use Bayesian spam filtering, an NLP statistical technique that compares incoming messages to a database and identifies spam;
• text editors like Microsoft Word or Google Docs use language processing to correct errors in words, not only grammatical, but also contextual;
• virtual keyboards in modern smartphones can predict the next words in the context of a sentence.
• voice assistants like Siri or Google Assistant can recognize the user, execute commands, transform speech into text, search the Internet, control smart home devices, and much more;
• accessibility applications on PCs and smartphones can read text and interface elements for visually impaired people thanks to speech synthesis algorithms;
• language models with a huge number of parameters like GPT-3 or BERT can generate texts of various lengths in various genres, help to search and predict a sentence by the first few words;
• Machine translation systems use statistical and language models to translate texts from one language to another.

What difficulties arise when using NLP technologies?

Often, when solving NLP problems, recurrent neural networks are used, which have a number of disadvantages, including:

• sequential processing of words;
• inability to retain a large amount of information in memory;
• problem susceptibility vanishing/exploding gradient;
• impossibility of parallel processing of information.

In addition, popular processing methods often misunderstand the context, which requires additional careful tuning of the algorithms.

Most of these problems are solved by large language models, but there are a number of difficulties with them. The first is their availability. A large language model, like GPT-3 or BERT, is difficult to train, but large companies are increasingly making them available to the public.

Also, many models work only with popular languages, ignoring uncommon dialects. This affects the ability of voice algorithms to recognize different accents.

When processing text documents using optical character recognition technology, many algorithms still cannot cope with handwritten fonts.

In addition to technological flaws, NLP can also be used for malicious purposes. For example, in 2016, Microsoft launched the Tay chatbot on Twitter, which learned how to communicate using the example of its human interlocutors. However, after only 16 hours the company turned off the robotwhen he started posting racist and offensive tweets.

In 2021, scammers from the UAE spoofed the voice of the head of a large company and convinced a bank employee to transfer $35 million to their accounts.

Similar case happened in 2019 with a British energy company. The scammers managed to steal about $243,000 by impersonating a company director using a fake voice.

Large language models can be used for mass spam attacks, harassment or misinformation. About it warned the creators of GPT-3. They also reported that their language model is biased towards certain groups of people. However, OpenAI reported that they reduced the toxicity of GPT-3, and at the end of 2021 they provided access to the model to a wide range of developers and allowed them to customize it.

What is natural language processing?

Natural Language Processing (NLP) is a set of techniques that help a computer system understand human speech.

NLP is a subset of artificial intelligence. This is one of the most difficult tasks of AI, which has not been fully solved so far.

When did NLP appear??

The roots of natural language processing go back to the 1950s, when the famous English scientist Alan Turing published an article “Computing Machines and Mind”by offering the so-called “Turing test”. One of its criteria is the ability of a machine to automatically interpret and generate human speech.

On January 7, 1954, scientists at Georgetown University demonstrated the power of machine translation. The engineers were able to translate more than 60 sentences from Russian into English in a fully automatic manner. This event had a positive impact on the development of machine translation and went down in history as Georgetown Experiment.

In 1966, an American computer scientist of German origin, Joseph Weizenbaum, developed the world’s first Eliza chatbot within the walls of the Massachusetts Institute of Technology. The program parodied a dialogue with a psychotherapist using the technique of active listening.

By and large, the system rephrased the user’s messages to give the appearance of understanding what was said. However, in fact, the program did not delve into the essence of the dialogue. When she couldn’t find an answer, she usually answered “I see” (“Understood”) and turned the conversation in a different direction.

In the same year, the Automatic Language Processing Advisory Committee (ALPAC) released report and concluded that decades of research fell short of expectations. As a result, funding for machine translation has been drastically reduced.

Over the next decades, there were no breakthroughs in the field of NLP, until the emergence of the first machine learning algorithms in the 1980s. It was around this time that statistical machine translation systems emerged, resurrecting research.

The boom in language processing came in 2010, when deep learning algorithms began to develop. During this period, many developments appeared that we still use today, such as chat bots, auto-correctors, voice assistants, and others. Most often, recurrent neural networks began to be used to solve these problems.

Another revolution in NLP systems occurred in 2019, when OpenAI presented the language model Generative Pre-Trained Transformer 2, or GPT-2. Unlike existing generators, this neural network was able to create long lines of related text, answer questions, compose poetry and compose new recipes.

A year later, OpenAI showed a new version GPT-3and large technology companies, one after another, began to demonstrate their own developments in the field of large language models.

How do NLP systems work?

To answer this question, it is necessary to pay attention to how we humans use natural language.

When we hear or read any phrase, several processes occur simultaneously in our subconscious:

• perception;
• understanding of meaning;
• response.

Perception is the process of translating a sensory signal into symbolic form. For example, we can hear a particular word or see it written in different fonts. Any of these types of obtaining information must be converted into a single: written words.

Understanding the meaning is the most difficult task, which even people with their natural intelligence cannot always cope with. Due to ignorance of the context and incorrect interpretation of the phrase, various embarrassments and sometimes serious conflicts can arise.

For example, in 1956, at the height of the Cold War between the USSR and the United States, the head of the Soviet state, Nikita Khrushchev, delivered a speech in which the phrase sounded “We will bury you.” The Americans took what was said too literally and regarded it as a threat of a nuclear attack. Although in fact, Khrushchev only meant that socialism will outlive capitalism, and the phrase itself is an interpretation of the thesis of Karl Marx.

The incident quickly escalated into an international scandal, for which Soviet diplomats and the Secretary General of the Communist Party had to apologize.

That is why it is very important to correctly understand the meaning of speech, the context of what was said or written, in order to prevent such situations that affect people’s lives.

A reaction is the result of a decision. This is a fairly simple task that requires the formation of a set of possible answers based on the meaning of the perceived phrase, context, and possibly some internal experiences.

Natural language processing algorithms work on exactly the same principle.

Perception is the process of translating incoming information into a machine-understandable set of characters. If this is text from a chatbot, then such an incoming set will be direct. If this is an audio file or handwritten text, then first you need to translate it into a convenient form. Modern neural networks successfully cope with this.

The text response problem was also successfully solved by weighing the alternatives and comparing the results with each other. For a chatbot, this can be a text response from its knowledge base, and for a voice assistant, it can be an action with some smart home object, for example, turning on a light bulb.

With understanding, things are somewhat different and this issue should be considered separately.

How do AI systems understand speech?

Today, the following types of analysis are common in solving problems of understanding the language:

• statistical;
• formal grammatical;
• neural network.

Statistical widely used in machine translation services, automatic reviewers and some chatbots. The essence of the method lies in “feeding” the model of a huge amount of an array of texts in which statistical patterns have been established. Then such models are used to translate texts or generate new ones, sometimes with understanding of the context.

Formal-grammatical the approach is a mathematical apparatus that allows you to accurately and unambiguously determine the meaning of a phrase in a natural language as much as possible for a machine. However, this is not always possible, since the meaning of some phrases is not clear even to people.

For developed languages ​​like Russian or English, an accurate and detailed description of speech in mathematical terms is an extremely difficult problem. Therefore, the formal-grammatical approach is more often used for the syntactic analysis of artificial languages, from which ambiguities are specifically removed during design.

V neural network approach, deep learning neural networks are used to recognize the meaning of the input phrase and generate the reaction of the AI ​​system. They are trained on stimulus-response pairs, where the stimulus is a phrase in natural language, and the response is the response of the AI ​​system in it or any actions of the AI ​​system.

This is a very promising approach, but it has all the negative qualities of neural networks.

What are NLP systems used for?

Natural language processing systems are used to solve many problems, from creating chatbots to analyzing huge text documents.

The main tasks of NLP include:

• text analysis;
• speech recognition;
• text generation;
• text-to-speech transformation.

Text analysis is the intellectual processing of large amounts of information, the purpose of which is to identify patterns and similarities. It includes data mining, search, utterance analysis, question-answer systems, and sentiment assessment.

Speech recognition is the process of converting text files or voice into digital information. A simple example: when you access Siri, the algorithm recognizes speech in real time and converts it into text.

Text generation is the process of creating texts using computer algorithms.

Text-to-speech is the reverse process of speech recognition. An example is reading information from the Internet with voice assistants.

Where are natural language processing systems used?

There are many ways to use NLP technologies in everyday life:

• email services use Bayesian spam filtering, an NLP statistical technique that compares incoming messages to a database and identifies spam;
• text editors like Microsoft Word or Google Docs use language processing to correct errors in words, not only grammatical, but also contextual;
• virtual keyboards in modern smartphones can predict the next words in the context of a sentence.
• voice assistants like Siri or Google Assistant can recognize the user, execute commands, transform speech into text, search the Internet, control smart home devices, and much more;
• accessibility applications on PCs and smartphones can read text and interface elements for visually impaired people thanks to speech synthesis algorithms;
• language models with a huge number of parameters like GPT-3 or BERT can generate texts of various lengths in various genres, help to search and predict a sentence by the first few words;
• Machine translation systems use statistical and language models to translate texts from one language to another.

What difficulties arise when using NLP technologies?

Often, when solving NLP problems, recurrent neural networks are used, which have a number of disadvantages, including:

• sequential processing of words;
• inability to retain a large amount of information in memory;
• problem susceptibility vanishing/exploding gradient;
• impossibility of parallel processing of information.

In addition, popular processing methods often misunderstand the context, which requires additional careful tuning of the algorithms.

Most of these problems are solved by large language models, but there are a number of difficulties with them. The first is their availability. A large language model, like GPT-3 or BERT, is difficult to train, but large companies are increasingly making them available to the public.

Also, many models work only with popular languages, ignoring uncommon dialects. This affects the ability of voice algorithms to recognize different accents.

When processing text documents using optical character recognition technology, many algorithms still cannot cope with handwritten fonts.

In addition to technological flaws, NLP can also be used for malicious purposes. For example, in 2016, Microsoft launched the Tay chatbot on Twitter, which learned how to communicate using the example of its human interlocutors. However, after only 16 hours the company turned off the robotwhen he started posting racist and offensive tweets.

In 2021, scammers from the UAE spoofed the voice of the head of a large company and convinced a bank employee to transfer $35 million to their accounts.

Similar case happened in 2019 with a British energy company. The scammers managed to steal about $243,000 by impersonating a company director using a fake voice.

Large language models can be used for mass spam attacks, harassment or misinformation. About it warned the creators of GPT-3. They also reported that their language model is biased towards certain groups of people. However, OpenAI reported that they reduced the toxicity of GPT-3, and at the end of 2021 they provided access to the model to a wide range of developers and allowed them to customize it.