Python Regex for Natural Language Processing

In this article we want to learn about Python Regex for Natural Language Processing, so natural language processing or NLP is a branch of computer science that deals with interaction between computers and humans using natural language. it involves many techniques such as tokenization, stemming, lemmatization and more. Python is one of the most popular programming languages for NLP. because it is easy and it has a lot of libraries, you can also use regular expressions or regex module for NLP tasks in Python.

Python Regex for Natural Language Processing

So first of all let’s learn about the basics of regex in Python, for using regex in Python, first we need to import that module in our code.

This is a simple example that uses regex to match a string.

import re

string = "Hello, world!"
pattern = r"world"
match = re.search(pattern, string)
print(match.group(0))

import re

string = "Hello, world!"

pattern = r"world"

match = re.search(pattern, string)

print(match.group(0))

In this example we have created a regex pattern that matches the string world and after that we have used re.search method to search for that pattern in the string. if the pattern is found, match will contain the matched string which we can print that.

This will be the result

So ow that we have learned about the basics of regex in Python, let’s learn that how we can use regex for NLP tasks, these are some examples of how regex can be used in NLP.

Tokenization: Tokenization is the process of breaking a text into individual words or tokens. this is an example of how we can use regex to tokenize a text.

import re

text = "Hello, geekscoders.com! This is a sample text."
tokens = re.findall(r'\b\w+\b', text)
print(tokens)

import re

text = "Hello, geekscoders.com! This is a sample text."

tokens = re.findall(r'\b\w+\b', text)

print(tokens)

In the above example we have used \b character to match word boundaries and \w+ character to match one or more word characters. after that we have used re.findall method to find all instances of this pattern in the text. this code will return a list of all the tokens found in the text.

This will be the result

Python Regex for Natural Language Processing

Removing punctuation: Punctuation can often be noise in NLP tasks. this is an example of how we can use regex to remove punctuation from a text.

import re

text = "Hello, geekscoders! Welcome to website."
clean_text = re.sub(r'[^\w\s]', '', text)
print(clean_text)

import re

text = "Hello, geekscoders! Welcome to website."

clean_text = re.sub(r'[^\w\s]', '', text)

print(clean_text)

In the above example we have used re.sub method to substitute all non word and non whitespace characters with an empty string. this code will return cleaned version of the original text with all punctuation removed.

This will be the result

Extracting named entities: Named entities are specific types of words or phrases that represent particular entities such as people, organizations or locations. this is an example of how we can use regex to extract named entities from a text.

import re

text = "Parwiz is a software engineer at Google."
pattern = r'(John Smith|Google)'
named_entities = re.findall(pattern, text)
print(named_entities)

import re

text = "Parwiz is a software engineer at Google."

pattern = r'(John Smith|Google)'

named_entities = re.findall(pattern, text)

print(named_entities)

This will be the result