Modified Atbash Cipher code in Python

(Photo credit: Wikipedia)

The Atbash cipher is a very specific case of a Monoalphabetic substitution cipher where the letters of the alphabet are reversed. In other words, all A's are replaced with Z's, all B's are replaced with Y's, and so on. 

I modified this general scheme somewhat and added support for encrypting digits and special characters too to make it stronger because reversing the alphabet twice will get you actual alphabet, you can encipher and decipher a message using the exact same algorithm.

Example:

Plaintext  : Ex094, You are a very Nice Guy!

Ciphertext: [liZeB?~uo?8r4?8?n4rk?_064?-okX

class AtBash:

   def __init__(self):
       self.alphabets = ' ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789~!@#$%^&*()_+|:"<>-=[];,.?/`'

   def encode(self, plaintext):
       cipher = ""
       for i in plaintext:
           index = self.alphabets.index(i)
           cipher += self.alphabets[abs(len(self.alphabets) - index - 1) % len(self.alphabets)]
       return cipher

   def decode(self, ciphertext):
       return self.encode(ciphertext)


if __name__ == "__main__":
   atbash = AtBash()

   enc = atbash.encode("Hello World!")
   print(enc)

   dec = atbash.decode(enc)
   print(dec)

I hope you liked this post. Share this post. Thank you.

Related articles

• Lightweight Introduction to Cryptography | Terminology, Security Goals and Cryptographic Attacks
• 100 Best Open Source Security Tools
• List Of Steganography Tools
• Cryptography | Identify Different types of Hashes
• 80+ Best Free Hacking Tutorials | Resources to Become Pro Hacker

Continue Reading →

Explosion Animation using Spritesheets in PyGame

I made an explosion animation using PyGame with Spritesheets. The following code serves as an example which shows how to use sprite sheets for making animations.

SpriteSheet Used




Output

Continue Reading →

Lyrister - Song Lyrics Scrapper

Hello Everyone,

Today I will share a little snippet I wrote to get scrap or download lyrics of your favourite songs and save on your hard drive as .txt files.

I have used python 2.7.6 for this and have used the requests python package for web requests

You can get the code here : https://github.com/AnimeshShaw/Lyrister

I hope I have taken care of most things that may cause you problems.

How to Use ?

On the terminal run the following and follow the instructions that follow:-

python lyrister.py

You have the option of giving command line arguments as the parameters. The following format is to be followed :-

python lyrister.py <name-of-song> <path-of-directory> <filename>

It takes three parameters. The first parameter is the name of the song for which you want to download the lyrics. Please note that if you intend to write the song name with spaces then use double quotes and your directory path must end with separator.

Example 1:

Here we download the song Carnival by Natalie Merchant. Since we have not provided the song name with space there is no problem.

python lyrister.py carnival /home/psychocoder/ carnivallyric

Example 2:

Here we download the song Carnival Of Rust by "Poets Of Fall". Since we have not provided the song name with space there is no problem.

Code:

python lyrister.py "carnival of rust" /home/psychocoder/ CarnivalOfRust

If a file with the same name exists in the directory then it won't let to save the song and will ask you to try another name to download the lyric.

If you want to save the file in some directory which doesn't have write permission then you wont be able to save the lyrics.

These checking have been done already and I hope the rest you can understand by yourself. I have given a good description for every step in the code itself and also how to resolve the errors. I hope you enjoy.

Related articles

• MangaScrapper - Download Manga in different Ebook formats
• Guess the Number CLI Game in Python
• Java Code to read Character File using FileReader.
• Read Character Files line by line in Java version 6 or less
• Write a File using Java NIO
• Image to ASCII ART converter in Java

Continue Reading →

Fuzzy String Matching or Searching with Python Code

Hello everyone, I have been recently reading about Text Mining and Natural Language Processing (NP). So, I will be sharing a few articles on TextMining from now on. So, that you can have an interest in this beautiful field of Computer Science.

Prerequisites

In order to understand the main content of the article you need some prerequisites which are as follows :- 

• Familiar with Programming.
• Know at least one programming language.
• Basics of Python (Optional but Recommended)
• Have some prior exposure to Data Structures and algorithmic techniques like Backtracking, Dynamic Programming etc.
• Have some gray cells and have patience.
• Passion for Computer Science.

Let's Start,

---

The first question that should arise in your mind when you hear the term TextMining is What is TextMining and What does it do ? hence we will understand first what is Text or Data Mining. Lets proceed.

Text Mining or Data Mining

Text mining, also referred to as text data mining, roughly equivalent to text analytics, refers to the process of deriving high-quality information from text. High-quality information is typically derived through the devising of patterns and trends through means such as statistical pattern learning.

Text mining usually involves the process of structuring the input text (usually parsing, along with the addition of some derived linguistic features and the removal of others, and subsequent insertion into a database), deriving patterns within the structured data, and finally evaluation and interpretation of the output.

To explain it simply

Data Mining is a processing of locating or finding important expression, phrases or sentences or events from data or text which can be further processed to get more useful information. Even though data mining tries to collect information from data, its different from Information Retrieval, instead of the fact that Data Mining, Information Retrieval or Natural Language Processing are often connected together and misunderstood as one, I will explain how they are different some other time and not here. If you wanna develop an application based on Text or where you have to deal with large amount of data and you need an intelligent system which will classify the data and sorts them into different categories or do different processes then you need these concepts a lot.

Real life examples of Data Mining

1. Google Search : Whenever you starting writing a query it automatically suggests the possible search queries. How does it do ? They use these concept to derive the most probable search query.

2. Spell Correction : You might have seen that if you type wrong spelling's most search engines or word processors mark them with a red underline which means that you have a spelling error. MS Word or Android apps has automatic spell correction. They use these NLP and Text Mining concepts. See the image it will clear a lot of doubts. See how Google does spell correction and the search results are an example of the query I entered.

3. Amazon or Online Shopping Stores : Most of you must have done online shopping a lot. Whenever you visit a site for the first time and search for some specific product. You see some thing written as recommended for you. Next time when you visit that site you might be astonished to see that the products you see when you open the webpage are similar to what you had searched in your last visit but more refined. So, how does it work ? the earlier mentioned concepts are used for these. They collect your browsing history and process the data and finally show you the most recommended products.

There are thousands of examples just Google for it :P
I hope by now you have somewhat understood what is data or text mining. Now we will come to today's real discussion that is fuzzy string searching. So lets continue.

---

Today's Discussion

In this section we discuss about String matching or string similarity algorithms. We will discuss the following algorithms today.

1. Damerau-Lavenshtein Distance.
2. Levenshtein Distance.
3. Wagner-Fisher Algorithm.

Strings are among the most important elements of computer science. There are literally 100's of research papers on strings. String searching and matching is one of he most important part in Text Processing and text analysis, since all the data to intelligent systems are fed as string. Strings are nothing but a sequence of characters joined together.

Fuzzy String Searching or Fuzzy String Matching

Fuzzy string search algorithms are algorithms that are used to match either exactly or partially of one string with another string. So, what exactly does fuzzy mean ? 

Fuzzy by the word we can understand that elements that aren't clear or is like an illusion. Therefore, in short we say that when there is a set of n elements and another set of m elements and they both have partially same elements then we can say that the relation between them is fuzzy.

(Funny example Most women have a fuzzy thinking that their men are cheating on them but instead of taking it as fuzzy they take it for granted :P)

By Fuzzy string searching (also known as approximate string matching) we identify those strings which are similar to a set of other strings. A number of conclusions can be drawn from these like Similarity co-efficient, similar words etc, etc.

The problem of approximate string matching is typically divided into two sub-problems: finding approximate sub-string matches inside a given string and finding dictionary strings that match the pattern approximately.

Fuzzy search algorithms (also known as similarity search algorithms) serve as a base for spell-checkers and search engines like Google or Bing or Yahoo etc. For example, these algorithms are used to provide the "Did you mean ..." functionality that is very common.

Example :- 

If you ask a machine to find the similarity between two words and correct them if required, say, "Looks" and "Lookes", the machine will return true for the first string "Looks" as its correct and will return the corrected string as "Looks" from the erroneous string "Lookes".

Simple Formulation of Fuzzy String Problems :-

"Find in the text or dictionary of a finite size n for all the words that match a word given as input (or starts with the given word), taking into account m possible differences (errors)."

Fuzzy search algorithms are characterized by metric - a function of distance between two words, which provides a measure of their similarity. A strict mathematical definition of metric includes a requirement to meet triangle inequality (p - metric function, Z - a set of words):



LaTeX code for above equation.

p(x,y)  \leq p(x,k) + p(k,y), where,  (x,y,k)  \varepsilon  Z

What is Edit-Distance ?

Definition From Wikipedia

Edit distance is a way of quantifying how dissimilar two strings (e.g., words) are to one another by counting the minimum number of operations required to transform one string into the other. Edit distances find applications in natural language processing, where automatic spelling correction can determine candidate corrections for a misspelled word by selecting words from a dictionary that have a low distance to the word in question.
Simply we can say that - The closeness of a match is measured in terms of the number of primitive operations necessary to convert the string into an exact match. This number is called the edit distance between the string and the pattern.

The primitive operations performed are, Insertion, Deletion, Substitution. Example :- 

1. Mad -- > Made, Insertion Operation Performed
2. Loooks -- > Looks, Deletion Operation Performed
3. Lick -- > Sick, Substitution Operation Performed

Now we will move to discuss the algorithms.

---

Damerau-Levenshtein Distance

Damerau-Levenshtein Distance is a distance (string metric) between two Strings, say String A and String B, which gives the minimum number of edit operations need to perform to transform String A to String B. Damerau's Algorithm can be used for spell correction with atmost 1 edit-distance.

There are four edit operations that can be performed with this algorithm. They are - 1. Insertion, 2. Deletion, 3. Substitution, and 4. Transposition. We have already seen the operation of the first three algorithms and so let me explain the last operation, that is transposition.

Transposition is the swapping of adjacent characters to bring them to a certain form.Example :-

String A := olko
String B := look

When the above two strings are feed to a machine that computes the Damerau-Lavenshtein Distance find that the edit distance is two. 

So how it operates ?
We have two string transpositions here :-

1.) olko -> loko 
2.) loko -> look

Hence we get the final result as :- look
And the number of edits performed is : 2

Please Note that Damerau-Levenshtein Distance doesn't follow the Triangle inequality. You can see that in the image below :-



"All Theory and No Code Makes Psycho a Dull guy"

I am giving the working function for now. You can see the Psuedo code on Wiki, its is well written.

Here's the code in Python. I would encourage you to write it by yourself too.

def damerau_levenshtein(s1, len_s1, s2, len_s2):
    """
    A function to calculate the Damerau-Levenshtein Distance.

    :param s1: Parent String
    :param len_s1: Length Of s1
    :param s2: pattern String
    :param len_s2: length of s2
    :rtype : int returns the damerau-levenshtein distance
    """

    d = {}
    len_s1 = len(s1)
    len_s2 = len(s2)
    for i in range(-1, len_s1 + 1):
        d[i, -1] = i + 1
    for j in range(-1, len_s2 + 1):
        d[-1, j] = j + 1

    for i in range(len_s1):
        for j in range(len_s2):
            if s1[i] == s2[j]:
                cost = 0
            else:
                cost = 1
                
            d[(i, j)] = min(
                d[i - 1, j] + 1,  # Deletion
                d[i, j - 1] + 1,  # Insertion
                d[i - 1, j - 1] + cost,  # Substitution
            )

            if i and j and s1[i] == s2[j - 1] and s1[i - 1] == s2[j]:
                d[i, j] = min(d[i, j], d[i - 2, j - 2] + cost)  # transposition
    return d[len_s1 - 1, len_s2 - 1]

You can get the complete source code here : https://github.com/AnimeshShaw/TextMin...nshtein.py
To see the Code in Execution :- http://code.hackerearth.com/0356acE
You can visualize how the steps or the algorithm works here : Click Me

Levenshtein Distance

Levenshtein distance is also a string metric calculate the edit-distance between two strings. They have found a lot of applications and in most of the real world systems an efficient and modified form of Levenshtein distance is used including various NLP toolkits like LingPipe or NLTK. or GATE. They use different data structures which reduces the time complexity.

The only difference between Lavenshtein and Damerau-Lavenshtein is that this algorithm doesn't supports transposition and we have only can perform only three operations namely, insertion, deletion and substitution.

Mathematically, it can be written as :-



LaTeX code for above equation

\qquad\operatorname{lev}_{s,t}(i,j) = \begin{cases}
  \max(i,j) \text{ if} \min(i,j)=0, \\
  \min \begin{cases}
          \operatorname{lev}_{s,t}(i-1,j) + 1 \\
          \operatorname{lev}_{s,t}(i,j-1) + 1 \\
          \operatorname{lev}_{s,t}(i-1,j-1) + 1_{(s_i \neq t_j)}
       \end{cases} \text{ else,}
\end{cases}

Another significant difference between Levenshtein Distance and Damerau-Levenshtein Distance is that the former satisfies the Triangle Inequality whereas the latter does not. It is visible from the image given below where the numbers represent the edit distance between two nodes :-

Okay, lets do some programming now. We can follow the above math relation an we can formulate a recursive solution :-

def recursivelevenshtein(s1, len_s1, s2, len_s2):
    """
    A function to calculate the Levenshtein Distance using recursion.

    :param s1: Parent String
    :param len_s1: Length Of s1
    :param s2: pattern String
    :param len_s2: length of s2
    :rtype : int returns the levenshtein distance
    """
    if len_s1 == 0:
        return len_s2
    if len_s2 == 0:
        return len_s1
    return min(
        recursivelevenshtein(s1, len_s1 - 1, s2, len_s2) + 1,  # Deletion
        recursivelevenshtein(s1, len_s1, s2, len_s2 - 1) + 1,  # Insertion
        recursivelevenshtein(s1, len_s1 - 1, s2, len_s2 - 1) + (s1[len_s1 - 1] != s2[len_s2 - 1])  # Substitution
    )

Full Source :- https://github.com/AnimeshShaw/TextMin...nshtein.py
See Execution :- http://code.hackerearth.com/4ac2f4u

(I haven't given the visualize python link here, but you can do it by yourself)

I hope you understood the process. But this backtracking technique is very very inefficient for larger strings. As the complexity is cubic. The space complexity is large. In order to improve the performance we use the Wagner-Fisher algorithm.

Wagner-Fisher Algorithm
Wagner-Fisher algorithm presents before us a Dynamic Programming approach to solve the Levenshtein Distance Problem in Quadratic time, O(m * n) where m and n are the length two input strings respectively.

Lets get to the code already :-

def wagner_fisher(s1, len_s1, s2, len_s2):
    """
    A function to calculate the Levenshtein Distance using
    Wagner-Fisher Algorithm.

    :param s1: Parent String
    :param len_s1: Length Of s1
    :param s2: pattern String
    :param len_s2: length of s2
    :rtype : int returns the levenshtein distance
    """
    d = {}

    #Make to use O(min(n,m)) space
    if len_s1 > len_s2:
        s1, s2 = s2, s1
        len_s1, len_s2 = len_s2, len_s1
    for i in range(-1, len_s1 + 1):
        d[(i, -1)] = i + 1
    for j in range(-1, len_s2 + 1):
        d[(-1, j)] = j + 1

    for i in range(len_s1):
        for j in range(len_s2):
            if s1[i] == s2[j]:
                d[i, j] = d[i - 1, j - 1]
            else:
                d[(i, j)] = min(
                    d[(i - 1, j)] + 1,  # deletion
                    d[(i, j - 1)] + 1,  # insertion
                    d[(i - 1, j - 1)] + 1,  # substitution
                )
    return d[len_s1 - 1, len_s2 - 1]

Full Source :- https://github.com/AnimeshShaw/TextMin...-Fisher.py
See Execution :- http://code.hackerearth.com/75dc9aJ

(I haven't given the visualize python link here, but you can do it by yourself)
Lets See how the lookup matrix is generated for the strings : "LAD" and "LLUDO"
Here's the final Matrix, just follow the algorithm step[ by step and you should get the following matrix, if you don't then you have made some mistakes.

+------+---+---+---+---+
| S1 → |   | L | A | D |
+------+---+---+---+---+
| S2 ↓ | 0 | 1 | 2 | 3 |
+------+---+---+---+---+
| L    | 1 | 0 | 1 | 2 |
+------+---+---+---+---+
| L    | 2 | 0 | 1 | 2 |
+------+---+---+---+---+
| U    | 3 | 1 | 1 | 2 |
+------+---+---+---+---+
| D    | 4 | 2 | 2 | 2 |
+------+---+---+---+---+
| O    | 5 | 3 | 3 | 3 |
+------+---+---+---+---+

The bottom-right element gives the Levenshtein edit distance. This process is better than recursive. Now my question to you is that, Can we improve further. Well I leave this to you. Find a method and write a code to find the edit distance using this algorithm which performs better than the present algorithm. Comment in the section below.

Using NLTK

If you're using NLTK the you can calculate it using :-

Conclusion

So, all folks we have come to an end of our first encounter with text Processing. If you made it till the end you probably have understood a nice algorithm. I will be posting more articles on these topics as soon as I have time. If you have some doubt then please ask but before that go through the tutorial properly and execute the codes by yourself and practice them by yourself.

References

http://en.wikipedia.org/wiki/Levenshtein_distance
http://en.wikipedia.org/wiki/Wagner-Fischer_algorithm
http://en.wikipedia.org/wiki/Fuzzy_string_searching
http://en.wikipedia.org/wiki/Damerau%E2%...n_distance
https://www8.cs.umu.se/kurser/TDBA77/VT0...NODE46.HTM

Online Tools Used

Online LaTeX : http://www.sciweavers.org/free-online-latex-equation-editor
Draw Diagrams : https://www.draw.io/
Text Table Generator : http://www.tablesgenerator.com/text_tables

Credits to @XxGreenLanternxX to header banner image.


Thank you,
Sincerely,
Psycho_Coder.

Related articles

• A Gentle Introduction to TextBlob for NLP
• String Permutations Algorithm with example code in C/C++
• Comma as Sequential-Evaluation Operator in C/C++
• Insertion Sort using Binary Search in Python
• Hash Algorithm Identifier | Identify CryptoHash Types
• Math Expression Parser in Java

Continue Reading →

Scraping List of all Mangas with Link in Python

Snapshot of a text file saved under the comma-separated value format (Photo credit: Wikipedia)

I wrote a little script that scrapes all the Manga names and their respective homepage URLs as a CSV file.

Here's the code :-

from lxml.html import parse

__author__ = 'Psycho_Coder'


def main():
    with open("mangalist.csv", "w") as f:
        tree = parse("http://www.mangapanda.com/alphabetical")
        manga_name_list = tree.xpath("//ul[@class='series_alpha']/li/a/text()")
        manga_url_list = tree.xpath("//ul[@class='series_alpha']/li/a/@href")
        f.write("\"Manga Name\", URL\n")

        for i in range(len(manga_name_list)):
            f.write("\"{0}\", http://www.mangapanda.com{1}\n".format(manga_name_list[i].replace("\"", ""), manga_url_list[i]))

if __name__ == "__main__":
    main() 

CSV : https://github.com/AnimeshShaw/MangaScrapper/blob/master/resources/mangalist.csv
Code On Github https://github.com/AnimeshShaw/MangaScrapper/blob/master/resources/MangaList.py


I hope you liked this script. Please share this post in your social network :)

Related articles

• Math Expression Parser in Java
• Hash Algorithm Identifier | Identify CryptoHash Types
• Visual Stack Application in Java
• Learning About the Stack Data Structure
• Image to ASCII ART converter in Java
• A Gentle Introduction to TextBlob for NLP

Continue Reading →

A Gentle Introduction to TextBlob for NLP

English: Python logo Deutsch: Python Logo (Photo credit: Wikipedia)

TextBlob is a Python (2 and 3) library for processing textual data. It provides a consistent API for diving into common natural language processing (NLP) tasks such as part-of-speech tagging, noun phrase extraction, sentiment analysis, and more. TextBlob objects can be treated as if they were Python strings that learned how to do Natural Language Processing. TextBlob heavily depends on Python NLTK and pattern module by CLIPS. Corpora used by NLTK is the default corpora for TextBlob as well. For Installation insturctions Click Here

Today we will have an overview of this library in this part and our main focus will be to cover the different properties and methods of BaseBlob class.

Basic's of TextBlob and Tokenization

---

First we import the TextBlob class which can be said as the most important class.

In [2]:

# We import the most important class TextBlob
from textblob import TextBlob

We will analyse and apply all the methods and functions of TextBlob on the following paragraph and sometimes with some additional sentences.

In [4]:

data = """
Hello, My name is Animesh Shaw and I am an undergraduate and studying Computer Science (upcoming Graduation in 2015). I Love programming and Computer science subjects of topics. 
My field of interest include Computational Linguistics. I Love watching anime specially One Piece and Naruto Shippunden. Animes specially those two shows
an extravagant amount of dedication, passion love, and amibition towards achieving one's goal and aim's in life. These have always inpired me a lot.
Giving up on your own dreams to fulfill others and the same feeling that other carry along with friends or something which I call as an eternal bond.
I recommend everyone to watch Naruto and One Piece. I have learnt a lot from there. "People are not always born intelligent or powerfull but with hard work
great heights can be achieved in life." Yes its true, dedication and hard work are the key goals to success less than 1% people are born with the blessing
of being a prodigy. The world was shaken mostly by those non-prodigy people which have had a massive impact in every individuals lives. 
With great goals and constant dedication and passion you can achieve 
the unachievable.
"""

To use any functions or methods of TextBlob we first create a TextBlob object

In [5]:

tblob = TextBlob(data)

We will store all the words of the paragraph along with their POS tags in a variable tags. tags is a property in TextBlob class which returns a list of tuples. The tuple format being (, ). All strings or return values in TextBlob are unicode encoded.

In [6]:

tags = tblob.tags #We have stored the words of the text with the respective parts of speech tags

In [8]:

tags[:6] # Now that the tags are stored we will display the first 6 tags.

Out[8]:

[(u'Hello', u'UH'),
 (u'My', u'PRP$'),
 (u'name', u'NN'),
 (u'is', u'VBZ'),
 (u'Animesh', u'NNP'),
 (u'Shaw', u'NNP')]

NNP stands for proper noun. It is used for name, place, animals etc. etc. PRP stands for Pronoun.

Now lets prints all the tags which was earlier stored in tags variable. We will just print the first 20 tags.

In [12]:

for tag in tags[:20]:
    print(str(tag[1]) + " ")

UH 
PRP$ 
NN 
VBZ 
NNP 
NNP 
CC 
PRP 
VBP 
DT 
JJ 
CC 
VBG 
NNP 
NNP 
JJ 
NNP 
IN 
CD 
PRP 

We can do the above by writing a single line of code too.

In [15]:

print("\n".join([tag[1] for tag in tags[:20]]))

UH
PRP$
NN
VBZ
NNP
NNP
CC
PRP
VBP
DT
JJ
CC
VBG
NNP
NNP
JJ
NNP
IN
CD
PRP

Now let us have a look at all the tags in the data above

In [16]:

print(" ".join([tag[1] for tag in tags]))

UH PRP$ NN VBZ NNP NNP CC PRP VBP DT JJ CC VBG NNP NNP JJ NNP IN CD PRP NNP NN CC NNP NN NNS IN NNS PRP$ NN IN NN VBP NNP NNP PRP NNP VBG NN RB CD NNP CC NNP NNP NNP RB DT CD VBZ DT JJ NN IN NN NN NN CC NN IN VBG CD POS PRP NN CC NN POS PRP IN NN DT VBP RB VBN PRP DT NN VBG IN IN PRP$ JJ NNS TO VB NNS CC DT JJ NN IN JJ VB IN IN NNS CC NN WDT PRP VB IN DT JJ NN PRP VB NN TO VB NNP CC CD NNP PRP VBP NN DT NN IN EX NNS VBP RB RB VBN JJ CC NN CC IN JJ NN JJ NNS MD VB VBN IN NN UH PRP$ JJ NN CC JJ NN VBP DT JJ NNS TO NN JJR IN CD NNS VBP VBN IN DT NN IN VBG DT NN DT NN VBD VBN RB IN DT JJ NNS WDT VBP VBD DT JJ NN IN DT NNS NNS IN JJ NNS CC JJ NN CC NN PRP MD VB DT JJ

In [17]:

# Now let us have a look at the total no of tags. We store all the tags in a variable named pos_tags
pos_tags = [tag[1] for tag in tags]
#Now we will print the length
print("No. of tags : " + str(len(pos_tags)))

No. of tags : 199

In [21]:

#if you have noticed in entry no. 16 that a lot of tags are repeating. We would like to get all the unique tags from them.
#We can simply use he set() data structure to do so which will remove the duplicates.
unique_poses = set(pos_tags)
print(" ".join([ i for i in unique_poses ]))
print("\nNo of unique POS's : " + str(len(unique_poses)))

PRP$ VBG VBD VBN VBP WDT JJ VBZ DT NN POS TO PRP RB NNS NNP VB CC CD EX IN MD JJR UH

No of unique POSes : 24

So now you can see that there are only 24 POS tags which have been used and the rest are just repetition. Using TextBlob we can even print all the noun phrases in the sentence.

In [23]:

# print all the noun phrases
tblob.noun_phrases

Out[23]:

WordList(['hello', u'animesh shaw', 'computer', 'graduation', 'love', 'computer', u'science subjects', u'computational linguistics', 'love', 'piece', u'naruto shippunden', 'animes', u"'s goal", u"aim 's", u'own dreams', u'eternal bond', 'naruto', 'piece', u'hard work', u'great heights', u'hard work', u'key goals', u'% people', u'non-prodigy people', u'massive impact', u'great goals', u'constant dedication'])

We can get all the words as a WordList as well, by using the words property as follows which returns a list of all words as a class of WordList. WordList is a list-like collection of words. Its no different from Python lists but with additional methods.

In [26]:

tblob.words #returns the data as word tokenized form in a list.

Out[26]:

WordList(['Hello', 'My', 'name', 'is', 'Animesh', 'Shaw', 'and', 'I', 'am', 'an', 'undergraduate', 'and', 'studying', 'Computer', 'Science', 'upcoming', 'Graduation', 'in', '2015', 'I', 'Love', 'programming', 'and', 'Computer', 'science', 'subjects', 'of', 'topics', 'My', 'field', 'of', 'interest', 'include', 'Computational', 'Linguistics', 'I', 'Love', 'watching', 'anime', 'specially', 'One', 'Piece', 'and', 'Naruto', 'Shippunden', 'Animes', 'specially', 'those', 'two', 'shows', 'an', 'extravagant', 'amount', 'of', 'dedication', 'passion', 'love', 'and', 'amibition', 'towards', 'achieving', 'one', "'s", 'goal', 'and', 'aim', "'s", 'in', 'life', 'These', 'have', 'always', 'inpired', 'me', 'a', 'lot', 'Giving', 'up', 'on', 'your', 'own', 'dreams', 'to', 'fulfill', 'others', 'and', 'the', 'same', 'feeling', 'that', 'other', 'carry', 'along', 'with', 'friends', 'or', 'something', 'which', 'I', 'call', 'as', 'an', 'eternal', 'bond', 'I', 'recommend', 'everyone', 'to', 'watch', 'Naruto', 'and', 'One', 'Piece', 'I', 'have', 'learnt', 'a', 'lot', 'from', 'there', 'People', 'are', 'not', 'always', 'born', 'intelligent', 'or', 'powerfull', 'but', 'with', 'hard', 'work', 'great', 'heights', 'can', 'be', 'achieved', 'in', 'life', 'Yes', 'its', 'true', 'dedication', 'and', 'hard', 'work', 'are', 'the', 'key', 'goals', 'to', 'success', 'less', 'than', '1', 'people', 'are', 'born', 'with', 'the', 'blessing', 'of', 'being', 'a', 'prodigy', 'The', 'world', 'was', 'shaken', 'mostly', 'by', 'those', 'non-prodigy', 'people', 'which', 'have', 'had', 'a', 'massive', 'impact', 'in', 'every', 'individuals', 'lives', 'With', 'great', 'goals', 'and', 'constant', 'dedication', 'and', 'passion', 'you', 'can', 'achieve', 'the', 'unachievable'])

See its so easy. Noun Phrases gives us a lot of important and relevant information which can be further used to analyse the meaning. When we use the tblob.noun_phrases it returns the noun_phrases as an WordList which a class used to store words and manipulate them or operate with different functions etc. etc. Lets do something more.

Language Detection and Translation

---

Suppose that you want to detect the language used in the text above, TextBlob provides a detect_language() method to detect language used. The methods uses the Google Langauge Translate API for the purpose.

In [26]:

tblob.detect_language()

Out[26]:

u'en'

Lets try some more and in different ways.

In [31]:

#Okay lets try some more.
TextBlob("Bonjour").detect_language()

Out[31]:

u'fr'

In [36]:

#Another one
TextBlob("Ciao").detect_language()

Out[36]:

u'it'

In the last two "fr" stands for french and "it" stands for italian. Now lets move on. Now that you know that TextBlob can detect language. You might have a question whether it can even do the translation or not. As a matter of fact it can Lets take a simple example we will Convert "Thanks" in english to Japanese.

In [38]:

TextBlob("Thanks").translate(to="ja")

Out[38]:

TextBlob("感謝")

You can see we got the translated text in Japanese.

Lets try another example with a bigger sentence

In [41]:

TextBlob("Hello, My name is Animesh P Shaw. I will become the Programming King").translate(to="fr")

Out[41]:

TextBlob("Bonjour , Mon nom est Animesh P. Shaw . Je vais devenir le roi de programmation")

You might be having a doubt whether these returned values are true or not. Well you can always Google you know. Lets see if the last french translated sentence is detected as french or not.

In [42]:

TextBlob("Bonjour , Mon nom est Animesh P. Shaw . Je vais devenir le roi de programmation").detect_language()

Out[42]:

u'fr'

Ta da! The langauge of the above sentence has been detected as french since fr is the french langauge code. ## Raw Text Handling
Let's explore more and see what we have got. We will print the complete text as raw which means all the escape characters like or or * will also be printed. There is an builtin property for that purpose.

In [44]:

tblob.raw

Out[44]:

'\nHello, My name is Animesh Shaw and I am an undergraduate and studying Computer Science (upcoming Graduation in 2015). I Love programming and Computer science subjects of topics. \nMy field of interest include Computational Linguistics. I Love watching anime specially One Piece and Naruto Shippunden. Animes specially those two shows\nan extravagant amount of dedication, passion love, and amibition towards achieving one\'s goal and aim\'s in life. These have always inpired me a lot.\nGiving up on your own dreams to fulfill others and the same feeling that other carry along with friends or something which I call as an eternal bond.\nI recommend everyone to watch Naruto and One Piece. I have learnt a lot from there. "People are not always born intelligent or powerfull but with hard work\ngreat heights can be achieved in life." Yes its true, dedication and hard work are the key goals to success less than 1% people are born with the blessing\nof being a prodigy. The world was shaken mostly by those non-prodigy people which have had a massive impact in every individuals lives. \nWith great goals and constant dedication and passion you can achieve \nthe unachievable.\n'

raw_sentances is another property which returns a list of raw sentences which means all the escape characters like or or will also be printed

In [45]:

tblob.raw_sentences #

Out[45]:

['\nHello, My name is Animesh Shaw and I am an undergraduate and studying Computer Science (upcoming Graduation in 2015).',
 'I Love programming and Computer science subjects of topics.',
 'My field of interest include Computational Linguistics.',
 'I Love watching anime specially One Piece and Naruto Shippunden.',
 "Animes specially those two shows\nan extravagant amount of dedication, passion love, and amibition towards achieving one's goal and aim's in life.",
 'These have always inpired me a lot.',
 'Giving up on your own dreams to fulfill others and the same feeling that other carry along with friends or something which I call as an eternal bond.',
 'I recommend everyone to watch Naruto and One Piece.',
 'I have learnt a lot from there.',
 '"People are not always born intelligent or powerfull but with hard work\ngreat heights can be achieved in life."',
 'Yes its true, dedication and hard work are the key goals to success less than 1% people are born with the blessing\nof being a prodigy.',
 'The world was shaken mostly by those non-prodigy people which have had a massive impact in every individuals lives.',
 'With great goals and constant dedication and passion you can achieve \nthe unachievable.\n']

Let us look at another property which is sentences. Now this is different from raw_sentences. The former will return a list of all the sentences of class Sentence(). We will have a look at it.

In [46]:

tblob.sentences

Out[46]:

[Sentence("
Hello, My name is Animesh Shaw and I am an undergraduate and studying Computer Science (upcoming Graduation in 2015)."),
 Sentence("I Love programming and Computer science subjects of topics."),
 Sentence("My field of interest include Computational Linguistics."),
 Sentence("I Love watching anime specially One Piece and Naruto Shippunden."),
 Sentence("Animes specially those two shows
an extravagant amount of dedication, passion love, and amibition towards achieving one's goal and aim's in life."),
 Sentence("These have always inpired me a lot."),
 Sentence("Giving up on your own dreams to fulfill others and the same feeling that other carry along with friends or something which I call as an eternal bond."),
 Sentence("I recommend everyone to watch Naruto and One Piece."),
 Sentence("I have learnt a lot from there."),
 Sentence(""People are not always born intelligent or powerfull but with hard work
great heights can be achieved in life.""),
 Sentence("Yes its true, dedication and hard work are the key goals to success less than 1% people are born with the blessing
of being a prodigy."),
 Sentence("The world was shaken mostly by those non-prodigy people which have had a massive impact in every individuals lives."),
 Sentence("With great goals and constant dedication and passion you can achieve 
the unachievable.
")]

Sentiment Analysis with TextBlob

---

TextBlob is specially helpful for Sentiment Analysis with all the built in methods and properties which you can modify by configuring and extend with different taggers or Analyzers.

What is Senitiment Analysis ?

Sentiment analysis (also known as opinion mining) refers to the use of natural language processing, text analysis and computational linguistics to identify and extract subjective information in source materials. With TextBlob we can see both the Polarity and Subjectivity of the information in a sentence or data.
Now lets do something interesting and important. Note the following produces important results. We will now see how to measure the polarity of a sentence. Now what is polarity. Polarity is a measure which gives a numerical value depending on which we can understand whether a sentence is postive or negetive. Its more like someone says bad about you feel sad and it means negetive and when someone praises you, you feel joy which is positive polarity.

In [20]:

for sent in tblob.sentences:
    print(sent.sentiment.polarity)

0.0
0.5
0.0
0.428571428571
0.428571428571
0.0
0.158333333333
0.0
0.0
0.436111111111
0.0383333333333
0.25
0.4

A value of 0.0 indicates neutral, 0.5 indicates positive. Note that the word "Love" indicates postiveness. Values which fall in between 0.4 and 0.5 are almost undecidatble or more or less positve. Let's consider the second last value 0.25, it is low because of the words shaken or massive impact which infuses a negetive sense.
Now lets display both the polarity and subjectivity. The sentiment property returns a namedtuple of the form Sentiment(polarity, subjectivity). The polarity score is a float within the range [-1.0, 1.0]. The subjectivity is a float within the range [0.0, 1.0] where 0.0 is very objective and 1.0 is very subjective.

In [21]:

for sent in tblob.sentences:
    print(sent.sentiment)

Sentiment(polarity=0.0, subjectivity=0.0)
Sentiment(polarity=0.5, subjectivity=0.6)
Sentiment(polarity=0.0, subjectivity=0.0)
Sentiment(polarity=0.4285714285714286, subjectivity=0.5857142857142856)
Sentiment(polarity=0.4285714285714286, subjectivity=0.5857142857142856)
Sentiment(polarity=0.0, subjectivity=0.0)
Sentiment(polarity=0.15833333333333333, subjectivity=0.5)
Sentiment(polarity=0.0, subjectivity=0.0)
Sentiment(polarity=0.0, subjectivity=0.0)
Sentiment(polarity=0.4361111111111111, subjectivity=0.7305555555555555)
Sentiment(polarity=0.03833333333333332, subjectivity=0.45166666666666666)
Sentiment(polarity=0.25, subjectivity=0.75)
Sentiment(polarity=0.4, subjectivity=0.5416666666666666)

---

Dumping Data Properties as JSON

Suppose that you want to get all the properties together as one in some format which is efficient and easy to parse. To solve such cases TextBlob provides a way to dump all the properties as a JSON file. For this example we will create a text blob instance with a smaller sentence "Nico Robin is the most sexy anime character I have ever encountered."

In [39]:

blob = TextBlob("Nico Robin is the most sexy anime character I have ever encountered.")
print(blob.json)

[{"polarity": 0.5, "stripped": "nico robin is the most sexy anime character i have ever encountered", "noun_phrases": ["nico robin", "sexy anime character"], "raw": "Nico Robin is the most sexy anime character I have ever encountered.", "subjectivity": 0.75, "end_index": 68, "start_index": 0}]

If you want to represent the JSON data in a serialized manner then you can do this in the following manner.

In [25]:

blob.serialized

Out[25]:

[{u'end_index': 68,
  u'noun_phrases': WordList([u'nico robin', u'sexy anime character']),
  u'polarity': 0.5,
  u'raw': 'Nico Robin is the most sexy anime character I have ever encountered.',
  u'start_index': 0,
  u'stripped': 'nico robin is the most sexy anime character i have ever encountered',
  u'subjectivity': 0.75}]

Now lets test something nice. Suppose that we add the following " and beautiful lady " after "sexy" in In [39] then what changes do you expect to happen. As you know that beautiful is a postive word and so what it does is it increases the polarity value. This technique can be used in different ways in research.

In [40]:

blob = TextBlob("Nico Robin is the most sexy and beautiful lady anime character I have ever encountered.")
print(blob.json)

[{"polarity": 0.6166666666666667, "stripped": "nico robin is the most sexy and beautiful lady anime character i have ever encountered", "noun_phrases": ["nico robin", "beautiful lady anime character"], "raw": "Nico Robin is the most sexy and beautiful lady anime character I have ever encountered.", "subjectivity": 0.8333333333333334, "end_index": 87, "start_index": 0}]

---

Summary

So thats all we have came to the end of our first encounter with TextBlob. As you can see I have explained the stuffs in a very detailed manner. This is definatly in more depth than what has been covered in the official tutorials. Stay tuned for more and I will continue this series and explain almost everthing in the TextBlob library. Next time we will discuss about Spelling Corrections, N-Grams, Taggers and maybe lemmatization.
Thank you for reading and I hope you have had a nice read.

Read this Article in IPython-Notebook format here.

 

Related articles

• Recommended Books to become a Java Hero from Java Zero
• String Permutations Algorithm with example code in C/C++
• Comma as Sequential-Evaluation Operator in C/C++
• Insertion Sort using Binary Search in Python
• List Of Steganography Tools
• How To copy a Very large file to a flash drive in Linux

Continue Reading →