r/learnpython•
greyoda

str.isalnum() not detecting non-alphanumeric characters?

I've written a program that reads a text file and outputs each word's frequency. To do so I needed to turn all words into lowercase and remove all non-alphanumerics. For that I used following code:

norm_word = ""
for char in word:
    if char.isalnum():
        norm_word += char

(At first I used the following, which I found on the internet but because I wasn't sure if it was causing my error I changed it for the above (it wasn't the cause btw))

''.join(char for char in word if char.isalnum())

Anyways, the program mostly works except that it doesn't remove all non-alphanumerics. For example:

For the test run I used Moby Dick from the Gutenberg Project. The output file contains the term "dome" twice and the term "dome;" once. Using CTRL + F in a text editor on the moby_dick text file I find the term "dome" three times.

Any idea why that ";" isn't being removed? Thanks!

Posted Jan 5, 2017, 11:22 PM UTC

18
Loading comments…