str.isalnum() not detecting non-alphanumeric characters?
I've written a program that reads a text file and outputs each word's frequency. To do so I needed to turn all words into lowercase and remove all non-alphanumerics. For that I used following code:
norm_word = ""
for char in word:
if char.isalnum():
norm_word += char
(At first I used the following, which I found on the internet but because I wasn't sure if it was causing my error I changed it for the above (it wasn't the cause btw))
''.join(char for char in word if char.isalnum())
Anyways, the program mostly works except that it doesn't remove all non-alphanumerics. For example:
For the test run I used Moby Dick from the Gutenberg Project. The output file contains the term "dome" twice and the term "dome;" once. Using CTRL + F in a text editor on the moby_dick text file I find the term "dome" three times.
Any idea why that ";" isn't being removed? Thanks!
Posted Jan 5, 2017, 11:22 PM UTC