How to determine/classify whether a given piece of string is a grammatically correct sentence or not?
Hi all! I'm a fresher and got lucky enough to join research as an assistant/intern. I have a task assigned to me that I thought was 'pretty basic', in terms of NLP, but I'm stuck now having no idea. I guess that's quite normal about interns :D
Let me explain the situation...
Context:
-
There's a dataset of scraped text from web pages, primarily cyber security related. Mostly, only the rendered text from each webpage is converted into a separate txt file.
-
Some txt files seem to include few HTML tags as well...
-
The researcher aims to train an NLP model to detect sentences that have important details regarding cyber threats.
-
For that to be achieved, we need a training dataset that has columns as follows
- needed
- Complete sentences/meaningful phrases that have information about cyber threats - unneeded
- Complete sentences/meaningful phrases that don't have information relevant to cyber threats.
- needed
My problem:
- As you might have guessed, I have to extract only the sentences from that scraped data.
- Individual words/grammatically wrong gibberish or other phrases that don't have meaning in this context such as 'About Us', and 'Contact Us' should be omitted.
- Text from any ads, punctuation marks, special symbols etc. should be omitted as well.
How do I achieve this? Are there any prebuilt frameworks that can do this with a prebuilt function, ideally like:
is_sentence(piece_of_string) -> Bool
Anything else would be of great help as well.
I've tried frameworks like Stanza, Spacy and NLTK. Though they have a sentence
class(?), it looks like it's purely based on punctuation alone, and not anything intelligent. Hence this question.
If I tried to achieve this using rules based on RegEx and punctuation, the output might include elements that should have been omitted.
Please help me figure this out. Your ideas would be of great help to me and help me advance in my career.
Posted Jan 4, 2023, 6:17 PM UTC