Correct way to use a lemmatizer in package tm

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Correct way to use a lemmatizer in package tm

Ashim Kapoor
Dear All,

I want to do lemmatization using the tm package and textstem package.

The following is how I am doing it currently :-


filePath = < Path to any text file >
text <- readLines(filePath)
docs <- Corpus(VectorSource(text))

# Convert the text to lower case
docs <- tm_map(docs,content_transformer(tolower))

# Remove numbers
docs <- tm_map(docs, removeNumbers)
# Remove english common stopwords
docs <- tm_map(docs, removeWords, stopwords("english"))
# Remove punctuations
docs <- tm_map(docs, removePunctuation)
# Eliminate extra white spaces
docs <- tm_map(docs, stripWhitespace)
# Text Lemmatization
docs <- tm_map(docs, content_transformer(lemmatize_words))

My query : Is the above line the correct way to do lemmatization ? Can
someone please confirm?

For the sake of giving a complete example I am giving the following code as

dtm <- TermDocumentMatrix(docs)
m <- as.matrix(dtm)
v <- sort(rowSums(m),decreasing=TRUE)
d <- data.frame(word = names(v),freq=v)
head(d, 10)
wordcloud(words = d$word, freq = d$freq, min.freq = 1,
          max.words=200, random.order=FALSE, rot.per=0.35,
          colors=brewer.pal(8, "Dark2"))
Thank you,

        [[alternative HTML version deleted]]

[hidden email] mailing list -- To UNSUBSCRIBE and more, see
PLEASE do read the posting guide
and provide commented, minimal, self-contained, reproducible code.