banner-bg-min

Content analysis and text mining software for fast and precise processing of large amounts of unstructured information

banner-bg-min

Content analysis and text mining software for fast and precise processing of large amounts of unstructured information

previous arrow

next arrow

Sentiment dictionaries

View PDF version

What is sentiment analysis?

Automated sentiment analysis is an application of text analytics techniques for the identification of subjective opinions in text data. It commonly involves the classification of text into categories such as "positive", "negative" and in some cases "neutral". Over the final five years, we have seen a tremendous increment in demand for sentiment analysis tools by companies willing to monitor people's opinions of the visitor and on its products and services but also past social scientific discipline researchers. To fulfill the increasing demands for such tools, more and more researchers and companies are releasing products to perform sentiment assay, many of them claiming to be able to perform sentiment analysis of whatever type of certificate in every domain. Unfortunately, feel has shown united states of america that, an "out-of-the-box" sentiment assay tools working beyond domains does non yet exist. The chief reason sentiment analysis is and so difficult is that words often take different meanings and are associated with singled-out emotions depending on the domain in which they are being used. The use of a give-and-take like "fingerprints" may represent a major quantum in a criminal investigation but a major headache for smartphone manufacturers. "Freezing" is skilful for a refrigerator but pretty bad for software applications. You want the stock market or your car to exist "predictable" simply not necessarily the flick you are well-nigh to picket. In that location are even situations where different forms of a single word will be associated with different sentiments. For example, nosotros found in client feedback that the give-and-take "improved" was associated with positive comments, but "amend" was more often used in negative ones.

All sentiment assay tools rely, at varying degrees, on lists of words and phrases with positive and negative connotations or are empirically related to positive or negative comments. We have used such a listing in the by for sentiment assay tasks, all the same we have never fabricated our sentiment dictionary bachelor for several reasons. Such lists cannot be used equally is, but need to be customized to specific domains in order to provide reliable results. A lot of endeavour is needed to develop a domain-specific sentiment dictionary and to identify the proper vocabulary associated with the expression of positive and negative feelings. Many people are non necessarily willing to spend time performing such customization and validation tasks. They want something that they believe volition work right away and they would exist ready to pay a lot for such a tool.

We believe in that location is a adventure some people may use our sentiment dictionary equally is, without attempting to validate it or customize information technology to their ain type of information. Those who are aware of the limitations of such lists may still take no idea how such customization could be achieved and need some guidance. However, despite the potential misuse of sentiment analysis discussion lists, we have decided to make our WordStat Sentiment Lexicon available to the public. 1 of the reasons that made us change our minds was the publication of two articles. The offset of these, written by Loughran and McDonald (2011), stresses the danger of using dictionaries similar ours without any attempt to conform them to the intended domain, in their case accounting and fiscal news. Those researchers adult their own domain-specific sentiment dictionaries and describe, in some detail,, the process by which they selected words and validated their results. The second paper, published past Immature and Soroka (2011), too presents the structure and validation process of a sentiment dictionary simply this time customized for the analysis of political news. Both papers represent laudable efforts and are worth reading past anyone who would similar to learn how to create a context-specific sentiment analysis dictionary.

The Loughran and McDonald Financial Sentiment Dictionary

The Loughran and McDonald (2011) article provides a clear sit-in that applying a general sentiment word listing to accounting and finance topics tin atomic number 82 to a high rate of misclassification. They institute that nigh three-fourths of the negative words in the Harvard Four TagNeg dictionary of negative words are typically not negative in a financial context. For instance, words like "mine", "cancer", "tire" or "uppercase" are ofttimes used to refer to a specific industry segment. These words are non predictive of the tone of documents or of financial news and simply add noise to the measurement of sentiment and attenuate its predictive value. These authors created custom lists ofnegative andpositive words specific to the accounting and financial domain. Another benefit of the lexicon they propose is that it shows how quantitative content analysis can move beyond mere dichotomous differentiation typical of sentiment analysis and can also be used to measure additional dimensions of involvement. Ii noteworthy additions are theUncertainty discussion list that attempts to measure out the general notion of imprecision (without an explicit reference to risks), and theLitigiousness word list that may exist used to identify potential legal problem situations. They too includedWeak Modal andPotent Modal word lists. The following table illustrates the various categories of the Loughran and McDonald fiscal sentiment dictionary.

Loughran and McDonald Financial Sentiment Dictionary

Scale

No. of words

Sample words

Negative

ii,337

 termination, discontinued, penalties, misconduct, serious, noncompliance, deterioration, felony

Positive

353

 attain, attain, efficient, improve, assisting

Uncertainty

285

 approximate, contingency, depend, fluctuate, indefinite, uncertain, variability

Litigiousness

731

 claimant, deposition, interlocutory, testimony, tort

Weak Modal Words

27

 could, depending, might, perchance

Stiff Modal Words

19

 always, highest, must, will


Downloading the dictionary
The original version of the sentiment dictionary, too as a WordStat version of information technology, tin be downloaded here. Please note that the lexicon cannot be used for commercial purposes without potency. For more data on this sentiment lexicon or to become say-so for commercial employ, delight contact the authors at the higher up spider web page.

WordStat Sentiment Lexicon 2.0(01/26/2018)

The WordStat Sentiment Dictionary was actually designed by combining negative and positive words from the Harvard IV dictionary, the Regressive Imagery Lexicon (Martindale, 2003), and the Linguistic and Word Count dictionary (Pennebaker, 2007). The WordStat dictionary edifice utility program was then used to expand its word list by automatically identifying potential synonyms and related words as well as any inflected forms. We ended upward with more than 9526 negative and 4669 positive discussion patterns. Actually, sentiment is not measured with those two lists of words and word patterns but instead with two sets of rules that effort to accept into account negations that may precede those words. For example, negative sentiment is measured past using the following two rules:

  1. Negative words non preceded past a negation (no, not never) within iv words in the same sentence.
  2. Positive words preceded by a negation within iv words in the same judgement.

Positive sentiment is measured in a similar way by looking for positive words non preceded past a negation likewise equally negative terms following a negation. However, our own experiences suggest that this terminal dominion has less predictive value and may fifty-fifty slightly deteriorate the measurement of sentiments. But there may be some situations where such a rule could help predict positive sentiments. Nosotros decided to keep this last rule and let the user decide whether it should exist applied or disabled.

Downloading the Dictionary

Y'all tin can download the latest version of the WordStat Sentiment Dictionary from here.  To use the dictionary in WordStat, extract it in the My Provalis Research Projects\Dictionaries folder located in your master Documents folder.

Recommended Use

Nosotros DO Non RECOMMEND USING THIS DICTIONARY AS IS. We strongly believe that doing and then will not present very accurate results. We recommend instead customizing this lexicon past applying the following procedures:

REMOVE DOMAIN-SPECIFIC WORDS – Identify and remove frequent words that may be specific to your domain of interest and that unremarkably do non have positive or negative connotations. Reviewing all of those words may be time-consuming, then a more fourth dimension-efficient way to do this would be to apply this dictionary to a big set of documents in your domain area and identify words that appear frequently. You should so utilise the keyword-in-context features of WordStat to assess how those words are being used.

Identify WRONGFUL PREDICTIONS – If you have a set of documents that take already been categorized as positive or negative, or comprise satisfaction scores or whatsoever other writer-sentiment indicator, nosotros propose using the WordStat cross-tab feature to assess the correlation between frequent positive and negative words and those indicators. From such a list, pay close attention to any word that seems to be inversely related to the expected prediction. Using the keyword-in-context characteristic, examine how those words are existence used. If they are usually preceded past a negation (within three words), you can go on those words in the dictionary since WordStat contains rules that will accept those into account.

Add DOMAIN-SPECIFIC SENTIMENT WORDS AND PHRASES – Quite frequently there are specific words in your domain area that are used to refer to positive or negative aspects or features. For case, if you lot sell smartphones, items like "fingerprint," "noise", "drop" or "sound quality" may exist highly associated with positive or negative feedback. For car manufacturers, "bullheaded spot" "hard plastic" "chug" "whiplash" "billowy" or any mention of "air current" or "legs" may also exist related to specific opinions about a specific car. If you have admission to a collection of positive and negative evaluations, ane easy way to place those domain-specific words would exist to correlate the well-nigh frequent words with satisfaction scores and identify those that are highly predictive of negative and positive scores. There is, however, a trap to avoid when selecting those predictors based on their loftier correlation to satisfaction scores: The obtained sentiment measure may become insensitive to changes. For example, if many people complain about the poor sound quality of a cellphone, then the phrase "sound quality" will likely be highly predictive of negative comments. If in reaction to those evaluations the manufacturer releases a new version with improved audio quality, then any new positive comments about this improved audio quality may be wrongly classified as negative. This lack of sensitivity to changes is also a pitfall of many machine-learning approaches to sentiment analysis.

Nosotros volition be updating the WordStat Sentiment Dictionary from time to time. If you believe words or phrases are missing or if you identify whatever errors that should be stock-still to amend the dictionary's accuracy, please let united states of america know. Also, if you take developed any customized version of this lexicon, we would very much like to know about your efforts.

 Click here for more than data virtually WordStat
Download costless trial versions

References

Loughran, T. & McDonald, B. (2011). When is a liability not a liability? Textual Analysis, Dictionaries and 10-Ks. The Periodical of Finance, 66(i), 35-66.