3
$\begingroup$

I have a dataset which consists of like.. 400000 sentences and I want give each sentence to ChatGPT so it classifies each sentence as positive or negative. My question is, where can I find a reliable / trusted prompt to do that? and provide evidence that the prompt I used gives reliable labels for the problem?

If I just create my own prompt like.. "Please conduct a sentiment analysis on the following text and tell me if the sentiment expressed is positive or negative: [insert text here]"

How can I convince someone that my prompt was good? This is what I mean :/ So later no one can criticize like "Hey maybe you just used a bad prompt and ChatGPT could have performed better!" :c

I hope my question is clear.

Improve this question
$\endgroup$
2
  • $\begingroup$ Is there a reason not to use something like a RoBERTa model specifically trained on sentiment classification? Such a model would always return a machine-digestible output (versus ChatGPT, which would not necessarily), would probably have higher accuracy due to being trained on just this one task, and could be fine-tuned specifically for your domain (versus ChatGPT which is basically closed-source now). You could then feed that data to ChatGPT or another such chatbot for a human-readable summary if necessary. $\endgroup$
    – Obie 2.0
    Jul 13 at 23:08
  • $\begingroup$ You would also know that your inputs were not being used as training data for a public-facing model, so they could not be estimated from an attack on the model or sold for advertising to third parties or whatever. $\endgroup$
    – Obie 2.0
    Jul 13 at 23:31

2 Answers 2

Reset to default
3
$\begingroup$

The problem with using ChatGPT with sentiment analysis is exactly that; it can't be relied upon. But, we can increase our chances of correct output by using chain-of-thought prompting with few-shot examples.

If you don't know CoT prompting and few-shot prompting, please refer to this.

Now, for your case, you can use a prompt like this-

Classify the sentence as positive or negative.

Think step-by-step, and provide reasoning why the text is identified as positive or negative.

For example,

Sentence: 'Same S##t, different day'

Sentiment: The sentence uses s##t. S##t is publically accepted as vulgar or profane. So, the sentiment is Negative.

and

Sentence: 'This product has changed my life, I am so much more productive now'

Sentiment: The product that is being talked about has made the user more productive. Productivity is generally associated with positivity. So, the sentiment is Positive.

Sentence: <<Paste your own sentence here>>

Sentiment:

Here, we are asking ChatGPT to provide reasoning for its decision and also use our examples to remember how step-by-step reasoning works.

Now, while this should work well, it still will give bad results once in a while as ChatGPT is finetuned on human textual data and separately forced to accept some things as positive or negative.

Consequentially, a sentence that you consider as positive/negative might not be looked upon in the same way by ChatGPT.

UPDATE: According the comment, it is absolutely correct that even seemingly vulgar words can be used in a positive context, a modified prompt in that case will look like this-

Classify the sentence as positive or negative.

Think step-by-step, and provide reasoning why the text is identified as positive or negative.

For example,

Sentence: 'Same S##t, different day'

Context: The context seems to be that of frustration where the speaker is frustrated and/or angry at the metaphorical same s##t or mundane tasks.

Sentiment: The feeling is that of frustration and boredom about something. So, the sentiment is Negative.

and

Sentence: 'This product has changed my life, I am so much more productive now'

Context: The context seems to be that of admiration of a product that helped the user in some wah and made their life easier.

Sentiment: Productivity is generally associated with positivity. So, the sentiment is Positive.

Sentence: <<Paste your own sentence here>>

Sentiment:
Improve this answer
$\endgroup$
1
  • $\begingroup$ I mean... I definitely wouldn't train it that anything using vulgar language is negative... c.f. "this product is f###ing sh#t", vs "this product is the f###ing sh#t". :D $\endgroup$
    – neminem
    Jul 13 at 17:45
2
$\begingroup$ 
You are given the text of reviews delimited by triple quotes.  
Your task is to determine the sentiment of the review in the steps:
1, Identify a list of emotions that the writer expresses, not just positive nor negative.
2, Quote the phrases that are related to the identified emotions.
3, Give the sentiment of the review in a single word either "positive" or "negative" and verify that the sentiment is correct.
4, Format your sentiment in json. The permissible values of sentiment are "positive" and "negative".

"""I have mixed feelings about this fitness tracker. On one hand, it accurately tracks my steps and heart rate, and the app is easy to use. However, the battery life is not very long and I find it uncomfortable to wear for long periods of time. I also think it's overpriced for what it does. So, while it has some good features, I'm not sure if I would recommend it."""

Give it steps to take time to think to avoid jumping to conclusion. The outputs from step 1 and 2 can be used for spot check its accuracy.

It would be optional but safe to ask it to verify it for you.

Finally, provide you an easy way to parse the output.

The test result with ChatGPT-3.5 is provided for your reference:

  1. Emotions: mixed feelings, accuracy, ease of use, discomfort, overpriced, uncertainty
  2. Quotes: "mixed feelings about this fitness tracker" "accurately tracks my steps and heart rate" "the app is easy to use" "the battery life is not very long" "find it uncomfortable to wear for long periods of time" "overpriced for what it does" "while it has some good features, I'm not sure if I would recommend it"
  3. Sentiment: Negative
  4. Formatted sentiment: {"sentiment": "negative"}

The sentiment of the review is negative because while the fitness tracker has some good features such as accurately tracking steps and heart rate, and having an easy-to-use app, the negative aspects such as short battery life, discomfort, and overpricing outweigh the positive ones. Additionally, the uncertainty expressed by the reviewer about recommending the product further indicates a negative sentiment towards it.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .