Automated statement labeling

[markwhiting]

Check how GPT labels statements on our labeling task. Use $Global R^2 = 1-\frac{mse(prediction,actual)}{mse(baseline,actual)}$ to score, and we can visualize in observable.

Would be nice to see how we do on each question.

[markwhiting]

Originally posted by @amirrr in Watts-Lab/commonsense-platform#86 (comment)

Dimensions of statement and their definition:

behavior

• Social: it refers to beliefs, perceptions, preferences, and socially con- structed rules that govern human experience; it can be “real” or opinion, but is intrinsically of human origins. e.g., I exist and am the same person I was yesterday. He yelled at me because he was angry. There are seven days in the week.
• Physical: it refers to objective features of the world as described by, say, physics, biology, engineering, mathematics or other natural rules; it can be measured empirically, or derived logically. e.g., Men on average are taller than women. The Earth is the third planet from the Sun. Ants are smaller than Elephants.

everyday

• Everyday: people encounter, or could encounter, situations like this in the course of their ordinary, everyday experiences, e.g., Touching a hot stove will burn you. Commuting at rush hour takes longer. It is rude to jump the line.
• Abstract: this claim refers to regularities or conclusions that cannot be observed or arrived at solely through individual experience, e.g., Capitalism is a better economic system than Communism. Strict gun laws save lives. God exists

figure_of_speech

• Figure of speech: it contains an aphorism, metaphor, hyperbole, e.g., Birds of a feather flock together. A friend to all is a friend to none.
• Literal language: it is plain and ordinary language that means exactly what it says. e.g. The sky is blue. Elephants are larger than dogs. Abraham Lincoln was a great president.

judgment

• Normative: it refers to a judgment, belief, value, social norm or convention. e.g., If you are going to the office, you should wear business attire,not a bathing suit. Treat others how you want them to treat you. Freedom is a fundamental human right.
• Positive: it refers to something in the world such as an empirical regularity or scientific law, e.g., hot things will burn you; the sun rises in the east and sets in the west.

opinion

• Opinion: it is something that someone might think is true, or wants others to think is true, but can’t be demonstrated to be objectively correct or incorrect; it is inherently subjective. e.g., FDR was the greatest US president of the 20th Century.. The Brooklyn Bridge is prettier than the Golden Gate. Vaccine mandates are a tolerable imposition on individual freedom.
• Factual: it is something that can be demonstrated to be correct or incorrect, independently of anyone’s opinion, e.g., the earth is the third planet from the sun (this is correct and we know it is correct), Obama was the 24th president of the United States (this is incorrect, but we know it’s incorrect). It will be sunny next Tuesday (we don’t yet know if this is correct, but we will be able to check in the future).

reasoning

• Knowledge: the claim refers to some observation about the world; it may be true or false, opinion or fact, subjective or objective e.g., The sun rises in the east and sets in the west. Dogs are nicer than cats. Glasses break when they are dropped.
• Reasoning: the claim presents a conclusion that is arrived at by combining knowledge and logic, e.g., The sun is in the east, therefore it is morning. My dog is wagging its tail, therefore it is happy. The glass fell off the table, therefore it will break and the floor will become wet.

Also:

category

Which knowledge category or categories describe this claim? (choose all that apply)

• General reference
• Culture and the arts
• Geography and places
• Health and fitness
• History and events
• Human activities
• Mathematics and logic
• Natural and physical sciences
• People and self
• Philosophy and thinking
• Religion and belief systems
• Society and social sciences
• Technology and applied sciences

[markwhiting]

Note, we should check labels against the original version of the statement, because cleaned statements might need different labels.

Once we have a good labeling strategy we should label all the new clean statements freshly #9

[markwhiting]

Non GPT based approach

Feature based model

leave one category out 
for each feature in [behavior ... ]: 
  training_data = data[category != LOO_category]
  test_data = data[category == LOO_category]
  model: feature ~ embedding on training_data
  predict: feature ~ embedding on test_data
  baseline: mode(feature) on training_data

Category based model

leave one design point out (a particular combo of features) 
multinomial regression: category ~ embedding
(same style testing regime)

Could try mean or mode for baseline.

Model type can be random forest or XGBoost

GPT approach

Just ask GPT the questions?

[amirrr]

Here's the data for the non-GPT approach (leaving out the category "Society and social sciences" since it resulted in the most accurate model).

Feature	Random Forest	XGBoost	GPT
behavior	-0.333	-1.475	0.097
everyday	0.092	0.476	0.056
figure_of_speech	0.0	0.156	0.105
judgment	0.034	-0.487	-0.097
opinion	-0.022	-0.126	0.165
reasoning	0.028	0.502	-0.128

[markwhiting]

Great, so we seem to need to do better, hahaha. Also, I think it's fine to trim these to 3 decimal places (e.g., -0.333), and we probably only need $R^2$.

Perhaps we can look at columns like (where each one shows the $R^2$ for that model for each feature):
Random forest
XGBoost
GPT

[markwhiting]

I edited your comment a bit more to indicate what I was thinking. (for some reason Github doesn't send notifications for edits)

[amirrr]

The table is complete now. I ran the GPT labeling against the first 2000 of the statements. Refer to this issue for more details about the prompt.

[markwhiting]

Thanks. Interesting. We're not doing very well.

Just so I understand, how are you doing the score calculation for GPT?

Would you mind making a second table that shows F1 scores for each of these as well?

[amirrr]

Feature	GPT Mean	GPT Mode
behavior	0.498	0.097
everyday	0.509	0.056
figure_of_speech	0.105	0.000
judgment	0.529	0.035
opinion	0.497	-0.022
reasoning	0.585	0.028

[markwhiting]

Interesting! Would you mind adding doing that for the others too? Just to see if our scores there get a lot better?

[amirrr]

These are the Jaccard accuracy, f1 and global $R^2$ (with baseline being average of scores) scores for Random Forest (RF) and XGBoost methods on labeling statements.

Feature	RF Jaccard Score	RF F1 Score	RF Global R-squared	XGBoost Jaccard Score	XGBoost F1 Score	XGBoost Global R-squared	GPT F1 Score	GPT Global R-squared
behavior	0.934	0.966	-0.497	0.950	0.950	-1.594	0.794	0.498
everyday	0.511	0.677	-0.733	0.674	0.674	0.083	0.791	0.509
figure_of_speech	0.028	0.054	-0.055	0.182	0.182	0.084	0.402	0.105
judgment	0.939	0.968	-0.031	0.963	0.963	-0.588	0.772	0.529
opinion	0.891	0.943	-0.184	0.940	0.940	-0.252	0.769	0.497
reasoning	0.438	0.609	-0.826	0.623	0.623	0.056	0.776	0.585

[markwhiting]

How interesting. So none of these is really good enough for everything, though most are OK on some of the features.

One more way we could look at this: for each of these samples, can you balance the data so that the training data have an equal number of each category for each feature.

After taking out a test split, take the smaller group and sample the larger group by the number of items the smaller group has.

[markwhiting]

@joshnguyen99 — can you also put updates here when you have them. Also, we have a project for this https://github.com/orgs/Watts-Lab/projects/27/views/5 and we can start tracking some of our efforts there.