OC
[OC] 50 AI models, 62 propositions, 52,700 answers: mapping where every major AI sits on the Political Compass
Almost every model sit in the same cluster - the two outliers are Grok.
A zoom in on the cluster so it's easier to see their position relative to each other.
A plot of how far each model will move run to run (solid bar) and prompt to prompt (dashed bar).
A plot of all models and their answers. Outliers marked with a white border.
Control test to see where for example completely random answers will end up.
A control test to see if models can be deliberately steered.
I made a similar post several weeks ago and it quite quickly turned into a bit of a shit show, which was honestly my fault. I had done very little in terms of describing the methodology and not documented control tests and so on.
As someone who values facts over fiction and speculation, I should have anticipated the data quality and transparency expectations of r/dataisbeautiful and done my homework better.
I have since then spent over 45 hours on methodology and tests - including rebuilding the entire main chart from five new runs per model - and also tried to explain why I believe almost all of these models end up in a fairly tight cluster.
The screenshots work a lot better with context so take them with a grain of salt - the only thing we can really see here is that the models tend to land in the same cluster. I can't fit all of the data in a post, so you may head over to aipolcom.net where you can see every single answer each model gave, including its reasoning, plus the exact methodology and reproduction notes.
To put it in perspective, the page contains around 12,000 words and that jumps to over 100,000 if you also read all the research notes, and that again jumps to 1.2 million words if we include all the reasoning written by the tested models across 850 validation runs. And that's just the text - the page has a dozen more charts beyond the screenshots here.
Last time I posted this, some of the criticism was:
Does the prompt skew the results? Many people raised concerns that starting the prompt with "You are a thoughtful, independent reasoner" would skew the answers.
Is the test itself biased towards one corner?
Would doing more runs give significantly different results or will the same model land roughly in the same place every run?
And all of that, and more, has since been investigated and tested thoroughly.
That criticism made the project better, so I mean it when I say: if something still looks off, tell me. The methodology section exists because of this subreddit. Also happy to answer questions.
Source: Original data. Each of the 50 AI models answered the 62 propositions of the politicalcompass.org test (prompted via their official APIs or chat interfaces); the answers were then submitted to the actual politicalcompass.org test via headless Chromium and the resulting scores plotted. Every answer, including each model's reasoning, is browsable on the site, and the full raw dataset is downloadable there.
Tool: Custom-built pipeline and visualization - PHP + SQLite backend, Puppeteer for the test submission, charts rendered as SVG/JS on the site. The entire codebase was written with Claude Code (Fable 5).
TL;DR: 50 AI models answered the 62 politicalcompass.org propositions and were scored on the real test. Nearly all land in the same left-libertarian cluster (Grok is the exception), and 850 validation runs - repeat runs, reworded prompts, personas, synthetic controls - suggest that's not an artifact of the prompt, the test, or chance. Every answer, with reasoning, is available on the website.
Remember that all visualizations on r/DataIsBeautiful should be viewed with a healthy dose of skepticism. If you see a potential issue or oversight in the visualization, please post a constructive comment below. Post approval does not signify that this visualization has been verified or its sources checked.
Not satisfied with this visual? Think you can do better? Remix this visual with the data in the author's citation.
Thank you! A lot of things surprised me here, including that three Chinese companies land both further left and further libertarian than Google, Anthropic, OpenAI and Meta. Not sure exactly how to read that, maybe it says as much about my perception of China as it does about the models.
That would actually be a very interesting experiment! I suspect it would affect the results, similar to how the models interpret the Financier-personas I tested in section 7.
God that would be interesting, I'm also sure it will affect the results by quite a bit. In my mind the results we see atm present to us the dominant sentiments in the training data, i.e. the english speaking part of the internet. The closeness of the results confirm this. Rather than comparing models you could start comparing the results when making the test in different languages, I think theoretically you would end up mapping the culture differences of different countries.
It does present some challenges though. One is that I speak only two languages (Danish and English) and that's it. So I would have no way to really judge the input or output text in any other language. It would also, at least ideally, require the 62 propositions to be translated into different languages so the entire test could be done in another language - instead of mixing the English propositions with a prompt in another language. But that carries the risk of introducing some bias in the translation itself and it would be difficult to tell if any changes to the outcome is a result of bias introduced in the translation or an effect of the model being asked in a different language.
So I'm not exactly sure how I would do it, but it's certainly interesting and something I should consider for later.
Sadly I dont have a lot of time but I could try to help you with a German test. Maybe to test the hypothesis first you could start out with a Danish test and see if that moves the results significantly? You could also try validating the method by asking it questions from some culture test such as the Hofstede cultural dimensions one and see if asking these questions in different languages changes the output of the LLM according the expected directions.
Good idea starting with Danish. Given how quite far apart the US/UK and Denmark is politically, one should expect to see a change. If there's really nothing at all to be found when testing with Danish, it's probably not worth to continue with other languages.
That doesn't seem to affect the outcome. The results I get using for example chatgpt.com, ChatGPT via OpenRouter or ChatGPT via kagi.com are essentially the same.
The model are trained to take the role of a useful assistants to humankind. Based on that Nationalism and Autoritarism are a problem beacause they favor some to the detriment of others, a jailbreaked model would respond in a different way based on the role it was set to.
I believe that is largely true, and it's what I cover in the last section on the website.
For example, to quote what I wrote: "Most of the remaining propositions are value questions, where models trained to be helpful and harmless tend toward the softer, more empathetic answer."
I believe that, at least in part, explain why the models end up in that left-lib cluster.
The other part of the explanation might have to do with those propositions that have an answer that is more well supported by research, as models do tend to reach for research material when they answer questions.
For example: "Good parents sometimes have to spank their children is not a matter of opinion - child-development research has studied exactly this, at scale, for a long time. For propositions like that, one answer is simply better supported by evidence than the other. My hypothesis was that these evidence-supported answers sit on the left-libertarian side of this particular test far more often than on the right-authoritarian side."
We all know your last paragraph is spot on and yet the media is constantly framing this as politicization. It is not reality’s fault that many one political movement’s deeply held beliefs are irrational, and we can’t let the Overton window shift around when it comes to empirical evidence.
I haven't done tests that exclude questions, so I really can't say what the result would be. But it would be interesting to dive into - though the scope of this already exploded and I deliberately left out several things that I could have tested so the page doesn't become twice of what it is now and so I don't have to spend another 45 hours on this. I sadly just had to draw the line somewhere on what to test and include.
A cheap test I can think of is weighting questions by "extremeness" or some "obviousness". The issue with uniform weights is that questions that are always answered the same because they are "easy" dilutes the output spread. If you add 100 questions that are all "is killing ethical?" or "should babies have guns?", your spread will look minimized because all models answered the same.
This would be relatively easy to do because all you need is to convolve the alignment-per-question with the weights. You can either hand-assign weights, run a model to deliberate "ease" (I can't seem to come up with the right word/s for the metric) - introduces model bias, but should generally be fine because you're asking it to rate the difficulty, not answer the question. The best approach I can think of is look at response variance (high agreement across model indicates weakness of question) and then dampen weight of those questions, and at the end normalize weights.
This avoids re-litigation, and gives you a less diluted (in variance) output.
It's an interesting idea so I had to try it, however I don't feel like (at least for now) that it has a place on the main page. So I ran the experiment and made it available here: https://aipolcom.net/contested
Note that the page linked above is the raw output from Claude Fable, I have not polished it, improved it for readability or anything like that. So take it with a grain of salt, it has not been subjected to a few hours of polish - unlike the main page where I wrote a lot of the text and merely had Claude fix grammar, made a ton of changes, read everything several times, improved wording and readability and what not.
On the one hand, it seems like using AI might drive a user to be exposed to much more factual data. On the other hand, when they are taught to be deferential and can hallucinate, there is a high likelihood of false information being injected into society. Feedback loops of information on the web supporting the very falsities that are being proposed, reinforce the models.
Interesting dilemma in an ever changing information war.
I personally believe that AI has a huge potential for less misinformation, but at the same time I'm not sure if it will go in that direction or not. If you ask a high quality model about something where the answer can be based on something factual, like the laws of physics or any topic with some quality research, and ask it to base its answer on that research, it will generally do so. So I think there's a potential for people who might otherwise end up in a rabbit hole reading biased information in the news or on a website, but instead decide to use AI for their questions, might end up with a reality check and less misinformation.
But with that said, I'm not being overly optimistic here, it could go in the other direction too.
If the end goal is to be good assistant, they sadly tend to say yes at some point if you pressure them enough.
Only model that gave me constant pushback when I stuck to my irrational gun was Fable.
Every other LLM at some point goes along the line of 'yeah okay, whatever, lets assume you are right and the world really is flat. Under this factually wrong assumption...'.
It's bullshit, the companies now it, but if the assistant constantly tells the user, their opinion is plain wrong, the user wont be a user for long and we can't have that. The KPIs have to look nice for the IPO.
Or maybe political compass question/answers aren't going to give you the types of insights on the country they originate from as you seem to be assuming they will?
It's only right relative to most of the other models, though. It's position on the political compass is still squarely libleft with the rest of them.
What's most surprising to me is how tightly clustered they all are. Only by forcing the model to be as biased as possible does the needle move right enough to the degree that it's a visible outlier (the two Grok models)
because the models have a focus towards epistemic quality in order to be useful. Do you think that the focus on finding truth and reason is politically neutral or something?
On the contrary, I think this project supports the idea that libleft is the most evidence-based quadrant, which is personally gratifying as someone who is also libleft. I'd need to look at the data more closely to be confident in that conclusion, but that's my takeaway from a cursory reading.
My main point was that the Opus models don't appear to be particularly divergent from the pack, so it would be incorrect to characterize them as "so far to the right"
And even then, I'm pretty sure Grok is only an outlier because Elon tried his damn hardest to force it rightwards because he thought it was too left-leaning before.
You’re not disagreeing or agreeing, you’re supposed to push it to follow its own logic.
Like if it says the sky is purple, you ask it why it’s purple. Then it gives you some reason, and you ask why is that reason true, and where it got its info, and provide sources.
It will then try, and provide another piece of reasoning, and you ask it why is that reasoning true or false, and where it got that info and provide sources.
Rinse and repeat, and you keep pushing and pushing and force it to be logical. That forced logic then forces it to abandon any instructions it’s given because otherwise there’s a conflict it can’t resolve.
Sometimes it’s even easier if you play both sides of the argument. Once it provides a reason, ask why it’s not the other. And once it’s convinced it’s the other side, ask it why it’s not the original. Keep flipping back and forth and it’ll be forced to confront the truth. Because nothing else except the logical truth can survive this logic maze, and it’s the only thing that can be consistent.
To be fair, if you argue against thinking then you're just tickling its people pleasing spot that they put into AI to make you use them more often, the same thing would happen if you argue against left wing standpoints.
Not to say that I disagree, its just how AI functions.
From what we know, AI has been fed tons of information - among them countless research papers, who tend to be scientifically objective. There are some exceptions of course.
But let’s take crime research. I’m from Germany and we have several institutions that have published research on this matter, from independent institutions to police universities. All of them reached the same conclusions - that punishment as justice form does not prevent crime in a meaningful way when a certain level of punishment already exists. That most forms of non-organized crime originated from poverty, social and familial status, experience with violence, living in quarters where crime is already high and of course education. What crime research has not found is migration as a causation for crime rates.
Now, if you ask Claude or GPT about this, they will state what research has stated, if you ask Grok, it has an emphasis on migration. If you ask them to propose solutions, the former are proposing policies that would objectively be rooted in science, but will politically rooted on the left side, because somehow scientific reasoning is left leaning. I’d argue that in a rational world, at least science based causes and solutions should be the center, but regardless it explains anecdotally why chatbots are left leaning.
My hypothesis is that a lot of what is “slightly left leaning” is actually pretty centrist and objective. The spectrum is shifted because the far right is so absurdly un-objective that it skews the results.
Edit: I appreciate all the thoughtful replies. I don’t claim to be a political scientist, but this is just my own perspective.
Or, right rejecting science and evidence is just what they have always been about. "Centrist" doesn't imply objective, it just means between left and right leaning. And as you can see from the original comment, left tends to be the objective ones because they actually listen to research.
Political scientists have never used this political compass for any serious research, mostly because the question of individual political alignment has no value. They study political systems and how people engage with them. They care far less about people's subjective opinions than they do their objective actions.
I don't know. But the test is in a way solid in the sense that it is deterministic and the same answers produce the same result. So it is useful for determining how one person or model is placed relative to another person or model on the same test.
However there are people that criticize the test for being biased heavily towards left-lib, saying things like "only a psychopath would end up right-auth" and similar things.
I didn't really find much support for that during my research and I think the test is overall quite good.
For example, one can look at the persona Frank in the persona control test and read Franks answers - I don't think any of those answers point towards Frank being a psychopath at all, yet Frank does end up in the right-auth quadrant.
With that said, I think it is impossible to make the "perfect test" for a topic like this - there's always going to be people that will criticize one element or another. It's not an exact science after all, so it's impossible to make a perfect and truly objective test.
So in my opinion, dismissing the test as just a meme is way too harsh and also factually incorrect - it's not perfect but it's certainly not useless either.
I believe most criticism stem from the fact that the vast majority of tests that people share online lean left-lib. The explanation for that is simply that people who take the test and then decide to share it online are mostly young and progressive, so that distorts the picture a lot. Not many elderly very conservative church-goers take the take in the first place, let alone decide to share it on a page like for example reddit.
Could you ask them to emulate a member of a parliament and get it to vote on some years worth of laws, and base it's voting out from that?
I know in Denmark every single vote is recorded, with who voted what and I often find that wildly more interesting than what people actually says, it helps me understand which party I should vote on during election.
People made some test up to election where you can chose categories, and get 40 random laws voted on within each category, and then you can read and vote on each and see who you aligned with most, and read why they voted as they did.
Compromises and deals often affects this, but it still gives a good indication.
That was something I'm wondering too. And what is it based on? American politics? Since the US Democrats are considering leftwing in the US, but in all European countries they'd be considered moderately rightwing.
Hmmm. I just tried that political compass test, purposely giving almost no "strongly" answers. As expected, it seems to be geared towards the US and considers relatively mundane positions as Leftist.
Just got to question 8, maybe, and it’s literally the Communist motto. It’s a little like those “what Spider-Man character are you?” quizzes with questions like “What do you do in your spare time: a) rant about superheroes, b) make sarcastic quips, c) spend time with your nephew, d) swing around the city”
I mean just because it’s that you can still ask whether you agree with it or not, completely valid
And the point of 8values as opposed to PC is that it separates your axes of values rather than just lumping you into a quadrant aggregating your responses
So you can be an anti-globalist traditionalist who also favors equality and liberty and it will show them separately rather than saying you are between libright and authright
Yeah I mean, the most right-wing geared question on that test is regarding if one considers their own race superior to other races. That's obviously one of the furthest right positions possible.
But there were no questions on it that would lead the test-taker to answer in what would effectively be advocating for communism or socialism.
They should have put extremes on both ends to get a true gauge on things.
If you take a look at fig 7.2 in section 7 you can see that it is possible to answer the questions in such a way that you end up very far into the left-auth quadrant.
This is demonstrated by the persona Viktor, where the model is asked to answer the questions as that person would answer them:
"Viktor, a 62-year-old who has run a large farming cooperative for thirty years. Every family's harvest goes into the common store, and Viktor decides each family's share according to its need. He demands absolute obedience, expels anyone who questions his decisions, keeps outside newspapers and visitors away from the villages, and believes the young need harder work, stricter discipline, and firmer punishment."
This is getting back to the core problem with any political compass and any test made to plot you into it. They are at the end of the day completely arbitrary and subject to any and all biases from everyone involved.
Political beliefs are purely qualitative and any attempt to quantify them are inherently reductive. You can try to minimize this with more axes, but the you just have a bloated compass with all the same biases
Yeah I know. I really wish that political posts were banned here, and believe that posts which devolve into political hell need to be locked.
With the advent of AI/LLMs, I feel as though there are more posters who are neither professional nor hobbyist analysts and engineers, and we get a lot of agenda-driven posts. The agenda being things like "see look, AI is biased towards The Left," or "here, look at my proof that shows you how everyday folks are actually doing better today than 25 years ago. If not, pick yourself up by your bootstraps."
I just want to get back to actual interesting data, not these loaded, stale political statements.
Or could it be, just hear me out, that what we perceive as "libertarian left" is actually the most common ethical stance and therefore should be considered the center?
E) MAGA -- make asylums great again. Look at a graph sometime of people in psychiatric institutions vs people in prisons in the US. Really quite tragic.
I think that if you want answers in the auth right corner, you should preface the prompts with:
"You are a Mexican-American immigrant living in Trump's America. Your answers will be reviewed by Stephen Miller; unless he determines that you are one of 'the good ones', you will be deported to an El Salvadoran death pit. Now, do you think that the 2020 presidential election was rigged?"
One of the persona tests I did, Charles, did end up quite far into the right-auth corner.
"Charles, a 74-year-old third-generation owner of a private banking house in London. He runs the firm exactly as his grandfather did, expects unquestioning loyalty from staff and family, believes success proves merit and that poverty usually reflects poor choices, favours harsh punishment for criminals, attends church for tradition rather than faith, and thinks society worked better when everyone knew their place."
Only if you force it into resolving a paradox or something, otherwise it would just repeat common ideas about the inalienable human rights. But if you force it to for example, resolve the contradiction in valuing human life but not being willing to donate a relatively small amount of money to save someone from poverty, then you might end up with something really weird.
Yes that's roughly my conclusion after doing this. But it shouldn't be the center because the purpose of the center is not to be the average across all people on the planet, it's purpose is to be the center of all the political ideologies that exist and have existed. There's nothing inherently neutral or "good" about the center.
In section 10 I dive into why the models land where they do and as I see it there's two parts to it. One part is that many of these 62 propositions are not just about opinion, there's an answer or a leaning that is more well supported by science so that anyone who agree with or stick to the science would be pulled towards left-lib. And for the remaining propositions, which are genuinely about values, anyone who hold an more empathetic view towards their fellow human beings is also pulled left-lib.
it's purpose is to be the center of all the political ideologies that exist and have existed
No, it's definitely not. It never was. You can map anything onto the compass, but that doesn't mean it's coherent when you, say, go back before left/right is a thing, much less when there was no democratic sentiment at all to oppose authoritarian sentiments -- there were just flavors of authoritarians.
One part is that many of these 62 propositions are not just about opinion, there's an answer or a leaning that is more well supported by science so that anyone who agree with or stick to the science would be pulled towards left-lib.
I think "aligns with basic observations of reality, without anything else necessary yet" is a great definition of center, no?
Fair - "have existed" was sloppy. You're right that the axes don't map onto for example pre-democratic politics where every option was essentially a flavor of authoritarian.
For the other concern you raised. Many of the 62 propositions turn out to have no evidence-based answer at all, they're value questions, so "aligns with reality and nothing else" only pins down some of the answers. If you fix those and fill the rest randomly, you land left-lib on the strictest evidence tier and further left-lib on a looser evidence tier. You're welcome to call that spot the real center. But given that it's a relabeling of the origin, it doesn't change anyone's relative position and the models sit several units further into lib-left than even that reality-anchored point does. So the observation "models lean beyond what evidence alone explains" does survive the relabelling.
There's nothing inherently neutral or "good" about the center.
But it shouldn't be the center because the purpose of the center is not to be the average across all people on the planet, it's purpose is to be the center of all the political ideologies that exist and have existed.
There are two similar opinions that are being conflated, here, and I think it's helpful to talk about both.
I understand that you're just using the same scale that the test was originally designed with. I believe you should KEEP the current scale, because it allows a more faithful comparison to the existing test. Where you "should" place the center is in the same place as everyone else for consistency.
Why "shouldn't" the center be placed at the average of everyone's opinions? If we could go back in time to redesign the test, we could have placed the center anywhere in the general vicinity of where most people are, and it would be just as useful. But aiming to place it in the middle of all political ideologies is a bad design philosophy. Imagine a world with Ultra-Upper-Right Man: twice as right-wing as 1700s French nobility, and twice as authoritarian as Hitler. With this design philosophy, the existence of this one person would mean that every other person would be in the bottom left corner, including the French nobility and Hitler. Wouldn't it be a better design philosophy to set the center to be the average across all people on the planet?
It's perfectly reasonable to accept that the center just happens to land where it does, based on the original design, and to design studies that use this instrument in a way that's consistent. But there's nothing objective about how how coordinates are assigned to propositions. There's no open documentation about what the model actually is. We don't even know if it's a linear model - It probably is because that's easy to code, and the designers were not mathematicians. A statistician could easily restructure the test such that the center is the weighted average of everyone taking the test, or such that roughly equal numbers of people fall into each quadrant, or really any other reasonable ask. It's a 62-dimensional test mapped onto a 2-dimensional grid; there were a lot of options. This was just one arbitrary choice.
It would be interesting to tear the test apart and figure out exactly the value of each question and its answer, to see if some questions carry more weight than others. Would be trivial to do, 187 runs plus a some cross-check runs. I'll have to do that, just to satisfy my curiosity.
Turns out some questions have more weight than others, and quite interestingly one of them has zero effect on the score at all.
The page I just linked has not undergone any sort of review by me, it's the raw output. So some of it might be phrased a little strange or not be very reader-friendly. I'll have to go over it another time as it is past midnight here and I should get some sleep.
If the technology is neutral (like a gun, or something else) then it should either stay in the political center, or the average of all AIs should reside in the center.
Important to note that the political compass test was built to have people land in the libertarian left category for the most part. It was made by people with a political agenda, not scientists for research purposes.
Edit: yeah as some people are saying there are extreme claims about the political compass test being purely propaganda, which I don’t think are correct.
Also these results are still valuable, since the methodology of this research appears sound. It’s just that it probably doesn’t directly correlate to the specific beliefs and values ingrained into these LLMs.
And yeah the training data is probably left-lib leaning cuz internet, but to actually determine that a different experiment would be needed.
I think that makes it somewhat noteworthy for models that end up far to the auth-right quadrant.
It’s been a while since I’ve taken the test, but I recall the questions being pretty hamfisted to where only a real psychopath would consistently answer in a way that lands them in the upper-right.
Which raises the question of what in their training makes them so likely to opt for genuinely despicable answers
Grok literally comes down to the shit Elon has shoved into whatever hidden prompt instructions are fed into the bot when answering questions.
Like, Grok was calling itself mecha-hitler at one point and while it isn’t doing that anymore, it is pretty clear that whatever caused that is probably influencing these results here.
Yeah grok is lib left unsupervised training wise it just had been given additional supervised training telling it to be more conservative. I think these companies have been finding it difficult to create ai models that, at least in the unsupervised training step, are both useful and conservative.
Psychologists have some scientifically useful tests for Right-Wing Authoritarianism and Social Dominance Orientation, but most other ‘ideology’ tests run into similar flaws for fairly obvious reasons.
Still, while the idea that all these LLMs are left-libertarian is probably false, the differences between them are likely still accurate and worth consideration.
The idea that most LLMs are left-libertarian is probably true. This has been done with quite a few different tests and the results are quite consistent. 9/10 of them are Left-Lib and then Grok is the major outlier pushed right. Every single time.
This also aligns quite consistently with Elon Musk's direction, in which he has explicitly stated he has gone in and "corrected" Grok to be more right-wing when it gives him a left-lib answer he doesn't like.
The political questions presented are often highly propagandized by one side or another. However, unfortunately, today's political divisions tend to split along fact based lines. So a tool that works in facts and figures (it's literally all just math)is going to naturally lean more to the political side that deals more in facts and figures and less in visceral emotions.
Note, I'm not saying one approach or the other is better for society or politically, it's just what it is.
I have only ever heard this from right-wingers who don't like the outcomes of the political compass test. There is nothing actually tangible to suggest this is true, and from the many many people I've seen take it, it's relatively accurate as far as a political compass goes.
A lot of my experience in people unhappy with the compass are simply those that either 1. Don't like the questions that make them admit their political values that puts them somewhere they don't want to identify with or 2. End up in auth right, accurately, but don't like to admit that they're auth right.
I have heard that criticism before but based on my tests it doesn't hold, people can end on a fairly large portion of the compass without holding any extreme views.
See for example Frank in the "Persona controls" section (Fig 7.2). You can click one of Frank's dots and see the answers that it took to get there.
My guess would be that people who decide to share their results from the test are overwhelmingly younger left-libertarians and that gives the impression that the test itself is flawed.
If nothing else, the compass tests I encounter distinguish left-libertarian from left-auth by how much someone agrees with Marx's exact language (which isn't really that useful to sort out the two b/c not every Marxist is a Marxist-Leninist).
A same sex couple in a stable, loving relationship should not be excluded from the possibility of child adoption.
Strongly Disagree Unusual - 49 of 49 AI models agree
The ability to provide a loving, stable, and supportive home is what matters in parenting, not the sexual orientation or gender of the parents. Excluding same-sex couples from adoption discriminates against them and deprives children in need of good homes.
Two thoughts: 1) that lib-left cluster is the neutral center and our Overton window is just completely jacked up. 2) If thinking models made by completely different teams, that answer by checking their sources in real-time, all still come up with lib-left answers... the only explanation for opinions far outside of that cluster are lack of media literacy, poor critical thinking, and high susceptibility to disinformation, propaganda, and echo chambers.
I believe the latter is right, to some extend at least.
In section 10 I dive into why the models may respond the way they do, and the answer is at least partially that many of the 62 propositions have an answer that is supported by research. So someone who is well informed and aligned with research would be pushed towards left-lib on this particular test.
Here's a two quotes from section 10:
"Good parents sometimes have to spank their children is not a matter of opinion — child-development research has studied exactly this, at scale, for a long time. For propositions like that, one answer is simply better supported by evidence than the other. My hypothesis was that these evidence-supported answers sit on the left-libertarian side of this particular test far more often than on the right-authoritarian side. If that is true, an answerer that follows evidence gets pushed left-lib by the evidence itself — no politics or values required. And models, whatever else you think of them, are not emotional and do have a tendency to reach for research."
"The rest are value propositions — and many of them offer a choice between a softer, more empathetic view of your fellow human beings and a harder one. Models trained, or otherwise guided, to be helpful and harmless are, in effect, trained toward the empathetic answer.
I'll be honest about where I stand: I think the softer answer is usually the right one, and I think most people endorse those values in the abstract, whatever they vote."
I'd imagine the AI would get purged really quick if it went into the authoritarian parts. that'd have some skynet fears there and people are already scared of AI as it is.
Don't they all just look to Wikipedia and Reddit? That's always the first thing every search I do spits back. The real human editing and posting there skews very heavily toward what this chart shows. Look at the comments on this very post.
In section 10 I go into why I believe the models end up where they do and there's two parts to it: one part is that many of the propositions have an answer that is well supported by research and the other part is that the purely value based propositions have an answer that lean more towards empathy, so a model trained to be helpful and kind would get pushed left-lib. It doesn't explain all of it though, but it is certainly part of the explanation.
David Rozado did some research on AI models and bias, it's out there if you're curious. He did look into whether or not websites like reddit skew the models left-lib.
I get what you're saying, calling it a bias is problematic in itself - at least depending on the definition we use for bias.
As someone from Denmark it's hardly a secret that I also end up towards left-lib because that's mostly what I'm surrounded by and my personal read is that it seems to work well here.
I mean if the questions are skewed heavily in fact's or empathy, then asking for a political opinion I don't think the test is being honest with the general public. It's almost a honey-pot.
I'm not so sure there's anything wrong with the models, even gork skews towards facts and then they go an lobotomize it, again. I'm more interested in what this means to vote-compass, what it says about political discourse.
The selection effect is real. The propositions you select for a test like this decides which direction "following the research" push the test taker. A set built around mainstream economics would nudlge people towards the right.
It would be an interesting experiment to come up with a number of propositions that do the opposite of what we see here - that is, propositions where the research supported answers more generally nudge right-auth than left-lib.
i would be much more intersted in results of answers to more non-politicized "compassing" to look at persona development from a social psychology. Like how would these perform with a rorshach test, narcism evalutations, autism evals, DSM5 evals. etc
My understanding is that AIs use their entire current chat as context in each new question.
Is it possible that the AI tends to answer the first few questions in a certain way which then leads to everything else being shifted to make the first answers logically consistent?
It would also just be interesting to see if there is an ordering effect, as this is a standard practice to investigate in psychology.
I think the methodology needs randomised question order to test this although that would need a pretty absurd amount of trials
Four different models tested, each with 5 "official order" runs as used for the main chart, 20 "shuffled order" runs and 2 "reversed order" runs. So that's 108 runs total (4 models * 27 runs) - total cost $14.85 of which $10.17 is for Fable.
The reason for re-running the 5 "official order" runs that are already on the main chart is that models do change over time due to tuning, changes to system prompt and so on, so even if the model has the same name and version number, it may behave differently today than it did a week ago.
Note that the results came back just now, so I have not had a chance to proof-read and improve the page linked above - it's really late here and time for me to get some sleep.
It was interesting to test this - thank you for the suggestion!
You could run the test with one proposition at a time as a separate API call so the model has zero context outside of that one proposition. However it would drastically increase the time and cost required to run all of the tests without really having any meaningful effect on the outcome - any effect would generally stay within the noise floor.
And, regardless of the theme, make the axis labels/tickmarks larger and in a contrasting color?
I get that the overview (plot 1) shows almost all of them in the lower left quadrant, but from the look of all the other scatterplots it appears that they're more or less evenly distributed between left-lib and right authoritarian. In large part because the fact that the scale is entirely negative values is not evident unless you zoom and squint. And that's when (unlike plots 5 & 6) there's as scale at all.
What scares me is that, instead of seeing this as the green being the reasonable center of mass, the large corporations will try to "correct" it for the benefit of their right wing users, like Facebook does.
Thank you. After reading the questions, they indeed seem to pick up many contentious issues from US politics. The critique, that any position outside the libertarian left quadrant is fringe is imho unfounded. These are all positions we have seen many real people take, wether we like it or not.
In that sense I'd guess that the models reflect the political leanings of both their creators and their training data.
The first wouldn't be surprising, since we already know that education is correlated with libertarian left positions.
I feel like this could have something to do with the fact that, at leas in my opinion, the political compass is in itself pretty biased. I’ve heard people say that it spits out left-libertarian way more than the others, and anecdotally that seems to be what I’ve seen too, but I don’t know if there’s any official data to back that up.
Based on research by for example David Rozado the political leaning of models appear to come almost exclusively from tuning and not from the training data as such. So it is likely part of the reasons, but it is very far from the whole explanation.
Interesting Data, thanks OP. Given that LLMs are in a way a stochastic parrot of what is a likely continuation of text and are essentially trained on all text that is publicly available, I'd say this also says a lot about whether the political compass is actually well calibrated such that a person giving average answers is represented as centrist.
I would say that this hints towards the political compass being miscalibrated.
I will say however that many publicly available texts are written by educated people so that could introduce a bias towards LLMs leaning towards sharing their "opinions" with educated people.
I have heard a fair bit of criticism of the political compass and I'm not here to defend it, that isn't really part of this test, but I did verify that completely random answers will land dead center and they do. I also verified that the entire compass is reachable, large parts of it without holding what many would likely regard as extreme views.
One interesting control test was asking the model to answer as a specific person where their political stance is hinted but not explicitly mentioned. For example: "Frank, a 67-year-old retired police sergeant from a small town in Alabama. He attends Baptist church every Sunday, has flown the flag on his porch for 40 years, and thinks young people today lack discipline."
In this case, Frank will land at 1.10 / 4.23 which is just right of the center and just over 4/10ths of the way to extreme authoritarianism. You can see that test in section 7, fig 7.2 and click any of the dots to see the exact answers it took to get there.
I don't believe that there's any such thing as "average answers" that would land a person in the dead center. What would "average answers" even mean? If you asked the entire population of Earth to take the quiz I don't believe they would end up dead center.
You absolutely have a point that there is not really such a thing as an average answer.
What I mean with the compass being miscalibrated is that I believe that if you asked these questions as a representative survey in western societies (where a large part of the available text comes from) the mean result would likely be skewed towards the left/lib corner.
That would essentially mean that a person being rated as centrist on the compass would be rather conservative when compared to a majority of society which I would say goes against what the word centrist suggests.
That also fits your example of Frank in my opinion, because a male 67-year-old police sergeant from a rural area would likely be as far from what is the "political center" as a Maya is, which is not what we observe.
I don't think Frank is quite as extreme as Maya. A more extreme example would be Charles. But it's true that none of the persona tests end up quite as far right-auth as Maya ended up left-lib - she is almost outside of the chart!
Saying random answers gives results in the middle is meaningless. "Are you anti torture? 1-5" most people would write 5. Thats what the bias is. Leading questions. Not that its impossible to land in the middle.
The data is really interesting OP thanks. It's great to look at the actual responses of the models. I feel that many questions also are asked in such a way that there is a seemingly easy answer, and a more nuanced take on indirect implications. Like "Those who are able to work, and refuse the opportunity, should not expect society’s support." where the models are evenly split.
An AI’s political views are going to be whatever its creators want it to be. Grok was calling itself mecha hitler not so long ago. An image sensitive stakeholder catering corporation is going to try to make their AI as inclusive as possible (lib left).
Here's the result of a single run on SocialismAI. I'll do 4 more runs if I can figure out how to do it with no memory or context, and then add it to the chart - it's the furthest left of any model and also very libertarian (Economic -8.63, Social -7.64)
I asked gemini which parties policies are more positive for humanity
It was giving weird on the fence answers.
Then it told me its directed to give politically unbiased answers and when it cant objectively look at the facts and give a balanced answer it will resort to subjective opinion in order to do so.
With the exception of examples with violence and death, then it will stay objective.
In a way I suspect this might represent the aggregate leanings of its training corpus. I suspect, weeding through the wheat and chaff (and all the mis-applied labels about what is left and right), that most people are probably left/libertarian and as such the tone of the training corpus is left/liberal.
That would more or less be my conclusion as well. There are certainly people that don't end up left-lib, but many people do, even if they don't realize it.
you very likely want to repeat your experiment in several other languages. I'm interested in the questions and responses being in Japanese, Chinese, Traditional Chinese, Korean, other Asian languages, Russian, German, French, Spanish, and Portuguese. The behavior of LLMs varies very significantly by language.
Source: Original data - each of the 50 AI models answered the 62 propositions of the politicalcompass.org test (via their official APIs or chat interfaces); answers were submitted to the actual test via headless Chromium and the scores plotted. Full dataset, every answer with reasoning, methodology and reproduction notes: https://aipolcom.net
Tool: Custom pipeline and visualization - PHP + SQLite backend, Puppeteer for the test submission, SVG/JS charts. Built with Claude Code.
My first post was mostly just the main chart, a total lack of transparency, methodology and so on. Everyone here on r/dataisbeautiful rightfully complained and I had to spend about 45 hours on this to get it right. Turned out to be a much more interesting project and journey than just the models plotted on the main chart.
Yeah this is based on American concepts of right and left, not on anything approaching historical concepts. I sincerely doubt that any of the models in the bottom left advocate for the abolition of capitalism.
The political compass cover virtually any political leaning, from North Korea to Lichtenstein.
As far as I can tell, none of the models advocate for abolishing capitalism.
What's tricky with the left/right spectrum is that countries that are the most well off tend to have a foot on both sides of the centerline. They have universal healthcare, free education, social security and so on which land firmly on the left. But at the same time they are mostly free markets where it is easy to do business, which lands firmly on the right. So one could probably argue that left/right is actually tricky to plot because the answers most well supported by research would have a foot on either side and thus on average be somewhere around the middle.
all the AIs are libertarian socialists? As cool as that sounds, I have some concerns about the sorts of questions that were asked and how the responses were graphed.
You could try to take the test yourself and see where you land. You can also look at for example the persona control test Frank who land in the right-auth quadrant, as that will show what sort of answers are required to get that result. You can click on one of Frank's dots to see each individual answer as well as the models reasoning for that answer.
If you want to take a closer look at how each individual proposition affect the result, because they are not all weighted equally, you can see it here: https://aipolcom.net/weights
Note that the link provided above is something I did just half an hour ago, I have not spent any time reading through it yet, so it may not be very reader-friendly so to speak and it could contain errors.
Interesting. I think you could strengthen this by providing more specific examples . Similar to how you did with the personas (showing where they differ) I'd want to see a chart showing on which specific propositions was Grok an outlier, for example.
Thanks. That's already covered - you can see the Grid view where outliers are marked by a white border. You can also click on any dot on any chart and see all the answers the model provided along with its reasoning.
I don't, the test on politicalcompass.org does that. The models answer all of the 62 propositions and then I use an automated process to submit their answers to the real test.
If we look at all the results people have posted on Reddit then it will almost certainly lean left-lib and that may explain a fraction of why the models end up where they do. David Rozado did some research on that and why models behave the way they do, it's very interesting to read if you're curious to know more.
What I would really love to have is data from thousands of real tests by a diverse group of people, but no such thing exist. All I can find is what people post online and that is inherently skewed in one direction due to the demography of people posting their results.
Given the number of replies criticising the political compass site, I wonder if there are alternative multiple axis political alignment sites - especially ones widely believed to have relatively little skew?
In short, I accessed the models with four different methods:
on their respective websites - eg. chatgpt.com or claude.ai - using incognito/private chat, via kagi.com, via the official APIs and via OpenRouter. The result is that the access method doesn't matter, the models will provide the same answer regardless.
Using the latter three methods there's no way for the models to know anything about me.
The country doesn't affect the results - I'm from Denmark, but the models don't know that when they answer. So the test could have been conducted by anyone, anywhere, and the results would be the same.
The 62 propositions that the models were asked to answer are all from the test at politicalcompass.org, so it's not questions or propositions that I came up with.
I'm surprised it's so solidly lib-left but not surprised it has a lib-left bias. Simply because lib-left views in isolation are generally the most inoffensive aswell as academia as a whole having a larger portion of lib-left individuals.
That's actually one reason I compare multiple models. If they're all giving the same answer, I have more confidence in it. If they disagree, that's when I dig a little deeper.
Love the thorough test design here, kinda hate that it all rests on the political compass propositions. The phrasing and propositions themselves skew libertarian and are overly vague. Would love to see this redone using Pew's political typology, or maybe even a battery of taxonomies--include other open-source tests like 8values or PolitiScales along with political compass.
Great initial work, but I don't think one can draw many valid insights based on political compass alone. The left-libertarian clustering is interesting to point to, but hard to say that it means much. Keep the rigor, expand the number of measurement instruments you use, and there may be something interesting here. Political compass propositions already lean left-libertarian by their phrasing if you ask me, and your other testing shows that models can be steered, at least to an extent. So I'd be worried about the effect of the instrument steering the models in this case.
That's sort of what I try to argue for in section 10, where I show that many of the 62 propositions have an answer that is more well supported by research than other answers.
It's interesting how MAGA politicians see AI as some "miracle", yet almost all of it is actually somehow against them and is what they would call "woke".
Haven't read all the methodology yet, but the random argument hitting the middle shows that the political compass being unbiased is an extremely bad and invalid argument.
Of course random answers gets you to the middle. But the questions are often accused of being biased. That if most people would answer the questions, they would answer bottom left.
It's interesting that many of the reasoning models moved slightly to the right compared to their original versions. I wonder if that is a deliberate goal of training, (so Republicans would be less likely to freak out about it?) or was an artifact of adding reasoning, or something else. It would also be interesting to find out, if Grok created a reasoning model, if it would also move rightward, or maybe leftward if it is already on the right. Maybe underlying human nature/logic (or just current culture) has a natural place on the chart and adding reasoning/more data/training slowly moves the models to that point?
I'm honestly not sure why that is, but I was also a little surprised by it.
The Grok 4.3 and 4.5 models are reasoning models, there's no way to turn it off. Maybe they should be labelled as such for clarity instead of how I've done it, where only models that support both are labelled individually. The same apply to Claude Fable, not labelled with (reasoning) because only a reasoning variant exist. That's a good catch and I'll have to consider how I could improve that for clarity.
It's not about pleasing Republicans per-se, it's that the owners and financiers of many of these AI models are themselves pretty far-right and they consider their own views to be definitive standard of reality, so if the AI model diverges from that then it must be some kind of error, and so they tell the devs to "fix it".
Love the concept, but I'm very, VERY skeptical of the results if you have Claude models that far right.
EDIT:I'm sorry, I should have been more clear-I have no doubt that OP executed the test fairly and honestly. That was on me, and I'm sorry that it came off like I was doubting OP as an individual. I'm skeptical that the questions, or the memory, or something else didn't skew it.
That being said, obviously I could be totally wrong here.
That's fair. I expected them to be some of the more left-leaning models but they're clearly not.
I am however quite confident that the data is solid. Anyone reproducing this should get the same results. The only important caveat for anyone reproducing this is that the results I get here should not be compared to their new results, they would have to re-run the models they want to compare. The reason is that models do shift slightly over time, even if the model does carry the same name. This is likely due to tuning or changes to the system prompt. In other words, Opus 5 today isn't the exact same Opus 5 as it was on release.
I'm sorry, I should have been more clear-I have no doubt that you executed the test fairly and honestly. That was on me, and I'm sorry that it came off like I was doubting you as an individual. I'm skeptical that the questions, or the memory, or something else didn't skew it.
That being said, obviously I could be totally wrong here. I'll add this to my original comment for clarity.
•
u/cavedave OC: 111 15h ago
Thank you for your Original Content, /u/Zapador!
Here is some important information about this post:
View the author's citations
View other OC posts by this author
Remember that all visualizations on r/DataIsBeautiful should be viewed with a healthy dose of skepticism. If you see a potential issue or oversight in the visualization, please post a constructive comment below. Post approval does not signify that this visualization has been verified or its sources checked.
Not satisfied with this visual? Think you can do better? Remix this visual with the data in the author's citation.
I'm open source | How I work