Our New Arbiters of Truth: Google, Twitter and Cues of Online Credibility

In our modern era of splintered trust and millions of media niches, how do we decide that something is credible and trustworthy? A little funny episode of mistaken identity on twitter gave me an interesting view into these issues.

I come at this from several angles.

First, as a cognitive psychologist, I am interested in the thought processes that cause us to rate some sources or facts as true and trustworthy. I am always curious about the actual signals of credibility, rather than the stories we tell ourselves.

Second, as a teacher of students in college, I am interested in helping students gain skills in searching (as much as such skills exist) and hone their BS detectors so that they can leave college with the ability to not only recognize credible sources from hucksters and unscientific charlatans, but to see a spectrum of credibility and bias.

Finally, as a parent, I want to help my kids grow up savvy to a world of media where I won’t be there to tell them that chain letters on instagram are not true, whereas what they read on ESPN probably is.

I was reminded of the problems with teaching online credibility detectors recently, when after the Supreme Court handed down the Hobby Lobby decision, many people angry about the decision took to twitter to denounce the judges. Unfortunately, many did not realize that the Supreme Court has no twitter presence. These people surely did not realize that the Supreme Court is not exactly a high tech kind of place. These angry tweeters denounced what they thought was the Supreme Court, but was instead a blog which covers the Supreme Court. SCOTUSBlog, instead of correcting people, played along and mocked them mercilessly.

Of course SCOTUSBlog is not the official twitter feed of the Supreme Court.

Of course the Supreme Court does not have a twitter feed, they haven’t even “gotten to email yet:

By their own admissions, the justices on the highest court of the land, the US Supreme Court, are not the most tech-savvy bunch. But despite Justice Elena Kagan’s own acknowledgement that the court hasn’t “gotten to” e-mail yet—instead communicating with one another via memos printed on ivory paper—the justices have been tasked with solving some of the most thorny technology debates of our time.

But we shouldn’t mock these tweeters, but seek to understand why they were so misled. And when we do, we should be concerned that new arbiters of credibility have emerged, and they are not necessarily the ones we expect.

To begin, let’s look at the twitter profile of SCOTUSBlog. The bio says: A private blog about the Supreme Court of the U.S. Contact: feedback@scotusblog.com. 

So, that should be open and shut: “private” as in, not public. As in, not official.

But what about other indicators of credibility?

SCOTUSBlog has a shiny blue verified check. Twitter says this identities authenticity, but I think this spreads well beyond a simple: “This account is who it says it is.” That blue check means legitimacy and credibility to many many people.

SCOTUSBlog has 185,000 followers. Certainly follower count doesn’t equal credibility, but it sure helps.

What if we look at SCOTUSBlog’s webpage header? This is what our Hobby Lobby tweeters saw on June 30th:

Header of SCOTUSBlog on July 1st, courtesy of the internet archive

Header of SCOTUSBlog on July 1st, courtesy of the internet archive

This looks pretty official to me. Although the “sponsored by Bloomberg Law” should be a give-away that this is not an official government site, the name Bloomberg and the bureaucratic design make me think it is perhaps less of a give-away than it should be.

And now, they have changed the web header to:

Web header of the SCOTUSBlog today

Web header of the SCOTUSBlog today

Even more official looking!

I am pointing this out not to say that SCOTUSBlog is being misleading, I think they are doing fantastic legal journalism, and I applaud them for it. Their webpage is simple and well designed.

But this little incident of internet mistaken identity presents an interesting object lesson, and I would submit that follower counts, cues from professional web design, and that twitter verified logo can overwhelm other cues to credibility. It reminds me of a passage in Danah Boyd’s excellent chapter on “digital natives” in her new book “It’s Complicated: The Social Lives of Networked Teens:”

Everywhere I went, I heard parents, teachers, and teens express reverence toward Google. They saw Google as a source of trusted information in a digital ecosystem filled with content of dubious quality. More important, many of the people I met believed that Google was neutral, unlike traditional news sources such as Fox News or the New York Times. Most people take for granted that someone, typically the editor in chief, chooses what stories appear on the front page of a newspaper or which are covered in a TV segment.  Conversely, people naively assume that algorithms, procedural sets of instructions for calculating an output, such as the ones produced by
Google, must not have nearly the same biases as an editor.

When people don’t have specific knowledge, that the Supreme Court is famously untechnological, that SCOTUSBlog was a great journalistic source for interpretation of the fabulously complex decision about the Affordable Care Act, that an official government page will never use the words “sponsored by Bloomberg Law,” they trust other cues that are generally applicable to many sources. These general cues are now being provided by companies, like Twitter and Google, who have the reputation of passively transmitting communication, but are not passive, and indeed, are not even transparent in the ways that they provide this credibility:

If you think you meet the criteria for verification and have not yet received a badge, please be patient. We are working within key interest areas to verify accounts that are sought after by other Twitter users. We don’t accept verification requests from the general public, but we encourage you to continue using Twitter in a meaningful way, and you may be verified in the future. Please note that follower count is not a factor in determining whether an account meets our criteria for verification.

Rather than chuckling at the naivete of the angry Hobby Lobby tweeters, this all makes me even more resolved to teach such things to my students and my kids, and to do so in the context of all the misleading cues to credibility out there.

Posted in web | Leave a comment

Please allow me to (Re)Introduce Myself

Hello readers! Welcome to the new home of Cedar’s Digest.

I have decided to integrate my blog with my professional website, no longer putting a wall between my blog and my website that presents a more professional and scholarly profile.

I began a blog in 2005, posting some book reviews that I submitted for the Virginia Quarterly Review (many did not actually get printed, as a graduate student, you could get a free review copy if you pledged to write a 500 word review). Here’s a good one on Rebekah Nathan’s My Freshmen Year, in which an anthropology professor goes back to college (undercover) and tells of her experience. I also posted random stuff I found that I thought was cool. Like the Chronicles of Narnia rap. Or extolling the virtues of McSweeney’s Lists.

As time has gone on, I have found my blog becoming more formal and bigger part of my professional and scholarly work. I find myself trying to restrict my blogging to issues where my training and expertise give me something unique to contribute, and therefore, while still often personal, it has become a lot more professional.

So now, in the tabs above, you can see how I structure some of the facets of my professional life. I hope that people who read or find the blog might also find some of the rest of the site useful as well. If you find yourself drawn into a blog post about higher education advising, you could read about my advising philosophy, or even find some useful forms that I use to guide my students through the process of asking for a recommendation.

One final note, I have also moved, from the wordpress.org website, to my own hosted site. I have been inspired by the Open VA conference, and wanted to take more control of my own web identity. My entire site in now hosted by Reclaim Hosting, a wonderful little company run by people associated with Mary Washington University, Jim Groom and Tim Owens. I have had a great experience with them so far, and I would recommend very Reclaim Hosting highly for anyone, especially academics, looking to take control of their own web identity. One benefit for you, dear reader, is that there will no longer be any ads on any pages.

Ok, thanks for reading, and I would welcome any comments on any aspect of the site.


Posted in Uncategorized, web | Tagged , , | 2 Comments

Social Media as a Catalyst for Psychological Science

Can social media serve as a catalyst for psychological science?

I think many scientists are rightly skeptical of social media as a replacement for other normal scientific processes. Peer review will not be replaced by Tweep Review. Methods sections will not be replaced by “Storifies.”

However, it is equally clear that social media engagement can boost the spread of scientific information, and enhance scientific communication, both within a scientific community as well as between scientists and the public. Social media can be a catalyst for psychological science.

Last week Melanie Tannenbaum, Jorden Cummings, Stuart Ritchie and I presented a symposium at the Association for Psychological Science 2014 annual convention on Social Media as a Catalyst for Psychological Science, laying out both the why and how we have used social media to enhance our science.

Here is a pdf of my presentation, with slides and what I said, hosted by the Open Science Framework. I also expanded that into a blog post.

Here is Melanie Tannenbaum’s presentation (“Social Media can be for Science!“) on slide share. I thought Melanie did a great job of showing how facebook can be used to spread scientific knowledge to friends and family, but also to humanize science to people who might not otherwise encounter someone they consider a scientist in everyday life. When Melanie was curating and trimming her facebook friends, hoping to tailor what friends in different areas of her life saw (would runner friends want so much social psychology? Would psychologists want to see so many cat pictures?), she asked her friends if they wanted to be removed from any lists. One of her old high school friends wrote a poignant comment: “I dropped out of college so your posts are the closest thing I’ll get to an education, please keep me on all your lists.” It also struck me that while facebook links to scientific articles are no substitute for higher education, on Melanie’s facebook page they are more than just a clipped article, because they include the original article as well as discussion among Melanie’s social psychologist community. Eavesdropping on professional or expert dialogue is one of the unheralded benefits of social media to me, and Melanie had a number of great examples of good dialogue as well as the benefits to the rest of her extended social network.

Jorden Cummings talked about how a conversation on twitter led to a research project. Cummings Day APS 2014 - Jorden nicely laid out the process of recruiting participants through twitter, negotiating IRB protocols (she had an approved tweet to use for recruitment) and disseminating the results. I thought her experience also showed very nicely how twitter can link people from relatively disparate backgrounds. Jorden is an academic clinical psychologist who studies interpersonal relationships and psychopathology at the University of Saskatchewan, and her colleague on this project is T. Eugene Day, a systems engineer who studies quality improvement and patient flow, as well as care delivery as a Senior Improvement Advisor and Principal Investigator at the Children’s Hospital of Philadelphia.

Stuart Ritchie closed our session with “4 tips for making psychology on twitter less annoying.” Ritchie – APS 2014 Twitter Talk - 1) Beware the echo chamber. Stuart pointed out that psychologists can lean heavily liberal, and we risk enclosing ourselves in echo chambers. Luckily for us, while moving to towns or neighborhoods that are more ideologically diverse might be difficult, following people who we don’t agree with on twitter is not that hard. 2) Think before you tweet… but also resist the urge to pile on. Highlighting the idiotic and prejudiced tweet by Geoffrey Miller, Stuart agreed that this was a bad idea, and we should avoid such a fate, but also cautioned against the momentum of outrage that can build on twitter. 3) Don’t be a replication bully. This episode has received even more attention since then, but Stuart reminded us what might seem to be a small bit of snark can be spread far and wide on twitter. 4) Finally, Stuart showed us how a bit of complaining on twitter about the poor methodology of a paper between himself, Keith Laws, Tim Smits and Daniel Lakens led to a published critique of the article. I thought this was a great example for two reasons. First, it shows how conversations on twitter can be “kindling” for longer, deeper, more substantial conversations on other channels. People on twitter often recognize the limits of 140 characters, and know you aren’t going to get a roaring fire from a few sticks, but sometimes that’s what you need to get it started. Second, I thought it showed how twitter can level status cues. Stuart is a graduate student and each of the other authors are professors. All too often the water cooler talk is segregated along status lines, and twitter can be an entry for junior colleagues to contribute. Perhaps this isn’t always the case, but I find that on twitter people might not take the time to glance at your conference name tag, or guess your age, and are more likely to react to the content of what you are saying.

Ok, that’s all. I hope that these presentations can help convince some of you psychologists not on twitter to get on, and maybe give a few tips to those of you who are already on or convinced that it is worthwhile.

Posted in psychology, science, web | Tagged , , , | 1 Comment

Social Media as a Catalyst for Psychological Science – My presentation

At the recent Association for Psychological Science annual convention, I co-chaired a symposium on “Social Media as a Catalyst for Psychologist Science.” In my next post I will give some context to the entire session, but first, here is my presentation, in blog form. This presentation is also available as a pdf: Riener – Social Media as a Catalyst for Psychological Science

NT homepage decline

I begin the motivation for using social media to spread psychological science with this chart, from the New York Times Innovation Report, a leaked internal document explaining the external challenges, internal struggles, and plans for the future for the New York Times to continue to be a successful media outlet. This chart illustrates how social media is coming to dominate the web, even “official” or “old media” sites like the New York Times, and how this is a very recent change (notice the years are from 2011 to 2013). This chart shows how visits to the NYT homepage have gone from 160 million a month to less than 80. People are finding the articles they want to read through social media.

These charts to the right (also from the same report) reinforce that point, but looking at the whole web. Visits to home pages are declining while social media referrals are climbing, all over the internet. People have NYTimes Innovation Report - Referralschanged the way they access the web, and shifting to getting referred from social sites.

Why communicate science online? I begin with how to be a scientist on social media by referencing an excellent primer by the biologists Holly Bik and Miriam Goldstein. Their Figure 2 shows the different scientific purposes and time commitments of various social media activities. For this presentation, I will focus on twitter.

As Bik and Goldstein note, different social media are well suited for different purposes. However, one thing that they note in the article that is not clear from the figure, is that these interact. Twitter can help spread a blog post, and a conversation on a blog post can spill out into twitter and facebook. But this is a good framework for how to see the social media landscape and science. For this talk I will be focusing on twitter, which is my preferred social media network.

Liz Neely - twitter as cocktail party at conference

For how to think of twitter, I turn to Liz Neeley, who succinctly frames twitter as a cocktail party at a conference. It isn’t only a presentation, or a poster session (where people are only strictly business) but it also isn’t just your friends who are interested in everything personal. The way I approach it, scientists on twitter shouldn’t be afraid of sharing a few personal details here and there, but most of the people who follow you will do so because of what your professional expertise or interests are, not personal reasons. This of course varies, but I recommend starting out thinking of twitter this way, curating your friends this way, and putting that kind of filter on your communication.

Riener - Who Am I slide

This is a brief snapshot of who I am, what I tweet about, and what I am interested in. So on twitter I will follow people in each of these spheres. I follow other teachers of General and Intro Psychology, historians of psychology (and some historians), perception researchers, teachers, and other people involved in higher education policy debates and faculty governance.

With gratitude to Raul Pacheco, who has listed five things he thinks twitter is useful for in academic contexts, I decided on four for the few minutes I have with you today.

Here are 4 things I’ll talk about today that I find twitter useful for: news and discovery, building scholarly networks, quick help, and giving psychological science away.

News and Discovery

First, as a personalized newspaper. Here are some examples of things I found interesting and useful that came across my twitter feed recently, that I would have not seen otherwise. Holcombe - retina evolutionAlex Holcombe, a fellow perception researcher, came across a fact about the evolution of the retina (in a Nature Reviews Genetics article, a journal I do not read or track) that non-visual cells evolved into visual cells, and visual cells evolved into non-visual. Cool!

Chris Crew - Columbia PhDJay Van Bavel, a social psychologist at Columbia, tweeted a picture of Chris Crew giving a presentation (with Kenneth and Mamie Clark pictured on a slide in the background) and noted Crew was the first African American man to earn a Ph.D. in psych from Columbia since Kenneth Clark. Interesting (and sad) fact that I remember and pass on to my history of psychology class.

ENTJ destroyer of worldsFinally, a colorful example from the Onion, to use when I trash the Myers Briggs Trait Inventory.



Building Scholarly Networks

I also use twitter to build scholarly networks. As a cognitive psychologist in a small school, in a small town in Virginia, we don’t have frequent psychology journal clubs or symposia. So I use twitter to connect with psychologists I would not otherwise be able to have a conversation with. riener - pashler - nature picsHere I am having a brief conversation with Hal Pashler. I will also say that I think twitter often acts as a leveler. If you are a junior faculty or graduate student, you can still have a conversation with a senior faculty member, and the mediated nature of the communication can sometimes reduce some of the cues to power and status that are often present in real world contexts.

I also like twitter to broaden and diversify my scholarly network, beyond the academy, and beyond my areas of interest and research. Sanjay Srivastava is a social psychologist, Audrey Watters is a education journalist who is an excellent (and respected) voice on educational technology, Mark Changizi is a perception researcher who has left academia, but remains quite active in applying perception research to real life problems through his company 2AI labs. Bashir9ist is a pseudonym for a graduate student in neuroscience who I have had a number of conversations with about neuroscience, graduate training in psychology, and other shared interests over the course of a year or two. As he went on the job market, I came to do some informal mentoring and advice. He may not have needed it exactly, (he got an excellent job) but he has not been the only person who I have served in this role. For me, building this scholarly network is not just what I can take from my network, but what I can give back.

Quick Help

Another example, though it may seem trivial, is the quick, non-google-able answer to a question. I am thinking about training myself up on R this summer, but I am a bit daunted, and it was helpful to discover that Jonathan Goya, a computational biologist in my network, is willing to help. Another biologist friend, Jeremia Ory (of the perfect twitter name: DrLabRatOry), chimed in to agree that Jonathan was very helpful.

Giving Psychological Science Away

Me and Nyhan and KonnikovaMy final example of using twitter is to spread psychological science and interact with non-scientists about psychological science. This might be in interacting with a science journalist (as I did here with Maria Konnikova, who is actually a scientist herself so perhaps not a perfect example) or a fellow academic in another related field (Brendan Nyhan is a political scientist who studies false beliefs in politics), but it also might be hosting a chat session with teachers. In the case on the left I hosted a #psychat session and asked AP psychology teachers about how much writing their students did. I also shared some of my conceptions of college readiness.

I’ll close with what many people might think is the first reason to use social media as a psychological science: to spread their own work. My article on learning styles, attempting to translate some of the basic findings in cognitive psychology to show that learning styles are not educationally useful, was spread very widely on social media, and I can track when it is spread and encourage people to ask questions, or answer concerns. I will also add that the other ways that I use twitter contribute to this final use. Because I have built my scholarly networks, asked and provided quick help, interacted with science journalists, the people in my network are more likely to read and spread my science.

That observation bears repeating, that each of the many uses of twitter (and you will certainly find some more yourself) interact with each other. Promotion helps you build your network, and robust networks help you more effectively promote. Offering quick help builds up good will, but sometimes, so does asking for it. Sometimes someone else in your network had that same question and didn’t think to ask. Curating interesting news and discovering things you find cool often means you spread that stuff to your network, and you come to be trusted and respected for your unique perspective.

Finally, I’ll close by saying that twitter and social media have a built in “baby pool.” Jump in, follow ten people, spend ten minutes a day checking it, and let your network grow organically. Don’t feel like you have to do it all at once. Thank you for your time, and I hope to see you on the twitters!

Posted in psychology, science | Tagged , , , , | 4 Comments

On Controversial College Speakers, Debating and Critical Thinking

Azusa Pacific University recently disinvited Charles Murray from giving a talk there. The talk had been scheduled for months, but apparently no one realized the full extent of his bad reputation. He wrote an open letter in response.

Although regular readers will know that my politics are quite opposite from Murray’s, I agree with Murray here. Here’s where I think they went wrong.

First of all, don’t invite someone unless you know who you are inviting and what they are going to say. Doing some due diligence on speakers seems not only like college PR 101, but college 101. Would we assign a book to our students without reading it ourselves, or at least having a really good sense of what was in it? Of course not.

Second, if you have invited someone who you discover has views you consider abhorrent, recognize that you have already given them publicity with the invitation. By my mind, rescinding the invitation doesn’t necessarily reduce their stature, all it does it prevent your students from engaging with their ideas on your campus.

Third, we should value engaging with ideas we disagree with in general, and in particular people like Charles Murray. Even when we disagree with conclusions, in higher education we should value people who convey their ideas clearly and with evidence, even if we see them as cautionary tales. I would much rather we engage with Charles Murray, because he often clearly lays out the ideas that many conservatives hold without explicitly stating them, than with vacuous celebrities such as Jenny McCarthy.

One of the things that I try to model in my psychology classes is how we disagree and settle arguments as scientists. Many of my students and I disagree on whether spanking works or not. That’s fine, but how should we disagree? As scientists, we should talk about what “works” means. Does it make children behave better? How would we measure that? Does it make them more obedient? How would we measure that? What sorts of studies could we do? Can we compare households who engage in spanking and those that don’t? What measures should we use?

It is very difficult to train critical thinking or even engage in it without any disagreement. Without disagreement, our claims turn into assumptions, and these assumptions melt away into the air we breathe.  This lesson is the one big lesson of perception. Just because we all see something the same way, doesn’t mean that it isn’t an assumption or a guess. Our visual system is full of assumptions that we don’t experience as assumptions.

We don’t realize that we have assumptions until someone asks “why do you think that?” Certainly it is possible to ask why without disagreeing, my kids do it all the time, but as adults we aren’t as easily led to the “why” questions without disagreeing with someone. You don’t think we need affirmative action? Why do you think that? You think over the past 50 years we have achieved meritocratic sorting in higher education? Why do you think that? You think you can write a book about class divergence as if it can be isolated from race? Why do you think that?

In my last post, I cited an essay by Murray on higher education. I find it useful in my own thinking to really grapple with the ideas here. Murray makes the claim that too many people are going to college.  He argues that we should make K-12 the grades where we impart liberal education and the core knowledge of our society, and that higher education should be more focused and targeted on people who want to be professionals and need training in reading and writing. I found myself agreeing that it is a problem that our society gives everyone the expectation that a college education is absolutely necessary to get a decent paying job, and we end up with many many students who are only in college to get a job, and are not motivated by the knowledge itself. Both Murray and I (and Mike Rose, I’d guess) would agree that our society should respect the trades far more than it does. I think where we’d split is that Murray seems to label trades as “good with your hands” and suitable for our low-IQ, unfit for college underclass, whereas Rose and I see trades as often just as cognitively demanding as professions, but simply not recognized as such. Here’s Murray:

But while it is true that the average person with a B.A. makes more than the average person without a B.A., getting a B.A. is still going to be the wrong economic decision for many high-school graduates. Wages within occupations form a distribution. Young people with okay-but-not-great academic ability who are thinking about whether to go after a B.A. need to consider the competition they will face after they graduate. Let me put these calculations in terms of a specific example, a young man who has just graduated from high school and is trying to decide whether to become an electrician or go to college and major in business, hoping to become a white-collar manager. He is at the 70th percentile in linguistic ability and logical mathematical ability—someone who shouldn’t go to college by my standards, but who can, in today’s world, easily find a college that will give him a degree. He is exactly average in interpersonal and intrapersonal ability. He is at the 95th percentile in the small-motor skills and spatial abilities that are helpful in being a good electrician.

I think he is drastically simplifying both managing, which requires a great deal more than linguistic and logical mathematical ability and being an electrician, which isn’t just small motor and spatial skills.

But that’s not the point. The point is that in laying out his argument, in claiming that the relevant distinguishing characteristic of this young man is ability (whether academic or small motor ability), Murray forces me to confront and sharpen my views on why I think people should go to college, why I think some people are unmotivated about college, and whether some people are “unfit” for college.

Rather than disinviting Charles Murray, why doesn’t Asuza Pacific University use this as a teaching moment, and ask someone else whose views are contrary to Murray’s? Why not spend some money and invite Ta-Nehisi Coates or a Isabel Wilkerson to talk about how the legacy of white supremacy still haunts us? I am not saying that everything is a debate, or that we should always give equal footing to each side of the argument, but if you have already invited someone, and that someone is clearly representative of a large body of views on one side of the political spectrum, why not debate instead of canceling entirely?

Posted in science | 2 Comments

Thumb on the PayScale

In my last post, I took issue with the PayScale college rankings, as well as with how economics reporters framed these rankings, citing their low calculated Return on Investment as evidence that these colleges “make” students poor. Jordan Weissmann has graciously responded to my critique. The Senior Vice President from PayScale itself has also responded in a comment on my post, defending the rankings as useful to high school seniors who later tell him, “I wish I had known this before.” I’m glad for the dialogue, and I think this issue connects to a larger debate on the value of higher education in general, as well as the relative value of different colleges, so I am going to keep the conversation going with a more extensive response.

Weissmann is not alone in his framing of PayScale’s rankings. While his headline at Slate reads “Which College Will Make You Poorest?” Derek Thompson at the Atlantic treads similar ground: “Which College—and Which Major—Will Make You Richest?” and later, “These U.S. Colleges and Majors Are the Biggest Waste of Money.” Walt Hickey goes with a more straightforward, but still uncritical view at 538: “Study Names Colleges with Best Return on Investment.”

My argument was first that salaries from art schools and engineering schools are not directly comparable, since students are not often choosing between Harvey Mudd (#1) and Maryland Institute College of Art (#1301). Chris Chabris mentions this point in his post criticizing 538′s coverage of the rankings. Second, I argued that PayScale and many journalists who wrote about the rankings, are overstating the role of the college in shaping its students salaries and understating other factors in what determines a students future salary, such as the role of selective admissions, constraints in the labor market and choices of the students themselves. Weissmann acknowledges various limitations (including selection bias) of PayScale’s methodology, but he doesn’t see these as reason enough to avoid phrasing like: “Which college will make you poorest?” or “Yes, the art school really is making students poorer if it delivers a negative ROI.”

The subhead for Weissmann’s recent defense of ranking colleges based on graduate salaries reads: “Academics might not like it, but schools should be held accountable.” Since I am the only named academic in the article, and my quote is framed as “Like many in higher education…” I can assume that Weissmann sees my criticism as based on my identity as an academic. True, I am an academic. I do feel solidarity with my fellow providers of higher education, whether they be at the University of Virginia (apparently THE best value! Wahoo!) or at arts schools like Maryland College of Art and Design, or Historically Black Colleges and Universities like Shaw and Fayetteville State University. However, my criticism of this particular set of rankings owes more to my training as a social scientist (and as a psychologist in particular) than to my allegiance with all institutions of higher education. Here’s why:

Claim a Cause? You’ve Got to Earn It

All this writing about salaries reminds me of another kind of earnings. When a scientist such as myself reads a headline with a claim that something causes something else, I ask whether the speaker has earned the right to make that claim.  Can the data cash the check that the causal claim makes? A few examples: “Smoking causes cancer,” “miracle diet causes weight loss,” “stress causes ulcers,” and “vaccines cause autism.”  While many educated people may start with a skeptical “correlation does not equal causation,” most scientists (especially social scientists) know that not all correlations are created equal, some studies are better than others. For a long time, the only good data on the harmful nature of smoking was epidemiological: those who smoked tended to have worse health than those who didn’t. Notice the lack of a causal claim, a study might conclude that a heavy smoker is 50 times more likely to develop lung cancer than a non-smoker, but it is still possible that smokers in general lead more stressful lives, exercise less, and have less healthy diets. In this case, rather than causing cancer, smoking is merely another marker of an unhealthy lifestyle. More recently studies have narrowed in on the biological mechanisms of smoking’s harmful effects on the body. What is it precisely that causes the harm? Is it the nicotine? the tar? What are the cellular mechanisms of the carninogens in cigarettes? These questions now have much more specific answers based on carefully controlled experiments, and even though there still seems to be some randomness to the overall pattern of carcinogens causing cancer (yes, some people can smoke for 60 years and not get cancer), we know a lot more about the biochemical effects of smoking.

But even without research into the cellular mechanisms, the epidemiological, correlational research was able to earn its causal claims. One landmark study compared British male doctors who smoked with …. British male doctors who did not smoke, and followed them for over 40 years. Keeping many other variables constant, this study still finds incredibly harmful effects of smoking.

A similar search has been conducted for a connection between vaccines and autism. Many children exhibit the first signs of autism spectrum disorders at around the same time they are vaccinated. Yet this is a misleading correlation. Epidemiological research would seem to be impossible. How can we compare levels of autism in non-vaccinators and vaccinators if everyone vaccinates? Except now that the fear has spread, we can. Populations that don’t vaccinate their children have the same incidence of autism as those populations who do vaccinate. Amazingly enough, however, populations who choose not to vaccinate DO have higher risk of an outbreak of whooping cough. A long search has concluded that vaccines do not cause autism, through both epidemiological work, as well as the search for biological mechanisms of harm.

Which brings us back to PayScale and the economics reporters. Have PayScale and these reporters earned their causal claims? Are these rankings actually useful to high school seniors deciding where to go to college, as Barnaby Dornfman comments on my previous post, or merely perceived to be?

I remain deeply skeptical.

The Median is not the Message*

*with apologies to Stephen Jay Gould

There are (at least) three classes of factors that could predict a high school student’s future salary. First, a student’s individual characteristics such as preferences, abilities, motivation and even demographic variables such as race and gender affect their future salary. Second, that person’s education and training of course influence what they earn. This is where college fits in, but it might also include high school preparation as well as out of school preparation. Third, students future salaries are dependent on the labor market they enter as they leave college. As many unemployed or underemployed PhD’s are now discovering, someone can have great achievements in the other two factors, but if there are few to no jobs, one’s salary is severely constrained. The PayScale rankings understate these factors in comparing salaries of graduates to the median salary for someone who has only a high school diploma.

The reporters who have covered the rankings, and to a lesser degree, PayScale itself, make two critical errors that undermine their causal claims. First, they gloss over the impact of student and labor market factors on salaries. Second, they also understate how inseparable colleges are from the students who attend them. The rhetorical effect of the apparently straightforward reporting on these rankings are a pair of bold pronouncements “Get thee to a college that doth begin with H-A-R-V, and drink thy fill from the trough of increased economic outcomes” and “But lo! if thy find thyself in Pellville Directional State University, fly fly away, and become a manager at Ye Olde Chipotle, for thy riches will grow more from avoiding useless, expensive learning than from seeking it.” The truth is a lot more complicated. Aggregate salaries depend far more on student characteristics present before they enter college and the labor market that they choose to compete in as they exit college, than what differentiates the colleges that choose to admit them.

There are a few ways to see this in the rankings themselves. I’ll admit that I don’t know much about the Colorado School of Mines, but what is making their students rich is not necessarily the school itself, but the students’ decision to become engineers in Colorado (or miners, maybe Clementine’s dad had it right after all!). When PayScale compares their salaries to what a median high school graduate makes, this isn’t the best comparison, given that these students have already decided to become engineers, it seems with a focus on energy and environment. Likewise, for the number 1 ranked school, (Harvey Mudd), before Harvey Mudd does any educating, it has selected certain students to admit, and some subset (approximately 30%) of those students have made a decision to attend a small engineering school in California. Of these students who choose to attend Harvey Mudd, 14% qualify for Federal Pell Grants and 46% qualify for any federal student loan aid. For these students, the average total federal student loan aid is around $6,400.  I am certain that Harvey Mudd is a fantastic student experience, but I think it is fair to say that they enroll a student body that is privileged, motivated, ambitious, and focused on a high salary occupation from the moment they arrive on campus. These students have decided to become scientists or engineers before they entered Harvey Mudd, and it doesn’t make much sense to compare their return on investment to what the median high school graduate makes.

Perhaps to defenders of the value of these rankings, they are fine with the advice boiling down to: Want to make more money? Become an engineer, not an artist. Here is hard data on how poor you will be as an artist. Going to college won’t help you if you choose a low paying career. As I said before, I am dubious of the value of such advice, although PayScale Senior Vice President Barnaby Dorfman has experienced sending the PayScale rankings to many people who say “I wish I had known!” However, for the I-wish-I-had-known camp to improve economic outcomes by going into engineering, we need enough engineering jobs to accommodate increased labor supply. But this is not the case. There is no STEM skills gap. There is no labor shortage for STEM jobs. Perhaps a better way to improve outcomes would be to forgo college altogether, and, as Weissmann suggests, work in a Ford factory or manage a Chipotle. This assumes that the student choosing these jobs instead of college will make greater than the median salary of high school graduates, which Weissmann acknowledges may not be the case, due in part to “weaker skills.” Even though Weissmann and others briefly consider the other flaws in this model of salary prediction, I think it is worth it to drill down deeper.

The Worst Performers

What if we do this kind of analysis for Derek Thompson’s so-called “biggest wastes of money?” The students entering these colleges quite often already have a deck stacked against them, and not just in college choice. A student entering Shaw University, the oldest historically black college in the South, located in Raleigh, North Carolina, is making a hopeful investment in a college education. But comparing their future salary to the median high school graduate does not take into account the racial disparities in our current labor market. Further, as Raj Chetty and colleagues have shown, economic mobility is constrained not just by demography of the individual, but by geography as well. In his testimony submitted yesterday to the Senate Budget Committee, Chetty describes one pattern with this lack of mobility: “The first pattern we document is that upward income mobility is significantly lower in areas with larger African-American populations.” He also notes that if mobility is measured by a child’s chances of moving from having parents with the lowest quintile in income (poorest 1/5th of population) to themselves having upper quintile income, one of the “commuting zones” lowest in intergenerational economic mobility? You guessed it, Raleigh, North Carolina with 5%, considerably lower than San Francisco, San Jose and San Diego, California,, all with odds more than twice that. See the map he created below:

Chetty map

To connect geographical mobility to higher education, we must understand how far students are likely to travel for college.  Most students stay close to home. Look closely at this data from the ACT.  The median distance to college (for those who took the ACT) is 51 miles. But if you separate groups by test scores, those who score lowest, stay closest to home. If you score between 1 and 15 on the ACT, the median distance to college is 18 miles. The same pattern can be seen depending on parental characteristics. For students whose parents have no college education, the median distance they travel to go to college is 24 miles (parents with graduate degrees? median distance of 95 miles). Poor students are less physically mobile, which is influenced by, and contributes to, their lack of economic mobility.

Take My Advice, Don’t Try it Twice, if You’ve Got but 50 Cents

So the PayScale rankings would have you believe that students should choose higher ranked colleges, instead of those that “make them poor,” like art schools and HBCUs. The advice is not “Don’t be black, because structural racism will still hurt your progress” or “Don’t be poor in the south, because intergenerational economic mobility there is very low” or “don’t find value in art, because your society doesn’t support that,” but rather: “Watch out for these schools, they waste your money!”

Here is where I notice a difference between what I will call the neoliberal economic approach from a conservative one like Charles Murray’s.  Neoliberals are often confident in the power of greater information to improve equality and efficiency, whereas conservatives are openly pessimistic that all students are intellectually capable enough to complete college.  Here’s Charles Murray:

So even though college has been dumbed down, it is still too intellectually demanding for a large majority of students, in an age when about 50 percent of all high school graduates are heading to four-year colleges the next fall. The result is lots of failure.

Whereas Weissmann closes his original reporting of the PayScale rankings with this:

But it does reinforce why we so desperately need high-quality consumer information about higher education. In the past, the higher-ed lobby has stood in the way of allowing the Department of Education to track college graduates over the long term to keep tabs on their lifetime earnings—what’s known as a “unit-record system.” And as a result, we have to rely on less complete government surveys, or less-than-ideal crowdsourced databases like Payscale’s. As a result, some students are going into college financially blind, and they could be ending up poorer for it—literally

Notice another causal claim: the lack of data is what is causing students to go into college financially blind. What goes unsaid here is that if better data existed, then students and their parents would view that data and incorporate it into their model of how college works. Then, armed with better financial vision, they would made better financial decisions. This same confidence underlies the concern about undermatching, in which high achieving poor students choose to attend lower status institutions, when they would apparently have better outcomes at a wealthier, more prestigious college. This is a topic I have written about here and here. Sara Mayeux makes some excellent points about Yale’s tone-deaf and misguided efforts to address undermatching.

A common theme across the educational spectrum emerges. Yes, economic inequality has grown and states continue to disinvest in higher education, but a cheap, easy and effective way of improving students’ economic situation is through educating people who experience low workforce success (i.e. poor people) to help them make more informed decisions. The implication is clear: Low performing schools (whether colleges or urban public schools) don’t need resources, they need to be held accountable. Low performing graduates and their parents don’t need better economic opportunities, they need better data. Graduates of HBCUs don’t need laws and policies to counteract the persistent effects of racist housing programs, racist tax policy, racist hiring practices, racist underfunding of public K-12 schools, they need bourgious norms, invigorating moral culture, and better salary data. Apparently these students should be informed that an HBCU, which may once have been a proud part of familial identity, a cause of pride in black scholarship, a shining beacon of advancement through higher education, well, now big data science shows it offers a low ROI. Prospective students should be advised to go to a better school, or an engineering school, or work at a factory instead.

I know that Weissmann, PayScale, and everyone involved in this debate wants a better system of higher education in these United States of America. We all should want, to paraphrase Lincoln, a higher education system of the students, for the students, and chosen by the students. One that is accountable to the students. But in search of this goal, they are willing to forgive imperfections of their scale, convinced of the desperate need for accountability. But as a psychologist well versed in some of the shameful consequences of imperfect scales in psychology’s past, I am skeptical. Instead of progress and accountability, I see such rankings of colleges based on assumptions of privilege and status, devised by privileged tech boom MBA’s, for the software and energy engineers. I see this scale as one that does not encourage accountability, but merely reflects existing social inequities and blames society’s economic losers, just as existing social policies make their fates increasingly unavoidable. Don’t tell me you are holding colleges accountable, while holding your thumb on the scale.

Posted in education, higherEd, psychology, science | Tagged , , , | 20 Comments

The Absurdity of Ranking Colleges by Graduate Salaries

Jordan Weissman has moved from the Atlantic, and is now covering economics at Slate. He has a post up provocatively titled (it is Slate, after all) “What College Will Leave you Poorest?” which covers the Payscale college salary rankings, in which Payscale.com ranks colleges based on the cost of the college offset by the reported salaries of their graduates. In other words, a return on investment for each college. What a grand idea.

Ok, I think it is a terrible idea for a lot of reasons, but in efforts to keep this blog post under 1000 words, I am going to focus on one seemingly minor issue, which to me invalidates this entire ranking system. To illustrate this, humor me first on this brief tangent into basic statistics and my own education in cognitive psychology.

I was a history of science major, so I only took a few psychology classes as an undergraduate, and these were all in the clinical or neuroscience area. So when I got to graduate school in cognitive psych, I had to catch up to learn some basic concepts. One of these didn’t occur to me until I took a class on “Advanced Research Methods in Cognitive Psychology.” We were discussing the merits of how to graph and analyze different kinds of data, and we were talking about time. We reviewed the different types of measurement scales. When a measurement is on an nominal scale, there is no order to the results of the measurement at all. These as categories, and it would make no sense to say that they are in order, or that the differences between them can be interpreted in some quantifiable way. Gender, race, religion, types of fruit, etc etc.  The next type of scale is ordinal. In this scale, the results are ordered, but the difference between the orders doesn’t necessarily make sense. This could be rankings, like eldest, middle and youngest child. We can put them in order, but the difference between the eldest and the middle is not necessarily the same as the difference between the middle and the youngest. Many likert-style (“neutral, ok fine I guess, agree, strongly agree, DEAR GOD YES”) survey items are on an ordinal scale. We can say that someone who chooses option 5 agrees more than option 3, but is the difference between 5 and 3 the same as between 3 and 1? Probably not, and we should analyze our data as if that is the case. The next scale is interval scale, in which the scores are ordered, but also the differences between ranks are equal intervals. For interval scales, the difference between scores of 8 and 10 is the same as between 2 and 4. Time is often given as an example of an interval scale. Finally, a ratio scale has a true zero point, where a score of zero is real and meaningful, such as wealth or income.

We were talking about these scales, and the issue of measuring reaction time came up. It seems like it could at least be an interval scale, if not a ratio. But as we were talking, the professor reminded us that even if the physical scale of time was an interval scale (the difference between 1 and 2 seconds is the same as the difference between 20 and 21 seconds) this did not mean that the psychological dimension of reaction time was also an interval scale. This struck me as an interesting way of thinking about a central insight in scientific psychology, since its inception. We have to be careful in how we treat our measurements, taking into account that the psychological scale does not always map cleanly onto the more physical or concrete scale (whether it be light, sound, time, or money).

Here’s my contention: while money might be an interval scale, salary is not.

Sure, a dollar is a dollar is a dollar, but saying that a mineral engineer makes twice what an artist does, and therefore this particular art school isn’t worth it, that just seems absurd, but it is exactly the logic that their ROI tanking system encourages, and that Weissman’s article adopts in using language like “to be blunt, these schools make students poorer.” No they don’t. I imagine that most students entering art school are fully informed (by parents, teachers, classmates, strangers in the grocery store) that their choice is disastrous and they will never find a lucrative job doing art. They choose to pay for training anyways. Is the art school making them poorer?

The salaries of graduates from Harvey Mudd (#1), the Colorado School of Mines (#11), UNC-Asheville (#1306) and Maryland Institute College of Art (#1301) are not comparable and should not be compared as if they have anything to do with the schools. Instead, this scale is not just a proxy for which careers graduates choose upon leaving the college, but even what career they were interested in upon entering that particular college.

A given measurement scale supports a certain kind of statistical analysis. If one has an nominal scale, it makes no sense to make ordinal claims of greater than or less than (are oranges less than apples, maybe Honeycrisps). If a measurement is on a ordinal scale you can report frequencies, but it makes no sense to analyze means or standard deviations (what is the mean and standard deviation of mild, medium, spicy and Native Thai?).

Perhaps the payscale people should consider replacing their college rankings with a big simple headline that says “Engineers make more than artists.” and “People who apply and are accepted by Harvard are going to make more money than people who apply and are accepted by Morehead State.”

I could go on and on. No matter how seemingly sophisticated these rankings seem, they have not found a way to disentangle a school from the students who choose to attend. Just because we have found a number (salary) that allows us to compare Williams (#16) with Virginia Tech (#69) doesn’t mean that those 53 spaces mean anything at all.

Posted in education, higherEd, science | Tagged , | 6 Comments

Grit and Galton: Is psychological research into traits inherently problematic?

Is all psychological research on individual differences racist?

Can psychologists ever separate our shameful past of scientific racism from the methods, techniques and questions that have grown from it?

A recent post criticizing the concept of “grit” (and Angela Duckworth, the researcher responsible for its popularization) made me consider these questions. While grit might be a specific research topic in psychology, it offers a useful case study in how findings in psychology get applied to education and policy settings.

The author of the post in question, Lauren Anderson, an education professor at Connecticut College, recounts discovering that Duckworth was awarded a MacArthur genius grant, and was disturbed as she read more about the roots of grit. From reading the research statement on Duckworth’s website, Anderson sees only one scholar mentioned in the first paragraph: Francis Galton, the father of eugenics.

With that, Anderson lays out what is essentially a guilt-by-association set of questions about the connection between Duckworth and Galton:

“What are we to make of a 2013 “genius” award winner quoting unproblematically the ‘founding father’ of eugenics in the opening paragraph of her research statement, even as her research engages young people of color?”

First, I am going to fully defend Duckworth herself, and any researchers of individual differences (who are also painted as “problematic” with this broad brush). But second, I am going to return to Anderson’s perspective, and try to see how it reflects neither ill-intentioned character assassination, nor intellectual laziness, but rather disciplinary differences, and a split between how people define science.

For the defense: here is Duckworth’s opening paragraph of the research statement:

The Duckworth Lab focuses on two traits that predict success in life: grit and self-control. Grit is the tendency to sustain interest in and effort toward very long-term goals. Self-control is the voluntary regulation of behavioral, emotional, and attentional impulses in the presence of momentarily gratifying temptations or diversions. On average, individuals who are gritty are more self-controlled, but the correlation between these two traits is not perfect: some individuals are paragons of grit but not self-control, and some exceptionally well-regulated individuals are not especially gritty. While we haven’t fully worked out how these two traits are related, it seems that an important distinction has to do with timescale: As Galton (1892) suggested, the inclination to pursue especially challenging aims over months, years, and even decades is distinct from the capacity to resist “the hourly temptations,” pursuits which bring momentary pleasure but are immediately regretted.

It is clear that Duckworth’s work focuses on traits. There is a great deal of modern research on personality psychology that focuses on individual differences in traits. Some people are shy, some people are outgoing, some people are messy, others are fastidious. The research questions are often framed as Duckworth does above. Are there fundamental and relatively stable personality traits? How are they related? How is shyness related to introversion? What kinds of future behavior can personality models predict? Here, in Duckworth’s opening paragraph, Galton is quoted with a very specific claim: there are two kinds of self-control, separated by time scale. One is the ability to resist an immediate momentary pleasure, and another that helps one to pursue a goal over years and decades. What makes Jiro dream of sushi, while some college students dream of pizza and video games?

But much of the research on traits began with white male researchers that were avowed racists. Galton is but one, there is Karl Pearson, a mathematician, pioneer of statistics, and one who saw eugenics as a natural outgrowth of his work on measurement and analysis of individual differences:

“History shows me one way, and one way only, in which a high state of civilization has been produced, namely, the struggle of race with race, and the survival of the physically and mentally fitter race. If you want to know whether the lower races of man can evolve a higher type, I fear the only course is to leave them to fight it out among themselves, and even then the struggle for existence between individual and individual, between tribe and tribe, may not be supported by that physical selection due to a particular climate on which probably so much of the Aryan’s success depended.”

And many of the early trait researchers also did research on individual differences in cognitive ability (or intelligence), which was also often bound up in racism. But does this mean that all trait research is stained? Was eugenics itself a bad seed, as it were, leading to an infected tree of modern trait research?

Absolutely not. If Galton said that some people are more shy than others, would shyness researchers be prohibited from “unproblematically” citing him? Of course not. Galton invented the dog whistle, and we don’t see any racist undertones with that. What’s more, we don’t condemn the transistor simply because its co-inventor, William Shockley, used the credibility granted by his Nobel Prize in Physics in 1956 to embark on an effort to apply the science of intelligence to improving human population through eugenics.

In reading Duckworth’s research statement, Anderson bemoans the “familiar-sounding narrative deployed to rationalize a turn toward individualistic, “objective measures.” But instead, she is simply reading about the existence of an area of psychology research into individual differences, whether they be neuroticism, intelligence, shyness or openness. Of course people are affected by the environments they were raised in and the situation they currently find themselves in. But individual differences exist, and studying them and invoking those who have studied them in the past, does not make one a racist. This is not a “turn toward individualistic objective measures” but an effort to study the dimensions of how people are different from each other. A complete disavowal of the role of individual differences plants us in another moment in psychology’s past, with Watson, Skinner and radical behaviorism, which of course has its own successes, failures, and interesting utopian theories.

Now I am going to step back a moment and consider why Anderson might have seen grit and Galton as problematic, but also the way in which I can agree that the application of the science of grit can actually be racist, even if the science itself is not. As I see it, Anderson sees Galton’s theories as far more coherent than any modern psychologist, including Duckworth, would. She seems to ascribe special meaning to the fact that the quoted section of Galton comes from a book which also has a section on the comparative worth of different races. If one sees Galton as a theorist, as a scientist devoted to forming a coherent, unifying, comprehensive view of human psychology, then each thread is connected to every other thread. Pull on the self-control thread long enough, you will see that it is connected to racist ideology.

However, this is clearly not how many modern psychologists treat Galton, or many of our other pioneers. They made a set of observations, sometimes connected to a theory, sometimes not. Pull on the dog whistle thread, and what do you get? A better understanding of the limits of the human ear, and how it ages. That’s it. For a unified theory of self-control, we might have to include some of Galton’s observations, but also Julian Rotter, Albert Bandura, Walter Miscel, Carol Dweck or Ellen Langer among many others. Put simply, we don’t have a model of how traits, attitudes, beliefs and situations interact. It is too complicated. Simply because we have a lack of a unified theory of human behavior does not mean that psychology isn’t a science. We do have sets of observations on ways in which each of these matter. The person matters, but so does the situation.

So where do I agree with Anderson? Duckworth has collected data showing that

For example, in prospective longitudinal studies, grit predicts final ranking at the Scripps National Spelling Bee, persistence at the U.S. Military Academy at West Point, and graduation from Chicago public high schools over and beyond standardized achievement test scores; likewise, self-control predicts report card grades and improvements in report card grades over time better than measured intelligence

So one’s success in these school situations (and spelling bees) is due more to what we might call “non-cognitive” factors (things that are not measured by traditional intelligence tests) than by cognitive ability. So at least within these contexts, hard work works. Of course, intelligence tests still do explain some variation (it helps to have high cognitive ability) but non-cognitive factors explain more. But we shouldn’t stop there.

I agree with Anderson that we should be careful and skeptical about applying these findings to school policy settings. Because if we take “hard work works” too far, it becomes a just-world fallacy, and blames those who haven’t succeeded for not working hard enough. If Sally wins the Spelling Bee because she was grittier than Tommy, then does one school succeed because it is better at imparting grit? Do schools fail because they don’t successfully transmit grit? Of course it isn’t that simple, nor would Duckworth say it is. She is interested in her little piece of the causes of success. This does not mean that she thinks social structures doesn’t exist, or poverty doesn’t exist, or racism doesn’t exist, just that she is studying something else. The problem becomes when school policy tilts to far in one direction, ignoring others. Students need more practice reading? Skip recess. I’m sorry, but reading researchers are not responsible for short-sighted school administrators who think that 1st graders don’t need recess every day.

Research psychologists such as Duckworth would do well to understand the context (and yes, the narratives) that can drive public acceptance and promotion of their science. But equally, policymakers and interpreters of psychological science should seek to situate the scientific evidence within both its scientific context as well as its social and institutional context. The existence and power of traits does not deny the power of situational and motivational context.

I’ll close with a quote from TaNehisi Coates, which Paul Thomas (see an excellent post here on separating the grit narrative from grit research) saw as relating to grit:

There is no evidence that black people are less responsible, less moral, or less upstanding in their dealings with America nor with themselves. But there is overwhelming evidence that America is irresponsible, immoral, and unconscionable in its dealings with black people and with itself. Urging African-Americans to become superhuman is great advice if you are concerned with creating extraordinary individuals. It is terrible advice if you are concerned with creating an equitable society. The black freedom struggle is not about raising a race of hyper-moral super-humans. It is about all people garnering the right to live like the normal humans they are.

I would rephrase Coates’ distinction between superhuman African-Americans and basic justice. Personality and individual difference researchers might be interested in what makes African-Americans (or any other human beings) different from each other, not necessarily superhuman. I agree that it is terrible advice for creating an equitable society, but I am ok with scientists studying things that don’t necessarily and immediately lead to an equitable society. I find fault with policymakers, who should be in the business of improving social equity (and justice), but instead act as if they can’t because people are different from one another. Policymakers should not seek to improve (or blame) individuals, but rather focus on improving their social circumstances. Duckworth is not in charge of healthy school lunches, or equal resources across schools, or improving teachers working conditions, nor would she deny that these things matter. Duckworth and fellow psychological researchers should be free to investigate the vast diversity of influences on human behavior, including cognitive ability, self control and other traits, but school officials and policy makers shouldn’t act as if grit is a magic bullet.

Posted in education, psychology, research, science | Tagged , , , , | 6 Comments

On Treating the Unprepared as if they were Unmotivated and Unworthy

An academic job market story yesterday reminded me of the perils of interpreting lack of preparation with lack of worth. A philosophy candidate for a job at Nazareth College in Rochester was offered the job, made some fairly common requests, and then promptly saw the job offer rescinded.  UPDATE: The candidate has revealed more details. As I hope should be clear below, she seems like a real class act to me, if perhaps misinformed.

Below I’ll first go over my agreement with many commentators that whether this was legal or not, it was unethical and outside of normal academic job market conventions, but then lay out why the specifics of her requests might have seemed quite unreasonable to the committee and administration. Finally, I’ll close with how I think Nazareth could have handled the situation better.

First, this represents a disregard for what most candidates understand the process of job interviewing to be, and what I would think is a common labor practice. The candidates interview, the committee deliberates, then the top candidate is offered the job. From the candidates point of view, at that point they heave a sigh of relief, and begin negotiation. The interview is over, they have been found meritorious, and now (as I was advised) they have the most power that they will ever have in academia. As I have read and heard many stories (and as fits the excellent book “Getting to Yes”) the power one has at this negotiation is directly related to “the best alternative to non-agreement,” or BATNA. If one is in a field where there were 200 applicants, and not much separation between them, then the employer may be more likely to say, sorry, no negotiation is possible, here is the offer, take it or leave it. However, if one is in, say, accounting, with 10 applicants or so, then a candidate is likely to have more power. But my understanding (and the understanding of every candidate) is that an offer puts a pole in the ground and moves the process to another stage. What this development (even if legal) means is that Nazareth College treated the negotiation as a continued part of the interview. “Sorry, candidate, you failed the “negotiating for the job” part of the interview, no job offer for you.” There are all sorts of reasons this is chilling, as it represents further erosion of any power or rights that the prospective academic labor force has. It leads me to consider all sorts of macabre scenarios: What if a dean found three (or five) equal candidates and put the job offer up for auction? How low can you go? What if, as seems to be a new trend for internships, candidates had to bid on how much they would pay to get the job? Admittedly, these are horrible. They have to be illegal, don’t they? (Please. Someone tell me they are illegal. Seriously.)

I think it is clear that the college acted unjustly and outside the normal bounds of job process negotiation, but their actions relate to a larger phenomena, seeing behavior that reflects unpreparedness or unfamiliarity with the situation, and treating this behavior as if it reflects poorly on the person. This is a kind of bias called the fundamental attribution error, and it has particularly pernicious effects in education. In this case, given what I know of the job search process at small liberal arts colleges, this candidate was quite unprepared for this negotiation,and her requests, while they might have seemed common to her, and indeed to many of my colleagues at large research places, do reflect a drastic lack of preparation on her part. To get at why this is (and to perhaps help future candidates) I think it is necessary to get into the details of her email and the specifics of the job.

Here are her requests:

As you know, I am very enthusiastic about the possibility of coming to Nazareth. Granting some of the following provisions would make my decision easier.

1) An increase of my starting salary to $65,000, which is more in line with what assistant professors in philosophy have been getting in the last few years.

2) An official semester of maternity leave.

3) A pre-tenure sabbatical at some point during the bottom half of my tenure clock.

4) No more than three new class preps per year for the first three years.

5) A start date of academic year 2015 so I can complete my postdoc.

I know that some of these might be easier to grant than others. Let me know what you think.

And here are why they are each highly unlikely, if not absurd.

First, the salary. Candidates, check out the AAUP Salary Survey for that college. Here is Nazareth College’s entry. Assistant professors (pre-tenure) make an average of $58,000. This collapses across the 6 years one is commonly an assistant professor. If it was done purely on seniority, with regular pay raises (which is unlikely, but a good baseline) this would mean that most year assistant professors after 3 or 4 years there are making $58,000 in years 5 and 6 probably closer to $60-62,000, but in years 1-2 closer to $52,000. Associate professors (after tenure) make on average $69,000, which collapses across the at least 6 years people spend at that rank. Going up for full professor at exactly 6 years is not as common as the tenure clock, so associate professor probably includes some people who have been at the college for 6-10 or more years. So, long story short, the candidate is asking for $65,000, which is probably about what most faculty make after they get tenure at Nazareth College. The chair of the philosophy department is an associate professor, who has a book out in 2012, looks like he could be recently tenured. Long story short, she could very well be asking for more than the chair makes, which is probably not a good place to start.

Second, the official semester of maternity leave. This is what struck some of my academic colleagues on twitter as evidence of gender discrimination and a motivation for a lawsuit. I am not saying that there isn’t gender discrimination here, but this seems to me like it might be a more thorny request than it seems. I wondered first what is the current policy at Nazareth on maternity leaves? I am not a lawyer, but it strikes me that something like this may not be subject to negotiation, and for good reason. For example, I know that many colleges in the past offered free tuition to children of faculty as a perk. The problem was that they excluded staff. A Supreme Court case found that colleges could not separate their different kinds of staff in this way, so now if a college wants to offer a tuition benefit, they must do so for all staff, not just faculty. In practice what this meant was that many large school immediately dropped this benefit. Many small schools retain it since we don’t employ the small city of staff that a large research university does. If maternity leave falls into this category, it may well be illegal to offer paid maternity leave to faculty and not to staff. Either way, it might be a standard benefit less subject to negotiation than the candidate thinks. But that is speculation on my part…

But even if the leave weren’t paid maternity leave, but unpaid, or a research leave of some sort, as she is asking for in item 3, it does represent a significant cost to the college, and a severe misreading of the kind of place Nazareth is. From the comments of the original post, it looks like the standard teaching load at Nazareth is 4-4. If the candidate is asking for a semester leave pre-tenure (which I assume is not included, given that she had to request it) the college will have to replace those 4 courses, either with adjuncts or with half an instructor salary, which seems to be how they are staffing a lot of courses. The philosophy department has four full-time, tenure track (all tenured, in fact) faculty, and five lecturers. Depending on how much they pay adjuncts or lecturers, that would amount to anywhere between 12K and 25K, which is not a small request at a place like that. For example, I would not be surprised if the price tag for a leave like that was close to the startup budget for the entire entering cohort of tenure track faculty.

Item 4 strikes many readers as totally reasonable, and it may be, but it would likely be something to work out with the department chair. How many preps do philosophy faculty usually have? I know in small departments that are also in small schools that have to cover both general education requirements and major classes, one could have three preps in a semester (with a 4-4 teaching load).

Item 5 is common, I have no problem with asking that, but also a common thing to turn down, or negotiate down to starting a semester late.

Ok, but given that I think a few of these requests are pretty far out of the range of what is possible, why do I still think the college was clearly in the wrong? Here is what they said:

Thank you for your email. The search committee discussed your provisions. They were also reviewed by the Dean and the VPAA. It was determined that on the whole these provisions indicate an interest in teaching at a research university and not at a college, like ours, that is both teaching and student centered. Thus, the institution has decided to withdraw its offer of employment to you.

Thank you very much for your interest in Nazareth College. We wish you the best in finding a suitable position.

If we take them at their word (unlikely, but a good exercise) the Search Committee and the VPAA and Dean interpreted these provisions as lack of interest in a teaching school, and not a lack of preparation for the negotiation. I would say that what they likely decided was that the provisions indicated a lack of suitability for this job, not just a lack of “interest.”  But whether interest or inherent suitability for the job, their decision indicates several of their failures.

First, a failure of the philosophy department (and indeed the whole interview process) to educate the candidate and adjust her expectations for what kind of college she was considering. I am sure they have some sort of pre-tenure research opportunities, which they no doubt told her about. But they should have been clear about what a 4-4 teaching load meant and what was typical about the duties and responsibilities of pre-tenure members of the department.

Second, a failure of imagination. This candidate may be a great philosopher and a great teacher, but she clearly has not been advised by someone familiar with small liberal arts colleges. Is that a personal failing on her part? I am not so sure. To me it points to a lack of mentoring in graduate school. Could she have googled as much as I have above and adjusted her salary expectations? Sure. But someone needed to sit down with her earlier in this process and be candid about what it means to work at a place with a 4-4 teaching load, a department of 4 full time faculty in a school with 2200 undergraduates and a low endowment in tough economic times. The fact that she did not have an mentor that did that for her does not indicate her unworthiness or lack of motivation, just that she was enrolled in a typical doctoral program at an elite large university with a faculty who know little or nothing about small liberal arts colleges. She shouldn’t be punished for (incredibly common) shortcomings of her training.

I’ll close by relating this to students. More and more students are coming to college with less and less preparation for both the academic rigors and social rules of college. As college professors we can treat them as unworthy and unmotivated, or we could take a deep breath, take them aside and remind ourselves that in more cases than not, it is not that they don’t have the inherent desire or interest to be successful in college, they just don’t know how to do college. We need to take them aside and say “Our college has found you a worthy student by admitting you, but you don’t know how to do this yet, here are some clear instructions on how to do college here.” I think the search committee at Nazareth would do well to take a page out of their own teaching philosophy and said to this candidate “Our college has found you a worthy candidate by offering you the job, but you don’t know how to negotiate for a job at a SLAC, here’s some information on how to do that.” Who knows, maybe they say they can’t honor any of those provisions and she turns them down. But then they go to their next candidate, and she doesn’t spill the beans on the internet.

Posted in science | Tagged , , | 12 Comments

How much does it matter how students feel?

As I prepare my tenure portfolio, I am catching up on entering in my student evaluation data and comments into my big spreadsheet. While I don’t think student evaluations should serve as the only data by which to judge teachers, they are full of valuable information about teaching, as long as you are willing to read between the lines. For simple 15 minute reflections at the end of a semester, they are often rich with insight into how students perceive their own learning. I don’t think they are nearly as rich with data about how much students actually learned, but how the experience felt to them. Which leads me to the question in the title: if students mostly report their feelings about the class, how do these feelings relate to what and how much they learned? Like the answers to most questions about a complicated craft like teaching, my answer is that emotions do matter in the classroom, but probably not as much as you think, and it depends.

I think we can being by staking down the posts at either end of the spectrum and rejecting them. If a student hates their experience throughout and finds no meaning in it (“this was miserable, but it got me through to where I can study cool stuff”), then I would I would conclude that the course failed for that student. I see this as a possible outcome and one to be avoided. Learning is connected to emotions, and if a student disengages emotionally from a course, it is a problem. If we put our students through stress, tedium and horrible struggle, but then don’t help them feel rewarded (emotional and otherwise) at the end, students will continuously feel that school is a place to be endured, rather than engaged.

However, I think the other pole is equally toxic. If we conclude that no learning can take place unless students are perfectly comfortable, always engaged and having fun, we do a grave disservice to our students. As Dan Willingham reminds us in a chapter heading in his wonderful “Why Don’t Students Like School?” - “People are naturally curious, but we are not naturally good thinkers.” In other words, thinking is hard. If given the choice, we would rather avoid thinking. This is why, for example, this sentence, which requires you to keep a bit more in working memory, specifically devised as an example for the claim that thinking is hard, since I would never write this on purpose (ok maybe I would but I would edit it afterwards, probably) is harder to understand than the one that follows. When we avoid thinking, we save energy and get on with our lives. Unfortunately, if we spend all semester in a glorified focus group, always attendant to feelings and engagement and not to learning, the learning does not happen all by itself. Reminding students what they already know, and trying to boost their confidence does not teach them anything. As it turns out, it doesn’t even boost their self-esteem. I know that there are teachers who lead their students through emotional transformations, and I so seek to connect with students’ emotions, but I fear we can go too far in fostering student engagement and attention.

So, I am left somewhere in the middle. School is not entertainment, and sometimes steps in learning something are not enjoyable. But if students are entirely emotionally disengaged, then they probably aren’t learning either. As I try to balance the fun and the tedium, the struggle and the joy, I try to help my students use the fun to stay curious and dig deeper (“yes it’s amazing, but why does that illusion look that way? What does it mean about your eyes?”) and frame the struggle as necessary and productive. I make sports metaphors about running and lifting waits and “just getting your reps in.” But ultimately, sometimes the way that they frame their own learning is so powerful it can’t be overcome. In this way, the associated question is just as important “How important do students think their feelings are to their learning?” I think the answer to this question is that students drastically overestimate the importance of their own feelings on their learning, because they often interpret these feelings (of struggle, of discomfort, of failure) through incorrect theories of the mind.

Related to student attitudes on learning, a few comments jumped out at me recently, and reinforced to me how the myth of learning styles can have corrosive consequences. These comments inevitably begin with “I am an x learner.” Sometimes they seem positive, like “I am a visual learner, so I loved the videos and visual examples.” Other times they are negative “I prefer more discussion, so the lectures bored me” or “I prefer to learn through lecture, so the discussions didn’t work for me.”  But ultimately even the positive ones suggest the same underlying attitude. When met with challenge, they have thoughts with the same, tragic, logical form of  “I found this element challenging due to a stable preference or personality trait of mine, therefore I disengaged because there was nothing I could do.”

While some of my colleagues who are fans of progressive education could claim that this attitude is solely a consequence of the student-as-consumer mentality, in which a market model is supposed to honor every preference, no matter how trivial. But I think a second factor contributing to this disengagement with struggle is a vague egalitarianism which seeks to minimize the role of ability, but ends up backfiring. When we tell a student “Oh, don’t worry about memorizing that speech, you’re a visual learner try this instead,” we’re trying to help them find engagement, but just as often we’re helping them avoid struggle. We might be trying to avoid telling them that Johnny just has a better working memory than Billy and can memorize these things easier. But what we should be saying is this: “You might have to work a little harder, but you can still learn this just as well.” I know this isn’t exactly what the data say, but as a teacher I feel like it is better to frame ability as “time to practice to reach competence.”  We might each have a different pace in learning how to walk, talk, read and write, but (with the notable exception of certain disabilities) we can each reach competence in these tasks. We’d be far better off if more students believed the same thing about calculus and neuroscience.

So, in the end, I am happy with a few comments of disgruntled students, groaning that the course was too hard, or had too much reading. And my heart is warmed by the student who remarks “The tests are a bit hard, but I think that is how they should be” and “I liked the pass/fail grade style because it helps me learn without killing my grades.” But the most depressing comment is not the aggrieved student who feels cheated out of a few points and denigrates the whole course because of it, but the resigned one who says “class seemed good, but not for me, was useful for only certain kinds of learners.”

Posted in education, higherEd, psychology | Tagged , , , | 7 Comments