Privacy paranoia at NSF?

Inside Higher Ed has an article with Orwellian-sounding overtones: Data on Minority Doctorates Suppressed. The gist of it is that NSF has tightened up its privacy rules and will no longer be reporting information on the ethnicity of doctorates when the cell size is 5 or smaller.

The trouble is that basic reporting on the demographics of small fields runs up against the new rules, particularly for groups such as Native Americans:

So while we know that in 2005, six black people earned doctorates in earth, atmospheric and marine sciences, the NSF won’t reveal how many earned the degrees in 2006 (covered by the most recent report). Information about the number of Latinos earning degrees in some engineering fields is gone, as are data about a number of categories for black Ph.D.’s. For Native Americans, where the base is smaller, the impact of the new policy is especially dramatic. The report was stripped of information on how many doctorates were awarded to all but 6 of the 35 subfields for which data were collected.

Commenters on the story are aghast, and some rail about Bush administration-style information suppression going on in the NSF. I don't think the motives are quite so sinister. But I am wondering if I might be partly responsible for the changes.

In late 2006 I requested a big batch of Survey of Earned Doctorates data from NSF for the Graduate School Guide. It was expensive and a pain to get, largely because the existing privacy rules blocked access to lots of interesting things, but eventually I managed to get the numbers. The one thing I had no trouble getting was information on PhD demographics. Sex, ethnicity, and citizenship were considered public information, and no cell size restrictions applied. (It's not clear from the article whether the tightening applies only to ethnicity or to sex and citizenship as well, but I'd assume everything.)

Apparently nobody had ever requested institution and discipline-level data before. Although my request was approved, I heard through the grapevine afterwards that the data set caused great consternation within NSF and provoked lots of meetings and arguments. I asked for some additional data later on, but never got an answer.

I suspect that at least part of what happened was that NSF decided after the fact that they were uncomfortable with some of the information they gave me, but that they had no rules in place at the time to prevent the request. I'm speculating that the tightening (at least in part) is to address the perceived problem. If so, I'd be curious to hear what else has been changed.

NSF has an important responsibility to protect the privacy of the people who participate in the Survey of Earned Doctorates. They are asking for data from individuals, and they want to make sure that those participating never have reason to be concerned about their information being released inappropriately. The NSF does a very good job of protecting the privacy of those participating - the rules for data access are quite draconian.

The trouble is that the NSF protects their information so zealously that, at least in my opinion, they compromise the larger mission of their organization. Suppressing things like the number of Native American geophysics PhDs granted is just silly. In what possible way does that reveal anything interesting about any individual? (The only thing I can think of is that you might be able to learn whether a particular individual participated. But you could learn the same thing for cell sizes larger than 5 when there is full participation by the subgroup.)

Computer security professionals have a maxim that the only truly secure computer is one that is off. Some of the new privacy changes sound like a step in the direction of pulling the plug to prevent software viruses.

A more productive approach would be to modify the terms of the survey's privacy policy to favor greater data access, not less. Sure, they might lose a handful of responses from the truly paranoid, but my guess is that people who are that concerned about their privacy aren't going to fill out the survey anyway. Greater access could allow some NSF to do some truly useful things with their data.

There is some good news for any of you who are interested in PhD demographics: there is a better place to get the data. The IPEDS data set contains complete information on the demographics of doctorates, with no data suppression. What's more, IPEDS also has data on master's students and on undergraduates, their data are not limited to science and engineering, they have a much more detailed taxonomy of disciplines than NSF, and they make their data available at the institution rather than the national level. IPEDS data are much more timely than NSF's. And, best of all, you can get the full data set without having to pay NORC lots of money. The NSF may have just done you a big favor.

[0 comments]

PE for grad students, part 2

This month's Wired has more on the exercise/cognitive function connection : aerobic exercise helps, weight lifting doesn't.

They also have a rundown on drugs that allegedly enhance cognitive functioning, handy for any who are thinking about emulating the 20% of scientists admitting to using brain-boosting drugs in a recent, informal survey in Nature.

The drugs have questionable efficacy, are expensive, and have nasty side effects (which Wired details - my favorite, for methamphetamines: "Prison"). Exercise, in contrast, works well, is free, and is beneficial pretty much all around. Forget the drugs - put on some sneakers and go run.

[0 comments]

PE for grad students?

Do the following strike you as things that graduate students or postdocs excel at?

• make plans
• keep track of time
• keep track of more than one thing at once
• meaningfully include past knowledge in discussions
• engage in group dynamics
• evaluate ideas
• reflect on our work
• change our minds and make mid-course and corrections while thinking, reading and writing
• finish work on time
• ask for help
• wait to speak until we're called on
• seek more information when we need it.

For many of these items, particularly the ones relating to planning and time management, my experience has been that these are things at which grad students and postdocs are particularly bad. And they're all important to career success. These kinds of activities are aspects of a broad measure of cognitive capability called "executive function" (the list was taken from here ).

I just saw a fascinating talk at Google by John Medina on his book, Brain Rules. The book is a summary of findings from neuroscience that are relevant to everyday life. In the first half of the talk Medina examines the relationship between exercise and cognitive function. Short summary: modest amounts of aerobic exercise (~20 minutes 3-4 times per week) have a big, quantifiable effect on executive function. It also helps prevent and treat depression, a condition to which grad students are especially prone.

One interesting implication is that exercise is an important component of education. Perhaps graduate programs should incorporate a mandatory PE requirement? At the very least, smart institutions should be making sure their grad students and postdocs have access to the gym, and they should encourage physical activity through things like intramural sports leagues for grads/postdocs. Students and their advisors shouldn't view exercise as time away from the bench, but rather as an investment in higher quality output.

Check out the talk.

[0 comments]

"How Scientific Gains Abroad Pay Off in the U.S."

An interesting piece in today's NY Times: http://www.nytimes.com/2008/04/20/technology/20ping.html

Quick summary: it's getting easier for US companies to farm out research tasks to low wage countries. America is becoming a "postscientific society": our future value-add will be in "product design, marketing and finance" not in scientific innovation.

In the short-term at least, higher spending on scientists by India and China could create a glut of them in these countries, driving wages down further and making the costs of acquiring science even lower.

Not to worry, though:

For the foreseeable future, United States companies will need their own highly paid scientists “to evaluate the purchase of foreign science and to make sense of it in their own labs,” says Daniel Sarewitz, director of the Consortium for Science, Policy and Outcomes at Arizona State University.

The implication is that a very different skill set will be needed by US scientists in the future. Prospects will continue to worsen in basic research as the cost of doing domestic research becomes prohibitively expensive relative to doing so elsewhere. The real opportunities will be in figuring out how to bring discoveries to market. We'll need some basic research to keep up some core skills and for teaching purposes, but increasing emphasis may be placed on people doing applied work, translational work, and so on.

[0 comments]

Graduate Education at Stanford

Mark Horowitz, Associate Vice Provost for Graduate Education at Stanford, gave a talk at Google a few weeks ago about some of the things Stanford is working on to enhance the quality of its graduate programs. After many years (a decade or more?) of having no senior leadership with responsibility for graduate education at the university level, Stanford has finally created an office of graduate education. Mark's talk was an overview of the initiatives coming out of the new office.

Stanford's new activity in graduate education is based on an internal report. The basic thrusts are

• more interdisciplinary education,
• greater diversity, and
• leadership training.

These are all fairly standard ideas, but a few things struck me about Stanford's approach.

• Stanford has a ridiculous amount of financial resources. It appears that the university is backing this initiative wholeheartedly. This is not just a few seminars cobbled together by an underfunded graduate or postdoc office - it looks like the real deal.

• Stanford has amazing courses throughout the institution, but usually access to courses in one department is limited to students in that department. The new Office is doing the smart thing of trying to better leverage existing resources. To give a sense of how serious the university is about this, they are talking about things like moving the Law School from a semester to a quarter system (maybe the other way around? this is from memory) so that students outside the Law School can more easily take graduate courses and vice versa.

• One problem people have with taking classes outside their department is that it's hard to find appropriate classes in other fields because there are so many classes out there. The catalog is paper based (!) and there is not much descriptive material on courses. Stanford has set up a web site called CourseRank, which is an exercise in using collaborative filtering to help students find relevant courses outside their departments. I suspect that CourseRank may be an exercise in trying to set up a social networking site for the sake of cool points, but the idea seems sound. One lower tech and much cheaper way to accomplish some of the same goals would be to do some basic mining of data from the registrar. Looking at past cross-disciplinary enrollments would be a simple way to provide students with suggestions for possible good courses to take outside their departments.

• Stanford is setting up career development as formal classes during the summer and before fall quarter (the Stanford Graduate Summer Institute ). Students have to apply, and in some cases pay a fee. Having an application process is smart - it makes the course into a desirable thing that you have to compete for rather than a freebie extra thing that people have to be dragged to. The fee accomplishes the same purpose, plus provides resources to sustain the effort.

• Career development workshops are being pitched as "leadership training". It's the same thing, but "leadership" is much sexier and probably easier to sell. I suspect that because of the word's additional usage as "leader in a field", "leadership" has positive connotations even for the most unreconstructed students-must-always-be-at-the-bench faculty.

• Stanford is trying to set up mentors for students outside of the regular faculty advisors. The vision appears to be pairing students with area professionals, e.g. people at dot-coms, biotechs, etc. Definitely a good way to provide some guidance that many faculty are ill-equipped to do.

All in all a good effort. It's especially promising because places like Stanford tend to inspire places that want to be like Stanford. I'm looking forward to seeing the results and to seeing other universities try to follow in their footsteps.

[0 comments]

Graduate School Guidance

One key data set I used in building the Graduate School Guide was IPEDS, which is put out by the Department of Education's National Center for Education Statistics. IPEDS is incredibly useful for this kind of thing: the data set contains a near-complete list of all colleges and universities in the country together with detailed information about student enrollments and degrees granted. The data are well-documented, easy to access (you can download everything in CSV files or grab custom subsets of the data from an online tool), and relatively recent (2006).

NCES delivers tremendous value with IPEDS in several ways:

1) They provide a lot of summary statistics from the data to give an overview the current state of post-secondary education.

2) They make it easy for others to get ahold of the data, and they provide thorough documentation so that once you have the data, it's easy to work with. As a result, third parties such as The Chronicle of Higher Education delve more deeply into the data and deliver additional interesting insights.

3) They help participating institutions use the data set to compare themselves to peer institutions. In so doing, they implicitly give guidance on good questions for institutions to ask about themselves (e.g. how much financial aid do we give relative to our peers? how well do we retain students?)

4) They provide data to third parties who help students choose colleges. It's virtually certain that US News, Peterson's, Fiske, etc, all draw upon IPEDS data.

5) They run College Opportunities OnLine, a site that provides useful information directly to prospective college students.

6) They combine multiple data sets in ways that increase the value of all the components. For example, on the COOL web site they combine their core data on universities and graduations with data on libraries and on campus crime.

The NSF has similarly interesting sets of data in their Survey of Earned Doctorates, Survey of Doctorate Recipients, and the Survey of Graduate Students and Postdoctorates in Science and Engineering, but they have not been anywhere near as effective at extracting value from them.

Here's how NSF stacks up to NCES:

1) The NSF provides data reports in the form of very basic, annual (or bi-annual) summary statistics and in Science and Engineering Indicators.

2) NSF has a decent tool for extracting additional summary statistics in WebCaspar. Unfortunately, the data are aggregated at such a high level (nationally for most things), that you can't get at the most interesting information. Basically you can't get anything beyond a count of individuals at the level of a single institution.

It is possible to get ahold of more detailed data than what's in WebCaspar, but the process is difficult, expensive, and time-consuming. The data we used on our site cost $7,000 and took months for NORC to generate. There is a do-it-yourself alternative if you have a facility that satisfies NSF's security criteria (stringent), can handle NSF's audits, and have a license for SAS (expensive).

NSF has a responsibility to protect the privacy of survey participants, and to their credit, they take that responsibility very seriously. However, I think they tend to err so far on the side of caution that they detract from their own mission.

3) As far as I know, NSF doesn't make any effort to help universities use the data they generate in useful ways. I think this is an area in which NSF could take a real leadership role. For example, people have been expressing dismay for years about the length of time it takes to earn a PhD. The NSF has detailed data on exactly how long it has taken to earn pretty much every single PhD in the country. Providing institutions with stats on their own times to degree relative to their peers could be a powerful catalyst for inspiring improvements. They could provide similar motivation on placement rates, funding levels, and so on.

The value the NSF could provide here is not just in providing the data - it's in getting institutions to ask the right questions of themselves. I have talked to several deans who participated in the recent CGS study of attrition. One common thread I have heard is that they were able to make progress on reducing their attrition rates - the big problem they had was that nobody had ever looked at the issue before.

4) People love to hate US News's rankings. The chief complaint is the exclusive reliance on reputation. There has been sporadic talk over the years about institutions proposing alternatives, but the talk has never amounted to anything. As we have demonstrated here at phds.org, NSF's data can form part of a more balanced approach. Surely a set of US News rankings that incorporated outcome measures as well as reputational measures would be more helpful to students and less objectionable to faculty members than their current approach.

5) As far as I know, NSF does not use their data to provide any services targeted at prospective graduate students. I think that's a reasonable call on their part, since I don't think it's a great fit for the skill set that the organization possesses. However, if they were to make department level data sets available (aggregated over sufficient numbers of years to protect privacy) on a regular basis to third parties such as US News, Peterson's, phds.org, etc, they could ensure that would-be students would benefit from the NSF's hard work.

6) As graduate-school.phds.org demonstrates, the NSF data sets are much more interesting when combined. One frustration I had in working with the NSF's data sets is that they are not designed to work well with each other or with outside data such as IPEDS, so combining them took months of work.

Each of NSF's data sets uses a different, incompatible taxonomy of disciplines, all of which are different from the one used by IPEDS. The NSF uses an outdated list of institutions (FICE codes) that results in information for entire state university systems being lumped together and makes it difficult to combine with information from IPEDS, which uses an up-to-date list. Rethinking the NSF's field codes and updating their institution lists would make their data more valuable to end users and would simplify the lives of the people at the institutions who have to answer both NSF and NCES surveys.

So here's what I think NSF could do to remedy things:

• Take a leadership role in defining metrics of interest, and provide institutions with summary statistics about their own performance relative to their peers.

• Rework their various classification schemes so that their data is easy to combine and compatible with outside data.

• Provide better quality control over department-supplied self-classifications and consider ways to handle multidisciplinary departments (e.g. a department of Mathematics and Statistics)

• Build a list of departments and strive to provide department-level data when practical.

• Release regular department-level data sets (aggregated over multiple years and suppressed as needed to protect participant privacy) for use by third parties.

I think the result would be universities receiving information they can use to make valuable improvements to their programs and prospective students getting better guidance on choosing suitable programs. That's a pretty big impact.

[0 comments]

Pardon the dust

I've spent the last 6 weeks of vacation time and weekends sequestered in a secure undisclosed location busily upgrading phds.org. The long-promised upgrade to our graduate school guide is now out, and the whole site has been moved over to a much faster (and hopefully more reliable) server. You should now be seeing a much more responsive site with much less down time. The server consolidation and upgrade is just the beginning of a series of improvements to the site, so stay tuned!

In the process of the move and upgrade of the back end, a few minor things have broken. I'm busily fixing things, and hope to have the main things nailed down in the next week or so.

[0 comments]

Fixing the NIH grant-making process

There's a piece in this week's Chronicle about some possible changes in the NIH's grant-making process. About 6 months ago, the NIH solicited suggestions from the general public for ways to improve the process, and an advisory committee has been sifting through the thousands of ideas they received.

The article describes a few broad classes of ideas:

• Streamlining the application process by reducing the length of grants from 25 pages to 15
• Limiting the number of proposals a person can submit
• Basing funding decisions more on an individual's than on specifics of their proposal
• Providing more affirmative action for younger scientists

These ideas aren't yet official recommendations - those won't be out until later this week - but they are likely indicative of the kinds of things the NIH will actually do. Many of these ideas are good ones; I'm just not convinced they will have the hoped-for effects.

Reducing the amount of effort required to submit a proposal sounds great. People invest huge amounts of effort on their proposals; I'd much rather have them spending their time doing science than chasing money. The trouble is that making it easier to submit a grant, will probably mean that people will submit more grants, driving the success rate down even more. Going back to the lottery ticket/grant analogy: during the budget doubling, the NIH increased both the odds that a ticket would win and the amount of money paid out by a winning ticket. Not surprisingly, people bought a lot more tickets. Streamlining the proposal process, while a worthy goal, effectively cuts the price of a ticket, which again increases the net payoff. If this happens, I predict we'll see even lower success rates in the future.

Limiting the number of proposals someone can submit is a non-starter, I suspect, despite the AAMC's endorsement of the idea. The idea has some merits: it would probably reduce the number of proposals the NIH receives and force people to submit only their best ideas. However, I think that there are legitimate scientific reasons for some larger labs to be submitting multiple proposals per year. Zerhouni opposes the idea. A better alternative might be to impose a surcharge, like publication fees charged by journals, to cover review costs on proposals after the first. This would reduce the number of people submitting multiple proposals while still making it possible to do so.

Judging proposals on the reputation of the submitter rather than on their content is a recipe for all sorts of trouble. Sure, it would make life more convenient for the elites, but I suspect that the result would be complacency, not greater willingness to take risks - just throw something over the fence and you get your funding, so why make the effort? Younger scientists are a source of a lot of crazy new ideas, but they don't have much of a track record, so this kind of scheme could shut them out of funding even more than they already are.

There's an alternative approach that I think avoids most of these difficulties - more next time.

[1 comment]