Engineering Science - Graduate School

Privacy paranoia at NSF?

2008-05-03T16:48:00Z

Inside Higher Ed has an article with Orwellian-sounding overtones: Data on Minority Doctorates Suppressed. The gist of it is that NSF has tightened up its privacy rules and will no longer be reporting information on the ethnicity of doctorates when the cell size is 5 or smaller.

The trouble is that basic reporting on the demographics of small fields runs up against the new rules, particularly for groups such as Native Americans:

So while we know that in 2005, six black people earned doctorates in earth, atmospheric and marine sciences, the NSF won’t reveal how many earned the degrees in 2006 (covered by the most recent report). Information about the number of Latinos earning degrees in some engineering fields is gone, as are data about a number of categories for black Ph.D.’s. For Native Americans, where the base is smaller, the impact of the new policy is especially dramatic. The report was stripped of information on how many doctorates were awarded to all but 6 of the 35 subfields for which data were collected.

Commenters on the story are aghast, and some rail about Bush administration-style information suppression going on in the NSF. I don't think the motives are quite so sinister. But I am wondering if I might be partly responsible for the changes.

In late 2006 I requested a big batch of Survey of Earned Doctorates data from NSF for the Graduate School Guide. It was expensive and a pain to get, largely because the existing privacy rules blocked access to lots of interesting things, but eventually I managed to get the numbers. The one thing I had no trouble getting was information on PhD demographics. Sex, ethnicity, and citizenship were considered public information, and no cell size restrictions applied. (It's not clear from the article whether the tightening applies only to ethnicity or to sex and citizenship as well, but I'd assume everything.)

Apparently nobody had ever requested institution and discipline-level data before. Although my request was approved, I heard through the grapevine afterwards that the data set caused great consternation within NSF and provoked lots of meetings and arguments. I asked for some additional data later on, but never got an answer.

I suspect that at least part of what happened was that NSF decided after the fact that they were uncomfortable with some of the information they gave me, but that they had no rules in place at the time to prevent the request. I'm speculating that the tightening (at least in part) is to address the perceived problem. If so, I'd be curious to hear what else has been changed.

NSF has an important responsibility to protect the privacy of the people who participate in the Survey of Earned Doctorates. They are asking for data from individuals, and they want to make sure that those participating never have reason to be concerned about their information being released inappropriately. The NSF does a very good job of protecting the privacy of those participating - the rules for data access are quite draconian.

The trouble is that the NSF protects their information so zealously that, at least in my opinion, they compromise the larger mission of their organization. Suppressing things like the number of Native American geophysics PhDs granted is just silly. In what possible way does that reveal anything interesting about any individual? (The only thing I can think of is that you might be able to learn whether a particular individual participated. But you could learn the same thing for cell sizes larger than 5 when there is full participation by the subgroup.)

Computer security professionals have a maxim that the only truly secure computer is one that is off. Some of the new privacy changes sound like a step in the direction of pulling the plug to prevent software viruses.

A more productive approach would be to modify the terms of the survey's privacy policy to favor greater data access, not less. Sure, they might lose a handful of responses from the truly paranoid, but my guess is that people who are that concerned about their privacy aren't going to fill out the survey anyway. Greater access could allow some NSF to do some truly useful things with their data.

There is some good news for any of you who are interested in PhD demographics: there is a better place to get the data. The IPEDS data set contains complete information on the demographics of doctorates, with no data suppression. What's more, IPEDS also has data on master's students and on undergraduates, their data are not limited to science and engineering, they have a much more detailed taxonomy of disciplines than NSF, and they make their data available at the institution rather than the national level. IPEDS data are much more timely than NSF's. And, best of all, you can get the full data set without having to pay NORC lots of money. The NSF may have just done you a big favor.

PE for grad students, part 2

2008-04-24T02:46:00Z

This month's Wired has more on the exercise/cognitive function connection : aerobic exercise helps, weight lifting doesn't.

They also have a rundown on drugs that allegedly enhance cognitive functioning, handy for any who are thinking about emulating the 20% of scientists admitting to using brain-boosting drugs in a recent, informal survey in Nature.

The drugs have questionable efficacy, are expensive, and have nasty side effects (which Wired details - my favorite, for methamphetamines: "Prison"). Exercise, in contrast, works well, is free, and is beneficial pretty much all around. Forget the drugs - put on some sneakers and go run.

PE for grad students?

2008-04-22T14:17:00Z

Do the following strike you as things that graduate students or postdocs excel at?

make plans

keep track of time

keep track of more than one thing at once

meaningfully include past knowledge in discussions

engage in group dynamics

evaluate ideas

reflect on our work

change our minds and make mid-course and corrections while thinking, reading and writing

finish work on time

ask for help

wait to speak until we're called on

seek more information when we need it.

For many of these items, particularly the ones relating to planning and time management, my experience has been that these are things at which grad students and postdocs are particularly bad. And they're all important to career success. These kinds of activities are aspects of a broad measure of cognitive capability called "executive function" (the list was taken from here ).

I just saw a fascinating talk at Google by John Medina on his book, Brain Rules. The book is a summary of findings from neuroscience that are relevant to everyday life. In the first half of the talk Medina examines the relationship between exercise and cognitive function. Short summary: modest amounts of aerobic exercise (~20 minutes 3-4 times per week) have a big, quantifiable effect on executive function. It also helps prevent and treat depression, a condition to which grad students are especially prone.

One interesting implication is that exercise is an important component of education. Perhaps graduate programs should incorporate a mandatory PE requirement? At the very least, smart institutions should be making sure their grad students and postdocs have access to the gym, and they should encourage physical activity through things like intramural sports leagues for grads/postdocs. Students and their advisors shouldn't view exercise as time away from the bench, but rather as an investment in higher quality output.

Check out the talk.

Graduate Education at Stanford

2008-04-16T02:41:00Z

Mark Horowitz, Associate Vice Provost for Graduate Education at Stanford, gave a talk at Google a few weeks ago about some of the things Stanford is working on to enhance the quality of its graduate programs. After many years (a decade or more?) of having no senior leadership with responsibility for graduate education at the university level, Stanford has finally created an office of graduate education. Mark's talk was an overview of the initiatives coming out of the new office.

Stanford's new activity in graduate education is based on an internal report. The basic thrusts are

more interdisciplinary education,
greater diversity, and
leadership training.

These are all fairly standard ideas, but a few things struck me about Stanford's approach.

Stanford has a ridiculous amount of financial resources. It appears that the university is backing this initiative wholeheartedly. This is not just a few seminars cobbled together by an underfunded graduate or postdoc office - it looks like the real deal.
Stanford has amazing courses throughout the institution, but usually access to courses in one department is limited to students in that department. The new Office is doing the smart thing of trying to better leverage existing resources. To give a sense of how serious the university is about this, they are talking about things like moving the Law School from a semester to a quarter system (maybe the other way around? this is from memory) so that students outside the Law School can more easily take graduate courses and vice versa.
One problem people have with taking classes outside their department is that it's hard to find appropriate classes in other fields because there are so many classes out there. The catalog is paper based (!) and there is not much descriptive material on courses. Stanford has set up a web site called CourseRank, which is an exercise in using collaborative filtering to help students find relevant courses outside their departments. I suspect that CourseRank may be an exercise in trying to set up a social networking site for the sake of cool points, but the idea seems sound. One lower tech and much cheaper way to accomplish some of the same goals would be to do some basic mining of data from the registrar. Looking at past cross-disciplinary enrollments would be a simple way to provide students with suggestions for possible good courses to take outside their departments.
Stanford is setting up career development as formal classes during the summer and before fall quarter (the Stanford Graduate Summer Institute ). Students have to apply, and in some cases pay a fee. Having an application process is smart - it makes the course into a desirable thing that you have to compete for rather than a freebie extra thing that people have to be dragged to. The fee accomplishes the same purpose, plus provides resources to sustain the effort.
Career development workshops are being pitched as "leadership training". It's the same thing, but "leadership" is much sexier and probably easier to sell. I suspect that because of the word's additional usage as "leader in a field", "leadership" has positive connotations even for the most unreconstructed students-must-always-be-at-the-bench faculty.
Stanford is trying to set up mentors for students outside of the regular faculty advisors. The vision appears to be pairing students with area professionals, e.g. people at dot-coms, biotechs, etc. Definitely a good way to provide some guidance that many faculty are ill-equipped to do.

All in all a good effort. It's especially promising because places like Stanford tend to inspire places that want to be like Stanford. I'm looking forward to seeing the results and to seeing other universities try to follow in their footsteps.

Graduate School Guidance

2008-02-06T02:09:00Z

One key data set I used in building the Graduate School Guide was IPEDS, which is put out by the Department of Education's National Center for Education Statistics. IPEDS is incredibly useful for this kind of thing: the data set contains a near-complete list of all colleges and universities in the country together with detailed information about student enrollments and degrees granted. The data are well-documented, easy to access (you can download everything in CSV files or grab custom subsets of the data from an online tool), and relatively recent (2006).

NCES delivers tremendous value with IPEDS in several ways:

1) They provide a lot of summary statistics from the data to give an overview the current state of post-secondary education.

2) They make it easy for others to get ahold of the data, and they provide thorough documentation so that once you have the data, it's easy to work with. As a result, third parties such as The Chronicle of Higher Education delve more deeply into the data and deliver additional interesting insights.

3) They help participating institutions use the data set to compare themselves to peer institutions. In so doing, they implicitly give guidance on good questions for institutions to ask about themselves (e.g. how much financial aid do we give relative to our peers? how well do we retain students?)

4) They provide data to third parties who help students choose colleges. It's virtually certain that US News, Peterson's, Fiske, etc, all draw upon IPEDS data.

5) They run College Opportunities OnLine, a site that provides useful information directly to prospective college students.

6) They combine multiple data sets in ways that increase the value of all the components. For example, on the COOL web site they combine their core data on universities and graduations with data on libraries and on campus crime.

The NSF has similarly interesting sets of data in their Survey of Earned Doctorates, Survey of Doctorate Recipients, and the Survey of Graduate Students and Postdoctorates in Science and Engineering, but they have not been anywhere near as effective at extracting value from them.

Here's how NSF stacks up to NCES:

1) The NSF provides data reports in the form of very basic, annual (or bi-annual) summary statistics and in Science and Engineering Indicators.

2) NSF has a decent tool for extracting additional summary statistics in WebCaspar. Unfortunately, the data are aggregated at such a high level (nationally for most things), that you can't get at the most interesting information. Basically you can't get anything beyond a count of individuals at the level of a single institution.

It is possible to get ahold of more detailed data than what's in WebCaspar, but the process is difficult, expensive, and time-consuming. The data we used on our site cost $7,000 and took months for NORC to generate. There is a do-it-yourself alternative if you have a facility that satisfies NSF's security criteria (stringent), can handle NSF's audits, and have a license for SAS (expensive).

NSF has a responsibility to protect the privacy of survey participants, and to their credit, they take that responsibility very seriously. However, I think they tend to err so far on the side of caution that they detract from their own mission.

3) As far as I know, NSF doesn't make any effort to help universities use the data they generate in useful ways. I think this is an area in which NSF could take a real leadership role. For example, people have been expressing dismay for years about the length of time it takes to earn a PhD. The NSF has detailed data on exactly how long it has taken to earn pretty much every single PhD in the country. Providing institutions with stats on their own times to degree relative to their peers could be a powerful catalyst for inspiring improvements. They could provide similar motivation on placement rates, funding levels, and so on.

The value the NSF could provide here is not just in providing the data - it's in getting institutions to ask the right questions of themselves. I have talked to several deans who participated in the recent CGS study of attrition. One common thread I have heard is that they were able to make progress on reducing their attrition rates - the big problem they had was that nobody had ever looked at the issue before.

4) People love to hate US News's rankings. The chief complaint is the exclusive reliance on reputation. There has been sporadic talk over the years about institutions proposing alternatives, but the talk has never amounted to anything. As we have demonstrated here at phds.org, NSF's data can form part of a more balanced approach. Surely a set of US News rankings that incorporated outcome measures as well as reputational measures would be more helpful to students and less objectionable to faculty members than their current approach.

5) As far as I know, NSF does not use their data to provide any services targeted at prospective graduate students. I think that's a reasonable call on their part, since I don't think it's a great fit for the skill set that the organization possesses. However, if they were to make department level data sets available (aggregated over sufficient numbers of years to protect privacy) on a regular basis to third parties such as US News, Peterson's, phds.org, etc, they could ensure that would-be students would benefit from the NSF's hard work.

6) As graduate-school.phds.org demonstrates, the NSF data sets are much more interesting when combined. One frustration I had in working with the NSF's data sets is that they are not designed to work well with each other or with outside data such as IPEDS, so combining them took months of work.

Each of NSF's data sets uses a different, incompatible taxonomy of disciplines, all of which are different from the one used by IPEDS. The NSF uses an outdated list of institutions (FICE codes) that results in information for entire state university systems being lumped together and makes it difficult to combine with information from IPEDS, which uses an up-to-date list. Rethinking the NSF's field codes and updating their institution lists would make their data more valuable to end users and would simplify the lives of the people at the institutions who have to answer both NSF and NCES surveys.

So here's what I think NSF could do to remedy things:

Take a leadership role in defining metrics of interest, and provide institutions with summary statistics about their own performance relative to their peers.
Rework their various classification schemes so that their data is easy to combine and compatible with outside data.
Provide better quality control over department-supplied self-classifications and consider ways to handle multidisciplinary departments (e.g. a department of Mathematics and Statistics)
Build a list of departments and strive to provide department-level data when practical.
Release regular department-level data sets (aggregated over multiple years and suppressed as needed to protect participant privacy) for use by third parties.

I think the result would be universities receiving information they can use to make valuable improvements to their programs and prospective students getting better guidance on choosing suitable programs. That's a pretty big impact.

Meanwhile in the Senate...

2007-05-10T16:24:00Z

I've just taken a look at the Senate's hefty bill to reauthorize the NSF, S 761. Like the House equivalent, there are some good and interesting things in the bill.

There is a lot of new money for graduate fellowships:

IGERT - Increased funding to the IGERT program, like in the House bill. This would fund provide funding for more graduate students, but IGERT programs appear to be much better suited to providing people with more of a range of career options than traditional PhD programs. The stipends, which are semi-portable traineeships rather research assistantships, are pretty hefty, too. $30K/year is more than some postdocs are paid - perhaps it will stimulate some upward growth in S&E graduate stipends overall.

The Senate's numbers are a lot lower than the House numbers - $22M/year in 2008 to $55M/year in 2011 (as opposed to ~$75M-$150M/year for the House). Since there was pushback from the administration on the way the House was proposing to fund IGERT (as a fixed percentage of the NSF's budget), I imagine something more like the Senate's version will go through.

Graduate fellowships - The Senate bill also allocates a good chunk of money for new old-style fellowships, ranging from $24M/year in 2008 to $60M/year in 2011. The House bill doesn't provide any new money for fellowships apart from the overall budget doubling.

DOE graduate fellowships - $9M/year-$35M/year in fellowships for "students pursuing a doctoral degree in a mission area of the Department [of Energy" - energy and nukes, presumably.

So altogether the Senate is proposing increased funding for graduate fellowships in the amount of $140M by 2011, which translates to about 2,800 new doctoral students per year.

A little digging in WebCASPAR shows that NSF funds about 20,000 full-time graduate students (about 5% of all full-time graduate students). If we make the assumption that the NSF funds primarily doctoral students and that NSF funding is spread over 5 years, we find that NSF funds on the order of 4,000 doctoral students/year.

So the combination of a budget doubling plus the new fellowships, if allocations are held constant, would increase NSF-funded doctoral students by about 4,000+2,800 = 6,800. This estimate is probably on the high side, since IGERT stipends are pretty high, some NSF funding probably goes to master's students, and the new money for fellowships may result in reallocations of other funds away from graduate fellowships; let's say there will be 5,000 new NSF-funded doctoral students per year. In 2005 there were 29,000 S&E PhDs granted, 13,000 of which were in the physical sciences and engineering. NSF's funding is concentrated in the physical sciences and engineering - NSF's existing 4,000 or so fellowships combined with 5,000 new ones would mean that NSF could end up funding most doctorate recipients in those fields.

Will this increase the number of doctorates granted per year? Maybe - Richard Freeman has some interesting work that suggests so - but perhaps not by 5,000. Once a department has its teaching needs covered, the incentive to enroll graduate students is reduced. New money from NSF frees up existing departmental funds that pay TAs for other purposes. Labs will be staffing up once new NSF grants start flowing, and I'd bet that they'll want to hire postdocs, not graduate students. I predict an increase in the total number of postdocs in many math and physical sciences fields. In fields, like CS and engineering, in which most people go to industry after graduating, postdocs will probably be harder to come by, so they'll end up with more grad students.

There is a modest amount of money to fund Professional Science Master's degree programs.

Professional Science Master's programs were created by grants from the Sloan Foundation back in 1997. I have met several people who have been through PSM programs, and they sound great - they're essentially hybrid-MBA/science degrees. If the country needs more scientists to spur economic growth, this is a great way to get them. The programs graduate people who know enough science to do useful things in theory, but who also know enough about navigating companies to actually accomplish things in practice.

I'm very encouraged to see the possibility of the NSF picking up PSM funding. Once institutions get into the business of bringing real, professional skills into science curricula, it's only a matter of time before the ideas diffuse into more mainstream science programs. There's nothing in the House bill about PSM programs, so I think this is iffy, but since the amount of money is modest ($9M/year in 2008 to $20M in 2011), I'm hoping it will pass.

There is a ton of additional material in both the Senate and the House bills about teacher training. More on that in a future post.

More Graduate Students - Brought to You by IGERT

2007-04-26T20:31:00Z

The increase in graduate students discussed earlier in the week just came a step closer to reality: the House and Senate just passed a set of bills that will steer a big chunk of funding toward new graduate fellowships, among other things. I assume there will be some negotiation in conference over the final form, but the boost to graduate numbers is a lot closer to reality.

One caveat: the bills just authorize increased spending, but they don't actually provide it. So there is room for things to be cut by failing to be funded in appropriations bills.

I have started looking through a few of the bills, and as best I can tell, there is some interesting and good stuff in them. There are also some things that are disappointingly omitted. As with HR 1453, the bills smack of AAAS Fellow influence - hats off to any of you who are reading this.

The House passed 3 bills:

HR 1867, the National Science Foundation Authorization Act of 2007, which doubles the NSF budget,
HR 363, the Sowing the Seeds Through Science and Engineering Research Act, which funds a bunch of undergraduate and graduate fellowships, and
HR 362, the 10,000 Teachers, 10 Million Minds Science and Math Scholarship Act, which funds S&E teacher training.

The Senate passed a single, 200+ page bill, S 761, the America Competes Act (more formally, the America Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science Act).

I've just started looking into these, and I imagine they'll take a few posts to digest.

HR 363 is the House bill most relevant to grad students. It passed 397 to 20, so at least some portion of it seems pretty likely to happen. Doubling the NSF budget will likely increase spending on grad students as well, but probably not in ways qualitatively different from current expenditures.

I notice in GovTrack that authorization for funds has been stripped out of the bill. So I'd guess that it will be funded at a lower level than the bill calls for.

Section 4 of the bill is the interesting bit:

SEC. 4. INTEGRATIVE GRADUATE EDUCATION AND RESEARCH TRAINEESHIP PROGRAM.

(a) Funding- For each of the fiscal years 2008 through 2012, the Director of the National Science Foundation shall allocate at least 1.5 percent of funds appropriated for Research and Related Activities to the Integrative Graduate Education and Research Traineeship program.

(b) Coordination- The Director shall coordinate with Federal departments and agencies, as appropriate, to expand the interdisciplinary nature of the Integrative Graduate Education and Research Traineeship program.

(c) Authority to Accept Funds From Other Agencies- The Director is authorized to accept funds from other Federal departments and agencies to carry out the Integrative Graduate Education and Research Traineeship program.

I'm disappointed that there are no provisions for measuring efficacy or for linking expenditures to the state of the labor market, so it ends up being a command-and-control type of program (like most of the rest of science, alas). That being said, funding IGERT rather than more traditional NSF fellowships seems like a promising way to go.

The good thing about IGERT (from the IGERT FAQ):

"A major objective of NSF's IGERT program is to train students in areas where industry, government and academic institutions are experiencing a shortfall. IGERT graduates may work in industries ranging from pharmaceutics to petrochemicals, government laboratories devoted to health, commerce or energy, small teaching colleges and major research universities. An important benefit of the IGERT programs is that most students have opportunities to sample these locations during their training. This makes it easier to decide which career environment is right for you."

So IGERT is not about creating a bunch of new professors.

Another good thing about IGERT he IGERT money goes to traineeships rather than research assistantships. The funding is not as portable as what Romer was calling for, but funding tied to the department is a lot more portable than funding tied to a researcher. I think it's a reasonable compromise between full portability and the increasingly common no portability.

(An aside: There have been a number of National Academy reports that have called for more portable funding, but they always seem to get shot down. I've spoken to some National Science Board members about the issue, and the story I have gotten from them is that they are terrified that given full portability, students will all end up at some program other than their own. This discussion gives a sense of the "debate." It's total BS - people say they don't want more portability because it might screw up the wonderful system we have now. But there is no analogous concern raised when funding shifts away from portability as it has steadily for the last couple of decades. Moreover, they complain about the lack of data documenting benefits of portability, but they never actually try running any experiments to gather any. It's not rocket science. In fact, there is actually a lot of data available in the SED and the SDR that would let one compare career outcomes for people funded by portable vs. non-portable funding, but nobody has bothered to run the stats. As best I can tell, nobody wants to know.)

The less good thing is that IGERT seems to emphasize interdisciplinarity for its own sake. Googling "IGERT" turns up some goofy sounding programs - "Interactive Digital Multimedia" at UC Santa Barbara (you can get a PhD in that?), "Biological Invasions"(!) at UC Davis, and so on. But plenty of sensible things, too.

IGERT seems to be about creating new programs, too, but maybe I'm just not understanding correctly. I'd be happier if there were provisions for existing programs getting their acts together in terms of providing professional development for their students, but maybe IGERT would lead to some diffusion.

It's interesting that the bill steers a fixed fraction of the NSF's budget toward IGERT rather than a specific amount. I'm guessing that's so that IGERT would benefit proportionally if the NSF budget is doubled and so that there isn't some convenient dollar amount to target once appropriations committees take their knives to the bill? 1.5% of a $5 billion dollar budget is $75 million. IGERT currently gets about $12 million / year, so that's a hefty boost. And $150 million if the NSF's budget doubles.

IGERT students get $30K stipends (more than NSF gave their postdocs back in my day, stingy things), plus tuition (call it $20K) plus maybe some overhead for health insurance, etc (say $10K). Probably there is some faculty and institutional money in IGERT (after all, creating new programs isn't cheap). So let's say half the money goes to students at $60K apiece. $75 million buys you a grand total of ... 625 students. And 100 of those were already funded. Even doubled, we're talking a not very large number of people. Romer's proposal is for an extra 17,000 new PhDs per year, so this is a tiny fraction of what he's talking about.

The administration has indicated that they don't like the idea of having a fixed fraction of the NSF's budget allocated to a particular program (because the other 98.5% just isn't enough). I assume this is because of pushback from senior people who don't like the idea of either portable funding or their pot of grant money being diminished - similar complaints were heard during the NIH doubling.

More Graduate Students?

2007-04-24T16:52:00Z

There have been a few bills working their way through Congress that seek to significantly increase the number of graduate students. Why now, at a time when people are asking, "Are we training too many PhDs?"

Much of the current impetus comes from the National Academies report, Rising Above the Gathering Storm -- the bills in question also address other Gathering Storm recommendations -- so the question becomes, where did the Gathering Storm report get the idea? Because the report was put together in very short order (10 weeks), the idea almost certainly came from elsewhere.

I recently dug up a 2000 working paper by Stanford economist Paul Romer that I think may contain the seeds of the current push to increase the ranks of graduate students. Here is a good piece on Romer's efforts to get Congress to implement some of his ideas. Romer is a smart guy, and while I'm not convinced by everything he says in the paper, it's a very interesting piece of work.

S&E research has been linked to overall economic growth. The question Romer addresses in the paper is that of, "What is the best way to increase the amount of science and engineering research done in the US?"

Several federal programs try to stimulate research activity by bolstering demand. Romer argues that there are 2 problems with a demand-side approach. First, demand subsidies don't necessarily increase the overall amount of research done. Unless the supply of researchers increases in response to demand, demand subsidies will just push wages up. Second, structural features of universities cause the supply of researchers to fail to respond to demand. This second point is the crux of the paper: because demand doesn't trigger increased supply, Romer argues that the government needs to subsidize supply instead of demand by creating large numbers of new graduate fellowships.

Romer discusses 2 reasons for the decoupling of S&E supply from demand:

First, S&E graduate programs provide no information on outcomes or salaries for their graduates, which prevents prospective students from being able to respond to demand signals in their enrollment choices:

"The lack of information that is available to students who are making decisions about careers in science and technology suggests that our existing educational institutions may not lead to the kind of equilibration that we take for granted in many other contexts. If students do not have information about what wages will be, it will be much harder for them to adjust their career decisions in response to wage changes."

Romer did an experiment in which he had an graduate assistant start the application process at the top 10 programs in business, law, and 6 different S&E fields. The assistant requested information on recent graduates' salaries from all programs, and got information from 80% of the business schools, 70% of the law schools and 0% of the 60 S&E programs. This problem seems straightforward to remedy - the Graduate School Guide provides some program level outcome information (but no salary information yet). The NSF is currently experimenting with a salary question on the Survey of Earned Doctorates, and should the test prove successful, it should be straightforward to assemble program-level post-graduation salary information.

Second, the response of competitive undergraduate institutions to increases in demand for S&E's is interesting. Elite liberal arts colleges gain prestige by being very selective. Suppose undergraduates respond to increases in demand for S&E's by enrolling in more classes that will prepare them for S&E careers. One response might be to hire more S&E faculty and to accept more S&E undergraduates. That's expensive and risky (if demand decreases, universities are stuck with excess faculty and facilities) and reduces selectivity. A simpler, alternative response:

"A liberal arts university that has a fixed investment in faculty who teach in areas outside of the sciences and that faces internal pressure to maintain the relative sizes of different departments may respond to this pressure by making it more difficult for students to complete a degree in science. Faculty in the departments that teach the basic science courses will be happy to 'keep professional standards high' and thereby keep teaching loads down. Faculty in other departments will be happy to make study in their departments more attractive, for example by inflating the average grade given in their courses. There is clear evidence that this kind of response currently operates on campuses in the United States."

So part of the reason that relatively few Americans pursue S&E careers may be the incentives under which universities operate.

Romer argues that immigration has provided a way around this undergraduate bottleneck and that S&E immigration levels have been much more responsive to changes in demand than domestic supply.

As an alternative to increasing reliance on foreign-born S&Es, Romer proposes creating large subsidies for both undergraduates and graduate students pursuing S&E degrees. These appear to be what we are seeing in current bills to increase the number of S&Es.

The details of his proposed subsidies are interesting, and it's worth looking into whether the current bills capture some of Romer's key points. More later in the week.

Student journalism

2007-04-18T16:04:00Z

I was on the faculty of the math department at Dartmouth for 4 years. Dartmouth has a wonderful department, full of smart, engaging people doing great work. One of the things I liked best about the department was that the local culture strongly values teaching in addition to research. That's true of the College as a whole, but I think it's particularly true in math.

One of the most interesting features of the graduate program is a summer-long teaching boot camp in which grad students learn theory of pedagogy, develop curricula, lecture to each other and get lots of feedback, have themselves videotaped teaching so they can study their delivery, and teach in the local high school. I have never heard of any other graduate program that does anything remotely similar. At NYU, we were thrown into the classroom on our own with no preparation whatsoever. Only if people were really terrible was there any "training," and that consisted primarily of having a less terrible graduate student observe your classes and give you some feedback.

In the eyes of US News's grad school rankings, however, all this effort and innovation gets boiled down to a single number between 1 and 5 that somehow encompasses everything that everyone is doing. My take is that their rankings tend to emphasize primarily research output. While Dartmouth has some really good people, they do spend time on things other than writing papers all day. It's not Princeton or Harvard, so in the eyes of US News, it's not a 5.0 (I have no idea what the actual US News number is). The real problem is that US News (and other ranking systems) boil everything down to a single, 1-dimensional scale. And that scale doesn't pick up different kinds of strengths.

One of the things that I'm hoping to achieve with the Graduate School Guide is to help people find programs that have strengths other than just the traditional cranking-out-tons-of-research. Only a handful of PhDs end up in positions where they do nothing but research, and it's important that the available training reflect that reality. I'd like to see programs like Dartmouth's be top-ranked given at least some people's priorities.

I tried to explain all this in a phone interview with a writer for The D, the student paper at Dartmouth. And what comes out? A story that talks about a former Dartmouth professor named "Geoff Parker" who says that Dartmouth does not have a "top-tier department". Yeah, thanks. Nice talking to you, too. Sigh.

So to any of my former Dartmouth colleagues who may be reading, I think you're great. Sorry about the article.

The Scientific Communications Act of 2007

2007-04-09T19:36:00Z

There's an interesting new bill working its way through the House: The Scientific Communications Act of 2007. (A tip of the hat to the fine folks over at ArsTechnica for cluing me in to its existence).

The bill allocates $50 million over 5 years to the NSF to improve the communication skills of S&E graduate students. The rationale offered in the bill is pretty straightforward:

Grad students often don't get training in communication skills, and these skills would help scientists talk to:

the public (i.e. increase public support for science)
policy makers (i.e. help scientific findings be incorporated into public policy), and
business leaders (i.e. help businesses use science to create new / better products)

All very sensible stuff, it seems to me. I am left wondering a few things:

First and foremost, how did this bill come to be? The lousy quality of graduate training in communication skills doesn't seem to be the kind of thing that most members of Congress spend their time worrying about. I wonder if this is the work of a AAAS fellow? Is there a good conduit for similar ideas to make their way up to the Hill?
Is this level of micromanagement unusual? Or does Congress go in and make NSF do little projects like this all the time? If the former, then I have rather mixed feelings about the bill. Despite the fact that I think communication skills should be taught, I'd be concerned that a different Congress might impose other, less desirable missions on the NSF, say, a set of grants to shore up the foundations for Creation Science.
Is this likely to go anywhere? Or is this just some kind of gesture?
Finally, why doesn't the NSF do this already? This seems like a no-brainer, and it's entirely in the self-interest both of the NSF as an institution and of scientists, since better public support of science presumably means more funding.

I have a couple of contacts on the Hill (not very good ones, unfortunately) will see if I can track down some answers. If you have any pointers to good sources, please contact me!

Career Outcomes and Rankings

2007-04-02T00:36:00Z

First of all, Happy April Fool’s Day – I think this day should be a nationally recognized holiday for PhDs, not only because so many of the best April Fool’s jokes have come from the ranks of scientists but also because of the lamentable condition many PhD grads find themselves in from an employment standpoint.

But on to the news…

“I wonder why top-ranked research universities DON"T track career outcomes for their graduates? Could it be that that information wouldn't make for good advertisement?”

Eric posted this rhetorical question last week after Geoff went public with his latest and greatest version of the Grad School Rankings guide.

I think this question hits at multiple levels of our discussion so I want to bring it out on stage.

About a decade ago I got a most unusual publication for the Geology Department of the University of Alaska, Fairbanks. They had tracked down EVERY PhD and Masters graduate back to something like 1955, or tried to, at least. Most of the people from those early years were “whereabouts unknown” – which, in Alaska, isn’t hard. But the department had earnestly tried to contact even those who were not “in the field” anymore, and their profiles were really amazing: an airline pilot, a private investigator, novelist, trapper… it was really eclectic. The publication illustrated something that anyone involved in PhD science education knows: most people who get a PhD eventually leave research science for some other field.

It turns out that quite a bit of data is available on where PhDs end up – through the university alumni relations office. While these data are not publically available, if you have permission, you can pull up the best known addresses and latest employers for every PhD grad in a program, going back as far as they have data. University Alumni offices pay close attention to these data – one of the reasons they send out free alumni magazines it to keep track of where you have gone. If you end up quitting that miserable post-doc at NRL and joining a small start-up like, say, Google, they know it and, years later, when you’re a fat juicy alum to hit up for donations, they find you!

But, getting directly to Eric’s point: departments don’t track grad student outcomes because:

A. they don’t need to (unless they have their own fund-raising efforts – and some do, like my undergraduate Geology Dept. at Princeton University that itself has an endowment that rivals those of many other universities).

B. They don’t really want to know what happens to the 80% or so of their graduates who eventually leave research science.

Academic departments, like individuals, have personae – visions of themselves that they try to project to the rest of the world. And the typical science department at a R1 university has a vision of itself as a producer of outstanding research and outstanding researchers. Nearly all their income (in the form of research grants) comes from the outside world’s acceptance of this persona. The “institutional reputation” factor that makes up so much of the US News and World Report’s program rankings plays off this issue directly. Would UCLA’s Chemistry Department seem a little less spectacular if it were widely known that more than 50% of its PhDs were working outside of research science after 10 years? Would MIT’s Mechanical Engineering department seem a little MORE impressive if 10% of its graduates end up as CEOs of companies (a higher percentage than Harvard Business School)?

Once you start asking this question you immediately start slipping into other intellectual quagmires. What is the “legitimate” employment outcome for a science PhD? What do we really CARE about with respect to outcome data anyway: how many graduate students end up becoming crusty old professors or how many graduates feel their department prepared them well for their future careers? What if every graduate of a program ends up in a high-powered academic job but 75% are miserable and lonely?

In the end, the only metric I can rationalize is whether graduates from the program feel the program satisfied their professional needs. PhD programs strenuously insist they are NOT a “professional degree” like those pitiable MBAs and MDs, but rather something higher, more noble and more pure. But in the end, their graduates end up in one profession or another, and the degree to which those alumni feel their PhD experience prepared them well or not should be, in my mind, of great interest to prospective students to that program.

phds.org in the News

2007-03-30T15:47:00Z

Scott Jaschik has written a nice piece on the new rankings at Inside Higher Ed. Here's the story.

New Graduate School Rankings

2007-03-29T21:56:00Z

Things have been a bit quiet here because I have been busy working on an expansion of phds.org: a new set of graduate school rankings. Take a look!

Here is what's new:

Program profiles - We now have detailed profiles of ~6,000 graduate programs at 418 US universities; altogether there are more than 25,000 pages of data!
Outcome data! - For years, people (myself included) have been calling for programs to make data on post-PhD outcomes available to prospective students. Relatively few programs collect this information, and even fewer make it available. However, it turns out that the National Science Foundation has been collecting outcome data for decades, but no one has ever looked at it at the program level. So for the first time ever, we have detailed, program-level information on placement rates at graduation, employment sector for recent graduates, time to degree, source of funding, graduate debt, and more.
More data - In addition to the outcome data, we also have comprehensive information on costs, student demographics, and more.
New UI - The user interface for the rankings has undergone a major overhaul. It's now even easier to generate rankings designed just for you.

So why do this?

There have been a number of papers that show that the US News rankings influence university behavior. See, for example, Monks and Ehrenberg and the introduction to this great paper by Avery et al. (they also cheat -- see this Wall Street Journal article ). If universities respond to rankings by changing their behavior, why not introduce elements into the rankings that encourage positive changes? Student outcomes should not be very difficult to improve - investments in better career services, processes that get students to think about their careers earlier in the PhD process, and curricula that better match what employers are looking for should all pay off. And the results benefit everyone: students get good jobs, departments get happy alumni, and the country gets scholars who are productively employed.

Another reason is to see if other providers of rankings can be influenced. If, say, US News decided to copy some of these ideas for their own rankings, then again, everyone comes out ahead (except for me, perhaps).

US News's rankings come out tomorrow - this should be interesting!