Graduate School Guidance

One key data set I used in building the Graduate School Guide was IPEDS, which is put out by the Department of Education's National Center for Education Statistics. IPEDS is incredibly useful for this kind of thing: the data set contains a near-complete list of all colleges and universities in the country together with detailed information about student enrollments and degrees granted. The data are well-documented, easy to access (you can download everything in CSV files or grab custom subsets of the data from an online tool), and relatively recent (2006).

NCES delivers tremendous value with IPEDS in several ways:

1) They provide a lot of summary statistics from the data to give an overview the current state of post-secondary education.

2) They make it easy for others to get ahold of the data, and they provide thorough documentation so that once you have the data, it's easy to work with. As a result, third parties such as The Chronicle of Higher Education delve more deeply into the data and deliver additional interesting insights.

3) They help participating institutions use the data set to compare themselves to peer institutions. In so doing, they implicitly give guidance on good questions for institutions to ask about themselves (e.g. how much financial aid do we give relative to our peers? how well do we retain students?)

4) They provide data to third parties who help students choose colleges. It's virtually certain that US News, Peterson's, Fiske, etc, all draw upon IPEDS data.

5) They run College Opportunities OnLine, a site that provides useful information directly to prospective college students.

6) They combine multiple data sets in ways that increase the value of all the components. For example, on the COOL web site they combine their core data on universities and graduations with data on libraries and on campus crime.

The NSF has similarly interesting sets of data in their Survey of Earned Doctorates, Survey of Doctorate Recipients, and the Survey of Graduate Students and Postdoctorates in Science and Engineering, but they have not been anywhere near as effective at extracting value from them.

Here's how NSF stacks up to NCES:

1) The NSF provides data reports in the form of very basic, annual (or bi-annual) summary statistics and in Science and Engineering Indicators.

2) NSF has a decent tool for extracting additional summary statistics in WebCaspar. Unfortunately, the data are aggregated at such a high level (nationally for most things), that you can't get at the most interesting information. Basically you can't get anything beyond a count of individuals at the level of a single institution.

It is possible to get ahold of more detailed data than what's in WebCaspar, but the process is difficult, expensive, and time-consuming. The data we used on our site cost $7,000 and took months for NORC to generate. There is a do-it-yourself alternative if you have a facility that satisfies NSF's security criteria (stringent), can handle NSF's audits, and have a license for SAS (expensive).

NSF has a responsibility to protect the privacy of survey participants, and to their credit, they take that responsibility very seriously. However, I think they tend to err so far on the side of caution that they detract from their own mission.

3) As far as I know, NSF doesn't make any effort to help universities use the data they generate in useful ways. I think this is an area in which NSF could take a real leadership role. For example, people have been expressing dismay for years about the length of time it takes to earn a PhD. The NSF has detailed data on exactly how long it has taken to earn pretty much every single PhD in the country. Providing institutions with stats on their own times to degree relative to their peers could be a powerful catalyst for inspiring improvements. They could provide similar motivation on placement rates, funding levels, and so on.

The value the NSF could provide here is not just in providing the data - it's in getting institutions to ask the right questions of themselves. I have talked to several deans who participated in the recent CGS study of attrition. One common thread I have heard is that they were able to make progress on reducing their attrition rates - the big problem they had was that nobody had ever looked at the issue before.

4) People love to hate US News's rankings. The chief complaint is the exclusive reliance on reputation. There has been sporadic talk over the years about institutions proposing alternatives, but the talk has never amounted to anything. As we have demonstrated here at phds.org, NSF's data can form part of a more balanced approach. Surely a set of US News rankings that incorporated outcome measures as well as reputational measures would be more helpful to students and less objectionable to faculty members than their current approach.

5) As far as I know, NSF does not use their data to provide any services targeted at prospective graduate students. I think that's a reasonable call on their part, since I don't think it's a great fit for the skill set that the organization possesses. However, if they were to make department level data sets available (aggregated over sufficient numbers of years to protect privacy) on a regular basis to third parties such as US News, Peterson's, phds.org, etc, they could ensure that would-be students would benefit from the NSF's hard work.

6) As graduate-school.phds.org demonstrates, the NSF data sets are much more interesting when combined. One frustration I had in working with the NSF's data sets is that they are not designed to work well with each other or with outside data such as IPEDS, so combining them took months of work.

Each of NSF's data sets uses a different, incompatible taxonomy of disciplines, all of which are different from the one used by IPEDS. The NSF uses an outdated list of institutions (FICE codes) that results in information for entire state university systems being lumped together and makes it difficult to combine with information from IPEDS, which uses an up-to-date list. Rethinking the NSF's field codes and updating their institution lists would make their data more valuable to end users and would simplify the lives of the people at the institutions who have to answer both NSF and NCES surveys.

So here's what I think NSF could do to remedy things:

• Take a leadership role in defining metrics of interest, and provide institutions with summary statistics about their own performance relative to their peers.

• Rework their various classification schemes so that their data is easy to combine and compatible with outside data.

• Provide better quality control over department-supplied self-classifications and consider ways to handle multidisciplinary departments (e.g. a department of Mathematics and Statistics)

• Build a list of departments and strive to provide department-level data when practical.

• Release regular department-level data sets (aggregated over multiple years and suppressed as needed to protect participant privacy) for use by third parties.

I think the result would be universities receiving information they can use to make valuable improvements to their programs and prospective students getting better guidance on choosing suitable programs. That's a pretty big impact.

Meanwhile in the Senate...

I've just taken a look at the Senate's hefty bill to reauthorize the NSF, S 761. Like the House equivalent, there are some good and interesting things in the bill.

• There is a lot of new money for graduate fellowships:

IGERT - Increased funding to the IGERT program, like in the House bill. This would fund provide funding for more graduate students, but IGERT programs appear to be much better suited to providing people with more of a range of career options than traditional PhD programs. The stipends, which are semi-portable traineeships rather research assistantships, are pretty hefty, too. $30K/year is more than some postdocs are paid - perhaps it will stimulate some upward growth in S&E graduate stipends overall.

The Senate's numbers are a lot lower than the House numbers - $22M/year in 2008 to $55M/year in 2011 (as opposed to ~$75M-$150M/year for the House). Since there was pushback from the administration on the way the House was proposing to fund IGERT (as a fixed percentage of the NSF's budget), I imagine something more like the Senate's version will go through.

Graduate fellowships - The Senate bill also allocates a good chunk of money for new old-style fellowships, ranging from $24M/year in 2008 to $60M/year in 2011. The House bill doesn't provide any new money for fellowships apart from the overall budget doubling.

DOE graduate fellowships - $9M/year-$35M/year in fellowships for "students pursuing a doctoral degree in a mission area of the Department [of Energy" - energy and nukes, presumably.

So altogether the Senate is proposing increased funding for graduate fellowships in the amount of $140M by 2011, which translates to about 2,800 new doctoral students per year.

A little digging in WebCASPAR shows that NSF funds about 20,000 full-time graduate students (about 5% of all full-time graduate students). If we make the assumption that the NSF funds primarily doctoral students and that NSF funding is spread over 5 years, we find that NSF funds on the order of 4,000 doctoral students/year.

So the combination of a budget doubling plus the new fellowships, if allocations are held constant, would increase NSF-funded doctoral students by about 4,000+2,800 = 6,800. This estimate is probably on the high side, since IGERT stipends are pretty high, some NSF funding probably goes to master's students, and the new money for fellowships may result in reallocations of other funds away from graduate fellowships; let's say there will be 5,000 new NSF-funded doctoral students per year. In 2005 there were 29,000 S&E PhDs granted, 13,000 of which were in the physical sciences and engineering. NSF's funding is concentrated in the physical sciences and engineering - NSF's existing 4,000 or so fellowships combined with 5,000 new ones would mean that NSF could end up funding most doctorate recipients in those fields.

Will this increase the number of doctorates granted per year? Maybe - Richard Freeman has some interesting work that suggests so - but perhaps not by 5,000. Once a department has its teaching needs covered, the incentive to enroll graduate students is reduced. New money from NSF frees up existing departmental funds that pay TAs for other purposes. Labs will be staffing up once new NSF grants start flowing, and I'd bet that they'll want to hire postdocs, not graduate students. I predict an increase in the total number of postdocs in many math and physical sciences fields. In fields, like CS and engineering, in which most people go to industry after graduating, postdocs will probably be harder to come by, so they'll end up with more grad students.

• There is a modest amount of money to fund Professional Science Master's degree programs.

Professional Science Master's programs were created by grants from the Sloan Foundation back in 1997. I have met several people who have been through PSM programs, and they sound great - they're essentially hybrid-MBA/science degrees. If the country needs more scientists to spur economic growth, this is a great way to get them. The programs graduate people who know enough science to do useful things in theory, but who also know enough about navigating companies to actually accomplish things in practice.

I'm very encouraged to see the possibility of the NSF picking up PSM funding. Once institutions get into the business of bringing real, professional skills into science curricula, it's only a matter of time before the ideas diffuse into more mainstream science programs. There's nothing in the House bill about PSM programs, so I think this is iffy, but since the amount of money is modest ($9M/year in 2008 to $20M in 2011), I'm hoping it will pass.

There is a ton of additional material in both the Senate and the House bills about teacher training. More on that in a future post.

More Graduate Students - Brought to You by IGERT

The increase in graduate students discussed earlier in the week just came a step closer to reality: the House and Senate just passed a set of bills that will steer a big chunk of funding toward new graduate fellowships, among other things. I assume there will be some negotiation in conference over the final form, but the boost to graduate numbers is a lot closer to reality.

One caveat: the bills just authorize increased spending, but they don't actually provide it. So there is room for things to be cut by failing to be funded in appropriations bills.

I have started looking through a few of the bills, and as best I can tell, there is some interesting and good stuff in them. There are also some things that are disappointingly omitted. As with HR 1453, the bills smack of AAAS Fellow influence - hats off to any of you who are reading this.

The House passed 3 bills:

• HR 1867, the National Science Foundation Authorization Act of 2007, which doubles the NSF budget,

• HR 363, the Sowing the Seeds Through Science and Engineering Research Act, which funds a bunch of undergraduate and graduate fellowships, and

• HR 362, the 10,000 Teachers, 10 Million Minds Science and Math Scholarship Act, which funds S&E teacher training.

The Senate passed a single, 200+ page bill, S 761, the America Competes Act (more formally, the America Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science Act).

I've just started looking into these, and I imagine they'll take a few posts to digest.

HR 363 is the House bill most relevant to grad students. It passed 397 to 20, so at least some portion of it seems pretty likely to happen. Doubling the NSF budget will likely increase spending on grad students as well, but probably not in ways qualitatively different from current expenditures.

I notice in GovTrack that authorization for funds has been stripped out of the bill. So I'd guess that it will be funded at a lower level than the bill calls for.

Section 4 of the bill is the interesting bit:

SEC. 4. INTEGRATIVE GRADUATE EDUCATION AND RESEARCH TRAINEESHIP PROGRAM.

(a) Funding- For each of the fiscal years 2008 through 2012, the Director of the National Science Foundation shall allocate at least 1.5 percent of funds appropriated for Research and Related Activities to the Integrative Graduate Education and Research Traineeship program.

(b) Coordination- The Director shall coordinate with Federal departments and agencies, as appropriate, to expand the interdisciplinary nature of the Integrative Graduate Education and Research Traineeship program.

(c) Authority to Accept Funds From Other Agencies- The Director is authorized to accept funds from other Federal departments and agencies to carry out the Integrative Graduate Education and Research Traineeship program.

I'm disappointed that there are no provisions for measuring efficacy or for linking expenditures to the state of the labor market, so it ends up being a command-and-control type of program (like most of the rest of science, alas). That being said, funding IGERT rather than more traditional NSF fellowships seems like a promising way to go.

The good thing about IGERT (from the IGERT FAQ):

"A major objective of NSF's IGERT program is to train students in areas where industry, government and academic institutions are experiencing a shortfall. IGERT graduates may work in industries ranging from pharmaceutics to petrochemicals, government laboratories devoted to health, commerce or energy, small teaching colleges and major research universities. An important benefit of the IGERT programs is that most students have opportunities to sample these locations during their training. This makes it easier to decide which career environment is right for you."

So IGERT is not about creating a bunch of new professors.

Another good thing about IGERT he IGERT money goes to traineeships rather than research assistantships. The funding is not as portable as what Romer was calling for, but funding tied to the department is a lot more portable than funding tied to a researcher. I think it's a reasonable compromise between full portability and the increasingly common no portability.

(An aside: There have been a number of National Academy reports that have called for more portable funding, but they always seem to get shot down. I've spoken to some National Science Board members about the issue, and the story I have gotten from them is that they are terrified that given full portability, students will all end up at some program other than their own. This discussion gives a sense of the "debate." It's total BS - people say they don't want more portability because it might screw up the wonderful system we have now. But there is no analogous concern raised when funding shifts away from portability as it has steadily for the last couple of decades. Moreover, they complain about the lack of data documenting benefits of portability, but they never actually try running any experiments to gather any. It's not rocket science. In fact, there is actually a lot of data available in the SED and the SDR that would let one compare career outcomes for people funded by portable vs. non-portable funding, but nobody has bothered to run the stats. As best I can tell, nobody wants to know.)

The less good thing is that IGERT seems to emphasize interdisciplinarity for its own sake. Googling "IGERT" turns up some goofy sounding programs - "Interactive Digital Multimedia" at UC Santa Barbara (you can get a PhD in that?), "Biological Invasions"(!) at UC Davis, and so on. But plenty of sensible things, too.

IGERT seems to be about creating new programs, too, but maybe I'm just not understanding correctly. I'd be happier if there were provisions for existing programs getting their acts together in terms of providing professional development for their students, but maybe IGERT would lead to some diffusion.

It's interesting that the bill steers a fixed fraction of the NSF's budget toward IGERT rather than a specific amount. I'm guessing that's so that IGERT would benefit proportionally if the NSF budget is doubled and so that there isn't some convenient dollar amount to target once appropriations committees take their knives to the bill? 1.5% of a $5 billion dollar budget is $75 million. IGERT currently gets about $12 million / year, so that's a hefty boost. And $150 million if the NSF's budget doubles.

IGERT students get $30K stipends (more than NSF gave their postdocs back in my day, stingy things), plus tuition (call it $20K) plus maybe some overhead for health insurance, etc (say $10K). Probably there is some faculty and institutional money in IGERT (after all, creating new programs isn't cheap). So let's say half the money goes to students at $60K apiece. $75 million buys you a grand total of ... 625 students. And 100 of those were already funded. Even doubled, we're talking a not very large number of people. Romer's proposal is for an extra 17,000 new PhDs per year, so this is a tiny fraction of what he's talking about.

The administration has indicated that they don't like the idea of having a fixed fraction of the NSF's budget allocated to a particular program (because the other 98.5% just isn't enough). I assume this is because of pushback from senior people who don't like the idea of either portable funding or their pot of grant money being diminished - similar complaints were heard during the NIH doubling.

More Graduate Students?

There have been a few bills working their way through Congress that seek to significantly increase the number of graduate students. Why now, at a time when people are asking, "Are we training too many PhDs?"

Much of the current impetus comes from the National Academies report, Rising Above the Gathering Storm -- the bills in question also address other Gathering Storm recommendations -- so the question becomes, where did the Gathering Storm report get the idea? Because the report was put together in very short order (10 weeks), the idea almost certainly came from elsewhere.

I recently dug up a 2000 working paper by Stanford economist Paul Romer that I think may contain the seeds of the current push to increase the ranks of graduate students. Here is a good piece on Romer's efforts to get Congress to implement some of his ideas. Romer is a smart guy, and while I'm not convinced by everything he says in the paper, it's a very interesting piece of work.

S&E research has been linked to overall economic growth. The question Romer addresses in the paper is that of, "What is the best way to increase the amount of science and engineering research done in the US?"

Several federal programs try to stimulate research activity by bolstering demand. Romer argues that there are 2 problems with a demand-side approach. First, demand subsidies don't necessarily increase the overall amount of research done. Unless the supply of researchers increases in response to demand, demand subsidies will just push wages up. Second, structural features of universities cause the supply of researchers to fail to respond to demand. This second point is the crux of the paper: because demand doesn't trigger increased supply, Romer argues that the government needs to subsidize supply instead of demand by creating large numbers of new graduate fellowships.

Romer discusses 2 reasons for the decoupling of S&E supply from demand:

First, S&E graduate programs provide no information on outcomes or salaries for their graduates, which prevents prospective students from being able to respond to demand signals in their enrollment choices:

"The lack of information that is available to students who are making decisions about careers in science and technology suggests that our existing educational institutions may not lead to the kind of equilibration that we take for granted in many other contexts. If students do not have information about what wages will be, it will be much harder for them to adjust their career decisions in response to wage changes."

Romer did an experiment in which he had an graduate assistant start the application process at the top 10 programs in business, law, and 6 different S&E fields. The assistant requested information on recent graduates' salaries from all programs, and got information from 80% of the business schools, 70% of the law schools and 0% of the 60 S&E programs. This problem seems straightforward to remedy - the Graduate School Guide provides some program level outcome information (but no salary information yet). The NSF is currently experimenting with a salary question on the Survey of Earned Doctorates, and should the test prove successful, it should be straightforward to assemble program-level post-graduation salary information.

Second, the response of competitive undergraduate institutions to increases in demand for S&E's is interesting. Elite liberal arts colleges gain prestige by being very selective. Suppose undergraduates respond to increases in demand for S&E's by enrolling in more classes that will prepare them for S&E careers. One response might be to hire more S&E faculty and to accept more S&E undergraduates. That's expensive and risky (if demand decreases, universities are stuck with excess faculty and facilities) and reduces selectivity. A simpler, alternative response:

"A liberal arts university that has a fixed investment in faculty who teach in areas outside of the sciences and that faces internal pressure to maintain the relative sizes of different departments may respond to this pressure by making it more difficult for students to complete a degree in science. Faculty in the departments that teach the basic science courses will be happy to 'keep professional standards high' and thereby keep teaching loads down. Faculty in other departments will be happy to make study in their departments more attractive, for example by inflating the average grade given in their courses. There is clear evidence that this kind of response currently operates on campuses in the United States."

So part of the reason that relatively few Americans pursue S&E careers may be the incentives under which universities operate.

Romer argues that immigration has provided a way around this undergraduate bottleneck and that S&E immigration levels have been much more responsive to changes in demand than domestic supply.

As an alternative to increasing reliance on foreign-born S&Es, Romer proposes creating large subsidies for both undergraduates and graduate students pursuing S&E degrees. These appear to be what we are seeing in current bills to increase the number of S&Es.

The details of his proposed subsidies are interesting, and it's worth looking into whether the current bills capture some of Romer's key points. More later in the week.

Student journalism

I was on the faculty of the math department at Dartmouth for 4 years. Dartmouth has a wonderful department, full of smart, engaging people doing great work. One of the things I liked best about the department was that the local culture strongly values teaching in addition to research. That's true of the College as a whole, but I think it's particularly true in math.

One of the most interesting features of the graduate program is a summer-long teaching boot camp in which grad students learn theory of pedagogy, develop curricula, lecture to each other and get lots of feedback, have themselves videotaped teaching so they can study their delivery, and teach in the local high school. I have never heard of any other graduate program that does anything remotely similar. At NYU, we were thrown into the classroom on our own with no preparation whatsoever. Only if people were really terrible was there any "training," and that consisted primarily of having a less terrible graduate student observe your classes and give you some feedback.

In the eyes of US News's grad school rankings, however, all this effort and innovation gets boiled down to a single number between 1 and 5 that somehow encompasses everything that everyone is doing. My take is that their rankings tend to emphasize primarily research output. While Dartmouth has some really good people, they do spend time on things other than writing papers all day. It's not Princeton or Harvard, so in the eyes of US News, it's not a 5.0 (I have no idea what the actual US News number is). The real problem is that US News (and other ranking systems) boil everything down to a single, 1-dimensional scale. And that scale doesn't pick up different kinds of strengths.

One of the things that I'm hoping to achieve with the Graduate School Guide is to help people find programs that have strengths other than just the traditional cranking-out-tons-of-research. Only a handful of PhDs end up in positions where they do nothing but research, and it's important that the available training reflect that reality. I'd like to see programs like Dartmouth's be top-ranked given at least some people's priorities.

I tried to explain all this in a phone interview with a writer for The D, the student paper at Dartmouth. And what comes out? A story that talks about a former Dartmouth professor named "Geoff Parker" who says that Dartmouth does not have a "top-tier department". Yeah, thanks. Nice talking to you, too. Sigh.

So to any of my former Dartmouth colleagues who may be reading, I think you're great. Sorry about the article.