32 Comments
Guest post by Bruce Reed, Lystranne Maynard Smith, Cibu Thomas, originally released on the NIH Center for Scientific Review’s Review Matters blog
Have you applied for, sponsored, or reviewed NIH fellowship applications? We would like to hear your thoughts on what works, what doesn’t, and how the process could be improved.
National Research Service Award (NRSA) Fellowship (F) awards are intended to support training that will enhance pre- and post-doctoral trainees’ potential to develop into productive, independent research scientists. In 2021, CSR handled the review of more than 5500 of the approximately 6800 NRSA F applications received by NIH. We recently convened a CSR Advisory Council working group, charged with evaluating the fellowship review process and making recommendations to make it as effective and fair as possible for all.
The working group has noted multiple concerns, many of which center around the challenges of discerning the potential of the applicant and the value of the training planned, as opposed to the general reputation of the school and sponsor. There are concerns that applicants from smaller and less-resourced schools sometimes face a higher bar, that grades can unfairly haunt strong applicants, and that junior faculty are hesitant to sponsor applications, feeling their chances are slim.
Now the working group would like to hear your thoughts about the fellowship review process. What are its strengths and weaknesses? How could it be improved? In answering, think about the characteristics of strong applicants, sponsors, and training programs and the challenge of identifying the applications that have the greatest potential to develop independent, productive research scientists.
It may be useful to refresh your memory on the criteria NIH uses now. The five main review criteria are: Fellowship Applicant; Sponsor(s), Collaborators, and Consultants; Research Training Plan; Training Potential; and Institutional Environment & Commitment to Training. (PA-21-051 is a typical funding announcement and details review information in section V.) However, you need not be bound to these criteria; creativity is welcome.
Share your ideas via the comments box, or by emailing feedback@csr.nih.gov. We cannot respond individually to most comments, but we promise that if received by January 24, 2022, they’ll be read and that the information you provide will help the working group and CSR strengthen the review of NRSA fellowship applications.
Grades are not relevant. Some schools curve to an A-, some to a B-. Some programs give only A’s because of this. An application should not be criticized for poor grades in undergraduate, or even in graduate school and straight A’s should not be considered a particular strength.
Many of our F applications are coming back with the criticism that the applicant hasn’t published yet in graduate school. This is fine for fields where the life cycle of a publication is 1 year, but in systems neuroscience that life cycle of a publication is at least 3-4 years. It is not standard practice for students to publish in the first 3 years of graduate school. This is an unfair criticism and creates bias against entire fields.
Almost every F application from a junior faculty lab is coming back with criticisms of a lack of track record in training. We try to remedy this by putting a senior faculty member on the application as a “co-mentor”. It would be better to have an explicit request for a section on “Mentoring the Mentor”, rather than forcing the artificial co-mentoring arrangement. If there is a solid plan for mentoring the mentor, then we should trust our junior faculty.
There is no doubt that applications from “big-name” schools fare better. We benefit from that to some degree. But, I suspect that applicants from “big-name” undergraduate schools also fare better. We tend to draw our graduate students mainly from smaller liberal arts colleges, mostly not R1. Our students have not had nearly the same opportunity as undergraduates as those from R1 universities. Study sections should not use criteria that discriminate based on undergraduate research opportunity, for example publications (see above).
I want to echo the comment about the complaints about “not enough publications”. We recently had a student from our PhD program submit his F award with 3 publications already within the past 18 months, and that was deemed “not enough”. Then what is enough?
I came here to say exactly this about grades. It is not relevant and, in the case of undergraduate grades, punishes those from financially disadvantaged backgrounds that could not afford retaking a class simply to raise the grade the second time around.
+10000 about the undue emphasis on grades, especially undergrad. It’s simply ridiculous at this point.
I agree completely with Dr. Liberger’s comments regarding grades and would expand that sentiment to include anything in the applicant’s past. Grades can’t be changed. Neither can past mentors can’t be changed. Criticizing an applicant for these is useless to the applicant. Moreover, past performance may have been influenced by circumstances or conditions that are out of the control of the applicant. This is often the case with underrepresented minorities.
I also agree with other commentors who raise the issue of junior faculty mentors. Junior faculty mentors are frequently criticized for not having enough of a track record to justify funding application where they are the sponsor. Being a junior faculty member does not automatically make that person an unfit mentor. Likewise, being an established investigator doesn’t guarantee being a proficient mentor. The true value of a mentor should be evident by how involved the mentor is in…well mentoring…the applicant and this should be evident in the training plan, the research project, and how these dovetail with the applicant’s future goals.
Unfortunately, the structure of the review form tends to focus reviewer’s attention on details that do not address the value training plan to the advancement of the applicant’s future career. More relevant review forms might focus attention on:
• How the applicant and mentor interact?
• Will this interaction improve the applicant’s chances of success now and in the future?
• What resources will the mentor bring to bear on the applicant’s training?
• Will the research proposal advance the applicant’s scientific knowledge and provide them with the skills needed for their future scientific objectives.
• Is there evidence of interaction between the applicant and mentor in designing the research project?
I only participated once on a review panel for fellowship applications. I found the process very unsatisfactory and extremely time consuming. I am more used to reviewing R series applications and know where to find the information I need to make an assessment. But for the K applications it seemed disorganized, I did not know where to go look and the information seemed very redundant. The applicant, mentor and school all addressed how great the mentoring was going to be and many times if felt like the same information was presented over and over. I think the applications could be significantly shortened.
With respect to the expectations of the other reviewers, I too felt dismayed that they expected the early stage graduate students and postdocs to have published in the lab from which they were applying. This seems crazy. It can take several years just to set up a system or land on a project that is viable. This is not just the situation for systems neuroscience as mentioned by the comment above. It was an eye opener for me that anyone would even expect this!
There is a lot of inconsistency in the reviews that is never resolved. I had a MD/PhD student apply for an F30 last year. One reviewer stated that his proposal was “overambitious” and the other reviewer said “not ambitious enough”. Because it was triaged, this was not hashed out in discussion, which was a shame. As it turned out, this student worked so hard that he so far has 2 papers published from his dissertation and 1 (that will probably be the highest impact) in final prep prior to peer review. I’ve seen this happen to a lot of the students in our PhD program.
Another comment is not specific to F awards but to minority supplements to funded R01s. Last year we applied for one, and were told it would only take a couple of months to review. It took 9 months to hear anything back, only to get back very vague comments with a rejection. That is a lot of time lost. We were told we could submit an F award in parallel, but it would have to be a completely different proposal, and for a student to have two different proposals is a big ask. If you really want to support increasing the number of underrepresented minorities, the review of those supplement proposals should be handled better.
I agree with Dr. Lisberger’s points. Grades are not helpful as predictors of success, and cannot be changed so comments are not helpful for a resubmission. This just gives applicants a sense of futility. Prior publication is very dependent on the field and specific project, as well as the publication pattern of the PI (a few big papers or multiple small ones). It is often not something the applicant can change in the timeframe of a resubmission.
As for the mentors, I think you can tell a lot about the mentorship by seeing how closely the applicant’s and mentor’s plans align. If very close, then clearly they communicate well, but if the mentor’s plan is very generic then there’s a problem. The number of prior trainees is also very dependent on the institution, department, etc, and may not reflect the quality of the mentor at all. In contrast, you can tell a lot about the mentors from their sponsor statements as well as the quality and scope of the research plan. No trainee should submit an F grant without their mentor’s help. Therefore, the proposal itself is a better reflection of the mentor than their previous trainees, and I would suggest de-emphasizing this factor.
One of the major concerns that I’ve seen with peer review of Fs is the expectation that predoctoral or postdoctoral candidates will be proposing R-level research. While a fair degree of rigor should be expected, I often find the reviewer comments on the research portion of the application read as if reviewers expect an R01 type of methodology rather than focusing on how the proposed research (which should be rigorous, but not unattainable) compliments the training portion of the application. This defeats the very purpose (and what is possible given no funding for research) of F applications. It is not uncommon for reviewers, of any mechanism, to only review them as if everything was an R01. Reviewers of Fs are no different.
I agree-the panels I have served on have largely focused on the details of the proposal rather than taking a holestic view on how the training will prepare the applicant for a successful career.
Just as standardized tests have gone by the wayside in graduate school admissions, undergraduate grades are often not reflective of student performance or scientific aptitude. Graduate programs that have chosen to acknowledge this and place more emphasis on research experience and recommendations for admission are then placed in the position where it is difficult for their students to receive F fellowships due to this criteria. It is disheartening to tell student applicants that they have to “justify” their undergraduate performance in their biosketch to even have a shot at funding in a competitive environment when everything else (research experience, publications, mentorship) stacks up equally.
Several F applications from our institution have come back citing that the mentor (junior faculty) did not have a sufficient number of publications within the first 3-5 years of their academic appointment, even if they were able to secure NIH funding during that time. Graduate students often to help build a new investigator’s research program and are the ones producing data for the publications. Often, the addition of a co-mentor to the application is not a sufficient remedy in the eyes of the study section. This produces a bias toward funding trainees in more established labs. The pandemic has only amplified this issue as many non-COVID-focused labs have experienced delays in finishing/revising manuscripts, yet “productivity” is assessed the same.
I will echo what others have said about junior faculty and mentoring. The idea of having a place on the application to clearly demonstrate what the junior faculty is doing for mentorship of themselves and of their students could be extremely helpful to the application. My first F31 proposal was not discussed and a big red flag was my PI being junior faculty – even though we had an established faculty as co-sponsor. In the resubmission, we asked one of my recommenders heavily emphasize how my PI was a strong faculty mentor even being a junior PI in their letter. That application ended up being funded. It is not fair to hold a faculty’s experience over a student and prevent them from receiving a grant – especially if they have a co-sponsor and a solid research proposal.
Completely agree with Dr. Hinkle and others. In fact, the stage of faculty should be irrelevant as some early stage faculty have excellent mentoring track record. It seems that some reviewers see “Assistant Professor” and reflexively ding the application, which is sloppy and unfair to the trainee and the sponsor.
I have not reviewed Fellowship applications, but have helped my students/postdocs apply for them. The F31 Diversity process went very well, even during covid. The F32 process, on the other hand, was largely a disaster. The postdoc applied for an F32 April 2019. Reviews came back Aug. 2019. So far, so good. But it was not until April 2020 that NIH finally decided to fund it. And it wasn’t until May 2020 that the postdoc was able to get paid via the F32. By then, because university seed money supporting the postdoc in the meantime was dwindling, she had to commit to a different job. Sure, covid (sort of) got in the way. But if it takes >13 months for a postdoc to get fellowship funds, how are we expected to recruit and retain a new generation of scientists?
Review processes have generally been very good. Sometimes it is clear the reviewer is not in the field, however, and make mistaken or misleading comments based on this lack of background. Of course not everyone can understand everything from a field-diverse pool of applicants. It would be best if each application received a scientific evaluation from at least a couple (ad hoc?) experts in that field, as well as an evaluation of the candidate by reviewers comparing the personal qualities training environments etc. of applicants. Expectations of productivity (bean-counting) can vary considerably across fields.
I definitely echo all of the comments here so far. I’ll also mention two major concerns as someone who as applied and gotten an F32 and has mentored a graduate student on writing one.
1. I know that the NIH review process is lengthy across the board. But graduate students have to wait 18 months between FIRST submission and funding. Summary statements are released really close to the next deadline, forcing students to skip a deadline before resubmitting… for example, my first submission was in August, I received my summary statement at the end of November with no time to revise for the December 8 deadline. I resubmitted in April and the soonest that I could I start my funding was in March of the following year! (And I graduated that May!) Fortunately I was able to fill the gap between September and March with another fellowship, but that might not be a possibility for many students. This timeline forces students to either prepare a fellowship application before they’re ready (especially if they haven’t published extensively, which others have mentioned is a problem with the review process) or expect to have a large gap in their funding.
2. Reviewers really ding applications where faculty sponsors don’t currently have NIH funding, even if they’ve had extensive NIH funding in the past. My primary sponsor was the PI on a large NIH P2C center grant but because she didn’t have her own funding at the time, my score was hurt. I understand that NIH wants students to receive mentorship on grantsmanship and have other sources of funding available to them. Applicants and reviewers should receive clarification about why NIH wants sponsors to have NIH funding so that they can better write and evaluate them, respectively. For example, applicants whose sponsors don’t have funding can include co-sponsors who do or add additional training in grantsmanship to their training plan.
Point 1 here is really good. This timeline just doesn’t work for PhD student and their time to degree.
I also came here to comment on timing. I applied for an F32 in December (less than a year after starting my postdoc) and was successful, but was told that I would receive only 2 years of funding because (by the time the decision was made) I was in my second year. The typical NIH turnaround time doesn’t work well for the training stages if fellowships expect to see some prior productivity in that stage for applications to be competitive.
A suggestion to help a bit with this problem: could you allow a one-page update like NIH was doing for applications at one point during the pandemic, so trainees can update their applications to reflect what might be substantial progress in the months between initial submission and panel review? This might help push people to apply earlier.
I had reviewed over 150 fellowship applications over the years and I would admit that the system is never perfect. However, I don’t think some of the complains here are valid. Ignoring the applicant’s academic record? How are you going to differentiate a strong applicant from a weak one? By flipping a coin?
Any competent graduate program director should steer students toward faculty with mentoring experience and track record. The goal of NRSA is to identify future scientific leaders in the field, not to provide cover for funding gaps. Finally, prestigious institutions do not have an automatic advantage. A weak candidate from Harvard will have a harder time than an outstanding candidate from UMass or Tuft.
I have limited experience in revieing fellowships, but am a standing revierwer in R-type NIH/NCI study sections. have had two students writing an F31. The first was criticized because her average had dipped from undergraduate to graduate school, and I thought that was an unfair criticism since she made the jump from a primarily African American undergraduate institution to a top 20 graduate school. She ended up successfully obtaining her Ph.D. and publishing on an excellent journal. Her paper is one of the most cited in the lab. I do not think that the undergraduate record should be taken into consideration. However, I think that the graduate record should be important since we do not want to fund a student that is struggling or failing. On the other hand, I also had a student in my current smaller, Hispanic-serving institution with a flawless record as an undergrad and grad school, who had published a review in my lab previous to obtaining an F31. She published three reviews as primary author, has one research publication, and another on the way. Recently, she also obtained her Ph.D. I think the criteria for obtaining an F31 should be a combination between grades in graduate school, a record in presentations and at least a review paper and good preliminary data, as well as taking into consideration the track record of the advisor. They should be no biases against smaller, minority-serving institutions since some of the Ph.D. students in our smaller, less-known institutions have had very successful academic careers.
I would like to echo support for Stephen Lisberger’s comments regarding grades and publications as well as add a few of my own:
1) The expectation that students already have publications must be calibrated according to field. The timeline for publication is multiple years in many fields, such as structural biology. Requiring publications simply skews fellowships towards fields where papers are published earlier and more frequently.
2) The focus on perfect grades does not account for differences between colleges in grading curves and hurts the chances of many outstanding students who excel in the laboratory. Students whose grades were negatively impacted because they were working multiple jobs to put themselves through school are also at a disadvantage.
3) The proposal sections other than the research strategy are a nightmare to write and an even worse nightmare to review. Up to 15 pages are devoted to sections on: Applicant’s background and goals for fellowship training (limit 6), Selection of sponsor and institution (limit 1), Respective contributions (limit 1), Training in the responsible conduct of research (limit 1), and Sponsor and co-sponsor statements (limit 6). The guidelines for writing these sections are vague and there is excessive redundancy between the student and sponsor statements. These sections are also far too long, with the generous page limits inviting long descriptions that make it difficult to compare proposals. These sections should be drastically shortened and given a prescribed structure that elicits the necessary information and facilitates comparisons between proposals.
I agree with all comments. I specially agree with Dr. Pollitt’s second point (faculty sponsors with no NIH funding). With the current review rubric this is a major deal-breaker. What is the point here? Grantsmanship training is part of many (if not all) graduate programs, and PIs, funded or not, are constantly submitting grants proposals. Other sources of funding available to the applicant should be recognized as much as the specific requirement for PI’s NIH funding. Applicants add a co-sponsor with NIH funding to tackle this matter, but (as also commented here) this may largely serve the purpose of checking the “NIH funding” box. It is (was) my understanding that the goal is to support future scientists across the board. These are not R-series proposals, focus should be on the consistency and robustness of the training plan.
I have sponsored an F32 application for a post-doc in my lab which was in part not funded due to the criticism that I as a the Sponsor don’t have a record of promoting anyone in my lab to an independent academic career. By this logic, my only option is to launch my post-docs without the help of a fellowship, which ensures that my grant will not go “as far” and limit our overall progress as a lab. In fact, this logic (along with the ding on Assistant level faculty) ensures that Fellowships go to well-established labs, not labs with fewer people and less money, those who would most benefit from the additional financial support. During resubmission we emphasized that the trainee has a post-doc advisory committee which includes more seasoned advisors. We shall see if that helps.
As faculty who has had many students successfully receive F fellowships as well as others who haven’t, as a Reviewer, and as a training Program Director, I concur with many of the statements here. The timeline in particular is difficult and we see a constant holding back of student applications due to the emphasis on publications early on. I also agree with many others on the undue focus on perfect grades, and in the past the GRE, due to the many experiences I have personally seen with students who had to overcome multiple obstacles in undergraduate studies due to their background, language and cultural disadvantages as pointed out by others. Definitely the application would benefit from reducing some of the current repetition although I understand that some of this is to ensure that all members of the training team are aligned in their expectations and goals. I strongly feel that more emphasis and orientation must be given to reviewers beforehand as to expectations for these fellowships and an increased awareness that a large ‘research-intensive’ institution does not necessarily equate to an excellent overall individualized training plan that will ensure that the trainee succeeds and moves on to their next career phase. Unfortunately I have seen reviews where assumptions are made purely based on locale which is particularly unfair to the applicant, and have been on panels where reviewers stated upfront that they have no experience with reviewing this type of fellowship and basically critiqued it as an R01.
Many of the prior comments reflect my perspective on the application and review process – but I’d emphasize the timeline needs to be shortened and there needs to be specific reviewer training. With respect to timeline: I have had sponsored several successful and several unsuccessful applications – two of which were funded after re submission and that was a further delay, but actually very good training for the applicant. A faster process is really needed – maybe even expedited review cycles.
Regarding reviewer training: There is a benefit for better known institutions – an assumption that the training will be less at a lesser known place is also at issue I think. A prior comment from an experienced F reviewer illustrates this by stating that a weak Harvard applicant will fare less well than an outstanding one from UMass (which is actually yet another well-known school as is Tufts). That is not the problem – the issue is a strong candidate from a Harvard-type school vs. an excellent candidate from a less-resourced or lesser known school like mine and so many more. A second issue that can perhaps be addressed by reviewer training is what makes a good training plan. The application from a lesser-resourced institution is unlikely to include the most ‘cutting-edge’ techniques (which the reputation institution will have), but the training potential may in fact be better because it is not the shiny new technologies that drive rigorous science, particularly when it comes to trainees who need to first learn and then solidify fundamental principles of research design, data collection and analysis, and robust reporting/publication. Training fellowships need to be reviewed a bit differently even in the research design, not just the other sections.
I am a frequent reviewer of fellowship applications and have served as a mentor to several trainees with F31/F32 fellowships. I echo many of the concerns about the review process in the comments above. Specifically:
1. Grades in graduate/undergraduate school are not relevant to the evaluation of research and training potential and should not be part of the application or the review process. Trainees are often highly demoralized by Reviewer comments about poor grades. I have also witnessed Reviewers commenting negatively on a gap in training. This is also not relevant. Some students have had to take an academic break because of medical or financial reasons. They should not have to explain or justify poor grades or why they had to take a break in their education.
2. I have often found some of the best mentors to be junior faculty. They should not be “punished” for being too junior to mentor trainees. They should not have to go and search for more established mentors who are often not as invested in training the applicant as the primary faculty sponsor.
3. On a related note, junior faculty, and faculty who are women or from under-represented groups are often criticized for not having enough “experience” in an area (despite publications proving the contrary). For example, as a neurobiologist who started studying how nerves affect islet biology, two of my trainees’ applications were critiqued (and triaged) because I was considered to not have enough experience in the islet field. This, despite me having published 4 papers in islet biology as last and corresponding author, and having NIH funded grants on the topic.
4. Sponsor funding should be de-linked from trainee fellowship evaluations. Talented and creative trainees should not be “punished” for having the misfortune to be in a lab where the sponsor may have a temporary gap in funding.
5. For F31 fellowships, applicants in their early years in a program (yrs 1-3) should not be critiqued for not having publications. Most students take time to publish and in their early formative years, they are learning the experimental and communication skills necessary to work toward a publication. Publications in early years of grad school are also field-specific, and the focus on publications in fellowship reviews work against students who did not have easy access to undergrad research opportunities.
I echo the comments on the need to shorten the timeline. I have sponsored three students for F31 applications and two received scores ~30 and subsequently determined there was too little time remaining in their graduate work to bother re-submitting. The third was funded on the first submission. In all cases we waited to apply once the applicant had publications, pushing the process towards the end of the graduate training.
The majority of comments above suggest grades aren’t relevant to graduate research training and I tend to agree to a certain extent. One of my trainees had a C in his undergraduate transcript and received a comment about poor grades despite having earned an MS and 3 original research articles as first author and two others as middle author. Thus in the context of evident research success, grades should be less weighted. However, in the absence of publications (difficult to provide early in training) or other evidence of research success, it’s not clear to me what metrics a reviewer would have to compare applicants as opposed to mentors and institutions. I wonder if there are data that might illuminate the link between grades and research success?
I also agree that mechanism-specific training should be required of reviewers to avoid the default R01-style review.
I had a great experience with the application process, and on the other side, the program officers were kind, responsive, and helpful. I am fortunate to be at an institution where we take a class on grant-writing for public health to help us prepare our applications, which was invaluable.
Two things that might be helpful on the NIH side:
-Provide guidelines for reading/interpreting scores, which you receive at the time of submission. This is one thing I was scrambling to learn when I got my score!
-Provide guidance on how to identify a study group for your application (to identify in your cover letter), which can make a difference in scoring
Thanks for providing this outlet for feedback!
I agree with everything that has been said here related to grades, publications, and junior faculty. I want to add that penalizing F applications (31s and 32s) due to a sponsor being an assistant professor, not only is unfair for the applicant, but it will cause great students to avoid junior labs, which will in turn hurt junior faculty. I understand that the competition is fierce, and that you have to somehow select the ones that are well on track for success, but focusing so much on the track record of the sponsor will inevitably cause all Fs awards to be from the same labs/institution. The emphasis should be on the applicant’s research, training potential, and the opportunity for growth, regardless of the title of the sponsor.
I know I am reiterating points already made but felt I had to add my voice. I have mentored 7 F31-funded students since I was an Assistant Professor in 2000; I am currently mentoring two F31-funded students now. As others have stated, there is an over-emphasis on undergraduate (UG) grades and on publications. UG grades reflect past events and may not reflect the applicant’s current motivation. In my view, motivation is everything — I can teach facts and procedures but I cannot teach motivation and I’ll take a super motivated “B” student (with a few “C”s!!) over an unmotivated “A” student any day. Motivation can be assessed by the application’s text/letters; UG grades are irrelevant in assessing these applicants. With regard to publications, since the first application I sponsored over 2 decades ago, there has been a regrettable increased emphasis on the applicant’s publications. The point of an F31 is to train the applicant and part of that training involves how to write and how to publish. In that context, why are we expecting trainees to already have learned the skills in which they will be trained? Again, I’d happily work with a motivated student with no publications rather than an unmotivated one with 5 publications; publications will come with the training but should not be a pre-requisite for it. Finally, there is a growing insistence on the part of reviewers that sponsors outline plans to help the applicant prepare and submit an application for post-doctoral training. I believe this insistence is, to be blunt, ludicrous. The role of the doctoral mentor is to prepare the applicant with the skills they need to move forward in their career successfully. That means, among other things, training them to do rigorous science, training them to publish their rigorous science, and mentoring them as they move forward (e.g., positioning them to apply for post-doctoral positions or faculty positions, etc). The role of the graduate mentor is not to help doctoral candidates write applications for post-doctoral training; rather, the post-doctoral mentor can work with their new trainee to write these applications.
I agree with what has been said and would like to emphasize that many portions of the F31 fellowship are very redundant. For instance, both the applicant, mentor, and school have to provide documents on programs and workshops that the school offers. This could all be condensed down into 1 document that the school provides. Another example of redundancies are the numerous sections which cover the facilities and equipment available to the researcher. The applicant is forced to rehash what they, the school, and the mentor have already written on the topic. Further, the large span of time between when the fellowship application is submitted and when feedback is received is too long and prevents timely resubmissions. For instance, if an app was submitted in December, then the feedback won’t become available until right before the next fellowship deadline (April), which means that the fellowship can not be resubmitted until the August deadline, and if it is accepted then the fellowship doesn’t start until months later.
First, I want to extol this committee being drawn together to address this problem. I have become reluctant to have my students write a fellowship to the F31 or F32 programs because they are a phenomenal amount of work. When the reviews come back with an off hand remark about grades, publications, or other aspects outside the student’s control, the student can be demoralized. I have watched extremely promising young scientist become unproductive and develop a negative opinion about their future in science after receiving NRSA fellowship reviews. This is a very important problem that should be fixed.
Here are my specific thoughts (based on recent experience as a mentor and a reviewer):
1) I agree with all the comments regarding asking for undergraduate grades. These are irrelevant and can discriminate against all kinds on nontraditional students.
2) Criticizing 2nd and 3rd year students for lack of a publication is unfair, particularly criticizing students for lack of a paper from undergraduate research. Students in early years of training rarely have done a body of research that leads to a robust paper, and publishing preliminary data in a low tier journal can hurt the student later from having the depth of work necessary for a more meaningful paper. Some labs put every person who even tangentially worked on the project as authors while others have a very high bar for inclusion as authors. Some candidates may have inherited a very advanced project, while others must first prepare all their reagents or generate transgenic mice or develop a novel technique. So the relevance of having or not having a first or middle authorship paper is rather meaningless to discriminate candidates. Only preliminary data in the proposal should be used to judge the merit and quality of a candidate’s first years of training and that should be judged against the progress from the starting point of the student taking over the work.
3) I most want to echo the comment about the just the absolute weight of these proposals. There are about 60 pages of material that needs to be reviewed for each candidate. This is a burden on reviewers and puts a heavy burden on our youngest and most vulnerable scientists to pull together. As a reviewer, I found the amount of material provided too much openings for useless criticism. The process is now so onerous, many larger programs have formal programs to guide students through this maze with input from faculty who have sat on review panels. This further discriminates against those at lesser resourced institutions.
I would like to see much of the requested info replaced by an online form and all the redundancy removed. Here are few of the most egregious. Why does the student plus the mentor plus the program directors need to describe the “environment” when there is a standard Facilities and equipment page? The entire section on Ethics training can be replace by a check box that asks if the student has taken a minimum of 20-hours training in Ethics. The student, the mentor, the program director etc all provide the same info about journal clubs and seminars…blah blah blah. Students have been criticized for “generic” descriptions from the Graduate Program directors or university staff, but is it reasonable that directors would write anything other than boilerplate text? Why is there a Biosketch complete with the standard 5 sections of past research and the identical information in a section to summarize past research? And this goes on and on.
As a specific recommendation, I sat on a DP2 panel that reviewed early stage faculty on much of the same criteria and the entire research proposal, career goals, and training plans fit in 8 pages and was sufficient to discriminate these candidates. That coupled with the (1) the standard Facilities and Equipment pages, (2) a bio sketch (no grades, just education, honors, and past research summaries), (3) a bio sketch from the mentor with a personal statement about training philosophy and past mentorship (or who is mentoring the mentor) and a 2 page letter of support from the mentor that includes 1-2 sentences on how the student will have resources for supplies and materials to complete the work (funded grants? Start up funds? Institutional funds? any source is OK). Additional material should be 2 letters of recommendations and a short 1-2 page description of the specific training program for this student and this should be a single document jointly prepared by the mentor and student with some boilerplate text about the general program from the director. This would be sufficient material to judge that student compared to others.
I hope this committee will take bold moves and dramatically revise this program, focusing on keeping the applications very short, limited to the material that will most discriminate qualified candidates.
Fellowship applicant:
1. Undergraduate academic record does not describe an applicant’s potential for excellent research at all. It is time to recognize that grades are not standardized across schools so a B at one school could be an A at another. Additionally, applicants who are from backgrounds that are underrepresented in the sciences (first-generation college students, for example) are more likely to have a harder time adjusting to college. I have seen instances of applicants mentioning this fact in their Biosketch and still being judged about their less than stellar grades, even though it is well known that adjusting to college takes time and many students’ grades suffer because of this.
2. Productivity (papers published) varies widely depending on the field and specific culture of the applicant’s previous and current labs. For neuroscience, it is common to publish one big first-author paper at the end of the Ph.D., yet I repeatedly see reviewers commenting that an applicant hasn’t published anything from their lab in the first 3 years. Furthermore, some labs add lots of authors for their contributions, and some labs keep their author lists short, excluding anyone who did not contribute heavily to intellectual advancements of the work (even if they did substantial experimental work on the project). Unless there is a way to tell which kind of authorship culture existed in the applicant’s previous and current labs, the authorship measure does not reveal much about the applicant’s potential for excellent research/contributions.
Sponsors, Collaborators, Consultants:
1. Requirement of the mentor to have a track record of mentoring individuals at a similar stage: this means that more fellowships will be given to applicants from established labs vs. newer labs. There should instead be a requirement for the sponsor to get mentorship training (either through a program or with a co-mentor/sponsor). It should be noted that there are benefits to being in a newer lab: (1) the lab is more likely to be small, meaning more time for the applicant to engage with and be trained by the sponsor, and (2) learning from an assistant professor is very relevant training for an applicant who wants to have an independent research career – the applicant will soon be an assistant professor themselves.
2. Requirement of the mentor to have substantial funding (an R01): this will again bias fellowships toward established labs (and away from newer labs). New faculty members often negotiate very substantial start up packages in order to kickstart their research before receiving an R01 and can be well equipped to support a graduate student’s research.
Research Training Plan:
1. The strongest fellowship applications are those that represent an original direction in research developed by the trainee. Since faculty are under some pressure to publish a sufficient body of work that can be evaluated for tenure, the fellowship is most valuable for risky, potentially important work that might otherwise not be pursued.