Feedback to Applicants: 2018 Exploration

The multidisciplinary review panel members provided comments about the applications they evaluated that were submitted to the New Frontiers in Research Fund (NFRF)—Exploration inaugural competition.

Review process

A total of 1,315 eligible applications were submitted to the Exploration inaugural competition, launched in December 2018. Each application was assigned to three reviewers, who each reviewed approximately 30 applications. The reviewers provided ratings for each application for the High Risk, High Reward, and Feasibility criteria, according to the evaluation grids. They also identified what they considered to be the top 25 per cent of applications within those they reviewed (typically six to eight). The applications that received a top 25 per cent designation from at least two reviewers, or were identified as the best or second-best applications from any reviewer, were selected for further discussion and assessment at the multidisciplinary review panel meeting.

The multidisciplinary review panel members met in person in Ottawa on March 26 and 27, 2019, to review the 321 top-rated applications. This meeting allowed the reviewers to discuss the strengths and weaknesses of each application according to the evaluation criteria. Reviewers identified a list of applications to recommend for funding at the end of the meeting. The list was presented for approval to the Canada Research Coordinating Committee.

Application ratings

Based upon the ratings provided by the multidisciplinary review panel members, each application was assigned an overall score for each of the three criteria. An application’s overall rating for a criterion is equal to the mean of the reviewer scores for that specific criterion prior to the multidisciplinary review panel meeting; therefore, these ratings do not represent the consensus ratings of the application. Overall ratings provide a sense of the way reviewers perceived an application, but were not used in isolation to identify applications to be funded.

Applications that were discussed at the meeting were also reviewed against the Equity, Diversity and Inclusion (EDI), and Interdisciplinarity criteria by two tri-agency internal review committees. These applications have a rating of Pass or Fail for these criteria; the final scores for EDI and Interdisciplinarity were established after discussion with the panel.

Application strengths and weaknesses

The following summarizes some of the elements that reviewers commonly noted in their discussions, according to the criteria.

As with all funding opportunities, well-written applications that clearly and concisely present the proposed project, while addressing the program criteria, are more likely to be understood and well-received by reviewers. For NFRF, where applications are evaluated by multidisciplinary review panels, reviewers noted it is even more important to provide context (where necessary) and avoid the use of jargon.

High Risk

Applications that stood out during review were those that proposed new and interesting approaches involving novel combinations of disciplines, innovative and creative ways of integrating multiple disciplines, and approaches combining multiple techniques common to different disciplines. Applications that proposed to integrate disciplines that seldom interact were highly regarded. Successful applications clearly explained the elements that make the proposed work high risk and distinguished the work from incremental extensions of previous or ongoing research projects.

High Reward

Applications that reviewers assessed as having potential for high reward included those that, if successful, would have a significant societal, economic, technological or health impact; affect a large population; substantially change current practices (defy paradigms); impact on numerous fields or applications; or resolve major challenges.

Reviewers were impressed by applications that proposed to address critical questions, completely new areas, or that would open a new field of research in Canada. Opportunities for rich and unique training opportunities for highly qualified personnel were also seen as contributing to the high reward potential. Successful applications clearly established the importance of resolving the problem, distinctly identified the project’s intended outcomes, and considered the realities of “real-world application” in describing the potential impact.

Feasibility

Several factors were considered in assigning a high rating for feasibility. One of the main elements reviewers looked for was that the team composition reflected the skills and disciplines required to successfully execute the planned research. Applications were highly regarded when they highlighted the roles of each team member and the contributions/methods from the multiple disciplines within it. Collaborations, industry connections, links with knowledge users, and engagement with Elders and Indigenous communities (where appropriate) were also identified as strengths. For large teams, an indication of how the members would work together and/or how each person would contribute strengthened the application. Reviewers looked for a well-developed project with clearly defined objectives and a sound plan for achieving them within the project’s timeframe, including some information about the methodology.

Reviewers expected that research teams consider Sex- and gender-based analysis, where appropriate, and use it to form the methodological approach. Proposals involving Indigenous research were also expected to address engagement with First Nations, Inuit and/or Métis peoples. Given the high-risk nature of the proposals, the inclusion of contingency plans was seen positively.

Equity, Diversity and Inclusion

Reviewers considered EDI when selecting their top 25 per cent of applications and selecting the applications to be funded. A tri-agency internal review committee also reviewed the EDI section of applications. This committee looked for at least one concrete measure to address each of three areas: team composition and training, recruitment, and inclusion. Reviewers noted applications that exhibited a meaningful and demonstrated commitment to EDI, indicated by EDI being incorporated and embedded into the team’s activities. Reviewers appreciated teams that encouraged diversity and inclusivity and that articulated commitments to training.

Interdisciplinarity

Both a tri-agency internal review committee and the multidisciplinary review panel assessed the interdisciplinarity of applications. The internal review committee evaluated applications according to the criterion’s definition: verification that research codes from at least two groups (Canada Research and Development Classification codes) were included and that these were reflected in the proposal. The rating of Pass or Fail refers only to this technical review.

The panel members reviewed the Interdisciplinarity of applications and considered it in their assessment of the High Risk criterion. Given the multidisciplinary nature of the review panel, the applications that stood out were those that pushed the boundaries in terms of interdisciplinarity. Applications that were regarded highly included those that brought together many different fields and/or included a unique combination, integration or amalgamation of disciplines. In all cases, reviewers looked for the interdisciplinarity to be reflected in the team composition and in the approach.

Reviewers appreciated clear explanations regarding the contributions of each team member and how different practices would be integrated in the research plan. Applications that proposed advances in multiple fields were particularly valued. Reviewers assigned higher ratings to applications that proposed projects that were conceived and designed from a truly interdisciplinary perspective, as opposed to projects where an interdisciplinary component was added to a more conventional project.