Assessment of the Centers for HIV/AIDS Vaccine Immunology and Immunogen Design

Authors

Liberty A. Walton, Brandie K. Taylor, Larry S. Solomon, Stuart Z. Shapiro & Angela Malaspina

Abstract

The National Institute of Allergy and Infectious Diseases (NIAID) of the National Institutes of Health uses several approaches to support scientific research. One method that NIAID recently deployed is the large-scale, high-resource, “big science” approach.  This method allows the government to fund and manage the research of large, complex issues that require input from several disciplines. NIAID uses the approach to support the Centers for HIV/AIDS Vaccine Immunology and Immunogen Discovery (CHAVI-ID). Funding for CHAVI-ID ends in fiscal year 2019, at which point NIAID aims to support a third iteration of the CHAVI program. For this reason, NIAID supported a comprehensive program evaluation to assess CHAVI-related outcomes and identify areas for program improvement. This evaluation also aimed to understand the value of CHAVI’s “big science” approach within the larger field of HIV vaccine research. This poster illustrates the process and outcome evaluation methodology, key study findings, and strengths and limitations.

Evaluation Purpose

  • Assess program outcomes
  • Identify areas to improve the program implementation and design
  • Understand the impact of this large-scale program within the greater HIV vaccine research field

Areas of Interest

  • Areas for program improvement
  • Scientific outcomes of the program to-date
  • Impact of the program on the HIV vaccine field
  • Focus of future program initiatives
  • Role of large-scale program for advancement of program goals
  • Development of infrastructure and knowledge base to help sustain or expand the vaccine field
  • Collaborations of the program scientists with other researchers in the vaccine field

Program Logic Model

The activities section of the logic model highlights CHAVI/CHAVI-ID key activities. See image caption for details.

The activities section of the logic model highlights CHAVI/CHAVI-ID key activities, including the three major areas of focus ( Full caption on https://www.niaid.nih.gov/about/assessment-chavi-id)

Credit: NIAID

Methods

  1. Portfolio Analysis
    1. Included a bibliometric analysis, network analysis, analysis of other outcomes data (e.g., patents, subsequent NIH funding, achievements of junior personnel), and content analysis
    2. Data sources included Internal NIH Databases (e.g., IMPAC II), iCite, NIH ExPORTER & RePORTER, PubMed, SPIRES, USPTO Database, Web of Science, and CrossRef
  2. Interviews
    1. 8 NIH program staff engaged in CHAVI/CHAVI-ID grant oversight
    2. 8 Non-federal employees currently or previously affiliated with CHAVI/CHAVI-ID
    3. 3 HIV vaccine subject matter experts
    4. 54-minute interviews, on average

Comparison Groups

Grants were selected for the comparison groups based on: grant activity code, year of initial program funding, supporting NIAID Branch, the number of grants and subprojects, percentage of grants with Early Stage Investigator and New Investigator principal investigators, as well as the percent of funding for each category along the product development pathway. In addition, investigators were excluded from the comparison groups if they were associated with CHAVI/CHAVI-ID.

  1. R01 HIV Vaccine: A collection of HIV vaccine-related NIH R01 grant funding that are not collaborative, co-located, or co-funded.
    1. Comparison of traditional R01 funding versus big science funding
    2. Identify the impact of big science on outcome measures (e.g., productivity)
  2. P01 HIV Vaccine: Co-located, organized, or integrated programs of HIV vaccine-related research funded by NIH P01 grant funding (i.e., not funded by a big science mechanism).
    1. Takes co-location and natural collaboration outside of the big science approach into account
    2. Identify the unique effects attributable to the big science mechanism (e.g., collaboration)
  3. Malaria Vaccine: A collection of NIH grants (e.g., R, P, U) in a non-HIV vaccine-related field that has not received big science funding.
    1. Comparison of big science approach and traditional funding among two different vaccine research fields
    2. Identify the unique effects attributable to the big science approach on larger field of vaccine research

Publication Productivity

Group CHAVI/CHAVI-ID R01 HIV Vaccine P01 HIV Vaccine Malaria Vaccine
# of Distinct Publications 817 300 149 150
Average Publication Cost $539,373 $235,597 $417,974 $239,266
Publications per $10M 18.5 42.4 23.9 41.8
Impact Factor per Publication (Mean) 10.1 8.9 8.8 5.5
Impact Factor per Publication (Median) 5.2 4.6 4.9 3.5
Relative Citation Ratio (RCR - Mean) 3.20 2.07 2.23 1.61
RCR (Median) 1.70 0.92 0.94 1.23

Top HIV Vaccine Research Articles

The chart shows publication trends from 2005 to 2016 comparing the top 100 HIV Vaccine research articles. See image caption for details.

The chart shows publication trends from 2005 to 2016 comparing the top 100 HIV Vaccine research articles. (Full caption on https://www.niaid.nih.gov/about/assessment-chavi-id)

Credit: NIAID

Success of Junior-level Investigators

Metric CHAVI/CHAVI-ID R01 HIV Vaccine P01 HIV Vaccine Malaria Vaccine
Junior-level personnel 481 187 162 86
Any grant-specific publications 269 (56%) 104 (56%) 57 (35%) 43 (50%)
Average time to first publication 3.4 years 2.7 years 2.8 years 3.3 years
Publications per person (among those with any publications) (median) 4 2 2 2
Junior investigators with Subsequent NIH Funding 31 (6.4%) 6 (3.2%) 10 (6.2%) 3 (3.5%)

Limitations

  • Differences in funding levels between groups
  • Metrics only accounted for NIAID funding
  • Overlap between groups (i.e., collaborate on other grants)
  • Bibliometric measures were used to assess publication quality, but they have limitations. For example, impact factor may not necessarily indicate the impact of individual articles within a journal.
  • Interview findings may not be generalizable
  • Interview participants were selected by NIAID and most were currently or previously affiliated with CHAVI/CHAVI-ID

Future Considerations

  • Continue to offer a flexible funding mechanism
  • Maintain the current program structure; consider minor changes
  • Engage with HIV vaccine researchers outside of CHAVI/CHAVI-ID
  • Maintain balance of current program focus, but shift toward development activities and beyond broadly neutralizing antibodies
  • Continue to share resources with the HIV vaccine research field
  • Prepare for manufacturing challenges
  • Sustain junior investigator involvement; focus on collaborative opportunities

Contact Information

NIAIDEvaluation2@niaid.nih.gov

Acknowledgements

DAIDS Staff; Ripple Effect Communications, Inc.

This work has been funded in whole or in part with Federal funds from NIAID, under contract HHSN272201600016I.

Poster presented at the 2018 American Evaluation Association Annual Meeting

  • Walton, L., Taylor, B., Solomon, L., Shapiro, S., & Malaspina, A. (2018, October). Assessment of the Centers for HIV/AIDS Vaccine Immunology and Immunogen Design at the National Institute of Allergy and Infectious Diseases. Presented at the Annual AEA Conference Meeting, Cleveland, OH.
Content last reviewed on