Authors
Liberty A. Walton, Brandie K. Taylor, Krystal A. Tomlin, Dione I. Washington, Larry S. Solomon, and Jane C. Lockmuller
Abstract
As the volume of data collected by researchers continues to expand rapidly, the National Institutes of Health (NIH) seeks to provide ways for data to be shared and used in meaningful ways. The Immunology Database and Analysis Portal (ImmPort) System, developed for the Division of Allergy, Immunology, and Transplantation (DAIT), of the National Institute of Allergy and Infectious Diseases (NIAID), is one such way that NIH provides a unique and meaningful platform for the scientific community to share and explore clinical and basic research data. To determine how this data repository is being used in the scientific community and how to improve the system’s efficiency and effectiveness, The Madrillon Group Inc., under contract and in collaboration with NIAID’s Strategic Planning & Evaluation Branch and DAIT, conducted a process evaluation of ImmPort. This poster illustrates the evaluation design, provides key findings, and outlines lessons learned from this evaluation.
Background
ImmPort provides the means for scientists to easily access, store, analyze, and exchange complex, high-quality data sets. ImmPort:
- has been under development and deployment by Northrop Grumman (NG) since 2005;
- advances the discovery and generation of new hypotheses for immune-mediated diseases and further understanding of innate and adaptive immunity;
- houses clinical and basic research data;
- offers unique analytical tools;
- is accessible to anyone with a valid email address; and
- provides a platform where data submission is mandatory for most DAIT-funded projects.
Areas of Interest
- Quality Assurance and Quality Control
- Data Use and Utility
- User Satisfaction
- Outcomes Identified in Publications
- Interoperability with Other Platforms
- Infrastructure Usability
- Tool Use and Comprehension
Evaluation Questions
In relation to ImmPort, NIAID pursued the following questions:
- What are the characteristics of this platform and its users?
- What are appropriate existing metrics for evaluating the utility and outcomes of ImmPort, and what are the best approaches for collecting this data in the future?
- How are the data repository and associated tools being used in the immunology community?
- To what degree are participants satisfied with their experience, and what aspects could be improved to enhance its efficiency and effectiveness?
- What computing and knowledge-base infrastructure components would promote the innovative reuse of ImmPort data, but are currently missing, if any?
- How effective are the outreach and education efforts?
Data Sources & Methodology
- Literature Review
- Search engines: MEDLINE, PubMed, and Google Scholar
- Search terms: database, data registry, data repository, research repository, metrics, evaluation, measurement, data quality, immunology, and biomedical
- Snowball strategy using the bibliographies cited in original search results
- Information also pulled from NIH websites and publications and journals specific to this content area
- Search yielded 64 articles
- Semi-Structured Phone Interviews
- Recruited 11 NIAID Project/Review Officers and 9 ImmPort Staff
- Interviewed 7 NIAID Project/Review Officers and 9 ImmPort Staff
- 80% Response Rate
- Interviews lasted approximately 30 minutes
- Web-based Surveys
- Two unique surveys for (1) people who use ImmPort data or tools and (2) people who submitted data to ImmPort
- Some topics surveyed: types of use; ease of use; types of data submitted and accessed; use of data, user satisfaction, suggestions for improvement
- Open- and close- ended questions in electronic survey using SurveyGizmo
- Received PRA/OMB Clearance via NIAID Fast Track Generic Clearance
- 2 invitation and 3 reminder emails were sent to registered ImmPort accounts with password protected links to the survey
- Average of 15-21 minutes to complete survey
- Surveys were fielded for 20 days
- 14% Response Rate (222 completed surveys/ 1,578 invitations)
- Analytics of Archival User Data
- Northrop Grumman, the ImmPort developer, routinely collects web analytic data on ImmPort users and system use
- Archival data fields: user characteristics, reasons for registering for the system, tools used, data submitted, and data downloaded by user email address
- Data were available from 2005-2015
- NIH Grants and Contracts Database
- “ImmPort” was searched in the title, summary statement, and abstract in IMPAC II QVR of all applications for grants, contracts, and intramural projects submitted to NIH for FY 2007-2015
- Full text search of the application was performed for “ImmPort” to determine if the applicant proposed using ImmPort data, submitting data to ImmPort, and/or working with the ImmPort team
Key Findings
Figure shows ImmPort Users’ (N=83) perceptions on system access. Eighty-eight percent of Users stated they did not have difficulty accessing tools. In contrast, 12% of users had some level of difficultly accessing tools.
Figure shows ImmPort Users’ (N=83) perceptions on system access. Eighty-eight percent of Users stated they did not have difficulty accessing tools. In contrast, 12% of users had some level of difficultly accessing tools.
This graphic illustrates submitter experiences with the data submission process. Thirty-nine percent of submitters believed the process was a lot more time consuming than expected. Thirty-two percent felt it took somewhat more time than expected. While 14% of respondents stated submission times matched their expectations. Equally, 14% of respondents found submission time to be somewhat less time than expected.
This figure presents likelihood that Users (N=81) will access data from ImmPort in the future. Most users (47% of the 81 respondents) stated they were very likely to access data from ImmPort. The remaining responses were divided among “Somewhat likely” (38%), “Somewhat unlikely” (11%), and “Very unlikely” (4%).
Figure presents respondents perceptions of the submission process. Respondents stated Submission process complexity as the biggest (66%) barrier to the submission process. Thirty-one percent identified both Lack of system training and lack of data organization in my group as a barrier. The remaining barriers were identified as Lack of data documentation by collaborators (24%), Lack of funding (21%), Lack of customized ImmPort assistance (14%), Other (10%), Lack of programming skills (7%), and None (7%).
Figure presents users' suggestions for improvements to ImmPort (N=37). Most respondents stated ImmPort would benefit if data was easier to find or browse (22%). The remaining suggestions were identified as Simplify downloads (16%), Make it easier to import to other software programs (14%), standardize formats (11%), Beta site is good (11%), Add more or newer data (8%), Add or keep tutorials and help (8%), Other (8%), Improve data quality (5%), and Increase awareness in the community (5%).
Recommendations
- Improve data submission process
- Improve data searching features
- Enhance interoperability and infrastructure
- Collect more data on outcomes
- Expand outreach efforts to broader communities and junior investigators; create an active online user group; increase references to ImmPort in high-profile publications
Lessons Learned
- Not surprisingly, self-reported data does not always match automated records
- When inviting all registered users of a system to participate in a survey, it is acceptable to have a low response rate due to a variety of factors, such as changes in contact information over time and individuals not actually having experience with the system after registration – which limits the pool of potential respondents
- Some institutions may have a strong firewall that prevent the delivery of survey invitations, further limiting the response rate
- Active and consistent engagement and input from SMEs and Program Staff are invaluable to having the evaluation run effectively and efficiently
Contact Information
NIAIDEvaluation2@niaid.nih.gov
Acknowledgements
The Madrillon Group, Inc.
Poster Presented at the 2016 American Evaluation Association (AEA) Annual Meeting: Walton, L., Taylor, B., Tomlin, K., Washington, D., Solomon, L., & Lockmuller, L. (2016, October). Evaluation of the NIAID/NIH Immunology Database and Analysis Portal (ImmPort). Presented at the Annual AEA Conference Meeting, Chicago, IL.