Measuring the Soft Underbelly of Science

I wasn’t sure where the phrase “soft underbelly” came from, but it was the first thing that came to mind to describe a problem in the improvement of scientific practice I’ve casually worked on for the past few years.

I had coffee the other day with Chris Shillum , the Executive Director of ORCID, and I was trying to describe a part of the problem I thought he might have part of a solution to. Nearly all scientists attend professional meetings and conferences, and almost all scientists take workshops at one time or another to build or acquire professional skills. These conferences and training sessions are short and potentially impactful, but we have very few ways to quantify their impact—especially long after the event. 

The phrase “soft underbelly” (attributed to Churchill ) refers to a vulnerable spot of an seemingly invulnerable opponent. For me, the task of closing gaps in professional development for researchers is that daunting opponent. The potential reward for victory? Accelerating the progress of scientific innovation—based on the premise that investing in developing the scientific talent we have today will yield exponential return on impactful science (see Reinventing Scientific Talent).

In our paper on the Bicycle Principles, we make several recommendations for what is needed to achieve greater impact from professional development (e.g., courses and workshops) and extending a bit further, improve other components of the research lifecycle that aren’t research in themselves (e.g., good mentorship, networking and exchange, and professional conferences).

If conferences and workshops add value to the scientific enterprise, we should be able to measure this.

“if you can’t measure it, you can’t improve it.”

Aphorsisim

Part of the problem is that there is no central database of the workshops and conferences scientists attend, and no comprehensive way to understand how participation at these events lead to future impacts. If we had this information, we could identify gaps and inequities by taking steps to support what made the biggest difference and rethinking what was unproductive.

What if researchers participating in workshops or conferences registered with their ORCID?

ORCIDs are free, unique identifiers that researchers can use to digitally label their works (e.g., publications). There are many benefits to doing this. Relevant to this discussion, on a researcher’s ORCID profile page, a bibliography of their work can be automatically compiled, and the ecosystem of digital research tools allows publications to be easily collated based on their association with an author’s ORCID. Aside from an author having to enter their ORCID, no other special effort is required.

If ORCID profiles captured the workshops a researcher completed or conferences they submitted abstracts to, it would generate a dataset that would expose trends and patterns for participation in the career growth and scientific output of researchers. For example, what courses enrolled in by researchers five years ago are associated with publications today? Which conferences are impactful in starting careers for scientists who we are struggling to include? Institutions and funders who support these activities should be able to justify their investments by quantitatively demonstrating that funding led to impact. As further incentive, the updating of biographical information submitted for grant proposals (e.g. SciENcv for life scientists in the US) could be a way to ensure researchers more easily get credit for their professional development. Tenure committees and award programs should also incentivize professional development by favorably considering these efforts as evidence a researcher is systematically working to update their skills and knowledge. 

For those of us who deliver training, we generally don’t have this information because it’s very difficult to collect more than a few weeks after an event, and everyone collects this information in different, incompatible, and unshared ways. This makes it frustrating to measure long term impact and find ways to improve the quality of training delivered (see Bikeprinciples.org recommendations B, E, H, J, and N). 

Adding ORCIDs to course and conference registrations would be a low-effort way to measure at least some impacts that would not otherwise be meaningfully measured. In talking with Chris, there would be few, if any, technological barriers to implementing this. What is needed now is dialogue and planning to fully characterize the potential benefits and mitigate any unintended consequences. Very important would be developing the social and professional incentives to implement. As usual, the social challenges to adoption are more onerous than technical ones. 

Using ORCIDs to track this difficult-to-measure underbelly of scientific practice could highlight the importance of participation in and access to professional development. This would be a one more step in recognizing another of the “non-traditional” outputs that researchers should get credit for while also helping funders maximize their investments.