The Bioinformatics Mastery Rubric – Interview with Rochelle Tractenberg

Meet Rochelle!

Rochelle Tractenberg (PhD, MPH, PhD, PStat®, FASA, FAAAS) is the current chair of the American Statistical Association Committee on Professional Ethics, Director of the Collaborative for Research on Outcomes and -Metrics, Professor in the Departments of Neurology; Biostatistics, Bioinformatics & Biomathematics; and Rehabilitation Medicine at Georgetown University and creator of the of Mastery Rubric for Bioinformatics (the reason I wanted to do this interview!).

---

The paper

The Mastery Rubric for Bioinformatics: supporting design and evaluation of career-spanning education and training

(BioRxiv pre-print; in review) Rochelle E. Tractenberg, Jessica M. Lindvall, Teresa K. Attwood, Allegra Via

---

The Interview

I caught up with Rochelle as we had our annual meeting for the Genomics Education Alliance. We chatted about the Mastery Rubric, and I thought we should put together a 150-200 word blurb/blog post and this is the result. I don’t think I’ve ever had a short conversation with Rochelle (there is always too much to learn) so I shouldn’t have been surprised. We settled on a few questions here that I hope give some context to the paper (and entices you to read it). Thanks to Rochelle for her time in providing these written answers.

Where did your interest in stats/cognitive science come from?

Those who never struggle can be so ineffective as teachers – they just have no idea how we (the strugglers) don’t automatically “see” what they mean…

I’m a professionally-accredited statistician and have been practicing in biostatistics (clinical trial design, as well as outcomes development) since 1997. But my first PhD (1997) was in cognitive sciences. As I have struggled with teaching hard things (like stats – and research ethics) to people who don’t always see much value in learning them, or in the effort required to learn them (graduate students and physicians in my experience) – I became interested in what makes learning some topics so hard for some people, while these same people find other topics very easy to learn.

Math and stats have always been incredibly hard for me, while learning languages has always been super easy. The difficulties I have had in overcoming my total inability to do or learn math (up until about 1993) have also helped me understand why those who never struggle can be so ineffective as teachers –  they just have no idea how we (the strugglers) don’t automatically “see” what they mean.

Knowing what I know about how adults learn (and how American education encourages students to do what they do well and stop struggling in courses or with topics they don’t automatically do well in) comes from cognitive sciences. Knowing how to help people get past their prior struggles also comes from cognitive sciences; and I apply these in my educational development efforts and in how I teach statistics (and ethics).

So, what is a “Mastery Rubric”?

A Mastery Rubric (MR) is like a traditional rubric, but instead of being specific for individual assignments like usual rubrics are, a MR describes the entire curriculum. Then, individual assignments (and their associated rubrics!) can be created by any instructor so that learners can show they are progressing according to the performance expected at different stages according to the MR. The desired knowledge, skills, and abilities (KSAs) that a learner should achieve are outlined together with performance levels – from less expert (novice) to more expert/independent (journeyman).

Creating a MR requires three key steps:

1. Identifying the KSAs that a curriculum should deliver, or that are the targets of learning
2. Identifying recognizable stages for the KSAs in a clear developmental trajectory that learners and instructors can identify, and that instructors can target in their teaching and assessment
3. Observable Performance Level Descriptors (PLDs) on each KSA at each stage, describing evaluable changes in performance from less to more expert

Essential feature of the MR: The name of the construct derives from the European Guild structure where a “Master” is a confirmed expert –confirmed by evidence of sustained, reliably superior, performance in the domain. However, in K-12 education, “mastery learning” has a very different meaning: that a student must “master” or pass a test with at least 80-90% correct answers, on target materials before the student can move onto the next set of materials. Not ever working with kids and never learning anything about k-12 education, I chose the guild structure for my modeling – for a great reason! – but now if you Google “Mastery Rubric” you don’t only get actual MRs, you may get them but you’ll also get a lot of info on rubrics for mastery learning – so ignore all of that! All Mastery Rubrics use the Guild Structure for staging, although (as is described later) not all of them include a Master level. Moreover, the transition from journeyman to Master is a challenging one in all disciplines, even for the MRs that do have a Master Level. So in 2017 I started building a Mastery Rubric specifically for getting people to the Master Level (in any discipline they choose, whether or not there is a specific MR for that curriculum). It (the MR-ML) is in its final refinements now.

Importantly – like a typical rubric, a MR is a construct. The construct of a rubric conveys information: that there will be several dimensions of the assignment on which the student will be graded; that there will be at least two levels of grade (probably more); only work that presents every gradable element at the highest level will receive that highest grade overall; etc. But knowing that an instructor will use “a rubric” does not tell you anything specific about the assignment or the instructor’s learning objectives, and until you see the rubric to be used, you wouldn’t know what the instructor is interested in, or how they’ll grade. The MR is similar: there isn’t just one  – there are 8 different applications of the MR construct so far. Each MR was developed specifically based on the target content or domain of the curriculum for which it was created.

The Mastery Rubric for Ethical Reasoning (MR-ER, published in 2012) supports the initiation and development of ethical reasoning and its instruction, spanning a scientist’s entire career. The Mastery Rubric for Bioinformatics (MR-Bi, published to BioRxiv in 2019 and currently in peer review) describes 12 KSAs as they develop across the career stages from true novice to fully-independent scientist. The Mastery Rubric for Evidence Based Medicine (MR-EBM, published in 2016) spans the clinician’s career from medical school (pre-clinical coursework) through residency and into any teaching the physician will do.

The MR-ER and MR-EBM include Master Levels, but the MR-Bi does not, because teaching (the support of apprentices, novices and journeymen) can be done by those at all performance levels, so it just wasn’t straightforward to include a “final” Master level in the MR-Bi. Another MR, describing the nurse practitioner curriculum (in the US, the MR-NP –currently in peer review) doesn’t have a Master level, but the main reason for its absence in this MR is that the NP competencies do not include teaching (unlike for EBM). Each MR was created for a specific curriculum where there have been difficulties in teaching, or assessing (or both).

What is the difference between knowledge, skills, and abilities?

There are some differences between these 3 “things” but not enough to justify any of the fits that seem to break out whenever they’re used. Here are general definitions:

• Knowledge is typically considered factual material; however, knowing that “science” is simply the currently-best-supported ideas rather than the “truth” in many cases is quite complex knowledge –while still being factual!  To give a practical example, a surgeon knows what “a suture” is. They would also know what kinds of wounds need sutures (and what kind don’t). This knowledge doesn’t mean they are able to perform sutures successfully.

• Skills are typically very specific and concretely observable or describable behaviors. For example, a surgeon could have the skills to execute a specific type of suture. This skill may develop (and be observed) without the individual knowing what type of wounds do/do not need sutures.

• Abilities are typically skills sequenced together, usually as a result of the application of knowledge. To complete our suturing example, the ability to suture would be recognized in that surgeon who has the knowledge to recognize that a wound is of a type that needs suturing, perform the suture, and also ensure that their suturing was done successfully (e.g., will be less likely to get infected/show a scar, etc).

The most important things to know about KSAs are:

1. When you’re talking about KSAs, the “A” should not be used to abbreviate “attitudes” in educational contexts, because attitudes are NOT observable. There’s no point in targeting attitudes in your teaching. Some people use “behaviors” and abbreviate these as KSBs; that way, there’s no chance people would mistake the A for attitudes. BUT the original educational researchers used KSAs, so that’s what I use.
2. Skills are agreed to be “less complex” than abilities – these ARE different. However, apart from US Federal and some state-level jobs descriptions, where they’re clearly differentiated, people don’t usually worry about the skills-abilities continuum. It’s just not very important – apart from the fact that skills are generally agreed to be highly specific while abilities are more complex and less specific than skills.
3. The expression “KSAs” is shorthand for two ideas:
   • Typically, instruction does not only target one of these (K or S or A) –typically it’s more than one. So, saying “KSAs” means
     “whatever you’re targeting with your instruction or your evaluation of whether a person is qualified”, and is not intended to mean “the specific knowledge and skill(s) and abilities you have in your mind right now”. “KSAs” is shorthand!
   • Sometimes people will disagree about whether “suturing” is a skill or an ability – it does not matter, and/but to avoid that argument if suturing is the target of instruction, you just say “KSAs”. Apart from ensuring that everyone knows you do not mean “attitudes” with your A, “KSAs” means “the elemental bits that your instruction is intended to convey”.

Can you explain what a learning trajectory is?

With a MR (and not with anything else I have ever seen), everyone – learners, instructors, curriculum developers, admin, parents everyone – can see where you are on each KSA. Since you can see where you are, you can also see what additional learning/development you need to get to the next stage of “sophistication” or independence…

I use the term “developmental trajectory” specifically to describe how a learner grows (develops) in their sophistication with respect to the given curriculum or domain. I also use Bloom’s Taxonomy (1956, not Anderson et al. 2001) exclusively to describe exactly what “more sophisticated” looks like; e.g., memorization (Bloom’s 1) is less sophisticated than application (Bloom’s level 3). This is handy for an instructor, or someone thinking about instruction over time (including a self-directed learner!), because Bloom’s verbs help you describe how these levels of performance can be distinguished in concrete and observable ways. Because a MR describes each KSA in the curriculum as it changes (or as the learner progresses along that KSAs developmental (or learning) trajectory), each row (KSA) of a MR constitutes one developmental trajectory. (It’s important to note that K-12 learning there’s a similar-sounding construct, a “learning progression” (or LP). Rather than going into the differences, I’ll just say here they aren’t exactly the same thing; you can read more about this on p.5 of the paper in SocArXiv describing all MRs, 10.31235/osf.io/qd2ae).

You can be super-fast developing on one KSA and quite slow on another – the point is, with a MR (and not with anything else I have ever seen), everyone – learners, instructors, curriculum developers, admin, parents everyone – can see where you are on each KSA. Since you can see where you are, you can also see what additional learning/development you need to get to the next stage of “sophistication” or independence. A Mastery Rubric is specifically for higher, post-graduate, and professional education, and much longer time scales –like your career as a scientist. Important to note: the MR is concrete and all verbs are specifically intended to support observable performance of any KSA in the curriculum; this is quite different from any other curriculum representation you might find (in higher/post-graduate education). I developed the construct because all the stakeholders should know what is expected of them, and what they should expect of the curriculum – and of themselves as they progress through it. MRs take a really long time to create, mainly because of the Performance Level Descriptors (PLDs). They have to be general enough for two different learners to be able to demonstrate their achievement using different materials, but specific enough that two independent observers (evaluators) would agree that any given work product does meet the PLD for a given stage.

A lot of biologists rely on workshops to get started on the path to learning bioinformatics – is that really a reasonable goal? Can workshops work?

It depends. They can totally work for people who have correctly identified:

• A) both a specific need they have to learn something and a learning opportunity where that will actually be learned;
• B) the level they need instruction to be pitched at;
• C) instructional opportunities that correctly describe their actual level of instruction!

I have many different examples in my own life and work where I did one or two but not all three of these; learning R – which I have done in the workshop but then never managed to keep in my brain! – is one of these. The problem for me – an accredited professional statistician with 22 years of experience as an actual biostatistician – is that I literally do not NEED R (this is item A). Whenever I need something done that my software (yes, I use SPSS! But also MPlus and BayesiaLab) can’t do (or won’t as easily as R does), I either ask a colleague for help or find my own old R code that did this thing, and revise it and re-run it. So, some biologists may feel, “I should really learn about bioinformatics!” – but in my (long) experience, this is simply not sufficient. Cognitive scientists would agree: a feeling that you should “learn something” is not going to lead to real, enduring learning – not without a huge effort and a lot of other ancillary support!

A whole other set of problems arise from B and C: people who offer “workshops in bioinformatics” do not know anything about requirement a!!! not only that, but R (if that’s what the workshop is about) isn’t supposed to be used just for the one thing you need to get done right now/today! You’re supposed to be learning to use it from now on so you don’t have to pay all those other license fees! But making that change, from “I don’t use R” to “I totally use R” takes a lot more than one workshop – when you leave the place where exercises have been carefully crafted so there’s an answer, and you have a high likelihood of getting that one answer, you will go into a place (your workplace) where… there’s no structure. And since you’re no longer in a workshop, there may be no help whatsoever. So, everything you learned about R evaporates as you return to what you do know will actually get the job done. Even the best-described, most correct workshop cannot help you once you’re back in the uncertain-but-time-crunched context of real work! Aanecdote: I spent a full week, 40 hours, learning MPlus during a semester long course crammed into one week – the course had both B and C, but I had only part of A. What I learned in that workshop was a general idea of how this software worked, and what kinds of problems it could solve. I spent the next year ruminating, but not using the software. I had to go to that same course again about 18 months later – with actual problems I thought the software could solve – before I was able to integrate it into my skill set and mental habits. If I had done two back-to-back weeks, that wouldn’t have happened; and if I hadn’t been able to conceptualize problems that software could solve, I also would never have ended up using it. So I feel strongly, from my own experience and also from what we know about how adults learn, that you need all three of these for a workshop to work for you!!

In addition,

1. learners have terrible trouble doing B (because you usually don’t know what you don’t know; and also, Dunning-Kruger 1999!) (google that!) and
2. instructors are often simply unable to do C. They describe what they HOPE they’re targeting but if you see my answer to question 2, people who are good at stuff can be really bad at understanding the trouble that people who are NOT good at stuff will have/do have.

I have met some people who did successfully teach themselves/use workshops to learn many things in bioinformatics. I would guess that they already had A) –at least implicitly; or, they’re already really good at what bioinformatics needs you to be good at (understanding computation and how to program to get the answer to a question). I have met many more people who never got what the training was meant to provide – and then they feel bad or “stupid” because they already had the training! Why don’t they know bioinformatics!? (BTW, this breakdown of workshop success (minority) and failure (vast majority) is also true of statistical software training, including R!)

Follow up – If you had to choose one thing for an instructor to do to improve C – correctly describe their actual level of instruction – what would it be? In this case is the MR-Bi a first step for them?

What a great question!! I absolutely intended the MR to help instructors – that is the reason they’re for curricula, not classes, workshops, or courses, because I got a job as a curriculum director and needed to be able to get all the instructors to both offer exactly what students needed throughout the curriculum and also describe exactly what students would be able to do at the end of their specific parts of that curriculum. To correctly describe what your course or workshop would do, if there is no MR that exists for your domain (which is possible, given that there are only 8!), I would focus on Bloom’s verbs and on the three questions Samuel Messick articulated for valid assessment (https://onlinelibrary.wiley.com/doi/pdf/10.1002/j.2333-8504.1992.tb01470.x – p. 17):

1. What are the knowledge, skills, and abilities (KSAs) the curriculum (or course/workshop) should lead to?
2. What actions/behaviors by the students will reveal these KSAs?
3. What tasks will elicit these specific actions or behaviors (that reveal KSAs)?

Whether or not the instructor who seeks to improve their C is working in bioinformatics, the MR-Bi does describe performance on each KSA at different levels concretely, and those performance level descriptors were painstakingly developed using Bloom’s and Messick explicitly. The method for PLD drafting is shown and discussed in the supplemental materials of that pre-print. So maybe that effort can help – model the process, or actually provide guidance, for how to “correctly describe their actual level of instruction!” The PLDs in the MR-Bi are intentionally not content specific, meaning both that a) any instructor in bioinformatics should be able to use it for a wide variety of content; and b) performance at any level on any of the KSAs can be demonstrated with diverse tools, techniques, and materials. 

But using the verbs and PLDs in the MR-Bi (or any MR) is only one part of ensuring your actual level of instruction is described correctly – the description then also has to match the instruction as delivered. So, the interested instructor can just consider the 3 Messick questions (what exactly is their instructional objective?) and then look at what actually happens in their class, and make sure to be objective about whether or not there truly is a match. (and consider asking an independent observer to confirm your descriptions – describing their observations as concretely as possible (i.e., not “yeah, looks good”, but more like, “you’ve stated you want to augment the learner’s knowledge and you present a series of facts, so that’s a match”).

A lot of the educational principles you discuss in the pre-print and elsewhere in your work have been established and accepted for decades, but it seems like many in the biology/bioinformatics community are just getting familiar with them. Any idea why?

If you get your PhD in chemistry (or anything), where you’ve shown your ability to do independent research, for some reason this is also interpreted worldwide as also evidence that you can teach…

Two ideas why:

1. If you get your PhD in chemistry (or anything), where you’ve shown your ability to do independent research, for some reason this is also interpreted worldwide as also evidence that you can teach. That’s just how it has always been. SO when someone comes along and says, “hey – you know, 100s of PhDs have been awarded in educational psychology and cognitive science – and there are decades worth of science – the currently-best-accepted theories about how adults learn! – that you could be using! And more of your students would GET what you’re teaching!”, some people really do not appreciate that. (people have actually told me, “why should I make this easier on the next generation of (economists)?. It was incredibly hard for me. Let them struggle, it’ll build character.” And my personal favorite, “I’m worried that you’re going to discover that I’ve been doing it wrong” (all this time). Also – some other disciplines prefer to believe that the humans in their programs “learn differently” than the humans that have been the subject of every educational psychology and cognitive science study. When you put it like that, it seems pretty stupid – again, people really do not appreciate finding out that they’ve been struggling to re-invent a wheel that has existed for decades, but also, that they lack the basic tools (understanding of human learning; principles of assessment) and are unlikely to have ever re-invented an actual wheel.
2. There is a ridiculous hubris in some science fields: educational or psychological research isn’t “real”, or it isn’t “hard”. As a journal editor and reviewer, I have observed educational “research” from many disciplines carried out with no consideration of educational theory or methodology. If you look at what passes for “educational research” within any specific field that is not education, you would be likely to find that authors have near ZERO qualification to engage in this particular type of research. In some fields (e.g., nursing and medicine), people are expected to do, and rewarded for doing, “educational research”. However, given that these clinicians do not have research degrees (or training), and almost universally do not have degrees or training in educational research, as you might imagine, they are just as qualified to do educational research as they are to do research in mathematics –but no one would ever suggest that clinicians should engage in mathematics research! (it’s so much harder than education research!). Because some people think educational research isn’t “real”, its results must be similarly not real. Why bother reading about that!? It does come back to the fact that most of us had terrible teachers –at least, in the topics we ended up abandoning because we did so much better in other fields (with better teachers? Probably not!) – but WE managed to get through it without using cognitive psychology, and our institutions clearly labeled us as “competent to teach”, so why would we need to go into a whole other field to learn how to teach better? But when you look at the literature about teaching and learning, these ideas DO make sense and CAN be very useful! So I do hope that people find, at least, that Bloom’s (1956) and Messick (1992) are important and helpful to think about in their own teaching.

Follow up question: Do these ideas meet with resistance?

I find that they do for those who have deeply invested in the idea that they or their students or their field are special – so nothing that a psychologist could have discovered could possibly apply to them. Some are resistant to the point of aggression (e.g., vitriolic ad hominem attacks)! (very interesting to me as a psychologist, while frustrating to me as a cognitive scientist!) Also, following what I said earlier, I have found that “psychology” is sometimes not perceived to be a “real science” by some other types of scientists, so, I mean, whatever the psychologists say, that can’t be right!

Also, speaking of hubris, some people just think that they’re special –and although it took ME 6 years to get my PhD in cognitive science, they can learn to do cognitive research in a matter of hours. I’m not saying I’m the sharpest tool in the shed, but…still. (This is also my experience with statistics, by the way – I have had people question why anyone (of their PhD students) needs to know more than they do about statistics, which is nothing besides how to get a p-value out of Excel for one t-test at a time (paraphrasing, but not by much). I have also been asked to “teach – these students – statistics” in one 3-hour lesson – during which time they cannot be asked to do any activities, because not all the students will be present. So – there’s resistance and there’s passive resistance.)

However, people newer to teaching – and those who already have tenure, so may feel (suddenly) able to focus on how they teach – may be experiencing what I did when I first realized that there might be a reason why I could not learn math: “gee this is SO HARD! Why!?!? Hm, hasn’t anyone studied/discovered how to teach better?” –the answer is YES, all the people in educational psychology and cognitive science! So, people who are interested in teaching better without re-inventing every wheel seem much more interested/less aggressively resistant! So, although I seem to have encountered all the jerks in academia (ever), I have also met with a lot of people who are really engaged, and really committed to teaching the best they can – at all levels. I want to encourage people who feel like they could be teaching better! There are a lot of tools and theories they can utilize – irrespective of whether their advisors tell them (as I promise you, one said to me), “If you have extra time to worry about your teaching, you don’t have enough to do”. (ALLLLL the jerks in academia)

Can you tell us about your dogs?

Atticus Finch and Ellen Ripley are two Rhodesian Ridgeback litter mates we adopted in 2012 at 10 weeks old. Atticus is 100 lbs and full of personality. He enjoys chicken and celery –and anything sweet -, long walks where no other dogs or people are encountered, but there are squirrels, and having a blankie on his head. Ripley is 85 lbs and enjoys long walks – sometimes without Atticus. She loves chicken and pretty much anything else edible that is not a green vegetable, and having a blankie on her person (doesn’t have to be her head, unlike Atticus, who cries if the blankie is on his person but NOT on his head). Ripley hates celery and going for rides in the car.

Both puppies feel that I do not wash my face sufficiently to be seen on video/Skype calls, and regularly clean my face when it is my turn to talk, unless they’re already comfortably under blankets.