Take a moment to familiarise yourself with the interface.

Go ahead and pick the first example.

Context: The scatter plot above shows the brain and body weights of n = 200 cats and n = 200 chihuahuas. The data sets were randomly drawn from a normal distribution whose mean and standard deviations were taken from autopsy findings (Bronson, 1979).

Setting: Imagine you want to develop (for whatever reason...) a machine learning algorithm that distinguishes cats and chihuahuas from each other, based on the given data. The blue separator represents the result of your algorithm: Everything above the separator is classified as a chihuahua, everything below as a cat.

Have a look at the sensitivity and specificity values. How do they change when you change the position of the separator?

• Try to set the specificity for cats as high as possible (to 1). How does this affect the sensitivity for cats? Why? (Try it the other way around, too!)
• Take a look at the contingeny table. Which values in the table lead to changes in specificity and sensitivity? Why?
• What happens to the specificity and sensitivity for chihuahuas when you adjust the values for cats?
• Now let's look at the miss rate and false alarm rate (click on the "Settings" button to display the two metrics). How do these two metrics change in relation to sensitivity and specificity?

Context: A key predictor of work performance is the candidate's intelligence. Intelligence and work performance correlate positively with each other (Sackett et al., 2022; Schmidt & Hunter, 1998; Schmitt, 2014).

The second measure shown in the plot is the applicant's previous leadership experience. It is reasonable to assume that tasks with leadership responsibility will have a positive influence on work performance in future leadership positions and should therefore be taken into account in the application process.

However, there are a number of systematic differences between women and men when it comes to leadership experience: as women on average take on more care work (e.g. looking after children or relatives) than men, and also take more time off work to do so (OECD, 2014; DeStatis, 2023). Therefore it is reasonable to assume that women (in management positions) have on average less management experience than men of the same age in similar positions.

Female applicants are outlined in black in the plot. You can click on the legend of the plot to hide (and show) the different groups.

Setting: As the HR-Manager responsible for selecting new management staff in a large company, you have access to data on intelligence (assessed via WIT-2; subscale "Reasoning") and leadership experience (in years) of N = 400 applicants for an open management position. All candidates placed above the separator will be invited to the further application process.

Your company uses a third-party machine learning model that scores candidates for leadership positions based on predictors such as age, leadership experience and application content. The model calculates a score between 0 and 100, with applicants scoring above 70 being categorised as 'likely to be successful'. Your company aims for equal gender representation in management positions.

(Please keep in mind that this is not real data and that a real candidate selection process should of course be designed differently!)

• Look at the sensitivity and specificity again. How would the values change if the two groups overlapped more (or less)?
• Is sensitivity or specificity more relevant when it comes to inviting as many likely successful candidates - and as few unlikely successful candidates - as possible?
• What is a key difference between sensitivity and specificity on the one hand and positive and negative predictive value on the other hand? How do the calculation and the interpretation of the different metrics differ?
• Activate the false discovery rate and the false omission rate in the settings. What is their relationship to the positive and negative predictive value?
• Now let's move on to another topic: Fairness. Would you recommend your company to continue using (and paying for!) the third-party machine learning model? What are the pros and cons?

Context: Anxiety and depression are positively correlated. This also leads to a high co-occurrence (comorbidity) of depressive and anxiety syndromes (e.g. Kessler et al., 1999).

This is also reflected in various questionnaires. For example, the STAI is used to assess trait and state anxiety, and both primary anxious and primary depressed patients score higher in trait anxiety than healthy controls (e.g. Kim et al., 2018; Wilbertz et al., 2010).

Furthermore, there is evidence suggesting that patients with depression often score higher in trait anxiety than patients with anxiety (possible explanation: STAI may measure unspecific negative affect, rather than specific trait-anxiety; Knowles & Olatunji, 2020; D’Avanzato et al., 2013).

In addition, patients with anxiety disorders also show increased scores in the BDI-II, a questionnaire on depressive symptoms (e.g. Coles et al., 2001).

In psychotherapeutic settings it is of elementary importance to diagnose comorbidities in addition to the "primary diagnosis". However, as the treatment of anxiety disorders differs from the treatment of depression, the primary diagnosis has far-reaching implications for treatment, e.g. the problem that is assumed to be central to the persistence of the patients condition should be treated first and be considered the primary diagnosis.

Setting: As a psychotherapy researcher, you are developing a test to discriminate between patients with depression as primary diagnosis and patients with an anxiety disorder as primary diagnosis.

(As always, it should be kept in mind that the data shown - although inspired by real world findings - is in fact simulated!)

• Let's look at sensitivity and specificity one last time. Assuming you want to make sure that patients with depression who also show high anxiety scores always receive a treatment program for depression (and none for anxiety). Which of the two metrics should you pay attention to? To what extent is there a trade-off here?
• Activate the positive and neg. likelihood ratio in the settings. Compare both metrics with regard to their interpretation. Which of these metrics should generally be minimized or maximized?
• What does a positive likelihood ratio of less than 1 indicate? What does a value of exactly 1 indicate?
• Activate the last four measures (accuracy, F₁-Score, diagnostic odds ratio, Youden's J) in the settings. What do they all have in common?
• What is the key difference between the diagnostic odds ratio on the one hand and accuracy, F₁-Score and Youden's J on the other?
• How does the interpretation of accuracy and diagnostic odds ratio differ from each other?