class: center, middle, inverse, title-slide # 05
interactivity and immediate feedback ## 💫 with learnr
🔗
bit.ly/teach-ds-wsc
###
dr. mine çetinkaya-rundel
dr. colin rundel ### 23 june 2021 --- class: middle, inverse .pull-left[ .huge-text[why] ] .pull-right[ .larger[ auto feedback ] ] --- class: middle .pull-left[ .center[ .huge[👉] <br> .large[**Nudging**] <br> students towards the right answer, especially in formative assessments ] ] --- .sample-question[ Suppose 10 means from a simulated sampling distribution is stored in a vector called `means`. ```r means ``` ``` ## [1] -1.21 0.28 1.08 -2.35 0.43 0.51 -0.57 -0.55 -0.56 -0.89 ``` What is the value of the first mean? ] -- <br> .pull-left-wide[ .sample-answer[ ```r mean[1] ``` ``` ## Error in mean[1]: object of type 'closure' is not subsettable ``` ] ] -- .pull-right-narrow[ <img src="img/student.png" title="Illustration of student with a speech bubble saying 'Object of type what now?'" alt="Illustration of student with a speech bubble saying 'Object of type what now?'" style="display: block; margin: auto 0 auto auto;" /> ] --- ## Nudging .hand[Not all feedback is useful, at least not for beginners...] <br> Providing helpful feedback can help them nudge them towards success: ```r mean[1] ``` ``` ## x `mean` is a function and a function doesn't have elements that can be subsetted with square brackets. ``` ``` ## ℹ `means` is the vector of sample means calculated earlier. ``` --- .sample-question[ Visualise the relationship between city and highway mileage of cars from the `mpg` dataset, conditional on year of manufacture. ] -- <br> .sample-answer[ There is a strong, positive, linear relationship between the city & highway mileage. Year does not seem to be related to either variable. ```r ggplot(mpg,aes(x = hwy, y = cty, fill=year)) +geom_point()+geom_smooth() ``` <img src="05-learnr_files/figure-html/unnamed-chunk-7-1.png" title="Scatterplot of highway mielage and city mileage where points are colored by year and a linear model is represented with a line with a positive slope on the plot." alt="Scatterplot of highway mielage and city mileage where points are colored by year and a linear model is represented with a line with a positive slope on the plot." width="70%" /> ] --- class: middle .pull-left-wide[ .sample-feedback[ - You mention a linear relationship, however your plot uses a loess fit to visualise the relationship between city and highway mileage. Also, the plot displays the uncertainty around the fit, but you haven't addressed it in your narrative. - Year should be mapped to the `color` aesthetic, not `fill`. - Plot styling: Use informative axis labels, noting units of measurement. Also, give an informative title to your plot. - Code styling: Use consistent spacing around operators (e.g ` = `) and line breaks after `+` in each layer of your ggplot. ] ] .pull-right-narrow[ <img src="img/teacher.png" title="Illustration of a teacher with a speech bubble that says 'So much to say, so little time...'" alt="Illustration of a teacher with a speech bubble that says 'So much to say, so little time...'" style="display: block; margin: auto;" /> ] --- class: middle .pull-left[ .center[ .huge[👉] <br> .large[**Nudging**] <br> students towards the right answer, especially in formative assessments ] ] .pull-right[ .center[ .huge[➕] <br> .large[**Scaling**] <br> up efficiency of grading faster than (human) resources ] ] --- ## Scaling .hand[Our courses are growing, and that's a good thing, right?] -- - Students turning in their work as R Markdown documents makes collecting submissions including code and narrative straightforward. -- - Providing feedback on both the code and narrative is not scalable unless (human) resources dedicated to your course grow proportionally with enrolments. --- class: middle, inverse .pull-left[ .huge-text[how] ] .pull-right[ .larger[ auto feedback ] ] --- <img src="img/puzzle-pieces.png" title="Two puzzle pieces that fit together, one says learnr and the other gradethis." alt="Two puzzle pieces that fit together, one says learnr and the other gradethis." width="70%" style="display: block; margin: auto;" /> --- ## 📦 learnr - **learnr** is an R package that makes it easy to create interactive tutorials from R Markdown documents. - Tutorials can include: - Narrative, figures, illustrations, and equations - Code exercises (R code chunks that users can edit and execute directly) - Multiple choice questions - Videos (YouTube, Vimeo) - Interactive Shiny components - learnr is on CRAN ```r install.packages("learnr") ``` --- ## 📦 gradethis - Companion to the learnr package, **gradethis** provides multiple methods to grade learnr exercises - gradethis is not yet on CRAN ```r devtools::install_github("rstudio-education/gradethis") ``` --- .huge-text[demo] .large[ [**`[tutorial]`**](https://minecr.shinyapps.io/penguins) [**`[code]`**](https://github.com/mine-cetinkaya-rundel/teach-ds-wsc-2021/blob/main/materials/05-learnr/penguins/penguins.Rmd) ] --- class: middle, inverse .huge-text[learnr] --- ## YAML ```r --- title: "Palmer penguins" output: learnr::tutorial: progressive: true allow_skip: true css: "css/font-size.css" runtime: shiny_prerendered --- ``` - `runtime: shiny_prerendered` - `progressive: true` for "Continue" buttons between subsections - `alow_skip: true` to allow skipping exercises --- ## Narrative - Markdown section and subsection headings with `##`, `###`, etc. - Text, figures, illustrations, and equations. - Videos: supported services include YouTube and Vimeo ```r ### Learning goals - Practice developing exercises with learnr. - Practice writing code checking tests with gradethis. ### Packages In this assignment we will work with the **palmerpenguins** and **tidyverse** packages. So let's go ahead and load them! ``` --- ## Multiple choice questions ```r question( "How awesome are penguins?", answer("Super awesome!", correct = TRUE), answer("Awesome"), answer("Meh"), answer("Not awesome"), answer("Super not awesome") ) ``` --- ## Text entry questions ```r question_text( "Please enter the word 'Penguin' below:", answer("penguin", message = "Don't forget to capitalize"), answer("pEnguin", message = "Did you capitalize properly?"), answer("Penguin ", message = "Make sure you do not have a trailing space"), answer("Penguin", correct = TRUE), allow_retry = TRUE, trim = FALSE ) ``` --- .your-turn[ *Recommend one person in each group share their screen and everyone work together to work through the document.* - Go to RStudio Cloud - Start the assignment titled `04 - Interactive tutorials` - Open `penguins.Rmd` and click on **Run Document** - Read the instructions under `Multiple choice questions` - See help for `?question` and `?quiz` and don't hesitate to call for help! ]
10
:
00
--- class: middle, inverse .huge-text[gradethis] --- ## Automated feedback - Use a code chunk with the same label, suffixed with `-check` - `.result` refers to the resulting output - Think about ways things can go wrong and write test cases for them - Write a "catch all" test case for everything else --- ## Checking options - `grade_this()`: Grade result of exercise code - `grade_this_code()`: Grade code against a solution - See `gradethis::gradethis_demo()` for examples --- ## Code exercises - demo - Go to RStudio Cloud - Start the assignment titled `Interactive tutorials` - Open `penguins.Rmd` and click on **Run Document** --- ## Keep in mind! - Exercise chunks run in independent sessions, they don't actually work like R Markdown chunks (i.e. they don't remember what happened before) - Use setup chunks to make the tutorial experience feel more like a data analysis story - Leverage this feature to write robust code and checks --- class: middle, inverse .huge-text[sharing] --- ## Sharing with students - You could share the R Markdown document (and all accompanying files) but that’s probably not what you want to do... - Deploy on - shinyapps.io - RStudio Connect (free for academic use, requires setup) - Distribute as a package - See the [publishing instructions](https://rstudio.github.io/learnr/publishing.html) on the learnr website for step-by-step instructions --- class: middle, inverse .huge-text[recording] .huge-text[data] --- ## Recording attempts - A "good enough" solution for formative exercises: embed a Google/Microsoft/etc. Form at the end and ask students to "submit" their work. - This only records that the student reached the end of the tutorial and not how (or even if) they answered any of the questions or exercises. - **Tip:** Add a free-text question to the form asking students to reflect on the exercises they just completed - you can then analyse the free-text data to gain insights into what students are struggling with -- "minute paper". .large[ [**`[example]`**](https://minecr.shinyapps.io/04-wrangleviz/#section-finish-up) ] --- ## Recording submissions The [`learnrhash`](https://github.com/rundel/learnrhash) package builds on the previous method by providing a way for students to submit their answers by generating a text "hash" which can be copy and pasted into the web form. ```r devtools::install_github("rundel/learnrhash") ``` <br> .large[ [**`[example]`**](https://minecr.shinyapps.io/penguins/#section-submit) ] .footnote[ See also the [**submitr**](https://github.com/dtkaplan/submitr) package by Danny Kaplan for a different approach to recording event data in learnr tutorials. ] --- class: middle, inverse .huge-text[closing] .huge-text[thoughts] --- ## Best practices for automated feedback -- - Measure twice, cut once (verify the correctness of your tests) 📏📏 ✂️ -- - Use rounding & type coercion to write robust tests 💪 -- - Don't give automated feedback on everything, asking narrative questions that can't be auto checked but gets the student thinking and writing has pedagogical benefits 🙈 -- - Consider peer feedback where automated feedback is not feasible (e.g. interpretation, narrative) but scalability is an issue 👯 --- .large[ .hand[Q: What is an approachable way to get started?] ] Build a tutorial where students build develop their analysis in exercise code chunks (that are not checked) and only multiple choice questions are used for assessment. [**`[example]`**](https://minecr.shinyapps.io/04-wrangleviz) <img src="img/learnr-simple.png" title="Screenshot of a sample learnr tutorial." alt="Screenshot of a sample learnr tutorial." width="85%" style="display: block; margin: auto;" /> --- .large[ .hand[Q: I've built simple tutorials already. How do I make the jump to code checking and providing automated feedback that is actually useful?] ] - Replicate `gradethis::gradethis_demo()`, then make incremental changes - Read the [Testing](https://r-pkgs.org/tests.html#test-tests) chapter in R Packages (Wickham and Bryan) - Also read the [Metaprogramming](https://adv-r.hadley.nz/metaprogramming.html) section in Advanced R (Wickham) <img src="img/learnr-learn-more.jpeg" title="Screenshots of resources listed in bullet points in slide." alt="Screenshots of resources listed in bullet points in slide." width="75%" style="display: block; margin: auto;" /> --- .large[ .hand[Q: Sounds great, but can it handle my class size and usage?] ] - First, chances are you're not using these live, but you might be... - If so, make sure to max out your instances and instance size on shinyapps.io. - Essential reading: - [Publishing learnr Tutorials on shinyapps.io](https://cran.r-project.org/web/packages/learnr/vignettes/shinyapps-publishing.html) by Angela Li - [Teach R with learnr: a powerful tool for remote teaching](https://education.rstudio.com/blog/2020/05/learnr-for-remote/) by Allison Horst --- .large[ .hand[Q: Where can I find more examples?] ] * Both the learnr package and gradethis have a large number of examples, e.g. ```r gradethis::gradethis_demo() ``` * [RStudio Cloud Primers](https://rstudio.cloud/learn/primers) - these are all learnr tutorials and their code is available on [github](https://github.com/rstudio-education/primers) * [Feedback at scale](https://www.rstudio.com/resources/rstudioglobal-2021/feedback-at-scale/) talk at rstudio::global(2021) * [Data Science in a Box](http://datasciencebox.org) - code for all tutorials available in the [dsbox](https://github.com/rstudio-education/dsbox) package