class: center, middle, inverse, title-slide # Feed me back now: ## Tales of automated feedback
in R and Python ### Mine Çetinkaya-Rundel & Tiffany Timbers ### eCOTS 2020 / 2020-05-20
🔗
bit.ly/feed-me-back
--- background-image: url(images/robots-background.jpeg) background-position: center background-repeat: no-repeat background-size: contain ## Assumed background --- background-image: url(images/robots-what-why.jpeg) background-position: center background-repeat: no-repeat background-size: contain class: middle, inverse # Automated feedback: # what and why? --- ## The what and the why -- .pull-left[ .center[ .huge[👉] <br> .large[**Nudging**] <br> students towards the right answer, especially in formative assessments ] ] -- .pull-right[ .center[ .huge[➕] <br> .large[**Scaling**] <br> up efficiency of grading faster than (human) resources ] ] --- .sample-question[ Suppose 10 means from a simulated sampling distribution is stored in a vector called `means`. ```r means ``` ``` ## [1] -1.21 0.28 1.08 -2.35 0.43 0.51 -0.57 -0.55 -0.56 -0.89 ``` What is the value of the first mean? ] -- <br> .pull-left-wide[ .sample-answer[ ```r mean[1] ``` ``` ## Error in mean[1]: object of type 'closure' is not subsettable ``` ] ] -- .pull-right-narrow[ <img src="images/student.png" style="display: block; margin: auto 0 auto auto;" /> ] --- ## Nudging .hand-blue[Not all feedback is useful, at least not for beginners...] <br> Providing helpful feedback can help them nudge them towards success: ```r mean[1] ``` ``` ## x `mean` is a function and a function doesn't have elements that can be subsetted with square brackets. ``` ``` ## ℹ `means` is the vector of sample means calculated earlier. ``` --- .sample-question[ Visualise the relationship between city and highway mileage of cars from the `mpg` dataset, conditional on year of manufacture. ] -- <br> .sample-answer[ There is a strong, positive, linear relationship between the city and highway mileage of cars. Year does not seem to be related to either variable. ```r ggplot(mpg,aes(x = hwy, y = cty, fill=year)) +geom_point()+geom_smooth() ``` <img src="feed_me_back_now_files/figure-html/unnamed-chunk-7-1.png" width="70%" /> ] --- class: middle .pull-left-wide[ .sample-feedback[ - You mention a linear relationship, however your plot uses a loess fit to visualise the relationship between city and highway mileage. Also, the plot displays the uncertainty around the fit, but you haven't addressed it in your narrative. - Year should be mapped to the `color` aesthetic, not `fill`. - Plot styling: Use informative axis labels, noting units of measurement. Also, give an informative title to your plot. - Code styling: Use consistent spacing around operators (e.g ` = `) and line breaks after `+` in each layer of your ggplot. ] ] .pull-right-narrow[ <img src="images/teacher.png" style="display: block; margin: auto;" /> ] --- ## Scaling .hand-blue[Our courses are growing, and that's a good thing, right?] -- - Students turning in their work as R Markdown documents makes collecting submissions including code and narrative straightforward. -- - Providing feedback on both the code and narrative is not scalable unless (human) resources dedicated to your course grow proportionally with enrolments. --- background-image: url(images/robots-options.jpeg) background-position: center background-repeat: no-repeat background-size: contain class: middle, inverse # Options for # automated feedback # in R and Python --- background-image: url(images/robot-r.jpeg) background-position: center background-repeat: no-repeat background-size: contain class: middle ## learnr / gradethis --- ## learnr - **learnr** is an R package that makes it easy to create interactive tutorials from R Markdown documents. - Tutorials can include: - Narrative, figures, illustrations, and equations - Code exercises (R code chunks that users can edit and execute directly) - Multiple choice questions - Videos (YouTube, Vimeo) - Interactive Shiny components --- background-image: url(images/learnr-first-look.png) background-position: center background-repeat: no-repeat background-size: contain class: middle --- ## gradethis Companion to the learnr package, **gradethis** provides multiple methods to grade learnr exercises: - `grade_code()`: Grade code against a solution - `grade_conditions()`: Grade all specified conditions - `grade_result()`: Grade result of exercise code --- background-image: url(images/robots-demo.jpeg) background-position: center background-repeat: no-repeat background-size: contain class: middle ### Demo: learnr / gradethis [**`[tutorial]`**](https://minecr.shinyapps.io/nobel-laureates-learnr) [**`[code]`**](https://github.com/mine-cetinkaya-rundel/feed-me-back/blob/master/examples/01-learnr-gradethis/nobel-laureates-learnr.Rmd) --- background-image: url(images/robot-r.jpeg) background-position: center background-repeat: no-repeat background-size: contain class: middle ## R Markdown / GitHub Actions --- ## R Markdown <img src="images/rmarkdown.png" width="80%" style="display: block; margin: auto;" /> --- ## GitHub <img src="images/github-repo.png" width="80%" style="display: block; margin: auto;" /> --- background-image: url(images/github-actions.jpeg) background-position: center background-repeat: no-repeat background-size: contain ## GitHub Actions Cause-and-effect automation tool for orchestrating workflows based on any event (e.g. student pushing their work to repo, deadline passing, etc.) on GitHub. --- ## Workflow - Each assignment is a *private* repository on GitHub created by the instructor with appropriate permissions given to individual students -- - At the beginning of the assignment, instructor places - template R Markdown document for students to fill in and - workflow files that outline when GitHub actions should be run and what they should check - badges to inform students of the status of these checkes -- - Students get feedback as they work on the assignment or after the deadline --- background-image: url(images/robots-demo.jpeg) background-position: center background-repeat: no-repeat background-size: contain class: middle ### Demo: R Markdown / GitHub Actions [**`[repo]`**](https://github.com/ghclass-demo/nobel-laureates) [**`[actions]`**](https://github.com/ghclass-demo/nobel-laureates/tree/master/.github/workflows) --- background-image: url(images/robot-r-python.jpeg) background-position: center background-repeat: no-repeat background-size: contain class: middle ## Jupyter / nbgrader --- ## What is Jupyter? .pull-left[ <img src="images/jupyter-r.png" width="150%" style="display: block; margin: auto;" /> ] -- .pull-right[ <img src="images/r-python.png" width="100%" style="display: block; margin: auto;" /> ] --- ## Tests as automated feedback <img src="images/tests-as-feedback.png" width="75%" style="display: block; margin: auto;" /> --- ## Autograding via nbgrader <img src="images/grading.png" width="75%" style="display: block; margin: auto;" /> --- ## Autograding via nbgrader <img src="images/grades_as_csv.png" width="75%" style="display: block; margin: auto;" /> --- ## Autograding via nbgrader .pull-left[ <img src="images/feedback-forms.png" width="130%" style="display: block; margin: auto;" /> ] -- .pull-right[ <img src="images/feedback-forms2.png" width="130%" style="display: block; margin: auto;" /> ] --- background-image: url(images/robots-demo.jpeg) background-position: center background-repeat: no-repeat background-size: contain class: middle ## Demo: Jupyter + nbgrader with R [**`[R notebook]`**](https://mybinder.org/v2/gh/ttimbers/jupyter-demo/master?filepath=jupyter-nbgrader-r%2Frelease%2Fworksheet_01_r%2Fworksheet_01_r.ipynb) [**`[R code]`**](https://github.com/mine-cetinkaya-rundel/feed-me-back/tree/master/examples/03-jupyter-nbgrader-r) --- background-image: url(images/robots-demo.jpeg) background-position: center background-repeat: no-repeat background-size: contain class: middle ## Demo: Jupyter + nbgrader with Python [**`[Python notebook]`**](https://mybinder.org/v2/gh/ttimbers/jupyter-demo/master?filepath=jupyter-nbgrader-python%2Frelease%2Fworksheet_01_python%2Fworksheet_01_python.ipynb) [**`[Python code]`**](https://github.com/mine-cetinkaya-rundel/feed-me-back/tree/master/examples/04-jupyter-nbgrader-python) --- class: middle, inverse # Closing thoughts --- ## Best practices for automated feedback -- - Measure twice, cut once (verify the correctness of your tests) 📏📏 ✂️ -- - Use rounding & type coercion to write robust tests 💪 -- - Use hashing to hide solutions for visible tests 🕵️ -- - Test your tests on the students compute environment 💻 -- - Abstract your tests to a script that is sourced/imported into the students literate code document 📁 -- - Don't give automated feedback on everything 🙈 -- - Consider peer feedback where automated feedback is not feasible (e.g. interpretation, narrative) but scalability is an issue 👯 --- background-image: url(images/robots-faq.jpeg) background-position: center background-repeat: no-repeat background-size: contain ## FAQ: learnr / gradethis --- .large[ .hand-blue[Q: What is an approachable way to get started?] ] Build a tutorial where students build develop their analysis in exercise code chunks (that are not checked) and only multiple choice questions are used for assessment. [**`[example]`**](https://minecr.shinyapps.io/04-wrangleviz) <img src="images/learnr-simple.png" width="85%" style="display: block; margin: auto;" /> --- .large[ .hand-blue[Q: Is it possible to record student data?] ] Yes, though not trivial, at this point. See [recording events](https://rstudio.github.io/learnr/publishing.html#Recording_Events) section on the learnr documentation and the [**submitter**](https://github.com/dtkaplan/submitr) package by Danny Kaplan. A solution that may be "good enough" for formative exercises: embed a Google/Microsoft/etc. Form at the end and ask students to "sign in" their work. **Tip:** Add a free-text question to this form where students reflect on the exercises they just completed, and you can easily capture and analyze. [**`[example]`**](https://minecr.shinyapps.io/04-wrangleviz/#section-finish-up) <img src="images/learnr-logging.png" width="60%" style="display: block; margin: auto;" /> --- .large[ .hand-blue[Q: I've built simple tutorials already. How do I make the jump to code checking and providing automated feedback that is actually useful?] ] - Replicate `gradethis::gradethis_demo()`, then make incremental changes - Read the [Testing](https://r-pkgs.org/tests.html#test-tests) chapter in R Packages (Wickham and Bryan) - Also read the [Metaprogramming](https://adv-r.hadley.nz/metaprogramming.html) section in Advanced R (Wickham) <img src="images/learnr-learn-more.jpeg" width="75%" style="display: block; margin: auto;" /> --- background-image: url(images/robots-faq.jpeg) background-position: center background-repeat: no-repeat background-size: contain ## FAQ: R Markdown / GitHub Actions --- .large[ .hand-blue[Q: How can I get started with having my students use Git and GitHub to manage and collaborate on their work?] ] - Read the [get started](https://rundel.github.io/ghclass/articles/ghclass.html) guide on the **ghclass** package website - Test out [ghclass](https://rundel.github.io/ghclass/index.html), an R package for managing a GitHub class organisation <img src="images/ghclass.png" width="85%" style="display: block; margin: auto;" /> --- .large[ .hand-blue[Q: Can I see more GitHub actions in action?] ] .pull-left[ We're collecting examples of GitHub actions useful for checking data science assignments at [github.com/rundel/ghclass-actions](https://github.com/rundel/ghclass-actions). ] .pull-right[ <img src="images/ghclass-actions.png" style="display: block; margin: auto;" /> ] --- ## Resources - learnr / gradethis: - Package documentation: [learnr](https://rstudio.github.io/learnr), [gradethis](https://rstudio-education.github.io/gradethis/) - CAUSE Webinar: [Interactive R tutorials with learnr](https://www.causeweb.org/cause/webinar/teaching/2018-05) (May 2018) - [Source code for RStudio Primers](https://github.com/rstudio-education/primers) - learnr for remote teaching: [Blog post](https://education.rstudio.com/blog/2020/05/learnr-for-remote/) + [sample tutorial](https://allisonhorst.shinyapps.io/missingexplorer/) featuring custom styling by Allison Horst - R Markdown / GitHub actions: - Still not using R Markdown in your courses? [R Markdown: Integrating A Reproducible Analysis Tool into Introductory Statistics](https://escholarship.org/uc/item/90b2f5xh) (Baumer et al., 2014) - Need Git convincing? [Infrastructure and Tools for Teaching Computing Throughout the Statistical Curriculum](https://www.tandfonline.com/doi/abs/10.1080/00031305.2017.1397549) (Çetinkaya-Rundel and Rundel, 2017) and [Implementing version control with Git as a learning objective in statistics courses](https://arxiv.org/abs/2001.01988) (Beckman et al., 2020) - ghclass resources: [package documentation](https://rundel.github.io/ghclass/index.html), [sample actions](https://github.com/rundel/ghclass-actions) - Jupyter / nbgrader: - [nbgrader documentation](https://nbgrader.readthedocs.io/en/stable/) - [Examples of R Jupyter notebooks with automated feedback from UBC's DSCI 100 course](https://github.com/UBC-DSCI/dsci-100-assets/tree/master/2019-spring/materials) - [Examples of Python Jupyter notebooks with automated feedback from Berkley's data8 course](https://github.com/data-8/data8assets/tree/gh-pages/materials/su17/lab) --- background-image: url(images/robots-last.jpeg) background-position: center background-repeat: no-repeat background-size: contain .pull-left[ ### Mine Çetinkaya-Rundel .large[ <svg style="height:0.8em;top:.04em;position:relative;fill:#0F4C81;" viewBox="0 0 512 512"><path d="M502.3 190.8c3.9-3.1 9.7-.2 9.7 4.7V400c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V195.6c0-5 5.7-7.8 9.7-4.7 22.4 17.4 52.1 39.5 154.1 113.6 21.1 15.4 56.7 47.8 92.2 47.6 35.7.3 72-32.8 92.3-47.6 102-74.1 131.6-96.3 154-113.7zM256 320c23.2.4 56.6-29.2 73.4-41.4 132.7-96.3 142.8-104.7 173.4-128.7 5.8-4.5 9.2-11.5 9.2-18.9v-19c0-26.5-21.5-48-48-48H48C21.5 64 0 85.5 0 112v19c0 7.4 3.4 14.3 9.2 18.9 30.6 23.9 40.7 32.4 173.4 128.7 16.8 12.2 50.2 41.8 73.4 41.4z"/></svg> cetinkaya.mine@gmail.com <svg style="height:0.8em;top:.04em;position:relative;fill:#0F4C81;" viewBox="0 0 512 512"><path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"/></svg> minebocek <svg style="height:0.8em;top:.04em;position:relative;fill:#0F4C81;" viewBox="0 0 496 512"><path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z"/></svg> mine-cetinkaya-rundel ] ] .pull-right[ ## Tiffany Timbers .large[ <svg style="height:0.8em;top:.04em;position:relative;fill:#0F4C81;" viewBox="0 0 512 512"><path d="M502.3 190.8c3.9-3.1 9.7-.2 9.7 4.7V400c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V195.6c0-5 5.7-7.8 9.7-4.7 22.4 17.4 52.1 39.5 154.1 113.6 21.1 15.4 56.7 47.8 92.2 47.6 35.7.3 72-32.8 92.3-47.6 102-74.1 131.6-96.3 154-113.7zM256 320c23.2.4 56.6-29.2 73.4-41.4 132.7-96.3 142.8-104.7 173.4-128.7 5.8-4.5 9.2-11.5 9.2-18.9v-19c0-26.5-21.5-48-48-48H48C21.5 64 0 85.5 0 112v19c0 7.4 3.4 14.3 9.2 18.9 30.6 23.9 40.7 32.4 173.4 128.7 16.8 12.2 50.2 41.8 73.4 41.4z"/></svg> tiffany.timbers@stat.ubc.ca <svg style="height:0.8em;top:.04em;position:relative;fill:#0F4C81;" viewBox="0 0 512 512"><path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"/></svg> TiffanyTimbers <svg style="height:0.8em;top:.04em;position:relative;fill:#0F4C81;" viewBox="0 0 496 512"><path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z"/></svg> ttimbers ] ] <br><br> .center[ ## 🔗 [bit.ly/feed-me-back](https://bit.ly/feed-me-back) ]