Thursday, September 13, 2012

The Math of Khan Part 1 - Going through Khan Academy's Math program in an hour a day

MOOC - Massively Open Online Courses is the acronym of Summer 2012.  As someone who works in an educational institution it's rare for a day to go by without someone talking about distance learning of some kind and specifically the "free" kind.  Whether it's "passive" systems like MIT's Open Courseware and iTunesU or "active" systems like Udacity, edX, Coursera and Khan Academy.  What this means is almost every day someone wants to tell you their implicit presuppositions about education through a comment on how MOOC is great, horrible, the future of education and/or it's demise.

I admit some of the courses certainly look appealing but as soon as I start think about taking one of them, the voices start.  Not the ones that tell me where to hide the bodies of vendors who make mathematical errors so horrible they border on insulting (perhaps that's the subject for another post).  No, the first voice to speak up is my technical criticism "These things are mostly video aren't they?  Don't you already loathe how video is massively overused as a delivery medium?" This is, in fact true (perhaps also a topic for another post there) I can think of nothing more dull than having to sit through a series of video lectures.

Right after that my "Shouldn't you be doing something more productive" voice speaks up and reminds me of how I, at this moment have about 79,642 other endeavors currently on the go.  Any of which is probably a better use of my time than taking an interest course.  This usually talks me out of it and from there I get back to surfing the web which for reasons unexplained both voices consider a neutral activity.

Delta, a part-time math teacher over at Angry Math recently reviewed Udacity's statistics 101 course and found it to be somewhere between bad and awful.  Some of these criticisms are squarely leveled at whomever developed the course materials. Others might be more generally applied to the medium as a whole.  Delta's critique started me wondering if there was a way to sample material from a MOOC without taking a whole course and wading through hours of *shudder* video lectures.

Then it struck me, what if I just took the exams and tests?  While this wouldn't help evaluate the teaching material but it *might* be a reasonable proxy for evaluating the skills we can expect students to gain from the course.  From there it was an easy leap to Khan Academy.  Khan Academy has a lot of material aimed at the Elementary to High-School curriculum (which means it should be easy) but it's also known for having a pretty sophisticated testing engine and methodology.

How testing works at Khan:

As far as I can see there is no document describing their testing methodology in detail but here's what I've gathered from my first few hours on their site.  Mathematics is broken up into a series of topics which are displayed in a "knowledge map" giving students an idea how the topics are interrelated.

For example addition and subtraction is shown as a prerequisite for multiplication.  Each topic is broken up into a series of skills which represent individual elements to be tested on. To give you an idea I've reproduced the list of skills for the Addition and Subtraction topic below:

  1. 1 Digit Addition
  2. Number Line 1
  3. Representing Numbers
  4. 2 Digit Addition
  5. 1 Digit Subtraction
  6. Number Line 2
  7. 2 and 3 Digit Subtraction
  8. Number Line 3
  9. Subtraction with Borrowing
  10. Addition with Carrying 
  11. Ordering Negative Numbers
  12. Addition and Subtraction Word Problems
  13. 4 digit Addition with Carrying
  14. 4 digit Subtraction with Borrowing
  15. Adding Negative Numbers
  16. Adding and Subtracting Negative Numbers
  17. Negative Number Word Problems

Each of these skills has at least one kind of test question associated with it.  When a student chooses to practice a topic they are given a "stack" - a series of questions which test their knowledge on some subset of the skills in the topic.  Primarily this will contain topics that the student has yet to reach "mastery" in (or has started to show some need to review) however it appears that there is also some minimum number of topics.  So as you master the majority of the topic skills you will still be tested on some topics which you have mastered.  Alternatively you can choose to focus on practicing one specific skill.

Mastery, appears to occur when the student has collected enough "leaves" in a particular topic.  Up to three of these "leaves" are awarded for answering a question on a particular topic correctly.  The scoring system appears to go like this:  Three leaves are awarded for answering the question correctly on the first attempt.  If they answer incorrectly they may continue to attempt the same question.  If they succeed two leaves are awarded.  I'm not sure what grants you only one leaf but there is a button on each question page where a student can request a "hint".  Presumably that drops your score.

Anyway, at the end of the "stack" the student's leaves are tallied and their progress in each topic is displayed.  It's worth noting that if students have a lot of trouble with a particular skill they can *lose* mastery of it.  In which case the skill shows up as "needing review".  According to the Khan documentation this can also happen over a long period of inactivity.

How I will be blogging about this:

Starting from the "Telling Time" topic I will, for a full hour each workday do nothing but consecutive testing until I achieve mastery in a topic.  Once I do I will move on to another topic.

I will, unless specifically instructed or provided an opportunity by the Khan software be doing these calculations in my head or on paper.

From there I'll post some highlights about what is needed to pass these tests and some of my limited commentary of their pedagogy and perhaps some contrasts with how I teach (or might teach) math to my kids at home.  I'll try to give a "thumbs up", "neutral" or "thumbs down" score in the following two areas:

Accuracy: I'll be asking myself on one hand if the material is ambiguous enough to cause confusion with a related concept.  For example Delta at angry math correctly noted that the Udacity course didn't clearly differentiate between the symbols for population mean and the sample mean.  Which is a huge thing.  The other side of that coin will be asking if the student is being taught an overly strict definition that may cause confusion later on.   Taking into account how far away the related concept is in the students education.  e.g. Teaching someone that they can't take the square root of -1 if they're five years away from learning about complex numbers will be looked at differently than if they're going to learn that in the next topic.

Practicality: This is where I ask "Is the concept being taught applicable to a real-world situation?".  By "real world" I may mean "in a particular career".  For example I went through Junior High in the aftermath of the New Math, which meant I was taught "Thou shalt know how to do math in an arbitrary radix (sometimes called "base") notation".  e.g. 0F + 01 = 10 in base 16 (or radix 16). Knowing how to count in and convert between base 16, base 10, base 8, and base 2 are still fairly important to a career in computer programming. So I'd consider those useful skills.  However I can remember a couple of classes where we spent time on arbitrary bases like base 12 or doing inter-base multiplication (e.g. What is 423 in base 6 multiplied by 12 in base 3) which are somewhere between uncommon and non-existent in computing today.

Hour One:  Telling time.

While this isn't strictly a math skill it is something commonly taught as math to elementary school students.  Khan's tests consist of exactly two kinds of question. i) Here is a clock, type the time in a box

 and ii) Here is a clock, set it to this time.
It's hard to come up with much in the way of criticism here but if I were to choose something that was perhaps a little interesting was how the writer of the courseware focused on the hour hand moving proportionally to the minute hand.  So in the type ii) questions they demand you set the hour hand in pretty close to exact proportion.  So the following answer would be considered wrong:
The proportional movement of the hour hand *is* an accurate description of how the vast majority of chronographs work but it's worth noting that there are some notable exceptions.  Some tower clocks and high-end multi-time zone watches use a "stepping hour-hand" mechanism where the hour hand jumps from one hour-mark to the next (Not to be confused with a "jumping hour" mechanism where the hour hand is replaced by a dial which displays the hour).  So thinking about my criteria, is this a practical skill?  Absolutely?  Is the topic handling too loose? Nope.  Too strict?  Only in the mildest possible way. It's clearly important for a student to be able to read clocks where the hour hand moves proportionally to the minute hand as this is the predominant form of timepiece. However this "feature" is simply an artifact of the mechanism the designers used to move the hands.  While it doesn't seem likely, if this *were* to somehow teach students that the configuration of hands on a multi-time zone watch are incorrect that would be a bad outcome.  As I said I doubt that happens much so I'd still give this unit:


Still Hour One: Addition and Subtraction.

It's worth noting that it became clear working through this unit that Khan's testing software neither employs purely static set of questions - that is you get different questions each time you ask for a "stack".  More interestingly it also doesn't appear to be simply drawing a random question from a large pool of static questions.  What it actually appears to be happening is the question is derived from a template.  Where the basic form of the question e.g. "Sally has ___ apples and eats ___. How many apples does sally have left?" is defined and the blanks are filled in by a randomly picked number.  The reason I think this is because some of the word problems involve situations one would not readily think about like sledding at -40 C (or F :-) ).  I suppose it's also possible that the questions are static but generated from a template.  The result in either case is pretty much the same.  It's exceedingly rare in some topics to get the same question twice.  Which is nice when compared to a lot of online courseware or even textbook test banks.  It will be interesting to see how this holds up with more complicated subjects. i.e. When you do integration by hand your professor has hand picked the functions you are integrating because not all are integratable by the same technique.

There's little to complain or laud here.  "Ordering negative numbers" seemed like something of an odd duck compared to the other "skills" but I suppose negative numbers might be counter-intuitive for someone who is just learning.  Doing four digit addition and subtraction was a little tedious and IMHO doesn't teach you anything you didn't learn with two and three digits.  That said...


Not Yet Hour Two: Absolute Value.

The main thing to praise here is that the topic is short.  I was taught absolute value in high-school and I even recall having a TA go over the notation in my first Calculus tutorial.  The idea that anyone would need much more of a lesson than "throw the minus sign away" is mystifying.  Anyway one unit, about 15 test questions.  Finished in under eight minutes.



So that wraps up our first day:  Insofar as Addition, Subtraction, Absolute Value and Telling Time the Khan Academy doesn't appear to be doing anything crazy.   In case you're interesting in exactly how much work it takes to achieve "mastery" in these subjects.  I took a look at my "Skill Progress" page and found out that I had answered just under 400 questions - that's about one every 9 seconds!  Our next unit is multiplication and division which I expect to take more than a single hour since it involves multi-digit multiplication which I will have to do by hand.  *sigh*

No comments:

Post a Comment