FUQed: False Positives, False Negatives, and Authentic Assessment

FuQed: False Positives, False Negatives, and Authentic Assessment¶

False positives and false negatives¶

Last year I asked an econometrics class for a write-up on the basic meaning of causal inference. At that point we had only covered simple regression with dummy variables. The answer I wanted was narrow. When treatment is randomly assigned, compare the treated group with the comparison group. That comparison gives the average treatment effect.

What came back were fluent pages on compliers, defiers, instrumental variables, difference-in-differences, and synthetic control. I had put those words in white type on a white slide background. They never appeared on the screen in class.

A false positive is accusing a student of cheating when that student actually learned the material and can explain it. A false negative is treating the paper as proof of learning when the student cannot state the basic point. In that case the essay came from hidden slide text plus a chatbot.

Either way, this situation is FuQed.

Joshua Angrist won the 2021 Nobel Prize in Economic Sciences. In his book Mostly Harmless Econometrics, he defines FuQed as a Fundamentally Unidentified Question.^[1]

Broccoli and brownies¶

John Paul Rollert’s Atlantic essay argues that this habit can harden into a society of cheats.^[2] Many universities answer by telling students to use AI responsibly. Rebecca Winthrop says that is not enough. We need responsible products.

She puts it as leaving a six-year-old at the dining table with broccoli and brownies. You say do not eat the brownies. Then you walk away for eight hours. The child eats the brownies.

Gary Becker’s economics of crime says the same thing in another language. When the benefit of the easy action outweighs the cost, people can be a rational criminal. Teaching willpower is necessary. It is not sufficient.

Our job is to build a bridge that can take harsh weather and bad drivers. It is not a bridge designed only for a good day.

Not the job market of 2022¶

We should not train students for the job market of 2022. The price of code is falling. Jevons’ paradox says cheaper tools invite more use, not less. When computers arrived, accountants had to learn the machine. The same pressure now sits on every major.

Andrew Ng’s line is that a marketing student should leave as a marketing engineer. A supply chain student should leave as a supply chain engineer. That person can see an end-to-end process and ship a working product, not only a dashboard.

Domain knowledge still comes first. Without it, a student cannot tell when a model is making things up. That student also cannot do systems thinking about where a piece of content belongs. Faculty have that hard-earned expertise. Most students do not, not yet.

Our job as an teachers is still to validate, authenticate, and credential. The labor market pays for people who can test an idea, connect it, and finish a live problem.

Teaching as research¶

Josh Hall, Milan Puskar Dean at West Virginia University, was one of my professors during my Ph.D. He used to tell me to make teaching into research. I have been tinkering with my teaching ever since. I document what I try. I write papers from it.

This semester I dropped homework, midterms, finals, and the course book as the spine of the class. I am teaching the absolute theoretical minimum. Lev Landau used that standard in physics. Leonard Susskind’s Theoretical Minimum lectures are on YouTube. That is the version I could follow with no physics background.

Statistics¶

In statistics I teach the sample mean, the sample standard deviation, the t-statistic, and simple linear regression. I do not teach the z-statistic. In applied work we do not know the population standard deviation. Hypothesis tests, p-values, and confidence intervals all use the t-statistic.

A two-sample t-test is a simple linear regression with a dummy variable. In simple linear regression, the F-statistic is the square of the t-statistic. A more detailed F-statistic can be taught in multiple linear regression.

Econometrics¶

In econometrics we start with the basics of R programming language. Then we go straight to data frames. I teach data management with a short set of seven verbs. Those verbs are select, filter, mutate, group by, summarize, pivot longer, pivot wider, and join. For graphs I teach a trend line, a density plot, and a county-level and state-level map of the United States.

We then move to linear regression. We spend most of the remaining time on its properties, its assumptions, and its interpretation. I teach the for-loop. With that tool they can run Monte Carlo simulations, bootstrapping, and randomization inference.

We then spend a long stretch on the philosophy of causal inference. Next come directed acyclic graphs, with confounders and colliders. Then I teach tnstrumental variables, difference-in-differences, and two-way fixed-effects models.

Oral exams¶

Every two or three lectures, students defend these minimum out loud. Both courses require oral and written examinations. I built a scalable online oral-exam system called, “My Merit Guide”.^[3]

Professors upload their course material. The system questions each student on that material within a set time window. It grades the oral responsed based on the professor’s materials only. It flags cases that may raise an integrity concern. Faculty can read the transcript and override a grade.

In my class the window is 45 to 60 seconds. The questions come from my own reading packet. Students cannot wander off the assigned pages. Later we sit down face to face to do actual oral examination.

We also run a peer-review loop. A student writes a narrative. Two or three peers respond. The author revises.

Then the system questions them on their own pages. A chatbot can help with the draft. It cannot sit in for the defense.

Such online scalable oral examination platform does not replace the professor. It points to where attention is needed in a large or online section. It takes the logistics of oral exams off the professor’s desk. The job is teaching, not policing.

Agentic AI after the minimum¶

Once student know the absolute minimums, we shift gear to use agentic AI. In statistics I have them bring their CV. They load it into an agentic system. Then they learn hypothesis testing against their own work history.

A student who works in a daycare asked when arrivals spike. They also asked how an app should behave. A student who works as a barista asked about wait times and how to deliver coupons for customer retention. Here the learning transforms from standardization to personalization.

In time-series forecasting, standard undergraduates are taught up to ARIMA. We move past that. My students go to Nobel Prize winner Harry Markowitz’s portfolio optimization math in a handful of R lines with for-loops. Then they go to Black-Scholes and to calibration for risk management.

Last year the final was a mathematical take on Scott Galloway’s The Algebra of Wealth. Pull a long S&P 500 history. Draw random 30-year spans.

Invest a dollar every year recursively. Run a Monte Carlo. Then tell me by which years you are unlikely to lose principal.

There is no textbook chapter for that exact question. The floor underneath it is compound interest and a for-loop.

The line between using an AI based design tool in powerpoint slides and using a AI to write a paper is fuzzy. I am less interested in policing that line. I am more interested in whether we send out graduates who confuse familiarity with understanding.

What’s next¶

The AI will affect jobs, no doubt. Some jobs will be lost, and that pain will be real. But economists call this creative destruction.

Nobel Laureate Edmund Phelps writes about creative destruction in Mass Flourishing. He says creative destruction brings dynamism in capitalist institutions. That dynamism leads ordinary people to put new talent to work. AI will churn jobs. The live question is whether students use that churn to grow a new skill, or only to finish the assignment.

No model replaces the person who can find the right people. That person can ask the right question. They stay pleasantly persistent to find the right answer and get the job done.

Our 2028 graduate should not be the student who picked a lane and waited. That student should take a problem apart. They should build end to end process with code. They should defend what they built. They should be able to finish the work with other people and machines.