{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Your first message classifier\n", "\n", "**Artificial Post \u00b7 20\u201330 minutes \u00b7 basic Python**\n", "\n", "By the end, you will fit a tiny classifier, compare it with a baseline, and explain one error on messages kept out of development.\n", "\n", "Run the cells in order in an existing Jupyter environment with Python 3. The code uses only Python's standard library: no pip installs, API keys, network calls, or dataset downloads. Jupyter itself is a separate prerequisite. All messages are fictional. The example teaches evaluation; it is not a production spam filter.\n", "\n", "Our task: predict **spam** or **ordinary** from a message. Both kinds of mistake count equally in the headline accuracy. We also count them separately so you can see what accuracy hides.\n", "\n", "## 1. Keep the jobs separate\n", "\n", "Training data fits the classifier. Validation data helps choose a setting. Test data appears later, after that choice is fixed. All sets are tiny and balanced; their scores cannot establish real-world reliability.\n" ], "id": "message-classifier-1" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from collections import Counter\n", "import math\n", "import re\n", "\n", "TRAIN = [\n", " (\"claim your free cash prize\", \"spam\"),\n", " (\"win cash now click to claim\", \"spam\"),\n", " (\"limited offer buy cheap tickets\", \"spam\"),\n", " (\"exclusive prize claim today\", \"spam\"),\n", " (\"click for a free bonus offer\", \"spam\"),\n", " (\"buy now huge discount sale\", \"spam\"),\n", " (\"urgent cash reward click now\", \"spam\"),\n", " (\"free prize winner claim reward\", \"spam\"),\n", " (\"sale ends today buy cheap\", \"spam\"),\n", " (\"special discount click for tickets\", \"spam\"),\n", " (\"are we meeting for lunch today\", \"ordinary\"),\n", " (\"please bring your notes to class\", \"ordinary\"),\n", " (\"the library meeting starts tomorrow\", \"ordinary\"),\n", " (\"can you review the project notes\", \"ordinary\"),\n", " (\"thanks for dinner see you tomorrow\", \"ordinary\"),\n", " (\"please send the meeting agenda\", \"ordinary\"),\n", " (\"your book is at the library\", \"ordinary\"),\n", " (\"let us plan lunch after class\", \"ordinary\"),\n", " (\"the project review is on friday\", \"ordinary\"),\n", " (\"can we discuss the book tomorrow\", \"ordinary\"),\n", "]\n", "VALIDATION = [\n", " (\"claim a cash bonus today\", \"spam\"),\n", " (\"click now for a cheap prize\", \"spam\"),\n", " (\"exclusive sale buy tickets\", \"spam\"),\n", " (\"win your free reward\", \"spam\"),\n", " (\"please review tomorrow meeting notes\", \"ordinary\"),\n", " (\"see you for lunch at the library\", \"ordinary\"),\n", " (\"are you free for our project meeting\", \"ordinary\"),\n", " (\"thanks for the book and dinner\", \"ordinary\"),\n", "]\n", "LABELS = (\"ordinary\", \"spam\")\n", "\n", "def tokens(text):\n", " # A deliberately simple representation: lowercase English word counts.\n", " return re.findall(r\"[a-z]+\", text.lower())\n", "\n", "assert not ({text for text, _ in TRAIN} & {text for text, _ in VALIDATION})\n", "print(\"Training:\", Counter(label for _, label in TRAIN))\n", "print(\"Validation:\", Counter(label for _, label in VALIDATION))\n" ], "id": "message-classifier-2" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Fit a small model\n", "\n", "Multinomial Naive Bayes counts words within each label. It combines a label's frequency with the likelihood of each word under that label. The simplifying assumption is that word occurrences can be treated independently given the label; this ignores word order and much of meaning.\n", "\n", "`alpha` adds a small count to every vocabulary word so an unseen word within a label does not give a zero probability. Words entirely outside the training vocabulary are ignored. Scores are sums of logarithms, not calibrated confidence percentages.\n" ], "id": "message-classifier-3" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def fit(rows, alpha=1.0):\n", " if alpha <= 0 or not rows:\n", " raise ValueError(\"Use non-empty training data and positive alpha\")\n", " counts = {label: Counter() for label in LABELS}\n", " label_counts = Counter()\n", " vocabulary = set()\n", " for text, label in rows:\n", " if label not in LABELS:\n", " raise ValueError(\"Unknown label: \" + label)\n", " words = tokens(text)\n", " counts[label].update(words)\n", " vocabulary.update(words)\n", " label_counts[label] += 1\n", " if not vocabulary or any(label_counts[label] == 0 for label in LABELS):\n", " raise ValueError(\"Both labels and some words are needed\")\n", " return {\"counts\": counts, \"labels\": label_counts, \"vocabulary\": vocabulary,\n", " \"alpha\": alpha, \"size\": len(rows)}\n", "\n", "def predict(model, text):\n", " scores = {}\n", " for label in LABELS:\n", " score = math.log(model[\"labels\"][label] / model[\"size\"])\n", " denominator = sum(model[\"counts\"][label].values()) + model[\"alpha\"] * len(model[\"vocabulary\"])\n", " for word in tokens(text):\n", " if word in model[\"vocabulary\"]:\n", " score += math.log((model[\"counts\"][label][word] + model[\"alpha\"]) / denominator)\n", " scores[label] = score\n", " # A tie uses the first label, ordinary. State this rule instead of hiding it.\n", " return max(LABELS, key=lambda label: scores[label])\n", "\n", "def evaluate(model, rows):\n", " predictions = [(text, label, predict(model, text)) for text, label in rows]\n", " correct = sum(actual == guess for _, actual, guess in predictions)\n", " errors = [(text, actual, guess) for text, actual, guess in predictions if actual != guess]\n", " return {\"correct\": correct, \"total\": len(rows), \"accuracy\": correct / len(rows), \"errors\": errors}\n", "\n", "model = fit(TRAIN)\n", "print(\"Vocabulary size:\", len(model[\"vocabulary\"]))\n", "print(\"Prediction:\", predict(model, \"claim a cash prize\"))\n" ], "id": "message-classifier-4" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Compare on validation data\n", "\n", "The baseline always predicts the most common training label (ordinary wins a tie). It does not learn word patterns. A useful comparison asks whether our model improves on that baseline.\n", "\n", "Try the three settings below **using validation only**. We choose the first setting in a tie. Do not choose a model using the final test results.\n" ], "id": "message-classifier-5" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "baseline_label = max(LABELS, key=lambda label: model[\"labels\"][label])\n", "baseline_correct = sum(label == baseline_label for _, label in VALIDATION)\n", "print(\"Validation baseline:\", baseline_correct, \"of\", len(VALIDATION))\n", "\n", "candidates = []\n", "for alpha in (1.0, 0.25, 4.0):\n", " candidate = fit(TRAIN, alpha)\n", " result = evaluate(candidate, VALIDATION)\n", " candidates.append((candidate, result))\n", " print(\"alpha =\", alpha, \"validation:\", result[\"correct\"], \"of\", result[\"total\"])\n", "\n", "FINAL_MODEL, validation_result = max(candidates, key=lambda pair: pair[1][\"accuracy\"])\n", "print(\"Chosen alpha:\", FINAL_MODEL[\"alpha\"])\n", "print(\"Validation errors:\", validation_result[\"errors\"])\n" ], "id": "message-classifier-6" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Pause before the final test\n", "\n", "Predict first, then write two sentences in a new cell or your notes:\n", "\n", "1. Why would getting every training message right be insufficient?\n", "2. Would the word **free** always mean spam? Think of an ordinary message containing it.\n", "\n", "Check your reasoning: a model can memorize familiar examples; an ordinary invitation can contain \u201cfree.\u201d A single word does not determine intent. Your next task is to inspect where this simple model breaks.\n", "\n", "## 4. Evaluate once on the held-out test\n", "\n", "Freeze the model choice before running the next cell. The labels below come from the fictional sender's intended use. Some ordinary messages deliberately contain promotional words to expose the model's limits.\n" ], "id": "message-classifier-7" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "TEST = [\n", " (\"cash prize click to claim today\", \"spam\"),\n", " (\"buy a special discount ticket now\", \"spam\"),\n", " (\"free reward exclusive bonus\", \"spam\"),\n", " (\"cheap sale claim your offer\", \"spam\"),\n", " (\"winner click now to buy\", \"spam\"),\n", " (\"the class project review is tomorrow\", \"ordinary\"),\n", " (\"please bring the library book to lunch\", \"ordinary\"),\n", " (\"thanks for sending the meeting agenda\", \"ordinary\"),\n", " (\"I have free tickets for you\", \"ordinary\"),\n", " (\"can you buy dinner today\", \"ordinary\"),\n", "]\n", "assert not ({text for text, _ in TEST} & {text for text, _ in TRAIN + VALIDATION})\n", "result = evaluate(FINAL_MODEL, TEST)\n", "print(\"Final toy test:\", result[\"correct\"], \"of\", result[\"total\"])\n", "false_spam = sum(actual == \"ordinary\" and guess == \"spam\" for _, actual, guess in result[\"errors\"])\n", "missed_spam = sum(actual == \"spam\" and guess == \"ordinary\" for _, actual, guess in result[\"errors\"])\n", "print(\"Ordinary messages marked spam:\", false_spam)\n", "print(\"Spam messages missed:\", missed_spam)\n", "for text, actual, guess in result[\"errors\"]:\n", " print(repr(text), \"expected:\", actual, \"predicted:\", guess)\n" ], "id": "message-classifier-8" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Your independent variation\n", "\n", "Write **eight fresh fictional messages**, four for each label, including at least one ordinary message containing a promotional word. Label them before inspecting predictions. Do not reuse the test messages above.\n", "\n", "1. Predict where the model will fail before running it.\n", "2. Use `evaluate(FINAL_MODEL, YOUR_MESSAGES)` without fitting on those messages.\n", "3. Explain one error using the representation: word counts, no word order, and ignored unknown words.\n", "4. If you revise training data or settings after seeing these errors, treat this set as development data. Get another untouched set for a final estimate.\n", "\n", "An independent check: if you change the model until it gets all ten test messages above right, can you still call that score an independent test? **No.** Test feedback is now influencing development.\n", "\n", "## Keep your claim proportional\n", "\n", "Record the training/validation/test sizes, chosen alpha, correct/total, the two error counts, and one failure example. A score on ten fictional messages is a demonstration, not an estimate for a real inbox. Which error should cost more in a real product is a separate decision; accuracy alone cannot answer it.\n", "\n", "If a cell says `NameError`, restart the kernel and run the cells from the top. No network call is needed. If you change the representation, rerun the fitting cells so the model and vocabulary match.\n", "\n", "Save your notebook with your predictions and explanation. Artificial Post does not receive the notebook or automatically mark a lesson complete.\n", "\n", "### Algorithm references\n", "\n", "- [scikit-learn: Multinomial Naive Bayes](https://scikit-learn.org/stable/modules/naive_bayes.html#multinomial-naive-bayes)\n", "- [Google: Training, validation, and test sets](https://developers.google.com/machine-learning/crash-course/overfitting/dividing-datasets)\n", "\n", "The code and fictional data in this notebook are original Artificial Post teaching examples.\n" ], "id": "message-classifier-9" } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.10" } }, "nbformat": 4, "nbformat_minor": 5 }