Build Your First Machine Learning Model From Scratch: Decision Tree vs. K-Nearest Neighbors
Data Science Journey 2026: Step 6
We have already learned Python, SQL, maths: linear algebra, matrices, coordinate geometry, calculus, probability and statistics. And we have seen your data through Exploratory Data Analysis.
We looked at the Iris dataset from every angle: bar plots, box plots, histograms, scatter plots, a pair plot, a correlation heatmap and come to conclusion that “Petal Length and Petal Width” were the two features that separated the three Iris species almost perfectly.
Today, that insight finally pays off. We’re going to take it and build a step by step real machine learning model, trained it, evaluate it to predict on test data. We’ll train our data on two machine learning models: a Decision Tree and a K-Nearest Neighbors classifier, so you can see that the three-step scikit-learn ritual (create, fit, predict) works the same way no matter which model you reach for.
By the end of this post, you’ll have trained your first two classifiers, and you’ll understand what they’re actually doing, not just which functions to call.
What Is a Machine Learning Model?
In Step 5, when we looked at the box plot of Petal Length by species, we could already tell that if Petal Length is under 2 cm, it’s almost certainly Setosa. We did that with our eyes with the help of graphs.
A machine learning model does the exact same thing, except it finds that rule automatically, from data, across four features at once instead of just one, and across millions of rows instead of just 150.
That’s the entire secret of machine learning. It isn’t magic, and it isn’t a black box. It’s pattern-finding, the same pattern-finding we were doing by hand in EDA. ML is just handed over to an algorithm so it can do it faster and at a scale you never could by hand.
A model has one job: look at inputs called features and learn a rule that predicts an output called the label or target. Today, our features are the four flower measurements: Petal Length, Petal Width, Sepal Length and Sepal Width. Our target is the ‘Species’.
