In this tutorial, we will cover subunits of Unit 3: Evaluating Models: 3.1 Importance of Model Evaluation, Train-Test Split, and Accuracy and Error of the AI-417 course for Class 10.
In Unit 2, we learned how to build AI models, rule-based, learning-based, and even how a neural network’s perceptron makes a decision. Now that we know how to build a model. Now the question arises:
How do we know if the model we built is any good?
Our this article answers this question.
Model Evaluation
Model evaluation is the process of using different evaluation metrics to understand how well a machine learning model performs.
Importance of Model Evaluation
An AI model gets better with constructive feedback, just like the report card you receive at school. Your report card has grades, percentages, percentiles, and ranks, which together tell you exactly where you’re doing well and where you need to improve. Model evaluation does the same thing for an AI model. It tells us the model’s strengths and weaknesses so we can improve it further.
The Model Evaluation Feedback Loop

This feedback loop matters because it’s the only way to know whether a model is actually suitable for the task it’s meant to solve, and it guides us on exactly which part of the model needs improvement.
Train-Test Split
Now, we know why evaluation matters, the next step is figuring out how to actually evaluate a model. This is where the train-test split comes in.
Train-test split is a technique for evaluating the performance of a machine learning algorithm. It can be used for any supervised learning algorithm and works best when there is a sufficiently large dataset available. The procedure involves taking a dataset and dividing it into two subsets:
- Training Dataset: Used to make the model learn, this is the data the model is fitted on.
- Testing Dataset: The input elements of this dataset are given to the trained model, which makes predictions on them. These predictions are then compared against the expected, correct values to judge performance.
The objective is to estimate how the model is likely to perform in general, specifically, on new data that was not used to train it, and for which we don’t already know the correct answer.
The Overfitting Warning- Very Important
It is not recommended to use the same data you used to build the model to also evaluate it. If you do, the model already knows the correct answers for every example in that dataset, so it will appear to perform perfectly even if it would actually fail on new, unseen data. This is known as “overfitting”, and it’s exactly why the training and testing datasets must be kept separate.
Accuracy and Error
Let’s understand this with “The Bob and Billy Story”:
Bob and Billy went to a concert together. Before going, Bob guessed the entry fee would be Rs 300, and Billy guessed it would be Rs 550. When they arrived, they found out the actual entry fee was Rs 500 per person.
Discussion Question: Who was more accurate in their estimate: Bob or Billy? How much error did each of them make?
The Answer:
- Bob’s error = |Actual − Predicted| = |500 − 300| = Rs 200
- Billy’s error = |Actual − Predicted| = |500 − 550| = Rs 50
Since Billy’s error is smaller, Billy was more accurate in his estimate. This little story captures the core relationship in model evaluation:
The smaller the error, the higher the accuracy.
What is Accuracy?
Accuracy is an evaluation metric that measures the total number of predictions a model gets right. Accuracy and performance are directly proportional to each other, a better-performing model produces more accurate predictions.
What is Error?
Error can be described as an action that is inaccurate or wrong. In the context of a machine learning model, error tells us how far off a model’s prediction is from the actual outcome. Based on the error, we can decide which machine learning model performs best for a given dataset.
Example: Disease Prediction
Consider a model built to predict whether a person has a certain disease.
If the model predicts a person does not have the disease when they actually do, that is an error. It quantifies how far off the prediction is from the real outcome.
If the model correctly predicts disease or no-disease across a given set of cases, it achieves 100% accuracy for that data set.
Key Points
- The goal of any AI model is to minimize error and maximize accuracy.
- Real-world data can be messy and unpredictable, even well-built models make mistakes.
- Accuracy alone isn’t always the best measure. In medical diagnosis, for instance, a model with a slightly lower overall accuracy but a strong focus on not missing an actual disease case might be the more responsible choice.
- The right metric to focus on, accuracy, error, or something else is always depends on the specific task and its real-world consequences.
Test Yourself
- Why is model evaluation compared to a report card?
- What is the purpose of splitting a dataset into training and testing subsets?
- What is overfitting, and why does testing a model on its own training data cause it?
- In the Bob and Billy story, why is a smaller error associated with higher accuracy?
- Why might a model with very high overall accuracy still be a poor choice for a task like medical diagnosis?
- Can you think of a real-world scenario where a small error could still have serious consequences, even if the model’s accuracy looks high on average?
Stay Tuned!!
Stay tuned for next part of this chapter, where we cover Classification Metrics: Confusion Matrix, Precision, Recall and F1 Score in detail.
Click below for the YouTube video for Part 1 of Unit 3:
For any queries and clarification, kindly email us at datasciencehorizon@gmail.com
Keep learning and keep implementing!!


