Evaluating a CNN Model Like a Pro
There's much more to evaluating a model over metric evaluation and predicting a batch and checking manually. These are the numbers you'd put out to signify how great your model is, if you were to write a publication - however, it's still a black box system in which we have no clue as to why the street before was classified as a building and vice versa. We do know that there's an overlap, but what has the model learned to make it misclassify in a relatively obvious image like the one above?
Note: Before going further, let's unshuffle the test set again. I know - it's tedious, and I wish there were a shorter way to do this, but there isn't. Some of the visualizations down the line we'll make are affected by the order of data.
test_generator = test_datagen.flow_from_directory(config['TEST_PATH'], target_size=(150,150), batch_size=32, shuffle=False, class_mode='categorical', seed=2) y_preds = model.predict(test_generator)
Identifying Wrong Predictions
Let's start out by identifying the wrong predictions. The
test_generator.classes property is a NumPy array of the classes - one class for each instance. It's the length of our test set (3000) and can be used to directly compare against the most confident predictions in our