In this study, Julian Backa, FIT M.Sc. graducate and former team member, tested and compared four often applied machine learning algorithms, the:

  • k-Nearest Neighbour (kNN),
  • Maximum Likelihood (ML),
  • Random Forest (RF)
  • Support Vector Machine (SVM) classifiers.

Julian Backa performed the accuracy assessment and further statistical tests in R. He designed his research using preprocessed Sentinel 2A data subsets of 25 x 15 km from 2018 and implemented 6 different training data sets. Finally, he compared the processed results with the latest EU CORINE Land Cover (CLC) dataset as validation data.

The principle research question was

  • which classification algorithm achieves the best results based on a standardised amount of training data in three different investigation areas across Germany.

And subsequent questions were:

  • Do the same classifiers, applied in different software environments achieve equal results using the default settings?
  • Is the CORINE Land Cover map applicable as validation dataset?

The research was implemented according to the following workflow:

The standardised training data was selected in 6 differently sized training polygons of 50, 80, 110, 140, 170, 200 polygons of pixel per class and a number of pixels per polygon ranging from 10 to 50. All classification results were compared with each investigation area using the Overall Accuracy (OA) and the KAPPA index, within a first comparison round. The second comparison round compared the two-best classifiers from the three study areas using confusion matrices, the OA, the KAPPA index, the Producer Accuracy (PA) and the User Accuracy (UA).

The classification results showed that the small sampling design was insufficient given the CLC as reference data. Hence, the results from the large sampling are given higher credibility. The SVM classifier applied in ArcGIS obtained the highest OA results on the Eberswalde and Grafing investigation areas (78.7% and 86.7%) and the RF classifier from the SAGA software on the Meschede study area (69.8%). Furthermore, the highest accuracy results had been mostly obtained with the 170 polygons training dataset. The CLC can be used as validation data, but only on a larger extent regarding the low resolution.

Overall accuracy (OA) assessment and result table for the Eberswalde region: