Machine learning is one of the important lanes of AI which is very spicy hot subject in the research or industry. Companies, universities devote many resources to advance their knowledge.Cognitive learning theory in the classroom recent advances in the field propel very solid results for different tasks, comparable to human performance (98.98% at traffic signs - higher than human-).Cognitive learning theory in the classroom

First step toward prevalent ML was proposed by hebb, in 1949, based on a neuropsychological learning formulation. It is called hebbian learning theory. With a simple explanation, it pursues correlations between nodes of a recurrent neural network (RNN).Cognitive learning theory in the classroom it memorizes any commonalities on the network and serves like a memory later. Formally, the argument states that;

Let us assume that the persistence or repetition of a reverberatory activity (or "trace") tends to induce lasting cellular changes that add to its stability.… when an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A' s efficiency, as one of the cells firing B , is increased.[1]

cognitive learning theory in the classroom

After 3 years later, widrow [4] engraved delta learning rule that is then used as practical procedure for perceptron training. It is also known as least square problem.Cognitive learning theory in the classroom combination of those two ideas creates a good linear classifier. However, perceptron's excitement was hinged by minsky[3] in 1969 . He proposed the famous XOR problem and the inability of perceptrons in such linearly inseparable data distributions.Cognitive learning theory in the classroom it was the minsky's tackle to NN community. Thereafter, NN researches would be dormant up until 1980s

There had been not to much effort until the intuition of multi-layer perceptron (MLP) was suggested by werbos[6] in 1981 with NN specific backpropagation(BP) algorithm, albeit BP idea had been proposed before by linnainmaa [5] in 1970 in the name "reverse mode of automatic differentiation".Cognitive learning theory in the classroom still BP is the key ingredient of today's NN architectures. With those new ideas, NN researches accelerated again. In 1985 - 1986 NN researchers successively presented the idea of MLP with practical BP training (rumelhart, hinton, williams [7] - hetch, nielsen[8])

cognitive learning theory in the classroom

At the another spectrum, a very-well known ML algorithm was proposed by J. R. Quinlan [9] in 1986 that we call decision trees, more specifically ID3 algorithm.Cognitive learning theory in the classroom this was the spark point of the another mainstream ML. Moreover, ID3 was also released as a software able to find more real-life use case with its simplistic rules and its clear inference, contrary to still black-box NN models.Cognitive learning theory in the classroom

One of the most important ML breakthrough was support vector machines (networks) (SVM), proposed by vapnik and cortes[10] in 1995 with very strong theoretical standing and empirical results.Cognitive learning theory in the classroom that was the time separating the ML community into two crowds as NN or SVM advocates. However the competition between two community was not very easy for the NN side after kernelized version of SVM by near 2000s .(I was not able to find the first paper about the topic), SVM got the best of many tasks that were occupied by NN models before. In addition, SVM was able to exploit all the profound knowledge of convex optimization, generalization margin theory and kernels against NN models.Cognitive learning theory in the classroom therefore, it could find large push from different disciplines causing very rapid theoretical and practical improvements.

NN took another damage by the work of hochreiter's thesis [40] in 1991 and hochreiter et.Cognitive learning theory in the classroom al.[11] in 2001, showing the gradient loss after the saturation of NN units as we apply BP learning. Simply means, it is redundant to train NN units after a certain number of epochs owing to saturated units hence nns are very inclined to over-fit in a short number of epochs.Cognitive learning theory in the classroom

Little before, another solid ML model was proposed by freund and schapire in 1997 prescribed with boosted ensemble of weak classifiers called adaboost.Cognitive learning theory in the classroom this work also gave the godel prize to the authors at the time. Adaboost trains weak set of classifiers that are easy to train, by giving more importance to hard instances.Cognitive learning theory in the classroom this model still the basis of many different tasks like face recognition and detection. It is also a realization of PAC (probably approximately correct) learning theory.Cognitive learning theory in the classroom in general, so called weak classifiers are chosen as simple decision stumps (single decision tree nodes). They introduced adaboost as ;

Another ensemble model explored by breiman [12] in 2001 that ensembles multiple decision trees where each of them is curated by a random subset of instances and each node is selected from a random subset of features.Cognitive learning theory in the classroom owing to its nature, it is called random forests(RF). RF has also theoretical and empirical proofs of endurance against over-fitting. Even adaboost shows weakness to over-fitting and outlier instances in the data, RF is more robust model against these caveats.(for more detail about RF, refer to my old post.).Cognitive learning theory in the classroom RF shows its success in many different tasks like kaggle competitions as well.

As we come closer today, a new era of NN called deep learning has been commerced.Cognitive learning theory in the classroom this phrase simply refers NN models with many wide successive layers. The 3rd rise of NN has begun roughly in 2005 with the conjunction of many different discoveries from past and present by recent mavens hinton, lecun, bengio, andrew ng and other valuable older researchers.Cognitive learning theory in the classroom I enlisted some of the important headings (I guess, I will dedicate complete post for deep learning specifically) ;

With the combination of all those ideas and non-listed ones, NN models are able to beat off state of art at very different tasks such as object recognition, speech recognition, NLP etc.Cognitive learning theory in the classroom however, it should be noted that this absolutely does not mean, it is the end of other ML streams. Even deep learning success stories grow rapidly , there are many critics directed to training cost and tuning exogenous parameters of these models.Cognitive learning theory in the classroom moreover, still SVM is being used more commonly owing to its simplicity. (said but may cause a huge debate 🙂 )

Before finish, I need to touch on one another relatively young ML trend.Cognitive learning theory in the classroom after the growth of WWW and social media, a new term, bigdata emerged and affected ML research wildly. Because of the large problems arising from bigdata , many strong ML algorithms are useless for reasonable systems (not for giant tech companies of course).Cognitive learning theory in the classroom hence, research people come up with a new set of simple models that are dubbed bandit algorithms [27 - 38] (formally predicated with online learning) that makes learning easier and adaptable for large scale problems.Cognitive learning theory in the classroom