In the otherwise dreadful year 2020, I got the idea to start an educational project on Machine Learning and Deep Learning (ML&DL). Building on materials that I had already developed for an Advanced Datamining course that I teach, my goal was to develop a package that implements some of the most prominent machine learning algorithms in common use, including (but not limited to) Artificial Neural Networks. Other similar projects exist that have inspired me to think that such a thing may be possible and timely, and I will surely refer to these in the course of this project. But at the same time these made me realize that I would use some slightly different approaches here and there, and I would not make these different choices if I didn’t think they provide a useful complement to all the available material out there already.
If I allow myself to be ambitious, my aims for this endeavour are:
- to explain the theory and inner workings of ML&DL in a way that does not avoid some level of rigor, but still remains accessible for a public with intermediate knowledge of mathematics and programming (e.g. advanced high-school or starting undergraduate level);
- to implement a range of common ML&DL algorithms in one (or more) popular programming language(s) in such a way that a moderately experienced programmer (e.g. 1 year exposure to a programming language of some kind) can follow along, but can also choose to deviate and make slightly other design choices as he or she sees fit;
- to implement ML&DL algorithms in a way that may not compete with available packages (like tensorflow), but is nevertheless sufficiently fully featured that the most generic use cases can be handled, sufficiently optimised that it is usable in practice, and employing a syntax that is sufficiently similar to other packages to allow these to be easily learned;
- to apply ML&DL to a range of “classic” problems (from irises to handwritten digits) that illustrate their strenghts and weaknesses;
- and – last but certainly not least – to allow both you and me to learn a thing or two in the process, experiment with perhaps naive but neat ideas, and make ML&DL (even more) interesting and fun!
I chose to name this brain child Dave Langers’ Deep Learning Demonstration Library, abbreviated as DL³. Partly because of the alternatingly alliterating acronym of course, but also because it emphasizes some key characteristics: it is authored by me, it covers deep learning and machine learning more broadly, and it is meant as an educational programming project.
I envisage this project to consist of a number of sources: a series of videos on some accessible platform, a number of blog posts on this site, and a code repository with code and notebooks. I hope to start with some YouTube video’s soon using python to write a package with modules as well as Jupyter notebooks, and a git repository on BitBucket. I have more than enough ideas to keep me occupied for quite some time to come, but sometimes that prevents one from just starting somewhere. Hence this message stub.
Topics on my short-term list certainly will include the following:
- Rosenblatt’s perceptron and linear regression;
- neurons with general activation- and loss-functions;
- derivatives and gradient descent;
- multilayer perceptron with input, hidden, and output layers;
- forward- and back-propagation;
- softmax and cross-entropy;
- cross-validation, avoiding over- and underfitting;
- early stopping, L1– & L2-regularization, dropout, augmentation;
- batch normalization;
- momentum, adaptive moments, and nesterov optimizers;
- …
On our way we’ll encounter other classical algorithms like logistic regression, support vector machines, non-linear regression, and perhaps we’ll do dimensionality reduction and clustering using unsupervised or semi-supervised learning too. Hopefully at some point we may endeavour into other network topologies, like convolutional or recurrent neural networks. Other important classes of algorithms comprise decision trees and naive Bayes and some of their variants, and perhaps we may need to spend some time on the basic representation of various types of (numeric and categorical) data too, but I don’t yet see how this will tie in with other topics. Anyway, let’s first try to get there.
If you’re up to the challenge to follow along and mutually inspire and learn from each other, don’t hesitate to drop me a message. In any case, I hope to see you here more often in the future!