Deep Learning

Major: System Analysis
Code of subject: 6.124.03.E.111
Credits: 5.00
Department: Information Systems and Networks
Lecturer: Professor Peleshchak Roman Mikhailovich
Semester: 8 семестр
Mode of study: денна
Learning outcomes: Methods and algorithms for teaching computer systems on based on experience and understanding of the world in terms of the hierarchy of concepts, their depth. Mathematical substantiation of methods and concepts of deep learning.
Required prior and related subjects: • Mathematical Statistics • Theory of probability •Discrete Mathematic
Summary of the subject: Lecture 1. Introductory lecture and Intro to GN: -organizational issues: implementation, submission and protection of labs -prerequisites (math analysis, statistics, linear algebra, MN) -recall the main terms from the MN -semi-supervised and self-supervised training Lecture 2. MLE for problems with the teacher. Concept of tensors. Introduction to TF (or PyTorch or Flax) -statistical definition of the task with the teacher -MLE and MAP -loss functions and metrics -tensors and operations on them (in one of the libraries) Lecture 3. Feedforward NN. Lecture 4. Training of neural networks. Optimization methods. -Method of (stochastic) gradient descent -backpropagation algorithm -automatic differentiation and calculation graphs -SGD with momentum -optimizers Adam, Lion Lecture 5. Introduction to CV. Convolutional neural networks. Lecture 6. Regularization of NN. Lecture 7. Skip-connections and ResNet. Lecture 8. Deep learning for NLP. Text classification. Language models and text generation. -preprocessing and tape preparation methods -Tokenization and embedding methods -the concept of a language model and methods of generation -seq2seq models Lecture 9. [Transformers, part 1] Mechanism of attention. Lecture 10. [Transformers, part 2] Transformer model. Pre-training and fine-tuning models. Lecture 11. [Transformers, h3] LLM: BERT, GPT, T5. BPE Tokenizers. Ecosystem HuggingFace. Lecture 12. Text embedding from LLM. Introduction to prompt engineering. Zero-shots and few-shots inference. -embedding models and sentence similarity -Sentence-transformers for zero-shot problems -prompt engineering -zero-shots and few-shots fine-tuning and inference Lecture 13. Learning without a teacher from GN. Autoencoders and latent spaces. Lecture 14. Diffusion models for image generation (simplified) Lecture 15. Ethics and regulation of generative AI. Preparation for the exam. Laboratory works (approximate list) (50 points): 1. Classification of images by a fully connected neural network (4 points): 2. Analysis of optimization algorithms (4 points): 3. Classification of images from CNN (4 points): 4. Regularization of neural networks (5 points): 5. Fine-tuning ResNet for image classification (6 points): 6. Text classification by neural networks (6 points): 7. Fine-tuning of speech models-transformers (6 points): 8. Mini-project - regression problem (6 points): 9. Mini-project - classification task (6 points): 10. Generation of images by diffusion models (3 points)
Assessment methods and criteria: • Current control (40%): written reports on laboratory work, essay, oral examination; • Final control (60% of exam): in written, verbally.
Recommended books: 1. Golovko V. Neural Networks: Training, Organization and Application / V. Golovko. – Moscow: IPRZHR, 2001. – 256 p. 2. Hinton G. E. A practical guide to training restricted Boltzmann machines / G. E. Hinton // Tech. Rep. 2010-000. – Toronto: Machine Learning Group, University of Toronto, 2010. – https://www.cs.toronto.edu/~hinton/absps/guideTR.pdf 3. Bengio Y. Learning deep architectures for AI / Y. Bengio // Foundations Trends Mach. Learning. – 2009. – Vol. 2(1). – P. 1–127. – https://www.iro.umontreal.ca/~lisa/pointeurs/TR1312.pdf