Deep Learning

This course covers deep learning, a highly influential area of machine learning that has achieved remarkable performance in numerous applications. The course begins with fundamental concepts including multilayer neural networks, their modeling power, and training methods. It then introduces major architectures like CNNs and RNNs, along with advances in network design, optimization, generalization improvement, and training techniques. Generative models will be examined as an important branch. The course also covers notable deep networks developed in recent years, with emphasis on applications in computer vision and natural language processing.

1. Introduction to Artificial Neural Networks
2. Multi-layer Perceptron (MLP)
   • MLP as universal approximator
3. Error Back Propagation Algorithm
4. Optimization in Deep Networks
   • Overview of convex optimization
   • Optimization methods: SGD, Momentum, RMSProp, Adam, etc.
5. Deep Network Training, Design and Generalization Techniques
   • Generalization improvement techniques: regularization, dropout, data augmentation
   • Batch Normalization
   • Activation functions, weight initialization, input normalization, etc.
6. Convolutional Neural Networks (CNNs)
   • Convolution and pooling layers
   • Popular CNN architectures
   • CNN applications
7. Recurrent Neural Networks (RNNs)
   • Sequence modeling
   • Long Short-Term Memories (LSTMs)
   • Attention Networks
   • Language Modeling using RNNs
   • Other RNN applications in NLP and other domains
8. Transformer Architecture
9. Product-Sum Networks
10. Generative Models
    • Autoregressive models
    • Variational Autoencoders
    • Generative Adversarial Networks (GANs)
    • Flow-based models
11. Deep Reinforcement Learning
    • Deep Q-Learning
    • Policy Gradient approach
    • Actor-Critic approach
12. Adversarial Examples and Network Robustness
13. Advanced Topics
    • Dual Networks and Dual Learning
    • Graph Convolutional Networks
    • Self-supervised Learning

• Assignments: 30%
• Midterm: 20%
• Final Exam: 30%
• Quizzes: 10%
• Project or Research Work: 10%