| Description |
1 online resource (267 pages) |
| Series |
Synthesis lectures on computer architecture, 1935-3243 ; #53 |
|
Synthesis digital library of engineering and computer science.
|
|
Synthesis lectures in computer architecture ; #53.
|
|
Synthesis Lectures on Computer Architecture Ser
|
| Contents |
1. Introduction -- 1.1. Deep learning in action -- 1.2. AI, ML, NN, and DL -- 1.3. Brief history of neural networks -- 1.4. Types of learning -- 1.5. Types of topologies -- 1.6. Training and serving a simple neural network -- 1.7. Memory and computational analysis -- 1.8. Hardware design considerations -- 1.9. Software stack -- 1.10. Notation |
|
2. Building blocks -- 2.1. Activation functions -- 2.2. Affine -- 2.3. Convolution -- 2.4. Pooling -- 2.5. Recurrent units -- 2.6. Normalization -- 2.7. Embeddings -- 2.8. Attention -- 2.9. Dropout |
|
3. Models and applications -- 3.1. Recommender systems topologies -- 3.2. Computer vision topologies -- 3.3. Natural language processing topologies -- 3.4. Reinforcement learning algorithms |
|
4. Training a model -- 4.1. Generalizing from training to production datasets -- 4.2. Weight initialization -- 4.3. Optimization algorithms: minimizing the cost -- 4.4. Backpropagation -- 4.5. Training techniques -- 4.6. Transfer learning via fine-tuning. -- 4.7. Training with limited memory |
|
5. Distributed training -- 5.1. Data parallelism -- 5.2. Model parallelism -- 5.3. Federated learning -- 5.4. Collective communication primitives |
|
6. Reducing the model size -- 6.1. Numerical formats -- 6.2. Quantization methodology -- 6.3. Pruning and compression -- 6.4. Knowledge distillation |
|
7. Hardware -- 7.1. Moore, Dennard, and Amdahl -- 7.2. Memory and bandwidth -- 7.3. Roofline modeling -- 7.4. Processor designs -- 7.5. High-performance interconnects -- 7.6. Processors in Production -- 7.7. Platforms strengths and challenges -- 7.8. Evaluating devices and platforms |
|
8. Compiler optimizations -- 8.1. Language types -- 8.2. Front-end, middle-end, and back-end compilation phases -- 8.3. LLVM -- 8.4. Hardware-independent optimizations -- 8.5. Hardware-dependent optimizations |
|
9. Frameworks and compilers -- 9.1. Frameworks -- 9.2. TensorFlow -- 9.3. PyTorch -- 9.4. TVM -- 9.5. PlaidML -- 9.6. Glow -- 9.7. XLA -- 9.8. MLIR -- 9.9. Others |
|
10. Opportunities and challenges -- 10.1. Machine learning for DL systems -- 10.2. Democratizing DL platforms -- 10.3. Security -- 10.4. Interpretability -- 10.5. Society impact -- 10.6. Concluding remarks |
| Summary |
This book describes deep learning systems: the algorithms, compilers, and processor components to efficiently train and deploy deep learning models for commercial applications. The exponential growth in computational power is slowing at a time when the amount of compute consumed by state-of-the-art deep learning (DL) workloads is rapidly growing. Model size, serving latency, and power constraints are a significant challenge in the deployment of DL models for many applications. Therefore, it is imperative to codesign algorithms, compilers, and hardware to accelerate advances in this field with holistic system-level and algorithm solutions that improve performance, power, and efficiency. Advancing DL systems generally involves three types of engineers: (1) data scientists that utilize and develop DL algorithms in partnership with domain experts, such as medical, economic, or climate scientists; (2) hardware designers that develop specialized hardware to accelerate the components in the DL models; and (3) performance and compiler engineers that optimize software to run more efficiently on a given hardware. Hardware engineers should be aware of the characteristics and components of production and academic models likely to be adopted by industry to guide design decisions impacting future hardware. Data scientists should be aware of deployment platform constraints when designing models. Performance engineers should support optimizations across diverse models, libraries, and hardware targets. The purpose of this book is to provide a solid understanding of (1) the design, training, and applications of DL algorithms in industry; (2) the compiler techniques to map deep learning code to hardware targets; and (3) the critical hardware features that accelerate DL systems. This book aims to facilitate co-innovation for the advancement of DL systems. It is written for engineers working in one or more of these areas who seek to understand the entire system stack in order to better collaborate with engineers working in other parts of the system stack. The book details advancements and adoption of DL models in industry, explains the training and deployment process, describes the essential hardware architectural features needed for today's and future models, and details advances in DL compilers to efficiently execute algorithms across various hardware targets. Unique in this book is the holistic exposition of the entire DL system stack, the emphasis on commercial applications, and the practical techniques to design models and accelerate their performance. The author is fortunate to work with hardware, software, data scientist, and research teams across many high-technology companies with hyperscale data centers. These companies employ many of the examples and methods provided throughout the book |
| Bibliography |
Includes bibliographical references and index |
| Notes |
Description based on online resource; title from PDF title page (viewed December 14, 2020) |
| Subject |
Machine learning.
|
|
Artificial intelligence.
|
|
Artificial intelligence -- Data processing.
|
|
Computer algorithms.
|
|
Compilers (Computer programs)
|
|
Artificial Intelligence
|
|
Algorithms
|
|
Machine Learning
|
|
artificial intelligence.
|
|
algorithms.
|
|
Artificial intelligence
|
|
Artificial intelligence -- Data processing
|
|
Compilers (Computer programs)
|
|
Computer algorithms
|
|
Machine learning
|
| Form |
Electronic book
|
| ISBN |
1681739674 |
|
9781681739670 |
|
9783031017698 |
|
3031017692 |
|
