Chapter 1: The Distributed Computing Problem

Learning Objectives

• Describe the MapReduce computational model and identify where its communication bottleneck arises
• Quantify the impact of stragglers on the latency of a synchronous distributed computation
• Formulate distributed (synchronous) gradient descent and pinpoint the gradient-aggregation bottleneck
• State the three coupled challenges — communication efficiency, straggler mitigation, and privacy — that drive the rest of the book
• Recognize why these three challenges cannot be solved in isolation: every gain on one axis costs something on another