DSC 204A: Scalable Data Systems

Name: DSC 204A Fall 2025
Author: Hao Zhang

Instructor: Hao Zhang, UC San Diego, Fall 2025

Announcements

Week 11 Announcements

Dec 1 · 0 min read

Reading Summary for week 8 & 9 & 10 will due 12/3, 11:59pm. It should cover 5 papers with 2.5 pages in total.
Please finish the course evaluation. All students will receive 2 bonus points if 80% or more of the class complete the evaluation.

Week 1

Sep 23

0 No lecture

Sep 26

1 Introduction: Slides • Recording

Survey Beginning of Quarter Survey (Due: End of Week 3 - 10/10)

Readings N/A

Week 2

Sep 30

2 Computer organization, representation of data, floating points: Slides • Recording • Notes

Readings (Due 10/15)

Required: OSTEP Chapter 4
Optional: OSTEP Chapter 6, OSTEP Chapter 7

Oct 2

3 Floating points, processors, memory hierarhcy: Slides • Recording • Notes

Readings (Due 10/15)

Required: OSTEP Chapter 18
Optional: OSTEP Chapter 15, OSTEP Chapter 16

Week 3

Oct 7

4 Cache, OS, scheduling: Slides • Recording • Notes

Readings (Due 10/15)

Required: OSTEP Chapter 22
Optional: OSTEP Chapter 28, OSTEP Chapter 39

Oct 9

5 Memory, cloud computing: Slides • Recording • Notes

Readings (Due 10/15)

Required: Understanding the cloud computing stack: SaaS, PaaS, and IaaS

Week 4

Oct 14

6 Cloud, network basics, collectives: Slides • Recording • Notes

Readings (Due 10/29)

Required: Above the Clouds: A Berkeley View of Cloud Computing
Optional: The Datacenter as a Computer Designing Warehouse-Scale Machines Chapter 1

Oct 16

7 Collective communication 1: Slides • Recording • Notes

Readings (Due 10/29)

Required: Above the Clouds: A Berkeley View of Cloud Computing
Optional: The Datacenter as a Computer Designing Warehouse-Scale Machines Chapter 2

Week 5

Oct 21

8 Collective communication 2, data warehouse: Slides • Recording • Notes

Readings (Due 10/29)

Required: Design Data-intensive applications, Chapter 3, Page 90 - 101
Optional: Design Data-intensive applications, Chapter 3, the rest

Oct 23

9 Parallelism: Slides • Recording • Notes

Readings (Due 10/29)

Required: Google File System
Optional: The Snowflake Elastic Data Warehouse

Week 6

Oct 28

10 Batch processing, map-reduce: Slides • Recording • Notes

Readings (Due 11/12)

Required: MapReduce: Simplified Data Processing on Large Clusters
Optional: Delay Scheduling: A Simple Technique for Achieving Locality and Fairness in Cluster Scheduling

Oct 30

11 Streaming processing, RDD, Spark: Slides • Recording • Notes

Readings (Due 11/12)

Required: Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing
Optional: PowerGraph: Distributed Graph-Parallel Computation on Natural Graphs

Week 7

Nov 4

12 MLsys Overview, DL libraries: Slides • Recording • Notes

Readings (Due 11/12)

Required: Ray: A Distributed Framework for Emerging AI Applications
Optional: Discretized Streams: An Efficient and Fault-Tolerant Model for Stream Processing on Large Clusters

Nov 6

13 Guest Lecture - Prof. Eugene Wu: Recording • Notes

Readings (Due 11/12)

Week 8

Nov 11

Holiday

Nov 13

14 Guest Lecture - Shreya Shankar: Recording • Notes

Readings (Due 12/3)

Required: TensorFlow: A system for large-scale machine learning
Optional: PyTorch Distributed: Experiences on Accelerating Data Parallel Training

Week 9

Nov 18

15 Guest Lecture - Audrey Cheng: Recording • Notes

Readings (Due 12/3)

Required: Scaling Distributed Machine Learning with the Parameter Server
Optional: Petuum: A New Platform for Distributed Machine Learning on Big Data

Nov 20

16 Guest Lecture - Prof. Junchen Jiang: Recording • Notes

Readings (Due 12/3)

Required: Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Optional: GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism

Week 10

Nov 25

17 Autodiff, MLSys overview, LLM architectures: Slides • Recording • Notes

Readings (Due 12/3)

Required: FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Optional: Attention Is All You Need

Nov 27

Holiday

Readings (Due 12/3: 5 papers in total, 2.5 pages needed)

Required: Efficient Memory Management for Large Language Model Serving with PagedAttention
Optional: Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning

Week 11

Dec 2

18 LLM, attention: Slides • Recording • Notes

Readings No required readings!

Optional: Fast Inference from Transformers via Speculative Decoding

Dec 4

19 LLM inference and serving: Slides • Recording

Readings No required readings!

Optional: Deepseek-v3 Technical Report

Week 12

Dec 11

19 Make-up lecture: PagedAttention, ML parallelisms: Slides1 Slides2 • Recording