Term tag: Distributed Computing

Cloud Computing

Cloud computing is the on-demand delivery of computing power, database storage, applications, and other IT resources via the internet with pay-as-you-go.

Concurrency

Concurrency is structuring a program so independent tasks interleave. How it differs from parallelism, and why serving a model is a queueing problem.

Consensus Algorithm

A consensus algorithm makes separate machines agree on one value even when some fail. Why majorities always overlap, and why agreement costs a round trip.

Distributed Computing

Distributed computing splits work across separate machines that talk only by messages. Why partial failure, not speed, is the defining problem.

Distributed Training

How one model is trained across thousands of GPUs — data, tensor and pipeline parallelism, and why the network rather than the chip becomes the bottleneck.

Interconnect (NVLink, InfiniBand)

The wiring that moves data between AI chips: on-package memory, NVLink inside a server, InfiniBand between them — and why each tier is far slower than the last.

Parallel Processing

Running many computations at the same instant across many processing units. In AI it makes matrix multiplication fast, which is why GPUs train models.