You'll notice experienced engineers basically refuse to look at average latency. Why is that, and what do they look at instead?

Averages hide the tail, basically. If most requests are fast but some small fraction are really slow, the average still looks fine while those users are having a terrible time. p99 is the value that 99% of requests come in under, so it captures that slow tail, the worst case real users actually hit. That's why people watch percentiles like p95, p99, p99.9 instead of the mean. And to compute percentiles correctly you need the distribution of latencies, which is what histograms give you. They bucket the response times so you can calculate percentiles and aggregate across servers. A single average per server can't really be combined in any meaningful way. The tail matters too because in systems making lots of calls, the slow ones end up dominating the overall response time. So you track latency as a histogram and watch the high percentiles, and those tell you what your slowest users are experiencing.