According to Prometheus documentation in order to have a 95th percentile using histogram metric I can use following query:
histogram_quantile(0.95, sum(rate(http
I believe this is the code for it in prometheus The general idea is that you use the data in the buckets to extrapolate / approximate the quantiles Elasticsearch also does something similar (yet different/much simpler) in their rollup capabilities