What dominates latency in a RAG pipeline?

LLM generation is typically 70–90% of total latency for long responses. Retrieval adds 10–50 ms for an in-memory HNSW index. Reranking adds another 50–200 ms depending on cross-encoder model size.

How can I reduce RAG pipeline latency below 500 ms?

Use a small, fast embedding model (e5-small, 50 ms), an in-memory HNSW index (5 ms), skip reranking for most queries, and use a 7B model with speculative decoding or a fast API for generation.

RAG Pipeline Latency Estimator

Estimate end-to-end latency for a Retrieval-Augmented Generation (RAG) pipeline. Model query embedding, ANN retrieval, reranking, context assembly, and LLM generation latency.

Networking

Algorithms

Binary & Number

Systems

Dev Metrics

IP Subnet Calculator

IP Address

CIDR Prefix

Network

192.168.1.0

Broadcast

192.168.1.255

Subnet Mask

255.255.255.0

First Host

192.168.1.1

Last Host

192.168.1.254

Usable Hosts

254

Binary breakdown:

IP: 11000000.10101000.00000001.00000000

Mask: 11111111.11111111.11111111.00000000

Net: 11000000.10101000.00000001.00000000

How to Use

Enter the values for the patient or scenario you are assessing. Estimate end-to-end latency for a Retrieval-Augmented Generation (RAG) pipeline. Model query embedding, ANN retrieval, reranking, context assembly, and LLM generation latency. Use the rag latency result to inform your clinical assessment.

Related Calculators

ANN Recall/Speed

Technology

Reranker Latency

Technology

Inference Latency

Technology

RAG Pipeline Latency Estimator

IP Subnet Calculator

How to Use

Related Calculators

Frequently Asked Questions

What dominates latency in a RAG pipeline?

How can I reduce RAG pipeline latency below 500 ms?