Perplexity Details Its GPU Embedding Stack: How Ivy, Tulip and ROSE Serve pplx-embed

Retrieval quality in an AI search product is bounded by two things: how good the embedding model is, and how cheaply you can run it across an index. This week, Perplexity Engineering team published Fast Embeddings on GPUs, an under-the-hood account of the second — the serving infrastructure behind pplx-embed and the ranking models used across Perplexity Search, Computer and the API Platform.

Perplexity team states that embedding inference on the GPU side has largely converged across engines on mature Hopper and Blackwell hardware. The wins sit in the runtime and harness around the model: CUDA graph management, an async result-tracking abstraction, and a Rust request path.

Two traffic patterns, one engine

Perplexity frames embedding serving as two workloads. Batch embedding happens when building or re-indexing the vector database, where throughput minimizes cost. Online embedding happens at query time, where a short query must be embedded fast. Scoring sits in between: after vector search, large document batches are ranked, balancing both.

The key decision is that Perplexity did not build a separate embedding engine. Because embedding models are small Transformers, batch embedding resembles compute-bound prefill and online embedding, often a few tokens, resembles memory-bound decode. So the research team reuses the prefill and decode kernels from its LLM stack.

Ivy, Tulip and ROSE

Three services handle a request:

• Ivy is a Rust HTTP gateway. It does the CPU-side work — JSON parsing, tokenization, input templating, batch splitting — and translates requests into a custom gRPC protocol. It also splits large-batch requests into chunks and load-balances them across replicas, which corrects the load imbalance that arises when production payloads vary in size.
• Tulip is the inference server interface: a gRPC server built with Rust, tokio and tonic, handling scheduling and batching before dispatching to the engine.
• ROSE (Runtime-Optimized Serving Engine) implements model inference. It is primarily Python, provides kernels, layers and model definitions, manages CUDA graphs, and exposes a step() function to Tulip.

Why the scheduler is deliberately simple

Tulip picks sequences first-come, first-served while requests accumulate. That simplicity is justified by a measurement: for small embedding models at the sequence lengths Perplexity serves, the linear cost of dense layers dominates the quadratic cost of attention. Latency is therefore roughly proportional to token count, not sequence count. Once a batch saturates the GPU, around 512 tokens on a sub-billion-parameter model, packing in more sequences does not improve efficiency.

CUDA graphs and LazyTensors

On small batches, CPU-side kernel launching can outweigh GPU execution. Perplexity builds whole-model CUDA graphs for all embedding models, capturing every launch into a single driver call. Because embedding models are small, the inflection point where GPU work exceeds launch cost arrives at batches of thousands of tokens and tens of sequences. Some attention implementations block full-model graphs by depending on dynamic host-side inputs; Perplexity upstreamed changes to FlashInfer to enable capture.

Graphs must be captured per configuration, so token counts are padded to buckets that are multiples of 64 or 256. That still yields thousands of graphs and multiple minutes of capture per model. The fix is lazy capture: each configuration gets an eager warmup run, then triggers capture and replay on its second hit. This costs p99 latency at startup but spreads minutes of eager work across hours.

The second piece is the LazyTensor, which tracks a page-locked host buffer plus a cudaMemcpyAsync and a CUDA event. Instead of step() blocking on the device, it returns a LazyTensor, letting a Rust async task wait on batch N while the CPU enqueues N+1.

Send a request</button></div>
</div>
</div>

<div class=”pe-panel” id=”peP1″>
<div class=”pe-card”>
<div class=”pe-txt”>On small batches, CPU-side kernel launches can outweigh GPU work. A whole-model <b>CUDA graph</b> captures every launch into one call to the driver, so the CPU is freed to enqueue the next batch. Toggle the two modes.</div>
<div class=”pe-ctl” style=”margin:0 0 12px”>
<button class=”pe-btn ghost on” id=”peEager”>Eager launches</button>
<button class=”pe-btn ghost” id=”peGraph”>CUDA graph</button>
</div>
<div class=”pe-lane”><div class=”pe-lbl”>Host / CPU</div><div class=”pe-track” id=”peCpuT”></div></div>
<div class=”pe-lane”><div class=”pe-lbl”>Device / GPU</div><div class=”pe-track” id=”peGpuT”></div></div>
<div class=”pe-stats”>
<div class=”pe-stat”><div class=”v” id=”peLaunches”>—</div><div class=”k”>Driver calls</div></div>
<div class=”pe-stat”><div class=”v” id=”peGap”>—</div><div class=”k”>GPU idle gaps</div></div>
</div>
<div class=”pe-txt” style=”margin:12px 0 0;font-size:11.5px;color:#6E8285″>Schematic. Block widths illustrate the launch-overhead pattern described in the post, not measured timings.</div>
</div>
</div>

<div class=”pe-panel” id=”peP2″>
<div class=”pe-card”>
<div class=”pe-txt”>Reading results back normally forces a host sync. A <b>LazyTensor</b> tracks a page-locked host buffer plus an async device-to-host copy and a CUDA event, so Tulip can block on batch N while the CPU already prepares batch N+1.</div>
<div class=”pe-lane”><div class=”pe-lbl”>CPU — prepare / sync</div><div class=”pe-track” id=”peLzC”></div></div>
<div class=”pe-lane”><div class=”pe-lbl”>GPU — forward pass</div><div class=”pe-track” id=”peLzG”></div></div>
<div class=”pe-ctl”>
<button class=”pe-btn” id=”peLzRun”> Run 3 batches</button>
<button class=”pe-btn ghost on” id=”peLzOn”>Overlapped</button>
<button class=”pe-btn ghost” id=”peLzOff”>Blocking</button>
</div>
<div class=”pe-note” style=”margin-top:12px” id=”peLzNote”>Overlapped: while the GPU chews batch N, the CPU is already tokenizing and packing batch N+1.</div>
</div>
</div>

<div class=”pe-panel” id=”peP3″>
<div class=”pe-card”>
<div class=”pe-txt”>For small embedding models at these sequence lengths, the linear cost of dense layers dominates the quadratic cost of attention, so latency tracks <b>token count, not sequence count</b>. Past roughly <b>512 tokens</b> on a sub-1B model, the GPU is saturated and packing in more sequences stops helping.</div>
<div class=”pe-lbl” style=”margin-top:6px”>Tokens in batch: <span id=”peTokV” style=”color:#3FB6C4″>512</span></div>
<input type=”range” id=”peTok” min=”32″ max=”4096″ step=”32″ value=”512″>
<div class=”pe-lbl”>GPU utilisation</div>
<div class=”pe-bar”><div class=”pe-fill” id=”peUtil”></div></div>
<div class=”pe-stats”>
<div class=”pe-stat”><div class=”v” id=”peUtilV”>—</div><div class=”k”>Saturation</div></div>
<div class=”pe-stat”><div class=”v” id=”peState”>—</div><div class=”k”>Regime</div></div>
</div>
<div class=”pe-txt” style=”margin:12px 0 0;font-size:11.5px;color:#6E8285″>Illustrative curve. The ~512-token saturation point is the figure stated in the post; the shape between points is a stand-in, not a benchmark.</div>
</div>
</div>

<div class=”pe-foot”>
<span>Source: Perplexity Engineering, “Fast Embeddings on GPUs” (Sep 4, 2026)</span>
<span><a href=”https://www.marktechpost.com”>Built by Marktechpost</a></span>
</div>

<script>
(function(){
var R=document.getElementById(‘pplxEmbedExplainer’);
var NOTES=[
‘<b>Ivy</b> — parses JSON, tokenizes with the in-house unigram tokenizer, applies input templating and splits large batches, then translates to a custom gRPC protocol. It also load-balances chunks across replicas.’,
‘<b>Tulip</b> — Rust gRPC server on tokio and tonic. Requests accumulate while it dispatches or waits; sequences are picked first-come, first-served and packed into a batch for the accelerator.’,
‘<b>ROSE</b> — the Runtime-Optimized Serving Engine. Python-defined kernels and layers, CUDA-graph management, and a step() function that returns a handle to the GPU computation. No KV cache is allocated for embeddings.’
];
function q(s){return R.querySelector(s)} function qa(s){return R.querySelectorAll(s)}
/* tabs */
qa(‘.pe-tab’).forEach(function(t){t.addEventListener(‘click’,function(){
qa(‘.pe-tab’).forEach(function(x){x.classList.remove(‘on’)});t.classList.add(‘on’);
qa(‘.pe-panel’).forEach(function(p,i){p.classList.toggle(‘on’,i==+t.dataset.p)});});});
/* 1 flow */
var note=q(‘#peNote’),dot=q(‘#peDot’);
function sel(i){qa(‘.pe-node’).forEach(function(n,k){n.classList.toggle(‘hot’,k==i)});note.innerHTML=NOTES[i];}
qa(‘.pe-node’).forEach(function(n){n.addEventListener(‘click’,function(){sel(+n.dataset.i)})});
sel(0);
var busy=false;
q(‘#peGo’).addEventListener(‘click’,function(){
if(busy)return;busy=true;var steps=[[0,’3%’],[1,’40%’],[2,’76%’]],k=0;
dot.style.transition=’none’;dot.style.left=’3%’;dot.style.opacity=’1′;
sel(0);
var iv=setInterval(function(){k++;if(k>2){clearInterval(iv);dot.style.opacity=’0′;busy=false;return;}
dot.style.transition=’left .8s cubic-bezier(.4,0,.2,1)’;dot.style.left=steps[k][1];sel(k);},900);
});
/* 2 cuda graphs */
var cpuT=q(‘#peCpuT’),gpuT=q(‘#peGpuT’),mode=’eager’;
function blk(p,l,w,cls,txt){var d=document.createElement(‘div’);d.className=’pe-blk ‘+cls;d.style.left=l+’%’;d.style.width=w+’%’;d.textContent=txt||”;p.appendChild(d);}
function drawG(){
cpuT.innerHTML=”;gpuT.innerHTML=”;
if(mode==’eager’){
for(var i=0;i<6;i++){blk(cpuT,1+i*16.4,7,’pe-cpu’,’launch’);blk(gpuT,8.4+i*16.4,7.6,’pe-gpu’,’kernel’);if(i<5)blk(gpuT,16+i*16.4,7.2,’pe-idle’,’idle’);}
q(‘#peLaunches’).textContent=’6′;q(‘#peGap’).textContent=’5′;
}else{
blk(cpuT,1,12,’pe-cpu’,’graph launch’);blk(cpuT,15,26,’pe-cpu’,’prepare next batch’);
blk(gpuT,13.5,84,’pe-gpu’,’6 kernels — one replay, no gaps’);
q(‘#peLaunches’).textContent=’1′;q(‘#peGap’).textContent=’0′;
}}
q(‘#peEager’).addEventListener(‘click’,function(){mode=’eager’;q(‘#peEager’).classList.add(‘on’);q(‘#peGraph’).classList.remove(‘on’);drawG();});
q(‘#peGraph’).addEventListener(‘click’,function(){mode=’graph’;q(‘#peGraph’).classList.add(‘on’);q(‘#peEager’).classList.remove(‘on’);drawG();});
drawG();
/* 3 lazytensor */
var lzC=q(‘#peLzC’),lzG=q(‘#peLzG’),lzMode=’on’,lzBusy=false,lzNote=q(‘#peLzNote’);
function drawLz(){
lzC.innerHTML=”;lzG.innerHTML=”;
for(var i=0;i<3;i++){
if(lzMode==’on’){blk(lzC,2+i*32,14,’pe-cpu’,’prep ‘+(i+1));blk(lzG,17+i*32,26,’pe-gpu’,’batch ‘+(i+1));}
else{blk(lzC,2+i*32,12,’pe-cpu’,’prep ‘+(i+1));blk(lzC,15+i*32,16,’pe-idle’,’wait’);blk(lzG,15+i*32,15,’pe-gpu’,’batch ‘+(i+1));}
}}
function lzRun(){
if(lzBusy)return;lzBusy=true;drawLz();
var bs=lzG.querySelectorAll(‘.pe-blk’),cs=lzC.querySelectorAll(‘.pe-blk’);
[].forEach.call(bs,function(b){b.style.opacity=’.15′});[].forEach.call(cs,function(b){b.style.opacity=’.15′});
var all=[].concat([].slice.call(cs),[].slice.call(bs)),j=0;
var iv=setInterval(function(){if(j>=all.length){clearInterval(iv);lzBusy=false;return;}all[j].style.opacity=’1′;j++;},220);
}
q(‘#peLzRun’).addEventListener(‘click’,lzRun);
q(‘#peLzOn’).addEventListener(‘click’,function(){lzMode=’on’;q(‘#peLzOn’).classList.add(‘on’);q(‘#peLzOff’).classList.remove(‘on’);lzNote.innerHTML=’Overlapped: while the GPU chews batch N, the CPU is already tokenizing and packing batch N+1.’;drawLz();});
q(‘#peLzOff’).addEventListener(‘click’,function(){lzMode=’off’;q(‘#peLzOff’).classList.add(‘on’);q(‘#peLzOn’).classList.remove(‘on’);lzNote.innerHTML=’Blocking: every step() waits for the device, so the CPU sits idle and the GPU starts late.’;drawLz();});
drawLz();
/* 4 batch shape */
var tok=q(‘#peTok’);
function drawT(){
var v=+tok.value;q(‘#peTokV’).textContent=v;
var u=Math.min(100,Math.round(100*(1-Math.exp(-v/230))));
q(‘#peUtil’).style.width=u+’%’;q(‘#peUtilV’).textContent=u+’%’;
q(‘#peState’).textContent=v<512?’Under-filled’:(v<1200?’Saturated’:’Throughput-bound’);
q(‘#peState’).style.color=v<512?’#C7A24E’:’#3FB6C4′;
}
tok.addEventListener(‘input’,drawT);drawT();
})();
</script>
</div>

<script>
(function(){function h(){var e=document.getElementById(‘pplxEmbedExplainer’);if(!e)return;parent.postMessage({pplxEmbedH:e.offsetHeight+40},’*’);}
window.addEventListener(‘load’,h);setTimeout(h,300);setTimeout(h,1200);
document.addEventListener(‘click’,function(){setTimeout(h,250)});
document.addEventListener(‘input’,function(){setTimeout(h,120)});
if(window.ResizeObserver){var e=document.getElementById(‘pplxEmbedExplainer’);if(e)new ResizeObserver(h).observe(e);}})();
</script>
</body></html>”>

Kernels still matter

ROSE supports multiple attention backends for ragged inputs: FlashInfer 2, FlashInfer 3 and FlashAttention 4. Perplexity team reports FlashAttention 4 is generally faster, but FlashInfer 3 outperforms it on Qwen-based models at very long sequence lengths, so backend selection is made case by case. Notably, when serving an embedding model ROSE does not instantiate a KV cache and dispatches to ragged attention variants to avoid padding.

Benchmarks

Perplexity benchmarks against vLLM v0.22.0 in BF16 on real weights and eval-derived inputs, with warmup runs verifying cosine similarity divergence within 0.1%. Four suites are charted: low-latency embeddings (batch 1; 128/512/4096 tokens), low-latency scoring (batch 5/25/50 at 512 tokens), high-throughput embeddings (batch 100, four concurrent processes) and high-concurrency embeddings (1 to 16 concurrent requests, including Ivy tokenization and network overhead).

Key Takeaways

• Perplexity’s embedding stack reuses its LLM prefill/decode kernels rather than running a separate engine.
• Latency tracks token count, not sequence count; ~512 tokens saturates a sub-1B model.
• Whole-model CUDA graphs plus lazy capture cut launch overhead without minutes-long startup.
• LazyTensor overlaps CPU batch prep with in-flight GPU work instead of blocking on sync.
• Ivy, Tulip and ROSE are internal; pplx-embed is reachable via Perplexity’s Embeddings API.

Check out the Technical details. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us

The post Perplexity Details Its GPU Embedding Stack: How Ivy, Tulip and ROSE Serve pplx-embed appeared first on MarkTechPost.