Hybrid search runs a meaning-based search and a keyword search, then merges their scores into one ranking. Most systems run both searches in full, even though only a handful of results survive the merge.

The key observation is that the first search already reveals how high the bar for the final ranking is. Turning that bar into a minimum score lets the keyword search ignore everything below it.

Often the keyword search can be skipped entirely. When the bar is computed exactly, the final ranking stays the same, and no new indexes are needed.

The same idea applies wherever one stage of a pipeline can bound the work of the next. The paper is under review.

AI assistants increasingly gather evidence from many search services, such as document stores and ticketing systems. Calling all of them is slow and costly, but skipping the wrong one loses the answer.

I am building routers that decide which services to call and when to stop, using only what they know before calling, namely the query, what each service offers and what earlier calls returned.

Early results roughly halve the number of calls without losing the evidence that matters.

The open question is how much more a router can save from the same cheap signals.

Production services have hundreds of settings, and their best values drift as traffic and hardware change. Tuning them by hand cannot keep up.

OPPerTune treats each service as a black box. The service reports how well it is running, and OPPerTune uses that feedback to choose what to try next.

I led its design and development through to production at Microsoft. In our evaluation, it cut the tail latency of microservice applications by more than half compared with expert-chosen settings.

Deployment exposed a harder problem, which is specifying what operators want tuned in the first place.

Read the NSDI 2024 paper, try the code or watch the talk.

Operators say things like "keep latency low without adding machines." A learning-based tuner needs that as a single numeric reward, and writing one by hand is hard.

I started Reward Copilot to automate this step. A language model proposes candidate rewards, each one trains a tuning policy, and the policies are checked against the operator's requirements.

The best candidates then guide the next round of suggestions.

The broader question is when a language model's output can be used inside a system, and how to check it cheaply when it cannot.

Read the NeurIPS 2024 workshop paper.