Jev-X

帖子

Thread · 8 条帖子

本串介绍PostHog如何用决策模型替代部分LLM调用,以降低成本并提升速度,并说明哪些任务仍应使用LLM。

查看 8 条目录
PostHog@posthog · ↗

Building a product that requires constant LLM calls gets real expensive, real fast. Fortunately, decision models like Jev can do some of the work up to 444.6x cheaper and 193.6x faster. Here's how we use them at PostHog:

4115191146
PostHog@posthog · ↗

It would cost way too much to send every Replay Vision summary to an LLM to scrub personal data. Instead, we get a decision model to score session summaries, and only the ones that score below 0.4 for PII get rewritten by Gemini.

推文附图
1030222
PostHog@posthog · ↗

Our PostHog Slack bot uses one to determine which model it should use to complete a task. When it receives a message, it checks a list of models to decide which one makes the most sense. Each option in the list carries a short note from us on what that model's good for, like "A general coding model. Good for most code changes and debugging" for GPT Sol.

推文附图
1001174
PostHog@posthog · ↗

Decision models can also be used in combination with embeddings to more cheaply rank results than an LLM would. When a user searches for a theme like "onboarding friction" in Replay Vision, an embedding search pulls an initial list of session replays. It then passes that list to a decision model, which checks if the observation matches the initial search, then reranks them by probability. This improved ranking quality by about 25%.

推文附图
101078
PostHog@posthog · ↗

Decision models aren't so good for multi-step thinking, though. Before we landed on embeddings, we tried using them to evaluate two questions about each Session Replay: 1. Is this session worth watching? 2. If so, why is it worth watching? (Choices like “error on screen,” “slow or stuck,” “user gave up", etc.) The results were awful. Processing the two questions required an hourly job that sent 12-38k tokens each and flooded the model server and block other products' calls for hours. It's better to stick to LLMs for that kind of work.

100062
PostHog@posthog · ↗

We also enhanced our deterministic SQL query chart picker in PostHog. It used to only look at rules (ex: multiple rows, a number column and a date column meant a line chart). Now, for opted-in orgs, a decision model looks at the SQL itself and first three rows of results to determine the best visualization for the data.

推文附图
100060
PostHog@posthog · ↗

For open-ended jobs, LLMs should still be the default. New solutions for the middle-ground are well underway already, though. Our in-house AI research team built Jeeves for jobs that require more context than typical Jev use cases, but still need to be answered cheaply. It's a hybrid decision model that reasons before deciding when it's unsure. It's our default for decision calls at PostHog that don't name a model.

1002145

引用这条的帖子4 人

只统计本站收录的帖子,按点赞排序。