Scale, Specialization, and Your App

Before anyone tunes a model server, someone has to decide what the server must be good at. A voice assistant, a nightly document pipeline, and a code editor all call language models, yet each one needs a different kind of speed, a different bill, and a different promise about uptime. This lesson covers the thinking that comes before inference engineering: what "best" means for a product, when to rent a shared API and when to run dedicated GPUs, and how the type of application shapes every later choice.

Afterwards you can list the five facts to know about a product, compare shared and dedicated inference with numbers, classify an application as online or offline and as consumer or business, and name the compliance questions that limit where inference can run.

The lesson builds on What Is Inference Engineering. It uses two terms from there. A token is a small piece of text that models read and write. A GPU is the chip that runs the model.

Example 1: optimization is a balance, not a maximum

To optimize means to pick the best solution from a range of options. Inference engineering adds speed and scale to AI products by optimizing how generative models are served in production. Before improving performance, the team has to know what "best" means for its product, because most performance gains come from a trade-off among three things:

In practice optimization is usually about finding a balance, not about maximizing one factor.

An analogy: players in the NFL (American professional football) are big, fast, and strong. They are still not as big as sumo wrestlers, as fast as Olympic sprinters, or as strong as champion powerlifters. Their bodies and skills are tuned for what one position demands over a full season. An inference service is the same: it is tuned for the demands of its model, its product, and its traffic. The analogy stops at this point: an athlete trains for years, while a service can be re-tuned when the workload changes. The rule that carries over is that the more constraints are known, the better the outcome can be.

Example 2: five things to know about your product

Five questions describe the workload. Writing the answers down turns vague wishes into constraints.

Early in a product's life these answers are often unclear. At that stage it is usually better to use off-the-shelf APIs than to invest in dedicated inference. As the product scales, the requirements become clear, and inference engineering becomes worth doing.

Example 3: shared versus dedicated inference

There are two ways to add AI models to a product:

• Shared inference: send traffic to a public API endpoint for a model and pay per million tokens, or by some other consumption-based metric.
• Dedicated deployment: rent GPUs and set up an inference service used only by this application, paying per hour of GPU time. The other route is buying GPUs and installing them on premises (in the company's own data center).

Shared versus dedicated is not the same question as closed versus open models. Many shared endpoints serve open models, and many model labs offer large customers a dedicated setup for closed models. Still, one main reason to adopt open models is that they allow unrestricted dedicated inference.

Most AI products start with pay-per-token APIs, because the trade-offs fit the time of searching for product-market fit (finding out that enough people want the product) and early growth.

Over time, moving to dedicated deployments becomes necessary for three reasons:

• Scale: the traffic volume is large enough that paying per GPU is cheaper than paying per million tokens.
• Specialization: the product runs a custom or fine-tuned model, or it has specific latency or uptime requirements.
• Orchestration: the product uses several models and multi-step pipelines, and the team needs to minimize network latency and deployment complexity.

The switch makes the team responsible for its own inference engineering. It brings flexibility and control, and it also grows the engineering surface area and raises the floor of the monthly bill. The rule: switch only when there is a clear and immediate business need.

The calculation below shows the "scale" reason with numbers. The prices are made up for illustration and are not any provider's real prices. Shared inference costs the price per million tokens times the volume. A dedicated deployment costs a fixed amount per GPU-hour, whatever the volume, as long as the GPUs can keep up. The calculation assumes one GPU can serve 3,000 tokens per second and a month has 730 hours.

Assumptions (illustrative): shared price 0.60 dollars per million tokens,
GPU 2.50 dollars per hour, one GPU serves 3,000 tokens/s, month = 730 hours.

one GPU per month, cost     = 2.50 x 730            = 1,825 dollars
one GPU per month, capacity = 3,000 x 3,600 x 730   = 7,884 million tokens
break-even volume           = 1,825 / 0.60          = 3,042 million tokens per month

millions of tokens/month | shared cost | dedicated cost (GPUs)
                     100 |       60.00 |    1,825.00 (1)
                   1,000 |      600.00 |    1,825.00 (1)
                   3,000 |    1,800.00 |    1,825.00 (1)
                  10,000 |    6,000.00 |    3,650.00 (2)
                  20,000 |   12,000.00 |    5,475.00 (3)

shared cost    = millions x 0.60
dedicated cost = ceil(millions / 7,884) GPUs x 1,825

The first calculation gives the dedicated floor: one GPU costs 1,825 dollars a month whether it is busy or idle. The break-even figure divides that floor by the shared price: below about 3,042 million tokens a month, shared inference is cheaper, and above it a fully used GPU is cheaper. The table shows the shape. The shared cost rises in a straight line with usage. The dedicated cost is a step function, because GPUs are bought one at a time. At 100 million tokens the dedicated floor is much higher than the shared bill, which is why early products start with APIs. At 20,000 million tokens the shared bill (12,000) is more than twice the dedicated one (5,475). Real comparisons also count engineering time, idle capacity, and traffic peaks, which is why the switch needs a clear business need and not only a cheaper line in a table.

flowchart TD
    A["New AI product"] --> B{"Requirements clear and traffic large enough that paying per GPU beats paying per token?"}
    B -->|"no"| C["Shared inference: pay per token"]
    B -->|"yes"| D{"Custom model, strict latency or uptime, or multi-model pipelines?"}
    D -->|"yes"| E["Dedicated deployment: pay per GPU-hour"]
    D -->|"no, but volume is high"| E
    C -->|"product scales, need becomes clear"| B

The chart reads top to bottom. The loop back from shared inference reflects the advice that teams start with APIs and revisit the choice as the product grows.

Example 4: the use case drives every decision

An analogy: a sports coach recruiting at a school picks different students depending on the sport. The basketball coach wants the tallest children, and the gymnastics coach picks among the shortest. In the same way, how a system will be used determines how it is built.

Two kinds of teams must build at the highest level of generality:

• Foundation model teams: they trained their own model and sell consumption directly through a public shared inference API, so they must support many usage patterns.
• Inference platform teams: they build an inference platform, either internally or as a product, and must support any model and any use case.

Most AI-native applications are different. They are vertical apps, which are products built for one field, such as code editors or customer service agents, where AI creates a new kind of experience. For these, the advice is to add as many constraints as possible by being specific about the use case.

Example 5: categories of AI-native applications

Generative AI created a new class of applications across industries. Each category uses different models and modalities (a modality is a type of data, such as text, audio, or images) and needs tailored inference.

The list is a small sample, and it shows the breadth of considerations. As models become faster, cheaper, and smarter, uses that nobody has imagined yet will appear.

Read one row closely. In the agent row, one user action triggers many inference calls. If each call takes 2 seconds and an agent step needs 10 calls in a row, the user waits 20 seconds, so the speed of each single call is multiplied by the length of the chain. The next lessons define the measures used to talk about this.

Example 6: online versus offline workloads

One of the main trade-offs in inference engineering is latency against throughput. Lower latency makes the application faster. Higher throughput makes it cheaper at scale, because fewer GPUs serve the same number of users.

• Online applications run in real time with an impatient user waiting on the other end of each API call. Code completion, chat, and voice agents are online, and they are optimized for latency.
• Offline (batch) applications process large amounts of work without a waiting user. A single request may be too slow for a good user experience, but the system as a whole handles far more requests per hour in parallel. They are served best by high-throughput deployments.

Three typical offline workloads:

One model can serve both kinds. Whisper, a speech-to-text model, can power a real-time dictation app and a batch transcription job. If both have enough volume, it is more cost-effective to run two separate deployments of the same model: one tuned for latency and one tuned for throughput.

A small worked case shows why. Suppose a GPU can process 1 request at a time in 1 second, or 16 requests together in 4 seconds. One at a time, it finishes 1 request per second and each user waits 1 second. In groups of 16, it finishes 4 requests per second (16 divided by 4), four times more work per GPU, but each user waits 4 seconds. The numbers are illustrative. The trade-off is real: grouping raises throughput and raises waiting time. The online job picks the first setting and the offline job picks the second.

Example 7: consumer versus business applications

Applications for consumers and for businesses have different inference needs.

Consumer inference engineers therefore design for sudden spikes at low cost. Business inference engineers design for steady, dependable performance.

In both cases, compliance can limit infrastructure options, particularly in regulated industries (fields with strict legal rules, such as healthcare). Three questions come up:

• Data sovereignty: whether the GPUs are in a geographic region where the company may send user data.
• User privacy: whether the inputs and outputs of the model are kept private and secure.
• Regulatory compliance: whether the company and all of its providers comply with every relevant regulation.

Inference engineers have to work closely with security and legal specialists to confirm that the infrastructure they operate is compliant. The production lessons return to these topics.

Interview points

• Optimization means balancing latency, throughput, and quality for a specific product, not maximizing a single metric.
• The five facts to establish about a product are model requirements, application interface, latency budget, unit economics, and usage patterns.
• Shared inference means paying per token on a public endpoint. Dedicated deployment means paying per GPU-hour for a service used only by the application.
• Shared inference has no cold starts and almost no engineering work, but its cost grows linearly, its provider caps uptime, and it gives no control over latency, quality, or rate limits.
• The three reasons to move to dedicated deployments are scale, specialization, and orchestration. Switch only for a clear, immediate business need.
• Foundation model teams and inference platforms must build for the widest range of uses. Vertical apps should add constraints.
• Online workloads optimize latency. Offline workloads optimize throughput, and one model may need two deployments for the two kinds of use.
• Consumer apps stress marginal cost and flexibility for spiky traffic. B2B apps stress latency and uptime.
• Compliance questions are data sovereignty, user privacy, and regulatory compliance.

Summary

• Inference engineering adds speed and scale by optimizing the production serving of generative models, and "best" depends on the product.
• Early products should use off-the-shelf APIs. Dedicated inference becomes worthwhile when the requirements are clear.
• Shared inference trades control for zero overhead. Dedicated deployment trades engineering effort and a higher monthly floor for control and lower cost at scale.
• Application categories include agents, chat, voice, media, search, RecSys, completion, and moderation, each with its own main consideration.
• Latency against throughput is the central trade-off, and it separates online from offline workloads.
• Consumer and B2B products have different priorities, and compliance can restrict where inference may run.