Controversial Opinions: Greatest Hits

Chapter 21 of Part III, Interview Mastery & The Staff Evaluator. It continues from What Interviewers Are Actually Evaluating. The next chapter is Coming in Volume 2.

Controversial Opinions — Greatest Hits

Part III — Interview Mastery & The Staff Evaluator

Part III — Interview Mastery & The Staff Evaluator

Every chapter in this book is built around committed positions. This one drops the scaffolding entirely.

What follows are the opinions that experienced Staff and Principal engineers hold but rarely say aloud in company meetings — because they're uncomfortable, because they contradict received wisdom, or because saying them requires defending a position against someone who's built a career on the opposite assumption.

They're organized into five clusters: what we get wrong about databases and scaling, what we get wrong about distributed systems, what we get wrong about caching and queues, what we get wrong about security and compliance, and what we get wrong about engineering decisions themselves.

Each opinion is stated as bluntly as I believe it. Each is defended with specifics. None are offered as universal truth — they are the positions I would commit to in 30 seconds if asked.

You may disagree. Good. The point is not agreement. The point is that you should have a position.

Cluster 1: Databases and Scaling

"You probably don't need sharding."

Most teams that shard their database didn't need to. They needed read replicas, better indexes, connection pooling, or query optimization — and they reached for sharding because it sounds serious and it's what big companies do.

The operational cost of sharding is permanent. Cross-shard queries. Distributed transactions. Schema migrations that must be applied N times. Rebalancing events that consume entire engineering teams for months. Every application-layer query now needs routing logic. Every on-call engineer needs to understand which shard a record is on. The simplicity of a single-master database is gone forever.

Read replicas, connection pooling (PgBouncer handles 10x effective connections without touching the schema), table partitioning, and vertical scaling cover 90% of scaling needs with 10% of the operational burden. A single PostgreSQL instance on the largest available cloud hardware today — 192 vCPUs, 4TB RAM, 256K IOPS — handles more write throughput than most companies ever approach.

When do you actually need sharding? When you've exhausted vertical scaling AND read replicas AND query optimization, and your write throughput on a single master is the hard ceiling. That's it. Not storage — use object storage. Not reads — use replicas. Writes. Only writes.

The corollary: teams that argue for sharding should be required to demonstrate they've tried read replicas and query optimization first. "We're growing fast" is not a reason to shard. Growth projections are wrong by 5–10x more often than not.

"The shard key decision is irreversible — treat it that way."

You will never reshard. Teams do not re-shard. The organizational and technical cost is so high that companies live with a bad shard key for five or ten years rather than fix it — because fixing it means 6–12 months of a senior engineering team moving every row in the database while keeping the system live, at constant risk of data loss, with no way to test at production scale before you're doing it in production.

The teams that have resharded know this. Ask them. They will not recommend it.

The correct response to this reality: treat the shard key decision as permanent from day one. Spend a week evaluating candidates — not an afternoon. Run the access patterns. Talk to product about the next 18 months of features, not just the current ones. Write an Architecture Decision Record. Get explicit sign-off from engineering leadership.

This sounds like bureaucracy. It is, in fact, the minimum due diligence for an irreversible decision.

"Cross-shard joins are a design smell, not a query pattern."

If your application needs a cross-shard join, one of three things is true: the shard key is wrong, the data model is wrong, or you're using the wrong tool for the access pattern.

Analytics that need cross-shard queries should use an OLAP store — Redshift, BigQuery, Snowflake — not the sharded OLTP database. Admin dashboards that need cross-shard queries should query a read replica aggregate, not production shards. New features that need cross-shard queries signal that product and engineering didn't align on shard boundaries before the feature was designed.

Building cross-shard join capability is building infrastructure to work around a bad decision. The right response is: "Which access pattern changed, and should we create a denormalized read store for it?" That question is harder than building the join infrastructure. It is the right question.

"Exact rate limiting is a myth — and teams that claim it haven't looked at their own metrics."

At any meaningful scale, you are never enforcing the limit you think you're enforcing. Clock skew between nodes causes token refills to vary by 10–100ms. Network delay means coordination messages arrive late. Retry amplification means rejected requests generate additional load. Batched arrivals mean traffic doesn't come in smoothly. By the time you measure, you've already over-admitted.

The limit you configure is "approximately N +/- epsilon." The Staff question is not "how do we achieve exact enforcement" — it is "what is our epsilon, is it acceptable for this intent, and who has explicitly signed off on it?"

For abuse protection at 1,000 requests per minute, an epsilon of 10% — 100 extra requests per window — is acceptable noise. The intent is to block volumetric attacks, not enforce to the last token. For billing enforcement where every token has dollar value, epsilon must be near zero, which requires centralized enforcement and different infrastructure.

Teams that claim exact enforcement at scale either have very low traffic (exact enforcement is trivial with one instance) or haven't measured their bypass rate recently. Measure the bypass rate. You will be surprised.

Cluster 2: Distributed Systems

"Most teams don't need Raft. They need PostgreSQL with synchronous replication."

Distributed consensus — Raft, Paxos, Multi-Paxos — is the right solution to one specific problem: how do you achieve agreement among multiple nodes when any node can fail and messages can be lost? etcd, ZooKeeper, and Consul are the appropriate tools for this problem, and the problem appears in: cluster membership, configuration distribution, leader election, and service discovery.

It does not appear in: storing financial records, serving user profiles, tracking payment state, recording chat messages, or most of what applications actually do.

PostgreSQL with synchronous replication gives you: ACID transactions, complex queries and joins, 30 years of production-grade reliability, and the ability to run arbitrary consistency logic in the database layer. A database trigger enforcing the trial balance invariant is a single CREATE FUNCTION statement. You cannot write that in etcd.

The error: candidates who hear "we need strong consistency" and reach for Raft. Strong consistency in a relational database is called ACID. PostgreSQL has had it since 1985. Use the right tool.

"Exactly-once delivery is mostly a lie."

When someone says "exactly-once delivery," they almost always mean something weaker.

"Kafka exactly-once" means exactly-once within the Kafka Streams processing topology — Kafka to process to Kafka. Your downstream database write can still fail. "Exactly-once processing" means at-least-once delivery plus an idempotent consumer — not the same thing. "Transactional outbox" gives you exactly-once publish; the consumer still gets at-least-once delivery.

Where it actually breaks: when two producers publish the same event; when a consumer rebalances mid-processing; when intentional replay makes "exactly-once" mean "exactly-twice"; when the guarantee spans two systems that don't share a transaction log (you cannot have exactly-once between Kafka and PostgreSQL — they are not in the same transaction).

The Staff position: stop saying "exactly-once" unless you can name the precise boundary. Design for "effectively-once" instead: at-least-once delivery plus idempotent consumers that make reprocessing safe. This is achievable. It is honest. It is what every production system that claims "exactly-once" actually implements.

"A growing queue is not absorbing load. It is hiding a problem that will explode later."

The lie: "The queue gives us time to scale." The truth: the queue gave you time to not notice you were sinking.

The math is unambiguous. Producers at 10K messages/second. Consumers at 8K messages/second. Queue lag grows at 2K messages/second. After one hour: 7.2 million messages, 12 minutes of lag. After two hours: 14.4 million messages, memory pressure, broker beginning to slow. After three hours: GC pauses, producer timeouts, data loss begins. After four hours: incident declared. The post-mortem says "the queue failed." The queue did not fail. The queue did exactly what it was designed to do — it buffered. The producers and consumers were mismatched from the start, and the queue made that invisible until it was too late to absorb gracefully.

Backpressure — rejecting early, letting producers handle it — is often the correct design. The queue does not fix a throughput mismatch. It delays the consequence of one. The right question before adding a queue: if consumers can't keep up, who can afford to wait, and for how long? Name the answer explicitly. Then design accordingly.

"Global message ordering is an engineering vanity project."

Users only perceive ordering within a single conversation. Nobody notices that a message in Group A arrived 500ms before a message in Group B even though Group B's message was sent first. Nobody. Not ever.

Total ordering across conversations requires a single global sequencer — a single point of failure, a throughput ceiling, and an architectural commitment that scales to exactly one server before sharding the sequencer becomes its own distributed systems problem. And the guarantee purchased: users can't tell the difference between this and per-conversation ordering.

WhatsApp doesn't have global ordering. Slack doesn't have global ordering. Signal doesn't have global ordering. None of the messaging systems used by 2 billion people have global ordering, because global ordering is not a user requirement. It is an engineering assumption that masquerades as one.

Per-conversation ordering is the only ordering that matters. Invest zero effort in cross-conversation ordering. If you're building a compliance audit log — where the absolute order of events across all conversations is a legal requirement — that is a different system with different requirements, and it belongs in a separate store, not the message delivery pipeline.

"Distributed consensus is used where simpler tools would work."

The decision to use Raft or Paxos or ZooKeeper is often made by engineers who find distributed consensus interesting, not by engineers who have established that distributed consensus is necessary. I have reviewed systems designs where the team proposed Raft for: storing user preferences, tracking feature flag state, managing session tokens, and recording API usage counters. In every case, a single PostgreSQL instance with appropriate indexing would have handled the load for years.

The bar for distributed consensus is: you genuinely cannot tolerate the downtime of a primary failover, AND you need the primary failover to be automatic, AND the data being coordinated is the kind of data that genuinely requires linearizable reads across nodes. If any of those three conditions is absent, you probably want a primary-replica database with automatic failover instead.

Use etcd for cluster membership. Use ZooKeeper for distributed locks when you truly need them. Use PostgreSQL for everything else.

Cluster 3: Caching and Queues

"Your cache is probably serving stale data right now. You just don't know it."

No errors. No alerts. No latency spike. Just confidently wrong answers.

Stale data is the only failure mode that generates no technical signal. A cache serving 200-millisecond-old data returns HTTP 200 with a response time well within SLA. The only signals are user complaints — if users notice — and business metrics — if the data is business-critical. Most teams measure cache hit rate. Almost nobody measures cache correctness. If you cannot tell me the last time you validated your cache values against the source of truth, your cache correctness is an assumption, not a guarantee.

The corollary: high hit rate can mask correctness bugs. A pricing bug writes an incorrect value to the database. That incorrect value gets cached. For the next five minutes — the TTL — 99% of reads hit the cache. All of them get the wrong price. Without the cache, the bug affects 1% of reads (direct database reads); some users notice immediately. With the cache, the bug affects 99% of reads for the full TTL duration. The cache turned a transient bug into a widespread incident.

Hit rate is a performance metric. Correctness is the safety metric. Most teams measure the first. Almost none monitor the second.

"Removing a cache is often the right fix."

When is a cache worth removing? When hit rate is below 50%. When write rate approaches read rate, creating invalidation churn. When multiple incidents have been traced to cache staleness. When the origin can handle the load without it. When nobody on the team can explain the staleness contract — what data is cached, for how long, under what invalidation conditions.

Teams don't remove caches because "we already built it" and "it must be helping somehow." The implicit assumption: adding infrastructure is progress; removing it is retreat. This assumption is wrong. The most resilient cache is the one that doesn't exist. The cache that doesn't exist has zero invalidation bugs, zero thundering herd events, zero stale data incidents, and zero operational overhead.

Removing infrastructure is politically harder than adding it. It requires arguing that the status quo is wrong and that the team made a mistake by building the thing. The Staff engineer asks "what if we just didn't cache this?" and is willing to accept the awkward silence that follows.

"Cache invalidation is an ownership problem, not a technical one."

"Cache invalidation is hard" is treated as a law of nature. It is actually an organizational failure disguised as a technical problem.

Invalidation fails because: the writer doesn't know about the cache (no shared understanding of the cache contract), there are multiple caches and one was forgotten (cache proliferation without governance), nobody designed for concurrent write race conditions (technical oversight), or the cache contract was never documented (ownership gap).

The technical fixes — Change Data Capture, version stamps, pub/sub invalidation — exist and work. You can implement any of them in a sprint. The organizational fix is harder: every write path must have a named owner of its cache contract. Without that, you implement CDC, and then a new write path appears that bypasses the CDC pipeline because the engineer who wrote it didn't know the cache existed, and the bug returns.

When a staleness incident occurs, "TTL was too long" is never the root cause. It is the symptom. The root cause is one of: writer didn't know to invalidate, invalidation code had a bug, invalidation was asynchronous and lost, or nobody owned the cache contract. Fix the ownership. The TTL is irrelevant.

"Presence at scale is unsolvable — stop trying to solve it perfectly."

Real-time presence for 100 million users is a battle against physics. A 10-second heartbeat for 100M concurrent users is 10 million writes per second — just for presence. A 30-second heartbeat is 3.3 million writes per second. More write throughput than most companies' entire database handles, for a feature that delivers one bit of information per event: online or not.

And the information itself? Knowing your friend was online 30 seconds ago does not meaningfully change user behavior. The 30-second-ago signal is functionally identical to the 5-minutes-ago signal for the vast majority of use cases.

WhatsApp shows "last seen" — deliberately coarse because it reduces presence update frequency to once per user activity session rather than once per heartbeat. The engineering team made a deliberate decision to trade presence accuracy for infrastructure sanity. The users adapted. Nobody switched to a competitor because they could only see "last seen 5 minutes ago" instead of real-time presence.

Design presence to be lazy and scoped. Show "online" only to users who are actively viewing the conversation with that person. Use "last seen" granularity. The engineering savings fund features users actually notice.

Cluster 4: Security and Compliance

"PCI scope is the most consequential architecture decision in a payments system — and most teams make it without realizing it's a decision."

Candidates routinely propose "collect card data and call the payment provider API" without understanding that this single architectural choice commits the company to 12+ months of compliance work, a dedicated security team, annual QSA audits costing $50K–$500K, and permanent maintenance of a PCI-compliant environment.

The alternative — hosted payment fields (Stripe Elements, Adyen Drop-in) — reduces this to a 3-month self-assessment. Card data never touches your servers. The performance difference is negligible: under 50ms additional latency for the iframe render. The organizational cost difference: one additional security engineer versus none.

This is not a security engineering decision. It is a product and finance decision that masquerades as a security engineering decision. The question "should we collect card data directly?" should be answered by the CFO and the General Counsel, not by the engineering team. Engineering's job is to make the cost of each path visible. "Direct API handling requires 12 months of compliance work and $500K in annual audit costs. Hosted fields do not. The product capability difference is X and Y." Then let the decision-makers decide.

The Staff move: default to hosted payment fields and only justify direct API handling if there is a concrete business requirement that cannot be achieved any other way. That bar is very high.

"E2E encryption's operational cost is systematically underestimated — every time."

Teams say "we'll add E2E encryption" the way teams say "we'll add unit tests later" — as if it's a cleanup task that can be done without changing the fundamental nature of the system.

E2E encryption eliminates five capabilities: server-side search, server-side spam detection, server-side content moderation, seamless multi-device sync, and message backup. Each eliminated capability requires a separate, more expensive solution under E2E:

Client-side search is limited to messages downloaded to the device — not the entire conversation history unless the user has been online. Spam detection operates on metadata only — sender frequency, recipient report rates — not content. Content moderation requires user reporting and hash-matching for known illegal content — no proactive scanning. Multi-device sync requires per-device key management — a new phone cannot read old messages unless the user transfers the encryption key or has a cloud backup. Message backup requires a user-held backup key — if the user loses it, the messages are permanently unrecoverable. No support ticket fixes it. No engineer can help. The data is gone.

Any team that adds E2E encryption without an explicit conversation with Trust and Safety, product, and legal about each of these capability eliminations will spend the next two years rebuilding the capabilities they broke. This conversation must happen before a single line of encryption code is written.

"Most companies are dramatically over-rotated on confidentiality and under-rotated on availability."

Security training emphasizes confidentiality — protecting data from unauthorized access. This is important. But for most businesses, the more common and more costly failure mode is availability: the system is down, the data cannot be accessed by the people who are authorized to access it.

A payments system that is inaccessible for two hours costs significantly more than a payments system that had a minor data exposure affecting 100 records. A chat system that is down for 30 minutes loses more users than a chat system that had a message delivery failure affecting a small percentage of conversations.

The organizational implication: security budgets and engineering attention are systematically directed toward access controls, encryption, and threat modeling (all important) and systematically under-directed toward availability engineering, backup recovery testing, and degraded-mode operation (equally important, less glamorous).

The Staff question, when evaluating a security investment: "Does this improve confidentiality, integrity, or availability? What's the relative risk of failure across all three for this specific system?" The answer shapes the priority.

Cluster 5: Engineering Decisions

"Most payment bugs are accounting bugs, not engineering bugs."

When a payment system breaks, engineers look at the code. The code is usually correct. The accounting logic is wrong. A journal entry that debits the wrong account type. A refund that reverses the charge but not the platform commission. A multi-currency conversion that rounds to the nearest cent in the wrong direction, creating a $0.01 discrepancy that passes all unit tests and accumulates to thousands of dollars over six months. A partial refund that creates ledger entries that technically balance but represent an accounting model that makes no sense to a financial controller.

These bugs pass unit tests because the code does what the specification says. The specification was written by engineers who didn't consult a financial controller. The specification was wrong.

The Staff insight: involve a financial controller as a design partner, not a QA reviewer. A controller should review the journal entry specifications for every new payment flow before code is written. An accounting model error caught in a specification review costs an afternoon. The same error caught in month-end reconciliation costs $50,000 in manual corrections, a regulatory inquiry, and 200 hours of engineering time to remediate.

Most teams don't know a financial controller. Most Staff-level payment engineers at least know which questions to ask.

"The policy management problem is harder than the technical problem — for most infrastructure systems."

Rate limiting's enforcement infrastructure is a 2-week project. Rate limiting's organizational governance — getting 15 product teams to agree on policies, keep them updated, respond when limits are hit, and not override the platform team's guardrails when their customers complain — is a 6-month project.

The same pattern appears in every shared infrastructure system. API gateway configuration: 2 weeks to build, 6 months to align 15 teams on how to use it. Circuit breaker configuration: 2 weeks to instrument, 6 months to agree on thresholds. Database connection pooling: 2 weeks to deploy, 6 months to get every team to update their connection strings.

In practice: product teams set rate limits too high because they fear blocking customers — the limits become theater. Nobody updates limits when product behavior changes. The platform team becomes a bottleneck for every policy change, or abdicates policy ownership entirely and the policies drift into chaos.

Staff engineers who ship the technical implementation and declare victory have shipped 25% of the project. The other 75% is the governance model, the self-service API, the review process, the canary rollout procedure, and the organizational escalation path when a policy change causes an incident. That's the work that determines whether the infrastructure is actually used correctly.

"Reconciliation is the product. The payment flow is just the happy path."

Candidates spend 40 minutes designing the authorize -> capture -> settle flow with meticulous care. They mention reconciliation in the last two minutes as "a batch job that runs daily."

The payment flow is the easy part. Stripe's API handles 80% of it. The hard part is what you discover in reconciliation: the charge that succeeded at the provider but failed in your system and generated no webhook, leaving a customer charged for something they didn't receive. The refund processed twice because a webhook arrived twice with different event IDs. The settlement amount that differs from the capture amount by $0.02 due to currency conversion rounding, which accumulates to $8,000 per month at $500M processing volume.

At $500M per month, a 0.01% discrepancy rate is $50,000 per month of unresolved money. That money exists somewhere — it is either in the customer's account (they were not charged for something they received), in the company's account (they were charged for something they didn't deliver), or at the provider (a settlement amount that doesn't match). The reconciliation pipeline is the system that finds it, classifies it, and resolves it.

The Staff engineer's job is not building the payment flow. It's building the system that catches and resolves the 0.01%. The payment flow designs itself. Reconciliation requires actual thought.

"Decentralized consumer ownership often fails — and nobody admits it until the incident."

Distributed consumer ownership sounds empowering. In practice, it produces: 15 different consumer frameworks, 15 different alerting setups, nobody who owns the broker when it misbehaves, and pager dilution when a topic accumulates lag at 3 AM.

The promise: teams own their domain. The reality: nobody owns the shared infrastructure. The broker, the topics, the schemas — these require coordinated ownership. When Team A scales their consumer and takes all the partitions, Team B's consumer starves. When Team C deploys a consumer bug and the dead-letter queue fills, the lag increase affects every consumer group, regardless of which team caused it.

Consumer ownership works when there is a platform team that owns the broker, topics, and schemas; a shared consumer library that provides built-in metrics, DLQ handling, and circuit breakers; and central observability with clear escalation paths. If you're seeing 15 different alerting setups and nobody can describe the escalation path for a broker outage, decentralization has already failed. The incident that makes it obvious is coming.

"Complexity that requires documentation to explain should be questioned, not documented."

When a new engineer joins and can't understand the system by reading the code, the instinct is to improve the documentation. This instinct is wrong.

Documentation is the last resort when the system can't explain itself. If a new engineer can't understand why the system works by reading the code and the architecture decision records, one of three things is true: the design is more complex than the problem requires, the team has accumulated context that never got written down (which means the system is held together by institutional knowledge that will leave with the people who have it), or the design has evolved without a coherent narrative — features were added to an original design that no longer applies.

The right response to a confused new engineer is not documentation. It is: "What specifically don't you understand? And is that thing worth keeping?" Complexity that requires documentation to explain is complexity that should be questioned. Some of it survives the questioning. Most of it doesn't.

"The most important engineering decision is usually made in the first 30 days — before anyone realizes it's a decision."

The database choice. The shard key, or lack thereof. The authentication model. The event streaming model. The deployment topology. These decisions are made in the first month of a project, when the team is small and moving fast and trying to build something that works, and they are almost never revisited — because by the time they're worth revisiting, they've been built around by 30 more decisions that now depend on them.

This is not a failure of the engineers who made the decisions. They made the best decisions they could with the information they had. It is an argument for being more deliberate about identifying which decisions are load-bearing before making them — and for investing a week in evaluating the decision rather than an afternoon.

The Staff engineer's contribution in the first 30 days is not moving the fastest. It is identifying which decisions are irreversible, slowing down on those specifically, and moving quickly on everything else. Knowing which is which is most of the job.

The Last Opinion

There is one more opinion, and it is the most important one in this book.

"Staff level is not a reward for being a very good Senior engineer. It is a different job."

The Senior engineer optimizes for implementation. They write excellent code, design reasonable systems, and ship reliably. This is genuinely valuable. But it is not the Staff job.

The Staff job is to optimize for outcomes — including the parts you didn't build, the teams you don't control, and the failures you didn't anticipate. It requires designing systems that are understandable by the teams who will maintain them, that fail in ways that are recoverable, that can be changed without requiring coordination across twelve teams, and that are owned by people who know what they're responsible for.

The failure patterns this book identified in Chapter 1 — the engineer who walked out convinced they nailed the interview — trace back to one thing: optimizing for appearing capable rather than thinking like someone who will own this system for three years.

Three years. That is the frame. Not "will this work in the demo." Not "will this satisfy the requirements as stated." Not even "will this handle 10x the current traffic." The frame is: if I am responsible for this system in three years — if I am on-call for it, if I have to explain its failure modes to a new engineer, if I have to change it when the business changes — is this the right design?

Almost every opinion in this book follows from that question. Reconciliation is more important than the payment flow because you'll be debugging reconciliation discrepancies for three years. The shard key is irreversible because you'll be living with it for five. The cache invalidation is an ownership problem because nobody who wrote the cache contract will be around in two years to explain it. Governance matters more than implementation because the implementation will be refactored; the governance model will not.

The Staff Decision Loop — intent, fault lines, who pays, make the call, stress-test — is not a framework for passing interviews. It is a framework for thinking about systems that you're going to own. When that becomes your natural operating mode, the interview performance follows. The reverse is never true.

You cannot fake three years of ownership into a 45-minute session. But you can demonstrate, in 45 minutes, that you think like someone who would.

That is the whole book.

STAFF CHECKLIST: The Opinions Worth Holding

• I can state a committed position on every major design decision in my domain — not "it depends," a position.
• When someone proposes sharding, I ask what simpler alternatives were tried first.
• When someone claims "exactly-once delivery," I ask exactly where the boundary is.
• When an incident traces to a cache, I ask who owned the invalidation contract — not what the TTL was.
• When a new engineer is confused by the system, I ask whether the system is too complex — not whether the documentation is too sparse.
• When evaluating a security investment, I ask whether it improves confidentiality, integrity, or availability — and what the relative risk is across all three.
• When designing any system, I ask: if I own this in three years, is this the right design?