Security 101

Security is hard to add to a system after it's built. It decides how every request is authenticated, what each caller is allowed to do, how data is protected on the wire and on disk, and what happens when one of those defenses fails. A fast, scalable system that leaks user data, or falls over under the first denial-of-service attack, has still failed.

The primer's security section is a short list of concepts with links to checklists. This chapter covers the pieces that show up in system design: how users and services prove who they are (sessions, JWTs, OAuth 2.0, OpenID Connect, mTLS), how access is decided (RBAC and ABAC), what the edge and the API gateway defend against, how data is encrypted with keys held in a KMS, and the common application bugs that undo all of it.

It builds on the primer's security, application layer, reverse proxy and communication chapters, and on reverse proxy and API gateway 101.

Authentication and identity

Authentication answers who is making this request? In a distributed system there are two kinds of caller: people using a browser or mobile app, and services calling other services. They are usually handled differently.

Sessions vs tokens

When a user logs in, the system checks their credentials, such as a password plus a second factor. HTTP doesn't remember anything between requests, so the system then gives the client something to send with every later request that proves it already logged in. There are two common designs.

A JWT (JSON Web Token) has three base64url-encoded parts separated by dots: a header saying which algorithm signed it, a payload of claims (sub for the user ID, exp for expiry, aud for the intended audience, and so on), and the signature. The payload is only encoded, not encrypted, so anyone holding the token can read it. Never put secrets in it.

Most production systems combine the two. They use a short-lived access token (valid 5 to 15 minutes) that APIs check without any lookup, and a long-lived refresh token (days to weeks) that is stored and checked server-side. When the access token expires, the client sends the refresh token to the authorization server. That server checks that the account is still active and the refresh token hasn't been revoked, then issues a new access token. Many servers also rotate the refresh token on each use and treat reuse of an old one as a sign of theft.

So a banned account keeps access for at most one access-token lifetime, a few minutes, while ordinary requests never touch the token store.

Federated identity: OAuth 2.0 and OpenID Connect

Most systems don't want every component handling passwords. They hand that job to a dedicated identity provider, using two standards that are often confused:

• OAuth 2.0 is a framework for delegated authorization. It lets an application get limited access to a user's data on another service without ever seeing the user's password. For example, a photo-printing site can read your photos from a cloud photo service. The user approves the request at the provider, and the application gets an access token limited to certain scopes (such as "read photos") to call the provider's API with. OAuth 2.0 on its own says nothing about who the user is.
• OpenID Connect (OIDC) is an identity layer on top of OAuth 2.0 that adds authentication. Besides the access token, the provider returns an ID token: a signed JWT with claims about the login, including iss (who issued it), sub (a stable user ID), aud (which application it's for), exp, and optionally profile claims like email. "Sign in with Google" is OIDC. The ID token is meant for the application that requested it, to learn who logged in. It is not meant to be sent to APIs, which take access tokens.

For browser and mobile apps, the recommended flow is the authorization code flow with PKCE (Proof Key for Code Exchange, RFC 7636). The steps:

1. The app generates a random secret, the code_verifier, and computes code_challenge = BASE64URL(SHA-256(code_verifier)).
2. It sends the user to the provider's login page with the code_challenge.
3. After the user logs in and approves, the provider redirects back to the app with a short-lived authorization code.
4. The app exchanges the code for tokens, sending the original code_verifier. The provider hashes it and checks it matches the challenge from step 2.

Someone who intercepts the code in step 3 can't use it, because they don't have the verifier. The current OAuth security best practice (RFC 9700) recommends PKCE for every client and says not to use the old implicit flow, which returned tokens directly in the redirect URL.

Service-to-service identity: mutual TLS

Inside a private network or Kubernetes cluster, services also need to know who is calling them. Trusting a request because it comes from an internal IP address is weak: an attacker who takes over one container can then call anything, and IP addresses change constantly anyway.

In ordinary TLS only the server proves its identity. With mutual TLS (mTLS), both sides do, during the TLS handshake:

1. Service A connects to service B. B sends its X.509 certificate, as in normal TLS.
2. B asks for a client certificate, and A sends its own.
3. Each side checks that the other's certificate is signed by a trusted internal certificate authority (CA) and hasn't expired. Each also proves it holds the private key matching its certificate, by signing part of the handshake.
4. Both derive fresh session keys, and the rest of the connection is encrypted.

Now B knows exactly which service is calling, and can apply rules like "only the checkout service may call payments". Managing certificates by hand for hundreds of services doesn't work, so in practice a service mesh does it. Each service's sidecar proxy gets a short-lived certificate encoding the service's identity (often a SPIFFE ID such as spiffe://cluster.local/ns/shop/sa/checkout), renews it automatically, and handles the handshake. The application itself speaks plain HTTP to its local sidecar. See reverse proxy and API gateway 101.

Authorization: access control models

Once you know who the caller is, authorization answers may this caller do this action on this resource?

Role-based access control (RBAC)

RBAC grants permissions to named roles and gives roles to users. A user with the editor role gets documents:read and documents:write, for example.

RBAC is simple to build and easy to audit ("who can delete invoices?" is a lookup), but it struggles once rules depend on context:

• An editor can edit documents, but should they be able to edit another department's documents?
• What about only during working hours, or only from a managed laptop?

Encoding each of these as a role leads to role explosion: hundreds of roles like editor_finance_us_readonly_weekend that nobody can keep track of.

Attribute-based access control (ABAC)

ABAC decides using policies over attributes, usually in four groups:

• Subject: who is asking (user ID, department, role, clearance).
• Action: what they want to do (read, write, delete, approve).
• Resource: what they want to do it to (document ID, owner, department, classification).
• Environment: the context (time, network, device posture).

A policy might say: allow users in the Finance department to edit Budget documents whose region matches the user's region, from a managed device. One rule like that replaces dozens of roles. The cost is that "who can access this?" is harder to answer, because the answer depends on attributes at request time.

In practice the two mix: roles are just one attribute among many. For "users can edit documents shared with them" style rules, many systems use relationship-based access control (ReBAC), which checks a graph of who owns or was shared what. Google's Zanzibar paper describes the best-known example.

Larger systems split the work in two (the terms come from the XACML standard):

• The policy enforcement point (PEP) sits in the request path, at the gateway, a sidecar, or inside the service. It collects the attributes, asks for a decision, and then allows or blocks the request.
• The policy decision point (PDP) evaluates policies and returns a decision, typically allow or deny, sometimes with extra conditions. Open Policy Agent (OPA) is a common general-purpose PDP. Its policies are written in a language called Rego and kept in version control like code.

Keeping policy in one engine means rules are written once, tested, and changed without redeploying every service. PDPs are usually run next to the PEP (as a sidecar or library) so a decision doesn't add a network round trip to every request.

Network and boundary security

The guiding principle is defense in depth: put independent controls at each layer (the edge, the network, the application, the data), so that one control failing doesn't expose everything. The WAF misses an injection, but the query is parameterized. An attacker gets a database backup, but it's encrypted with a key they can't reach.

Edge, WAF and API gateway

Before traffic reaches application servers, it passes through several layers, each stopping a different kind of attack:

1. DDoS protection at the edge. CDN and edge providers (Cloudflare, Amazon CloudFront with AWS Shield, Fastly, Akamai) absorb volumetric attacks at layers 3 and 4, such as UDP floods and SYN floods. Their capacity is spread across many locations by Anycast, so the flood is split up long before it reaches you. See DNS and CDN 101.
2. Web application firewall (WAF). Inspects HTTP requests at layer 7 and blocks known attack patterns: SQL injection strings, cross-site scripting payloads, exploit signatures such as Log4Shell, and malformed requests. A WAF is a filter for known patterns, not a fix. The application still has to be safe on its own.
3. API gateway. Enforces rate limits per IP address, API key or user, authenticates callers, rejects requests that don't match the API's schema, caps body sizes, and strips headers that clients aren't allowed to set, such as internal identity headers.

Zero trust

The older model was castle and moat: everything outside the firewall is untrusted, everything inside is trusted. Its weakness is that one compromised laptop or container puts the attacker inside, with access to everything.

Zero trust (described in NIST SP 800-207) assumes the attacker may already be inside:

• Being on the internal network grants nothing. Every request is authenticated and authorized, service-to-service calls included, which is where mTLS comes in.
• Access follows least privilege: each user and service gets only the permissions its job needs, so one compromised credential has a small blast radius.
• Decisions can use context such as device health, and are made per request, not once per login.

Data protection: encryption and key management

Data needs protection in two states: moving across a network (in transit) and stored on disk (at rest).

In transit: TLS 1.3

TLS gives confidentiality, integrity and server authentication between two endpoints. TLS 1.3 (RFC 8446, 2018) is the current version. Compared with TLS 1.2 it brings:

• A faster handshake. A full handshake takes one round trip before application data can flow, instead of two in TLS 1.2. A client resuming a recent session can send data in its very first flight (0-RTT). That early data can be replayed by an attacker, though, so servers should accept it only for requests that are safe to repeat, such as a GET.
• Only modern cryptography. It removed RSA key transport, CBC-mode ciphers, compression and renegotiation. Every handshake uses an ephemeral Diffie-Hellman exchange, which gives forward secrecy: if the server's long-term private key is stolen later, previously recorded traffic still can't be decrypted, because the session keys were never derivable from it.

Terminate TLS at the edge for internet traffic, and use TLS or mTLS again on internal hops. Leaving traffic unencrypted inside the network is what zero trust tells you not to do.

At rest: envelope encryption with a KMS

Encrypting data with a key stored on the same disk, or in the app's config file, protects very little: whoever gets the data probably gets the key too. Production systems use envelope encryption, which separates two kinds of key:

1. Data encryption key (DEK). A key, typically 256-bit AES, used to encrypt the actual data: a file, a database record, a disk volume. Each piece of data, or each tenant, can have its own.
2. Key encryption key (KEK). A master key that lives inside a key management service (KMS), such as AWS KMS, Google Cloud KMS, Azure Key Vault or HashiCorp Vault's transit engine, usually backed by hardware security modules (HSMs). The KEK never leaves the KMS. Its only job is encrypting and decrypting DEKs.

Writing data:

1. The application asks the KMS for a new data key under a named master key. In AWS KMS this call is GenerateDataKey.
2. The KMS returns the DEK twice: in plaintext, and wrapped (encrypted under the KEK).
3. The application encrypts the data locally with the plaintext DEK, stores the ciphertext together with the wrapped DEK, and discards the plaintext DEK from memory.

Reading it back:

4. The application sends the wrapped DEK to the KMS (Decrypt). The KMS checks the caller's permissions, unwraps it with the KEK, and returns the plaintext DEK.
5. The application decrypts the data locally.

Why this design:

• Bulk encryption stays local. The KMS only ever handles 32-byte keys, not your gigabytes. AWS KMS won't even encrypt more than 4 KB directly. Your data never leaves your servers, and KMS request rates stay manageable. Applications often cache plaintext DEKs briefly to cut KMS calls further.
• Access is controlled and logged in one place. Every unwrap is a KMS call, checked against IAM policy and recorded in an audit log, so you can see and revoke who can decrypt what.
• Rotation is cheap. Managed KMS rotation keeps old versions of the master key, so existing wrapped DEKs still decrypt and nothing has to be re-encrypted. If you move to a completely new master key, you re-wrap the small DEKs (AWS has a ReEncrypt call for this), not the data.

Secrets management

Database passwords, API keys and private keys committed to source code or baked into container images are a common cause of breaches: they end up in git history, CI logs and image registries. Keep them in a secrets manager instead (HashiCorp Vault, AWS Secrets Manager, Google Secret Manager). Kubernetes Secrets are only base64-encoded unless you turn on encryption at rest, ideally with a KMS provider.

• Dynamic secrets. Instead of one database password shared by every instance, Vault can create a separate database login for each instance, with a lease that expires after a set time. A leaked credential stops working on its own, and the logs show which instance it belonged to.
• Rotation. Long-lived secrets are rotated on a schedule. Applications should fetch secrets at runtime and reload them when they change, so rotation doesn't need a redeploy.
• Auditing. Every read is logged with the identity of the caller.

Common application vulnerabilities and mitigations

Strong architecture is undone by a single injectable query. These are the bugs that keep appearing in the OWASP Top 10, and the standard defenses.

Injection (SQL, command)

Injection happens when untrusted input is pasted into a query or command, so the input can change what the query means.

• The bug: building SQL by string concatenation, as in "SELECT * FROM users WHERE email = '" + input + "'". An input of ' OR '1'='1 turns it into a query that matches every user.
• The fix: parameterized queries (prepared statements), or an ORM or query builder that uses them. The query's structure is fixed first and the input is sent separately as data, so it can't change the structure. Table and column names can't be parameters, so if they come from users, check them against a fixed allowlist. For shell commands, pass arguments as an array instead of building a command string.

Cross-site scripting (XSS)

XSS happens when an app puts untrusted data into a page without escaping it, letting an attacker run JavaScript in other users' browsers.

• Impact: the script runs as the victim on your site. It can read anything on the page, make requests with the victim's session, and steal tokens it can reach.
• Mitigations:
  • Escape output for its context (HTML body, attribute, URL, JavaScript). Modern templating frameworks do this by default. The danger is in the escape hatches, like raw-HTML helpers.
  • Keep session tokens in HttpOnly cookies so scripts can't read them through document.cookie. This limits the damage, though injected script can still act as the user while the page is open.
  • Send a strict Content Security Policy (CSP) header. A policy that allows scripts only from your own origin, or only with a per-response nonce, and doesn't include 'unsafe-inline', stops most injected scripts from running.

Cross-site request forgery (CSRF)

CSRF tricks a logged-in user's browser into sending a request to your site from another site, for example a hidden form that posts a money transfer. The browser attaches the user's cookies, so the request looks genuine.

• Mitigations:
  • Set SameSite on session cookies. With SameSite=Lax, the browser doesn't send the cookie on cross-site POSTs or on requests embedded in another site (images, iframes, fetch). It still sends it when the user follows a plain link to your site, so GET requests must never change state. SameSite=Strict doesn't send it on any cross-site request, which is safer but means users arriving from a link appear logged out. Most browsers now treat cookies without a SameSite attribute as Lax.
  • Use anti-CSRF tokens: a random value tied to the session, put in each form or sent in a custom header like X-CSRF-Token, and checked by the server on every state-changing request. Another site can't read the token, so it can't forge the request.
  • APIs that authenticate with a bearer token in the Authorization header aren't exposed to CSRF in the first place, because browsers don't attach that header automatically.

Server-side request forgery (SSRF)

SSRF happens when a server fetches a URL supplied by a user, such as a link preview or an image import, and doesn't restrict where it can go.

• Impact: the attacker makes your server call things only it can reach, like http://127.0.0.1:8080/admin or the cloud metadata service at http://169.254.169.254/, which on a misconfigured instance hands out temporary cloud credentials. This was the core of the 2019 Capital One breach.
• Mitigations:
  • Run URL fetching in an isolated service whose network egress is limited to the public internet.
  • Resolve the hostname, then refuse private and special addresses: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, loopback 127.0.0.0/8, link-local 169.254.0.0/16, and their IPv6 equivalents (::1, fc00::/7, fe80::/10). Connect to the IP address you checked, not the name, so a DNS answer that changes between your check and your request (DNS rebinding) can't get around the filter. Apply the same check to every redirect.
  • Protect the metadata endpoint itself. On AWS, require IMDSv2, which only hands out credentials after a PUT request that returns a session token, and by default allows only one network hop, so a simple GET forwarded through a vulnerable app can't reach it.

Check yourself

Further reading

• RFC 6749: The OAuth 2.0 Authorization Framework: the core OAuth 2.0 specification.
• RFC 9700: Best Current Practice for OAuth 2.0 Security: today's guidance, including PKCE for every client and dropping the implicit flow.
• OpenID Connect Core 1.0: the ID token and the authentication flows built on OAuth 2.0.
• RFC 8446: The Transport Layer Security (TLS) Protocol Version 1.3: the handshake, forward secrecy, and the replay caveats of 0-RTT data.
• AWS KMS concepts: KMS keys, data keys and envelope encryption as one provider implements them.
• NIST SP 800-207: Zero Trust Architecture: the reference description of zero trust.
• OWASP Top Ten and the OWASP Cheat Sheet Series: the most common web application risks, and practical guidance on preventing each.