End-to-End Encryption vs Encryption at Rest

A new feature on a project I've worked on for years needed one to one messaging between users. Before any code got written, one question came up: can our own team read these messages if a client asks us to?

The default answer was going to be yes. The platform already encrypts data at rest, keys managed through Vault, rotation automated, every box a security questionnaire asks about ticked. That default is good enough for most data. It is not the same promise as end to end encryption, and messaging is exactly the kind of data where the difference matters.

This is the design I worked through once "encrypted at rest" stopped being a good enough answer.

Encryption at rest is not end-to-end encryption

Encryption at rest solves a specific problem: if someone steals a hard drive, dumps a database, or gets hold of a backup file, they get ciphertext, not your data. That is a real threat and worth defending against. Tools like HashiCorp Vault or a cloud KMS do this well, and if you already have key rotation and access-controlled secrets, you are ahead of most teams.

What it does not solve: your own application, running with its normal credentials, can still decrypt the data. So can anyone with access to that application's environment: a developer with production access, a support engineer debugging a ticket, an attacker who compromises your app server rather than your disk. Encryption at rest changes who can read your data if they steal a file. It does not change who can read your data if they use your own system the way it is meant to be used.

That distinction gets lost constantly, including on security questionnaires that ask "is data encrypted" and accept a yes without asking who holds the key.

The one-line test for real end-to-end encryption

Here is the test I use now: can someone with full access to your database and your application servers still not read the content? If the answer is no, whatever you have built is encryption at rest with extra steps, not end to end encryption. True end to end encryption means the plaintext only ever exists on the two ends of the conversation, the sender's device and the recipient's device. Everything in between, including your own servers, only ever handles ciphertext.

That is a genuinely different, harder guarantee to build. It is also the one the client was actually asking for.

The core pattern: a public key, a private key, and a key that locks the key

The pattern is the same one every real end to end encrypted app uses, and it generalizes past messaging the moment you see the shape of it.

Every user gets a keypair, a public key and a private key, generated on their device. The public key gets stored in plain text in your database. Anyone can see it, that is the point of a public key, it is what lets someone else encrypt something only that user can read.

The private key is the part that matters. It cannot sit in your database in plain text, or you are back to encryption at rest with a different name. So instead of storing it directly, you encrypt it first, using a key derived from a secret only the user has. I call that a locking key: not the user's actual secret, but a key produced from it, strong enough to encrypt the real private key.

Once that pattern exists, sending a message is simple. The sender's client fetches the recipient's public key, encrypts the message locally, and sends only ciphertext to the server. The server relays it and stores it. It never has anything capable of decrypting it, because it never has the private key, only an encrypted version of one.

Swap "message" for "uploaded document" or "form response" and the pattern does not change. The only thing that changes is what gets encrypted before it leaves the device.

Where the private key actually lives, and why that's the whole game

There are three real places to put that locked private key, and the choice decides whether you actually built end to end encryption or just moved the encryption-at-rest problem one layer down.

Server/KMS holds the key. Your server can technically decrypt it. Lowest engineering cost, but this is encryption at rest, not E2E.

Password-wrapped key. Only the user can decrypt it, via their secret. Moderate cost, the practical middle ground.

Per-device keys (Signal/WhatsApp model). Only the user's specific devices can decrypt it. Highest cost, strongest guarantee, real multi-device complexity.

The first option is tempting because it is the one your existing secrets infrastructure already does well. Wrap the private key with a KMS-managed key, decrypt it server-side whenever a message needs sending or reading. It is easy to build, and it fails the test above immediately: your server can decrypt it, which means your server can read the message. I ruled this out for exactly that reason. It relocates the promise you are trying to keep, it does not keep it.

The third option is what Signal and WhatsApp actually run: every device, not every account, gets its own keypair, generated locally, private key never transmitted anywhere in any form. It is the strongest guarantee available, and it is genuinely heavy to build. Linking a new device becomes a key exchange event, not a login. Sending a message means encrypting a separate copy for every device the recipient owns. For a one to one messaging feature added to an existing product, that is more infrastructure than the feature justifies.

The second option, a password-wrapped key, is where I landed. It keeps the guarantee real, the server never holds anything it can decrypt on its own, while staying close to the account model most products already have.

The password-change problem nobody thinks about first

The obvious first instinct is to derive the locking key directly from the password. Do not do this. The moment a user changes their password, that locking key changes too, and now it cannot open a private key that was locked with the old one.

The fix is to treat the private key and its lock as two separate things that change at different times. The private key itself is generated once and never changes. Only its wrapper does. On a password change: unlock the private key with the old password's locking key, generate a new locking key from the new password, re-wrap the same private key with it. One small re-encryption, not a rewrite of message history. Every message stays exactly as encrypted as it was, because it was always encrypted to the public key, and that never moved.

SSO breaks the model unless you plan for it

Here is the gap that does not show up until you have real enterprise clients: SSO users never give your application a password. Authentication is delegated entirely to their identity provider. Your app never sees a secret to derive a locking key from, because there is not one.

The fix is to stop tying the locking key to login entirely. Every user, SSO or password, sets up a separate passphrase or PIN the first time they use the encrypted feature, used only for that purpose. It is one more secret to manage, and I do not love adding it, but it is what keeps the guarantee identical for every user regardless of which identity provider they log in through.

The tempting shortcut is to fall back to the KMS-held key just for SSO users. Resist that. It means some users get true end to end encryption and others get encryption at rest, silently, depending on how they log in. That inconsistency is a much harder conversation to have in a compliance review than the extra passphrase is to build.

What happens when someone forgets the passphrase

There is no way around this one: if there is no backdoor, there is no password reset that also recovers old data. Forget the passphrase, and the private key it was protecting is gone, along with everything encrypted to it.

The standard answer, the same one WhatsApp uses for encrypted backups, is a recovery code. Generate a second, high-entropy secret at setup time, wrap a second copy of the same private key with it, show the code to the user exactly once, and tell them plainly to save it somewhere safe. Never store the code itself, only what it unlocks. If the passphrase is forgotten, the recovery code becomes the way back in, after which you issue a new one and retire the old.

It does not close the gap completely. Lose both the passphrase and the recovery code, and the data is genuinely gone. That is not a bug to fix, it is the tradeoff you are making by building something real instead of something that just looks like end to end encryption.

Where the unlocked key lives on the client

One more decision that is easy to get wrong: once you have unlocked the private key, where do you keep it so the user is not typing their passphrase on every page load?

The honest answer is you mostly do not solve this, you accept it. Keep the unlocked key in memory only, gone on refresh, gone on tab close. That is exactly how Bitwarden and Signal's web client behave, and it is not a corner they cut, it is the correct tradeoff. Anything durable enough to survive a refresh, a cookie, local storage, session storage, is also durable enough for a cross-site scripting bug to read straight out of.

If refresh survival genuinely matters for your product, the Web Crypto API lets you import a key as non-extractable and hold it in IndexedDB. The key survives a refresh and can still be used for encryption and decryption, but no script, including an attacker's, can ever read the raw bytes back out. It is more work to build. I would only reach for it after confirming the simpler option is actually costing you users.

Applying this beyond messages

None of this is specific to chat. The pattern, a keypair per user, a locking key derived from a secret, a wrapped private key, applies to any field where the honest answer to "can your team read this" needs to be no: uploaded documents, form responses, anything a compliance questionnaire singles out by name.

The cost that shows up immediately once you generalize is search. A database field encrypted this way cannot be queried, filtered, or indexed the normal way, because your server never has the plaintext to index in the first place. That is a real architectural cost, not a footnote, and it is the main reason not to reach for this pattern everywhere by default. Internal analytics fields, anything your own team needs to read to do its job, are usually better served by encryption at rest. Save the heavier pattern for the data where the answer genuinely has to be no.

The questions to answer before you build this

• Does compliance actually require "cannot read," or does "encrypted at rest" already satisfy what you were asked?
• Does this data need to cross a boundary you have already promised is isolated, for example between two separate client organizations?
• What recovery tradeoff is the business willing to accept: lost data on a forgotten secret, or a weaker guarantee?
• Does your auth model include SSO, and if so, have you planned the separate passphrase step, or are you about to discover this gap in production?
• Does anything downstream, search, reporting, support tooling, need to read or query this field in plaintext?

If the answer to that last one is yes for a field you were about to encrypt this way, that is usually the sign the field does not need true end to end encryption. Encryption at rest already covers it.

What I'd tell you

End to end encryption is not a feature you turn on, it is a set of tradeoffs you accept on purpose: no password reset without a recovery code, a second secret for every SSO user, fields you can no longer search server-side. None of that is a reason to avoid it. It is a reason to be precise about which fields actually need it, and honest with anyone who asks whether your team can read their data, instead of letting "it's encrypted" answer a question it was never actually answering.

FAQ

Is encryption at rest the same as end-to-end encryption? No. Encryption at rest protects data if someone steals a disk or a backup, but your own application can still decrypt it. End to end encryption means only the sender and recipient's devices ever hold the plaintext, your servers never can.

Can you add end-to-end encryption to data that also needs to be searched? Not in the normal way. Once a field is encrypted client-side, your server has no plaintext to index or query. Plan for this before encrypting a field, not after.

What happens if a user loses both their passphrase and recovery code? The data is unrecoverable. There is no backdoor by design, that is what makes the encryption real, so this tradeoff has to be a documented product decision, not a surprise.

Does end-to-end encryption work with single sign-on? Yes, but not by deriving anything from the SSO login itself, since your app never sees a password. Use a separate passphrase or PIN set up specifically for the encrypted feature.

Is this the same thing Signal or WhatsApp do? Related but heavier. Signal and WhatsApp give every device its own keypair with no password involved at all. The password-wrapped approach here trades some of that strength for a much simpler fit with a normal account-based product.