October DealsAmazon USOctober deal check: compare before you payAmazon US: current deals, useful picks and tech finds.Check DealsPC HealthRecommendedCrashes, freezes, slowdowns? Check your PC nowSpot repairable issues before they interrupt work.Check PCOctober DealsAmazon USDeal season is back - check today's better picksAmazon US: current deals, useful picks and tech finds.See Picks×
Skip to content
World desk10 min

Building a C++ File Encryptor: Practical Cryptography and File I/O for Beginners

A beginner's guide to a C++ file encryptor built on OpenSSL EVP and AES-256-GCM, covering key derivation, binary file handling, the container format, and fail-closed decryption.

Free tools Windows power users keep installed

One-click scans. No signup required.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

A beginner’s C++ file encryptor should use OpenSSL’s EVP interface with AES-256-GCM, an authenticated mode that encrypts the data and also detects tampering. Derive the key from a password with PBKDF2 and a fresh random salt, store everything decryption needs in a versioned binary header and trailer, and release plaintext only after the authentication tag verifies. Treat the result as an educational project. It needs expert review before anyone uses it to protect sensitive data.

Decide what the program protects before writing code

Answer three questions in writing before you choose a cipher, because the answers change the design.

As an Amazon Associate I earn from qualifying purchases.

  • Who is the adversary? Someone who copies a file from a lost laptop or an unencrypted backup is a different threat from someone who can run code as your user account.
  • Who holds the key? A password-based tool suits one person encrypting their own files. Sharing with several recipients needs a key-distribution design (for example, public-key wrapping of a file key), which is a separate project from this one.
  • What happens if the password is lost? With a password-derived key and no escrow, the data is unrecoverable. The program’s help text and documentation should say so plainly.

Application-layer file encryption does not protect against a compromised process. Once the program is running as the user and holds the password or derived key in memory, malware in that session can read what the program reads. File encryption guards stored or copied data, such as a stolen disk or a copied backup. It does not protect a machine that is already compromised. The OWASP Cryptographic Storage Cheat Sheet is the baseline reference for these threat-model questions: OWASP, Cryptographic Storage Cheat Sheet.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Four values that beginners confuse

Most bugs in this kind of project come from mixing up four values that look alike. Each has a different job and a different reuse rule.

Value Source Secret? Stored in the file? Rule
Key Derived from the password and salt Yes No Never write it to disk or logs; overwrite it after use.
Salt Random, generated once per file No Yes, in clear Use a new salt for every file so each password-derived key is unique.
Nonce (IV) Random, generated for each encryption No Yes, in clear Never reuse with the same key under GCM.
Authentication tag Computed by GCM during encryption No Yes, after the ciphertext Release no plaintext until the tag verifies.

The salt belongs to the password-to-key step and the nonce belongs to the cipher. Storing them in clear is safe because the key, not the parameters, carries the secrecy.

Why AES-GCM instead of other modes

OWASP recommends AES with at least a 128-bit key, ideally 256-bit, and a secure mode. It prefers authenticated modes such as GCM or CCM when they are available, and it says a separate authentication step is required if you fall back to CBC or CTR (OWASP, Cryptographic Storage Cheat Sheet). The practical comparison looks like this:

Mode Built-in integrity IV or nonce rule Fit for this project
ECB No No IV Do not use for file encryption.
CBC No; needs a separate MAC Unpredictable IV for each message Avoid here. Composing a MAC correctly is easy to get wrong.
CTR No; needs a separate MAC Unique nonce per key; reuse exposes plaintext Avoid here for the same reason as CBC.
GCM Yes, with a 16-byte tag 96-bit nonce; never reuse with the same key Recommended. Works with chunked EVP updates.
CCM Yes Nonce length fixed at setup An authenticated alternative, but OpenSSL’s CCM needs the total message length before data is processed, which makes streaming awkward.

OWASP’s Cryptographic Storage Cheat Sheet handles homemade designs under its Custom Algorithms heading with one line: “Don’t do this.” The sentence has no named author, so cite it to the cheat sheet itself. Do not invent a cipher, and do not treat unauthenticated encryption as complete protection.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Choosing the library and setting up the build

Use OpenSSL’s high-level EVP interface rather than cipher-specific or deprecated low-level calls. The AES variants, including GCM, are listed in the OpenSSL EVP_CIPHER-AES 3.0 manual page, and the AEAD encrypt and decrypt behavior is described in the OpenSSL EVP_EncryptInit 3.1 manual page. Check the manual that matches your installed release before you rely on any function signature.

sudo apt install libssl-dev
g++ -std=c++17 -Wall -Wextra -o efile main.cpp -lcrypto

The package name is for Debian and Ubuntu; other distributions use different names. The -lcrypto flag links the OpenSSL crypto library.

Deriving the key from a password

A password is not a raw AES key. Turn it into one with a password-based key derivation function and a random salt. OpenSSL’s EVP_BytesToKey 3.1 manual page says newer applications should use PBKDF2 instead, so do not use EVP_BytesToKey in new code.

  1. Generate a 16-byte salt with RAND_bytes(salt, 16) and check that it returns 1. The RAND_bytes manual page, 1.0.2 version, describes the return status; confirm the current behavior in the manual for your release.
  2. Call PKCS5_PBKDF2_HMAC(password, password_len, salt, 16, iterations, EVP_sha256(), 32, key) to produce a 32-byte key for AES-256, and check its return value.
  3. Store the KDF identifier and the iteration count in the file header so the default can change later without breaking old files.
  4. Overwrite the password and key buffers with OPENSSL_cleanse when you finish.

Choose the iteration count by benchmarking on the machine that will run the tool and following current published guidance. Do not copy a number from an older tutorial; a value that was reasonable years ago may now be too low.

What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Designing the container format

Serialize the header field by field, with explicit sizes and big-endian integers. Do not write a C++ struct to disk, because padding and byte order vary across platforms and compilers. The layout below is a design suggested for this project. The OpenSSL and C++ references do not define a container format.

Field Bytes Contents
Magic 4 ASCII AEF1, identifying the file type
Version 1 Value 1; readers reject any other version
KDF ID 1 Value 1 means PBKDF2-HMAC-SHA256
Iterations 4 Unsigned big-endian iteration count
Salt 16 Random, generated per file
Nonce 12 Random, generated per encryption
Ciphertext Variable Written in chunks
Tag 16 GCM tag, written after the last ciphertext byte

Feed the serialized header to GCM as additional authenticated data. The header is then not encrypted but is covered by the tag, so any edit to it fails verification even in fields that do not change the key. Because the tag sits at the end of the file, a decryptor must confirm the file is long enough for a header and a tag, and read the last 16 bytes before it streams anything.

Encrypting: the EVP call sequence

  1. Generate the salt and nonce with RAND_bytes, check both return values, and derive the key as described above.
  2. Create a context with EVP_CIPHER_CTX_new() and check for NULL.
  3. Select the cipher with EVP_EncryptInit_ex(ctx, EVP_aes_256_gcm(), NULL, NULL, NULL).
  4. Set the key and nonce with EVP_EncryptInit_ex(ctx, NULL, NULL, key, nonce). The 12-byte nonce is GCM’s default length, so no extra length call is needed.
  5. Pass the header as associated data with EVP_EncryptUpdate(ctx, NULL, &outlen, header, header_len). This call produces no output.
  6. For each input chunk, call EVP_EncryptUpdate(ctx, out, &outlen, in, inlen) and write exactly outlen bytes. Size the output buffer to at least the input length plus EVP_MAX_BLOCK_LENGTH. For an empty file, skip this loop; the final step still produces a tag.
  7. Call EVP_EncryptFinal_ex(ctx, out, &outlen), check the return value, and write any final output.
  8. Retrieve the tag with EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_GET_TAG, 16, tag). This must happen after the final call.
  9. Write the tag, flush, and close the file, checking each step. Then call EVP_CIPHER_CTX_free(ctx) and OPENSSL_cleanse on the key.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Decrypting and failing closed

Decryption is where most security bugs appear. The rule is simple: any failure in parsing, key derivation, I/O, or authentication leaves no output file behind.

  1. Check that the file is at least header size plus 16 bytes, then read the trailing 16-byte tag.
  2. Parse the header. Check the magic, version, KDF ID, and iteration limit, and reconstruct the header bytes exactly as they were written. Reject bad values before you allocate memory or derive a key.
  3. Derive the key from the stored salt and iteration count.
  4. Run EVP_DecryptInit_ex(ctx, EVP_aes_256_gcm(), NULL, NULL, NULL), then EVP_DecryptInit_ex(ctx, NULL, NULL, key, nonce).
  5. Pass the reconstructed header with EVP_DecryptUpdate(ctx, NULL, &outlen, header, header_len).
  6. Create a temporary file in the same directory as the output path. For each ciphertext chunk, call EVP_DecryptUpdate(ctx, out, &outlen, in, inlen) and write the result to the temporary file.
  7. Set the expected tag with EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_TAG, 16, tag). This must happen before the final call.
  8. Call EVP_DecryptFinal_ex(ctx, out, &outlen). Treat any return value other than 1 as an authentication failure. Close and delete the temporary file, print one generic message such as “Decryption failed,” and exit with a nonzero status. The OpenSSL 3.1 manual describes failed finalization as an authentication failure whose output must not be used.
  9. On success, flush and close the temporary file with checks, then call std::filesystem::rename to move it onto the final name. Renaming within one directory avoids cross-device failures.

Writing to a temporary file has a trade-off. Unverified plaintext sits on disk briefly, and a crash can leave it there. Holding the whole output in memory avoids that exposure but works only for small files. For a learning project, the temporary-file route is the safer default, and the limitation should be documented.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Report every authentication and parse failure with the same message. Distinguishing a wrong password from a corrupted file tells an attacker which guess was close enough to pass the parser.

Reading and writing binary files safely

  • Open in binary mode. Use std::ifstream in(path, std::ios::binary) and std::ofstream out(path, std::ios::binary), then check that the stream opened. The cppreference basic_ifstream constructors page documents the open behavior.
  • Check every read. After in.read(buf, n), use in.gcount() to learn how many bytes arrived, and treat in.bad() as an I/O error. A short read near the end of the file is normal; a short read in the middle of the expected data means the file is truncated.
  • Check every write, flush, and close. Test the stream after out.write. Call out.close() explicitly and check the stream again, because buffered data can fail at that point and a destructor cannot report the error.
  • Do not trust the file size. std::filesystem::file_size reports the size of regular files only. Its error-code overload returns an error rather than throwing, and the cppreference file_size page documents this behavior. Use the size to locate the tag and to reject impossible lengths, but never to size a single allocation.
  • Use bounded chunks. A 64 KiB chunk is a reasonable starting value. It is a choice for this project, not a requirement, and it keeps memory use independent of file size.
  • Enforce limits before allocating. Cap the iteration count and any length fields, and reject values outside those caps before you derive a key or reserve memory.

Key and password handling

  • Do not put a password or key in source code. Compiled binaries and committed repositories expose hardcoded secrets.
  • Do not accept the password as a command-line argument or in an environment variable. Shell history, process listings, and the environment of the same user can expose them.
  • Read the password interactively with echo turned off: termios on POSIX systems, or the console API on Windows.
  • For a real deployment, consider an operating-system key store or a managed key service. Write the threat model first, because the right store depends on who must be prevented from reading the key.
  • Clear sensitive buffers with OPENSSL_cleanse. Note that clearing memory is best effort; the operating system may already have copied pages elsewhere.

Test the failure paths, not just the happy path

Round-trip tests prove only that the encoder and decoder agree. The failure cases prove the program fails closed. Build each case into an automated test suite.

Case Expected result
Round trip of an empty file, a one-byte file, a file of all 256 byte values, and a multi-chunk file Output matches the input byte for byte.
One ciphertext byte flipped Decryption fails; no output file exists.
One tag byte flipped Decryption fails; no output file exists.
One byte changed in the salt, nonce, or iteration field Decryption fails; no output file exists.
Wrong password The same generic failure as tampering.
File truncated inside the header, inside the ciphertext, or inside the tag Rejected before any plaintext is released.
Wrong magic, unknown version, or iteration count above the cap Rejected before key derivation.
Output directory not writable, or disk full during writing Error reported; no partial output remains.
Multi-gigabyte file Memory use stays near the chunk size plus fixed buffers.

Next steps for the project

  • Run the test suite under AddressSanitizer and UndefinedBehaviorSanitizer by compiling with -fsanitize=address,undefined.
  • Fuzz the header parser with malformed and truncated inputs, since it is the first code that touches untrusted bytes.
  • Read the OpenSSL manual for your installed release and confirm each function signature used above against it.
  • Treat multi-recipient sharing as a separate design, because it adds key wrapping and key distribution that this format does not include.

”

The Bottom Line

“”

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

Leave a Reply

Your email address will not be published. Required fields are marked *

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

More from the Wire

  1. World desk4 min
    How to Spot an AI Voice Scam Before Sending MoneyDon’t rely on how a caller sounds. Pause, call back through a known number, and verify the emergency with another trusted person before sending money.
  2. Mountain View desk4 min
    Google’s SynthID Detector: How to Check AI-Generated Images, Video and AudioGoogle’s SynthID Detector looks for an embedded watermark in supported images, video and audio. Here is what its results do—and do not—show.
  3. Redmond desk20 min
    How to create a link to File or Folder in Windows 11Windows 11 gives you several ways to point to a file or folder without moving or duplicating it. You can create a desktop shortcut,…
Recommended PC Tool
Recommended PC Tool
Windows Errors? Fix Them Before They SpreadFree repair scan
Crashes, No Sound, or Screen Glitches?Free driver scan

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.