Data Cleaning Microservice: FastAPI, pandas, and Docker for CSV Files

You can build a working CSV cleaning API with FastAPI receiving the upload, pandas parsing and cleaning it, and Docker packaging the result, in roughly one small application file plus a Dockerfile. The part that determines whether the service is trustworthy is not the code but the contract: which columns are required, how missing values are treated, what the caller gets back when a file is wrong, and how large an upload may be. This guide makes each of those decisions explicit, then shows the implementation that follows from them.

Define the API contract before writing any cleaning logic

FastAPI and pandas document the mechanics of uploads and parsing, but neither decides what your service should accept or promise. The FastAPI guides do not set upload size limits, and the pandas and Docker documentation do not prescribe a cleaning policy. Write these decisions down first, because every line of code below follows from them.

#	Preview	Product	Price
1		Data Cleaner: Techniques et méthodes de préparation des données pour un reporting rapide et...	$9.99	Buy on Amazon

As an Amazon Associate I earn from qualifying purchases.

Accept the upload with UploadFile

Browsers and curl -F send files as multipart form data, and FastAPI needs the python-multipart package to read that form. The FastAPI Request Files tutorial explains the two ways to receive a file. A parameter typed as bytes loads the entire upload into memory. A parameter typed as UploadFile uses a “spooled” file, which the guide describes as held in memory up to a limit and then written to disk, and it exposes a file-like interface plus metadata such as the filename.

For a CSV service, UploadFile is the better default because pandas can read from the file-like object directly, and you can check the size without loading the body into a Python bytes object first. Note that pandas still builds a full DataFrame in memory, so the upload limit matters even when the request layer spools to disk.

#1 Best Overall

Data Cleaner: Techniques et méthodes de préparation des données pour un reporting rapide et efficace (Données: traitement et reporting t. 1) (French Edition)

$9.99

Buy on Amazon

Install the dependencies

1. Create a virtual environment with python -m venv .venv and activate it.
2. Run pip install fastapi uvicorn python-multipart pandas.
3. After the application works, run pip freeze > requirements.txt so the Docker build installs the exact versions you tested.

The endpoint

import pandas as pd
from fastapi import FastAPI, File, HTTPException, UploadFile
from fastapi.responses import Response

app = FastAPI()

MAX_UPLOAD_BYTES = 10 * 1024 * 1024  # 10 MB, a project decision
REQUIRED_COLUMNS = ['customer_id', 'email', 'signup_date']
NULL_TOKENS = ['', 'NA', 'N/A', 'null']

@app.post('/clean')
async def clean_csv(file: UploadFile = File(...)):
    if not (file.filename or '').lower().endswith('.csv'):
        raise HTTPException(status_code=415, detail='Upload a file with a .csv extension.')

    handle = file.file
    handle.seek(0, 2)
    size = handle.tell()
    handle.seek(0)
    if size > MAX_UPLOAD_BYTES:
        raise HTTPException(status_code=413, detail='File exceeds the 10 MB limit.')

    try:
        df = pd.read_csv(handle, dtype=str, na_values=NULL_TOKENS, encoding='utf-8')
    except UnicodeDecodeError:
        raise HTTPException(status_code=422, detail='File must be UTF-8 encoded.')
    except (pd.errors.ParserError, pd.errors.EmptyDataError) as exc:
        raise HTTPException(status_code=422, detail='Malformed CSV: ' + str(exc))

    rows_in = len(df)

    df.columns = df.columns.str.strip().str.lower().str.replace(' ', '_')
    missing = [c for c in REQUIRED_COLUMNS if c not in df.columns]
    if missing:
        raise HTTPException(status_code=422, detail='Missing required columns: ' + ', '.join(missing))

    df = df.apply(lambda s: s.str.strip())
    df['email'] = df['email'].str.lower()
    df['signup_date'] = pd.to_datetime(df['signup_date'], format='%Y-%m-%d', errors='coerce')
    df = df.drop_duplicates()

    body = df.to_csv(index=False)
    return Response(
        content=body,
        media_type='text/csv',
        headers={'X-Rows-In': str(rows_in), 'X-Rows-Out': str(len(df))},
    )

Each stage has one job: check the request, parse the bytes, normalize and validate the structure, transform the values, then serialize. Keeping those stages separate makes it possible to test the cleaning function without an HTTP client, and it keeps error messages specific to the stage that failed.

Parse with explicit rules, not inference

The pandas read_csv reference accepts file-like objects and documents controls for delimiters, types, missing-value markers, date parsing, encodings, and malformed rows. Relying on defaults means a column of zip codes such as 02134 can silently become the integer 2134, and a column of IDs can turn into floats once a blank appears. The code above avoids this by reading every column as text and deciding afterwards what each column means.

Note that pandas releases have been changing the default string dtype, so after pinning a version run your sample files and confirm the output is unchanged.

Clean and validate in separate steps

Validate structure first

Header normalization runs before the required-column check, so Customer ID and customer_id are treated the same. If a required column is absent, the service stops with a 422 and names the missing columns. Validate structure before values: a file with the wrong shape should never reach the transformation step.

Transform values

• Whitespace: str.strip() runs on every cell, which removes accidental padding from exports.
• Email case: lowercasing is appropriate for most addresses but is a policy choice; some systems treat local parts as case-sensitive.
• Dates: pd.to_datetime with an explicit format rejects ambiguous inputs such as 01/02/2026. errors='coerce' turns unparseable values into missing dates instead of failing the whole file.
• Duplicates: drop_duplicates() runs after stripping, so rows that differ only in whitespace are recognized as duplicates.

Handle missing values deliberately

How pandas represents a missing value depends on the column’s dtype: numeric columns use NaN, datetime columns use NaT, and text columns hold NaN or None. The pandas missing-data guide describes isna() and notna() as the reliable way to detect these values regardless of representation. Use them to count blanks per column, for example df['email'].isna().sum(), and include those counts in logs or tests.

A worked example

Input file sent to the endpoint:

customer_id,email,signup_date
1, [email protected] ,2026-01-05
2,,2026-02-30
3,[email protected],N/A
1, [email protected] ,2026-01-05

Output body returned by the service:

customer_id,email,signup_date
1,[email protected],2026-01-05
2,,
3,[email protected],

The response headers are X-Rows-In: 4 and X-Rows-Out: 3. The fourth row is removed because it becomes identical to the first after whitespace is stripped. Row 2 keeps its place with an empty email, and its impossible date 2026-02-30 becomes an empty cell. Row 3’s N/A marker is treated as missing. The caller can compare the counts, but the service does not say which cells were blanked; if your callers need that detail, add a per-row error column or a separate report.

Choose a policy for each required field

Return actionable errors

• 415: the filename does not end in .csv. This check is by name only, so a renamed file can still reach the parser and fail there.
• 413: the spooled file is larger than MAX_UPLOAD_BYTES.
• 422: the file is not UTF-8, the CSV is malformed or empty, or required columns are missing. The detail field names the specific problem.
• 500: an unexpected failure. Log the traceback on the server and return a generic message to the caller.

To test these branches, send a file with a bad encoding, one with no header, and one with a misspelled required column, and confirm each returns the expected status code.

Package the service with Docker

Docker’s Python language-specific guide walks through a FastAPI container, a Compose configuration, and pinned requirements for reproducible builds. The Dockerfile below follows that pattern: dependencies are installed before the application code is copied, so changes to your code do not invalidate the dependency layer.

The Dockerfile

FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app/ ./app/

EXPOSE 8000

CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

Put the endpoint in app/main.py so the module path matches the CMD. Choose a Python image tag that your team supports and test it; the tag shown is an example, not a recommendation of a specific patch release.

Add a .dockerignore file containing .venv, __pycache__, .git, and *.csv so local virtual environments and sample data are not baked into the image.

Build, run, and test

1. Build the image: docker build -t csv-cleaner .
2. Run it: docker run --rm -p 8000:8000 csv-cleaner
3. Send a file and save the output, printing the response headers: curl -D - -F "[email protected]" http://localhost:8000/clean -o cleaned.csv
4. Open cleaned.csv and confirm the row count matches the X-Rows-Out header.

The FastAPI Docker deployment guide explains why this works: containers have their own isolated processes, file system, and network, which simplifies deployment and security. Inside the container, the service listens on 0.0.0.0 so the published port can reach it.

Choose a deployment route

The FastAPI guide names several ways to run a container in production: Docker Compose on a single server, Kubernetes, Docker Swarm, Nomad, and cloud services that deploy container images. It does not rank them for every case. The table below compares them on the axes that usually decide the choice.

Whichever route you choose, the FastAPI guide states that HTTPS is commonly handled outside the application container, and that replication should match the orchestration setup. Plan the TLS termination point before you expose the service.

Checks before exposing the service

• Transport: terminate HTTPS at a reverse proxy or load balancer, and set the proxy’s maximum request body size to match MAX_UPLOAD_BYTES, so oversized requests fail before reaching the application.
• Authentication: the endpoint above has none. Add an API key or token check before any public or shared deployment.
• Memory: pandas commonly uses more memory than the file occupies on disk once parsed. Measure peak memory with a file near your limit, then set the container’s memory limit with headroom.
• Disk: the spooled upload can be written to disk above its in-memory threshold. If the files are sensitive, point the container’s temporary directory at storage that matches your retention rules.
• Logging: log row counts, status codes, and processing time, but not cell values, since they may contain personal data.
• Restarts: configure a restart policy so a crashed container comes back, and verify that a health check or the root path responds after restart.

With the contract settled, the code above is a complete, testable starting point. Extend the cleaning functions and tests as your data demands, and revisit the framework and pandas documentation for the versions you pin.

Quick Recap

Bestseller No. 1

Data Cleaner: Techniques et méthodes de préparation des données pour un reporting rapide et efficace (Données: traitement et reporting t. 1) (French Edition)

$9.99

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.