PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteSome links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
Use pandas to clean data as a sequence of documented decisions, not a single “clean” button. Preserve the original file, inspect its structure, measure missing and inconsistent values, choose rules based on what each column means, validate the result, and save a separate output. The current pandas documentation identifies version 3.0.6 (dated September 17, 2026); examples below are written for that documented version.
What data cleaning in Python actually involves
Cleaning means making a dataset consistent and usable for a defined purpose while keeping track of what changed. A blank in an middle_name column may be harmless, while a blank in price may prevent a calculation. Two records with the same customer ID may be a legitimate update history or an accidental duplicate. The correct action depends on the field’s meaning and the analysis you plan to perform.
pandas is an open-source Python library for tabular analysis. Its DataFrame object lets you inspect, transform, and export rows and columns. Do not overwrite your source file during exploration: retain an untouched copy and write cleaned data to a new path.
1. Install pandas and load a protected copy
In a virtual environment, install pandas and a CSV engine if needed:
#1 Best Overall
python -m pip install pandasLoad a CSV, Excel workbook, or another supported format. Keep the original path separate from the output path:
from pathlib import Path
import pandas as pd
source = Path("orders_raw.csv")
out = Path("orders_clean.csv")
df = pd.read_csv(source)
# For Excel instead: df = pd.read_excel("orders_raw.xlsx", sheet_name=0)
print(df.shape) # (rows, columns)
print(df.columns.tolist())
print(df.head())
print(df.dtypes)
Check whether the import interpreted separators, encodings, headers, and decimal marks correctly. An apparently “clean” transformation cannot repair a file that was parsed into the wrong columns.
2. Profile before changing anything
Start with measurements that reveal problems without altering values:
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →print(df.info())
print(df.describe(include="all"))
# Missing values and distinct values
print(df.isna().sum().sort_values(ascending=False))
for col in df.select_dtypes(include=["object", "string", "category"]).columns:
print(f"n{col} values:")
print(df[col].value_counts(dropna=False).head(20))
# Exact duplicate rows
print("Exact duplicate rows:", df.duplicated().sum())
- Dimensions: unexpected row or column counts often indicate a bad import.
- Names: spaces, inconsistent capitalization, or duplicate column names complicate later code.
- Types: a number stored as text will not behave like a numeric measure; dates may also be strings.
- Ranges and categories: negative quantities, impossible dates, and spelling variants are questions to investigate, not values to erase automatically.
Record these baseline metrics so you can compare them with the cleaned output.
3. Decide what missing values mean
pandas represents missingness differently depending on dtype. Its documentation treats dropping and filling as separate operations. Before choosing either, classify the blank as unknown, not applicable, not collected, or an import error.
Measure missingness
missing = (df.isna().sum()
.rename("missing")
.to_frame()
.assign(percent=lambda x: 100 * x["missing"] / len(df)))
print(missing.sort_values("missing", ascending=False))
Drop only when the loss is justified
# Remove rows missing a field required for this analysis
analysis_df = df.dropna(subset=["order_id", "order_date"])
# Remove a column only when it is not needed and mostly unavailable
smaller_df = df.drop(columns=["unused_note"])
Dropping reduces the retained sample and can bias results if missingness is systematic. Keep a count of removed rows and the rule that removed them.
Fill with a value that represents the domain
# A documented business rule: no recorded discount means zero
work = df.copy()
work["discount"] = work["discount"].fillna(0)
# Preserve an indicator when absence itself carries information
work["phone_was_missing"] = work["phone"].isna()
Do not use a mean, median, zero, or “Unknown” merely because it is convenient. An imputed value changes the data; document the assumption and, when useful, retain an indicator or the original column.
Recommended Free Tools
4. Normalize text without merging different meanings
Whitespace, case, punctuation, and spelling differences can split one category into several labels. pandas provides vectorized string methods through .str; these methods generally skip missing values automatically.
work = df.copy()
# Keep the raw field and create a normalized analysis field
work["city_clean"] = (
work["city"].astype("string")
.str.strip()
.str.replace(r"s+", " ", regex=True)
.str.casefold()
)
work["status_clean"] = (
work["status"].astype("string")
.str.strip()
.str.casefold()
.replace({"in progress": "in_progress", "in-progress": "in_progress"})
)
print(work[["city", "city_clean"]].drop_duplicates().head(20))
Inspect before-and-after categories. “St.” and “Saint” may be equivalent in one dataset but distinct labels in another. Keeping the source column makes the transformation reversible and auditable.
5. Convert types with explicit checks
Convert only after inspecting exceptional formats. A failed conversion should be visible rather than silently turned into a missing value.
Numbers
raw_amount = work["amount"].astype("string").str.strip()
# Remove a known currency symbol and thousands separator
prepared = raw_amount.str.replace("$", "", regex=False).str.replace(",", "", regex=False)
work["amount_num"] = pd.to_numeric(prepared, errors="coerce")
failed = work[work["amount_num"].isna() & raw_amount.notna()]
print("Unparsed amounts:")
print(failed[["amount", "amount_num"]])
Review the failed rows before deciding whether they are malformed, genuinely missing, or expressed in another currency.
Dates
work["order_date_parsed"] = pd.to_datetime(
work["order_date"], errors="coerce"
)
print(work.loc[
work["order_date"].notna() & work["order_date_parsed"].isna(),
["order_date", "order_date_parsed"]
])
Ambiguous formats such as day/month versus month/day require a known convention; do not guess from a few rows.
Categorical data
work["status_clean"] = work["status_clean"].astype("category")
print(work["status_clean"].cat.categories)
Categoricals can save memory and expose unexpected labels, but define categories only after normalization and validation.
6. Find duplicates using the correct key
df.duplicated() detects identical full rows, not repeated entities. A customer may appear twice with different addresses; an order may have multiple line items. Decide which columns must be unique, then inspect conflicts.
# Exact rows
exact_dupes = work[work.duplicated(keep=False)]
# A domain key: one row per order
key = ["order_id"]
key_dupes = work[work.duplicated(subset=key, keep=False)].sort_values(key)
print(key_dupes)
Only remove records after reviewing the duplicate group. If one record is a correction or a later update, reconcile fields according to a timestamp or source priority rather than keeping an arbitrary first row.
Free tools Windows power users keep installed
One-click scans. No signup required.
# Safe only when the business rule says the latest timestamp wins
work = (work.sort_values("updated_at")
.drop_duplicates(subset=["order_id"], keep="last"))
7. Validate and export a separate result
Validation asks whether the transformation met your rules; pandas cannot determine your domain’s meaning automatically.
# Example checks
assert work["order_id"].notna().all(), "Required order IDs are missing"
assert work["order_id"].is_unique, "Order IDs are not unique"
assert (work["amount_num"] >= 0).all(), "Negative amounts need review"
print("Rows before:", len(df))
print("Rows after:", len(work))
print("Missing after:")
print(work.isna().sum().sort_values(ascending=False))
print("Status categories:", work["status_clean"].value_counts(dropna=False))
work.to_csv(out, index=False)
print(f"Wrote {out}")
Compare row counts, missingness, distinct categories, ranges, and key constraints before and after. Save the script, input filename, date, and transformation decisions so another person can reproduce the output.
A complete beginner cleaning script
This compact example combines the workflow while leaving domain-specific choices visible:
from pathlib import Path
import pandas as pd
source = Path("orders_raw.csv")
out = Path("orders_clean.csv")
df = pd.read_csv(source)
work = df.copy()
# Normalize text, preserving raw columns
work["status_clean"] = (work["status"].astype("string")
.str.strip().str.casefold()
.replace({"in progress": "in_progress", "in-progress": "in_progress"}))
# Parse and expose failures
raw_amount = work["amount"].astype("string").str.strip()
prepared = raw_amount.str.replace("$", "", regex=False).str.replace(",", "", regex=False)
work["amount_num"] = pd.to_numeric(prepared, errors="coerce")
work["order_date_parsed"] = pd.to_datetime(work["order_date"], errors="coerce")
# Apply only justified missing-value rules
work["discount"] = work["discount"].fillna(0)
# Review key duplicates before enforcing uniqueness
print(work[work.duplicated("order_id", keep=False)].sort_values("order_id"))
# Validate required fields and export separately
assert work["order_id"].notna().all()
work.to_csv(out, index=False)
Troubleshooting common cleaning failures
“My numbers are still strings”
Currency symbols, commas, spaces, or mixed text prevent numeric conversion. Inspect the rows that became missing with errors="coerce", remove only documented formatting, and re-run the check.
“Dates became missing”
The source contains invalid or ambiguous formats. Print the original values where parsing failed, establish the source’s date convention, and parse each known format explicitly instead of guessing.
“Dropna removed too much data”
You may have dropped on every column or on a field that is optional. Use subset=[...], measure the row count before and after, and decide whether to retain or impute the missing field.
“Duplicates remain”
Full-row comparison misses records that share a business key but differ elsewhere. Use duplicated(subset=[...]), inspect conflicts, and define which record wins or whether both are valid.
“A string operation changed categories unexpectedly”
Case folding, punctuation removal, or replacement rules may have merged meaningful labels. Compare unique values before and after, keep the raw column, and narrow the rule.
“The output overwrote my source”
Use separate input and output paths, write to a temporary filename first when appropriate, and retain the original until validation is complete.
Best Value
Performance, reliability, and cost considerations
Vectorized pandas operations such as .str, fillna, to_numeric, and boolean filtering are generally preferable to Python row loops. Read only required columns for very wide files, process large inputs in chunks when a full DataFrame does not fit memory, and avoid repeatedly copying large frames. These are engineering choices, not substitutes for validation.
Cleaning itself has no pandas usage fee: pandas is open-source. Your costs may come from storage, compute, or the service that supplies the data. Reproducibility is improved by pinning the pandas version used for a project and recording import options, rules, and validation results. Because the current documentation identifies pandas 3.0.6, check the documentation for the exact version installed before relying on behavior that matters to a production pipeline.
Or skip the browser setup
If your workflow also needs screenshots of source tables, dashboards, or rendered reports, ScreenshotNeo provides a website screenshot API and MCP server. It accepts consent banners before capture and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each step can be disabled. Bot checks, CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and response headers identify the page verdict and billing status.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
A single request returns PNG, JPEG, WebP, or PDF:
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpPython:
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)Node.js:
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);See the ScreenshotNeo documentation for all options, including full-page and element capture, device and retina settings, custom CSS or JavaScript, waits, request blocking, cookies and headers, PDFs, caching, signed links, webhooks, bulk capture, and the usage API. Its MCP server includes take_screenshot, get_page_info, and capture_pdf for Claude, Cursor, and other MCP clients. The Free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000. Create a free ScreenshotNeo account.
FAQ
Should I clean data in place?
No. Keep the source immutable and export a separately named result so you can audit or reverse decisions.
Is an empty string the same as a missing value?
Not necessarily. Profile empty strings and null-like markers, then define a consistent representation for your dataset before applying missing-value rules.
How do I know whether a duplicate is an error?
Use the domain key and the dataset’s business process. Identical rows, repeated entities, and legitimate history require different handling.
Frequently Asked Questions
Can pandas decide the correct cleaning rule automatically?
No. pandas supplies operations and diagnostics; whether to drop, fill, normalize, or reconcile values depends on what each field represents.
What should I keep for an audit trail?
Retain the original file, cleaning script, pandas version, input options, transformation rules, validation results, and the separate output filename.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
