Recommended Free Tools
To process CSV records without collecting the entire result set first, pipe Import-Csv into ForEach-Object and export the transformed objects once. That is record-by-record streaming, not chunking. If an operation requires fixed-size groups, accumulate and process a configurable batch; if independent tasks should run concurrently, use ForEach-Object -Parallel in PowerShell 7 or later. These are different approaches, and the example below shows when to use each.
Choose what “batch processing” means for your task
PowerShell pipelines pass command output downstream in order, with results available as they are generated. Microsoft describes the order this way: “In a pipeline, the commands are processed in order from left to right.” about_Pipelines
| Approach | What it does | When to use it |
|---|---|---|
| Record-by-record pipeline | Processes each row as it reaches the next command; no explicit chunk is assembled. | Each row can be transformed independently and the next command can accept individual objects. |
| Explicit chunking | Collects up to a chosen number of rows, processes that group, then starts another. | The operation requires a group, such as a bulk request with a maximum item count. |
| Parallel per-record processing | Runs multiple independent work items concurrently, up to a throttle limit. | Work can safely overlap and concurrency is useful for the particular task. |
Chunking is sequential unless you deliberately add concurrency. Parallel processing does not mean that the script creates groups for a bulk operation; it means that separate items can be active at the same time.
Start with a streaming CSV transformation
This example reads a CSV with Name as a column, adds a Processed value, and writes the results once. Change the paths and transformation to fit your file and task.
#1 Best Overall
- Book - powershell for sysadmins: workflow automation made easy
- Language: english
- Binding: paperback
param(
[string] $InputPath = '.input.csv',
[string] $OutputPath = '.output.csv'
)
Import-Csv -LiteralPath $InputPath |
ForEach-Object {
# Replace this with the task-specific transformation.
[pscustomobject]@{
Name = $_.Name
Processed = $true
}
} |
Export-Csv -LiteralPath $OutputPath -NoTypeInformationImport-Csv converts CSV rows into custom objects whose properties come from the file’s column headers. Its defaults expect a header row and a comma delimiter; use its header or delimiter options when your source differs. See Import-Csv.
The pipeline form avoids building a separate array of all transformed rows in the script. It does not establish a universal memory limit for every file, provider, or upstream command. Check the expected headers before transforming; handle an empty file, malformed rows, and missing values according to the requirements of the actual job. Keep status messages and diagnostics off the success-output stream so they are not treated as objects for CSV export.
Use explicit chunks when the operation needs a group
Set $BatchSize to the maximum number of rows your operation should receive at once. This example accumulates one chunk, processes it when full, and flushes any final partial chunk after input ends. Replace the processing block with an operation that accepts a collection of rows.
param(
[string] $InputPath = '.input.csv',
[int] $BatchSize = 500
)
if ($BatchSize -lt 1) {
throw 'BatchSize must be at least 1.'
}
$batch = [System.Collections.Generic.List[object]]::new()
Import-Csv -LiteralPath $InputPath | ForEach-Object {
$batch.Add($_)
if ($batch.Count -ge $BatchSize) {
# Replace with the operation that needs a group of rows.
foreach ($row in $batch) {
[pscustomobject]@{
Name = $row.Name
Processed = $true
}
}
$batch.Clear()
}
}
if ($batch.Count -gt 0) {
# Flush the final group, which may be smaller than BatchSize.
foreach ($row in $batch) {
[pscustomobject]@{
Name = $row.Name
Processed = $true
}
}
}The explicit buffer holds up to one chunk, in addition to any buffering or memory use of the input path and commands involved. The sample emits transformed rows to the success stream; pipe the complete script’s output to a single export command if you need a CSV result. Avoid writing with Export-Csv -Append for every row when one pipeline export will do. In Microsoft’s documented example using 2,100 CSV lines, moving export outside the loop changed the reported time from 15,968.78 ms to 42.92 ms, described there as 372 times faster. Those are timings for that specific example, not a general performance guarantee. PowerShell script performance considerations
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Rank #3
Run independent work concurrently in PowerShell 7 or later
For independent per-row work, PowerShell 7.5 documents ForEach-Object -Parallel and its -ThrottleLimit parameter. The throttle limits how many tasks run at once; Microsoft’s example uses four. Verify the installed version with $PSVersionTable.PSVersion before using this parameter set.
param(
[string] $InputPath = '.input.csv',
[string] $OutputPath = '.output.csv',
[int] $ThrottleLimit = 4
)
Import-Csv -LiteralPath $InputPath |
ForEach-Object -Parallel {
# Each item must be safe to process independently.
[pscustomobject]@{
Name = $_.Name
Processed = $true
}
} -ThrottleLimit $ThrottleLimit |
Export-Csv -LiteralPath $OutputPath -NoTypeInformationParallel completion order may differ from input order. Shared files, counters, APIs, and other side effects need deliberate synchronization or isolation; external services may also impose rate limits. Plan how failures and retries should work rather than assuming a failed item will be retried automatically. The PowerShell 7.5 ForEach-Object reference documents the parallel parameter set. The Windows PowerShell 5.1 reference does not list it, so use the sequential pipeline or explicit chunk approach in that environment.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Package reusable per-row logic in a function
When you want to reuse a transformation as a pipeline command, accept pipeline input and put the per-record operation in the function’s process block. Use begin for one-time setup and end for cleanup or final work. This keeps record handling distinct from work that should happen only once. See Microsoft’s about_Functions.
Quick Recap
Best Value
function Convert-InputRow {
[CmdletBinding()]
param(
[Parameter(ValueFromPipeline)]
[psobject] $Row
)
begin {
# One-time setup, if needed.
}
process {
[pscustomobject]@{
Name = $Row.Name
Processed = $true
}
}
end {
# One-time cleanup or final work, if needed.
}
}
Import-Csv -LiteralPath '.input.csv' |
Convert-InputRow |
Export-Csv -LiteralPath '.output.csv' -NoTypeInformationCheck the script before using it on a large file
- Confirm the input path, header names, delimiter, and expected handling of blank or malformed rows.
- Use record-by-record processing if each row can be handled alone; use a chunk only when the target operation requires a collection.
- Choose a batch size based on that operation’s limits and the available memory; there is no generally established best size.
- Use parallel processing only for independent work or work protected by an explicit synchronization strategy, and decide how output ordering and failures should be handled.
- Keep progress and diagnostics on appropriate streams rather than mixing them into objects destined for
Export-Csv. - Export once after the transformation when practical instead of reopening and appending to the output for every row.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →
