Batch Processing Large Data Sets With Spring Boot and Spring Batch

Large data sets often need to be imported, transformed, validated, exported, or synchronized without overwhelming application servers or databases. Spring Boot and Spring Batch provide a proven foundation for these workloads, combining production-ready configuration with a mature batch framework designed for repeatable, restartable, and observable processing.

Spring Batch structures work into jobs, steps, readers, processors, and writers, making it easier to process millions of records in controlled chunks instead of loading everything into memory. With transaction boundaries, retry and skip policies, job metadata, and restart support, it helps teams build reliable pipelines that can recover from failures and continue from a known state.

#	Preview	Product	Price
1		Spring Batch in Action	$32.84	Buy on Amazon
2		The Definitive Guide to Spring Batch: Modern Finite Batch Processing in the Cloud	$45.93	Buy on Amazon
3		140 Great Hikes in and near Palm Springs, 25th Anniversary Edition	$25.12	Buy on Amazon
4		Hiking Hot Springs in the Pacific Northwest (Regional Hiking Series)	$24.95	Buy on Amazon
5		Fighting for Forever (The Springs Book 10)		Buy on Amazon

As an Amazon Associate I earn from qualifying purchases.

Building scalable batch jobs requires more than wiring components together. Efficient readers and writers, thoughtful chunk sizing, database tuning, parallelism, monitoring, and operational controls all influence throughput and stability when volumes grow.

Spring Batch Architecture and Core Concepts

Spring Batch is organized around a small set of core abstractions that make large-scale data processing predictable, restartable, and observable. At the top level, a Job represents the complete batch workload, such as importing customer records, generating invoices, or reconciling transactions. A job is composed of one or more Steps, and each step performs a distinct part of the work. For example, one step might load records from a CSV file into a staging table, while another validates and writes approved records to the main business tables.

#1 Best Overall

Sale

Spring Batch in Action

• Used Book in Good Condition

$32.84

Buy on Amazon

In a Spring Boot application, jobs are configured as Spring beans and launched through Spring Batch infrastructure. The framework stores execution metadata in a JobRepository, including job parameters, start and end times, status, step counts, failures, and restart information. This metadata is usually stored in relational tables such as BATCH_JOB_INSTANCE, BATCH_JOB_EXECUTION, and BATCH_STEP_EXECUTION. Because the repository persists state, Spring Batch can distinguish between a new job run and a restart of a failed run.

Primary building blocks

• Job: The full batch process, identified by its name and job parameters.
• Step: A single phase of work inside a job, commonly implemented with chunk-oriented processing or a tasklet.
• JobLauncher: Starts a job with a set of parameters, often triggered by a scheduler, REST endpoint, command-line runner, or messaging event.
• JobRepository: Persists execution metadata and supports restartability, auditing, and status tracking.
• JobExecution: Represents one attempt to run a job.
• StepExecution: Represents one attempt to run a step and records read, write, commit, rollback, skip, and failure counts.
• ExecutionContext: A persistent key-value store used to save state between commits and across restarts.

Most high-volume jobs use chunk-oriented processing. In this model, a step reads a group of items, optionally transforms them, and writes them as a batch inside a transaction. The three main collaborators are an ItemReader, an ItemProcessor, and an ItemWriter. The reader pulls one item at a time from a source such as a database cursor, paginated query, flat file, JSON file, XML document, Kafka topic, or cloud object storage. The processor applies validation, enrichment, filtering, or mapping. The writer persists the processed items to a destination such as a relational database, file, API, search index, or message broker.

A typical step might read 1,000 rows at a time from a database, transform each row into a domain object, and write the transformed records using JDBC batch updates. The commit interval, also called the chunk size, controls how many processed items are included in a single transaction. If the chunk size is 500, Spring Batch reads and processes up to 500 items, calls the writer once with that list, and commits the transaction. If an error occurs before commit, the transaction rolls back and the step can retry, skip, or fail depending on the configured fault-tolerance rules.

Job flow and step types

Steps can be arranged sequentially or conditionally. A job can move from one step to the next only when the previous step completes successfully, or it can branch based on an exit status such as COMPLETED, FAILED, or a custom status. This is useful for workflows where failed validation should send records to a rejection report instead of continuing to downstream publication.

This architecture gives Spring Batch its strength for large data sets: clear separation of responsibilities, persistent execution state, configurable transactions, and reusable components. Instead of writing one large procedural script, you define a job as coordinated steps with explicit readers, processors, writers, and operational metadata. That structure becomes essential when jobs must process millions of records, recover from mid-run failures, and provide enough visibility for production support teams.

Setting Up a Spring Boot Batch Project

A Spring Boot batch project starts with a small set of dependencies and a clear separation between infrastructure configuration, job definitions, and domain-specific processing code. For most applications, include spring-boot-starter-batch for Spring Batch, a JDBC driver for the metadata repository, and the database starter that matches your persistence approach, such as spring-boot-starter-jdbc or spring-boot-starter-data-jpa. If the job reads or writes files, add any needed libraries for CSV, fixed-width, JSON, or XML handling.

Spring Batch uses a JobRepository to store job instances, executions, step executions, parameters, and status information. In development, an embedded database may be enough, but production workloads should use a persistent database such as PostgreSQL, MySQL, Oracle, or SQL Server. This repository is what allows failed jobs to be inspected and restarted safely, so it should be treated as operational infrastructure rather than temporary storage.

Typical project structure

• config: batch infrastructure settings, job and step bean definitions.
• reader: implementations that read from files, databases, queues, APIs, or object storage.
• processor: validation, enrichment, filtering, mapping, and business rules.
• writer: database writes, file exports, message publishing, or API calls.
• domain: input records, output records, entities, and DTOs.
• listener: job, step, chunk, skip, and retry listeners for auditing and diagnostics.

In current Spring Batch versions, jobs and steps are commonly defined as beans using JobBuilder and StepBuilder, wired with the JobRepository and PlatformTransactionManager. A minimal job usually contains one or more steps, and each step defines a reader, optional processor, and writer. Keeping these beans explicit makes it easier to test each part and to tune individual steps later.

Application configuration

Configure the batch metadata schema and launch behavior through Spring Boot properties. During local development, automatic schema initialization can be convenient. In controlled environments, create the Spring Batch tables through a migration tool such as Flyway or Liquibase so schema changes are versioned and repeatable. Many teams also disable automatic job execution on startup and launch jobs through a scheduler, REST endpoint, command-line runner, or orchestration platform.

Rank #2

Sale

The Definitive Guide to Spring Batch: Modern Finite Batch Processing in the Cloud

$45.93

Buy on Amazon

Job parameters deserve attention from the beginning. Spring Batch identifies a job instance by the job name and its identifying parameters. For example, a daily import might use businessDate=2026-05-26, while a file-based job might use inputFile=s3://bucket/orders-2026-05-26.csv. Choose parameters that represent the data being processed, not just the time the job was started, so restarts and duplicate-run protection behave predictably.

For scalable projects, avoid placing business directly inside configuration classes. Define readers, processors, and writers as focused beans, inject collaborators such as repositories or clients, and keep job configuration responsible only for assembling the flow. This creates a foundation that can grow from a single local CSV import into multi-step, restartable, observable batch pipelines that process millions of records reliably.

Reading, Processing, and Writing Large Data Sets

A Spring Batch step usually moves data through three roles: an ItemReader loads one record at a time, an ItemProcessor transforms or validates it, and an ItemWriter persists a group of processed records. This model works well for large data sets because records are streamed through the pipeline instead of being loaded into memory all at once. For example, a job can read ten million customer rows from a database, normalize phone numbers, filter inactive accounts, and write the remaining records to another table or file while keeping memory usage predictable.

Choosing the right reader

The reader should match the source system and access pattern. For relational databases, JdbcCursorItemReader streams rows using a database cursor, while JdbcPagingItemReader reads pages of rows using sort keys and is often easier to restart safely at scale. For JPA-based applications, JpaPagingItemReader can be used, but it should be configured carefully to avoid persistence context growth. For flat files, FlatFileItemReader reads delimited or fixed-width records line by line. For XML and JSON sources, Spring Batch also provides specialized readers, though very large JSON files may require streaming parsers to avoid memory pressure.

• Database imports: use paging readers with stable ordering, such as an indexed primary key.
• CSV exports: use FlatFileItemReader with a tokenizer and field mapper.
• Large table scans: select only required columns and avoid expensive joins inside the reader query.
• Restartable jobs: prefer readers that store progress in the Spring Batch execution context.

The processor is the place for business rules that operate on one item at a time. It can enrich a record, convert it into a different domain object, apply validation, or return null to filter the item out of the output. For large jobs, processors should be deterministic and lightweight. Expensive remote calls, repeated database lookups, or object-heavy transformations can quickly become bottlenecks. If enrichment data is small enough, preload it into a cache or map before the step starts. If it is large, use batched lookups or redesign the flow so that enrichment happens closer to the database.

Writing efficiently

Writers receive a chunk of items rather than a single item, which allows efficient batch output. JdbcBatchItemWriter can execute batched inserts or updates using prepared statements, reducing database round trips. JpaItemWriter works for entity persistence, but flushing and clearing must be managed to prevent the persistence context from retaining too many objects. FlatFileItemWriter is suitable for generating CSV, fixed-width, or line-based export files. When writing to messaging systems, object storage, or external APIs, use batching where the target system supports it and add clear limits for request size and timeout behavior.

A practical pattern is to keep each component focused and measurable. The reader should not contain business transformation rules, the processor should not open database cursors, and the writer should not silently discard failed records. Clear separation makes the job easier to test with small fixtures and easier to tune under production volume. For high-volume data sets, start with representative sample data, verify record counts at each stage, and capture timings for read, process, and write phases so the slowest part of the pipeline can be improved directly.

Chunk-Oriented Processing and Transaction Management

Chunk-oriented processing is the default model for scalable Spring Batch steps because it avoids loading an entire data set into memory. Instead of reading millions of records at once, a step reads a fixed number of items, optionally processes them, writes them as a group, and then commits the transaction. This pattern gives you predictable memory usage, efficient database writes, and a clear recovery boundary if the job fails midway through execution.

A typical chunk step is built from an ItemReader, an optional ItemProcessor, and an ItemWriter. If the chunk size is set to 1,000, Spring Batch repeatedly reads up to 1,000 items, passes them through the processor, sends the resulting list to the writer, and commits the transaction after the write succeeds. If an exception occurs before the commit, the transaction rolls back and the items in that chunk are not considered complete. Previously committed chunks remain durable, which is what makes restart behavior practical for large jobs.

Rank #3

Sale

140 Great Hikes in and near Palm Springs, 25th Anniversary Edition

$25.12

Buy on Amazon

Choosing an Effective Chunk Size

The chunk size controls how much work is performed inside one transaction. A very small chunk size, such as 1 or 10, increases transaction overhead and can slow down high-volume jobs. A very large chunk size, such as 50,000, can increase memory pressure, hold database locks longer, and make rollbacks more expensive. For database-backed jobs, common starting values are often between 500 and 5,000, then adjusted after measuring throughput, heap usage, lock duration, and write latency.

• Smaller chunks reduce rollback scope and lock duration but increase commit frequency.
• Larger chunks improve write batching and reduce transaction overhead but require more memory.
• Remote APIs or slow writers usually benefit from smaller chunks to avoid long-running transactions.
• Bulk database inserts or updates often perform better with moderate to larger chunks, combined with JDBC batching.

Transaction Boundaries in a Step

In a chunk-oriented step, the transaction normally begins before the chunk is processed and commits after the writer completes successfully. The reader, processor, and writer may all participate in the same transaction depending on the resource being used. For example, a JDBC cursor reader and JDBC batch writer can be coordinated through the configured PlatformTransactionManager. If the writer fails due to a constraint violation, timeout, or deadlock, Spring Batch rolls back the current chunk and can retry, skip, or fail the step depending on the configured fault-tolerance rules.

For file-based readers, transaction behavior is slightly different because a flat file cannot be rolled back like a database table. Spring Batch handles restartability through the ExecutionContext, storing metadata such as the current line number or item count. On restart, the reader can reopen the file and continue from the last committed checkpoint. This makes the commit interval especially significant: if a job fails after reading 900 records in a chunk of 1,000, those 900 records are read again on restart because the checkpoint was not committed.

Practical Configuration Guidelines

• Use one transaction manager that matches the primary transactional resource, such as DataSourceTransactionManager for JDBC workloads.
• Keep processors deterministic so reprocessing the same item after rollback produces the same result.
• Make writers idempotent where possible, especially when writing to external systems that are not part of the transaction.
• Avoid mixing database writes and non-transactional side effects, such as sending emails, inside the same chunk without a compensation strategy.
• Persist enough state in the ExecutionContext to allow safe restarts after failure.

Well-designed chunk and transaction settings directly affect correctness as well as performance. The goal is to commit often enough that failures are cheap to recover from, but not so often that transaction overhead dominates the job. In production, the best configuration is usually found by testing with realistic data volumes, indexes, network latency, and downstream system behavior rather than relying on a fixed default.

Handling Errors, Retries, and Skipped Records

Large batch jobs must assume that some records will fail. A customer row may contain an invalid email address, a referenced account may not exist, a downstream API may time out, or a database constraint may reject a duplicate key. Spring Batch provides fault-tolerant step configuration so a job can distinguish between recoverable failures, invalid data, and serious system problems that should stop execution. This keeps high-volume jobs from failing after millions of successful records because of a small number of bad inputs.

Fault tolerance is usually configured on a chunk-oriented step. When a step is marked as fault tolerant, you can define which exceptions are retryable, which are skippable, and the maximum number of skips allowed before the step fails. Retries are best for transient problems, such as temporary network failures, deadlocks, optimistic locking conflicts, or short-lived service outages. Skips are better for deterministic data problems, such as malformed CSV rows, validation failures, or records that violate business rules.

Configuring retries and skips

A typical configuration uses retry for exceptions that may succeed on a later attempt and skip for records that should be excluded from the current run. The retry limit should be small and intentional, commonly 2 to 5 attempts, because every retry increases job duration and can amplify load on an already unhealthy dependency. The skip limit should reflect operational tolerance: allowing 100 rejected records in a 10 million row import may be acceptable, while a single skipped payment record may require investigation.

• Retry transient exceptions: database deadlocks, connection timeouts, HTTP 503 responses, remote service timeouts.
• Skip bad input records: parse errors, missing required fields, invalid enum values, failed domain validation.
• Fail fast on infrastructure or logic errors: missing tables, invalid SQL, misconfigured files, serialization bugs.
• Use specific exception classes: avoid broad rules such as skipping all Exception, which can hide serious defects.

Spring Batch applies retry and skip behavior around item reading, processing, and writing, but each phase has different implications. A skipped read usually means the framework could not map an input record, such as a malformed line in a flat file. A skipped process item often indicates validation or enrichment failure. A skipped write can be more complex because writers often handle a chunk of items at once; if a batch insert fails, Spring Batch may need to isolate the bad item depending on the writer and transaction behavior.

Capturing rejected records

Skipped records should not disappear silently. Use skip listeners to capture the failed item, exception type, message, step name, job execution id, and input source position when available. Many teams write rejected records to a dedicated database table, a dead-letter file, or an object storage location such as S3. This makes reconciliation practical after the job completes and provides a clear audit trail for compliance-sensitive workloads.

Rank #4

Hiking Hot Springs in the Pacific Northwest (Regional Hiking Series)

$24.95

Buy on Amazon

For production jobs, combine fault tolerance with clear exit statuses and alerts. A job that completes with skipped records may still need a distinct status such as COMPLETED_WITH_SKIPS so schedulers, dashboards, and support teams can treat it differently from a clean success. Store skip counts, retry counts, and representative error messages in monitoring systems, and set thresholds that trigger notifications before data quality issues become routine.

Performance Tuning for High-Volume Jobs

High-volume Spring Batch jobs usually become slow because of database round trips, oversized transactions, inefficient readers, or too much work inside the processor. Start tuning with measurements rather than guesses: capture item throughput, chunk duration, commit time, read latency, write latency, CPU usage, heap usage, garbage collection pauses, and database wait events. A job that processes 50 million rows should expose enough metrics to show whether the bottleneck is the JVM, the database, the network, or an external service.

Choose an efficient chunk size

The chunk size controls how many items are read, processed, and written before a transaction commits. A very small chunk size, such as 1 or 10, creates excessive commit overhead and database chatter. A very large chunk size, such as 50,000, can increase memory pressure, hold locks longer, and make rollbacks expensive. For many database-backed jobs, a starting range of 500 to 5,000 items per chunk is practical, then adjust based on commit time, heap usage, and lock contention.

• Increase chunk size when commits are frequent and the database can handle larger batched writes.
• Decrease chunk size when transactions hold locks too long, memory grows rapidly, or rollback cost is high.
• Keep item objects small by avoiding unnecessary fields, nested graphs, and large temporary collections.

Optimize readers and writers

Readers and writers often decide the real throughput of a batch job. For relational databases, prefer paging or cursor-based readers designed for streaming large result sets instead of loading all rows into memory. Ensure the read query uses indexed columns for filtering and sorting, especially when processing data by status, date range, tenant, or numeric identifier. When writing, use batch updates, bulk inserts, or database-native upsert patterns where appropriate. With JPA, clear the persistence context regularly or use JDBC-based writers for very large volumes to avoid accumulating managed entities in memory.

Use parallelism carefully

Spring Batch supports several scaling patterns, including multi-threaded steps, partitioned steps, remote partitioning, and remote chunking. A multi-threaded step is simple to configure, but the reader, processor, and writer must be thread-safe or scoped correctly. Partitioning is often safer for large database workloads: split the input by ID ranges, dates, tenants, or file segments, then run partitions concurrently. This reduces contention and allows each worker to process an independent slice of data.

Parallel execution should match the capacity of downstream systems. Setting a task executor to 32 threads will not help if the database connection pool has only 10 usable connections or if the target API throttles requests. Tune thread count, connection pool size, queue capacity, and database resources together. Monitor deadlocks, lock waits, rejected connections, and slow queries after every concurrency change.

Reduce avoidable work

For very large jobs, small inefficiencies become expensive. Avoid per-item logging, repeated object mapper creation, repeated date formatter creation, and remote calls inside the processor. Replace row-by-row enrichment with bulk prefetching when the reference data is small enough to cache. If the job processes only changed records, use watermarks, status flags, or change tables instead of scanning the full source every run. Keep indexes balanced as well: the right indexes speed reads, but too many indexes can slow heavy writes.

Finally, tune the JVM and job repository for long-running workloads. Set realistic heap limits, watch garbage collection, and keep the Spring Batch metadata tables maintained with proper indexes and cleanup policies. The best-performing job is usually the one that processes predictable partitions, commits at a stable rhythm, writes in batches, and produces enough runtime metrics to make the next bottleneck visible.

Monitoring, Scheduling, and Restarting Batch Jobs

Once a batch job is moving millions of records, the operational model becomes as as the job code. Spring Batch stores execution metadata in its repository tables, including job instances, job executions, step executions, parameters, start and end times, exit statuses, and failure messages. These tables let operators answer practical questions: which file was processed, how many records were read and written, where the failure occurred, and whether a job can be restarted safely.

Best Value

Sale

Fighting for Forever (The Springs Book 10)

Buy on Amazon

In a Spring Boot application, expose batch health and metrics through Spring Boot Actuator. The health endpoint can show whether the application is ready to launch jobs, while Micrometer metrics can be sent to Prometheus, Grafana, Datadog, or another monitoring system. Track counters such as read count, write count, skip count, rollback count, commit count, duration per step, and records processed per second. For long-running jobs, also log progress at controlled intervals, such as every 10,000 records, so teams can distinguish a slow job from a stuck one without flooding log storage.

Scheduling job execution

Batch jobs can be launched in several ways depending on the environment. For simple deployments, Spring’s @Scheduled support can trigger jobs on a fixed delay or cron expression. In production, many teams prefer an external scheduler such as Kubernetes CronJob, Control-M, Airflow, Jenkins, Autosys, or a cloud-native scheduler. External scheduling usually gives better visibility, dependency management, retry policies, calendars, and alerting across mulle systems.

• Use unique job parameters: include values such as business date, input file name, tenant, or run id so Spring Batch can distinguish one job instance from another.
• Avoid overlapping runs: prevent the same job and parameter set from running concurrently unless the design explicitly supports partitioning or parallel execution.
• Validate inputs before launch: check file presence, database connectivity, target table availability, and required configuration before starting expensive processing.
• Emit clear exit codes: map completed, failed, stopped, and partial-success states to codes that schedulers and alerting tools can interpret.

Restarting failed jobs safely

Spring Batch restartability depends on the job repository and consistent transaction boundaries. If a chunk-based step fails after several commits, previously committed chunks remain complete, and the restarted execution continues from the last saved checkpoint when the reader supports state. This is especially useful for large files, paged database reads, and message-like workloads where reprocessing everything would be too expensive.

Restart-safe jobs need deterministic input and idempotent output. A restarted job should not duplicate payments, resend emails, insert duplicate rows, or overwrite newer data incorrectly. Common safeguards include natural keys, unique constraints, upsert operations, staging tables, processed-record flags, and audit columns that store the job execution id. For file-based jobs, keep the original input file immutable until the job finishes, and archive it only after a successful execution.

For controlled shutdowns, use Spring Batch’s stop capabilities rather than killing the process abruptly. A stopped execution can often be resumed cleanly, while a terminated process may leave external resources in uncertain states. Combine metadata, metrics, structured logs, and scheduler integration so every batch run is traceable from launch through completion, failure, restart, and final reconciliation.

Frequently Asked Questions

How do I choose the right chunk size for a large Spring Batch job?

Start with a moderate chunk size such as 500 or 1,000 records, then measure throughput, memory usage, database load, and rollback cost. Larger chunks reduce transaction overhead but increase memory pressure and the amount of work repeated after a failure. For database-heavy jobs, tune chunk size together with JDBC fetch size, batch insert size, indexes, and connection pool limits.

What is the best way to restart a failed Spring Batch job without reprocessing everything?

Use Spring Batch’s JobRepository with restartable jobs and readers that support saving state, such as JdbcCursorItemReader, JdbcPagingItemReader, or FlatFileItemReader with state enabled. Make sure each JobInstance is launched with stable identifying JobParameters, and avoid deleting batch metadata tables after failures. Writers should be idempotent or use upserts, unique constraints, or status columns so a restarted job does not create duplicate output.

Should I use JdbcCursorItemReader or JdbcPagingItemReader for very large database tables?

JdbcCursorItemReader streams rows from a single database cursor and can be efficient, but it may hold database resources for a long time and can be sensitive to connection timeouts. JdbcPagingItemReader reads data page by page, which is often safer for very large tables and restart scenarios, especially when you sort by a stable unique key. For high-volume production jobs, paging is usually preferred when the source table can be queried with an indexed ordering column.

How do I handle bad records without failing the entire batch job?

Configure fault tolerance on the step with skip rules for expected data problems, such as validation errors or parse exceptions, and set a skip limit that matches your operational tolerance. Use a SkipListener to write rejected records and error details to a dead-letter table, file, or monitoring system for later review. Keep retry for transient issues like network timeouts or deadlocks, not for records that are permanently invalid.

How can I speed up a Spring Batch job that takes hours to process millions of records?

First profile where time is spent: reading, processing, writing, database commits, or external service calls. Common improvements include increasing chunk size carefully, using JDBC batch writes, adding the right indexes, reducing per-record queries, and moving expensive lookups into caches or preloaded maps. For further scaling, consider partitioning, multi-threaded steps, or remote chunking, but verify that your reader, writer, and target system can handle concurrent access safely.

Bottom Line

Spring Batch gives Spring Boot applications a reliable foundation for processing large data sets through clear job configuration, chunk-oriented steps, restartability, fault tolerance, and flexible readers and writers. When designed carefully, batch jobs can handle high volumes efficiently without sacrificing consistency or operational visibility.

Start with a simple chunk-based job, measure its throughput, then tune commit intervals, parallelism, database access, and retry/skip rules based on real workload behavior. From there, add monitoring, restart strategy, and scheduling so your batch pipeline is not only fast, but dependable in production.

Quick Recap

SaleBestseller No. 1

Spring Batch in Action

Used Book in Good Condition

$32.84

SaleBestseller No. 2

The Definitive Guide to Spring Batch: Modern Finite Batch Processing in the Cloud

$45.93

SaleBestseller No. 3

140 Great Hikes in and near Palm Springs, 25th Anniversary Edition

$25.12

Bestseller No. 4

Hiking Hot Springs in the Pacific Northwest (Regional Hiking Series)

$24.95

SaleBestseller No. 5

Fighting for Forever (The Springs Book 10)

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.