A big data framework is software and supporting architecture for storing, managing, processing, and analyzing data whose scale or complexity challenges traditional approaches. The term can mean a specific distributed-processing tool, such as Hadoop, or a broader model for how data services and participants fit together. NIST’s Big Data Reference Architecture describes the broader model; it is not a software package.
What does “big data framework” mean?
There is no single universal meaning implied by the phrase. In practice, it is used for systems that organize the resources and processes needed to work with data at scale, and sometimes for the software that performs that work.
As an Amazon Associate I earn from qualifying purchases.
The underlying challenge is that data volume, speed of arrival, and variety can exceed the capabilities of conventional approaches. NIST describes scalable architectures as a way to address these characteristics. NIST Big Data Interoperability Framework: Volume 1, Big Data Definitions [Version 2]
Framework software versus reference architecture
A software framework is an implementation: it provides capabilities that applications can use to process or manage data. Hadoop is one example. Apache describes it as software for distributed processing of large data sets across computer clusters, using programming models such as MapReduce. Apache Hadoop training presentation
A reference architecture, by contrast, is a conceptual model for describing roles, activities, components, and their relationships. NIST’s Big Data Reference Architecture (NBDRA) is vendor-neutral: it helps explain how a big-data system can be organized, but it does not prescribe a particular product or implementation. NIST Big Data Interoperability Framework: Volume 6, Reference Architecture
Who does what in NIST’s reference architecture?
NIST’s model identifies five main roles. In a real deployment, one organization or system may perform more than one role.
- Data Provider: supplies data to the system.
- Big Data Application Provider: develops or integrates the application that processes and uses the data.
- Big Data Framework Provider: supplies the platform environment that supports the application, including relevant infrastructure, platform, and service capabilities.
- Data Consumer: uses the information or results produced by the system.
- System Orchestrator: coordinates the system and the relationships among its parts.
The architecture considers both information value—collecting, integrating, analyzing, and applying data—and the IT services that enable those activities, such as networking, infrastructure, platforms, and application tools.
Why management and security span the whole system
In the NIST model, management and security/privacy are cross-cutting concerns, not isolated pipeline stages. They apply across the roles and system activities. That framing matters because a system can process data successfully yet still be difficult to operate, govern, or protect if those concerns are treated as afterthoughts.
Examples of big data frameworks
Hadoop is a concrete example of distributed-processing software, and MapReduce is one of the parallel-processing frameworks associated with it. NIST’s reference-architecture report also names Spark and Storm as examples of frameworks. These examples illustrate the software sense of the term; their mention here does not establish a current feature or version comparison. NIST Volume 6 report (PDF)
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How to interpret the term when evaluating a system
First establish whether a discussion means a conceptual architecture or a specific software platform. If it means software, the right choice depends on the workload and operating environment rather than on the label “big data” alone. Relevant questions include:
Rank #4
- Does the workload process stored batches, continuously arriving streams, or both?
- Which data sources and formats must it support?
- How should storage and compute work together, and how will the system scale or recover from failures?
- Which programming interfaces and deployment environments are required?
- What security, privacy, and governance controls are needed?
- Can the team operate the system at an acceptable level of complexity and cost?
These are decision criteria, not a ranking of products. NIST’s model is useful for mapping responsibilities and system components; a software framework is the implementation selected to meet a particular workload’s needs.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Quick Recap
Best Value
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
