Modern Data Architecture with AWS Cloud
- data challenges of traditional data architecture
- gains from adopting modern data architecture
- modern data architecture implementation strategy
- modern data architecture orchestration
- modern data architecture AWS-based services.
In light of the current ceaseless expansion of data volume and the emergence of new data types, the modernization of data analytics systems is in increased demand. As it is not always transparent when modern data architecture is mandatory, and whether or how exactly your business may benefit from this kind of investment, we have compiled this article to clarify the link between poignant enterprise issues and the solutions suggested by modern data architecture, in particular, offered by Amazon Web Service (AWS).
The article intends to keep you posted on the process and consequences of modern data architecture implementation with a view to handling your data-related stumbling blocks, and boosting analytics which leads to new business outcomes.
Table of Contents
The problematique of data architecture
Over the years, enterprise data warehouses (EDW) have been preferable among numerous businesses as the single version of the truth. Offering a consolidated historical business information repository for data of various origins, EDW is referred to as the semantic layer of hierarchically centered data stored in files and folders.
Data lakes, on the other hand, store and process information within their flat architecture, whether in a file or object storage. Their signature feature is containing a substantial amount of raw information in its native format until it is to be analyzed.
EDW data reports have led many organizations by providing insights into business strategies and governing decision-making. The last two decades witnessed the two most common approaches to executing EDW:
Kimball approach (by Ralph Kimball)
Inmon approach (by Bill Inmon)
One commonly accepted outlook on data warehouse design is Ralph Kimball’s perception whereby it is a bottom-up process. The information is collected in data marts (or star schemas), each tied up to a specific business context yet unified under common business dimensions, that together constitute the data warehouse.
According to Kimball, information can be integrated into the data warehouse gradually through conformed data dimensions. New data marts are introduced with the addition of fact tables as the central element of a star schema as well as with the attachment of new dimensions to the present conformed data dimensions.
Let’s take a look at the arguments for and against preferring Kimball’s approach to creating a data warehouse.
- A purpose-built and logically grouped set of data marts.
- Gradually compiled for a business use case, containing all data.
- Made up of denormalized data.
- Designed for a specific business domain, thus comprehensible for non-IT users.
- Suitable for MPP Relational Database Management Systems (RDBMS).
- Appropriate for data lake implementations.
- A properly synchronized integration layer is virtually impossible to achieve due to the data duplication and data sync issues, alongside EDW being case-driven.
- Changes that lessen the impact on customers are delayed because of tight coupling with them.
- The tables’ large amounts of data need labor-intensive and time-consuming extract, transform, and load (ETL) operations.
In contrast with Kimball’s approach, Bill Inmon regards a top-down approach as optimal for EDW design. Having a normalized data model as his aim, he advocates producing a unified data model in accordance with the business context for all the upcoming information items to be translated into it.
More specifically, in order to enter the EDW, data from various sources need to be unified, formatted, typecasted, and semantically translated to comply with the previously defined unified data model established by the enterprise.
Further, we present the most commonly mentioned merits and demerits of Inmon’s EDW creation approach.
- Significantly normalized, with low data redundancy.
- Clear in data relationships and granularity, characterized by proper semantics.
- Independently evolving since it is decoupled from end users.
- Improved for the storage of historical data.
- A better fit for RDBS implementations.
- Given the data model complexity, substantial time and effort investment is required to prepare data from various sources to fit the unified model.
- Data sources dependency and minimized parallelism are brought about by the integrated data model.
- The computation required to do ad hoc analysis with the Third normal form (3NF) is quite high.
- As a rule, source data requires splitting to suit the EDW data model and the process of forming a final integrated dataset from various sources is even more complex.
- The best analytical query performance is still achieved with a denormalized in a star or snowflake schema presentation layer on top of the EDW.
- Inmon’s top-down approach is less adapted for data lake implementation.
So, the problem is
While both approaches to data warehouse design have delivered value through their advantages, modern data quantities cannot be managed by conventional on-premises systems for reasons including matters of scaling, flexibility and price.
Traditional data-storing systems were designed with the view to combining information of various origins and generating operational reports. They, however, were not created to accommodate the rapid expansion of event data (like log files, or machine-generated data from the Internet of Things devices).
Those factors, together with the current rapid expansion of data, make traditional data warehouses less agile and likely to lengthen the time-to-market for realizing solutions. In addition, modern data apps tend to employ a variety of tools to be built and deployed whereas conventional data warehouse models support only a limited number of such tools, e.g. SQL and BI workloads.
This, merged with the development of cloud and big data capabilities in recent years contributes to organizations moving toward a modern data architecture pattern. Customers now choose to build data lakes to store all of their data, both structured and unstructured, in one location and analyze this data with a wide range of tools provided.
And yet, to make the most of their data, end users are more and more often embracing a modern data architecture which can combine the benefits of both data lakes and purpose-built data storage. The practical decision, for now, incorporates storing all open-formatted data in data lakes and occasionally using conventional purpose-built data services to process and analyze them.
What is modern data architecture?
Technology advancements, ceaselessly increasing amounts of various data and fierce competition all force enterprises to gather the information, amass and process it to extract the maximum value in the shortest time possible. Data warehouses were a response to a particular type of use case, which was Online Analytics Processing (OLAP). Today, the EDW’s inherent one-size-fits-all data strategy is hardly sufficient to accommodate the needs of, let’s say, log analytics, predictive analytics, or big data processing, and more often than not impedes further scaling and development.
As aforementioned, a modern data architecture enables incorporating the benefits of both data lakes and purpose-built data stores. By adopting the modern data strategy referred to above, you can cut back on the data silos, and enable your teams to utilize the most suitable tools for conducting analytics or machine learning tasks. This approach also ensures that data accessibility is safeguarded with the necessary data governance controls.
While it sounds easy and straightforward, modern data architecture is extending beyond the simple integration of data lakes with data warehouses; it rather focuses on providing unified governance and smooth data movement without compromising its security.
A proper data foundation entails:
- Data infrastructure modernization.
- Unification of the benefits of data lakes and purpose-built data stores.
- Innovation and reimagining old processes with AI/ML.
At the same time, modern data architecture:
- Is scalable, performant and cost-effective.
- Includes purpose-built data services.
- Supports open-data formats.
- Manages decoupled storage and computing.
- Provides seamless data movements.
- Allows diverse consumption mechanisms.
- Ensures governing and security.
Why you need a modern data architecture
If your objective is to gain and retain a competitive position in the market, your data will benefit from a whole new approach, as well as cloud-native tools and other components to administer, integrate and process them.
While conventional on-premise data analytics is unable to process the steadily expanding data amount or scale fast
enough, modern data architecture ensures:
- Accelerated and improved decision-making by eliminating data silos.
- Enhanced customer experience, and thus loyalty.
- Data-driven insights for innovation and competitive advantage.
- Business situations understanding, more accurate future outcomes prediction.
- Business process optimization and operational cost reduction.
What business challenges modern data architecture solves
More data than ever generated
The unprecedented pace of data volume growth produces more obstacles with its storage, processing and analysis.
Dealing with diverse types of data
On-premise data storage is far from being the most cost-efficient, best-scaling option to manage recent types of data, such as structured data, unstructured data, or real-time streaming data.
Traditional data warehouses cannot cater to the functional requirements of machine learning if future-oriented innovation is your choice.
Data compliance & security
Traditional data solutions are incomparable to modern architecture apps with inherent data compliance and more robust security.
Six layers of modern data architecture
Modern data architecture is designed with the aim of assembling, storing, converting, and analyzing data, making the right information easily accessible to necessary systems and individual users. Such architecture is handy for business intelligence, more sophisticated analytics such as machine learning, or a central data hub.
The foundational components of modern data architecture represent a layered structure. Visualize a modern data architecture as a stack consisting of six distinct layers:
- Ingestion layer
- Storage layer
- Processing layer
- Consumption layer
- Visualization layer
- Governance & security layer
What modern data streaming architecture enables is receiving, managing and scrutinizing large amounts of rapidly-changing data for more responsive real-time customer experiences. Each layer of the modern data architecture contains purpose-built components, designed to tackle specific demands, and is to be described in detail further.
The Ingestion Layer’s key function is the transfer of information from its original source to the data platform. This process usually entails extracting the data, inspecting their quality, and keeping the data in the staging area of the platform. An important feature of the Layer is supporting the data source connections with the use of drivers or libraries.
Elicited from its name, the Storage Layer is supposed to retain or archive the data protected from disasters, malfunctions or user errors, at the same time providing easy access to authorized users. A Storage Layer is typically represented by solutions like cloud storage, NoSQL database or Graph database, and modern data architecture companies (e.g. Hadoop).
This Layer manages the data transformation so that they duly correspond to the previously established data model. Depending on the nature of the data source and the requirements for data availability, the processing can be executed in batches at a scheduled time or in real time. Should the data model be modified for any reason, the Processing Layer converts the information again to match the data model of the Storage Layer.
The Analytics Layer is where the properly prepared data are subjected to various types of analytical models, including ML, with the view of detecting trends and answers to business questions. The output of the analytics can be sent to the Storage Layer or the Visualization Layer. The Analytics Layer can be classified into two categories: Business Intelligence and Advanced Analytics. The former involves reports that present KPI results related to business performance, while the latter uses more advanced algorithms to generate results.
Once the data have been extracted, transformed, stored, and analyzed, they are ready to be revealed to the end-user within the Visualization Layer, which can take the form of different tools, also illustrated below. The aim is to withdraw valuable information that provides insights and assists in making informed business decisions.
Data visualization can appear in the following four types:
- Self-Service BI
- Embedded Analytics
Governance & Security Layer
Ensuring data security is crucial to any data platform, and the essence of the Governance and Security Layer. Speaking of the latter, user authentication and authorization, data encryption, and audit trails, among other means, can be engaged in the prevention of unauthorized access, modification or disclosure of information.
As per data governance, it refers to storing data, and metadata, in a centralized catalog that allows a unified logical view of the information across an organization, managing master and reference data, and maintaining the data lineage.
The strategy for modern data architecture
On your way to a more convenient, secure and accurate data architecture that best meets the present-day data-based challenges, you are likely to benefit from a modern data strategy. It is supposed to assist you in breaking down silos and entitling your team to use the most appropriate and effective tools or techniques in order to get the most out of data analytics.
The modern data strategy is reliant on these three pillars:
- Modernization – data infrastructure is best modernized through a cloud migration as opposed to self-management.
- Unification – data silos are to be broken down to make the information centralized and available across databases, data analytics, and ML services.
- Innovation – creating new experiences and enhancing existing processes lead to new business insights and outcomes.
The pillars of modern data strategy do not necessitate a specific implementation order. Your current requirements and objectives govern the process, even if it means applying the three pillars simultaneously. We will expand on the pillars in more detail thereafter.
Modernization, associated with cost optimization and performance improvement, is best attained through migration or adoption of cloud-native services. At the outset, one can open with either designing one’s own scalable data lake, or choosing among the suggested purpose-built data services. The major points to consider include security controls and centralized access to authorized systems or individuals, cost-efficient scalability, and integration of machine learning tools into the architecture.
Well-informed timely decisions based on the holistic view of the enterprise require scaling data stores to collect the emergent data, unified according to the data model, seamlessly connected and available across the organization’s various systems, to be further processed and scrutinized with the most advanced analytics algorithms and machine learning tools.
Any stage of modern data strategy can incorporate innovation. Any kind of modern data architecture services, whether it comes to databases: relational or non-relational; to data services: purpose-built or data lakes; to processing tools: sophisticated data analytics algorithms or machine learning solutions, – can be drawn by aiming at improved customer experience and innovation.
How modern data architecture on AWS is arranged
AWS data architecture allows the swift construction of one’s own data lakes and provides a wide collection of purpose-built data services to opt from. Whichever you pick, you are in for security compliance, unified data format, centralized access governance, seamless scaling and convenient data connectivity, regardless of their origin. AWS services combine the advantages of both data lakes and purpose-built data stores without having to choose between boosted performance and optimized cost.
So as to modern data architecture on AWS, it includes four principal components, which will be explained thoroughly:
- a scalable data lake
- purpose-built data services
- seamless data movement
- unified data governance.
AWS data lakes
The core of modern data architecture is a data lake. A more secure, scalable and agile version of a data hub, a data lake allows discovering, storing, sharing, governing and processing of all kinds of data – structured and unstructured alike. Analytics in AWS data lakes unlocks the possibility to choose the tools as well as the type of examination for the best-informed business decisions, including big data processing, real-time analytics, and machine learning.
The benefits of AWS data lakes
- Collect and store any type of data, at any scale, and at a low cost
AWS modern data architecture experts ensure that either relational or non-relational, either structured or semi-structured, just like unstructured data are managed cost-effectively, regardless of the data volume.
- Use a broad set of analytic engines
The more sophisticated analytical algorithms, including ad hoc analytics, real-time streaming, predictive analytics, ML and AI, are supported by modern data architecture engineering services available on either pay-as-you-go subscription or on-demand.
- Catalog, search, and find the relevant data in the central repository
Modern data architecture services are also able to operate on data directly, without data transfer. This is enabled by a centralized data catalogue that handles data within a data lake.
- Quickly and easily perform new types of data analysis
AWS offers more than 50 services to deal with data of any amount, velocity or type. AWS’s modern data architecture services and features are there for extracting, storing, processing, analyzing and visualizing data in the cloud.
- Secure the data and prevent unauthorized access
AWS modern data architecture consultants make sure that your cloud systems are compliant with such certifications as ISO 27001, FedRAMP, DoD SRG, and PCI DSS to help any workload or infrastructure meet the most rigorous requirements.
Purpose-built data services
The second key component of modern data architecture is purpose-built data services. Such tools are custom-built for specific use cases to optimize cost, performance, scaling and functionality so that you do not have to compromise either of those. Moreover, AWS tools have various deployment options to choose from for anyone to commence straight away. We will further review some of the extensive variety of such data tools offered by Amazon.
Data warehousing – Amazon Redshift
If what you require are rapid query results for structured data, you may benefit from using a data warehouse.
Revolutionizing the data warehousing economics, Redshift was the first data warehouse initially designed to function in the cloud, yet it is still the leading cloud data warehousing option offering three times the price performance of its competitors. Its consistently high performance combined with predictable costs makes Redshift one of the most reliable choices for cloud data warehousing. Numbers suggest that Redshift supports queries of up to an exabyte scale run against information from your data lake, and manage as much as petabytes of data within its clusters. Redshift is the answer to the request for easy analytics, irrelevant of the data storage locations, for limitless on-demand scaling, and for the first serverless computing analytics experience.
Interactive query – Amazon Athena
Should you require direct data querying in your S3 data lakes, Amazon Athena is the serverless solution you will benefit from. Designed with a simple SQL interface with JDBC and ODBC drivers, Athena does not need setting up or server managing: by directing data to Amazon S3 and defining the schema, one can quickly commence data querying with an integrated query editor and receive outcomes in various formats (e.g. CSV, JSON, Avro, Parquet, or ORC) and compression types (e.g. GZIP or BZIP).
Big data processing – Amazon EMR
For managing huge volumes of data, modern data architecture consulting firms recommend Amazon EMR. The incrementally scalable infrastructure allows EMR to handle data of any size by creating entirely managed data clusters with up to thousands of compute instances within such distributed frameworks as Spark or Presto.
A practical and fully administered service, EMR also provides clearly predictable per-second pricing model, which means you will only pay for the cluster’s active time. It proves optimal for periods of fluctuating usage by granting flexibility to manage your resources. Cost-efficient EMR makes a good match with Amazon S3 for storage, and with EC2 Spot Instances for computing with the use of on-demand instances.
Log and search analytics – Amazon OpenSearch
Amazon OpenSearch service is usually the best fit for those that monitor the production systems’ performance or troubleshoot problems by inspecting substantial volumes of log data while also prioritizing data security.
A fully maintained service, OpenSearch enables you to launch a production-ready OpenSearch or Elasticsearch-like cluster in a matter of minutes, not to mention that scaling occurs through a mere API call or via a console. OpenSearch supports a number of AWS services, including Logstash, Kinesis Data Firehose, IoT and CloudWatch Logs.
Real-time analytics – Amazon Kinesis & Amazon MSK
At last, real-time data streaming is possible with services like Amazon or Amazon MSK. The former is able to provide the quickest insights based on the mining and analyzing of real-time streaming data. While the focus of the orderly organized latter is placed on the security of the data within clusters, inasmuch as MSK patches up unhealthy components, this Apache Kafka service also boasts of its availability.
As pointed out above, numerous AWS purpose-built data services have been created to cater to the diverse analytical needs among various workloads. For a straightforward and quick start with data analytics, one had better choose among the services maintaining serverless options – these will help you put your data-driven insights into well-informed business decisions as soon as possible.
Seamless data movement
An indispensable element of modern data architecture execution is data movement: into and out of the data lake, and across the purpose-built data stores. A service which can provide seamless data movement is AWS Glue.
Amazon’s serverless data integration service facilitates data discovery, collection, cleansing, transfer and integration whether with the aim of analytics, ML, or solution design. Glue supports unified information catalogues with centralized governance of the connected array of more than 70 separate data sources. The service can take over the construction and management of ELT pipelines to smoothly move and load information among data lakes and purpose-built data stores.
More thoroughly, AWS Glue:
- Is a serverless tool for carrying out data integration jobs, which are at the core of effective information management. How Glue processes these jobs or end-to-end workflows can be twofold: either by event or as a pre-scheduled implementation.
- Supplies a fully managed metadata repository, the Glue Data Catalog, which enables ELT operations from a centralized metadata location, also handy for data processing and accessibility. Glue Crawlers is another feature that reinforces the narrated functions further to ensure automated information cataloguing and discovery across sources.
- Offers off-the-shelf connector features to ensure data connectivity, irrespective of their location, which can range from on-premise sources to a different cloud service provider. Apart from the integrated Python, JDBC, and ODBC connectors to get built-in access to multiple databases, including log and file-based data, Glue developed a marketplace with third-party connectors, counting BigQuery, Salesforce and other SaaS systems in.
- Assists you in constructing data integration jobs with a broad variety of personalized tools, such as data integration jobs visualization with Glue Studio; a data wrangling interface with Glue DataBrew; or an interactive notebook with Glue Notebooks.
Data integration jobs can further be facilitated by ML, automatic code generation and template-driven development supported by AWS Glue which goes beyond data transformation, data lake construction, or utilizing purpose-built data tools, being powered by modern data architecture.
Unified data governance
Collecting and transforming data within your data lakes and purpose-built data repositories aside, the information needs to be safeguarded; and it is simply assured by establishing unified data governance. Data security essentially encompasses two elements: centralized access control and regulatory obligations compliance. Both of these are provided by Lake Formation, developed by AWS to set up a valid unified data governance solution.
The primary feature of AWS Lake Formation is centralized data authorization, seamlessly functioning with machine learning and analytical tools.
Fundamentally, AWS Lake Formation enables:
- Building, managing, and protecting data lakes with prompt use of familiar database-like features.
- Streamlining governance on a larger scale, and authorizing permissions on both marketplace-level and asset-level across your data lake.
- Eliminating data silos and enhancing data availability with a data catalogue.
- Facilitating data accessibility at the scale of enterprise with cross-account data sharing.
Modern Data Architecture with AWS Cloud FAQ
AWS modern data architecture can significantly contribute to your business growth by providing scalable, cost-effective, and secure data management solutions. It allows you to handle increasing data volumes and diverse data types efficiently, enabling you to derive valuable insights for strategic decision-making. Moreover, the architecture's support for machine learning can facilitate the development of predictive models and automation, driving innovation and operational efficiency.
Implementing AWS modern data architecture may present challenges such as data migration, ensuring data security, and managing costs. These can be mitigated by careful planning, using AWS's robust security features, and leveraging AWS's flexible pricing models. AWS also provides extensive documentation and support services to assist businesses in navigating these challenges.
AWS analytics architecture can significantly enhance your business's data analysis capabilities by providing a wide range of powerful tools for data processing, storage, and analysis. These tools can handle large data volumes, real-time data streaming, and complex analytical tasks, enabling you to derive valuable insights quickly and efficiently. Additionally, AWS's machine learning services can help you develop predictive models to anticipate future trends and make data-driven decisions.
Some best practices for maintaining and optimizing AWS modern data architecture include regularly reviewing and updating your data management strategies, ensuring data security and compliance through AWS's governance features, and optimizing costs by leveraging AWS's flexible pricing models. Additionally, it's important to stay updated with AWS's new releases and features, as they can provide new opportunities for enhancing your data architecture.