How to build a
Modern Data Platform
The Modern Data Stack (MDS) represents the latest approach in data platforms.
However, this evolution didn't happen overnight.
The advent of big data technologies and cloud computing significantly reduced data processing costs.
This reduction enabled the development of more complex data tools capable of processing large volumes of data.
Algorithms, statistical models, and predictive models have become more accessible, turning data intelligence into a significant business opportunity.
This blueprint is designed to help you learn everything about the modern data platform that we have adopted at Indicium for clients, partners, and our internal use.
You will understand what the Modern Data Stack (MDS) is, along with its principles and characteristics.
Additionally, you will learn how to build and implement a data stack using cutting-edge tools to revolutionize data management within your company.
Enjoy!
competitive differentiation
more flexible methodologies
CHALLENGES IN MODERN DATA OPERATIONS
In the coming decades, the volume, speed and variety of data will reach unprecedented levels.
According to the International Data Corporation (IDC), the Global DataSphere is expected to grow by 500% by 2025. Data has never been as crucial in the business world as it is today.
Companies are becoming more agile in identifying signals within their data, optimizing outcomes as a result. This agility leads to more efficient decision-making, enabling swift responses to business challenges.
Customer behaviors, inventory, products, market trends, and many other data points can now be tracked and analyzed to provide businesses with critical real-time insights.
But it’s not all smooth sailing…
The more data is generated, the more challenges arise in organizing, integrating, and managing it. To begin your immersion in MDS, here are some basic needs that must be met:
.png){kind=icon}
More efficient cloud storage and computing
.png){kind=icon}
Integration of data architectures
.png){kind=icon}
Automation of routines with artificial intelligence
And you should know that, to meet these needs, data management solutions are under pressure to be…
.png){kind=icon}
Fast, efficient and capable of handling large volumes of information
.png){kind=icon}
Flexible, to incorporate multiple versions of the truth
.png){kind=icon}
Accessible, in order to add value and simple, for a successful user experience
Modern data platform:
transition in action
As companies grow, the complexity of their data operations, methodologies and systems, which once worked on a small scale, become obsolete and start to cause harmful friction to businesses.
Therefore, they must be replaced.
To increase analytical maturity and gain a competitive advantage in the market, organizations need to transform data into business assets, generating innovation, process improvement and cost optimization.
To put all of this into practice, we have created the POD, our methodology based on three fundamental pillars:
1 - People
2 - Organization
3 - Data
In that transition scenario, a question arises:
How can a company prepare its data operations and infrastructure to handle so many challenges?
The answer is by implementing a modern data management approach, known as the Modern Data Stack (MDS).
This innovative solution democratizes the collection, integration, and management of data for all stakeholders.
Discover the three main benefits of the modern data approach:
.png){kind=icon}
Teams have autonomy but operate under guidance and rules set by the central team.
Control and approvals are managed by the central team.
Departments with autonomy to accelerate development.
Speed with quality: better understanding of demand and business impact.
.png){kind=icon}
A combination of governance, speed, and quality allows for greater financial returns for companies.
A company needs to be prepared for increasingly complex data operations. This entails having a modern data platform like the one we use at Indicium.
You are going to learn about it next.
WHAT IS A Modern Data Stack?
The modern data platform we've adopted at Indicium goes by various names. Here are a few:
- Modern Data Stack (MDS)
- Modern Analytics Stack (MAS)
- Data Stack
- Analytics Stack
Despite different labels, they all refer to the same data methodology that emerged to address a central business problem: developing companies' analytical capabilities to meet everyday challenges.
What challenges?
The rapid advancement of new technologies, the increase in data volume and the growing complexity of businesses, for example.
How does the Modern Data Stack solve this?
There's no magic.
To support the scalable growth of modern data operations, we create adaptable, accessible, and flexible data infrastructures by combining these three factors:
.png){kind=icon}
Best practices in data science
Specific analytical tools
Innovative technologies
The modern data approach is not a standalone technology.
It integrates various technologies to address the data science, analytics and artificial intelligence challenges faced by modern businesses.
In practical terms...
A company with a structured Modern Data Stack uses independent, complementary tools and technologies that perform specific functions throughout the data cycle.
For instance, consider a company that has drastically increased its inventory and customer base and, therefore, needs a more robust data storage solution.
Thanks to the data stack, it can adapt its current solution or replace it with one that meets the new business demands.
The best part is that this can be done without completely overhauling the entire data infrastructure.
With the MDS, much like Legos, organizations have the flexibility and autonomy to replace pieces and make specific adjustments without dismantling their entire data structure.
THE PRINCIPLES OF THE MODERN DATA APPROACH
To build an efficient and modern data approach, it's essential to thoroughly understand its key characteristics and guiding principles.
With that in mind, we've listed seven principles of one of the most advanced concepts in the field of data. Take a look!
Cloud-based
Data storage is completely centralized in the cloud, utilizing scalable and flexible technology that allows for processing an infinite amount of data in an online and secure environment. This approach significantly reduces costs associated with infrastructure, installation, and maintenance.
1
Modularity
Separation of business rules from the ELT process stages, particularly in the extraction and loading phases, allows for the use of third-party tools to continue the data integration process with minimal investment.
2
Simplicity
Data transformation is driven by one or a few widely known programming languages and executed centrally. This approach brings benefits such as the democratization of information and the reduction of training and maintenance costs.
3
Governance
Easily accessible and centralized information simplifies data documentation and governance. This enables the creation of permission logic and the integrated management of sensitive data.
4
Versioning
Utilization of versioning best practices allows for collaborative work without conflicts in data projects, thanks to the modern ELT tools used in this approach.
5
Environment separation
Creation of distinct environments to separate raw, transforming, and final data, consequently facilitating access to different development stages, promoting collaborative work, and reducing production mistakes.
6
Testing
Execution of best practices in testing for data projects, similar to those in modern software development, ensuring the consistency and reliability of results.
7
Now that you're familiar with these fundamental principles of the modern data approach, let's talk about how to implement them in your company.
For more efficient results, data teams need to be structured and integrated to adopt the MDS effectively.
To structure the team, companies invest in data training, thus building a solid data-driven culture in the process.
BUILDING THE MODERN DATA APPROACH
Thanks to new technologies and accessible tools, companies of all sizes can now adopt a modern data approach.
However, it's essential to understand how all the pieces fit together to successfully implement these practices, from the stages of the data stack to defining the technologies and tools for their execution.
One of the most striking features of the modern data approach is the integration of various tools and technologies into a data stack.
Do you know what a data stack is?
The data stack is a collection of processes, tools, applications, and technologies responsible for automating data management in businesses across the entire data pipeline.
Do you know what a data pipeline is?
In data, pipelines encompass all the stages of data processing, from the input system to the final destination of the information. In other words, it's the entire composite of this process.
In practical terms...
The first step in implementing the MDS is structuring, followed by configuring the data stack.
With this setup, it's possible to respond to data operations demands in a highly efficient manner.
For your company to have an efficient data stack, there are five specific stages along the data pipeline. The main Modern Data Stack tools and tasks are summarized in the following diagram, categorized by their respective stages.
Data Stack: the five stages
Data storage and management
Now, the data is prepared and stored in data lakes and data warehouses. These scalable and secure structures enable large-scale information analysis and management, making them fundamental components of the data stack.
Data transformation
Here, due to the massive volume of data for processing, we use the ELT flow (extract, load, transform) in the Modern Data Stack, instead of the traditional ETL. This approach is faster and more flexible, as data transformation occurs shortly after the collection and integration of information into a centralized repository, rather than before, as in ETL.
This allows for the transformation of raw data into modeled data within the data warehouse.
Business intelligence and data analytics
Analytical intelligence is the final priority within the Modern Data Stack. After setting up the previous steps, the information and insights that add value to business decision-making become accessible in real-time.
As a result, business managers and departments can visualize data, identify trends, optimize processes, and act quickly with the help of business intelligence tools, interactive dashboards, and automated reports connected to a data warehouse.
Advanced analytics
In the final and most advanced stage of the MDS, it is possible to apply and develop highly complex techniques such as machine learning, artificial intelligence, and predictive modeling within the modern data infrastructure set up in the previous stages. Examples of these techniques include recommendation models and prescriptive modeling.
Data collection, integration and cleansing
Companies have their own processes for collecting relevant data. At this stage of the data stack, data can be collected and integrated from multiple sources such as CRMs, Excel spreadsheets, social media, and more, efficiently centralized in a data warehouse.
This is the moment when, with the help of the right tools, necessary adjustments are made to prepare the data for the next stage.
In practical terms...
Every data stack built on the basic operations described above fulfills the necessary requirements to support the scalable growth of modern data operations.
Hence, even though the architecture of a data pipeline may vary among companies, all of them should incorporate these processes.
The concepts addressed so far are fundamental for the upcoming explanations about the recommended tools and technologies for implementing the Modern Data Stack methodology in your business.
Data stack: THE MODERN DATA APPROACH's MAIN TOOLS
One of the most distinctive features of the Modern Data Stack is the integration of various data tools and technologies to meet the demands of current data operations.
In addition to the stages of the data flow, the tools used at each stage of the pipeline are essential elements that determine the success or failure of implementing the MDS.
Therefore, a modern data operation requires the combination of various services and tools into a data stack.
Check out Indicium's analysis of the main tools available in the market for building a modern, scalable, and flexible data approach below.
Data collection, deployment and transformation tools
Data collection and integration
Fivetran, Stitchdata, AWS Glue, and Google Cloud Dataflow are leading tools in data collection and ingestion. They enable the transfer of data from hundreds of sources, such as ERPs, CRMs, databases, REST APIs, and more, directly into a data warehouse (whether cloud-based or on-premises). Additionally, these tools can be combined, eliminating the need for significant investments in software licenses or implementation hours.
.png)
.png)
Deployment
Docker and Kubernetes are widely used tools for deployment, along with orchestrators like Airflow, and infrastructure management tools like Terraform. These tools ensure all the “Lego pieces” communicate harmoniously with each other, allowing data to flow in sync throughout the data structure.
.png)
.png)
.png)
Data transformation
dbt (data build tool), Dataform, Spark, Matillion, and Coalesce are among the leading tools in the Modern Data Stack for data transformation. Together, they enable the execution of the ELT process to transform raw data into modeled data within a data warehouse, which is a fundamental step in the modern data approach.
.png)
.png)
Data storage and management tools
Today, we have two practical, viable, and efficient options for data storage, both on-premises and in the cloud: data warehouses and data lakes. Both should be evaluated on a case-by-case basis, as they present technical and conceptual differences in terms of architecture and purpose.
Data warehouse
Amazon Redshift, Snowflake, Google BigQuery, Databricks, and PostgreSQL are among the top tools for data warehousing. They feature scalable cloud architectures that allow for quick storage and querying of massive data volumes. These tools are essential for building a modern data approach due to their efficiency and scalability.
.png)
.png)
.png)
Data lake
Dremio, Amazon S3, Apache Hadoop, Google Cloud Storage, and Azure Data Lake are highly recommended data lakes. These platforms can store all types of data—structured, unstructured, and hybrid—in one place, making them essential for a modern data approach.
.png)
.png)
Cloud computing
AWS, Google Cloud, and Microsoft Azure are the leading cloud computing providers today. You can choose among these options based on your company's needs for storing and managing your data.
.png)
.png)
Data analytics tools
Modern data analysis involves various activities that vary in terms of complexity and tools.
To facilitate understanding, they can be divided into two categories:
(1) business intelligence tools and (2) advanced analytics and generative AI.
Business intelligence
Metabase stands out among several robust open-source alternatives, while Microsoft Power BI, Looker, and Tableau excel as SaaS (Software as a Service) platforms.
With a modern data infrastructure in place, these business intelligence tools can be utilized to visualize, analyze, and generate insights from data, enhancing decision-making and strategic planning.
.png)
.png)
Advanced analytics
MLflow and Kedro assist in executing predictive and prescriptive models, optimizing the development process, and reducing the time between modeling and deployment.
Tools like Apache Spark, TensorFlow, and PyTorch also play crucial roles in advanced analytics.
Machine learning, artificial intelligence, and data modeling are techniques applied in advanced analytics for more complex analyses within the data stack, utilizing these tools along with various libraries in R and Python languages.
.png)
.png)
Generative AI
OpenAI, H2O.ai, Gemini, and Amazon Bedrock are leading platforms in the field of generative AI. These tools assist in creating sophisticated models capable of generating text, images, and other data types, significantly enhancing the development process and reducing the time between concept and deployment.
Generative AI leverages advanced machine learning techniques to produce new content, offering powerful solutions for complex data analysis and innovative applications within the data stack.
MODERN DATA APPROACH CASES
The digital revolution has driven companies of all sizes to seek innovation.
In this context, the Modern Data Stack is no longer an option but a necessity for companies to remain competitive.
This is where Indicium comes in: transforming not only the way data is managed but also the data-driven approach to business decision-making.
Companies with modern data approaches
Each case is a unique success story, highlighting the specifics of the practical application of advanced technologies in real-world scenarios.
Through the partnership with Indicium, companies from various sectors were able to:
.png)
+ accelerate decision-making with instant access to actionable insights.
.png)
+ improve operational efficiency through automation and process optimization.
.png)
+ enhance growth by identifying new market opportunities.
.png)
+ foster a data-driven culture in which every decision is based on informed analyses.
We work diligently to make a difference in various industry sectors with the Modern Data Stack.
Next, learn more about the applicability of MDS by sector.
Solutions tailored to your industry
Built for and used by customers just like you.
.svg)
.svg)
.svg)
.svg)
.svg)
.svg)
ConclusION
There are many decisions involved in a company’s data operations, from development to enhancement. Up-to-date information can be hard to find and even harder to understand.
A Modern Data Stack (MDS) solves this issue by creating an integrated business intelligence approach that collects, combines, analyzes, and delivers the value of data.
Fortunately, the components of the data stack are much cheaper and simpler to set up and handle. As a result, companies of all sizes can gain a competitive advantage and develop analytical maturity.
We can help you understand your current data management needs, assess your options, and guide you through the next steps.
.jpg)
.jpg)
.jpg)
.jpg)
.jpg)
Indicium is a global data service company headquartered in New York City, with over seven years of experience collaborating with prominent clients such as PepsiCo, Burger King, Bayer, Kenvue, and Novo Nordisk.
We specialize in the Modern Data Stack, supported by a robust delivery center in the Americas.
Our team is certified in leading modern tools and trained in-house to deliver standardized, high-quality work.
+6 YEARS
of experience with the MDS
+120
MDS platforms
+200
AI/ML models
+500
data products
+600
data strategy consultations
+2,000
professional certificates issued
R$ +1BI
ROI for clients
+150%
ROI per project
+10,000
training hours conducted
Along our journey, we believe that every piece of data has a story to tell.
At Indicium, we transform each of these stories into profitable and sustainable business strategies. If you wish to unlock the potential of your data with the Modern Data Stack, we can make that a reality for you.
Indicium is data-driven and specialized in creating data science, analytics and AI solutions
We are a data company globally recognized for our cutting-edge solutions.
.png){kind=icon}
.png){kind=icon}
Prepare your organization for decades of data-driven innovation.
Connect with us to learn how we can help.