The Power of Customer Insights + Synapse

One of the most common problems enterprises face today is disparate data. While most businesses have collected large amounts of customer data, it may be spread out over multiple tools or platforms, in separate data warehouses, or even in departmental siloes. Disparate data sources make deriving insights from all your data virtually impossible, which means that your analytics may not be giving you a complete or current picture of your customer.

Just over a year since its availability, Microsoft continues to enhance Synapse Analytics, making it an increasingly viable solution for big data processing and predictive modeling. Synapse enables interactive data exploration, and self-service and predictive analytics based on machine learning.

Synapse is a set of services and technologies that allow you to analyze and synthesize data from data warehouses and big data systems. You can use the Spark within the synapse environment, or you can use Synapse to analyze data coming out of another Spark-based environment. Combine this with Dynamics Customer Insights and the potential business benefits are exciting. You can start to ask yourself: “What is likely to happen in the future based on previous trends and patterns?”

Microsoft’s Customer Data Platform, Customer Insights, recently added Azure Synapse as a direct export destination or integration. Having this direct connection to export Customer Insights data to Azure Synapse relieves the burden of needing an extra ETL process or the additional overhead for data storage, because exported CI data is persisted in Synapse.

Customer Insights is already a great tool

Customer Insights on its own is a very useful tool for enriching segmented customer data. It’s a prepackaged solution, but it’s more than just a view. You can use it to manually dictate the relationships between ingested data sources with inherent ‘map, match and merge’ technology — which simply maps the data, matches it with another data source, and merges it into a customer profile.

When output to Synapse, CI data is stored in the Common Data Model format. A large and growing collection of solutions work seamlessly when data is stored in Common Data Model form. This means you can quickly implement new business processes and gain insights into your customers without friction or added complexity.

Why do we need Synapse?

The benefits of using Azure Synapse are abundant from a variety of perspectives, including integration, performance, and diagnostic capacity. In the remainder of this article, we’ll dig deeper into how Synapse works and opens a host of possibilities for your analytic efforts.

Fundamentally, Synapse streamlines data exploration and discovery by removing the need to transform data from one format to another or move data to other systems. This lets you experiment, by mapping and correlating the different datasets to produce curated datasets that are ready for consumption and for porting back into other platforms, such as Customer Insights.

But the challenge of joining highly curated relational data with a broad array of variable and semi-structured data in a way that provides meaningful and predictive insight is harder than it sounds, and Synapse aims to alleviate that complex difficulty.

It’s all about the data

Customer data comes from everywhere and is in everything: customer service calls, website visits, purchases in online and physical stores, and mobile app usage. These are just a few of the channels that generate large volumes of data every minute.

Azure Synapse Analytics can pull, store, and analyze this data in real-time or near-real time with using Azure Synapse Link, eliminating the extra layers of storage and compute required in traditional ETL pipelines for analyzing operational data. You can also use Azure Cognitive Services for mining unstructured data such as text, language, images, videos, and speech processing using prebuilt APIs. With all that data, you can then apply machine learning models to drive better predictive analytics and drive those models into Synapse.

Here’s a high-level diagram of how Synapse brings together unstructured and semi-structured data with the data generated by Customer Insights and Dynamics 365 (all underpinned by the common data model).

Single service interface

Data platforms typically need to address multiple data practitioners in an organization (data engineers, business analysts, data scientists) all with differing skillsets and needs. These demands are often oppositional and providing a single toolset with the capability to support everyone can be very difficult. This is where the Azure Synapse Analytics with its unified interface and flexible delivery options delivers.

The Synapse workspace (or studio, as it’s sometimes called) is a web-native experience that ties everything together so all data practitioners can play, all in one location. For example, business users such as marketers, customer experience (CX) professionals, and product owners can benefit from pre-packaged models such as customer churn and CLV analysis. Data scientists and skilled business analysts can use robust tools for building critical models themselves.

The following screenshot of the Synapse workspace illustrates how an existing Power BI report can be edited within Synapse. Any saved changes will be written back to the Power BI workspace.

How Synapse works

The main function of Azure Synapse Analytics is to transform and aggregate data into a format suitable for analytics processing. You can then apply to model and perform complex queries.

Import and storage

Synapse contains tools to create and schedule data-driven workflows (called pipelines) that ingest disparate data for storage in many different data lake options, including Cosmos DB and Azure Data Lake Storage Gen2. This stored data can include Customer Insights data along with operational data and unstructured/semi-structured data.

Azure Data Lake Storage Gen2 is dedicated to big data analytics, and is built on Azure Blob storage so you to easily manage massive amounts of data. Data engineers can either use the Synapse code-free data pipelines or build custom code using Jupyter notebooks on serverless Apache Spark pools to curate, correlate and transform data for building consumption-ready datasets.

Analyze and query

Using the models, data in the Gen2 lake, Azure Synapse SQL and Apache Spark pools can further transformed and enriched.

Azure Synapse SQL is a distributed query system that enables you to implement data warehousing and data virtualization scenarios using standard T-SQL experiences familiar to data engineers. Synapse SQL offers both serverless and dedicated resource models to work with both descriptive and diagnostic analytical scenarios. Serverless Apache Spark pools to curate, correlate and transform data to build consumption-ready datasets.

• A dedicated SQL pool offers T-SQL based compute and storage capabilities. After creating a dedicated SQL pool in your Synapse workspace, data can be loaded, modeled, processed, and delivered for faster analytic insight.
• Serverless SQL pools let you use SQL without having to reserve capacity. Billing for a serverless SQL pool is based on the amount of data processed to run the query and not the number of nodes used to run the query.

Synapse uses Massively Parallel Processing (MPP) to quickly run complex queries across petabytes of data. It is powered by Apache Spark, which is not only secure and reliable, but integrates deeply and is optimized for Azure databases and stores. SQL and Spark pools give you the freedom to query data on your terms, using either serverless or dedicated options—at scale. In the world of analytics, speed is paramount, because the value of data depreciates with time.

Here’s a high-level sample architecture diagram of how the technologies we’ve discussed fit together in a Synapse-driven environment.

As you can see, compute and storage workloads are functionally separated meaning that you only use (and therefore pay for) the compute services as and when you need them and when they are not in use they can be paused. This is especially necessary for analytics where workload demands are not uniform.

About Machine Learning

A machine learning model is a file that has been trained to recognize certain types of patterns. You train a model over a set of data, providing an algorithm that it can use to reason over and learn from that data. You could use machine learning to create a model that predicts certain key attributes around your customers.

Machine learning models can be integrated into the Synapse environment where they land in Azure Blob Storage and are accessed via Synapse SQL, along with the data in the Gen2 lake for processing as we discussed above.

Azure Synapse is truly a complete experience to ingest, prepare, manage, and serve data for immediate BI and machine learning needs. When combined with the 360° customer profiles in Customer Insights, organizations have exciting new opportunities to capture new market segments. For example, you can monitor and analyze social media as well as foot traffic in a store to implement a real-time customer feedback loop and make insight-driven decisions. The potential is really limitless.

Take the first step

“Data is only useful if you, in real-time, can predict something better, can automate something better, or gain an insight,” Microsoft CEO Satya Nadella said at the National Retail Foundation’s NRF 2020 Vision: Retail’s Big Show. “That is the true measure of your success with data.”

At Hitachi Solutions, we’ve been providing modern data platform solutions for our customers for many years. Our Intelligent Customer Experience in a Day workshop will get you started on the right path to understanding your data ecosystem and identifying key opportunity areas. We’ll provide an overview and demonstration of the joint capabilities of Customer Insights and Azure Synapse Analytics, then identify a potential use case and offer a sample architecture design and roadmap for moving forward.

Talk to us today.