SAP Data Replication

SAP Data Extraction: Useful Tools You Need to Know

SAP Data Extraction, its various modes and mechanisms, are the topic for this blog. SAP with its various systems and functionalities, has several data extraction methods and tools in place. We will discuss ODP, OData Services and ODQ that work together as a framework to provide real-time data and other SAP extraction tools. Also, see how using BryteFlow as an automated SAP data extraction tool (with flexible connections to SAP) can make your life much easier. About BryteFlow SAP Data Lake Builder

Quick Links

What is SAP Data Extraction?

SAP data extraction refers to the process of pulling out the data from SAP applications, databases, and other systems to enable users to consume it in a format that can be understood. The data can then be used for analytics, reporting or any other business-centric objective. SAP data extraction will probably be a part of the larger ETL or ELT process, where the data is loaded and then transformed in an on-premise or Cloud data warehouse. The SAP data extraction may involve data transformation, so it undergoes vital cleansing, structuring and preparation to render it usable and ensure data quality. SAP S/4 HANA Overview and 5 Ways to Extract S/4 ERP Data

SAP Data Extraction Challenges

SAP data exists in a multitude of places. In database tables, in ERP systems, in SAP CRM applications, the list goes on. The trick is to coax the SAP data out with a minimum of fuss. This of course is easier said than done, since the SAP ecosystem is quite complex. It has its proprietary language, multiple structures and formats, and numerous versions. Many SAP implementations have varying degrees of customization too, making the data even harder to extract. However, it’s worth the trouble given that SAP encompasses a huge range of an organization’s business aspects and operations. Data Extraction with BryteFlow SAP Data Lake Builder

Why Extract SAP Data?

Extracting SAP data will allow the data to be analyzed, offer a window into business performance metrics and enhance operational efficiency, further leading to better decisions and valuable business insights. It also enables data to be analyzed in conjunction with non-SAP data, since over a period it is highly probable that an organization may have other systems running too. SAP HANA to Snowflake (2 Easy Ways)

Very often large organizations throw up a lot of data and need it to be analyzed and generate reports in real-time. In such scenarios scalability and latency are paramount. That’s where automated SAP ETL tools like BryteFlow SAP Data Lake Builder come into their own. It automates every process including SAP data extraction, CDC, merging and masking, and preserves application logic when it extracts data from SAP applications. This means it creates logic and tables automatically on target as a replica of the source. Reliable SAP CDC and Data Provisioning in SAP

SAP Data Extraction Infrastructure

In the SAP ecosystem there are multiple mechanisms and tools to extract data. The methods and tools you use will largely depend on the SAP system you are using. For example, SAP Data Intelligence has its own connection type which can be used with ABAP-based source systems. In the case of an SAP S/4HANA system, you can extract data based on CDS views which is the recommended approach, especially if you have a Run-Time SAP license or are using SAP Rise. If you have an Enterprise SAP license, you can extract data from tables in an SAP S/4HANA system. In this situation the SAP Landscape Transformation Replication Server (SLT) can be used to enable SAP data extraction. CDS Views in SAP HANA and how to create one

Operational Data Provisioning (ODP)

The foundation of many SAP data extraction methods is the ODP (Operational Data Provisioning) framework which is part of SAP NetWeaver. ODP is a unified technical infrastructure that allows data to be extracted and replicated from SAP systems. It can be used to transfer data from SAP systems into a central storage repository. The data is compressed and stored in an Operational Delta Queue (ODQ). SAP applications (Subscribers / Consumers) can then subscribe to the delta queue and read data from it. The data is stored until the last subscriber calls it. ODP can be used with various SAP source systems including SAP ERP, SAP BW/4HANA and SAP BW systems. It also integrates with other SAP systems like SAP S/4HANA, SAP ECC, and SAP SLT. SAP Extraction using ODP and SAP OData Services (2 Easy Methods)

For data recovery ODP provides a recovery mechanism with data retention periods that can be configured. The data can be stored even after the last subscriber application has called it, depending on the setting you have provided – by default this is 24 hours.

ODP reduces load on source systems since only a single connection is made to the source system to distribute data to multiple subscriber applications. The data is extracted exactly once and distributed over the ODP framework as per the interval defined or in real-time.

SAP Tools to Extract Data

SAP also has its own tools for data extraction and replication. Some of these are:

SAP Data Services or SAP BODS (SAP Business Object Data Services)

SAP Data Services or SAP BODS (SAP Business Object Data Services) as it is also known, is an SAP ETL tool that is capable of extracting data from different sources including SAP and non-SAP systems, cleansing it and transforming the data as per requirements. It can then move the data to the target destination which could be an SAP application, data mart, data warehouse, flat file, or any operational database. It provides data extraction in real-time or in batches. SAP BODS (What makes it a great SAP ETL Tool)

SAP Data Services is a versatile tool that can process data at enterprise-scale and take care of multiple functions like data integration, data quality, data profiling, and data processing. It has a graphical user interface (GUI) to help developers build and implement complex data integration workflows easily without coding. Being an SAP product, SDS integrates seamlessly with SAP systems, such as SAP Business Suite, SAP BW (Business Warehouse), and SAP HANA. However, it is a commercial product and needs a license for use. Another drawback is that it has limited real-time data capabilities and getting high volumes of real-time data may prove challenging. SAP HANA to Snowflake (2 Easy Ways)

SAP Landscape Transformation Replication Server (SAP SLT)

A part of the SAP Business Technology Platform, SAP SLT or the SAP Landscape Transformation Replication Server replicates data from SAP and non-SAP systems in real-time to the SAP HANA database. SAP SLT delivers data in real-time by using trigger-based Change Data Capture. When data is populated in the logging table, it acts as a trigger and SAP SLT captures the deltas. SAP SLT in HANA: How to use SLT Transformation Rule

SAP SLT in HANA provides real-time and scheduled replication from SAP and non-SAP systems, both on-premise and in the Cloud. However, using SAP SLT requires a license. This is not needed if the target system’s license covers SLT functionality. If not, then a separate SLT full license will be needed. One drawback is that the tool does not have advanced transformation capabilities and can only do basic transformation. Learn about SAP on AWS

SAP Datasphere

SAP Datasphere is a key part of SAP Business Data Cloud (BDC). It enables you to merge data from multiple sources including SAP systems, and third-party on-premise and Cloud environments into a single managed Cloud environment. It acts as a flexible secure platform to support real-time analytics, governed data access, a data catalog and data science objectives like Machine Learning. SAP to AWS (Specifically S3) – Know as Easy Method

It also acts as an Object Store for loading, staging and preparing data cost-effectively. It has a large, built-in range of connections to SAP and non-SAP Cloud sources and on-premise sources. SAP Datasphere has graphic-based no-code / low code tools for self-service modeling requirements. It provides built-in high-powered SQL and data flow editors for advanced modeling and data transformation requirements, and supports third-party tools and other SAP IDEs.

Components of the SAP Data Extraction Process

While extracting SAP data, the extraction process involves several elements. These are:

Sources: These refer to the places where the data to be extracted resides, like databases in SAP ECC, S/4 HANA, and SAP BW. The data sources impact the structure of extracted data and contain valuable metadata that is needed to define the data for extraction and its structure in the data extraction process. SAP Extraction using ODP and SAP OData Services (2 Easy Methods)

Extraction Methods: This refers to the processes used to extract data, mechanisms like CDC or ODP, replication tools etc. Reliable SAP CDC and Data Provisioning in SAP

Moving the Data: This refers to the movement of the data extracted, which will be transferred through a secure connection to the destination.

Loading: This refers to loading the data into the destination system which could be SAP BW or a data warehouse either on-premise or on the Cloud like Snowflake, Amazon Redshift, Azure Synapse, SingleStore, or Databricks.

How is SAP Data extracted?

SAP has several data extraction mechanisms in place depending on the SAP system you are using. These could involve the use of SAP data extractors, SAP replication tools, ABAP extractors, CDS Views or even SQL queries. SAP S/4 HANA Overview and 5 Ways to Extract S/4 ERP Data

SAP Extractors

SAP Extractors are programs that collect specific datasets from source systems. In the extraction process the extractors move the content of the fields required using a transfer structure. They can combine and modify the data while it is being collected and deliver data from specific fields of a single table, multiple outer or inner joins between views, tables, or ABAP programs with their own business logic. Extractors are pre-defined and encompass a variety of modules and data types. They can handle transactional and master data and are often used together with SAP Business Warehouse (SAP BW) to provide data to analytical systems. How to create an SAP OData Service for SAP Data Extraction

Types of SAP Extractors

Transactional Data Extractors

SAP extractors can be categorized by function. For example, Transactional Data Extractors are created for the extraction of operational data (e.g. Purchase orders, Sales orders). The data is transferred to data warehouses like BW for generating reports and analytics. They rely on standard SAP mechanisms like FI, CO, LO Cockpit or custom-built extractors using Function Modules. Very often they support the extraction of deltas, extracting and transferring only data that has changed or is new. SAP ECC and Data Extraction from an LO Data Source

Master Data Extractors

SAP master data extractors are used in the extraction of basic, unchanging data from SAP systems, so it is available for use in other systems, such as SAP BW (SAP Business Warehouse) or SAP IBP (SAP Integrated Business Planning). These extractors can be generic and work with any table or view, or can be application-specific, to be used with designated SAP modules. These extractors are responsible for populating analytical and planning tools with the correct master data so data and structures remain consistent across IT systems and applications. SAP to Snowflake (Make the Integration Easy)

Business Content Extractors

These extractors are pre-defined by SAP and co-relate to the various modules within an SAP system. They are part of the standard Business Content (which needs to be first installed in BW) and come with pre-built structures and logic making them immediately ready to use. They can retrieve data from various ERP modules like Financials (FI), Controlling (CO), and Logistics (LO) and move it to SAP BW for reporting and analytics. Once the extractor is activated, its metadata gets stored in the metadata repository of SAP BW (not the actual data). How to Migrate Data from SAP Oracle to HANA

Customer Generated Extractors (Application Specific)

Customizations can be made to modules like LIS, CO-PA, FI-SL by organizations according to their operational requirements. SAP cannot provide a standard data source for applications like this, so customers generate their own SAP BW extractors, hence the name Customer Generated extractors. SAP ECC and Data Extraction from an LO Data Source

Generic Extractors (Cross Application): based on DB View, Infoset, Function Module

SAP’s standard extractors sometimes cannot fulfil the complete requirements of organizations. Business Content has the functionality to enable Generic Data Extractors that can support creation of DataSources built upon Database Views or InfoSet queries. An Infoset is very like a View but allows outer joins between tables. Generic Extractors are created using Tables and Views (for simple extractions) and using Infosets or using Function Modules (for complex extractions). Creating OData Services using ADSO and SAP BW Query

CDS Views

In SAP, CDS (Core Data Services) Views offer a way to define and consume data models within the database layer. These are just like database views but have enhanced features and functionality to model data. CDS Views provide semantically rich data models directly in the SAP HANA database. They are virtual data models based on specific data requirements and use SAP HANA’s powerful in-memory database to deliver fast or real-time performance by performing operations like resource-consuming calculations, aggregations and string operations at database level. They allow users to retrieve data fast, reducing application execution time. CDS views are highly extensible and can be extended to fulfil custom analytical requirements without impacting the underlying data sources. CDS Views in SAP HANA and how to create one

CDS Views are native to the SAP HANA database but of late are also used in ABAP environments as ABAP CDS Views that are database-agnostic. CDS views run on the ABAP SAP NetWeaver stack. However, running complex SQL statements on database versions lower than SAP NetWeaver ABAP 7.5 may cause performance issues since they are not optimized for these queries unlike the SAP HANA database. SAP on AWS – The Fundamentals

CDS Views can be consumed in different ways – for analytics, for creating an OData Service for data extraction, or even for building applications. CDS views are created with Open SQL, which can be enriched with annotations of different types – semantic (S4HANA Embedded Analytics), analytical (analytical tools like Bex, Lumira), UI annotations (FIORI applications). How to Migrate Data from SAP Oracle to HANA

SQL-Based Queries for Data Extraction

You can also use SQL (Structured Query Language) to directly access the SAP database layer. This method is appropriate for ad hoc queries and if data retrieval is not possible with standard extractors. It may prove an expensive proposition if SAP licensing for the underlying database is not in place. SAP to AWS (Specifically S3) – Know an Easy Method
Since the ABAP stack is the basic license, you may need to pay for other interfaces. It helps to be cognizant of authorization and the database structure first. Broadly, leveraging SQL queries for the SAP data extraction process involves establishing a database connection, instituting SELECT queries to extract relevant datasets, and ensuring the data extracted is in line with security and compliance policies. BryteFlow for SAP Database Replication

SE16 for Extraction of SAP Data Tables through SAP UI

Using transaction code SE16 will enable you to view data tables in the SAP system. You can easily and quickly access data from different tables with the SAP UI interface. This is especially beneficial when exporting data on the fly and does not need great technical skills. The process involves opening SE16 in the SAP GUI and typing in the name of the required table. You then need to define the criteria for selecting data and execute to view the contents of the table. The last step is to simply export the data as required. How to Migrate Data from SAP Oracle to HANA

Real-Time SAP Replication with ODP and OData Services

We spoke about the ODP (Operational Data Provisioning) earlier at length, now we will discuss how ODP in conjunction with OData Services and SAP tools can provide data in real-time. The ODP framework represents a unified technology infrastructure for extraction and replication of SAP data to multiple target applications and supports transference of deltas as well. It enables ETL services to targets like SAP BW, SAP BW/4HANA, SAP Data Services and SAP HANA SDI (Smart Data Integration). SAP Extraction using ODP and SAP OData Services (2 Easy Methods)

The key elements of ODP are:

ODP Providers

These are SAP applications that support ODP functionality for data extraction for e.g. SAP SLT, SAP Business Suite, SAP S/4HANA, SAP ECC, SAP BW/4HANA, and SAP BW. They are the source systems or components that enable data extraction through the ODP framework. SAP SLT can also be used to replicate data to SAP HANA and can be exposed as an ODP provider. How ODP helps in SAP CDC

DataSources

They represent the interface by which the SAP source system and the ODP framework interact. These objects are linked to source systems and can be exposed as OData Services to help with extraction of data. The interfaces include SAP DataSources (Extractors), ABAP CDS Views, SAP BW/4HANA systems, and SAP HANA Information Views, all of which can be exposed as OData Services to assist in extraction of data. DataSources are provided by SAP or can be created by customers. They communicate using an RFC connection (no longer allowed by SAP for customers) or publish data via HTTP protocols. No Need to Panic with SAP’s Ban of RFC

About OData: It should be noted that OData is a standard protocol for creating and consuming data APIs on the Internet. It allows SAP systems to consume and publish data via HTTP. How to Migrate Data from SAP Oracle to HANA

ODP Contexts and Relevant DataSources

SAP Business Suite / SAP S/4 HANA – SAP Extractors (DataSources), Tables

SAP S/4 HANA – CDS Views (ABAP Core Data Services)

SAP BW, SAP BW/4 HANA – BW InfoProviders, DataStore Objects (DSOs), CompositeProviders, etc.

SAP HANA – SAP HANA Information Views

SLT Source – SLT Replication Server

ODQ (Operational Delta Queue)

ODQ is an important part of ODP. The ODQ is a physical queue that serves to move data from the ODP source provider to the ODP consumer to support data extraction and replication. An update process automatically writes the deltas to the delta queue (transaction code ODQMON) via an extractor interface. SAP Extraction using ODP and SAP OData Services (2 Easy Methods)

With ODQ, subscribers can request multiple data types from the source since it provides multiple data request capabilities. It can be used to recover a canceled delta process, recover data that has low relevance, and to recover data with medium relevance. Users can ask for a full copy or a snapshot of data. This enables users to easily reload data at any given location or time. This functionality is part of ODQ and all the BW DataSources support it. SAP to Kafka with BryteFlow

The advantage of ODQ is that no activation or extra subscription is required. For deltas or to get changes in data, you need to start delta initialization, and this can be activated with a Push or Pull mechanism. Deltas in an ODQ can be accessed by multiple target systems which is convenient and makes data distribution efficient and consistent. Operational data is stored in ODQs in a compressed form which makes data processing faster. SAP ECC and Data Extraction from an LO Data Source

ODP Consumers

ODP Consumers are applications that receive data from the ODP framework and are the target systems where the data will be consumed for analytics or processed further. These are systems like SAP BW/4HANA or SAP BW that are ODP compatible. They are subscribers to the data delivered by ODP sources like DataSources, ABAP CDS Views, or SAP BW/4HANA.  5 Ways to Extract SAP S/4 HANA Data

ODP Consumers / Subscribers include:

SAP BW/4HANA and SAP BW

These are the target systems for ODP that provide data analytics or data warehousing platforms.

SAP Data Services

SDS can consume data from ODP providers and serves as a platform for data integration. SAP BODS (What makes it a great SAP ETL Tool)

SAP HANA Smart Data Integration (SDI)

Data can be consumed by SDI from ODP providers, to deliver real-time data integration and virtualization.

The 3 Step Process for Real-Time SAP Replication

In a nutshell, this is the 3-point process for replication of SAP data in real-time.

  • Define the DataSource for your relevant ODP Context
  • Select ODP as the source system (for SAP BW or SAP Data Services)
  • Replicate data in real-time using SAP SLT

BryteFlow as an SAP Data Extraction Tool

BryteFlow SAP Data Lake Builder is a third-party SAP data extraction tool that makes the SAP ETL process extremely simple, efficient and cost-effective. It automates SAP data extraction providing data in real-time or as required for analytics, MI, reporting and other integration scenarios.

Highlights of BryteFlow’s SAP Data Extraction

  • The tool extracts data from SAP applications keeping business logic intact and creates tables automatically on target. Extraction of data from an SAP database is also possible if you have licensed access to the underlying database.
  • It extracts data from SAP applications like SAP ECC, S/4HANA, SAP BW, SAP HANA using the Operational Data Provisioning (ODP) framework and OData services. SAP Extraction using ODP and SAP OData Services (2 Easy Methods)
  • BryteFlow is flexible and can connect to SAP in various ways including: Database Logs, ECC, HANA, S/4HANA , SAP SLT, and SAP Data Services.
  • There is no coding for any process, including SAP data extraction, CDC, merging, mapping, masking, schema, or DDL.
  • Preserves a record of every transaction with automated SCD Type 2 history.
  • Delivers consumption-ready data for Analytics and Machine Learning models with automated data type conversions. SAP HANA to Snowflake (2 Easy Ways)
  • Moves large volumes of enterprise scale SAP data with multi-threaded parallel loading and partitioning for both – initial full refresh and incremental data.
  • Low latency and high throughput, loads 1000000 rows in 30 seconds. How BryteFlow Works
  • Has automated upserts to merge deltas automatically with existing data to keep it continually updated on Snowflake, Redshift, Amazon S3, Google BigQuery, Databricks, Azure Synapse, ADLS Gen2, Azure SQL DB ( or SQL Server).
  • Provides built-in resiliency with automatic network catch-up

BryteFlow Blogs on How to Extract SAP Data

Here are some of our blogs on SAP data extraction. We hope you find them useful!

Conclusion

We have learnt how SAP data extraction works in different SAP systems and the tools and processes that drive it. We have also seen how BryteFlow as an automated SAP data extraction tool can extract data from SAP applications with logic intact, in a plug ‘n’ play mode without requiring a line of code. Contact us if you would like a demo of the BryteFlow SAP Data Lake Builder.