Oracle Data Replication

GoldenGate CDC and a GG Alternative (Cuts Licensing Costs)

What Oracle GoldenGate CDC is all about and how this cost-effective GoldenGate alternative can save you big money

This blog talks about Oracle GoldenGate, GoldenGate CDC replication and the processes involved. It also examines GoldenGate products, advantages and disadvantages, and presents BryteFlow as a worthy GoldenGate alternative, backed up by a comparison matrix.

Avoid expensive Oracle GoldenGate licensing with BryteFlow

Please note, with BryteFlow you do not need additional licensing for the Oracle GoldenGate XStream module, which GoldenGate requires to enable real-time CDC from an Oracle 19c database. This functionality was supported in the previous Oracle versions 12c and lower, with no extra licensing through Oracle Streams and the LogMiner Continuous Mining feature, both of which are now deprecated. Instead BryteFlow gets real-time data from Oracle 19c redo logs without using the licensed Oracle GoldenGate XStream product and is more performant and cost effective. Bryteflow for Oracle Replication

Quick Links

What is Oracle GoldenGate?

A versatile CDC and replication software, Oracle GoldenGate or simply GoldenGate as it is popularly known, enables replication of data across multiple heterogenous systems. It replicates, filters, and transforms data from one database to another. It allows users to replicate data from Oracle databases to other supported heterogenous databases and also between heterogenous databases. It enables replication to Flat Files, Java Messaging Queues and supports Big Data destinations like Amazon S3, Amazon Redshift, Azure Synapse, Azure Data Lake, Snowflake, Kafka, SingleStore, Teradata,  Google BigQuery etc. (in conjunction with Oracle GoldenGate for Big Data). Golden Gate is known for its real-time data integration, log-based CDC (Change Data Capture), transformations, high availability and validation between operations and analytical systems. It complements the Oracle Data Integrator Enterprise Edition to enable data integration. Oracle to Azure Cloud Migration (Know 2 Easy Methods)

Oracle GoldenGate – A Short History

GoldenGate was originally created by a San Francisco based company named GoldenGate Software Inc. It was designed for Tandem computers and was well received with the financial services industry on account of the fast and robust replication capabilities. Banks would use GoldenGate software to send transaction data from ATMs to the main servers. GoldenGate was acquired by the Oracle Corporation in 2009 and is today part of the Oracle Cloud Infrastructure (OCI) solution. It is used by over 80% of Fortune 500 companies. About Oracle to Postgres Migration

Oracle GoldenGate for Big Data

GoldenGate was supposedly the first data replication Cloud service to deliver a serverless experience with built-in scalability. It supports real-time data replication with log-based change data capture and integration to data warehouses, data marts from operational and analytical production systems, zero-downtime cloud migrations, reporting from transactional systems (On-premise and Cloud), and continuous availability. Oracle GoldenGate for Big Data was launched in February 2015. It is an Oracle GoldenGate product that streams data in real-time from heterogeneous data sources to Big Data platforms. GoldenGate for Big Data works as a standalone or integrated part of Oracle GoldenGate and can be deployed as a part of OCI GoldenGate, on-premise or through third-party cloud implementations. Oracle to SQL Server: Reasons, Challenges & Tools

GoldenGate for Big Data Replication supports many sources and targets

GoldenGate for Big Data can be used to build real-time ETL pipelines easily, acquire and analyze time-series and geo-spatial data and enable machine learning use cases. GoldenGate for Big Data supports more than 300 combinations of sources and targets, provides CDC (Change Data Capture) from Kafka, MongoDB, Cassandra, and GoldenGate and delivers data to targets like Kafka, Hadoop, HBase, MongoDB, Cassandra, Oracle NoSQL, and cloud environments, like AWS (S3, Redshift, Kinesis), Azure Cloud (Azure Data Lake, SQL Database, HDInsight, Azure Synapse, Event Hub), and Google Cloud (BigQuery), and Oracle Cloud Infrastructure (Object Storage). It also delivers data to SAP HANA, Elasticsearch, IBM PureData System for Analytics, Snowflake and Pivotal Greenplum. GoldenGate transforms relational data into hierarchical formats like XML, JSON, Avro and columnar formats such as ORC and Parquet. Within GoldenGate for Big Data there are JDBC and custom Java handlers to enable integration with various databases and applications. Oracle CDC to Kafka Methods

GoldenGate for Big Data supports real-time streaming analytics

Oracle Stream Analytics included with Oracle GoldenGate for Big Data allows users to execute queries against real-time event streams to identify events of interest. It provides ML/AI capabilities for geo-spatial and time-series analytics, allowing for real-time monitoring, transformation of streaming data and setting up alerts based on analysis of real-time streams via the creation of customized operational dashboards. Real time Oracle Replication

How GoldenGate CDC Replication works

GoldenGate CDC is essentially real-time, log-based CDC and based on the reading of transaction logs of the database. Basically, committed transactions are captured and can be filtered as they happen by reading the transaction logs. GoldenGate now offers two options for Oracle data capture: Classic and Integrated Capture. Connect Oracle to Databricks and Load Data the Easy Way

GoldenGate Classic Capture CDC

With Classic Capture CDC mode, the Oracle GoldenGate Extract method captures data changes from the Oracle Redo or Archive log files on the source database or from archived logs on a standby system. This was the original GoldenGate CDC method. Classic Capture can be used on any source Oracle RDBMS supported by GoldenGate except for multi-tenant container databases. Oracle vs Teradata (Migrate in 5 Easy Steps)

GoldenGate Integrated Capture CDC

With Integrated Capture CDC mode, the Oracle GoldenGate Extract method interacts directly with the database log mining server to get changes in data in the form of LCRs (Logical Change Records). An advantage of Integrated Capture is that it supports more data and storage formats than Classic Capture. Also, no additional setup is required to work with Oracle RAC, ASM, and TDE since Integrated Capture is completely integrated with the Oracle database. Even if a log file is absent due to a corrupted disk or hardware failure, Integrated Capture can handle it – provided extra copies of the archived and online logs are available. Unlike Classic Capture CDC, Integrated Capture CDC supports capture from a multitenant container database. It is also faster at filtering tables and handles point-in-time recovery and RAC integration better. It also retains the archive logs required by Extract automatically through the Oracle Recovery Manager (RMAN). Oracle CDC: 13 Things toKnow
Overall GoldenGate CDC has minimal impact on source systems, very low latency and upholds transactional integrity across databases. GoldenGate can handle CDC with high throughput in challenging business intelligence and operational scenarios. Oracle GoldenGate leverages Oracle LogMiner for both integrated capture and delivery and also uses XStream for CDC and streaming data. Oracle to Azure Cloud Migration (Know 2 Easy Methods)

Oracle GoldenGate Replication – Essential Processes

Extract or Capture Process

Through the database transaction logs, the Extract process captures committed transactions on the source database and writes them to GoldenGate’s trail files. Oracle to Postgres Migration (The Whys and Hows)

Pump Extract

The job of the Pump Extract is to protect replication from network or site failure. It is optional but its configuration is recommended. Pump Extract is a source side process and enables sending of trail data to multiple targets. Postgres to Oracle with BryteFlow

Replicat or Apply Process

Replicat is configured on the target side and is also known as Apply process. The trail files sent by the Extract process are received by Replicat and applied on the target database. Replicat ensures data integrity and zero data loss with a checkpointing mechanism. Real time Oracle Replication

Manager Process

The Manager process is present on both – the source and the target sides. Manager process controls the replication environment on both sides – starting / restarting Extract and Replicat, managing storage for trail files, reporting etc. Oracle to Redshift Migration Made Easy (2 Methods)data

Collector Process

The Collector receives the committed data sent by the source or Pump extracts and aggregates it in the target side trail. The resulting trail file is used by the Replicat process. Oracle to Snowflake: Everything You Need to Know

Trail Files

The trail files in GoldenGate can be likened to Redo Logs in the Oracle DB. Trail files aggregate the data captured by the Extract process from source and create similar trail files on the target, on which the Replicat process will apply the captured data changes. Trail files follow the Canonical format and are sequenced according to the Commit order of the captured transactions to ensure transactional integrity. GoldenGate also has a checkpointing process to ensure tracking of data being written to a trail file. Debezium CDC Explained and a Great Alternative CDC Tool

Parameter Files

GoldenGate configurations are varied and depend on the parameters defined for Extracts and Replicats. Parameters are configured in process parameter files which are editable plain text files and are stored in their own dedicated folder in GoldenGate.

GoldenGate CDC and Oracle LogMiner

Oracle LogMiner is a component included with the purchase of the Oracle database. It enables querying of online and archived redo log files via a SQL interface. Oracle LogMiner is preferred by data integration vendors since it does not need additional licensing. GoldenGate has access to LogMiner’s redo logs to determine changed records in the database and enable CDC. GoldenGate’s CDC functionality can detect and transmit data events including DDL (data definition language) and DML (data definition language) besides procedure events that can aid in syncing databases. GoldenGate CDC is used for replication to data warehouses, data lakes, maximum availability architecture and data streaming (e.g. Kafka).

LogMiner’s Continuous Log Mining for CDC feature deprecated

LogMiner was originally created more as a database forensics and recovery tool to track errors and as such could not analyze huge amounts of data changes, it could not go beyond 1 CPU of resource consumption to avoid strain on the database. Oracle then created the Continuous Log Mining feature that would enable users to subscribe to a LogMiner event stream to enable continuous ETL and get notifications when there were changes in the source database. This feature was used by many third-party CDC tools to replicate Oracle data for their customers. Unfortunately, this feature has been deprecated after Oracle 19C was introduced. Oracle Streams, the free native tool for Oracle CDC was deprecated as well in Oracle 12C. Cloud Migration (Challenges, Benefits and Strategies)

GoldenGate CDC and Oracle XStream API

The Oracle XStream API is licensed with Oracle GoldenGate and is made up of APIs (Application Programming Interfaces) and Oracle DB utilities that allow client applications to receive changes taking place in an Oracle database (via XStream Out APIs) and to send data changes to an Oracle database (via XStream In APIs). XStream enables high throughput CDC from the Oracle DB and is used by ISVs and third-party replication tools for this purpose. Oracle XStream API is useful for engineers who need high-speed, low-level access to database logs for transactional database replication. However, using XStream is expensive as compared to Oracle LogMiner since it involves additional licensing costs. Why Your DBA Will Love Oracle Autonomous Data Warehouse

GoldenGate Products

These are the products in the GoldenGate family and they require separate licensing.

Oracle GoldenGate for Big Data

Oracle GoldenGate for Big Data enables writing of operations data from Oracle GoldenGate trail records to multiple Big Data destinations (for e.g., HBase, Elasticsearch, Kafka, HDFS, JDBC, Cassandra, Flume, and MongoDB). Kafka CDC and Oracle to Kafka CDC Methods

Oracle GoldenGate Veridata

GoldenGate Veridata compares datasets at source and target and informs about data that is not in sync, so users can repair the data.

Oracle GoldenGate EMCC Plugin

Oracle GoldenGate Plug-in for EMCC is an Enterprise Manager Plug-in for Oracle GoldenGate and enables visual monitoring and managing of Oracle GoldenGate processes.

Oracle GoldenGate Monitor

Oracle GoldenGate Monitor is a web-based console for real-time online monitoring that provides a graphical view of GoldenGate instances, and the associated databases being used by an organization.

Oracle GoldenGate Application Adapters

Oracle GoldenGate Application Adapters are integrated with the GoldenGate core product when installed to provide Java Message Service (JMS) information or to deliver information as JMS messages or files.

Oracle GoldenGate for HP NonStop

Oracle GoldenGate for HP NonStop is a software that allows replication of data across heterogenous platforms and applications. Users can extract and replicate selected records and transactional changes, allowing for management of organizational data at a transactional level. CDC, high availability, real-time data integration, replication and data manipulation capabilities are part of the product.

Oracle GoldenGate Studio

Oracle GoldenGate Studio enables replication of data in high volumes and in real-time by automated table and column mappings, enabling custom drag and drop mappings, creating template-based best practice configurations and providing relevant help that is context-sensitive. It enables new users to start using GoldenGate replication for CDC without much effort.

GoldenGate Advantages and Disadvantages

Let us start with the advantages of GoldenGate which are many:

  • High performance CDC replication – GoldenGate replicates and delivers data in near real-time with sub-second latency. It offers the most stable and high-performing Oracle CDC replication among Oracle tools. Oracle to Azure Cloud Migration (Know 2 Easy Methods)
  • GoldenGate has deep integration with Oracle DB means backup, redo, logging, security, administration and integration with Data Vault and Data Guard are seamless.
  • GoldenGate sends committed data, so performance and consistency are maintained. Though uncommitted data is captured, it is discarded in case of a Rollback. Source to Target Mapping Guide (What, Why, How)
  • GoldenGate supports many different Oracle versions and multiple non-Oracle databases. It supports cross-platform migrations, and it is available for a wide range of operating systems, hardware, and Cloud platforms. Why Your DBA Will Love Oracle Autonomous Data Warehouse
  • GoldenGate replication and CDC is low-impact and does not affect the underlying database. Connect Oracle to Databricks and Load Data the Easy Way
  • GoldenGate has built-in data recovery mechanisms. It is highly available and has gap resolution for various types of failures like network or site failure. It is the only CDC tool certified for Oracle’s Platinum Tier of maximum availability architecture and tested for strict RPO (recovery point objective) and RTO (recovery time objective) conditions.
  • GoldenGate has its own proprietary format for maintaining committed files from the source, this makes the functioning independent of the database in use. Oracle to Snowflake: Everything You Need to Know
  • GoldenGate does not depend on Oracle Network services but utilizes the standard network between source and target database, this ensures there is no impact on the network performance by the data transfer process.
  • GoldenGate is fault-tolerant and has its own process to track how many transactions have been completed and how many are pending. The automatic gap resolution ensures zero data loss and is independent of the database. Oracle to Redshift Migration Made Easy (2 Methods)

Operational Advantages of GoldenGate

  • GoldenGate has a command line and a graphical web user interface which makes it a convenient CDC tool.
  • GoldenGate has easy installation or provisioning and is fast.
  • AdminClient offers command line access that can be run anywhere.
  • GoldenGate has RESTful APIs for DevOps.
  • GoldenGate Monitoring is built-in but also can be easily customized to work with your own monitoring.
  • Oracle GoldenGate can be run On-premise or on Cloud. Oracle to Redshift Migration Made Easy (2 Methods)
  • GoldenGate enables encryption of data on disk and over the wire protocol which is great for banking and finance customers.
  • GoldenGate is versatile and has adapters to extend its functionality, for e.g. the GoldenGate application adapter for Java can send data to a non-RDBMS target like the Java Messaging Service (JMS).
  • A standby database in sync with the source database is possible with GoldenGate. In case of a crash there would be an instant fail-over which means minimal impact on the business.
  • GoldenGate’s replication allows for use of a separate system for replication which could be used for reporting purposes, reducing the overhead on the source database.
  • Golden Gate ensures zero downtime for upgrades and migrations. Migrating ERP data to the Cloud
  • GoldenGate Configuration Options are another big advantage. Data can be delivered and received in various flexible configurations with GoldenGate. This is as per organizational demand and requirements.
  • One-to-One (Unidirectional Replication)
  • One-to-Many (Broadcast Replication)
  • Many-to-One (Consolidation Replication)
  • Bi-Directional (Active-Active Replication)
  • Multimaster (Peer-to-Peer Replication)
  • Cascading Data Marts

Disadvantages of GoldenGate

  • GoldenGate is an expensive product which requires licensing separately from the Oracle DB license. With Oracle GoldenGate, all its features can be licensed separately or as a package. For GoldenGate’s advanced replication features (Active Data Guard and XStream) to work, users would need to license the Oracle Enterprise Edition. These features are needed to process real-time data through an API and deliver it on target, but they are licensed options. A user will need to purchase licenses separately for Active Data Guard (disaster recovery) and XStream (real-time data capture) when he buys the license for the Oracle Enterprise Edition. Why Your DBA Will Love Oracle Autonomous Data Warehouse
  • Deployment of GoldenGate could need anywhere from days or weeks to setup depending on database and storage structures. Users may have to deal with some complexity in setting up.
  • Migrating data with Oracle GoldenGate replication could throw up character set replication issues.
  • Oracle GoldenGate’s Extract process during replication may consume a lot of memory. Cloud Migration Challenges
  • For XML data and HDFS there may be issues in data delivery. Data corruptions could take place. Ideally a script for detection and correction of XML data would need to be written as part of the Oracle GoldenGate replication process. Oracle to Azure Cloud Migration (Know 2 Easy Methods)

Oracle GoldenGate Alternative: BryteFlow

GoldenGate competes with multiple CDC software that can be used as alternatives for GoldenGate. Our very own BryteFlow is one of them. As mentioned earlier, Oracle recommends licensing the Oracle GoldenGate XStream module to capture real-time data during Oracle GoldenGate replication. This hurts users because for Oracle 12c and lower, this functionality was supported earlier with no additional licensing cost (via Oracle Streams and LogMiner). In contrast, BryteFlow can get real-time data from Oracle 19c redo logs without using the licensed Oracle GoldenGate XStream product, which means you can save big on GoldenGate and XStream licensing costs. BryteFlow for Oracle Replication
With the deprecation of LogMiner’s continuous log mining feature, BryteFlow has developed its own log parser tool for Oracle CDC that requires no additional licensing. BryteFlow’s CDC replication is also at least 6x faster than GoldenGate and moves 1,000,000 rows in 30 seconds approx. making the solution more performant and cost-effective than GoldenGate. Postgres to Oracle with BryteFlow

Highlights of BryteFlow as a GoldenGate Alternative

GoldenGate CDC vs BryteFlow CDC: A Comparison

Feature Oracle GoldenGate BryteFlow
Licensing

 

 Available as an add-on license option for Oracle Enterprise Edition Database. It can also be separately licensed for non-Oracle databases. GoldenGate Licensing

 

 BryteFlow Standard and Enterprise editions are available for AWS, Azure and other Cloud Platforms. View Pricing

 
Component Licensing Requires separate licensing for components like GoldenGate Veridata, GoldenGate Studio, GoldenGate Monitor etc. For real-time CDC, a license for the XStream module is also needed along with installation of Oracle Enterprise Edition Database, which adds up to sizable licensing costs. Bryteflow Enterprise Edition license includes tools for Replication (BryteFlow Ingest & XL Ingest ), Transformation (BryteFlow Blend ), Reconciliation (BryteFlow TruData ) and Monitoring (BryteFlow ControlRoom ). No additional licensing required.
Implementation and Maintenance May need GoldenGate experts to deploy GoldenGate and DBA involvement for maintenance. Implementation can be hard and expensive. Low maintenance CDC solution does not need DBA involvement. Business users can implement it with ease if prerequisites are met. Cost-effective.

 
Parallel log mining Yes Yes
Remote Log Mining Yes Yes
Deployment options On Premise and Cloud On Premise and Cloud
Delivery of data with history of every change (SCD Type2) Yes Yes
Log-based, real-time, Change Data Capture (CDC) Yes, needs XStream API installed to capture real-time data. Yes, CDC needs no third-party tool installation.
Coding requirements when data is incrementally captured May require some amount of coding. Completely point and click, no coding required.
Flexibility with Cloud Flexibility is low. For non-Oracle targets like Amazon S3, Snowflake, or other cloud platforms, a separate license for Oracle GoldenGate for Big Data is required. No other license is needed, BryteFlow is a Cloud-native tool and offers high performance Cloud replication.
Pipeline strength GoldenGate pipelines can be complex, with multiple integration points that can break often, impacting performance of downstream models, analytics & applications. Robust pipelines with automated network catchup.
Deployment time Weeks to months Within hours
Automated Partitioning and Compression for replicating data Yes Yes. Partitioning can be done by configured columns automatically. File types and compression supported include – ORC (snappy), ORC(zlib), Parquet(snappy), gzip, bzip2.
Automated Data reconciliation Yes, but need to install GoldenGate Veridata (additional licensing). Yes. With BryteFlow TruData you can automatically reconcile data with counts and checksums.
End to end monitoring Yes, but need to install Oracle GoldenGate Management Pack (additional licensing). BryteFlow ControlRoom monitors the pipeline and tracks every instance.
Large Volume data extraction from transactional sources Yes Yes. Can ingest petabytes of data with high performance. BryteFlow XL Ingest has smart partitioning mechanisms to ingest data in parallel.
Serverless Yes No. The software must be deployed on customer’s on-prem servers or on virtual servers on Cloud.
Data Formats and Conversions Supports XML, JSON, Avro and columnar formats such as ORC and Parquet. Supports Parquet, ORC and Avro.

If you are looking for a cost-effective, high performing GoldenGate alternative for CDC, contact us for a demo of BryteFlow. You can also try out BryteFlow free.