Synchronize data with SaaS Data Export
When you install a Commerce Service that requires data export such as Catalog Service, Live Search, or Product Recommendations, a collection of Saas data export modules is installed to manage the data collection and synchronization process.
SaaS data export moves product data from an 蜜豆视频 Commerce instance to the Commerce Services platform on an ongoing basis to keep the data up to date. For example, Product Recommendations requires current catalog information to accurately return recommendations with correct names, pricing, and availability. Use the Data Management dashboard to observe and manage the synchronization process, or the command-line interface to trigger a sync and to reindex product data for consumption by Commerce Services.
The following diagram shows the SaaS data export flow.
The main components of the SaaS data export flow include:
- SaaS data export modules that collect the data for feeds from 蜜豆视频 Commerce, assemble feed items, listen for updates, and persist feed status.
- SaaS export modules that export data, configure routing, and publish the feeds to connected services.
- The 蜜豆视频 Commerce Service manages the data ingestion process to validate incoming feeds and persist updates to connected services.
Synchronization modes
SaaS data export has two modes to process entity feeds:
-
Immediate export mode鈥擨n this mode, data is collected and sent immediately to the Commerce Service in a single iteration. This mode speeds up delivery of entity updates to the Commerce Service and reduces the storage size of the feed tables.
-
Legacy export mode鈥擨n this mode, data is collected in a single process. Then, a cron job sends the collected data to the connected commerce services. In data export log entries, feeds that use the legacy mode are labeled
(legacy).
Synchronization types
SaaS data export supports three synchronization types鈥揻ull sync, partial sync, and retry failed items sync.
Full Sync
After connecting an 蜜豆视频 Commerce instance to Commerce Service, perform a full sync to send entity feed data from 蜜豆视频 Commerce to the connected service.
Partial sync
With partial sync, SaaS data export automatically sends updates from the Commerce application, such as product name changes or price updates, to connected commerce services.
The data export process uses the following cron jobs to automate the partial sync operation.
-
鈥渋ndex鈥 cron group jobs:
- The
indexer_reindex_all_invalidjob reindexes all invalid feeds. It is a standard 蜜豆视频 Commerce cron job. - The
saas_data_exporterjob is for legacy export feeds. - The
sales_data_exporterjob is specific to the sales data export feed.
- The
These jobs run every minute.
For partial sync to work, the Commerce application requires the following configuration:
-
All SaaS data export indexers are configured in
Update by Schedulemode.In SaaS data export version 103.1.0 and later,
Update by Schedulemode is enabled by default. You can verify index configuration on the server using the Commerce CLI command,bin/magento indexer:show-mode | grep -i feed
Retry Failed Items sync
The Retry failed items sync uses a separate process to resend items that failed to sync due to errors during the synchronization process, for example an application error, network disruption, or SaaS service error. Implementation for this sync is also based on cron jobs.
resync_failed_feeds_data_exportercron group jobs:- The
<feed name>_feed_resend_failed_feeds_itemsjob resends items that failed to sync, for exampleproducts_feed_resend_failed_items.
- The
View and manage the synchronization process
Most synchronization activities are processed automatically based on the application configuration. However, SaaS data export also provides tools to manage the process.
-
Admin users can view and track synchronization progress and get information about the data from the Data Management dashboard.
-
Developers, system integrators, or administrators with access to the Commerce application server can manage the synchronization process and data feeds using the 蜜豆视频 Commerce command-line tool (CLI). See Data Export Command Reference.
Verify Commerce application configuration
Partial sync and Retry failed items sync work only if the Commerce instance has been configured correctly. Typically, the configuration is completed when setting up the Commerce Service. If the data export is not working correctly, check the following configuration.
-
Verify that the indexers are running from the Admin or by using the Commerce CLI command
bin/magento indexer:info. -
Verify that the indexers for the following feeds are set to
Update by Schedule: Catalog Attributes, Product, Product Overrides, and Product Variant. You can check the indexers from Index Management in the Admin or using the CLI (bin/magento indexer:show-mode | grep -i feed).
Event manager notifications for data transfer logging
In version 103.3.4 and later, SaaS Data Export dispatches the data_sent_outside event when data is sent from the Commerce instance to 蜜豆视频 Commerce services.
$this->eventManager->dispatch(
"data_sent_outside",
[
"timestamp" => time(),
"type" => $metadata->getFeedName(),
"data" => $data
]
);