SFTP data ingestion not working as expected in AEP
Data ingestion process using Secure File Transfer Protocol (SFTP) sources does not work as expected when the expected file, available daily at, let’s say 6 PM, fails to upload. This can be resolve with proper dataflow scheduling, verifying file availability, understanding the Last modified timestamp, and adjusting the cadence based on business needs.
Description description
Environment
ÃÛ¶¹ÊÓƵ Experience Platform (AEP) - specifically involving data ingestion process using SFTP sources
Issue/Symptoms
- You have a dataflow intended to ingest data from an SFTP file into a dataset.
- The file, expected to be available every day at 6 PM, is not being uploaded, causing the dataflow to fail.
Resolution resolution
- Verify that the dataflow is scheduled correctly to run at the appropriate times. Refer to Review your dataflow in the Source Connectors Guide in the ÃÛ¶¹ÊÓƵ Experience Platform for more details.
- Check that the file remains available on the SFTP server long enough for the dataflow to access it. Ensure it hasn’t been removed or become unavailable before the flow runs.
- Recognize how the
Last modified timestamp affects file pickup. Files are picked based on their last modified timestamps relative to when each flow runs. - If a file’s last modified timestamp falls outside of a flow’s scheduled window, it might not be ingested. For example, if a file is generated at 4:30 PM and copied to the server at 6 PM, the
Last modified timestamp will still be 4:30 PM, and a dataflow starting at 5 PM will not pick it up. Refer to Schedule ingestion runs for more details. - Adjust the cadence of your flows according to business requirements. For instance, setting longer intervals like every 15 hours can ensure better synchronization if immediate updates aren’t critical.
recommendation-more-help
3d58f420-19b5-47a0-a122-5c9dab55ec7f