Snowflake’s auto ingestion service Snowpipe has been around since Dec 2018 and, in my experience, has proven to be an excellent method for automatically ingesting incoming data landing in cloud storage into Snowflake.

Snowpipe with auto ingestion requires addition cloud services over and above the simple cloud storage containers your data lands in.

There are so many tools to migrate the data from your server to Azure blob storage. The tools may not work as you want. Here is the way to minimize your third-party cost and process your data in an effective way. Now, we are uploading CSV files from the folder recursively i.e. file inside the sub-folder too.

Screw the Business Key!…Say What Now?

The business key…. Well what is the business key? In the Data Vault methodology it is simply a unique key used to identify a record to either be inserted into the Data Vault (hub) or check against an existing record.

Data Vault, is it a replacement for traditional data warehouses and star schemas? This is a common question that not only often gets asked to me, but also a question that get gets asked by those interested or learning about Data Vault.

2024 © Data Engineers Ltd - All Rights Reserved