{"id":85097,"date":"2022-10-07T05:44:06","date_gmt":"2022-10-07T11:14:06","guid":{"rendered":"https:\/\/www.whizlabs.com\/blog\/?p=85097"},"modified":"2024-04-30T15:21:22","modified_gmt":"2024-04-30T09:51:22","slug":"snowpipe-in-snowflake","status":"publish","type":"post","link":"https:\/\/www.whizlabs.com\/blog\/snowpipe-in-snowflake\/","title":{"rendered":"What is Snowpipe &#038; how does it works?"},"content":{"rendered":"<p><b>Snowpipe<\/b><span style=\"font-weight: 400;\"> is the Continuous Data Ingestion service offered by <a href=\"https:\/\/www.whizlabs.com\/snowflake-snowpro-advanced-architect-certification\/\" target=\"_blank\" rel=\"noopener\">Snowflake<\/a>.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Snowpipe initiates Data Loading from files the moment they are available in a stage. This allows you to load data from files in micro-batches rather than manually executing COPY statements on a schedule to enable large load batches.<\/span><\/p>\n<p><b>Snowflake<\/b><span style=\"font-weight: 400;\"> supports loading data from any of the three cloud storage service providers, i.e. Amazon S3, Microsoft Azure, and Google Cloud Storage irrespective of the cloud platform.<\/span><\/p>\n<div class=\"ast-oembed-container \" style=\"height: 100%;\"><iframe title=\"What is Snowpipe: Learn about the Working and Advantages of Snowpipe | Snowflake SnowPro Core\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube.com\/embed\/JHdyIufxul4?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/div>\n<h3>How does Snowpipe works?<\/h3>\n<p><span style=\"font-weight: 400;\">Snowpipe loads data once it is available in Stage as per the COPY statement defined in the referenced pipe.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">A <\/span><b>Pipe<\/b><span style=\"font-weight: 400;\"> is a named, first class Snowflake object that is used by Snowpipe and carries a COPY statement. The COPY statement recognizes the source location of the target table and data files. Snowpipe supports all data types including semi-structured types like Avro and JSON.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Snowflake follows one of these methods for detecting staged files:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Calling Snowpipe REST endpoints<\/b><span style=\"font-weight: 400;\">: In this, the client application makes a call to a public REST endpoint using a pipe object name and a data filename list. If new data files that match the list are found in the stage referenced by the piped object, they get added in the loading queue. Thus, data is loaded from the queue using Snowflake-provided compute resources into a Snowflake table on the basis of parameters defined in the queue.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Automating Snowpipe with Cloud Messaging<\/b><span style=\"font-weight: 400;\">: Here, automated data loads enable event notifications for Cloud Storage that informs Snowflake about the arrival of new data files for loading. Snowpipe then copies these files into a queue, from where they get loaded into the target table in a continuous and serverless way on the basis of parameters defined in a specified pipe object.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">Snowflake supports the use of standard SQL for querying data files located in the internal stage, i.e. Snowflake, and named external stages which could be Google Cloud Storage, Microsoft Azure, and Amazon S3.<\/span><\/p>\n<blockquote><p>All you need to know about <a href=\"https:\/\/www.whizlabs.com\/blog\/snowflake-certifications\/\" target=\"_blank\" rel=\"noopener\">Snowflake Certifications<\/a><\/p><\/blockquote>\n<h4><b>Features that Snowpipe offer<\/b><\/h4>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Data Duplication<\/b><span style=\"font-weight: 400;\">: It prevents the reloading of same files and data duplication in tables, and does not load files with the same name even if they are modified later.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Serverless Compute Model<\/b><span style=\"font-weight: 400;\">: It provides a pipeline and enables loading of fresh data in micro-batches the moment it is available.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Single Queue Data Loading<\/b><span style=\"font-weight: 400;\">: For every pipe object, Snowflake creates a single queue for the sequencing of data files that are waiting to load.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">For supporting the creation and management of pipes, Snowflake offers special DDL commands which are,<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">CREATE PIPE<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">DESCRIBE PIPE<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">ALTER PIPE<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">DROP PIPE<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">SHOW PIPES<\/span><\/li>\n<\/ul>\n<h4><b>Differences between Snowpipe and Bulk Loading<\/b><\/h4>\n<table>\n<tbody>\n<tr>\n<td><\/td>\n<td><b>Snowpipe<\/b><\/td>\n<td><b>Bulk Loading<\/b><\/td>\n<\/tr>\n<tr>\n<td><b>Authentication<\/b><\/td>\n<td><span style=\"font-weight: 400;\">Requires key pair with JWT<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Client supported security options<\/span><\/td>\n<\/tr>\n<tr>\n<td><b>Transactions<\/b><\/td>\n<td><span style=\"font-weight: 400;\">Loads are split or combined into single or multiple transactions<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Single Transaction Loading<\/span><\/td>\n<\/tr>\n<tr>\n<td><b>Cost<\/b><\/td>\n<td><span style=\"font-weight: 400;\">Billed as per the compute resources used in Snowpipe Warehouse during file loading<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Billed for the time a Virtual Warehouse is active<\/span><\/td>\n<\/tr>\n<tr>\n<td><b>Compute Resources<\/b><\/td>\n<td><span style=\"font-weight: 400;\">Uses Snowflake-supplied Compute Resources<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Needs User-specified Warehouse<\/span><\/td>\n<\/tr>\n<tr>\n<td><b>Load History<\/b><\/td>\n<td><span style=\"font-weight: 400;\">Stored in the metadata of the pipe for 14 days<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Stored in the metadata of the target table for 64 days<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>&nbsp;<\/p>\n<p><span style=\"font-weight: 400;\">Data in Snowflake is maintained in databases. Each database consists of one or more schemas. Schemas are logical groupings of Database objects including views and tables. Databases carry Schemas and Schemas further comprise objects. Snowflake does not place any restrictions on the number of databases, schemas, or objects that could be created.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Tables store data in the form of columns and rows. Databases carry permanent tables that are persistent and stay in memory till the time you drop them.<\/span><\/p>\n<p><a href=\"https:\/\/www.whizlabs.com\/snowflake\/\"><b>Snowflake<\/b><\/a><span style=\"font-weight: 400;\"> allows the usage of permanent tables as the default table type when creating tables. However, tables can also be defined as either temporary or transient in Snowflake. These tables prove extremely useful when you need to store data that is not required to be maintained for a long period of time, that is, <\/span><b>Transitory Data<\/b><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Similar to relational databases, Snowflake supports the creation of <\/span><b>Temporary tables<\/b><span style=\"font-weight: 400;\"> that are able to hold the data that is not permanent. At multiple times, we require the data only for the current session and so temporary tables are needed. A temporary table is only visible in the current session, that is they exist only in the session that they have been created and are purged once the session ends. The data thus created cannot be recovered later. Like other table types, temporary tables also belong to a specific database and schema.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Even though you have a permanent table existing, you are allowed to create a temporary table with the same name since it is only session specific.<\/span><\/p>\n<p><b>Restrictions that come along<\/b><\/p>\n<p><span style=\"font-weight: 400;\">Whenever you wish to view the existing tables using the \u2018show tables\u2019 statement, you will only get to see the existing permanent tables and no temporary tables.<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Cloning is not allowed<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">No fail safe period, i.e. the moment session ends, the table and data vanishes<\/span><\/li>\n<\/ul>\n<h3>Transient Tables in Snowflake<\/h3>\n<p><span style=\"font-weight: 400;\">Snowflake also provides <\/span><b>Transient tables<\/b><span style=\"font-weight: 400;\"> which remain until dropped explicitly and stay available for the users as per the privileges that they hold. These tables are created for the transactory data that is required even after the current session.<\/span><\/p>\n<h4>Temporary Tables vs Transient Tables<\/h4>\n<table>\n<tbody>\n<tr>\n<td><b>Temporary Table<\/b><\/td>\n<td><b>Transient Table<\/b><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">Created using the keyword \u2018temporary\u2019<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Created using the keyword \u2018transient\u2019<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">Used for storing non-permanent, transitory data such as ETL data and session-specific data<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Used for the storage of transitory data that is required beyond a session<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">Could carry the same name as the permanent table<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Gives error, cannot be of the same name as the permanent table<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">Session-bound<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Not bound to a session<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">Not visible using \u2018show tables\u2019<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Visible just like permanent tables when using \u2018show tables\u2019<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">Cloning is not allowed<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Cloning is not allowed<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">No fail safe period, Data cannot be recovered once the session has ended<\/span><\/td>\n<td><span style=\"font-weight: 400;\">No fail safe period, Data cannot be recovered after the time travel retention period has passed\u00a0<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">Gets deleted once the session ends<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Has to be dropped explicitly when the session is finished<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><i><span style=\"font-weight: 400;\">The fail-safe period for a permanent table is 7 days.<\/span><\/i><\/p>\n<div class=\"ast-oembed-container \" style=\"height: 100%;\"><iframe title=\"Snowflake Tables Types: Permanent, Temporary, and Transient | Snowflake Tutorial | Whizlabs\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube.com\/embed\/3J_zKrYNKBY?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/div>\n<h3>Data Security in Snowflake<\/h3>\n<p><span style=\"font-weight: 400;\">When we think of migrating our data to the Cloud, the first thing that comes to our mind is Data Security. This is a major concern when you are new to Cloud and do not carry an experienced technical team that knows the whereabouts of Data Protection in Cloud.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">When you are using Cloud Storage, Cloud Computing and multiple other Cloud Technologies which run on public networks, it becomes extremely critical to secure and protect data.<\/span><\/p>\n<div class=\"ast-oembed-container \" style=\"height: 100%;\"><iframe title=\"What is Snowflake Data Security: Learn Data Encryption, Row level Security, Column level Security\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube.com\/embed\/7h_gH8ZMO_w?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/div>\n<p><span style=\"font-weight: 400;\">Snowflake offers multiple approaches to deal with Data Security. They are,<\/span><\/p>\n<p><img decoding=\"async\" class=\"alignnone wp-image-85105 size-full\" title=\"Datasecurity in Snowflake\" src=\"https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/datasecurity-in-snowflake.webp\" alt=\"datasecurity in snowflake\" width=\"1207\" height=\"610\" srcset=\"https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/datasecurity-in-snowflake.webp 1207w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/datasecurity-in-snowflake-300x152.webp 300w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/datasecurity-in-snowflake-1024x518.webp 1024w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/datasecurity-in-snowflake-768x388.webp 768w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/datasecurity-in-snowflake-831x420.webp 831w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/datasecurity-in-snowflake-640x323.webp 640w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/datasecurity-in-snowflake-681x344.webp 681w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/datasecurity-in-snowflake-150x76.webp 150w\" sizes=\"(max-width: 1207px) 100vw, 1207px\" \/><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Data Encryption<\/b><span style=\"font-weight: 400;\">: Snowflake provides end-to-end encryption which ensures that only those users are allowed to see data that are allowed through sufficient permissions. When a customer stages its documents into Snowflake\u2019s internal stage, Snowflake encrypts the data dynamically. It follows AES 256 bit encryption with a hierarchical key scheme. This model has keys on four levels,<\/span>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">Root Key<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">Account Master Keys<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">Table Master Keys, and<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">File keys<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">This system includes \u2018wrapping\u2019 wherein a parent key(higher layer) encrypts all child keys(keys in the layer below).<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The Snowflake Provider, command line users, drivers, and the Web UI interact through http connections secured at rest with TLS 1.2.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">When a master key is 30 years old, it gets rotated automatically. When you have Periodic Rekeying enabled, Snowflake generates a new encryption key and re-encrypts all the data after a year using this key.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">E2EE prevents data from being read at-rest or in-transit by third parties.<\/span><b><\/b><\/p>\n<ul>\n<li aria-level=\"1\"><b>CLS-Column Level Security or Data Masking<\/b><span style=\"font-weight: 400;\">: Snowflake\u2019s Column Level Security allows you to mask the columns holding sensitive data including Bank Account no.s, SSN no.s, among others. Snowflake offers data masking through,<\/span><\/li>\n<\/ul>\n<p><img decoding=\"async\" class=\"alignnone wp-image-85107 size-full\" title=\"Column Level Security in Snowflake\" src=\"https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/column-level-security.webp\" alt=\"column level security in snowflake\" width=\"1207\" height=\"591\" srcset=\"https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/column-level-security.webp 1207w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/column-level-security-300x147.webp 300w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/column-level-security-1024x501.webp 1024w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/column-level-security-768x376.webp 768w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/column-level-security-858x420.webp 858w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/column-level-security-640x313.webp 640w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/column-level-security-681x333.webp 681w, https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/column-level-security-150x73.webp 150w\" sizes=\"(max-width: 1207px) 100vw, 1207px\" \/><\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">External Tokenization: This enables the tokenization of sensitive data before it is loaded into Snowflake. It dynamically detokenizes the data while executing queries by making use of masking policies with External Functions.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">Dynamic Data Masking: Here, masking policies are used to mask the data in\u00a0 specific columns in a view or table. These policies are simple SQL statements\/Conditions that mask the data while executing the query. This allows the real data in the table to be the same so no static masking applies. Masking policies can be applied on the basis of User Role\/Group enabling masked data for unauthorized users and real data for users with permissions.\u00a0<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<ul>\n<li><span style=\"font-weight: 400;\"><span style=\"font-weight: 400;\"><b>RLS-Row Level Security: <\/b>Row level security is implemented through Row access policies by deciding which rows should be visible in the query result. A row access policy is a schema-level object that decides which row should be returned as a result of SELECT, UPDATE, MERGE, and DELETE statements.<b><\/b><\/span><\/span><span style=\"font-weight: 400;\"><span style=\"font-weight: 400;\"><b><\/b><br \/>\n<\/span><\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">Snowflake makes use of Views\/Secure Views for providing Row Level Security. Data Security can also be controlled by assigning Access Privileged Roles on database objects to users. Information is shared and hidden from the users through views. Data can be concealed or filtered using SQL statements in the definition of View.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Secure Views allows you to hide the SQL statement used in generating the view. A secure view can be constructed using the \u2018SECURE\u2019 keyword before the name of the view.<\/span><\/p>\n<h3>Summary<\/h3>\n<p>On a final note, you can clearly understand what is Snowpipe and it&#8217;s feature in snowflake that allows you to continuously load data into your snowflake account. This is especially useful if you have a lot of data that is constantly changing, as it allows you to keep your data up-to-date without having to manually load it each time.<\/p>\n<p>There are a few different ways to implement Snowpipe, so be sure to select the best method for your architecture. And, be sure to test the workflow on a small dataset first. This will allow you to accurately assess the cost-benefit of using Snowpipe.<\/p>\n<p>We hope this blog helps in your snowflake data journey. Feel free to comment us if you have queries.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Snowpipe is the Continuous Data Ingestion service offered by Snowflake. Snowpipe initiates Data Loading from files the moment they are available in a stage. This allows you to load data from files in micro-batches rather than manually executing COPY statements on a schedule to enable large load batches. Snowflake supports loading data from any of the three cloud storage service providers, i.e. Amazon S3, Microsoft Azure, and Google Cloud Storage irrespective of the cloud platform. How does Snowpipe works? Snowpipe loads data once it is available in Stage as per the COPY statement defined in the referenced pipe. A Pipe [&hellip;]<\/p>\n","protected":false},"author":371,"featured_media":85175,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_uag_custom_page_level_css":"","site-sidebar-layout":"default","site-content-layout":"default","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"default","adv-header-id-meta":"","stick-header-meta":"default","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"set","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[4881],"tags":[4929],"class_list":["post-85097","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-snowflake","tag-snowpipe-in-snowflake"],"uagb_featured_image_src":{"full":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/What-is-snowpipe-_-how-does-it-works.webp",1280,720,false],"thumbnail":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/What-is-snowpipe-_-how-does-it-works-150x150.webp",150,150,true],"medium":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/What-is-snowpipe-_-how-does-it-works-300x169.webp",300,169,true],"medium_large":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/What-is-snowpipe-_-how-does-it-works-768x432.webp",768,432,true],"large":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/What-is-snowpipe-_-how-does-it-works-1024x576.webp",1024,576,true],"1536x1536":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/What-is-snowpipe-_-how-does-it-works.webp",1280,720,false],"2048x2048":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/What-is-snowpipe-_-how-does-it-works.webp",1280,720,false],"profile_24":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/What-is-snowpipe-_-how-does-it-works.webp",24,14,false],"profile_48":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/What-is-snowpipe-_-how-does-it-works.webp",48,27,false],"profile_96":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/What-is-snowpipe-_-how-does-it-works.webp",96,54,false],"profile_150":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/What-is-snowpipe-_-how-does-it-works.webp",150,84,false],"profile_300":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/What-is-snowpipe-_-how-does-it-works.webp",300,169,false],"tptn_thumbnail":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/What-is-snowpipe-_-how-does-it-works-250x250.webp",250,250,true],"web-stories-poster-portrait":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/What-is-snowpipe-_-how-does-it-works-640x720.webp",640,720,true],"web-stories-publisher-logo":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/What-is-snowpipe-_-how-does-it-works-96x96.webp",96,96,true],"web-stories-thumbnail":["https:\/\/www.whizlabs.com\/blog\/wp-content\/uploads\/2022\/10\/What-is-snowpipe-_-how-does-it-works-150x84.webp",150,84,true]},"uagb_author_info":{"display_name":"Sweta Singhal","author_link":"https:\/\/www.whizlabs.com\/blog\/author\/sweta\/"},"uagb_comment_info":2,"uagb_excerpt":"Snowpipe is the Continuous Data Ingestion service offered by Snowflake. Snowpipe initiates Data Loading from files the moment they are available in a stage. This allows you to load data from files in micro-batches rather than manually executing COPY statements on a schedule to enable large load batches. Snowflake supports loading data from any of&hellip;","_links":{"self":[{"href":"https:\/\/www.whizlabs.com\/blog\/wp-json\/wp\/v2\/posts\/85097","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.whizlabs.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.whizlabs.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.whizlabs.com\/blog\/wp-json\/wp\/v2\/users\/371"}],"replies":[{"embeddable":true,"href":"https:\/\/www.whizlabs.com\/blog\/wp-json\/wp\/v2\/comments?post=85097"}],"version-history":[{"count":7,"href":"https:\/\/www.whizlabs.com\/blog\/wp-json\/wp\/v2\/posts\/85097\/revisions"}],"predecessor-version":[{"id":90062,"href":"https:\/\/www.whizlabs.com\/blog\/wp-json\/wp\/v2\/posts\/85097\/revisions\/90062"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.whizlabs.com\/blog\/wp-json\/wp\/v2\/media\/85175"}],"wp:attachment":[{"href":"https:\/\/www.whizlabs.com\/blog\/wp-json\/wp\/v2\/media?parent=85097"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.whizlabs.com\/blog\/wp-json\/wp\/v2\/categories?post=85097"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.whizlabs.com\/blog\/wp-json\/wp\/v2\/tags?post=85097"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}