A data ingestion architecture is the set of decisions that sit between a source system and the first table your platform owns: how changes are detected, how they travel, and how they land. "Batch or streaming" is a processing question; the ingestion question is what each source can show you. A 2018 post on the Debezium project blog names the limit of querying: "Naturally, polling will not allow you to identify any records that have been deleted since the last poll." So the extraction pattern is chosen per source, and the landing stays the same for every pattern. In our practice, a data ingestion architecture is finished when any table can be rebuilt from what landed, without asking the source again.
What a Data Ingestion Architecture Decides, and What It Leaves Downstream
Data ingestion is the boundary between a system you do not own and the first table you do. Its decisions put risk on different owners. Detection spends query capacity or log retention on a system that serves production traffic, so it concerns the source owner. Transport decides whether a record can arrive twice or out of order. Landing decides whether the platform can replay a table on its own.
Vendors group tooling the same way. The Lakeflow Connect documentation says that "Lakeflow Connect organizes connectors by source type, including database, SaaS, file, and streaming connectors." The source comes first because it limits which detection methods are possible.
Our overview of where ingestion sits among the four pipeline layers gives ingestion one paragraph. This article stays inside that layer and goes one level down, to how each source gives up its changes.
Five Ingestion Patterns and What Each One Can See
Every data ingestion architecture is built from five ways of detecting change: a full snapshot, a watermark query, log-based CDC, file arrival and an event stream. A managed SaaS connector delivers one of these rather than adding a sixth; Lakeflow Connect, for instance, ships both query-based and CDC-based database connectors.
The dividing line runs between querying a table and reading its log. The Debezium post describes the gap: "For instance it could happen that a record is inserted and deleted between two polls, in which case this record would never be captured by poll-based CDC." The log can also carry the old row state for updates and deletes, "Whereas with polling, you'll only get the current row state."
Delivery tools change; what each pattern can see does not.
Snapshots and Watermarks: When a Query Is Enough
In a data ingestion architecture, query-based extraction is the usual starting point, and for many sources the right one. Microsoft's Azure Data Factory guide to incremental copy defines the mechanism: "A watermark is a column that has the last updated time stamp or an incrementing key." Even the Debezium post concedes that when hourly propagation is enough and missing intermediate versions is acceptable, polling "can be perfectly fine."
Before accepting a watermark, we check where it breaks. It needs a column that somebody maintains, and any update path that skips the timestamp hides rows silently. Hard deletes never appear, because the row is gone before the next query runs. Every cycle also loads the production database, since "the queries used for polling cause load on the source database."
When deletes matter and the log is out of reach, compare periodic snapshots instead. Databricks' AUTO CDC APIs handle it with AUTO CDC FROM SNAPSHOT, meant for when "CDC is not enabled on the source database and only snapshots are available."
Log-Based CDC: Snapshot First, Then the Log, and the Costs in Between
Log-based CDC sees every committed change, and it is where a data ingestion architecture takes on the most operational cost. It begins with a snapshot: the Debezium PostgreSQL connector documentation notes that "The first time it connects to a PostgreSQL server or cluster, the connector takes a consistent snapshot of all schemas," and then streams from the log position it recorded. The seam produces duplicates: "New records captured by the snapshot can appear twice; first, as part of the initial snapshot, and then again in the streaming phase."
Re-reading a table later no longer means pausing the stream. Debezium's incremental snapshots capture each table "in a series of configurable chunks," 1024 rows each by default, while streaming continues. The 2020 DBLog paper from Netflix engineers described the technique and noted that "The watermark approach does not use locks and has minimum impact on the source."
The hidden cost sits in the source. A PostgreSQL replication slot keeps write-ahead log until its consumer confirms it, and the PostgreSQL documentation on logical decoding is blunt: "Replication slots persist across crashes and know nothing about the state of their consumer(s)." Debezium warns that WAL disk usage can "spike or increase out of usual proportions" and adds heartbeat messages so the database can reclaim it. When a CDC connector stops, the data goes stale and the disk on a production database keeps growing. We track slot lag next to data freshness, as part of freshness, lag and schema-drift monitoring for streaming ingestion.
Managed CDC carries no latency promise; the AWS DMS guide to ongoing replication states that "There are no SLAs for CDC latency." When the consumer is another service, the outbox pattern writes the event in the same transaction as the business change and lets CDC publish it.
Files and Event Streams: Ingestion Driven by Arrival
File and event sources reverse the direction: the producer pushes, and the data ingestion architecture reacts to arrival. Databricks Auto Loader "incrementally and efficiently processes new data files as they arrive in cloud storage," by listing directories or, in file notification mode, through a queue that "subscribes to file events from the input directory." It "does not guarantee the order in which files are discovered or processed," so ordering must travel inside the record. After a failure it resumes from its checkpoint with exactly-once guarantees "when writing data into Delta Lake." Snowpipe uses load metadata so it "ignores the file if it's staged again."
Event streams make duplicates the normal case. The Apache Kafka design documentation says that "Kafka guarantees at-least-once delivery by default." The idempotent producer, available since 0.11.0.0, stops retries from duplicating log entries, but consumers can still reread after a restart. More useful for ingestion: "In many cases messages have a primary key and so the updates are idempotent." Google Pub/Sub offers exactly-once delivery "within a cloud region," once the feature is enabled on a subscription.
Whatever the tool, design the landing as if every event can arrive twice and out of order, because the documentation says when it happens.
The Landing Contract: Raw, Keyed, Replayable
The landing contract is the part of a data ingestion architecture that does not change with the pattern. We apply five rules to the first table of every source:
- Append-only, as received. Land the raw record with its operation type and merge later; the AUTO CDC APIs build SCD Type 1 and Type 2 tables from such a feed.
- Source position on every row. Store the LSN, offset or file path, plus the ingestion time.
- A deduplication key. Primary key plus source position. Without it, at-least-once delivery turns into duplicate rows.
- Deletes as data. Land a delete as its own row with the key and the operation, even when Debezium also emits a tombstone with "the same key but a null value."
- An explicit schema-evolution mode. According to the Auto Loader schema documentation, "
addNewColumnsmode is the default when a schema is not provided, but none is the default when you provide a schema," so choose it on purpose and keep_rescued_data, which "rescues columns that don't match the schema instead of dropping them."
When the landing table follows these rules, backfill and replay become operations on the platform instead of requests to the source owner. We design it with ingestion, orchestration and observability designed as one platform, since a replay is only as safe as the jobs reading the landing table.
Choosing a Pattern per Source, and How We Build It
In a data ingestion architecture, the pattern is chosen table by table, after six questions to the source. The answers map onto the rows of the table above.
- Is there a change column that someone maintains?
- Do hard deletes matter to anyone downstream?
- Can we get log or replication privileges, and who watches log retention?
- How many rows change relative to table size? Small reference tables are often cheapest as full snapshots.
- Does the source already push files or events on its own?
- Do consumers need intermediate versions, for audit or SCD Type 2?
Whether the data is then processed in batch or as a stream is a separate decision, driven by how fresh the result must be; choosing batch or streaming for the processing that follows covers that trade-off. The extraction method above is decided by the source, and the two choices are often different for the same table.
At DS Stream we start with a source inventory that answers the six questions for each table before anyone picks a tool, with one landing contract for every pattern. To start, bring us your source inventory and the one table you cannot reload.
FAQ
What is the difference between query-based and log-based CDC?
Query-based CDC asks the table what changed, using a timestamp or incrementing column, on a schedule. Log-based CDC reads the transaction log, so it sees deletes and intermediate versions without adding query load. Query-based capture is simpler and often enough when hourly freshness is fine and deletes do not matter.
How do you ingest hard deletes without log-based CDC?
By comparing snapshots. Take a periodic snapshot of the keys, compare it with the previous one, and emit a delete record for every key that disappeared. The snapshot interval sets how late a delete can arrive.
Does change data capture replace the initial full load?
No, it starts with one. A CDC connector first takes a consistent snapshot and then continues from the log position it recorded, and the target starts complete. Records written during the switch can arrive twice, so the landing table needs a deduplication key from day one.
How do you backfill a table without stopping ingestion?
A data ingestion architecture that can backfill without a pause uses incremental snapshots: the table is re-read in primary-key chunks while change events keep streaming, and watermarks resolve rows that appear in both. Debezium, Flink CDC and DBLog all work this way. For files, re-run discovery over the existing directory; the checkpoint skips files already processed.
