Efficient Data Deduplication Using Hash Indexing

PDE By islever Editorial Last updated

Building the pipelines.

Short answer

D Maintain a database table to store the hash value and other metadata for each data entry to enable efficient, indexed deduplication checks against incoming payloads.

Question

Your company uses a proprietary system to send inventory data every 6 hours to a data ingestion service in the cloud. Transmitted data includes a payload of
several fields and the timestamp of the transmission. If there are any concerns about a transmission, the system re-transmits the data. How should you
deduplicate the data most efficiency?
  1. Assign global unique identifiers (GUID) to each data entry.
  2. Compute the hash value of each data entry, and compare it with all historical data.
  3. Store each data entry as the primary key in a separate database and apply an index.
  4. Maintain a database table to store the hash value and other metadata for each data entry. Correct Answer

Key Takeaways

The main trap is choosing between GUIDs and hashes; GUIDs generated on-the-fly fail to match duplicates, while hashing all history is slow, making a hash-indexed table the optimal balance of speed and accuracy.

Learn how to efficiently deduplicate cloud-injected inventory data by storing hash values and metadata in a database table rather than comparing full payloads or relying on non-deterministic identifiers.

Common mistake: Option A is often chosen because GUIDs sound 'unique', but without a pre-existing ID in the payload, generating a new GUID for every ingestion creates distinct IDs for identical re-transmitted messages, failing to deduplicate.

Expert Analysis

The question asks for the most efficient way to deduplicate data where the source system re-transmits identical payloads upon concern. The key constraint is efficiency in handling these re-transmissions.

Option A suggests assigning global unique identifiers (GUID) to each entry. While GUIDs are excellent for uniqueness, the option implies generating them at the ingestion point or during processing. If the original payload lacks a pre-assigned ID, creating one on arrival does not help identify that a duplicate has arrived unless you compare it against existing records, which defeats the purpose of using the ID as a primary key for immediate rejection without complex lookups if the ID is generated fresh. More importantly, if the system re-sends the same data, a newly generated GUID would be different from the previous one, failing to detect the duplicate unless the generation logic is deterministic based on content (which makes it a hash).

Option B proposes computing a hash and comparing it with all historical data. This is computationally expensive ($O(N)$) as the dataset grows, making it inefficient for large-scale, continuous ingestion scenarios.

Option C suggests storing entries as primary keys. This requires storing the full payload as the key, which is storage-inefficient and still requires a lookup mechanism similar to hashing but with larger I/O overhead.

Option D recommends maintaining a database table to store the hash value and other metadata for each entry. This is the standard industry practice for idempotent ingestion. By storing only the hash (a fixed-size string) and metadata (like timestamp), you create a lightweight index. When new data arrives, you compute its hash and check this index (typically via a unique constraint or indexed column). This allows for $O(1)$ or $O( ext{log } N)$ lookups, ensuring high efficiency and low storage overhead compared to storing full payloads. The comments support this, noting that storing hashes allows efficient detection without comparing full payloads or generating non-deterministic IDs. The ambiguity in some comments about GUIDs stems from whether the ID is intrinsic to the message or assigned later; Option D explicitly handles the "re-transmission" scenario by checking against known hashes, which is robust and efficient.

Exam Strategy

When asked for the 'most efficient' deduplication strategy in cloud ingestion pipelines, prioritize solutions that use fixed-size identifiers (like hashes) stored in indexed structures. Avoid options that require scanning full histories or generating non-deterministic IDs for identical inputs, as these introduce computational or logical failures.

Frequently Asked Questions

Why isn't a GUID the best choice for deduplication here?

A GUID generated at ingestion time will be unique for every request, even if the payload is identical. Without a pre-assigned ID in the source message, you cannot distinguish a duplicate re-transmission from a new record using a freshly generated GUID.

How does storing hash metadata improve efficiency over storing the full payload?

Storing only the hash (e.g., SHA-256) and metadata keeps the index small and fast to query. Comparing a short hash string is significantly faster and uses less memory/disk than storing and comparing entire data payloads.

What happens if two different payloads have the same hash?

This is a collision. While theoretically possible, modern cryptographic hashes make collisions extremely rare. In practice, the hash serves as a strong identifier, and metadata (like timestamps) can help resolve edge cases.

More PDE FAQ →

Community Votes

D
54%
A
35%
B
11%

54% of anonymous learners picked answer D. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 119 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 30 of 119

  1. dg63 👍 69
    The best answer is "A". Answer "D" is not as efficient or error-proof due to two reasons 1. You need to calculate hash at sender as well as at receiver end to do the comparison. Waste of computing power. 2. Even if we discount the computing power, we should note that the system is sending inventory information. Two messages sent at different can denote same inventory level (and thus have same hash). Adding sender time stamp to hash will defeat the purpose of using hash as now retried messages will have different timestamp and a different hash. if timestamp is used as message creation timestamp than that can also be used as a UUID.
  2. retax 👍 13
    If the goal is to ensure at least ONE of each pair of entries is inserted into the db, then how is assigning a GUID to each entry resolving the duplicates? Keep in mind if the 1st entry fails, then hopefully the 2nd (duplicate) is successful.
  3. ralf_cc 👍 12
    A - In D, same message with different timestamp will have different hash, though the message content is the same.
  4. omakin 👍 8
    Strong Answer is A - in another question on the gcp sample questions: the correct answer to that particular question was "You are building a new real-time data warehouse for your company and will use BigQuery streaming inserts. There is no guarantee that data will only be sent in once but you do have a unique ID for each row of data and an event timestamp. You want to ensure that duplicates are not included while interactively querying data. Which query type should you use?" This means you need a "uniqueid" and timestamps to properly dedupe a data.
  5. medeis_jar 👍 7 Answered A
    Transmitted data includes fields and timestamp of transmission. So, hash value changes with re-transmission ==> Option B & D are wrong.
Show 25 more comments
  1. Radhika7983 👍 6
    According to me the answer is A. the question is about a company sending inventory data every 6 hours to data ingestion service in cloud. Cloud data flow can run both in batch and streaming and hence data will be ingested using pub sub and will be processed by cloud pub sub in batch mode. One thing to learn here is Pub/Sub assigns a unique message_id to each message, which can be used to detect duplicate messages received by the subscriber. So de duplication will be handle only if pub sub tries to send the same message. In this case publisher is transmitting the message again and hence publishes need to provide a unique message identifier. The other options talk about data base. As the question is specific to data ingestion and not data storage, I am going with A. However, even in cloud spanner, we can generate UUID for each message. Generating unique identifier is the most efficient way.
  2. cetanx 👍 5
    Answer - D Key statement is "Transmitted data includes a payload of several fields and the timestamp of the transmission." So the timestamp is appended to message while sending, in other words that field is subject to change if message is retransmitted. However, adding a GUID doesn't help much because if message is transmitted twice you will have different GUID for both messages but they will be the same/duplicate data. You can simply calculate a hash based on not all data but from a select of columns (with the payload of several fields AND definitely by excluding the timestamp). By doing so, you can assure a different hash for each message.
  3. korntewin 👍 5 Answered A
    I would vote on A, assuming that the re-transmission will submit the same GUID. D can be confused and can cause duplication or incorrect data on two scenarios: 1. If the hash includes transmission timestamp, every data transmission will have different hash --> data re-transmission will have different hash --> duplicate 2. If the hash data doesn't include transmission timestamp --> the inventory data on different time can have the same hash --> no data with the same value can be inserted to the DB!
  4. MarcoDipa 👍 5
    Answer is D. Using Hash values we can remove duplicate values from a database. Hash values will be same for duplicate data and thus can be easily rejected. Obviously you won't check hash for timestmp. D is better thatn B because maintaning a different table will reduce cost for hash computation for all historical data
  5. Roxy2 👍 5
    D is incorrect , data reord include transmission timestamp field and incase of retransmission there would be different timestamp and thus different HASH thus you can't compare based on has untill you remove timestamp field - which is not given n question
  6. emmylou 👍 5
    If you add a unique ID aren't you by definition not getting a duplicate record. Honestly I hate all these answers.
  7. moonlightbeamer 👍 4
    Answer: A it is a proprietary application, insert a GUID with a record is easy. in cloud whenever see fame GUID, will discard new transmitted entry. so simple and elegant. I don't see why it wouldn't work. others are all too complicated and unnecessary.
  8. juliobs 👍 4
    Hard question. It's a proprietary system. Who guarantees we can even add a GUID? But if you can, it's definitely more efficient than calculating hashes (ignoring timestamp).
  9. Nirca 👍 4 Answered A
    The simplest option is A. And simple is is an advantage.
  10. jkhong 👍 3 Answered A
    Initially misinterpreted the answers.. Assumed all of them to be very different approaches. But essentially the question is a comparison of different ways to generate unique identifiers for entries. Ways to generate unique identifiers: 1. Hashing 2. indexing 3. GUID There are storage and compute considerations that we have to make while choosing these methods. GUID does not require calculation again compared to hashing or indexing. However storage of GUID is heavy compared to the two. Since we are only concerned of efficiency, GUID is the fastest. We also do not need to store as we can implement a deduplication when our window closes. We don't need to store the unique values in a database in this case, as aggregations will be performed after the windowing period ends.
  11. hellobot 👍 3
    Best Answer = B answer A, GUID will help in keeping all values unique but will not remove duplicates for transaction purpose. answer D is not correct because maintaining external database is a good approach but it will not be efficient when there are large transactions that needs to be processed quickly and additional cost for database. I dont wanna pay for it. Answer B is correct because: you just need to pass a check cloud function because i assume data has to be processed in real time, before loading the data to bigquery. ``import haslib def compute_hash(data): hash_value = hashlib.sha256(data.encode('utf-8')) # detecting duplicates and ensure data integrity before storing in BQ return hash_value.hexdigest() ``
  12. JustQ 👍 3
    B. Compute the hash value of each data entry, and compare it with all historical data. Explanation: Efficiency: Hashing is a fast and efficient operation, and comparing hash values is generally quicker than comparing the entire payload or using other methods. Space Efficiency: Storing hash values requires less storage space compared to storing entire payloads or using global unique identifiers (GUIDs). Deduplication: By computing the hash value of each data entry and comparing it with historical data, you can easily identify duplicate transmissions. If the hash value matches an existing one, it indicates that the payload is the same.
  13. vickutk 👍 3
    Answer A Message deduplication is required for exactly-once message processing. Dataflow deduplicates messages with respect to the Pub/Sub message ID. As a result, all processing logic can assume that the messages are already unique with respect to the Pub/Sub message ID. The efficient, incremental aggregation mechanism to accomplish this is abstracted in the PubsubIO API. If PubsubIO is configured to use custom message IDs, Dataflow deduplicates messages by maintaining a list of all the custom IDs it has seen in the last 10 minutes. If a new message's ID is in this list, the message is assumed to be a duplicate and discarded.
  14. Hungry_guy 👍 3
    Answer is B - although the time stamp is diff for each transmission - the hash value is computed for the payload, not for the timestamp - which is just an added field for transmission. So, has val remains the same for all transmissions of the same data - which is what we can use for comparision. So, much more efficient to just directly compare the hash values with the historical data - to check and remove duplicates - instead of again wasting space storing stuff - in option D
  15. sumanshu 👍 3
    I don't think - A is correct answer here - because A says - Assign global unique identifiers (GUID) to each data entry. (i.e. in source itself we need assign Unique ID to each data) - Who will assign this ? (if data source is cloud storage). Though, if it;s PUB-SUB (then it automatically assign message ID - but no where in question its mention we need to use PUB-SUB) - it's only mention ingestion service ie. DATAFLOW and DATAPROC
  16. kishanu 👍 3
    Its B. A - I feel its incorrect, as the question talks about re-transmiting the data. If you assing a unique ID to the data, how we would identify tf the data is duplicated. B - As the hash value of a similar entry would be same, this can be used to re-transmit the data C- Way far then the question. D - Close, but why use a database and create overhead.
  17. Alexej_123 👍 3
    The answer A is Assign global unique identifiers (GUID) to each data entry. My question who should assign GUID. In the description it is pretty cear, that client will not assign and transfer guid to the cloud. If the client will not assign and transfer guid, how guid assignment will hellp to deduplicate? So I think A is wrong, and correct is D.
  18. TOXICcharlie 👍 3 Answered A
    My answer is A. Inventory data can often be naturally duplicate. Assigning a unique GUID at sender's end is ensuring that we can track a unique record reliably at the recieving end and if there are issues which causes same field to be sent twice, we can easily dedup using the GUID with lesser hassle.
  19. JK007 👍 3
    But this question is not saying how you are calculating the hash algo. It says hash of "data entry" which includes other fields and timestamp field. So taking hash of the entire data entry the hash value will be different as timestamp will be different each time.
  20. aldno 👍 3
    The question hints that the transmission timestamp is set anew with every re-transmission. This makes A the only viable answer. B, C and D all rely on the "first" transmission timestamp being constant for every data point.
  21. PolyMoe 👍 3 Answered B
    I would go for B. Seems to be most intuitive. If you want to know whether the input is already in the database, you need to compare it with the current content of the database. How ? by using hash
  22. devric 👍 3
    It's D. A doesn't recognize duplicates (it just assign an ID to each row).
  23. i_b1 👍 3 Answered D
    It's a proprietary solution. Cannot modify input
  24. yellow28 👍 2
    Your publisher when publishing messages to Pub/Sub can generate duplicates due to at-least-once publishing guarantees. Such duplicates are referred to as ‘publisher generated duplicates’. Pub/Sub automatically assigns a unique message_id to each message successfully published to a topic. Each message is considered successfully published by the publisher when Pub/Sub returns an acknowledgement to the publisher. Within a topic all messages have a unique message_id and no two messages have the same message_id. If success of the publish is not observed for some reason (network delays, interruptions etc) the same message payload may be retried by the publisher. If retries happen, we may end up with duplicate messages with different message_id in Pub/Sub. For Pub/Sub these are unique messages as they have different message_id. https://cloud.google.com/blog/products/data-analytics/handling-duplicate-data-in-streaming-pipeline-using-pubsub-dataflow
  25. Jackalski 👍 2 Answered D
    goal is to avoid duplicates in case resending the same information. Assuming using key-value store to handle duplicated values correctly - would failed on insert etc ( however not shared details here in this question) tricky question is what are definition of GUID versus hash. calculated hash on data sent first time and resent again can be the same assuming timestamp of datapoints inslide payload are not changed. timestamp of transmission is another story and I dont assume it is consider as value here (assumption to resend). GUID usually means to use UUID function which is random - no option to validate uniqness if GUID is build on specific way - than it would require complex logic ... Hash gives easy way to calculate it on full payload received Question was about most effective ... so I vote on D

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

How to establish consistent BigQuery spend?

S-grade

Triggering Airflow DAGs Reactively in Private Cloud Composer

S-grade

How to resolve Cloud Composer 2 worker pod evictions and task failures?

S-grade

Ensuring Consistency in BigQuery Streaming Analytics

S-grade

← Back to PDE Study Guide