Improving Data Discoverability with Data Catalog

PDE By islever Editorial Last updated

Designing for a data platform.

Short answer

C Use Data Catalog to automatically catalog BigQuery datasets and Pub/Sub topics, and use custom connectors to manually catalog PostgreSQL tables.

Question

Your organization's data assets are stored in BigQuery, Pub/Sub, and a PostgreSQL instance running on Compute Engine. Because there are multiple domains and diverse teams using the data, teams in your organization are unable to discover existing data assets. You need to design a solution to improve data discoverability while keeping development and configuration efforts to a minimum. What should you do?
  1. Use Data Catalog to automatically catalog BigQuery datasets. Use Data Catalog APIs to manually catalog Pub/Sub topics and PostgreSQL tables.
  2. Use Data Catalog to automatically catalog BigQuery datasets and Pub/Sub topics. Use Data Catalog APIs to manually catalog PostgreSQL tables.
  3. Use Data Catalog to automatically catalog BigQuery datasets and Pub/Sub topics. Use custom connectors to manually catalog PostgreSQL tables. Correct Answer
  4. Use customer connectors to manually catalog BigQuery datasets, Pub/Sub topics, and PostgreSQL tables.

Key Takeaways

Data Catalog automatically catalogs BigQuery and Pub/Sub; for non-native sources like PostgreSQL, use community connectors instead of building manual API integrations to reduce effort.

Learn how to improve data discoverability across BigQuery, Pub/Sub, and PostgreSQL using Data Catalog's native auto-cataloging and community connectors to minimize development effort.

Common mistake: Choosing option B (using APIs for PostgreSQL) because users forget that community connectors exist for common databases like PostgreSQL, making APIs unnecessarily complex for this scenario.

Expert Analysis

Native Auto-Cataloging for Google Cloud Services

Google Cloud Data Catalog provides built-in, out-of-the-box integrations for specific managed services. When you create datasets in BigQuery or topics in Pub/Sub, Data Catalog automatically discovers these assets and ingests their metadata without any additional configuration. This native capability eliminates the need for manual intervention or custom code for these two data sources.

Handling Non-Native Sources: Custom Connectors vs. APIs

The PostgreSQL instance running on Compute Engine is not a native Google Cloud service supported by Data Catalog's automatic discovery. To catalog this data with minimal effort, you should use community-contributed connectors. These are pre-built connectors maintained by the community that allow you to ingest metadata from popular non-Google sources (like PostgreSQL) into Data Catalog with relatively low development overhead. While using the Data Catalog APIs to manually create entries is possible, it requires significantly more development effort compared to leveraging an existing connector.

Why Other Options Are Incorrect

Option A incorrectly suggests manual API usage for Pub/Sub, which is natively supported. Option D suggests manual cataloging for all sources, ignoring the automation available for BigQuery and Pub/Sub. Option B suggests using APIs for PostgreSQL; while technically feasible, it contradicts the requirement to keep development efforts to a minimum when a community connector exists. Therefore, leveraging native auto-cataloging for BigQuery and Pub/Sub, combined with a community connector for PostgreSQL, is the optimal solution.

Official Reference

https://cloud.google.com/data-catalog/docs/integrate-data-sources#in

This link opens the official vendor documentation in a new tab.

Exam Strategy

When designing data governance solutions, always check if a source is natively supported by Data Catalog first. If it is, no action is needed. If not, look for community connectors before resorting to manual API implementation, as this aligns with exam goals of minimizing effort and maximizing automation.

Frequently Asked Questions

Does Data Catalog automatically catalog PostgreSQL?

No, Data Catalog does not natively auto-catalog PostgreSQL. You must use a community connector or build a custom integration via APIs.

What is the difference between custom connectors and community connectors?

Community connectors are pre-built by the community for popular sources. Custom connectors are typically built by you using APIs when no community connector exists.

More PDE FAQ →

Community Votes

B
55%
C
45%

55% of anonymous learners picked answer B. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 32 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 30 of 32

  1. raaad 👍 16 Answered B
    - It utilizes Data Catalog's native support for both BigQuery datasets and Pub/Sub topics. - For PostgreSQL tables running on a Compute Engine instance, you'd use Data Catalog APIs to create custom entries, as Data Catalog does not automatically discover external databases like PostgreSQL.
  2. datapassionate 👍 14 Answered C
    Data Catalog is the best choice. But for catalogging PostgreSQL it is better to use a connector when available, instead of using API. https://cloud.google.com/data-catalog/docs/integrate-data-sources#integrate_unsupported_data_sources
  3. ML6 👍 5 Answered C
    Google Recommendation: If you can't find a connector for your data source, you can still manually integrate it by creating entry groups and custom entries. To do that, you can: - Use one of the Data Catalog Client Libraries in one of the following languages: C#, Go, Java, Node.js, PHP, Python, or Ruby. - Or manually build on the Data Catalog API. However, there is a connector for PostgreSQL, so option C.
  4. AllenChen123 👍 5
    Agree. https://cloud.google.com/data-catalog/docs/concepts/overview#catalog-non-google-cloud-assets
  5. fitri001 👍 4 Answered B
    BigQuery Datasets and Pub/Sub Topics: Google Data Catalog can automatically catalog metadata from BigQuery and Pub/Sub, making it easy to discover and manage these data assets without additional development effort. PostgreSQL Tables: While Data Catalog does not have built-in connectors for PostgreSQL, you can use the Data Catalog APIs to manually catalog the PostgreSQL tables. This requires some custom development but is manageable compared to creating custom connectors for everything.
Show 25 more comments
  1. tibuenoc 👍 4
    Agree. If it doesn't have a connector, it must be manually built on the Data Catalog API. As PostgreSQL already has a connector it's the best option is C
  2. baimus 👍 3 Answered C
    This is C. To clarify some issues below with B, the links provided by supporters of B actually do say that it's preferable to use a community connector where available, and to only use the API when the case is genuinely not supported by community connectors. In this case it's Postgresql, so it's supported, see here for full list: https://cloud.google.com/data-catalog/docs/integrate-data-sources#integrate_on-premises_data_sources So this would be B if it was something like Q+ or some genuinely unsupported database, but postgres is supported for community connector.
  3. joao_01 👍 3
    In the opction C, the expression "Use custom connectors to manually catalog PostgreSQL tables." is refering to the use case of Google when you want to use "Community-contributed connectors to multiple popular on-premises data sources". As you can see, this connectors are for ON-PREMISSES data sources ONLY. In this case the Postgres is in a VM in the cloud. Thus, the option correct is B.
  4. shanks_t 👍 3 Answered B
    Data Catalog automatically catalogs metadata from Google Cloud sources such as BigQuery, Vertex AI, Pub/Sub, Spanner, Bigtable, and more. To catalog metadata from non-Google Cloud systems in your organization, you can use the following: Community-contributed connectors to multiple popular on-premises data sources Manually build on the Data Catalog APIs for custom entries
  5. LaxmanTiwari 👍 3 Answered C
    I vote for c as per Integrate on-premises data sources To integrate on-premises data sources, you can use the corresponding Python connectors contributed by the community: under the link https://cloud.google.com/data-catalog/docs/integrate-data-sources
  6. saschak94 👍 3 Answered C
    If you can't find a connector for your data source, you can still manually integrate it by creating entry groups and custom entries. To do that, you can: - Manually build on the Data Catalog API.
  7. Matt_108 👍 3 Answered B
    Option B - Data Catalog automatically maps out GCP resources and dev efforts are minimized by leveraging the data catalog API to do the same for postgresql db
  8. Cassim 👍 2 Answered B
    Option B leverages Data Catalog to automatically catalog BigQuery datasets and Pub/Sub topics, which streamlines the process and reduces manual effort. Using Data Catalog APIs to manually catalog PostgreSQL tables ensures consistency across all data assets while minimizing development and configuration efforts.
  9. hanoverquay 👍 2 Answered B
    option B, there's no need to build a custom connector now, postgreSQL is now supported https://github.com/GoogleCloudPlatform/datacatalog-connectors-rdbms/tree/master/google-datacatalog-postgresql-connector
  10. GCP001 👍 2
    B. -- Looks much better option as needed low development efforts. -- C not looking right as it will need lot of dev efforts for custom connectors.
  11. virat_kohli 👍 2 Answered B
    B. Use Data Catalog to automatically catalog BigQuery datasets and Pub/Sub topics. Use Data Catalog APIs to manually catalog PostgreSQL tables.
  12. Y___ash 👍 2 Answered B
    Use Data Catalog to automatically catalog BigQuery datasets and Pub/Sub topics. Use Data Catalog APIs to manually catalog PostgreSQL tables.
  13. AWSandeep 👍 2 Answered B
    This section explains it clearly: https://cloud.google.com/data-catalog/docs/integrate-data-sources#integrate_unsupported_data_sources.
  14. Harshzh12 👍 2 Answered B
    Datacatalog API contain the connector for postgresql with using it developer don't have to create the custom connectors
  15. 987af6b 👍 2 Answered C
    I’m voting for C because the documentation states that Postgres is a custom connector developed by the community.
  16. carbonscave 👍 1 Answered C
    Why C is the right move Native Auto-Cataloging: Google Cloud Data Catalog (now part of Dataplex) has native, out-of-the-box integration with both BigQuery and Pub/Sub. The moment a BigQuery dataset or Pub/Sub topic is created, Data Catalog automatically discovers it and makes the metadata searchable with zero configuration or development effort required. This eliminates options A and D. Minimizing Development for External DBs: Because PostgreSQL is running unmanaged on a Compute Engine instance, Data Catalog cannot automatically see it. To bridge this gap with the "minimum development effort," Google provides open-source custom connectors (e.g., google-datacatalog-postgresql-connector). Connectors vs. APIs: These pre-built custom connectors handle the extraction and formatting for you. You only need to configure the connection string and deploy the script. This requires vastly less development effort than Option B, which suggests manually interacting with the Data Catalog REST APIs and writing the entire metadata extraction and ingestion codebase from scratch.
  17. af17139 👍 1 Answered C
    Automatic Cataloging: Data Catalog (now part of Dataplex) has built-in integrations to automatically discover and register metadata from Google Cloud managed services like BigQuery and Pub/Sub. This requires little to no configuration. Custom Connector for PostgreSQL: For the PostgreSQL instance running on Compute Engine, you'll need to develop a "custom connector." This typically involves writing a script or application that: Connects to the PostgreSQL database. Extracts schema information (tables, columns, types, etc.), potentially from information_schema. Transforms this information into the format required by the Data Catalog API. Uses the Data Catalog APIs to create or update entries for the PostgreSQL tables. This connector can be run on a schedule (e.g., using Cloud Scheduler and Cloud Functions) to keep the metadata in Data Catalog synchronized with the PostgreSQL instance.
  18. Abizi 👍 1 Answered C
    Why C is Correct? BigQuery datasets → ✅ Automatically cataloged in Data Catalog Pub/Sub topics → ✅ Automatically cataloged in Data Catalog PostgreSQL on Compute Engine → ❌ Not automatically cataloged Requires a custom connector to extract metadata and push it to Data Catalog. Option B (using Data Catalog APIs manually) is not enough because PostgreSQL metadata isn’t natively supported. Why Not B? Option B suggests using Data Catalog APIs manually for PostgreSQL. However, Data Catalog does not natively support PostgreSQL metadata extraction. You need a custom connector to first extract PostgreSQL schema information, then push it to Data Catalog.
  19. aaaaaaaasdasdasfs 👍 1 Answered B
    This is the correct approach. Data Catalog provides automatic discovery and metadata extraction for both BigQuery datasets and Pub/Sub topics as native Google Cloud services. You'll only need to use the Data Catalog APIs to manually catalog the PostgreSQL tables running on Compute Engine, as this is a non-native data source. This maximizes automation while minimizing development effort.
  20. d11379b 👍 1
    I think “custom connector” here may just infer that this is not official tools? as the doc mentioned “ connectors contributed by the community” And should not be B as “manually catalog by API “ this is a way even more basic than using connector
  21. 7787de3 👍 1
    I agree. On the linked page: If you can't find a connector for your data source, you can still manually integrate it by creating entry groups and custom entries. As we can find a connector there, it should be used.
  22. 987af6b 👍 1
    Changed my mind. B. -This is not on premise, so the custom connector should not be applicable -Question says keep manual dev and config to a minimum
  23. shanks_t 👍 1
    C. While similar to B, using custom connectors for PostgreSQL might involve more development effort than using the Data Catalog APIs directly.
  24. StefG_ITA 👍 1
    Not is B. "Custom Connectors" violating the "minimum development and configuration effort" requirement of the question.
  25. meh_33 👍 1
    raaad mostly correct and we can check his description supporting his answer so we can go with it .Cheers mate

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

How to establish consistent BigQuery spend?

S-grade

Triggering Airflow DAGs Reactively in Private Cloud Composer

S-grade

Bigtable Row Key Design for Time-Series Data

S-grade

Migrating Hadoop Spark Jobs to Dataproc Serverless and Cloud Storage

S-grade

← Back to PDE Study Guide