BigQuery - Connecting to External Sources

While the majority of the tutorial, up to this point, has concerned the UI and the cloud terminal, it's time to explore connecting to BigQuery through external sources.

Limitations of Writing Queries in the UI

Even though it may be convenient to write queries in BigQuery Studio, the truth is, this only fulfills a limited range of purposes −

• Initially developing a SQL query or script
• Debugging a query
• Conducting spot checks or quality assurance

Simply writing and running queries in the UI doesn't help deliver automated data solutions. This means that in the BigQuery SQL environment, you can't −

• Access the BigQuery API
• Integrate with Airflow
• Create ETL pipelines

External BigQuery Integrations

In the following chapters, we'll explore how to integrate BigQuery with −

• BigQuery Scheduled Queries
• BigQuery API (Python)
• Cloud Composer / Airflow
• Google Sheets
• BigQuery Data Transfers

External BigQuery integrations empower developers to leverage the power of SQL to perform the following tasks −

• Create automated extract load (EL)
• Extract transform load (ETL)
• Extract load transform (ELT) jobs