= Google BigQuery Component
:doctitle: Google BigQuery
:shortname: google-bigquery
:artifactid: camel-google-bigquery
:description: Google BigQuery data warehouse for analytics.
:since: 2.20
:supportlevel: Stable
:tabs-sync-option:
:component-header: Only producer is supported
//Manually maintained attributes
:group: Google
:camel-spring-boot-name: google-bigquery

*Since Camel {since}*

*{component-header}*

The Google Bigquery component provides access
to https://cloud.google.com/bigquery/[Cloud BigQuery Infrastructure] via
the https://developers.google.com/api-client-library/java/apis/bigquery/v2[Google Client Services API].

The current implementation does not use gRPC.

The current implementation does not support querying BigQuery, i.e., is a producer only.

Maven users will need to add the following dependency to their pom.xml
for this component:

[source,xml]
------------------------------------------------------
<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-google-bigquery</artifactId>
    <!-- use the same version as your Camel core version -->
    <version>x.x.x</version>
</dependency>

------------------------------------------------------

[[GoogleBigQuery-AuthenticationConfiguration]]

== Authentication Configuration

Google BigQuery component authentication is targeted for use with the GCP Service Accounts.
For more information, please refer to https://cloud.google.com/docs/authentication[Google Cloud Platform Auth Guide]

Google security credentials can be set explicitly by providing the path to the GCP credentials file location.

Or they are set implicitly, where the connection factory falls back on
https://developers.google.com/identity/protocols/application-default-credentials#howtheywork[Application Default Credentials].

When you have the **service account key**, you can provide authentication credentials to your application code.
Google security credentials can be set through the component endpoint:

[source,java]
--------------------------------------------------------
String endpoint = "google-bigquery://project-id:datasetId[:tableId]?serviceAccountKey=/home/user/Downloads/my-key.json";
--------------------------------------------------------

You can also use the base64 encoded content of the authentication credentials file if you don't want to set a file system path.

[source,java]
--------------------------------------------------------
String endpoint = "google-bigquery://project-id:datasetId[:tableId]?serviceAccountKey=base64:<base64 encoded>";
--------------------------------------------------------

Or by setting the environment variable `GOOGLE_APPLICATION_CREDENTIALS` :

--------------------------------------------------------
export GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/my-key.json"
--------------------------------------------------------

== URI Format

--------------------------------------------------------
google-bigquery://project-id:datasetId[:tableId]?[options]
--------------------------------------------------------

// component-configure options: START

// component-configure options: END

// component options: START
include::partial$component-configure-options.adoc[]
include::partial$component-endpoint-options.adoc[]
// component options: END

// endpoint options: START

// endpoint options: END

// component headers: START
include::partial$component-endpoint-headers.adoc[]
// component headers: END

== Usage

=== Producer Endpoints

Producer endpoints can accept and deliver to BigQuery individual and grouped
exchanges alike. Grouped exchanges have `Exchange.GROUPED_EXCHANGE` property set.

Google BigQuery producer will send a grouped exchange in a single api call unless different table suffix or
partition decorators are specified. In that case, it will break it down to ensure data is written with the
correct suffix or partition decorator.

Google BigQuery endpoint expects the payload to be either a map or list of maps. A payload containing a map
will insert a single row, and a payload containing a list of maps will insert a row for each entry in the list.

=== Template tables

Reference: https://cloud.google.com/bigquery/streaming-data-into-bigquery#template-tables

Templated tables can be specified using the `GoogleBigQueryConstants.TABLE_SUFFIX` header.

I.e. the following route will create tables and insert records sharded on a per-day basis:

[source,java]
------------------------------------------------------
from("direct:start")
  .header(GoogleBigQueryConstants.TABLE_SUFFIX, "_${date:now:yyyyMMdd}")
  .to("google-bigquery:sampleDataset:sampleTable")
------------------------------------------------------
Note it is recommended to use partitioning for this use case.

=== Partitioning

Reference: https://cloud.google.com/bigquery/docs/creating-partitioned-tables

Partitioning is specified when creating a table and if set data will be automatically partitioned into
separate tables. When inserting data a specific partition can be specified by setting the
`GoogleBigQueryConstants.PARTITION_DECORATOR` header on the exchange.

=== Ensuring data consistency

Reference: https://cloud.google.com/bigquery/streaming-data-into-bigquery#dataconsistency

An insert id can be set on the exchange with the header `GoogleBigQueryConstants.INSERT_ID` or by specifying
query parameter `useAsInsertId`. As an insert id need to be specified per row inserted the exchange header can't
be used when the payload is a list.
If the payload is a list the `GoogleBigQueryConstants.INSERT_ID` will
be ignored. In that case use the query parameter `useAsInsertId`.


include::spring-boot:partial$starter.adoc[]
