feat: Demonstrate creating code using Gemini CLI to migrate mongo to bigquery

gemini-codegen/java/migrateMongodbToBigquery.md

@@ -0,0 +1,65 @@
+# Migrating Data from MongoDB to BigQuery
+
+This document provides instructions on how to run a Spark job in Java to migrate a collection from MongoDB to BigQuery using Dataproc Serverless.
+---
+The text between the line above and line below was written by a human. The rest of the document was created by Gemini. The initial prompt to Gemini was:
+```
+Create Spark job in Java to to migrate a collection from mongodb to BigQuery. Provide instructions to run this job on serverless spark in migrateMongodbToBigquery.md and provide a summary of the session in migrateMongodbToBigqueryREADME.md.
+
+```
+Gemini generated the Java app, specifically the file `MongoToBigQuery.java` and the README file. A small change was required to Gemini updated `pom.xml` specifying the BigQuery connector version and the scope. No changes were needed to the Java code. The working gcloud command is:
+```
+gcloud dataproc batches submit spark --project dataproc-templates --region us-central1 \
+   --batch="mongodb-to-bigquery-$(date +%s)" --class com.customer.app.MongoToBigQuery --version=2.2 \
+   --jars=<JARS> -- <MONGODB_URI> <DATABASE> <COLLECTION> <BQDATASET.TABLE> <TEMP_GCS_BUCKET>
+```
+---
+## Prerequisites
+
+1.  **Google Cloud Project:** A project with the Dataproc and BigQuery APIs enabled.
+2.  **GCS Bucket:** A bucket to store the compiled JAR and for BigQuery's temporary data.
+3.  **MongoDB Instance:** A reachable MongoDB instance (e.g., MongoDB Atlas or on GCE).
+4.  **BigQuery Dataset:** A dataset where the table will be created.
+
+## Building the JAR
+
+Build the project using Maven to create the shaded JAR:
+
+```bash
+mvn clean package
+```
+
+The output JAR will be `target/spark-delta-to-iceberg-migration-1.0-SNAPSHOT.jar`.
+
+## Running on Dataproc Serverless
+
+Use the `gcloud dataproc batches submit spark` command to run the job.
+
+```bash
+gcloud dataproc batches submit spark \
+    --project=<PROJECT_ID> \
+    --region=<REGION> \
+    --batch=mongo-to-bq-job \
+    --class=com.customer.app.MongoToBigQuery \
+    --jars=target/spark-delta-to-iceberg-migration-1.0-SNAPSHOT.jar \
+    -- \
+    "<MONGO_URI>" \
+    "<MONGO_DATABASE>" \
+    "<MONGO_COLLECTION>" \
+    "<BQ_DATASET.TABLE>" \
+    "<TEMP_GCS_BUCKET>"
+```
+
+### Parameters:
+
+*   `<PROJECT_ID>`: Your Google Cloud project ID.
+*   `<REGION>`: The region where you want to run the batch (e.g., `us-central1`).
+*   `<MONGO_URI>`: The connection string for your MongoDB instance.
+*   `<MONGO_DATABASE>`: The name of the MongoDB database.
+*   `<MONGO_COLLECTION>`: The name of the MongoDB collection to migrate.
+*   `<BQ_DATASET.TABLE>`: The destination BigQuery table in the format `dataset.table`.
+*   `<TEMP_GCS_BUCKET>`: A GCS bucket used by the BigQuery connector for temporary data.
+
+## Note on Networking
+
+Ensure that the Dataproc Serverless environment has network access to your MongoDB instance. If MongoDB is outside of Google Cloud, you might need to configure Cloud NAT or a VPN. If it's on MongoDB Atlas, ensure the IP addresses of your Dataproc workers are whitelisted or use VPC Peering/Private Service Connect.

gemini-codegen/java/migrateMongodbToBigqueryREADME.md

@@ -0,0 +1,21 @@
+# Session Summary: MongoDB to BigQuery Migration
+
+This session focused on implementing a Spark job in Java to migrate a collection from MongoDB to BigQuery on Dataproc Serverless.
+
+## Key Changes
+
+1.  **Dependencies:** Added the following dependencies to `pom.xml`:
+    *   `org.mongodb.spark:mongo-spark-connector_2.13`
+    *   `com.google.cloud.spark:spark-bigquery-with-dependencies_2.13`
+2.  **Implementation:** Created `MongoToBigQuery.java` in `com.customer.app`. This job:
+    *   Reads from a specified MongoDB URI, database, and collection.
+    *   Uses `DataframeUtils` to add an `insertion_time` timestamp to the data.
+    *   Writes the transformed data to a BigQuery table.
+3.  **Instructions:** Provided detailed instructions in `migrateMongodbToBigquery.md` on:
+    *   Prerequisites for the migration.
+    *   Building the shaded JAR using Maven.
+    *   Submitting the job as a Dataproc Serverless batch using `gcloud`.
+
+## Running the Migration
+
+To execute the migration, build the project and use the `gcloud` command provided in `migrateMongodbToBigquery.md`. The command requires the MongoDB URI, database name, collection name, destination BigQuery table, and a temporary GCS bucket for the BigQuery connector.

gemini-codegen/java/pom.xml

@@ -64,6 +64,19 @@
             <version>3.1.0</version>
         </dependency>
 
+        <!-- MongoDB and BigQuery connectors -->
+        <dependency>
+            <groupId>org.mongodb.spark</groupId>
+            <artifactId>mongo-spark-connector_2.13</artifactId>
+            <version>10.4.0</version>
+        </dependency>
+        <dependency>
+            <groupId>com.google.cloud.spark</groupId>
+            <artifactId>spark-3.5-bigquery</artifactId>
+            <version>0.36.4</version>
+            <scope>provided</scope>
+        </dependency>
+
         <!-- JDBC Drivers -->
         <dependency>
             <groupId>org.postgresql</groupId>

gemini-codegen/java/src/main/java/com/customer/app/MongoToBigQuery.java

@@ -0,0 +1,48 @@
+package com.customer.app;
+
+import com.customer.util.DataframeUtils;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.SaveMode;
+import org.apache.spark.sql.SparkSession;
+
+public class MongoToBigQuery {
+
+    public static void main(String[] args) {
+        if (args.length != 5) {
+            System.err.println("Usage: MongoToBigQuery <mongo.uri> <mongo.database> <mongo.collection> <bq.dataset.table> <temp.gcs.bucket>");
+            System.exit(1);
+        }
+
+        String mongoUri = args[0];
+        String mongoDatabase = args[1];
+        String mongoCollection = args[2];
+        String bqDatasetTable = args[3];
+        String tempGcsBucket = args[4];
+
+        SparkSession spark = SparkSession.builder()
+                .appName("MongoDB to BigQuery Migration")
+                .getOrCreate();
+
+        // Read data from MongoDB
+        Dataset<Row> df = spark.read()
+                .format("mongodb")
+                .option("connection.uri", mongoUri)
+                .option("database", mongoDatabase)
+                .option("collection", mongoCollection)
+                .load();
+
+        // Add insertion time column
+        Dataset<Row> transformedDf = DataframeUtils.addInsertionTimeColumn(df);
+
+        // Write data to BigQuery
+        transformedDf.write()
+                .format("bigquery")
+                .option("table", bqDatasetTable)
+                .option("temporaryGcsBucket", tempGcsBucket)
+                .mode(SaveMode.Overwrite)
+                .save();
+
+        spark.stop();
+    }
+}