Scalable and Serverless Data Ingestion with OCI Functions

Build a scalable data pipeline: weather station CSV to Autonomous Database using OCI Functions, Object Storage, and Terraform.
Scalable and Serverless Data Ingestion with OCI Functions
Scalable and Serverless Data Ingestion with OCI Functions

In this article, we will leverage OCI’s capabilities to the fullest, embracing the following principles:

  • Scalability
  • Resilience
  • Flexibility
  • Security
  • Automation

The “loadfileintoadw” project is located in the same GitHub repository used so far to discuss OCI Function fn-examples .

This example will help you understand how to integrate multiple OCI services and make the most of the cloud provider.

We will simulate a series of weather stations that will write a CSV file with sampling data (temperature, humidity, etc.). The sensor will automatically upload the file to an Object Storage bucket.

Processing will be automatically invoked through a Function that, in this case, converts the file format from CSV to JSON and natively saves it to a serverless Autonomous Database.

The IaC code has been developed to perform the entire deployment, from infrastructure to function. Therefore, within the project folder, simply execute the command to create everything:

terraform apply

If you are not familiar with Terraform, we recommend checking theofficial documentation and our tutorial and video

Frequently Asked Questions

How do I build a serverless ETL pipeline on Oracle Cloud?

The standard OCI pattern uses Object Storage as the landing zone, an Event Rule to trigger an OCI Function on file upload, and the Function to parse and load data into Autonomous Database. Terraform provisions the full pipeline as infrastructure as code.

Can OCI Functions process large CSV files from Object Storage?

Yes. OCI Functions can download objects directly from Object Storage using the OCI Python SDK, parse CSV data with pandas or the standard csv module, and insert rows into Autonomous Database. For very large files, consider chunking or using OCI Data Integration instead.

Is a serverless data ingestion pipeline on OCI cost-effective?

Yes. With OCI Functions you pay only for actual execution time. Object Storage charges are minimal for typical CSV workloads. For intermittent or batch ingestion patterns, this approach is significantly cheaper than a continuously running VM or managed ETL service.

Related posts