Quickstart

Quickstart with Octopipe

Welcome to the Octopipe quickstart guide. This document will help you set up and run your first pipeline in a few simple steps. Follow these instructions to quickly see Octopipe in action and start processing your data.

Step 1: Initialize Your Project

Begin by initializing a new Octopipe project. This command sets up the basic structure and configuration files needed for your pipeline.

octopipe init --name my_pipeline --description "Pipeline for Sales Data"

Tip: The —name flag defines your pipeline’s name, and —description provides context about its purpose.

Step 2: Authenticate with Octopipe

Before interacting with the platform, you need to authenticate. Use your API key to log in:

octopipe login --api-key YOUR_API_KEY_HERE

• Note: Ensure your API key is active. Check the authentication section in the documentation if you encounter any issues.

Step 3: Add a Data Source

Configure your data source by adding a connector. For example, to add a sales API as a data source, run:

octopipe source add --name sales_api --type api --option url=https://api.sales.com/data --option token=YOUR_TOKEN

• Explanation:

• —name assigns a unique name to your data source.

• —type specifies the kind of source (e.g., api, database, or file).

• Additional options such as url and token provide connection details.

Step 4: Add a Data Destination

Now, set up where your data will be loaded. For instance, if you’re using PostgreSQL as your destination, run:

octopipe destination add --name sales_db --type postgres --option host=localhost --option port=5432 --option user=dbuser --option password=secret --option database=sales

• Detail: This command configures the database connection by specifying host, port, credentials, and the target database name.

Step 5: Define a Transformation

Create a transformation that maps the type safe API schema to the labeled database schema. You can define this mapping using a schema file:

octopipe transform add --name sales_transform --source sales_api --destination sales_db --schema-file ./schemas/sales_schema.json

• Clarification: The transformation aligns API fields with database fields, ensuring data consistency during the load process.

Step 6: Create the Pipeline

With your components in place, create the pipeline that orchestrates these elements:

octopipe pipeline create --name daily_sales --source sales_api --destination sales_db --transform sales_transform --schedule "0 0 * * *"

• Schedule: The cron expression “0 0 * * *” schedules the pipeline to run daily at midnight.

Step 7: Start the Pipeline

Launch your pipeline to begin processing data:

octopipe start daily_sales

• Observation: A successful start will display confirmation messages and log entries indicating that data processing has begun.

Step 8: Monitor the Pipeline

To view real-time updates and logs, use the following command:

octopipe logs daily_sales --follow

• Tip: The —follow option streams live logs to your terminal, allowing you to monitor progress and troubleshoot if needed.

Final Thoughts

Congratulations! You have now set up and started your first Octopipe pipeline. This quickstart guide covered initialization, authentication, source and destination configuration, transformation setup, pipeline creation, and monitoring.

If you need additional assistance, refer to the detailed documentation or connect with the community for support. Enjoy building robust data pipelines with Octopipe!