Destination

octopipe destination Command Reference

The octopipe destination command allows you to manage data destinations in your Octopipe projects. This command is crucial for defining where transformed data is loaded after processing.

Purpose

• Data Output: Configure and manage destinations where data is stored after processing.
• Centralized Management: Keep track of all data endpoints to ensure smooth data flow and integration.

Usage

octopipe destination <subcommand> [options]

Subcommands

add

• Purpose: Add a new data destination.

• Usage Example:

octopipe destination add --name sales_db --type postgres --option host=localhost --option port=5432 --option user=dbuser --option password=secret --option database=sales

• Options:

• --name <destination_name>: Unique identifier for the destination.

• --type <destination_type> : Destination type (e.g., postgres, mysql, s3).

• --config-file <path>: Optional configuration file for complex setups.

• --option <key>=<value>: Additional settings specific to the destination.

list

• Purpose: List all configured data destinations.

• Usage Example:

octopipe destination list

update

• Purpose: Update an existing destination’s settings.

• Usage Example:

octopipe destination update sales_db --option password=newsecret

remove

• Purpose: Remove a data destination.

• Usage Example:

octopipe destination remove sales_db

Detailed Behavior

• Validation:

When adding or updating a destination, Octopipe validates connectivity using the provided parameters.

• Security:

Sensitive data such as passwords should be managed carefully, using environment variables when possible.

• Configuration Files:

For destinations with numerous settings, a configuration file can simplify the process.

Examples

Adding a PostgreSQL Destination

octopipe destination add --name sales_db --type postgres --option host=localhost --option port=5432 --option user=dbuser --option password=secret --option database=sales

Adding an S3 Destination

octopipe destination add --name backup_storage --type s3 --option bucket=my_bucket --option region=us-west-2 --option access_key=ACCESS123 --option secret_key=SECRET123

Listing Destinations

octopipe destination list

Updating a Destination

octopipe destination update sales_db --option password=newsecret

Removing a Destination

octopipe destination remove backup_storage

Best Practices

• Secure Credentials:

Use environment variables or secret management solutions for passwords and keys.

• Consistent Naming:

Maintain consistent naming conventions for ease of management.

• Documentation:

Keep detailed records of each destination’s settings for troubleshooting and audits.

Troubleshooting

• Connectivity Issues:

Verify that hostnames, ports, and credentials are correct if a destination fails to connect.

• Configuration Mismatches:

Ensure that configuration files are in the correct format and contain all required parameters.

• Listing Errors:

If destinations are not listed as expected, review the log files for potential errors.

Conclusion

The octopipe destination command is essential for defining where your processed data is stored. By carefully managing your destinations using the guidelines and examples provided, you can ensure reliable and secure data delivery.