Managing Connections

Connections are the bridge between the Akili platform and external data systems. Every kind: source input port in inputs.yaml references a named connection. This guide covers creating, testing, and managing connections for all supported connector types.

Connection Types

Connector Type	Use Case	Protocol
postgres	Relational databases (CDC and incremental)	PostgreSQL wire protocol
mysql	Relational databases (CDC and incremental)	MySQL protocol
s3	Object storage (file ingestion)	S3-compatible API
redpanda / kafka	Event streams	Kafka protocol
http	REST APIs	HTTP/HTTPS

Creating a Connection

Connections are registered via the CLI with a JSON configuration object. The configuration fields vary by connector type.

PostgreSQL Connection

akili connection create \
  --name production-db \
  --connector-type postgres \
  --config '{
    "host": "db.example.com",
    "port": 5432,
    "database": "orders",
    "username": "akili_reader",
    "ssl_mode": "require"
  }'

Note

Connection credentials (passwords, API keys) are stored as Kubernetes secrets, not in the connection configuration. The config object contains only non-sensitive connection parameters. Credentials are linked to the connection by name in the cluster.

S3 / Object Storage Connection

akili connection create \
  --name data-lake-input \
  --connector-type s3 \
  --config '{
    "endpoint": "https://s3.example.com",
    "bucket": "raw-data",
    "prefix": "orders/",
    "region": "us-east-1",
    "path_style": true
  }'

Redpanda / Kafka Connection

akili connection create \
  --name event-stream \
  --connector-type redpanda \
  --config '{
    "brokers": ["broker-1:9092", "broker-2:9092"],
    "security_protocol": "SASL_SSL",
    "sasl_mechanism": "SCRAM-SHA-256"
  }'

HTTP / REST API Connection

akili connection create \
  --name weather-api \
  --connector-type http \
  --config '{
    "base_url": "https://api.weather.example.com/v2",
    "auth_type": "bearer",
    "timeout_seconds": 30,
    "rate_limit_rps": 10
  }'

Testing a Connection

Always test a connection after creation to verify connectivity and credentials:

# Test by connection ID
akili connection test conn-abc123

# Output on success:
# OK  Connection 'production-db' is reachable

# Output on failure:
# FAIL  Connection 'production-db': timeout after 30s

The test performs a lightweight probe appropriate to the connector type:

Connector Type	Test Method
postgres / mysql	SELECT 1 query
s3	HeadBucket API call
redpanda / kafka	Metadata request to broker
http	HEAD request to base URL

Discovering Resources

After a connection is established, use the discover command to see available resources (tables, topics, paths):

# Discover tables in a PostgreSQL connection
akili connection discover production-db

# Output:
# RESOURCE              TYPE    ROWS
# public.orders         table   1,284,302
# public.customers      table   45,891
# public.products       table   3,204
# public.regions        view    48

Inspecting Schema

Drill into a specific resource to see its column schema:

akili connection schema production-db public.orders

# Output:
# COLUMN          TYPE            NULLABLE
# order_id        uuid            NO
# customer_id     uuid            NO
# product_id      uuid            NO
# quantity        integer         NO
# unit_price      decimal(10,2)   NO
# total_amount    decimal(12,2)   NO
# region          text            YES
# order_date      date            NO
# created_at      timestamptz     NO

This schema information can be used to populate the columns field in inputs.yaml.

Listing Connections

# List all connections for the current tenant
akili connection list

# As JSON
akili connection list --json

Updating a Connection

Update connection parameters without recreating the connection:

akili connection update \
  --id conn-abc123 \
  --config '{
    "host": "new-db.example.com",
    "port": 5432,
    "database": "orders",
    "username": "akili_reader",
    "ssl_mode": "require"
  }'

Caution

Updating a connection does not automatically retest it. Always run akili connection test after updating to verify the new configuration works.

Deleting a Connection

# Attempt deletion (requires --confirm)
akili connection delete conn-abc123 --confirm

Deletion will fail if any deployed data product references the connection in its inputs.yaml. You must first undeploy or update those products to remove the dependency.

Referencing Connections in Manifests

Once created, connections are referenced by name in inputs.yaml:

inputs:
  - name: orders
    kind: source
    connection: production-db     # <-- connection name
    resource: public.orders
    strategy: cdc
    cursor_field: updated_at

The platform resolves the connection name to the full connection configuration (including credentials from Kubernetes secrets) at execution time. Manifests never contain connection details directly.

Connection Lifecycle

%%{init: {'flowchart': {'curve': 'basis'}}}%%
flowchart LR
    A[Create connection] --> B[Test connectivity]
    B --> C[Discover resources]
    C --> D[Reference in inputs.yaml]
    D --> E[Deploy product]
    E --> F[Connection used at runtime]
    F --> G[Rotate credentials]
    G --> B

Credential Rotation

When credentials need rotation:

1. Update the Kubernetes secret with new credentials
2. Test the connection: akili connection test <id>
3. No product redeployment is needed — credentials are resolved at runtime

Related

• Ingestion Patterns — ingestion strategies that use connections
• akili connection — full CLI reference