# Create API Key
Source: https://docs.dataraven.io/api-reference/api-keys/create-api-key
POST /v1/teams/{team_id}/api-keys
Create a new API key. The full key is returned ONCE in the response.
**Required Role:** ADMIN or OWNER
**Security Note:** The `full_key` field is shown only in this response.
Store it securely -- it cannot be retrieved again.
# Delete API Key
Source: https://docs.dataraven.io/api-reference/api-keys/delete-api-key
DELETE /v1/teams/{team_id}/api-keys/{api_key_id}
Hard delete an API key (permanent removal).
**Required Role:** ADMIN or OWNER
# Get API Key
Source: https://docs.dataraven.io/api-reference/api-keys/get-api-key
GET /v1/teams/{team_id}/api-keys/{api_key_id}
Get detailed API key information.
**Required Role:** VIEWER or higher
# List API Keys
Source: https://docs.dataraven.io/api-reference/api-keys/list-api-keys
GET /v1/teams/{team_id}/api-keys
List API keys for a team. Secrets are never returned.
**Required Role:** VIEWER or higher
# Revoke API Key
Source: https://docs.dataraven.io/api-reference/api-keys/revoke-api-key
POST /v1/teams/{team_id}/api-keys/{api_key_id}/revoke
Revoke an API key (soft delete -- key becomes unusable).
**Required Role:** ADMIN or OWNER
# Rotate API Key
Source: https://docs.dataraven.io/api-reference/api-keys/rotate-api-key
POST /v1/teams/{team_id}/api-keys/{api_key_id}/rotate
Rotate an API key's secret. Returns the new full key (shown ONCE).
**Required Role:** ADMIN or OWNER
Generates a new secret while keeping the same key ID, name, and scopes.
The old secret is immediately invalidated. All subsequent requests must
use the new key.
**Security Note:** The `full_key` field is shown only in this response.
Store it securely -- it cannot be retrieved again.
# List Audit Logs
Source: https://docs.dataraven.io/api-reference/audit-logs/list-audit-logs
GET /v1/teams/{team_id}/audit-logs
Get audit log feed for a team.
**Required Role:** VIEWER or higher
**Query Parameters:**
- `resource_type`: Filter by resource (task, execution, secret, location, member, notification, vault_connection, api_key)
- `event_type`: Filter by event (task_created, execution_completed, etc.)
- `actor_type`: Filter by actor (user, system, api_key)
- `start_date`: Filter events from this date (ISO 8601)
- `end_date`: Filter events until this date (ISO 8601)
- `page`: Page number (default: 1)
- `limit`: Items per page (default: 25, max: 250)
# Download Logs
Source: https://docs.dataraven.io/api-reference/executions/download-logs
GET /v1/teams/{team_id}/tasks/{task_id}/executions/{execution_id}/logs/download
Download full log file from AWS S3.
**Required Role:** VIEWER or higher
This endpoint streams the complete RClone output log file (gzipped).
The log file contains every line of RClone output including:
- All file transfer events
- Complete stats blocks (every 10 seconds)
- Error messages and stack traces
- RClone debug information
**Requirements:**
- Execution must be in terminal state: COMPLETED, DRY_RUN_COMPLETED, FAILED, or CANCELLED
- Returns 400 Bad Request if execution is still PENDING, QUEUED, or RUNNING
The file is streamed directly from AWS S3 (no pre-signed URLs).
# Dry Run
Source: https://docs.dataraven.io/api-reference/executions/dry-run
POST /v1/teams/{team_id}/tasks/{task_id}/executions/dry-run
Submit a task for dry-run execution.
**Required Role:** OPERATOR, ADMIN, or OWNER
Dry-run mode performs all checks and simulations without actually
transferring or modifying any files. This is useful for:
- Testing task configuration
- Previewing what files would be transferred
- Estimating transfer size and duration
Dry-run executions:
- Do NOT transfer or modify files
- Preview what would be transferred without actual data movement
- Complete with status DRY_RUN_COMPLETED instead of COMPLETED
- Show accurate file counts and size estimates in stats
**Note:** This endpoint requires no request body. Execution parameters are
determined server-side (is_dry_run=true, trigger=manual).
# Get Execution
Source: https://docs.dataraven.io/api-reference/executions/get-execution
GET /v1/teams/{team_id}/tasks/{task_id}/executions/{execution_id}
Get detailed execution information.
**Required Role:** VIEWER or higher
Returns comprehensive execution details including:
- Current status and timing information
- Complete RClone statistics (all fields)
- Error information (if failed)
- Task configuration snapshot (audit trail)
- Log file URL (if execution completed)
Use this endpoint to:
- Monitor real-time progress (check stats.progress_percentage, stats.eta)
- View active file transfers (stats.transferring array)
- Check error details after failure
- Access log file for download
**Real-time Log Streaming:**
For live log updates during execution, use /logs/stream endpoint:
- Subscribe when execution.status is RUNNING
- See /logs/stream documentation for usage details
# List Executions
Source: https://docs.dataraven.io/api-reference/executions/list-executions
GET /v1/teams/{team_id}/tasks/{task_id}/executions
List executions for a task with filtering and pagination.
**Required Role:** VIEWER or higher
Returns executions ordered by queued_at descending (most recent first).
Query Parameters:
- status: Filter by execution status (pending, queued, running, completed, failed, cancelled, dry_run_completed)
- page: Page number (1-indexed)
- limit: Items per page (max 100)
# Stop Execution
Source: https://docs.dataraven.io/api-reference/executions/stop-execution
POST /v1/teams/{team_id}/tasks/{task_id}/executions/{execution_id}/stop
Request cancellation of a running execution.
**Required Role:** OPERATOR, ADMIN, or OWNER
This sets the execution status to CANCELLED. The worker will detect
this change (polls every 10 seconds) and gracefully stop the RClone
process using SIGTERM.
**Note:** Cancellation is not instant. The worker needs to:
1. Detect the status change (up to 10 seconds)
2. Send SIGTERM to RClone process
3. Wait for graceful shutdown (up to 30 seconds)
4. Upload partial logs to object storage
5. Update final execution status
Can only cancel executions in PENDING or RUNNING status.
# Health Check
Source: https://docs.dataraven.io/api-reference/health/health-check
GET /v1/health
Health check endpoint.
Returns basic API status information.
This endpoint is public and does not require authentication.
# API Reference
Source: https://docs.dataraven.io/api-reference/introduction
DataRaven REST API
The DataRaven API lets you programmatically manage teams, locations, tasks, executions, and vault
connections.
## Base URL
```
https://api.dataraven.io
```
## Authentication
All API requests require a Bearer token in the `Authorization` header:
```bash theme={"theme":{"light":"github-light","dark":"poimandres"}}
curl https://api.dataraven.io/v1/teams \
-H "Authorization: Bearer YOUR_API_TOKEN"
```
## Rate Limits
The API enforces rate limits per IP address. If you receive a `429` response, respect the `Retry-After` header and retry with exponential backoff.
### Global Rate Limit
All endpoints are subject to a global rate limit:
* **120 requests per minute** per IP address
### Rate Limit Headers
When you make a request, the API returns rate limit information in response headers:
| Header | Description |
| ----------------------- | ------------------------------------------------------------- |
| `X-RateLimit-Limit` | The rate limit for the endpoint (requests per minute) |
| `X-RateLimit-Remaining` | The number of requests remaining in the current window |
| `X-RateLimit-Reset` | Unix timestamp (seconds) when the rate limit window resets |
| `Retry-After` | Seconds to wait before retrying (included in `429` responses) |
### Handling 429 Responses
When rate limited, the API responds with `429 Too Many Requests`:
```json theme={"theme":{"light":"github-light","dark":"poimandres"}}
{
"error": "Rate limit exceeded",
"message": "..."
}
```
### Best Practices
1. **Check the `Retry-After` header** before retrying a rate-limited request
2. **Implement exponential backoff** for retries
3. **Batch requests** when possible to reduce API calls
4. **Monitor rate limit headers** in responses to adjust request frequency proactively
Validation errors (`422`) include a `detail` array with specific field-level error information.
## OpenAPI Spec
The full OpenAPI 3.1 specification is available at:
```
https://api.dataraven.io/openapi.json
```
Interactive API documentation is available at:
```
https://api.dataraven.io/scalar
```
# Accept Invitation
Source: https://docs.dataraven.io/api-reference/invitations/accept-invitation
POST /v1/users/me/invitations/{invitation_id}/accept
Accept a pending invitation and join the team.
Creates a team membership with the role specified in the invitation.
# Create Invitation
Source: https://docs.dataraven.io/api-reference/invitations/create-invitation
POST /v1/teams/{team_id}/invitations
Invite a user to join the team by email.
**Required Role:** ADMIN or OWNER
**Tier Limits:**
- Free Tier: Maximum 1 team member (no invites allowed)
- Pro Tier: Maximum 20 team members (including pending invites)
**Invitation Behavior:**
- New users: Will receive an invitation email; they sign in via email OTP and auto-register
- Existing users: Will receive notification email (TODO)
- Both must explicitly accept the invitation
The invitation expires in 7 days.
# Decline Invitation
Source: https://docs.dataraven.io/api-reference/invitations/decline-invitation
POST /v1/users/me/invitations/{invitation_id}/decline
Decline a pending invitation.
The invitation is marked as declined and kept for audit purposes.
The team admin can send a new invitation if needed.
# Delete Invitation
Source: https://docs.dataraven.io/api-reference/invitations/delete-invitation
DELETE /v1/teams/{team_id}/invitations/{invitation_id}
Revoke a pending invitation.
**Required Role:** ADMIN or OWNER
Only works for invitations with status 'pending'.
# List Invitations
Source: https://docs.dataraven.io/api-reference/invitations/list-invitations
GET /v1/teams/{team_id}/invitations
List all invitations for a team.
**Required Role:** ADMIN or OWNER
Optional filters:
- status: Filter by invitation status (pending, accepted, declined, etc.)
# My Invitations
Source: https://docs.dataraven.io/api-reference/invitations/my-invitations
GET /v1/users/me/invitations
Get all pending invitations for the current user.
Returns invitations matching the user's email that are:
- Status: pending
- Not expired
Includes team name and inviter details for display.
# Resend Invitation
Source: https://docs.dataraven.io/api-reference/invitations/resend-invitation
POST /v1/teams/{team_id}/invitations/{invitation_id}/resend
Resend invitation email for a pending invitation.
**Required Role:** ADMIN or OWNER
Resets the expiration to 7 days from now.
# Create Location
Source: https://docs.dataraven.io/api-reference/locations/create-location
POST /v1/teams/{team_id}/locations
Create a new cloud storage location.
**Required Role:** ADMIN or OWNER
**Tier Limits:**
- Free Tier: Maximum 4 locations
- Pro Tier: Maximum 100 locations
Locations define cloud storage endpoints (buckets/containers). RClone configuration
options are set at the Task level, not here.
Example request body for S3:
```json
{
"name": "Production S3 Bucket",
"description": "Main production data storage",
"location_type": "s3",
"secret_id": "uuid-of-secret",
"bucket_name": "my-prod-bucket",
"region": "us-east-1"
}
```
Example for Azure Blob:
```json
{
"name": "Azure Backup Storage",
"location_type": "azure_blob",
"secret_id": "uuid-of-secret",
"bucket_name": "backup-container",
"region": "eastus"
}
```
# Delete Location
Source: https://docs.dataraven.io/api-reference/locations/delete-location
DELETE /v1/teams/{team_id}/locations/{location_id}
Delete a location.
**Required Role:** ADMIN or OWNER
**Validation:** Cannot delete if the location is used by any tasks.
You must delete the tasks first.
Returns:
- 204 No Content on success
- 404 Not Found if location doesn't exist
- 409 Conflict if location is in use by tasks
# Get Location
Source: https://docs.dataraven.io/api-reference/locations/get-location
GET /v1/teams/{team_id}/locations/{location_id}
Get detailed information about a location.
**Required Role:** VIEWER or higher
Returns:
- Full location configuration
- Usage statistics (tasks using this location)
- Verification status
# Get Provider Defaults
Source: https://docs.dataraven.io/api-reference/locations/get-provider-defaults
GET /v1/teams/{team_id}/locations/defaults/{location_type}
Get default rclone configuration for a provider type.
**Required Role:** VIEWER or higher
This endpoint shows which default settings will be automatically applied
when you create a Task using locations of this provider type. These defaults
improve compatibility and performance but can be overridden by providing your
own rclone_config values in the Task.
Use this endpoint to:
- See what defaults are applied for a provider before creating a task
- Understand which settings you can override in Task.rclone_config
- Learn about provider-specific optimizations
Returns a dictionary of default rclone config settings for the provider.
Returns an empty object if the provider has no special defaults.
Example responses:
```json
// S3_COMPATIBLE
{"s3_no_check_bucket": true}
// B2
{"transfers": 32, "b2_hard_delete": false, "fast_list": true}
// S3
{}
```
# List Locations
Source: https://docs.dataraven.io/api-reference/locations/list-locations
GET /v1/teams/{team_id}/locations
Get all locations for a team with optional filtering, sorting, and pagination.
**Required Role:** VIEWER or higher
Query parameters:
- location_type: Filter by provider type (s3, azure_blob, gcs, etc.)
- q: Search by name (case-insensitive partial match)
- sort_by: Field to sort by (name, created_at, last_verified_at)
- sort_order: Sort direction (asc, desc). Default: desc
- page: Page number (default: 1)
- limit: Items per page (default: 50, max: 100)
Returns paginated list of locations with metadata.
# List Location Verifications
Source: https://docs.dataraven.io/api-reference/locations/list-verifications
GET /v1/teams/{team_id}/locations/{location_id}/verifications
Get paginated verification history for a location (newest first).
**Required Role:** VIEWER or higher
Each record captures what rclone returned at a point in time — bucket
existence, permission grid (list/read/write/delete), region match,
and structured error codes on failure. Records cascade-delete with
their parent location.
Attribution (who triggered, when, from where) lives in the audit_logs
feed — this endpoint only returns the verification result payload.
# Update Location
Source: https://docs.dataraven.io/api-reference/locations/update-location
PATCH /v1/teams/{team_id}/locations/{location_id}
Update location properties.
**Required Role:** ADMIN or OWNER
**Updatable fields:**
- name
- description
- bucket_name
- region
- endpoint_url
- secret_id (must have same type as location_type)
**Immutable field:**
- location_type (cannot be changed after creation)
**Warning:** Changing bucket_name, region, endpoint_url, or secret_id will affect
all tasks using this location. Verify your changes before updating locations
used by active tasks.
# Verify Location
Source: https://docs.dataraven.io/api-reference/locations/verify-location
POST /v1/teams/{team_id}/locations/{location_id}/verify
Test connection to cloud storage location.
**Required Role:** OPERATOR or higher (useful for troubleshooting)
This endpoint runs a live rclone probe against the bucket, persists the
result to the verification history, and returns that newly-written row.
The response body is identical in shape to the records returned from
``GET /verifications``, so clients can use a single render path.
**Success response (200 OK):**
```json
{
"id": "...",
"location_id": "...",
"verified": true,
"bucket_exists": true,
"can_list": true,
"can_read": true,
"can_write": true,
"can_delete": false,
"region": "us-east-1",
"error_code": null,
"error_message": null,
"vault_field_name": null,
"vault_reference": null,
"vault_type": null,
"created_at": "2026-04-11T02:00:00Z"
}
```
Three-state permissions: ``true`` = probe succeeded, ``false`` = probe
failed, ``null`` = probe not run (e.g. empty bucket, delete skipped
because write failed).
**Client error (400 Bad Request):** Returned for user/configuration
errors (invalid credentials, bucket not found, access denied). The body
shape is the same as the success response, with ``verified=false`` and
``error_code`` / ``error_message`` populated.
**External vault resolution error (400 Bad Request):** When 1Password /
Doppler / Infisical secret resolution fails, ``error_code`` is one of
``VAULT_FIELD_NOT_FOUND``, ``VAULT_ITEM_NOT_FOUND``, ``VAULT_ACCESS_DENIED``,
``VAULT_INVALID_REFERENCE``, ``VAULT_CONNECTION_ERROR``,
``VAULT_SDK_NOT_INSTALLED``, or ``VAULT_UNKNOWN``. The ``vault_field_name``,
``vault_reference``, and ``vault_type`` fields identify exactly which
secret failed to resolve.
**Server error (500 Internal Server Error):** Returned only for truly
unexpected bugs. Expected failure modes (vault errors, connector build
errors, rclone errors) are caught inside the service layer, persisted to
the verification history, and returned as 400 responses with a normal
body shape.
# Bulk Create Notifications
Source: https://docs.dataraven.io/api-reference/notifications/bulk-create-notifications
POST /v1/teams/{team_id}/notifications/bulk
Bulk create notification configs for multiple event types with a single Apprise URL.
**Required Role:** ADMIN or OWNER
Creates one notification config per event type, all sharing the same Apprise URL.
Default message templates are applied automatically. Configs can be individually
edited after creation for custom filters or templates.
**Tier Limits:**
- Free Tier: Maximum 1 notification configuration total
- Pro Tier: Maximum 25 notification configurations total
# Create Notification
Source: https://docs.dataraven.io/api-reference/notifications/create-notification
POST /v1/teams/{team_id}/notifications
Create a new notification configuration.
**Required Role:** ADMIN or OWNER
**Tier Limits:**
- Free Tier: Maximum 1 notification configuration
- Pro Tier: Maximum 25 notification configurations
Supports event-driven notifications for:
- Task execution events (started, completed, failed, status changed)
- Task events (created, updated, deleted, status changed)
- Secret events (created, updated, deleted)
- Location events (created, updated, deleted)
- Usage events (threshold reached - requires `threshold_gb` in event_filters)
**Usage Threshold Example:**
```json
{
"name": "500GB Alert",
"event_type": "usage_threshold_reached",
"event_filters": {"threshold_gb": 500},
"apprise_url": "slack://token"
}
```
# Delete Notification
Source: https://docs.dataraven.io/api-reference/notifications/delete-notification
DELETE /v1/teams/{team_id}/notifications/{notification_id}
Delete a notification configuration.
**Required Role:** ADMIN or OWNER
# Get Notification
Source: https://docs.dataraven.io/api-reference/notifications/get-notification
GET /v1/teams/{team_id}/notifications/{notification_id}
Get a specific notification configuration.
**Required Role:** VIEWER or higher
# List Notifications
Source: https://docs.dataraven.io/api-reference/notifications/list-notifications
GET /v1/teams/{team_id}/notifications
List all notification configurations for a team.
**Required Role:** VIEWER or higher
**Query Parameters:**
- `q`: Search by name (case-insensitive partial match)
- `event_type`: Filter by event type
- `is_enabled`: Filter by enabled status
- `sort_by`: Field to sort by (name, event_type, created_at)
- `sort_order`: Sort direction (asc, desc). Default: desc
- `page`: Page number (default: 1)
- `limit`: Items per page (default: 50, max: 100)
# Test Notification
Source: https://docs.dataraven.io/api-reference/notifications/test-notification
POST /v1/teams/{team_id}/notifications/{notification_id}/test
Send a test notification to verify configuration.
**Required Role:** OPERATOR or higher
# Update Notification
Source: https://docs.dataraven.io/api-reference/notifications/update-notification
PATCH /v1/teams/{team_id}/notifications/{notification_id}
Update a notification configuration.
**Required Role:** ADMIN or OWNER
# Confirm Rclone Import
Source: https://docs.dataraven.io/api-reference/rclone-import/confirm-rclone-import
POST /v1/teams/{team_id}/rclone-import/confirm
Confirm and execute the rclone.conf import, creating secrets and locations.
**Required Role:** ADMIN or OWNER
This is step 2 of the import workflow. Send the confirmed list of remotes
(with any user modifications like name changes or bucket names) along with
the actual credential values.
For each remote, this endpoint creates:
1. A **Secret** with credentials stored in AWS SSM Parameter Store
2. A **Location** linked to that secret
Remotes are processed independently - if one fails, others still succeed.
Request body:
```json
{
"remotes": [
{
"name": "My S3 Remote",
"location_type": "s3",
"auth_method": "s3_access_key",
"region": "us-east-1",
"bucket_name": "my-bucket",
"credentials": {
"access_key_id": "AKIA...",
"secret_access_key": "wJal..."
}
}
]
}
```
Returns created resource pairs and any errors encountered.
# Parse Rclone Config
Source: https://docs.dataraven.io/api-reference/rclone-import/parse-rclone-config
POST /v1/teams/{team_id}/rclone-import/parse
Parse an rclone.conf file and return a preview of discovered remotes.
**Required Role:** ADMIN or OWNER
This is step 1 of the import workflow. Send the raw rclone.conf content
and receive a breakdown of:
- **remotes**: Successfully parsed remotes with their mapped types
- **errors**: Remotes that couldn't be parsed (missing fields, unknown types)
- **skipped**: Recognized but unsupported provider types (e.g., Google Drive, Dropbox)
Request body:
```json
{
"content": "[MyS3Remote]\ntype = s3\nprovider = AWS\naccess_key_id = AKIA...\n..."
}
```
The response includes metadata about each remote but NOT the actual credentials.
Credential values are only sent in the confirm step.
# Create Secret
Source: https://docs.dataraven.io/api-reference/secrets/create-secret
POST /v1/teams/{team_id}/secrets
Create a new secret with credentials stored in Vault or external vault.
**Required Role:** ADMIN or OWNER
**Tier Limits:**
- Free Tier: Maximum 5 secrets
- Pro Tier: Maximum 100 secrets
Secrets can be created in two ways:
**Option 1: Direct credentials** - Store credentials in AWS SSM Parameter Store
Request body should contain:
- name: Secret name (must be unique within the team)
- description: Optional description
- secret_type: Type of credentials (s3, azure_blob, gcs, etc.)
- auth_method: Authentication method for the provider
- credentials: Provider-specific credentials dict
**Option 2: External vault** - Reference secrets in 1Password, Doppler, or Infisical
Request body should contain:
- name: Secret name (must be unique within the team)
- description: Optional description
- secret_type: Type of credentials (s3, azure_blob, gcs, etc.)
- auth_method: Authentication method for the provider
- vault_connection_id: ID of the vault connection to use
- field_mappings: List of {field_name, reference} objects
Example for S3 (direct credentials):
```json
{
"name": "AWS Production Access",
"description": "S3 access for prod environment",
"secret_type": "s3",
"auth_method": "s3_access_key",
"credentials": {
"access_key_id": "AKIAIOSFODNN7EXAMPLE",
"secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}
}
```
Example for S3 (1Password external vault):
```json
{
"name": "AWS Production Access",
"description": "S3 access via 1Password",
"secret_type": "s3",
"auth_method": "s3_access_key",
"vault_connection_id": "uuid-of-1password-connection",
"field_mappings": [
{"field_name": "access_key_id", "reference": "op://DevOps/AWS-Prod/access_key_id"},
{"field_name": "secret_access_key", "reference": "op://DevOps/AWS-Prod/secret_access_key"}
]
}
```
Example for S3 (Doppler external vault):
```json
{
"name": "AWS Production Access",
"description": "S3 access via Doppler",
"secret_type": "s3",
"auth_method": "s3_access_key",
"vault_connection_id": "uuid-of-doppler-connection",
"field_mappings": [
{"field_name": "access_key_id", "reference": "AWS_ACCESS_KEY_ID"},
{"field_name": "secret_access_key", "reference": "AWS_SECRET_ACCESS_KEY"}
]
}
```
Example for S3 (Infisical external vault):
```json
{
"name": "AWS Production Access",
"description": "S3 access via Infisical",
"secret_type": "s3",
"auth_method": "s3_access_key",
"vault_connection_id": "uuid-of-infisical-connection",
"field_mappings": [
{"field_name": "access_key_id", "reference": "AWS_ACCESS_KEY_ID"},
{"field_name": "secret_access_key", "reference": "AWS_SECRET_ACCESS_KEY"}
]
}
```
Returns:
- 201 Created on success
- 400 Bad Request for validation errors
- 409 Conflict if a secret with the same name already exists in this team
The response will NOT include credentials.
# Delete Secret
Source: https://docs.dataraven.io/api-reference/secrets/delete-secret
DELETE /v1/teams/{team_id}/secrets/{secret_id}
Delete a secret.
**Required Role:** ADMIN or OWNER
**Validation:** Cannot delete if the secret is used by any locations.
You must delete or reassign the locations first.
Returns:
- 204 No Content on success
- 404 Not Found if secret doesn't exist
- 409 Conflict if secret is in use by locations
# Get Secret
Source: https://docs.dataraven.io/api-reference/secrets/get-secret
GET /v1/teams/{team_id}/secrets/{secret_id}
Get detailed information about a secret.
**Required Role:** VIEWER or higher
Returns:
- Secret metadata (no credentials)
- Number of locations using this secret
- List of locations using this secret
**Security Note:** Credentials are NEVER returned.
# List Secrets
Source: https://docs.dataraven.io/api-reference/secrets/list-secrets
GET /v1/teams/{team_id}/secrets
Get all secrets for a team with pagination.
**Required Role:** VIEWER or higher
**Security Note:** Credentials are NEVER returned in this endpoint.
Only metadata (name, description, type, timestamps) is returned.
Query parameters:
- q: Search by name (case-insensitive partial match)
- secret_type: Filter by provider type (s3, azure_blob, gcs, r2, b2, wasabi, etc.)
- sort_by: Field to sort by (name, created_at)
- sort_order: Sort direction (asc, desc). Default: desc
- page: Page number (default: 1)
- limit: Items per page (default: 50, max: 100)
Returns paginated list of secrets with metadata.
# Update Secret
Source: https://docs.dataraven.io/api-reference/secrets/update-secret
PATCH /v1/teams/{team_id}/secrets/{secret_id}
Update secret properties and/or rotate credentials.
**Required Role:** ADMIN or OWNER
**Updatable fields:**
- name: Rename the secret
- description: Update description
- secret_type: Change provider type (validates linked locations match)
- auth_method: Change auth mechanism (must be valid for secret_type)
- credentials: Rotate credentials (AWS SSM Parameter Store secrets only)
- field_mappings: Update vault references (external vault secrets only)
**Validation rules:**
- If changing secret_type, all linked locations must have matching location_type
- If changing auth_method, must be valid for the (new or existing) secret_type
- External vault secrets cannot have credentials updated (manage in your vault)
- AWS SSM Parameter Store secrets cannot have field_mappings updated (use credentials)
- Cannot provide both credentials and field_mappings in the same request
**Immutable field:**
- vault_connection_id (cannot change vault provider after creation)
**Security Note:** Credentials are stored encrypted in Vault.
The response will NOT include the credentials.
# Billing Portal
Source: https://docs.dataraven.io/api-reference/subscriptions/billing-portal
POST /v1/teams/{team_id}/billing/portal
Get Polar customer portal URL for billing management.
**Required Role:** ADMIN or higher
Creates a customer session on Polar and returns the portal URL.
The frontend should open this URL in a new tab.
The portal allows users to:
- View and download invoices
- Update payment method
- Cancel or resubscribe
- Update billing information
Note: Only teams with a subscription (PRO tier) can access the billing portal.
# Get Subscription
Source: https://docs.dataraven.io/api-reference/subscriptions/get-subscription
GET /v1/teams/{team_id}/subscription
Get team's subscription details for both FREE and PRO tiers.
**Required Role:** VIEWER or higher
Returns comprehensive subscription information for any tier:
- FREE tier: Basic status with null subscription fields
- PRO tier: Full subscription details from local cache (updated via Polar webhooks)
Use this for displaying subscription status in settings/billing pages and dashboard widgets.
# Upgrade
Source: https://docs.dataraven.io/api-reference/subscriptions/upgrade
POST /v1/teams/{team_id}/billing/upgrade
Create a Polar checkout session for PRO tier upgrade.
**Required Role:** ADMIN or higher
Creates a unique checkout session URL on Polar for the team to upgrade
to PRO tier. The frontend should open this URL in a new tab/window.
Returns 409 Conflict if team already has an active subscription.
Note: The checkout URL is unique per request and can only be used once.
# Verify Subscription
Source: https://docs.dataraven.io/api-reference/subscriptions/verify-subscription
POST /v1/teams/{team_id}/subscription/verify
Verify and sync subscription status from Polar API.
**Required Role:** VIEWER or higher
This endpoint fetches the current customer state directly from Polar's API
and syncs it to the local database. Use this after checkout to ensure the
subscription status is up-to-date (in case webhooks are delayed).
Returns the fresh subscription status after syncing.
# Archive Task
Source: https://docs.dataraven.io/api-reference/tasks/archive-task
POST /v1/teams/{team_id}/tasks/{task_id}/archive
Archive a task (soft delete).
**Required Role:** OPERATOR, ADMIN, or OWNER
Archived tasks:
- Stop running on schedule
- Are hidden from default task lists (filter with status=archived to view)
- Retain execution history
- CANNOT be enabled (must create a new task)
**Use Case:** Deprecate old tasks while preserving audit trail.
**Status Transitions:**
- ENABLED → ARCHIVED ✅
- DISABLED → ARCHIVED ✅
- ARCHIVED → ARCHIVED ❌ (error)
# Create Task
Source: https://docs.dataraven.io/api-reference/tasks/create-task
POST /v1/teams/{team_id}/tasks
Create a new task definition.
**Required Role:** OPERATOR, ADMIN, or OWNER
**Tier Limits:**
- Free Tier: Maximum 2 tasks
- Pro Tier: Maximum 50 tasks
A task is a blueprint for a transfer operation between two locations.
Tasks can be executed on-demand or run on a schedule.
Example request body:
```json
{
"name": "S3 to R2 Daily Backup",
"description": "Daily sync from production S3 to Cloudflare R2 backup",
"task_type": "sync",
"source_location_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"destination_location_id": "7b9c12e8-4a3f-4d5e-8c6b-9f2a1e3d4c5b",
"source_path": "/prod/data",
"destination_path": "/backup",
"rclone_config": {
"bwlimit": "100M",
"transfers": 4,
"checkers": 8,
"filters": {
"filter": ["+ *.jpg", "+ *.png", "- *.bak", "- *.tmp"],
"max_age": "30d"
},
"checksum": true,
"fast_list": true
},
"is_scheduled": true,
"schedule_cron": "0 2 * * *"
}
```
# Delete Task
Source: https://docs.dataraven.io/api-reference/tasks/delete-task
DELETE /v1/teams/{team_id}/tasks/{task_id}
Permanently delete a task.
**Required Role:** ADMIN or OWNER
**⚠️ WARNING: Tasks with execution history CANNOT be deleted!**
**Important:**
- Tasks with execution history return 400 Bad Request
- Use archive instead to preserve audit trail: `POST /tasks/{task_id}/archive`
- Only tasks with NO executions can be deleted
- Deletion cascades to remove notifications (if task never ran)
**Why this restriction:**
Execution history contains immutable audit records required for compliance.
Deleting executions would destroy historical usage metrics and transfer logs.
**Deleting a task:**
- Permanently removes the task definition
- Cascades to delete notifications (if any)
- CANNOT be undone
**Validation:**
- Task must exist and belong to the team
- Task must NOT have any execution history (enforced)
**Recommendation:** Use archive instead of delete to preserve history.
# Disable Task
Source: https://docs.dataraven.io/api-reference/tasks/disable-task
POST /v1/teams/{team_id}/tasks/{task_id}/disable
Disable a task to prevent all executions.
**Required Role:** OPERATOR, ADMIN, or OWNER
Disabled tasks:
- Will NOT run on schedule
- Cannot be executed manually
- Remain visible in task lists
- Can be enabled anytime
**Status Transitions:**
- ENABLED → DISABLED ✅
- DISABLED → DISABLED ❌ (error)
- ARCHIVED → DISABLED ❌ (cannot disable archived tasks)
# Enable Task
Source: https://docs.dataraven.io/api-reference/tasks/enable-task
POST /v1/teams/{team_id}/tasks/{task_id}/enable
Enable a disabled task to allow executions.
**Required Role:** OPERATOR, ADMIN, or OWNER
**Status Transitions:**
- DISABLED → ENABLED ✅
- ENABLED → ENABLED ❌ (error)
- ARCHIVED → ENABLED ❌ (cannot enable archived tasks)
# Get Task
Source: https://docs.dataraven.io/api-reference/tasks/get-task
GET /v1/teams/{team_id}/tasks/{task_id}
Get detailed task information including execution statistics.
**Required Role:** VIEWER or higher
Returns full task configuration plus aggregated execution statistics
(total runs, success/failure counts, last run time, etc.).
# List Tasks
Source: https://docs.dataraven.io/api-reference/tasks/list-tasks
GET /v1/teams/{team_id}/tasks
List all tasks for a team with filtering and pagination.
**Required Role:** VIEWER or higher
**Query Parameters:**
- `q`: Search by name (case-insensitive partial match)
- `status_filter`: Filter by task status (enabled, disabled, archived)
- `task_type`: Filter by task type (copy, sync)
- `source_location_id`: Filter by source location UUID
- `destination_location_id`: Filter by destination location UUID
- `is_scheduled`: Filter by scheduled (true) or manual (false) tasks
- `location_type`: Filter by provider type (matches source or destination)
- `sort_by`: Field to sort by (name, status, created_at, updated_at)
- `sort_order`: Sort direction (asc, desc). Default: desc
- `page`: Page number (default: 1)
- `limit`: Items per page (default: 50, max: 100)
**Example Response:**
```json
{
"tasks": [...],
"total": 42,
"page": 1,
"limit": 50,
"pages": 1
}
```
# Update Task
Source: https://docs.dataraven.io/api-reference/tasks/update-task
PATCH /v1/teams/{team_id}/tasks/{task_id}
Update a task configuration.
**Required Role:** OPERATOR, ADMIN, or OWNER
All fields are optional - only provided fields will be updated.
Example request body:
```json
{
"name": "Updated Task Name",
"rclone_config": {
"bwlimit": "200M",
"transfers": 16
},
"is_scheduled": false
}
```
**Note:** Updating a task does NOT affect currently running executions.
Changes apply to future executions only.
# Create Team
Source: https://docs.dataraven.io/api-reference/teams/create-team
POST /v1/teams
Create a new team.
**Global Limit:** Maximum 5 teams per user (abuse prevention).
The creating user automatically becomes the team owner.
New teams start on the FREE tier. Upgrade each team individually
to PRO for higher resource limits (tasks, locations, secrets, etc.).
# Delete Team
Source: https://docs.dataraven.io/api-reference/teams/delete-team
DELETE /v1/teams/{team_id}
Delete a team and all associated data.
**Required Role:** OWNER only
**Prerequisites:** All child resources must be deleted first:
1. Delete all tasks (this also removes executions and usage metrics)
2. Delete all locations
3. Delete all secrets
Returns 409 Conflict with details if resources still exist.
# Get Team
Source: https://docs.dataraven.io/api-reference/teams/get-team
GET /v1/teams/{team_id}
Get detailed information about a specific team.
**Required Role:** VIEWER or higher
Includes full member list with roles and pending invitations.
# List Teams
Source: https://docs.dataraven.io/api-reference/teams/list-teams
GET /v1/teams
Get all teams the authenticated user belongs to.
Returns teams with:
- Member count
- User's role in each team
- Pagination metadata
Query parameters:
- page: Page number (default: 1)
- limit: Items per page (default: 50, max: 100)
# Remove Member
Source: https://docs.dataraven.io/api-reference/teams/remove-member
DELETE /v1/teams/{team_id}/members/{user_id}
Remove a team member.
**Required Role:** ADMIN or OWNER
Restrictions:
- Cannot remove yourself (use leave team endpoint instead)
- Cannot remove the last owner
# Update Member Role
Source: https://docs.dataraven.io/api-reference/teams/update-member-role
PATCH /v1/teams/{team_id}/members/{user_id}
Update a team member's role.
**Required Role:** ADMIN or OWNER
Restrictions:
- Cannot change your own role
- Cannot change owner role (unless you're also an owner)
- Cannot set role to owner (use ownership transfer instead)
# Update Team
Source: https://docs.dataraven.io/api-reference/teams/update-team
PATCH /v1/teams/{team_id}
Update team metadata.
**Required Role:** ADMIN or OWNER
# Get Quota Usage
Source: https://docs.dataraven.io/api-reference/usage/get-quota-usage
GET /v1/teams/{team_id}/usage/quotas
Get resource quota usage (current counts vs tier limits).
**Required Role:** VIEWER or higher
Returns current usage counts and tier limits for each quota-limited resource
(tasks, secrets, locations, alerts, vault connections, API keys, team members).
# Usage Analytics
Source: https://docs.dataraven.io/api-reference/usage/get-usage
GET /v1/teams/{team_id}/usage/analytics
Comprehensive analytics endpoint.
**Required Role:** VIEWER or higher
Returns aggregate metrics, time-series data, top errors, and per-task breakdown
for the selected date range.
# Create Vault Connection
Source: https://docs.dataraven.io/api-reference/vault-connections/create-connection
POST /v1/teams/{team_id}/vault-connections
Create a new external vault connection.
**Required Role:** ADMIN or OWNER
**Tier Limits:**
- Free Tier: Maximum 1 vault connection
- Pro Tier: Maximum 10 vault connections
Connects your team to an external vault provider (1Password, Doppler, or Infisical).
Credentials are stored encrypted and never returned in API responses.
**1Password:**
- Requires a service account token (starts with "ops_")
- No additional configuration needed
**Doppler:**
- Requires a service token (starts with "dp.st.")
- Requires project and config
**Infisical (Universal Auth / Machine Identity):**
- Create a Machine Identity in Project Settings -> Access Control -> Machine Identities
- client_id: UUID from Machine Identity
- client_secret: Secret string from Machine Identity
- project_id: Project ID (UUID) from Project Settings -> General
- environment: Environment slug (e.g., "dev", "staging", "prod", "sandbox")
- Optional: secret_path (defaults to "/")
Example for 1Password:
```json
{
"name": "1Password Production",
"vault_type": "onepassword",
"access_token": "ops_xxx..."
}
```
Example for Doppler:
```json
{
"name": "Doppler Production",
"vault_type": "doppler",
"access_token": "dp.st.xxx...",
"project": "my-project",
"config": "production"
}
```
Example for Infisical:
```json
{
"name": "Infisical Production",
"vault_type": "infisical",
"client_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"client_secret": "st.abc123xyz...",
"project_id": "f1e2d3c4-b5a6-7890-abcd-ef1234567890",
"environment": "prod",
"secret_path": "/aws"
}
```
# Delete Vault Connection
Source: https://docs.dataraven.io/api-reference/vault-connections/delete-connection
DELETE /v1/teams/{team_id}/vault-connections/{connection_id}
Delete a vault connection.
**Required Role:** ADMIN or OWNER
**Validation:** Cannot delete if the connection is used by any secrets.
You must delete or migrate the secrets first.
Returns:
- 204 No Content on success
- 404 Not Found if connection doesn't exist
- 409 Conflict if connection is in use by secrets
# Get Vault Connection
Source: https://docs.dataraven.io/api-reference/vault-connections/get-connection
GET /v1/teams/{team_id}/vault-connections/{connection_id}
Get detailed information about a vault connection.
**Required Role:** VIEWER or higher
Returns:
- Connection configuration (excluding access token)
- Number of secrets using this connection
- Last verification timestamp
# List Vault Connections
Source: https://docs.dataraven.io/api-reference/vault-connections/list-connections
GET /v1/teams/{team_id}/vault-connections
List all vault connections for the team.
**Required Role:** VIEWER or higher
Returns all external vault connections configured for the team.
Access tokens are never included in the response.
# Rotate Token
Source: https://docs.dataraven.io/api-reference/vault-connections/rotate-token
POST /v1/teams/{team_id}/vault-connections/{connection_id}/rotate-token
Rotate the access token for a vault connection.
**Required Role:** ADMIN or OWNER
**Note:** Only works for 1Password and Doppler connections.
For Infisical, delete and recreate the connection with new credentials.
Use this to update the access token without recreating the connection.
All secrets using this connection will use the new token.
# Test Vault Connection
Source: https://docs.dataraven.io/api-reference/vault-connections/test-connection
POST /v1/teams/{team_id}/vault-connections/{connection_id}/test
Test vault connection by verifying access.
**Required Role:** OPERATOR or higher
Attempts to connect to the external vault and list accessible vaults/secrets.
Updates last_verified_at on success.
Returns:
- success: Whether the connection works
- message: Human-readable status
- vaults_accessible: Number of vaults (1Password)
- secrets_accessible: Number of secrets (Doppler/Infisical)
# Update Vault Connection
Source: https://docs.dataraven.io/api-reference/vault-connections/update-connection
PATCH /v1/teams/{team_id}/vault-connections/{connection_id}
Update vault connection properties.
**Required Role:** ADMIN or OWNER
Updatable fields:
- name
- description
To rotate the access token, use the /rotate-token endpoint.
# Connecting Storage Backends
Source: https://docs.dataraven.io/connecting-storage
Add and configure cloud storage locations as sources and destinations for transfer tasks.
Locations define cloud storage endpoints — buckets and containers — used as sources or destinations
in transfer tasks. Each location references a [Secret](/secrets) for credentials, and
connects to a specific bucket in a specific region.
RClone configuration options (transfers, chunk sizes, etc.) are set at the **Task** level, not on
locations. Locations define *where* to connect — tasks define *how*.
## Supported Providers
Each provider has its own page covering credentials, location setup, quirks, and rclone import
behavior:
Amazon Simple Storage Service
Azure Blob containers
Cloud storage with generous free tier
S3-compatible object storage
DigitalOcean Spaces Object Storage
Fastly's S3-compatible object storage
S3-compatible object storage on Filecoin
S3-compatible object storage, single global endpoint
GCS buckets
Hetzner Object Storage
EU-sovereign S3-compatible object storage
MEGA's S3-compatible object storage
Oracle S3-compatible endpoint
Rabata Object Storage
Railway-provided S3 buckets
Globally distributed S3-compatible storage
Hot cloud storage, S3-compatible
Any S3-compatible provider
## Creating a Location
Every location needs credentials. If you haven't already, [create a secret](/secrets) for your cloud provider. The secret's `secret_type` must match the location's `location_type`.
Go to **Locations** in your team dashboard and click **Add Location**.
Fill in the required fields:
| Field | Description | Required |
| ------------------ | ------------------------------------------------------ | ------------------ |
| **name** | Descriptive unique name (e.g., "Production S3 Bucket") | Yes |
| **description** | Optional notes about this location | No |
| **location\_type** | Cloud provider — must match the secret's `secret_type` | Yes |
| **secret\_id** | Reference to the credentials secret | Yes |
| **bucket\_name** | Bucket or container name | Yes |
| **region** | Provider region (e.g., `us-east-1`, `westus2`) | Varies |
| **endpoint\_url** | Custom endpoint URL | S3-compatible only |
Region and endpoint requirements vary — see the [provider comparison](#provider-specific-configuration) below, or jump straight to [your provider's page](#supported-providers) for the full setup.
After creation, DataRaven auto-redirects to the detail page and runs a [connection verification](#connection-verification) automatically.
## Connection Verification
DataRaven verifies that a location is reachable and properly configured by testing the actual
connection.
Verification checks:
* **bucket\_exists** — confirms the bucket/container is accessible
* **permissions** — tests list, read, write, and delete operations
### Verification Status
| Status | Meaning |
| ---------------- | ----------------------------------------------- |
| **Verified** | Successfully tested within the last 24 hours |
| **Stale** | Last successful test was more than 24 hours ago |
| **Never tested** | Verification has not been run |
Verification runs automatically after creation. Re-run it anytime from the location detail page or
via the API to confirm connectivity — especially after rotating credentials or changing bucket
settings.
For locations using [external vault (BYOV)](/vault-integration) credentials, verification
resolves the vault references to test the connection. If resolution fails, you'll receive specific
error codes: `VAULT_FIELD_NOT_FOUND`, `VAULT_ITEM_NOT_FOUND`, `VAULT_ACCESS_DENIED`, etc.
## Provider-Specific Configuration
Every provider needs the same three things — a matching [secret](/secrets), a bucket, and enough
information to reach the endpoint — but region and endpoint requirements differ. The comparison
below shows the shape at a glance; each provider's page has the full setup, including
credentials, worked examples, and quirks. DataRaven checks these requirements when you save the
location, so a missing endpoint or region is rejected then rather than when a transfer first runs.
| Provider | Region | Endpoint URL | Notable |
| --------------------------------------------------------- | ----------------------------- | ----------------------------------- | -------------------------------------------------------- |
| [AWS S3](/providers/aws-s3) | Required | Not needed | |
| [Azure Blob Storage](/providers/azure-blob) | Required | Not needed | SAS URL or account key auth |
| [Backblaze B2](/providers/backblaze-b2) | Optional | Not needed | Optimized transfer defaults |
| [Cloudflare R2](/providers/cloudflare-r2) | Not needed | Required — account-specific | Account ID lives in the endpoint |
| [DigitalOcean Spaces](/providers/digitalocean-spaces) | Not needed | Required — embeds region | |
| [Fastly Object Storage](/providers/fastly-object-storage) | Required — Fastly's own names | Derived from region | Full-access keys only; reserved bucket prefixes |
| [Fil One](/providers/fil-one) | Required | Derived from region | Access keys are region-scoped |
| [Filebase](/providers/filebase) | Not needed | Optional — global default | Bucket names are globally unique; no dots |
| [Google Cloud Storage](/providers/google-cloud-storage) | Optional | Not needed | Service account JSON auth |
| [Hetzner](/providers/hetzner) | Not needed | Required — embeds region | |
| [Impossible Cloud](/providers/impossible-cloud) | Required | Derived from region | Region must be the bucket's own; bucket names are global |
| [MEGA S4](/providers/mega-s4) | Required — picks endpoint | Derived from region | Global buckets; strict object-key and bucket-name rules |
| [Oracle Object Storage](/providers/oracle-object-storage) | Required | Required | |
| [Rabata](/providers/rabata) | Required | Required — embeds region | |
| [Railway](/providers/railway) | Varies | Required — Railway-provided | |
| [Tigris](/providers/tigris) | Not needed | Required — `https://t3.storage.dev` | |
| [Wasabi](/providers/wasabi) | Required | Required — embeds region | |
| [S3 Compatible](/providers/s3-compatible) | Varies | Required | Catch-all for unlisted providers |
## Provider Defaults
DataRaven applies sensible default rclone configuration per provider at the task level. Each
provider's page lists its defaults under **At a Glance**; you can also query them via the API:
```json Backblaze B2 theme={"theme":{"light":"github-light","dark":"poimandres"}}
{
"transfers": 32,
"b2_hard_delete": false,
"fast_list": true
}
```
```json S3-Compatible Providers theme={"theme":{"light":"github-light","dark":"poimandres"}}
{
"s3_no_check_bucket": true
}
```
Provider defaults are starting points. You can override any of these settings in the task's rclone
configuration. They are resolved each time a task runs rather than copied into it, so when a
provider's defaults change, every task that uses that provider picks up the change on its next run.
## Updating Locations
You can update the following fields on an existing location:
* `name`, `description`
* `bucket_name`, `region`, `endpoint_url`
* `secret_id` (new secret must have the same `secret_type` as the location's `location_type`)
**`location_type` is immutable.** You cannot change a location's provider after creation. Create a
new location instead.
Changing `bucket_name`, `region`, `endpoint_url`, or `secret_id` affects **all tasks** currently
using this location. Verify the connection after making changes.
## Deletion
A location **cannot be deleted** while it is referenced by any tasks. Attempting to delete returns a
`409 Conflict` error.
To delete a location:
1. Navigate to the location detail page to see linked tasks
2. Delete or reassign all tasks using this location
3. Delete the location
## Linked Tasks
The location detail page shows all tasks using this location — grouped by whether it's the
**source** or **destination**. Click any task to navigate directly to its configuration.
## Tier Limits
| Plan | Locations |
| -------- | --------- |
| **Free** | 4 |
| **Pro** | 100 |
Need more? [Upgrade your plan](https://app.dataraven.io/#/subscription) from the billing page.
## Required Roles
| Action | Minimum Role |
| ------------------------ | ------------ |
| View locations | VIEWER |
| Create / Update / Delete | ADMIN |
| Verify connection | OPERATOR |
The OPERATOR role can run verification without full admin access — useful for on-call engineers
troubleshooting connectivity issues.
# Core Concepts
Source: https://docs.dataraven.io/core-concepts
Understand the DataRaven data model
DataRaven organizes data movement around a few key primitives.
## Teams
Everything in DataRaven lives inside a **Team**. Teams are collaborative workspaces with:
* **Role-based access** — Owner, Admin, Member roles
* **Shared resources** — Locations, tasks, and vault connections are scoped to a team
* **Independent billing** — Each team has its own subscription (Free or Pro)
## Locations
A **Location** is a configured connection to a storage backend. Think of it as a named pointer to a
bucket, container, or directory on a remote provider.
Locations support 40+ backends including S3-compatible stores, GCS, Azure Blob, and more.
## Tasks
A **Task** defines a data movement operation between two locations. Tasks specify:
* **Source and destination** locations
* **Transfer mode** — copy, sync, or move
* **Filters** — include/exclude patterns
* **Schedule** — optional cron expression for recurring runs
* **Bandwidth limits** and other rclone flags
## Executions
An **Execution** is a single run of a task. Each execution captures:
* Start/end time and duration
* Bytes transferred and files processed
* Status (running, completed, failed, stopped)
* Full rclone logs (downloadable or streamable)
## Vault Connections
DataRaven's **zero-knowledge credential architecture** means your cloud credentials are never retained
on our systems. Instead, you connect a **Vault** — a secrets manager you already trust:
* **1Password** — Connect via Service Account
* **Doppler** — Project-level access
* **Infisical** — Machine identity tokens
Locations reference secrets stored in your vault. DataRaven resolves them at transfer time and
discards them immediately after.
## Secrets
**Secrets** are references to credentials stored in your connected vault. They map a secret name in
DataRaven to a path/key in your vault provider.
## Notifications
Configure **Notifications** to get real-time alerts when configurations change & task execution events. Supports webhooks and
integrations for monitoring your data pipeline.
# Creating & Managing Transfer Tasks
Source: https://docs.dataraven.io/creating-tasks
Complete guide to DataRaven tasks — the core resource for data transfer operations
Tasks are **the core resource** in DataRaven. They are blueprints for data transfer operations
between two locations, defining source → destination mapping, transfer modes, rclone options, file
filters, and optional scheduling. Tasks can be executed on-demand or automatically on a cron
schedule.
## Transfer Modes
DataRaven supports two fundamental transfer modes, each with different behaviors and safety
implications:
**Copy** (`copy`) transfers files from source to destination without deleting anything at the
destination. This is the **safest option** for one-way backups and data distribution. - Files
are copied from source to destination - Existing files at destination are preserved unless
overwritten by newer source files - Safe for backups — no data loss risk at destination 📖
[RClone Copy Documentation](https://rclone.org/commands/rclone_copy/)
**Sync** (`sync`) makes the destination mirror the source exactly. This means files that exist
on the destination but not on the source will be **deleted**.
**Sync mode deletes files on the destination** that don't exist on the source. Use with
extreme caution. Always test with dry runs first.
**Delete limit.** Every sync stops and fails if it would delete more than 1,000 objects
at the destination. This protects the destination when the source listing comes back
incomplete. Set `max_delete` on the task to change the limit, or to `-1` to remove it.
See [File Handling](#rclone-configuration) below.
* Destination becomes an exact mirror of source - Extra files at destination are permanently
deleted - Ideal for maintaining exact replicas - ⚠️ **High risk** — can cause data loss 📖
[RClone Sync Documentation](https://rclone.org/commands/rclone_sync/)
## Creating a Task
Before creating a task, ensure you have:
* At least 2 locations configured → [Connecting Storage Guide](/connecting-storage)
* Required secrets set up → [Secrets Management Guide](/secrets)
* Appropriate user role (OPERATOR or higher required)
Go to **Tasks → Create Task** in the DataRaven dashboard.
Configure the fundamental task properties:
**Required Fields:**
* **Name** (required, max 255 characters) — Human-readable task identifier
* **Task Type** — Choose `copy` or `sync` mode
* **Source Location** — Select from configured locations
* **Destination Location** — Select from configured locations (must be different from source)
**Optional Fields:**
* **Description** (max 2000 characters) — Document the task purpose
* **Source Path** — Specific path within source bucket/folder (defaults to root)
* **Destination Path** — Specific path within destination bucket/folder (defaults to root)
**Scheduling:**
* **Enable Scheduling** — Toggle scheduled execution
* **Cron Schedule** — 5-field UNIX cron expression or keywords
Source and destination locations **must be different**. This is validated by the system to prevent data corruption.
Configure performance, filtering, and transfer behavior. See [RClone
Configuration](#rclone-configuration) section below for detailed options.
Advanced options for comparison methods, transfer behavior, performance tuning, network settings, and provider-specific configurations.
## RClone Configuration
DataRaven provides extensive rclone configuration options to fine-tune transfer behavior,
performance, and filtering.
Control transfer performance and resource usage:
| Option | Type | Range | Description |
| ------------- | ------- | -------- | -------------------------------------------------- |
| `bwlimit` | string | - | Bandwidth limit with time-based scheduling support |
| `transfers` | integer | 1-64 | Number of parallel file transfers (default: 4) |
| `checkers` | integer | 1-64 | Number of parallel file checkers (default: 8) |
| `buffer_size` | string | - | In-memory buffer size per transfer |
| `max_backlog` | integer | 1-150000 | Max objects in sync backlog (default: 10000) |
**Bandwidth Limiting Examples:**
```
100M # Constant 100MB/s limit
10M:off # 10MB/s on weekdays, unlimited on weekends
Mon-00:00,512 12:00,1M # 512KB/s until noon, then 1MB/s on Mondays
Sun-20:00,off # Unlimited on Sunday after 8 PM
```
**Buffer Size Examples:**
* `16M` — 16 megabytes per transfer
* `32M` — 32 megabytes per transfer (good for high-latency connections)
Control which files are included or excluded from transfers. All filter options are nested under the
`filters` configuration key.
**Don't mix filter types!** Use either `filter` patterns OR the old-style `--include`/`--exclude`
flags, but never both together.
| Option | Type | Description |
| ---------- | ---------------- | ---------------------------------------------- |
| `filter` | array of strings | Include/exclude patterns with `+`/`-` prefixes |
| `max_age` | string | Only transfer files newer than this age |
| `min_age` | string | Only transfer files older than this age |
| `max_size` | string | Only transfer files smaller than this size |
| `min_size` | string | Only transfer files larger than this size |
**Filter Pattern Examples:**
```json theme={"theme":{"light":"github-light","dark":"poimandres"}}
{
"filters": {
"filter": [
"+ *.jpg", // Include all JPEG files
"+ *.png", // Include all PNG files
"- *.tmp", // Exclude temporary files
"+ important/**", // Include entire important directory
"- **" // Exclude everything else
]
}
}
```
**Age Filter Examples:**
* `24h` — Files from last 24 hours
* `7d` — Files from last 7 days
* `30d` — Files from last 30 days
**Size Filter Examples:**
* `100M` — Files under 100 megabytes
* `1G` — Files under 1 gigabyte
* `500K` — Files under 500 kilobytes
Configure how rclone determines if files need to be transferred:
| Option | Type | Description |
| ----------------- | ------- | ------------------------------------------------------------- |
| `checksum` | boolean | Verify file checksums (slower but safer) |
| `size_only` | boolean | Compare files by size only (faster, less accurate) |
| `ignore_size` | boolean | Don't compare file sizes |
| `ignore_checksum` | boolean | Skip checksum verification — compare by size and modtime only |
| `modify_window` | string | Time window for modification time comparison |
**Modify Window Examples:**
* `1s` — 1 second tolerance
* `5m` — 5 minute tolerance (useful for some cloud providers)
* `1h` — 1 hour tolerance
Use `checksum` for critical data where integrity is paramount. Use `size_only` for faster
transfers when you trust modification times aren't reliable.
Control how files are transferred and handled:
| Option | Type | Description |
| -------------------- | ------- | ------------------------------------------------- |
| `update` | boolean | Skip files that are newer on destination |
| `ignore_existing` | boolean | Skip all files that exist on destination |
| `immutable` | boolean | Fail if existing files have been modified |
| `use_server_modtime` | boolean | Use server-side modification times |
| `no_update_modtime` | boolean | Don't update modification times on destination |
| `no_traverse` | boolean | Don't scan destination directory (copy mode only) |
| `metadata` | boolean | Preserve file metadata when copying |
| `dry_run` | boolean | Preview transfer without actually moving data |
`no_traverse` can significantly speed up copy operations when you know the destination doesn't
have conflicting files.
Control file processing limits and behavior:
| Option | Type | Range | Description |
| ----------------- | ------- | ---------- | --------------------------------------------------------------------------------------------------------------------- |
| `max_depth` | integer | -1 to 100 | Directory recursion depth (-1 = unlimited, 0 = root only) |
| `max_transfer` | string | - | Total transfer size limit per execution |
| `max_delete` | integer | -1 or more | Sync only. Objects a sync may delete at the destination before it stops and fails (default: `1000`, `-1` = unlimited) |
| `max_delete_size` | string | - | Sync only. Total size of deletions at the destination after which the sync stops and fails (default: unlimited) |
**Max Transfer Examples:**
* `10G` — Stop after transferring 10 gigabytes
* `500M` — Stop after transferring 500 megabytes
* `1T` — Stop after transferring 1 terabyte
**Sync Delete Limit:**
A sync makes the destination match the source. If the source listing comes back short, because
a provider truncated the pages, a permission change hid a prefix, or an endpoint returned an
empty page under load, rclone cannot tell that from a real deletion at the source. Without a
limit it deletes the difference at the destination and reports success.
DataRaven therefore adds `--max-delete` to every sync. The default limit is 1,000 objects. When
a sync reaches the limit, rclone stops, the execution fails, and the error names the limit.
Deletions up to the limit have already happened; the rest have not. Dry runs apply the limit
too, so a dry run shows you whether a sync would trip it before anything is deleted.
* Set `max_delete` to the number of deletions you expect if your source legitimately removes
more than 1,000 objects between runs
* Set `max_delete` to `0` to allow a sync to copy and update but never delete
* Set `max_delete` to `-1` to remove the limit
* `max_delete_size` adds a second bound on the total size of deleted objects, for example `10G`
The limit is resolved when the task runs, not stored on the task. A task with no `max_delete`
always uses the current default.
Fine-tune network behavior and connection handling:
| Option | Type | Range | Description |
| ---------------------- | ------- | ------ | -------------------------------------------------- |
| `disable_http2` | boolean | - | Disable HTTP/2 for compatibility |
| `multi_thread_streams` | integer | 0-64 | Multi-threaded download streams (default: 4) |
| `tpslimit` | float | 0-1000 | HTTP transactions per second limit (0 = unlimited) |
| `tpslimit_burst` | integer | 0-1000 | Maximum burst for transaction rate limiting |
Special options that apply to specific cloud storage providers:
| Option | Type | Default | Applies To | Description |
| -------------------- | ------- | ---------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `s3_no_check_bucket` | boolean | `true` | All S3-compatible providers | Skip bucket existence check |
| `s3_upload_cutoff` | string | `200M` (rclone) | S3-family providers | File size above which uploads switch to multipart (max `5G`) |
| `s3_no_head` | boolean | `false` (rclone) | S3-family providers | Skip the post-upload HEAD check — saves one API call per object, but a failed upload can go undetected |
| `s3_list_version` | integer | `0` (rclone) | **Both** S3 remotes in the task | S3 listing API version — `0` auto, `1` ListObjects, `2` ListObjectsV2. On auto, rclone picks the version from the provider string. Most S3-compatible providers resolve to V1. |
| `b2_hard_delete` | boolean | `false` | B2 | Permanently delete vs hide files |
| `fast_list` | boolean | `true` | B2 and most S3-compatible providers | Use recursive directory listing |
Set `s3_list_version` only if both locations support the version. rclone applies one listing
API to **both** sides of the transfer, not only to the provider that you configure. Some
S3-compatible endpoints return all pages on one version, but fail on the other. Providers
that need a specific version, such as [Fil One](/providers/fil-one), set the value on the
remote. These providers need no task setting. [Filebase](/providers/filebase) stays on auto
deliberately — both versions are verified to return the same keys there.
Provider defaults are applied automatically based on your source and destination location types.
Destination defaults take precedence over source defaults, and your explicit configuration always
overrides defaults. They are resolved each time the task is read or run rather than copied into
it, so a task always gets the current defaults for its locations, and a task moved to another
provider picks up that provider's defaults. Only the values that differ from the defaults are
stored as your configuration; a value equal to the default is treated as the default.
## Scheduling Tasks
DataRaven supports flexible task scheduling with cron expressions and convenient keywords.
### Cron Expressions
Tasks use **5-field UNIX cron expressions** evaluated in **UTC timezone** by the Hatchet scheduler:
```
┌─────────────── minute (0 - 59)
│ ┌───────────── hour (0 - 23)
│ │ ┌─────────── day of month (1 - 31)
│ │ │ ┌───────── month (1 - 12)
│ │ │ │ ┌─────── day of week (0 - 6) (Sunday = 0)
│ │ │ │ │
* * * * *
```
### Schedule Keywords
For common scheduling needs, use these convenient keywords:
| Keyword | Equivalent Cron | Description |
| ----------- | --------------- | ---------------------------------------- |
| `@hourly` | `0 * * * *` | Every hour at minute 0 |
| `@daily` | `0 0 * * *` | Every day at midnight UTC |
| `@weekly` | `0 0 * * 0` | Every Sunday at midnight UTC |
| `@monthly` | `0 0 1 * *` | First day of every month at midnight UTC |
| `@yearly` | `0 0 1 1 *` | January 1st at midnight UTC |
| `@annually` | `0 0 1 1 *` | Same as @yearly |
### Examples
```bash theme={"theme":{"light":"github-light","dark":"poimandres"}}
# Every 15 minutes
*/15 * * * *
# Every day at 3:30 AM UTC
30 3 * * *
# Every Monday at 9 AM UTC
0 9 * * 1
# First day of every month at 6 AM UTC
0 6 1 * *
# Every 6 hours
0 */6 * * *
```
Need help with cron expressions? Use [crontab.guru](https://crontab.guru/) for interactive cron
schedule building and validation.
**Pro Tier Required:** Task scheduling is only available on Pro tier (\$29/mo). Free tier users can
only run tasks on-demand.
## Dry Runs
Before executing a task with real data transfer, use **dry runs** to preview exactly what would
happen without actually moving any data.
### Benefits of Dry Runs
* **Preview transfers** — See which files would be copied, updated, or deleted
* **Test filters** — Verify your include/exclude patterns work correctly
* **Validate configuration** — Catch configuration errors before real execution
* **No usage impact** — Dry run data transfer doesn't count toward usage metrics
### Running Dry Runs
Click the **"Dry Run"** button on any task to execute a preview.
`bash POST /tasks/{id}/executions/dry-run `
### Dry Run Output
Dry runs provide detailed logs showing:
* Files that would be transferred
* Files that would be skipped (already up-to-date)
* Files that would be deleted (sync mode only)
* Total bytes and file counts for the operation
**Always run a dry run first** when using sync mode or complex filter patterns. It's your safety
net against unintended data loss.
## Task Execution
### Running Tasks
**Web Dashboard:** Click "Run Now" button on any task **API:** `bash POST /tasks/{id}
/executions `
When scheduling is enabled, tasks automatically run according to their cron schedule. Scheduled
runs appear in the execution history with a "scheduled" trigger type.
### Execution Tracking
Every task execution creates a detailed execution record containing:
* **Real-time logs** — Stream transfer progress and debug information
* **Transfer statistics** — Bytes transferred, files processed, error counts
* **Performance metrics** — Duration, transfer rates, checkpoint timing
* **Downloadable logs** — Full execution logs for debugging and audit
### Execution States
| State | Description |
| ----------- | ---------------------------------- |
| `PENDING` | Execution queued, waiting to start |
| `RUNNING` | Currently transferring data |
| `SUCCESS` | Completed successfully |
| `FAILED` | Completed with errors |
| `CANCELLED` | User-cancelled before completion |
For detailed information about monitoring and managing executions, see the [executions API reference](/api-reference/executions/list-executions).
## Task Lifecycle & Status Management
Tasks can be in one of three states, each with different capabilities:
**Default state** for new tasks.
* ✅ Can be executed on-demand
* ✅ Scheduled runs are active (if configured)
* ✅ Can be disabled or archived
* ✅ Full edit capabilities
**Paused state** — temporarily inactive.
* ❌ Cannot be executed
* ❌ Scheduled runs are paused
* ✅ Can be re-enabled
* ✅ Can be archived
* ✅ Configuration can be edited
**Permanent soft delete** — cannot be reactivated.
* ❌ Cannot be executed
* ❌ Cannot be re-enabled
* ❌ Cannot be edited
* ✅ Execution history is preserved
* ⚠️ **Irreversible** —
create new task if needed
### Status Transitions
* Enabled → Disabled ✅
* Disabled → Enabled ✅
* Enabled → Archived ✅
* Disabled → Archived ✅
* Archived → Enabled ❌
* Archived → Disabled ❌
* Archived → *anything* ❌
## Task Deletion vs Archiving
### Deletion Rules
Tasks can only be **hard deleted** under specific conditions:
**DELETE only works for tasks with ZERO executions.** If a task has any execution history,
deletion will fail with a 400 error — you must archive instead.
* ✅ **Can delete:** Fresh tasks that have never been executed
* ❌ **Cannot delete:** Tasks with any execution history
* 🔐 **Requires:** ADMIN role or higher
### Why This Restriction?
Execution history serves as an **immutable audit trail** for:
* Compliance requirements
* Billing calculations
* Forensic analysis
* Data governance
## Task Duplication
The DataRaven UI supports **task duplication** for creating similar tasks with slight variations:
### What Gets Copied
* All configuration settings
* RClone options
* Filter patterns
* Scheduling settings
### What Gets Reset
* Task name (you must provide a new unique name)
* Execution history (new task starts clean)
* Created/modified timestamps
### Use Cases
* Creating staging vs production versions of the same transfer
* Setting up similar tasks for different source/destination pairs
* Testing configuration variations
Duplication is perfect for creating task templates. Set up one task with complex filtering and
performance tuning, then duplicate it for different data sources.
## Tier Limits & Performance
DataRaven applies different limits based on your subscription tier. **Limits are enforced at
execution time**, not task creation time.
**Subscription:** Free
**Task Limits:**
* Maximum 2 tasks total
* On-demand execution only (no scheduling)
**Performance Limits (applied at runtime):**
* `transfers`: 1 (single-threaded transfers)
* `checkers`: 1 (single-threaded checking)
* `bwlimit`: 1G (max 1GB/s bandwidth)
**Subscription:** \$29/month
**Task Limits:**
* Maximum 50 tasks total
* Full scheduling capabilities
**Performance Limits:**
* `transfers`: up to 64 (user configurable)
* `checkers`: up to 64 (user configurable)
* `bwlimit`: up to 10G (user configurable)
Pro tier respects your configured values. If you set `transfers=8`, it stays at 8. The limits
are maximums, not forced values.
### Tier Transitions
When you change subscription tiers, the effects are **immediate**:
* **Upgrade to Pro:** Existing tasks immediately get Pro limits on next execution
* **Downgrade to Free:** Existing tasks immediately get Free limits on next execution
* **No reconfiguration needed** — DataRaven handles the transition automatically
## Security & Validation
DataRaven implements multiple security layers to protect your data and systems:
### Path Validation
* **Command injection prevention:** Source and destination paths are validated against shell
metacharacters
* **Path traversal protection:** Paths cannot contain `../` or other directory traversal patterns
* **Character restrictions:** Paths must use safe, filesystem-compatible characters
### Configuration Validation
* **Filter pattern validation:** All filter patterns must start with `+` or `-` prefixes
* **RClone config size limit:** Configuration payload limited to 50KB maximum
* **Allowed flags only:** Only explicitly defined rclone flags are accepted — extra fields are
rejected
* **Provider defaults:** Safe provider-specific defaults are applied server-side at run time
### Data Protection
* **Payload size limits:** Request payloads are size-limited to prevent abuse
* **Rate limiting:** API endpoints are rate-limited to prevent overload
* **Audit logging:** All task operations are logged for compliance and debugging
## Required Permissions
Different task operations require different user roles:
| Operation | Required Role | Notes |
| ------------------- | ------------- | -------------------------------------- |
| View/List Tasks | VIEWER | Read-only access |
| Create Task | OPERATOR | Can create and configure |
| Update Task | OPERATOR | Can modify existing tasks |
| Run/Dry Run Task | OPERATOR | Can execute tasks |
| Enable/Disable Task | OPERATOR | Can change task status |
| Archive Task | OPERATOR | Can soft-delete tasks |
| Delete Task | ADMIN | Can hard-delete (zero executions only) |
Role inheritance applies: ADMIN users can perform all OPERATOR and VIEWER operations.
## Best Practices
* Always test with dry runs
* Use copy mode for backups
* Be extremely careful with sync mode
* Start with small file sets
* Tune transfers/checkers for your bandwidth
* Use appropriate buffer sizes
* Consider time-based bandwidth limiting
* Monitor execution performance
* Use descriptive task names
* Document complex filter patterns
* Group related tasks logically
* Archive old tasks instead of deleting
* Consider UTC timezone for cron
* Spread scheduled tasks across time
* Account for transfer duration in schedules
* Use keywords for common schedules
## Troubleshooting
### Common Issues
**Possible causes:**
* Task is disabled or archived
* Insufficient user role (need OPERATOR+)
* Source/destination locations offline
* Invalid rclone configuration
**Solutions:**
1. Check task status and enable if needed
2. Verify user permissions
3. Test location connectivity
4. Run a dry run to validate config
**Possible causes:**
* Low `transfers` or `checkers` values
* Bandwidth limiting too restrictive
* Small buffer size
* Network latency issues
**Solutions:**
1. Increase `transfers` (4-16 for most cases)
2. Increase `checkers` (8-32 for many small files)
3. Adjust `bwlimit` or remove entirely
4. Increase `buffer_size` for high-latency connections
**Possible causes:**
* Restrictive filter patterns
* Files already exist (with `ignore_existing`)
* Files newer on destination (with `update`)
* Size/age filters excluding files
**Solutions:**
1. Review filter patterns with dry run
2. Check comparison options settings
3. Verify source file timestamps
4. Test with minimal filters first
**Possible causes:**
* Files exist on destination but not source
* Filter patterns changed since last sync
* Source path configuration changed
**Solutions:**
1. Always dry run sync operations first
2. Verify source path hasn't changed
3. Check that source files still exist
4. Consider using copy mode instead
The execution failed with a message such as *"Sync stopped: it would delete more than 1000
objects at the destination (max\_delete limit)"*.
**Possible causes:**
* The source listing was incomplete, so objects that still exist looked deleted
* The source really did lose more objects than the limit allows
* The source path or filters changed, so the sync now sees a different set of objects
**Solutions:**
1. Check that the source still holds the objects you expect
2. Run a dry run to see which objects the sync would delete
3. If the deletions are intended, raise `max_delete` on the task, or set it to `-1` to remove
the limit, then run the task again
## Next Steps
* **[Executions API](/api-reference/executions/list-executions)** — Monitor and manage task runs
* **[Connecting Storage](/connecting-storage)** — Add more source/destination locations
* **[API Reference](/api-reference/tasks)** — Complete API documentation
* **[Secrets Management](/secrets)** — Secure credential storage
# API Keys
Source: https://docs.dataraven.io/developer-platform/api-keys
Programmatic access to the DataRaven API with scoped, team-bound API keys.
API keys let you authenticate with the DataRaven API without a browser session. They are the foundation for building automations, integrations, and developer tooling on top of DataRaven.
Each key carries only the scopes it needs — least-privilege by default.
Revoke a compromised key immediately. All in-flight requests fail instantly.
Rotate the secret while keeping the same key ID, name, and scopes. Update your secret store before deploying.
Every create, revoke, rotate, and delete is recorded in the audit log with IP and user agent.
## What's Coming
API keys are the first step toward a full developer platform. They unlock programmatic access today and power the integrations we're building next.
Official client libraries for Python, TypeScript, and Go.
Manage transfers, secrets, and tasks from the terminal.
Trigger and monitor transfers from GitHub Actions, Airflow, Dagster, and more.
Let AI agents orchestrate data movement across your infrastructure.
***
## Key Format
API keys follow a structured format that makes them easy to identify and parse:
```
dr__
```
| Part | Description |
| -------- | ------------------------------------------------------------------ |
| `dr_` | Fixed prefix — lets DataRaven distinguish API keys from JWT tokens |
| `key_id` | 12-character alphanumeric identifier (stable across rotations) |
| `secret` | 256-bit cryptographically random secret (URL-safe base64) |
The full key is shown **once** at creation time. Store it securely — it cannot be retrieved again.
If lost, rotate the key to generate a new secret.
## Authentication
Pass the API key as a Bearer token in the `Authorization` header:
```bash theme={"theme":{"light":"github-light","dark":"poimandres"}}
curl https://api.dataraven.io/v1/teams/{team_id}/tasks \
-H "Authorization: Bearer dr_aBcDeFgHiJkL_xYz..."
```
## Scopes
Every API key carries a list of scopes that control what it can access. Scopes follow a `resource:action` pattern and are validated at creation time.
### Example: Read-Only Monitoring Key
```json theme={"theme":{"light":"github-light","dark":"poimandres"}}
{
"name": "Monitoring Dashboard",
"scopes": ["tasks:read", "audit_logs:read", "usage:read"]
}
```
### Example: CI/CD Execution Key
```json theme={"theme":{"light":"github-light","dark":"poimandres"}}
{
"name": "GitHub Actions - Deploy Pipeline",
"scopes": ["tasks:read", "tasks:execute"]
}
```
### Example: Full Automation Key
```json theme={"theme":{"light":"github-light","dark":"poimandres"}}
{
"name": "Terraform Provisioner",
"scopes": [
"locations:create", "locations:read", "locations:delete",
"secrets:create", "secrets:read", "secrets:delete",
"tasks:create", "tasks:read", "tasks:update", "tasks:delete", "tasks:execute"
]
}
```
For a complete list of all available scopes, see the [Permissions Matrix](/security/permissions#scope-reference).
## Lifecycle
| Action | What Happens |
| ---------- | ------------------------------------------------------------------------------------------ |
| **Create** | Generates a new key. The full key (with secret) is returned once. |
| **Rotate** | Replaces the secret. Same key ID, name, and scopes. Old secret is immediately invalidated. |
| **Revoke** | Soft-delete — the key becomes unusable but remains visible in the dashboard for audit. |
| **Delete** | Permanent removal from the system. |
## Tier Limits
| Tier | Max Active Keys |
| ---- | --------------- |
| Free | 2 |
| Pro | 25 |
Only non-revoked keys count toward the limit.
## Security Best Practices
A key that only needs to trigger executions should have `tasks:read` and `tasks:execute` — not every scope. If a key is compromised, the blast radius is limited to its scopes.
Keys created for one-off migrations or contractor access should have an `expires_at` value. Expired keys are automatically rejected.
Use the rotate endpoint to generate a new secret without changing the key ID. The old secret is invalidated immediately, so update your secret store and redeploy before rotating.
Store API keys in your CI/CD platform's secret manager (GitHub Actions secrets, GitLab CI variables, etc.). The `dr_` prefix makes it easy to scan for accidental leaks.
Every API key action is logged. Filter the audit log by `actor_type: api_key` to see all programmatic activity across your team.
# Import rclone.conf
Source: https://docs.dataraven.io/guides/import-rclone-config
Migrate your existing rclone remotes into DataRaven in seconds
If you already use [rclone](https://rclone.org), you can import your `rclone.conf` file to
automatically create **secrets** and **locations** in DataRaven — no manual re-entry required.
## Prerequisites
* A DataRaven account with **Admin** or **Owner** role on the team
* An existing `rclone.conf` file (usually at `~/.config/rclone/rclone.conf`)
## Supported Providers
The importer supports object storage remotes:
| rclone type | DataRaven provider |
| ---------------------- | --------------------------------------------------------- |
| `s3` (AWS) | [AWS S3](/providers/aws-s3) |
| `s3` (Cloudflare) | [Cloudflare R2](/providers/cloudflare-r2) |
| `s3` (Wasabi) | [Wasabi](/providers/wasabi) |
| `s3` (Tigris) | [Tigris](/providers/tigris) |
| `s3` (Mega) | [MEGA S4](/providers/mega-s4) |
| `s3` (DigitalOcean) | [DigitalOcean Spaces](/providers/digitalocean-spaces) |
| `s3` (Hetzner) | [Hetzner](/providers/hetzner) |
| `s3` (Rabata) | [Rabata](/providers/rabata) |
| `s3` (ImpossibleCloud) | [Impossible Cloud](/providers/impossible-cloud) |
| `s3` (Fastly) | [Fastly Object Storage](/providers/fastly-object-storage) |
| `s3` (Other) | [S3 Compatible](/providers/s3-compatible) |
| `azureblob` | [Azure Blob Storage](/providers/azure-blob) |
| `gcs` | [Google Cloud Storage](/providers/google-cloud-storage) |
| `b2` | [Backblaze B2](/providers/backblaze-b2) |
Remotes using non-object-storage types (Google Drive, Dropbox, SFTP, etc.) are automatically
skipped with a friendly message — they won't cause errors. Note that rclone's `mega` type is the
consumer MEGA cloud drive, which is not supported; MEGA S4 object storage arrives as `s3` instead.
Some providers have no `provider =` value of their own, or gained one only recently, so real
configs write `provider = Other` (or omit it). [Fil One](/providers/fil-one),
[Filebase](/providers/filebase), [Tigris](/providers/tigris), and [MEGA S4](/providers/mega-s4)
remotes are recognized by their endpoint hostname in that case, rather than falling back to S3
Compatible.
## Step 1: Upload Your Config
In the dashboard, navigate to **Import rclone.conf** from the sidebar.
Drag and drop your `rclone.conf` file onto the upload area, or click to browse. The file is
parsed on the server — credentials are **not** stored at this stage.
Click **Continue** to parse the file and preview what will be imported.
## Step 2: Review and Edit
After parsing, you'll see a breakdown of your remotes:
* **Ready to import** — Recognized remotes with their mapped provider type and detected credentials
* **Skipped** — Recognized but unsupported types (e.g., Google Drive)
* **Errors** — Remotes that couldn't be parsed (missing fields, unknown types)
For each importable remote you can:
* **Edit the name** — Change the display name for the secret and location
* **Set the bucket name** — Required for all remotes (enter your bucket or container name)
* **Set the region and endpoint** — Prefilled from the file when it had them, editable either way.
Providers with a fixed region list get a dropdown; picking a region fills in its endpoint URL.
* **Deselect** — Uncheck any remote you don't want to import
Every selected remote requires a **bucket or container name**, and providers that cannot be
reached without a **region** or an **endpoint URL** (Fastly, Fil One, Wasabi, Cloudflare R2,
among others) require that too. The importer marks these fields with `*` and will not proceed
until every selected remote has them — the same rule the location form applies.
A remote that carries only an endpoint on a provider whose hostname encodes the region — Fastly's
`.object.fastlystorage.app`, Fil One's `.s3.fil.one`, Impossible Cloud's
`.storage.impossibleapi.net` — arrives with its region already filled in.
## Step 3: Confirm Import
Click **Import** to create resources. For each selected remote, two resources are created:
1. A **Secret** — Credentials are encrypted and stored in AWS SSM Parameter Store
2. A **Location** — Linked to the new secret, ready to use in tasks
Remotes are processed independently. If one fails, the others still succeed and any partially
created resources are cleaned up automatically.
## What's Next?
Once your locations are imported, you can immediately use them in tasks.
Set up sync, copy, or move operations between your imported locations.
# Cross-Region Bucket Replication
Source: https://docs.dataraven.io/guides/railway-cross-region-replication
Set up automated cross-region replication for Railway buckets
This guide walks through setting up automated cross-region replication for Railway Buckets using
DataRaven's scheduled tasks.
## Prerequisites
* A [Railway account](https://railway.app) with active project(s)
* Railway Buckets in at least two regions that you want to replicate
* DataRaven account with team access
Railway Buckets provide S3-compatible object storage. DataRaven treats them as object storage
locations, enabling automated replication across regions.
## Step 1: Gather Your Bucket Credentials
For each bucket you want to replicate, you'll need to collect the S3 credentials from the Railway
console:
1. Open your [Railway dashboard](https://railway.app/dashboard)
2. Navigate to your project and select the bucket
3. Go to the **Credentials** tab
4. Note the following values:
* **Bucket Name**: e.g., \`dr-bak-west-0iofqnrozari\`\`
* **Access Key ID**: Your S3 access key
* **Secret Access Key**: Your S3 secret key
* **Endpoint URL**: e.g., `https://t3.storageapi.dev`
Repeat this process for both your source and destination buckets.
## Step 2: Configure Source Bucket Location
### Via DataRaven Dashboard
1. Go to **Locations** in your team workspace
2. Click **Add Location**
3. Select **Railway** as the provider type
4. Fill in the following details:
* **Display Name**: e.g., "Railway US-East Bucket"
* **Bucket Name**: The bucket name from your credentials (e.g., `dr-bak-west-0iofqnrozari`)
* **Access Key ID**: Your S3 access key (or reference a
[vault secret](/vault-integration))
* **Secret Access Key**: Your S3 secret key (or reference a
[vault secret](/vault-integration))
* **Endpoint URL**: The endpoint from your credentials (e.g., `https://t3.storageapi.dev`)
5. Click **Test** to verify the connection
6. Click **Save**
## Step 3: Configure Destination Bucket
1. Repeat Step 2 for your destination bucket
2. Use the **Display Name**: e.g., "Railway EU-West Bucket"
3. Use the bucket name and credentials for your destination bucket (e.g.,
`dr-bak-west-0iofqnrozari`)
## Step 4: Create a Replication Task
1. Navigate to **Tasks** in your workspace
2. Click **Create Task**
3. Configure the task:
* **Task Name**: e.g., "Replicate US→EU Buckets"
* **Source Location**: Select your source Railway bucket
* **Destination Location**: Select your destination Railway bucket
* **Task Type**: Choose `sync` (one-way sync) or `copy` (copies new/changed files without
deleting)
* **Filters**: Optionally set path filters to replicate only specific directories or file types
### Schedule Options
Use cron expressions to set when your replication runs. Enter a standard cron schedule or use
shortcuts:
* `@daily` — Runs every day at midnight
* `@weekly` — Runs every Sunday at midnight
* Custom cron — e.g., `0 2 * * *` for 2 AM every day
**All jobs execute in UTC.** You can also run one-off syncs manually from the task dashboard.
To set up notifications for task completion or failures, configure them in **Notifications**.
## Step 5: Monitor Replication
### Dashboard Monitoring
1. Go to **Tasks** and select your replication task
2. View:
* **Last Execution**: When the sync last ran
* **Status**: Success/In-progress/Failed
* **Data Transferred**: Total bytes synced
* **Execution Time**: How long the sync took
### View Detailed Logs
1. Click on an execution in the task history
2. View logs including:
* Files synced/skipped
* Any errors or warnings
* Performance metrics
### Set Up Alerts
1. Navigate to **Notifications** in settings
2. Create a notification rule for your replication task
3. Choose channel: Email, Slack, or Webhook
4. Configure when to notify: On success, failure, or both
## Best Practices
### Performance Optimization
* **Filter by path**: If you only need to replicate specific directories, use path filters in your
task configuration to reduce transfer time and costs
* **Schedule during off-peak hours**: Run large syncs at times when traffic is lowest
* **Use Daily or Weekly for large datasets**: Hourly syncs can be expensive if your data is large
### Cost Management
* **Railway Bucket egress is free**: Cross-region transfers between Railway Buckets don't incur
egress charges
## Troubleshooting
### "Connection Failed" Error
**Cause**: Invalid S3 credentials or incorrect endpoint URL
**Solution**:
* Verify your Access Key ID and Secret Access Key are correct
* Confirm the Endpoint URL matches what's shown in Railway console (e.g.,
`https://t3.storageapi.dev`)
* Ensure the bucket name is exact (Railway generates unique names with hashes)
* Test the connection again after updating credentials
### Slow Transfer Speed
**Cause**: Free tier accounts have limitations on transfer performance
**Solution**:
* Free tier transfers are **single-threaded with a 1 GB/s speed cap**. Upgrade to Pro for parallel
transfers and faster speeds.
* Consider splitting large datasets into smaller sync jobs
* Run syncs during off-peak hours to avoid network congestion
## Next Steps
* Learn about [general task creation](/creating-tasks) for other use cases
* Explore [vault integration](/vault-integration) for managing credentials securely
## Additional Resources
* [Railway Documentation](https://docs.railway.com/storage-buckets)
# Introduction
Source: https://docs.dataraven.io/introduction
Cloud-agnostic data movement platform with enterprise controls
# Welcome to DataRaven
DataRaven is a cloud-agnostic data movement platform, built for reliable, production-grade
[rclone](https://rclone.org/) orchestration. It gives you enterprise controls, monitoring, and team
collaboration.
## Why DataRaven?
Moving data between cloud providers shouldn't require DevOps expertise, managing servers, stitching together scripts, managing
credentials in plaintext, or babysitting long-running transfers. DataRaven handles all of that.
S3, GCS, Azure Blob, Cloudflare R2, Wasabi, Backblaze B2, Railway, Oracle Cloud, plus any S3
compatible backend.
Bring Your Own Vault (BYOV) — connect 1Password, Doppler, or Infisical. We never see your
secrets.
Shared workspaces with role-based access, and flexible notification integrations.
Cron-based scheduling, real-time execution logs, webhooks.
## Use Cases
* **Cloud Migration** — Move data between providers with minimal downtime
* **Backup & Disaster Recovery** — Schedule cross-cloud backups and maintain off-site copies for
business continuity
* **AI/ML Data Pipelines** — Shuttle training data and model artifacts across storage backends
* **B2B Data Sharing** — Securely exchange datasets with partners
* **Multi-Cloud Resiliency** — Replicate critical data across providers
* **Cost Optimization** — Tier data to cheaper storage without manual effort
* **Compliance & Data Sovereignty** — Meet GDPR, HIPAA, and data residency requirements with
controlled geographic placement
Create your first data transfer in under 5 minutes.
# Notifications
Source: https://docs.dataraven.io/notifications
Set up alerts for task executions, resource changes, and more — delivered to Slack, Discord, email, PagerDuty, and 100+ other services.
DataRaven uses [Apprise](https://github.com/caronc/apprise) under the hood to deliver notifications
to virtually any service you already use. Configure a notification once, and DataRaven will alert
you whenever the events you care about occur.
## How It Works
Pick which event should trigger the notification (e.g., task execution failed).
Each notification service has a unique URL format that contains your credentials or webhook
token.
Narrow which events match using filters, and customize the message with a Jinja2 template.
Use the **Send Test** button to verify delivery, then enable the notification.
## Quick Setup
Quick Setup lets you create multiple notification configs at once — one per event type — all sharing
the same Apprise URL. This is the fastest way to get comprehensive alerting for a single
destination like a Slack channel or Discord webhook.
Go to **Notifications** and click **Quick Setup**.
Provide a name prefix (e.g., "Slack Alerts") and your Apprise URL. Each notification config will
be named **"\{prefix} - \{Event Name}"** (e.g., "Slack Alerts - Execution Failed").
Check the event types you want notifications for. Use the group checkboxes to select entire
categories, or **Select All** for full coverage.
Click **Create** to bulk-create all selected configs in a single request. Each config gets
a sensible default message template that you can customize later.
After Quick Setup, each notification config is fully independent — you can edit individual configs
to add event filters, change templates, or disable specific ones without affecting the others.
### Quick Setup via API
You can also bulk-create notifications programmatically using the
[Bulk Create endpoint](/api-reference/notifications/bulk-create-notifications):
```bash theme={"theme":{"light":"github-light","dark":"poimandres"}}
curl -X POST https://api.dataraven.io/v1/teams/{team_id}/notifications/bulk \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "Slack Alerts",
"apprise_url": "slack://tokenA/tokenB/tokenC",
"event_types": [
"task_execution_started",
"task_execution_completed",
"task_execution_failed",
"task_execution_cancelled"
]
}'
```
This creates one notification config per event type. The response includes all created configs with
their IDs, vault references, and default templates.
## Apprise URLs
Every notification service is configured through a single URL string. Apprise supports **100+
services** — see the [full list on the Apprise docs](https://appriseit.com/services/).
### Popular Services
```text Slack theme={"theme":{"light":"github-light","dark":"poimandres"}}
slack://tokenA/tokenB/tokenC
```
```text Discord theme={"theme":{"light":"github-light","dark":"poimandres"}}
discord://webhook_id/webhook_token
```
```text Microsoft Teams theme={"theme":{"light":"github-light","dark":"poimandres"}}
msteams://TokenA/TokenB/TokenC
```
```text Email (Gmail) theme={"theme":{"light":"github-light","dark":"poimandres"}}
mailto://user:password@gmail.com
```
```text PagerDuty theme={"theme":{"light":"github-light","dark":"poimandres"}}
pagerduty://integration_key@routing_key
```
```text Telegram theme={"theme":{"light":"github-light","dark":"poimandres"}}
tgram://bot_token/chat_id
```
Refer to the [Apprise wiki](https://github.com/caronc/apprise/wiki) for the exact URL format and
required tokens for your service.
## Event Types
DataRaven supports **18 event types** organized into five categories:
Events fired during task execution lifecycle.
| Event Type | Fires When |
| -------------------------- | ---------------------------------- |
| `task_execution_started` | An execution begins |
| `task_execution_completed` | An execution finishes successfully |
| `task_execution_failed` | An execution encounters an error |
| `task_execution_cancelled` | An execution is cancelled |
Events fired when tasks are created, modified, or removed.
| Event Type | Fires When |
| --------------------- | ------------------------------ |
| `task_created` | A new task is created |
| `task_updated` | A task's configuration changes |
| `task_deleted` | A task is deleted |
| `task_status_changed` | A task is enabled or disabled |
Events fired when secrets are managed.
| Event Type | Fires When |
| ---------------- | ----------------------- |
| `secret_created` | A new secret is created |
| `secret_updated` | A secret is updated |
| `secret_deleted` | A secret is deleted |
Events fired when storage locations change.
| Event Type | Fires When |
| ------------------ | ------------------------- |
| `location_created` | A new location is created |
| `location_updated` | A location is updated |
| `location_deleted` | A location is deleted |
Events fired when API keys are managed.
| Event Type | Fires When |
| ----------------- | ------------------------ |
| `api_key_created` | A new API key is created |
| `api_key_rotated` | An API key is rotated |
| `api_key_revoked` | An API key is revoked |
| `api_key_deleted` | An API key is deleted |
## Event Filters
Event filters let you narrow which events trigger the notification. Filters are a JSON object where
**all conditions must match** (AND logic). Leave empty (`{}`) to match all events of the selected
type.
### Filter Rules
* **Exact match** — `{"trigger": "scheduled"}` matches only scheduled triggers
* **Any-of match** — `{"status": ["failed", "completed"]}` matches if the value is any item in the
array
* **Boolean match** — `{"is_dry_run": false}` matches only real executions
* **Multiple conditions** — `{"trigger": "scheduled", "is_dry_run": false}` requires both to match
### Examples by Event Type
```json Execution Filters theme={"theme":{"light":"github-light","dark":"poimandres"}}
// Only failed and completed executions
{"status": ["failed", "completed"]}
// Only scheduled triggers (not manual)
{"trigger": "scheduled"}
// Only real executions (not dry runs)
{"is_dry_run": false}
```
```json Task Filters theme={"theme":{"light":"github-light","dark":"poimandres"}}
// A specific task by name
{"name": "my-task"}
// Only sync tasks
{"task_type": "sync"}
// Only enabled tasks
{"status": "enabled"}
```
```json Secret Filters theme={"theme":{"light":"github-light","dark":"poimandres"}}
// A specific secret
{"name": "my-secret"}
// Secrets of a specific type
{"secret_type": "s3"}
```
```json Location Filters theme={"theme":{"light":"github-light","dark":"poimandres"}}
// Azure blob locations only
{"location_type": "azure_blob"}
// A specific region
{"region": "us-east-1"}
```
```json API Key Filters theme={"theme":{"light":"github-light","dark":"poimandres"}}
// A specific API key
{"name": "my-api-key"}
// A specific key ID
{"key_id": "AbCdEf123456"}
```
## Message Templates
Notifications use [Jinja2 templates](https://jinja.palletsprojects.com/) to format the message body.
Each event type exposes different variables. If you don't provide a custom template, DataRaven uses
a sensible default.
### Available Variables
| Variable | Description |
| ------------------ | ------------------------------------- |
| `task_name` | Name of the task |
| `execution_number` | Sequential execution number |
| `status` | Current execution status |
| `duration_seconds` | Execution duration (completed events) |
| `error_message` | Error details (failed events) |
| Variable | Description |
| ------------- | ------------------------------ |
| `name` | Task name |
| `description` | Task description |
| `status` | Task status (enabled/disabled) |
| Variable | Description |
| -------- | ----------- |
| `name` | Secret name |
| Variable | Description |
| --------------- | --------------------------------------- |
| `name` | Location name |
| `location_type` | Storage type (e.g., `s3`, `azure_blob`) |
| Variable | Description |
| ------------------ | ------------------------------------- |
| `name` | API key name |
| `created_by_email` | Email of the user who created the key |
| `expires_at` | Expiration date (or `Never`) |
| `created_at` | Creation timestamp |
| `rotated_at` | Rotation timestamp (rotated events) |
### Default Templates
Here are the built-in templates DataRaven uses when you don't specify a custom one:
```text Execution Started theme={"theme":{"light":"github-light","dark":"poimandres"}}
🚀 Task execution started
Task: {{task_name}}
Execution: #{{execution_number}}
Status: {{status}}
```
```text Execution Completed theme={"theme":{"light":"github-light","dark":"poimandres"}}
✅ Task execution completed
Task: {{task_name}}
Execution: #{{execution_number}}
Duration: {{duration_seconds}}s
```
```text Execution Failed theme={"theme":{"light":"github-light","dark":"poimandres"}}
🚨 Task execution failed
Task: {{task_name}}
Execution: #{{execution_number}}
Error: {{error_message}}
```
```text Task Created theme={"theme":{"light":"github-light","dark":"poimandres"}}
📝 New task created
Name: {{name}}
Description: {{description}}
```
```text Location Created theme={"theme":{"light":"github-light","dark":"poimandres"}}
📍 New location created: {{name}} ({{location_type}})
```
```text API Key Created theme={"theme":{"light":"github-light","dark":"poimandres"}}
🔑 API key created
Name: {{name}}
Created by: {{created_by_email}}
Expires: {{expires_at or 'Never'}}
Created: {{created_at}}
```
```text API Key Rotated theme={"theme":{"light":"github-light","dark":"poimandres"}}
🔑 API key rotated
Name: {{name}}
Rotated by: {{created_by_email}}
Rotated at: {{rotated_at}}
```
You can customize templates with any valid Jinja2 syntax — conditionals, default values, formatting,
etc.
```jinja theme={"theme":{"light":"github-light","dark":"poimandres"}}
🚨 ALERT: {{task_name}} failed (Execution #{{execution_number}})
{% if error_message %}
Error: {{error_message}}
{% else %}
No error details available.
{% endif %}
```
## Testing Notifications
Before relying on a notification in production, use the **Send Test** button in the admin UI. This
sends a sample notification using your configured Apprise URL and message template so you can verify
delivery without waiting for a real event.
Test notifications use placeholder data for template variables (e.g., `task_name` = "Test Task").
The actual notification will contain real event data.
## Security
Apprise URLs contain sensitive credentials — webhook tokens, API keys, passwords. DataRaven
**never** stores these in plain text or returns them in API responses.
* Apprise URLs are encrypted and stored in **AWS SSM Parameter Store**
* API responses include only a vault reference ID (`apprise_vault_secret_id`), never the URL itself
* URLs are only decrypted internally when sending a notification
## Tier Limits
| Plan | Notification Configs |
| -------- | -------------------- |
| **Free** | 1 |
| **Pro** | 25 |
Need more? [Upgrade your plan](https://app.dataraven.io) from the billing page.
## Common Patterns
**Event type:** `task_execution_failed`
**Filters:** `{}`
**Result:** Get notified whenever any task fails.
**Event type:** `task_execution_completed`
**Filters:** `{"trigger": "scheduled"}`
**Result:** Notify Slack when scheduled tasks finish (ignore manual runs).
**Event type:** `task_execution_failed`
**Filters:** `{"is_dry_run": false}`
**Result:** Page on-call when real (non-dry-run) executions fail.
**Event type:** `secret_created`, `secret_updated`, `secret_deleted`
**Filters:** `{}`
**Result:** Track all secret lifecycle events for compliance.
**Event type:** `api_key_created`, `api_key_rotated`, `api_key_revoked`, `api_key_deleted`
**Filters:** `{}`
**Result:** Track all API key lifecycle events for security auditing.
You can create multiple notification configs for the same event type with different filters and
destinations — for example, send failures to both Slack and PagerDuty.
# AWS S3
Source: https://docs.dataraven.io/providers/aws-s3
Connect Amazon S3 buckets as sources and destinations for transfer tasks.
Amazon Simple Storage Service — the reference S3 implementation. Standard configuration with no
custom endpoint.
## At a Glance
| | |
| ----------------- | -------------------------------------------------------- |
| **Region** | Required — e.g., `us-east-1`, `eu-west-1` |
| **Endpoint URL** | Not needed |
| **Credentials** | Access key — `access_key_id`, `secret_access_key` |
| **Task defaults** | None — AWS S3 works well with rclone's standard behavior |
## Credentials
Create a [secret](/secrets) with the **AWS S3** provider type. The auth method is
`s3_access_key`, which requires:
| Field | Description |
| ------------------- | --------------------- |
| `access_key_id` | IAM access key ID |
| `secret_access_key` | IAM secret access key |
Create the key for an IAM user with permission to **list, read, write, and delete** objects in
the bucket — the same operations
[connection verification](/connecting-storage#connection-verification) tests.
Temporary STS credentials are not supported — they require a session token, which DataRaven
does not store. Use a long-lived IAM access key pair.
## Location Setup
| Field | Value |
| ----------------- | -------------------------------------------------- |
| **region** | Required — must match where the bucket was created |
| **endpoint\_url** | Leave blank |
Example:
| Field | Example value |
| -------------- | ---------------------- |
| name | `Production S3 Bucket` |
| location\_type | AWS S3 |
| bucket\_name | `acme-prod-backups` |
| region | `us-east-1` |
## Importing from rclone.conf
AWS remotes appear in `rclone.conf` as `type = s3` with `provider = AWS`. A `type = s3` remote
with **no** `provider` value is also assumed to be AWS S3. See the
[rclone import guide](/guides/import-rclone-config).
# Azure Blob Storage
Source: https://docs.dataraven.io/providers/azure-blob
Connect Azure Blob Storage containers as sources and destinations for transfer tasks.
Azure Blob Storage containers. The only provider with a choice of auth method — SAS URL or
account key.
## At a Glance
| | |
| ----------------- | ------------------------------------------------------------ |
| **Region** | Required — Azure region, e.g., `eastus`, `westus2` |
| **Endpoint URL** | Not needed |
| **Credentials** | SAS URL, **or** account name + account key |
| **Task defaults** | None — Azure Blob works well with rclone's standard behavior |
## Credentials
Create a [secret](/secrets) with the **Azure Blob** provider type. Azure is the one provider
where you choose between two auth methods:
| Auth method | Required fields |
| ------------------- | ----------------------------- |
| `azure_sas_url` | `sas_url` |
| `azure_account_key` | `account_name`, `account_key` |
**Prefer SAS URLs over account keys.** SAS (Shared Access Signature) URLs are scoped to specific
containers and operations, and they expire automatically. Account keys grant full access to the
entire storage account. Generate the SAS URL from the Azure portal with the minimum required
permissions (read/write/list) and a reasonable expiration window.
## Location Setup
| Field | Value |
| ----------------- | ------------------------------------------------------------------ |
| **region** | Required — the Azure region where your storage account is deployed |
| **endpoint\_url** | Leave blank |
Example:
| Field | Example value |
| -------------- | -------------------------- |
| name | `Azure Prod Archive` |
| location\_type | Azure Blob Storage |
| bucket\_name | `archive` (container name) |
| region | `eastus` |
## Importing from rclone.conf
Azure remotes appear in `rclone.conf` as `type = azureblob`. If the remote has a `sas_url`, it
imports with the SAS URL auth method; otherwise the `account` and `key` fields import as an
account-key secret. The remote's `container` value carries over as the bucket name. See the
[rclone import guide](/guides/import-rclone-config).
# Backblaze B2
Source: https://docs.dataraven.io/providers/backblaze-b2
Connect Backblaze B2 buckets as sources and destinations for transfer tasks.
Backblaze B2 cloud storage, using B2's native API with application keys.
## At a Glance
| | |
| ----------------- | ----------------------------------------------------------- |
| **Region** | Optional |
| **Endpoint URL** | Not needed |
| **Credentials** | Application key — `application_key_id`, `application_key` |
| **Task defaults** | `transfers: 32`, `fast_list: true`, `b2_hard_delete: false` |
## Credentials
Create a [secret](/secrets) with the **Backblaze B2** provider type. The auth method is
`b2_application_key`, which requires:
| Field | Description |
| -------------------- | --------------------- |
| `application_key_id` | B2 application key ID |
| `application_key` | B2 application key |
Create an application key from the Backblaze dashboard under **App Keys**, scoped to the bucket
you plan to transfer to or from.
## Location Setup
| Field | Value |
| ----------------- | ----------- |
| **region** | Optional |
| **endpoint\_url** | Leave blank |
Example:
| Field | Example value |
| -------------- | ------------------ |
| name | `B2 Media Backups` |
| location\_type | Backblaze B2 |
| bucket\_name | `acme-media` |
## Provider Quirks
B2 locations automatically receive optimized [task defaults](/connecting-storage#provider-defaults)
following Backblaze's own recommendations: higher concurrency (`transfers: 32`), fast listing for
large buckets, and `b2_hard_delete: false` so deletions hide file versions rather than
permanently removing them. Override any of these in the task's rclone configuration.
## Importing from rclone.conf
B2 remotes appear in `rclone.conf` as `type = b2`. The remote's `account` field imports as
`application_key_id` and `key` as `application_key`. See the
[rclone import guide](/guides/import-rclone-config).
# Cloudflare R2
Source: https://docs.dataraven.io/providers/cloudflare-r2
Connect Cloudflare R2 buckets as sources and destinations for transfer tasks.
Cloudflare R2 — S3-compatible object storage with zero egress fees, addressed through an
account-specific endpoint.
## At a Glance
| | |
| ----------------- | ---------------------------------------------------------- |
| **Region** | Not needed |
| **Endpoint URL** | Required — `https://.r2.cloudflarestorage.com` |
| **Credentials** | Access key — `access_key_id`, `secret_access_key` |
| **Task defaults** | `s3_no_check_bucket: true`, `fast_list: true` |
## Credentials
Create a [secret](/secrets) with the **Cloudflare R2** provider type. The auth method is
`r2_access_key`, which requires:
| Field | Description |
| ------------------- | ------------------- |
| `access_key_id` | R2 API token ID |
| `secret_access_key` | R2 API token secret |
Create an R2 API token from the Cloudflare dashboard (**R2 → Manage R2 API Tokens**) with
read/write permission on the bucket.
The **account ID** does *not* belong in the secret — it goes in the location's `endpoint_url`.
## Location Setup
| Field | Value |
| ----------------- | ---------------------------------------------------------- |
| **region** | Leave blank |
| **endpoint\_url** | Required — `https://.r2.cloudflarestorage.com` |
The location form asks for your **Cloudflare account ID** (shown on the R2 overview page in the
dashboard) and fills in the endpoint URL from it. Onboarding and the rclone.conf import review do
the same. The URL stays editable: a bucket in a jurisdiction uses that jurisdiction's host instead,
such as `https://.eu.r2.cloudflarestorage.com` for the EU.
Example:
| Field | Example value |
| -------------- | ----------------------------------------------- |
| name | `R2 Assets` |
| location\_type | Cloudflare R2 |
| bucket\_name | `acme-assets` |
| endpoint\_url | `https://a1b2c3d4e5f6.r2.cloudflarestorage.com` |
## Provider Quirks
R2 buckets are managed through the dashboard — the API exposes no `CreateBucket`, so R2 locations
default to `s3_no_check_bucket: true`. R2 also handles list operations very efficiently, so
`fast_list` is on by default.
## Importing from rclone.conf
R2 remotes appear in `rclone.conf` as `type = s3` with `provider = Cloudflare`. If the remote has
an `account_id` but no `endpoint`, the importer builds the endpoint URL from the account ID
automatically. See the [rclone import guide](/guides/import-rclone-config).
# DigitalOcean Spaces
Source: https://docs.dataraven.io/providers/digitalocean-spaces
Connect DigitalOcean Spaces buckets as sources and destinations for transfer tasks.
DigitalOcean Spaces Object Storage — S3-compatible, with the region encoded in the endpoint
hostname.
## At a Glance
| | |
| ----------------- | ------------------------------------------------- |
| **Region** | Not needed (embedded in endpoint) |
| **Endpoint URL** | Required — e.g., `nyc3.digitaloceanspaces.com` |
| **Credentials** | Access key — `access_key_id`, `secret_access_key` |
| **Task defaults** | `s3_no_check_bucket: true`, `fast_list: true` |
## Credentials
Create a [secret](/secrets) with the **DigitalOcean Spaces** provider type. The auth method is
`digitalocean_spaces_access_key`, which requires:
| Field | Description |
| ------------------- | ------------------------ |
| `access_key_id` | Spaces access key ID |
| `secret_access_key` | Spaces secret access key |
Generate Spaces access keys from the DigitalOcean control panel under **API → Spaces Keys**.
## Location Setup
| Field | Value |
| ----------------- | ------------------------------------------------------ |
| **region** | Leave blank — the endpoint hostname carries the region |
| **endpoint\_url** | Required — `.digitaloceanspaces.com` |
The endpoint URL includes the region (e.g., `nyc3.digitaloceanspaces.com`,
`sfo3.digitaloceanspaces.com`).
Example:
| Field | Example value |
| -------------- | ----------------------------- |
| name | `DO Spaces NYC` |
| location\_type | DigitalOcean Spaces |
| bucket\_name | `acme-space` |
| endpoint\_url | `nyc3.digitaloceanspaces.com` |
## Importing from rclone.conf
Spaces remotes appear in `rclone.conf` as `type = s3` with `provider = DigitalOcean`. See the
[rclone import guide](/guides/import-rclone-config).
# Fastly Object Storage
Source: https://docs.dataraven.io/providers/fastly-object-storage
Connect Fastly Object Storage buckets as sources and destinations for transfer tasks.
Fastly's S3-compatible object storage. Region names look like AWS's but are Fastly's own, and the
S3 API only works with full-access keys.
## At a Glance
| | |
| ----------------- | ----------------------------------------------------------------- |
| **Region** | Required — Fastly region name, e.g., `us-east-1`, `eu-central` |
| **Endpoint URL** | Derived from region — `https://.object.fastlystorage.app` |
| **Credentials** | Access key — `access_key_id`, `secret_access_key` |
| **Task defaults** | `s3_no_check_bucket: true`, `fast_list: true` |
## Credentials
Create a [secret](/secrets) with the **Fastly Object Storage** provider type. The auth method is
`fastly_access_key`, which requires:
| Field | Description |
| ------------------- | ----------------- |
| `access_key_id` | Fastly access key |
| `secret_access_key` | Fastly secret key |
Fastly's S3-compatible API requires an access key with **full access to all buckets and
read/write scope** — per-bucket or read-only keys do not work with the S3 API. Treat Fastly
secrets accordingly: the key grants account-wide object storage access.
## Location Setup
| Field | Value |
| ----------------- | ----------------------------------------------------------------------------- |
| **region** | Required — Fastly region name |
| **endpoint\_url** | Optional — derived from region as `https://.object.fastlystorage.app` |
Example:
| Field | Example value |
| -------------- | --------------------- |
| name | `Fastly EU Storage` |
| location\_type | Fastly Object Storage |
| bucket\_name | `acme-objects` |
| region | `eu-central` |
## Provider Quirks
* **Region names are Fastly's own.** They resemble AWS region names but are not the same set —
note `eu-central` has no `-1` suffix.
* **Bucket names cannot start with `fst` or `fastly`** — those prefixes are reserved.
## Importing from rclone.conf
Fastly remotes appear in `rclone.conf` as `type = s3` with `provider = Fastly`. A config that
carries only the endpoint is fine — the region is recovered from the hostname
(`.object.fastlystorage.app`). See the [rclone import guide](/guides/import-rclone-config).
# Fil One
Source: https://docs.dataraven.io/providers/fil-one
Connect Fil One buckets as sources and destinations for transfer tasks.
Fil One — S3-compatible object storage built on Filecoin, with always-on encryption at rest and
optional versioning and object lock. Access keys are region-scoped, which shapes how you organize
secrets, and the two regions differ in API surface **and behavior** in ways worth knowing before
a big migration — most importantly, `eu-west-1` cannot pass rclone's checksum verification, so
DataRaven sets `ignore_checksum` automatically on transfers that touch that region (see
[below](#eu-west-1-checksum-verification-fails-without-ignore_checksum)).
## At a Glance
| | |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------- |
| **Region** | Required — `eu-west-1` (EU, France) or `us-east-1` (US East, Michigan) |
| **Endpoint URL** | Derived from region — `https://.s3.fil.one` |
| **Credentials** | Access key — `access_key_id`, `secret_access_key` |
| **Task defaults** | `s3_no_check_bucket: true`, `fast_list: true`. On `eu-west-1`, also `ignore_checksum: true`. The remote itself sets `list_version = 2` |
| **Notable limits** | Objects up to 5 TB. Multipart parts from 5 MB to 5 GB (max 10,000). `DeleteObjects` batches of 1,000 or less |
| **Provider docs** | [docs.fil.one](https://docs.fil.one) · [S3 compatibility matrix](https://docs.fil.one/reference/s3-compatibility) |
`s3_no_check_bucket` is on by default because `CreateBucket` only exists in `us-east-1` — in
`eu-west-1` buckets are created through the Fil One dashboard — and is an explicit key permission
even where it exists. Skipping rclone's bucket-existence check avoids a spurious failure against a
bucket that is already there.
## Credentials
Create a [secret](/secrets) with the **Fil One** provider type. The auth method is
`filone_access_key`, which requires:
| Field | Description |
| ------------------- | ------------------------- |
| `access_key_id` | Fil One access key ID |
| `secret_access_key` | Fil One secret access key |
Fil One access keys are **region-scoped**: a key created for one region only authenticates
against that region's endpoint. If you store data in more than one Fil One region, create a
separate secret per region.
Keys are created in the Fil One dashboard and can be scoped to specific buckets and permissions —
worth doing for a transfer key that only needs one bucket. Note that `CreateBucket` and
`DeleteBucket` are explicit permissions a key must be granted; a default key cannot create
buckets even in `us-east-1`.
## Location Setup
| Field | Value |
| ----------------- | --------------------------------------------------------------- |
| **region** | Required — `eu-west-1` (EU, France) or `us-east-1` (US East) |
| **endpoint\_url** | Optional — derived from region as `https://.s3.fil.one` |
A bucket's region is fixed when the bucket is created and data cannot be moved between regions
afterwards, so the location's region simply matches where the bucket lives. Fil One requires
**path-style addressing** and HTTPS — the configuration DataRaven generates uses both, so there
is nothing to set up.
Example:
| Field | Example value |
| -------------- | -------------- |
| name | `Fil One EU` |
| location\_type | Fil One |
| bucket\_name | `acme-archive` |
| region | `eu-west-1` |
## Provider Quirks
### Bucket naming — no dots
Fil One does **not** allow dots (`.`) in bucket names, which S3 permits. An S3 bucket named
like a hostname — `backups.example.com` — needs a new name on the way in.
The full rules:
* 3–63 characters
* Lowercase letters, numbers, and hyphens only — no dots
* Must begin and end with a letter or number
* Unique within Fil One
DataRaven checks these rules when you create the location, so a non-conforming name fails
immediately with a clear message instead of surfacing later as an error from the service.
Object **key** handling depends on the region. In DataRaven's end-to-end transfer tests,
`us-east-1` accepted everything S3 does — keys with consecutive slashes (`//`), trailing slashes
on objects with content, Unicode, and 250+ character paths all copied in unchanged. `eu-west-1`
**rejects keys containing consecutive slashes** with a 400 (`Object name contains unsupported
characters`), so a transfer carrying such keys fails on those objects mid-run — check source
keys before a migration into `eu-west-1`.
### eu-west-1: checksum verification fails without `ignore_checksum`
In `eu-west-1`, the ETag returned for an uploaded object is **not the object's MD5** — the
storage layer encrypts at rest and returns a value that changes on every upload, even for
identical content. rclone's default post-copy verification compares the source MD5 against
that ETag, so **every uploaded object is reported "corrupted on transfer" and the failed copy
is then removed from the destination**: a default-configured transfer into `eu-west-1` fails
and leaves almost nothing behind.
DataRaven handles this automatically: every execution whose source or destination is a Fil One
`eu-west-1` location runs with `ignore_checksum: true`. The flag is resolved at execution time —
like plan limits — from the endpoint the transfer actually uses (an explicit endpoint URL wins
over the region field), so it follows a region change immediately; to opt out, set
`ignore_checksum: false` explicitly in the task's rclone configuration. Objects are still verified by size, and uploads still carry a `Content-MD5` that
the service checks on receipt — only the after-the-fact ETag comparison, which has no valid MD5
to compare against, is skipped.
`us-east-1` returns standard MD5 ETags and verifies checksums cleanly — no flag needed there.
Large objects uploaded via multipart are unaffected in both regions, because multipart ETags are
never plain MD5s and rclone skips that comparison automatically.
### DataRaven sets Fil One remotes to ListObjectsV2
Fil One has no dedicated rclone backend, so DataRaven configures it as `provider = Other`.
rclone picks the S3 listing API from that provider string. `Other` resolves to
**ListObjects (V1)**. The Fil One S3 compatibility page lists that API as a known risk in
`eu-west-1`.
The risk comes from the way V1 continues a listing. V1 continues with a `marker=` parameter.
The value of that parameter is a plain object key. In `eu-west-1`, that key carries a
server-internal suffix. The shape of the suffix changes at each page boundary:
```
page 1 -> 2 file-1000.txt[minio_cache:v2,return:]
page 2 -> 3 file-2000.txt[minio_cache:v2,id:d8582cd1-...,p:0,s:0]
```
Fil One recommends ListObjectsV2 with `ContinuationToken` instead. That token is an opaque
base64 string. Nothing between rclone and the service reads it as a key, so the suffix cannot
stop the listing.
We did not reproduce a failure. Under V1, a listing of 2,001 objects across three pages
completed correctly. This risk is latent, not an observed failure. DataRaven sets V2 because
Fil One documents the risk, and because V2 returns the same keys.
DataRaven therefore writes `list_version = 2` into every Fil One remote. We verified both
regions with 2,001 objects:
* Each request carries `list-type=2`.
* Each new page uses `continuation-token`.
* V2 returns the same key set as V1.
`us-east-1` returns bare keys as tokens, so that region does not have this risk. DataRaven
sets V2 in both regions, so the two regions keep the same config.
DataRaven writes this value into the **remote**, not into the task config. The
`--s3-list-version` flag in rclone is backend-global. rclone applies one value to every S3
remote in the transfer. This rule includes the remote on the other side of the transfer.
Some S3-compatible endpoints return all pages on V1, but fail on V2. A value set on the task
therefore applies V2 to the remote on the other side of the transfer. That remote can fail on
V2. The config file keeps V2 on the Fil One remote alone.
This design has two consequences:
* **V2 applies to every Fil One task, including tasks created before this change.** DataRaven
builds the remote again on each run, so there is no older task to correct.
* **V2 does not appear in the task config.** The detail page of the task shows no
`s3_list_version` value. That field stays empty until you set it.
To return to auto-detection in rclone, set `s3_list_version: 0` on the task. A value set on
the task has priority, because the command-line flag has priority over the config file.
### The two regions differ in API surface
`us-east-1` supports the larger slice of the S3 API. In `eu-west-1`:
* **Buckets are created and deleted in the dashboard only** — `CreateBucket` and `DeleteBucket`
are not available via the API (this is why `s3_no_check_bucket` is a task default).
* **ETags are not content MD5s** — see the checksum warning above; DataRaven defaults
`ignore_checksum` on for transfers touching this region.
* **Keys with consecutive slashes (`//`) are rejected** at upload with `Object name contains
unsupported characters`; `us-east-1` accepts them.
* **`CopyObject` is not confirmed.** Cloud-to-cloud transfers into Fil One never need it, but
rclone uses it to rewrite an object's metadata in place — updating only a modification time on
an otherwise unchanged object. If a sync keeps erroring on objects whose content hasn't
changed, this is the likely cause; setting `size_only` in the task config compares by size
alone and sidesteps the metadata rewrite.
* **Missing resources can return `AccessDenied` (403) instead of 404.** When troubleshooting,
a permission error against `eu-west-1` may actually mean a mistyped bucket or path.
* **V1 `ListObjects` markers carry internal state that changes** — see the listing section
above. DataRaven sets `list_version = 2` on the Fil One remote.
### Versioning and object lock are set at bucket creation — permanently
Versioning is opt-in when the bucket is created and can never be suspended afterwards. Object
lock (Governance or Compliance mode, with a default retention of 1 day to 100 years) requires
versioning and is likewise fixed at creation.
Two implications for transfer tasks:
* **On a versioned destination, every overwrite is a new full copy.** A sync task that
repeatedly rewrites changed objects grows the version stack, and every version is billed as
stored data. Deletes place a delete marker; prior versions remain and remain billed.
* **Locked versions cannot be overwritten or deleted.** On a bucket with object lock, sync
operations that would replace or prune an object inside its retention window fail with
`AccessDenied` — expected behavior, not a credentials problem.
## Performance Notes
Fil One publishes no numeric rate limits. `SlowDown` (503) responses are throttling, not
failure — rclone backs off and retries automatically; if they persist in execution logs, lower
`transfers` in the task's rclone configuration.
For large objects, throughput scales with multipart settings: raise `s3_chunk_size` (e.g.,
`64M`) and `s3_upload_concurrency` in the task's rclone configuration rather than file-level
`transfers`. Parts can be 5 MB–5 GB with up to 10,000 parts per object, so any object up to the
5 TB maximum is reachable with room to spare.
Fil One does **not** expire incomplete multipart uploads. If a large-object transfer is
interrupted partway, the already-uploaded parts remain in the bucket — invisible in listings
but billed as storage — until explicitly aborted. Fil One's
[multipart upload guide](https://docs.fil.one/storage/multipart-uploads) shows how to list and
abort stale uploads with the AWS CLI.
## What Transfers — and What Doesn't
Fil One stores object data and standard metadata, and supports more of the S3 feature set than
most S3-compatible providers — but not all of it (see the
[compatibility matrix](https://docs.fil.one/reference/s3-compatibility)). Migrating from S3:
| S3 feature | On Fil One |
| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Object data & metadata | Transfers |
| Versioning | Supported — but the current version of each object transfers; version history does not carry over. Enable versioning on the destination bucket at creation to keep history going forward |
| Object lock / legal hold | Supported — but per-object retention does not carry over. A destination bucket created with object lock applies its default retention to arriving objects |
| Object tags | Not supported today (planned) — tags do not carry over |
| ACLs / bucket policies | No ACL or policy model — canned ACL headers are silently ignored; all access is via authenticated keys |
| Server-side encryption config | Encryption at rest is always on, with keys held by Fil One; SSE-KMS and SSE-C are rejected |
| Storage classes / lifecycle | Single storage class; lifecycle rules are not supported today (planned) |
## Importing from rclone.conf
Fil One has no `provider =` value of its own in rclone, so real configs write `provider = Other`.
The importer recognizes Fil One remotes by their endpoint hostname (`*.s3.fil.one`) instead of
falling back to S3 Compatible, and a config that carries only the endpoint is fine — the region
is recovered from the hostname. Because access keys are region-scoped, each imported remote maps
cleanly to one region's secret. See the [rclone import guide](/guides/import-rclone-config).
# Filebase
Source: https://docs.dataraven.io/providers/filebase
Connect Filebase buckets as sources and destinations for transfer tasks.
Filebase — S3-compatible object storage with a single global endpoint, always-on encryption at
rest, and a generous 500 requests/second account limit. Two things shape how you use it: buckets
come in **two kinds** with very different egress bills (see
[below](#two-bucket-types-object-storage-and-ipfs)), and bucket names are **globally unique across
every Filebase account**, so a name that looks free may not be.
## At a Glance
| | |
| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| **Region** | Not needed (handled automatically as `auto`) |
| **Endpoint URL** | Optional — defaults to `https://s3.filebase.io` |
| **Credentials** | Access key — `access_key_id`, `secret_access_key` |
| **Task defaults** | `s3_no_check_bucket: true`, `fast_list: true`, `transfers: 32`, `checkers: 16`, `tpslimit: 400`. The remote itself sets `disable_http2 = true` |
| **Notable limits** | 500 requests/second per account. Objects up to 5 TB; single `PutObject` capped at 5 GB. Multipart parts 5 MiB-5 GiB, max 10,000 |
| **Provider docs** | [filebase.com/docs](https://filebase.com/docs/) |
## Credentials
Create a [secret](/secrets) with the **Filebase** provider type. The auth method is
`filebase_access_key`, which requires:
| Field | Description |
| ------------------- | -------------------------- |
| `access_key_id` | Filebase access key ID |
| `secret_access_key` | Filebase secret access key |
Find your access keys in the Filebase dashboard under **Access Keys**. Keys are account-wide,
with no per-region scoping to worry about.
## Location Setup
| Field | Value |
| ----------------- | ----------------------------------------------- |
| **region** | Leave blank |
| **endpoint\_url** | Optional — defaults to `https://s3.filebase.io` |
Leave the endpoint blank to use the default. Filebase supports both path-style and
virtual-hosted addressing, and HTTPS only — plain HTTP is redirected to 443.
Example:
| Field | Example value |
| -------------- | ------------------ |
| name | `Filebase Archive` |
| location\_type | Filebase |
| bucket\_name | `acme-backups` |
## Provider Quirks
### Two bucket types: object storage and IPFS
A Filebase bucket is created as either an **object-storage (S3)** bucket or an **IPFS** bucket,
and the choice is permanent. They do not behave the same:
| | Object storage (S3) | IPFS |
| ----------------------- | ------------------------------ | ---------------------------------------------------------- |
| Egress | Free on paid plans | **Metered and billed** |
| `x-amz-checksum-sha256` | Returned on every `GET`/`HEAD` | **Not available** — the CID comes back as `x-amz-meta-cid` |
| Gateway / Bitswap / RPC | n/a | Separately metered |
Pointing a large recurring transfer at an **IPFS** bucket bills egress on every read. If the
bucket is a transfer destination that something else later reads from, confirm which type it is
before the first run.
Not sure which kind a bucket is? Read any object's response headers: an `x-amz-meta-cid` header
means IPFS, an `x-amz-checksum-sha256` header means object storage.
The free plan includes **5 GB of storage (S3 + IPFS), free S3 egress, 5 GB of IPFS bandwidth,
500 IPFS pins, and 100,000 Class A / 1,000,000 Class B operations per month**. On a trial
migration the 5 GB storage ceiling is what you hit first — S3 egress is not metered on any plan,
so the 5 GB bandwidth figure applies to IPFS only.
### Bucket naming — no dots, and globally unique
Filebase does **not** allow dots (`.`) in bucket names, which S3 permits. An S3 bucket named
like a hostname — `backups.example.com` — needs a new name on the way in.
The full rules:
* 3-63 characters
* Lowercase letters, numbers, and hyphens only — no dots
* Must begin and end with a letter or number
* **Globally unique across all Filebase accounts**
DataRaven checks the first three when you create the location, so a non-conforming name fails
immediately with a clear message instead of surfacing later as an error from the service.
Global uniqueness is a separate failure mode and no validator can catch it: a perfectly legal name
may already belong to a stranger. When it does, Filebase answers `ListObjects` and `HeadObject`
with **403** rather than 404, because the bucket really does exist — it just is not yours. A
Filebase location that fails verification with "Access forbidden" is therefore as likely to have a
taken name as a permissions problem, and DataRaven's error message says so.
Object **keys** are case-sensitive UTF-8 up to 1,024 bytes. In DataRaven's end-to-end tests
Filebase accepted everything S3 does except keys with **non-normal path segments** — `//`, `/./`,
and `/../` are all rejected at upload. The status depends on the client: rclone sees a 403, the
AWS CLI a 301. Everything else round-tripped unchanged — `%`, `%2F`, `+`, `&`, `#`, `?`, embedded
newlines and tabs, NFC/NFD Unicode twins, case-only twins, Windows-reserved names, trailing dots
and spaces, a trailing-slash key, a 1,024-byte key, and a 256-byte path segment.
These keys are **not** normalized away in transit. A source bucket containing an object whose
key has `//` in it will fail that object on **every run**, permanently — the transfer reports
`AccessDenied`, which reads like a permissions problem but is a key-name rejection. Check source
keys before migrating into Filebase, and rename the offenders at the source.
### Checksums verify normally
Filebase returns real content MD5s as ETags for single-part uploads, so rclone's post-copy hash
verification works and DataRaven leaves it on. Of the 2,098 objects in our fixture transfer,
2,092 landed on Filebase and all 2,092 verified with zero hash differences. (The six that did not
land are key-name and source-side cases documented elsewhere on this page, not checksum failures.
One of the 2,092 matched on size alone, because its *source* ETag was a multipart ETag with no
rclone-written MD5 metadata to compare against — a property of the source object, not of
Filebase.)
Filebase also returns `x-amz-checksum-sha256` on every `GET`/`HEAD` of an object-storage bucket,
with `x-amz-checksum-type: FULL_OBJECT` set even for multipart objects — `ChecksumMode: ENABLED`
is a documented no-op because the header is always present. SHA-256 is the only algorithm
supported — but the others are not *rejected*. `PutObject` with
`--checksum-algorithm CRC32`, `CRC32C`, `SHA1`, or `CRC64NVME` returns `200`, and the object then
comes back with only a SHA-256 and `null` for every other algorithm. The requested algorithm is
discarded without an error.
Client-supplied checksums on `PutObject` are **accepted but not verified** — Filebase computes
its own. A `200` response is not the provider agreeing with your hash.
### Filebase accepts a lot that it silently ignores
This is the most important thing to know before a migration, and it is not visible from response
codes. Several S3 features return `200` and are then dropped, so a migration script that checks
only for errors will report complete success while losing the setting:
| Request | Response | What actually happens |
| -------------------------------------------------------------- | -------- | ---------------------------------------------------------------------------- |
| `--checksum-algorithm CRC32` / `CRC32C` / `SHA1` / `CRC64NVME` | `200` | Discarded; only SHA-256 is ever stored |
| `--storage-class GLACIER` / `DEEP_ARCHIVE` / `STANDARD_IA` / … | `200` | Discarded; `HeadObject` returns no storage class, listings report `STANDARD` |
| `--server-side-encryption aws:kms` / `AES256` | `200` | Discarded; `HeadObject` returns no `ServerSideEncryption` |
| `--acl public-read` | `200` | Discarded; `GetObjectAcl` still shows only the owner with `FULL_CONTROL` |
| `--tagging` | `200` | Discarded; see the tagging row in the table below |
A few reads are surprising in the same direction:
* **`GetBucketEncryption` returns an *empty* configuration**, not the AES-256 rule Filebase's
documentation describes — so it cannot be used to confirm encryption is on.
* **`GetBucketPolicy` ignores the `?policy` subresource entirely** and returns a
`ListBucketResult` — a bucket listing where a policy document belongs. Tooling that parses the
response without checking its shape will misread it.
* **`GetBucketLocation` returns the literal string `auto`**, consistent with the single global
endpoint.
* **`ListMultipartUploads` ignores the `prefix` parameter** and returns every in-progress upload
in the bucket. Filter the returned `Key` yourself, and abort each upload against its own key —
aborting one id against a different key fails with `NoSuchUpload`.
### DataRaven puts Filebase remotes on HTTP/1.1
Filebase serves HTTP/2 and recycles connections with a graceful `GOAWAY`. Because HTTP/2
multiplexes many requests onto one connection, a single `GOAWAY` lands on every upload in flight
on it, and Go cannot replay a `PUT` whose body it has already written. The result is a burst of
failures like:
```
cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY]
after Request.Body was written
```
Copying the same 1,500 objects twice at `transfers: 32` produced **18 failed uploads over HTTP/2
and none with HTTP/1.1**. rclone's retries did recover all 18, so this is wasted requests and log
noise rather than lost data — but those retries are not free against the 500 requests/second
ceiling, and a task configured with fewer retries would lose the objects outright.
DataRaven therefore sets `disable_http2` on the Filebase remote itself, not as a task-wide
setting, so the remote on the other side of the transfer keeps HTTP/2.
### Listing uses ListObjects V1, deliberately
Filebase has no `provider =` value of its own in rclone, so the generated remote says
`provider = Other`, which resolves to `ListObjects` V1. That is safe here: V1 and V2 return
identical key sets across page boundaries, and V1's `marker=` is a plain object key with no
server-internal suffix to misparse. Verified across 1,500 objects, and again with a one-key page
size so that every hostile key in the test set took a turn as the marker.
Set `s3_list_version` in the task's rclone configuration if you want V2 anyway.
### `UploadPartCopy` is not supported
Filebase supports the rest of the multipart surface — `CreateMultipartUpload`, `UploadPart`,
`CompleteMultipartUpload`, `AbortMultipartUpload`, `ListParts`, `ListMultipartUploads` — but not
`UploadPartCopy`.
This only matters for **Filebase-to-Filebase** transfers. rclone copies server-side when both ends
are the same provider, and reaches for `UploadPartCopy` above its own copy cutoff
(`--s3-copy-cutoff`, 4.656 GiB by default). Below that threshold plain `CopyObject` works normally.
**4.66 GiB is not a Filebase size limit** — it is simply where rclone starts calling an operation
Filebase does not implement. The call fails at every size: forcing it onto a 12 MiB object with
`--s3-copy-cutoff 5M` reproduces the failure exactly.
Filebase answers `UploadPartCopy` with **HTTP 200, `Content-Length: 0`, and an empty
`CopyPartResult`** — no ETag, and no error anywhere in the response. rclone's SDK rejects the empty
payload, cancels the copy, and reports:
```
operation error S3: UploadPartCopy, https response error StatusCode: 200,
deserialization failed, received empty response payload
```
Retries do not help, but rclone does fail loudly and leaves nothing behind.
Tooling that does not notice the missing `CopyPartResult` gets something worse than an error. The
part is registered as a **zero-byte part** — `ListParts` reports it with the MD5 of the empty
string — and `CompleteMultipartUpload` then **succeeds**, returning a normal-looking multipart
ETag for an object that is 0 bytes. Every call in the sequence returns `200`. A migration script
that checks only status codes will report success and have copied nothing.
There is no task setting that works around this today — DataRaven does not expose rclone's
`--disable copy` or `--s3-copy-cutoff`. A Filebase-to-Filebase task carrying objects larger than
\~4.66 GiB will fail on those objects. Transfers where only one side is Filebase are unaffected,
because rclone streams those through the worker rather than copying server-side.
## Performance and Rate Limits
Filebase allows **500 requests per second per account**, shared across the S3 API and the Platform
API. Higher limits are available on request from
[hello@filebase.com](mailto:hello@filebase.com). What that means for transfer tasks:
* **Every HTTP call counts as one request, regardless of size.** A 100-part multipart upload is
102 requests. Each `ListObjectsV2` page is one. Many small objects hit the request cap long
before bandwidth becomes the limit.
* **Anonymous reads from public buckets through the CDN are not counted** against the account.
* **`SlowDown` (503) responses are throttling, not failure** — and the only error Filebase
documents as retryable. rclone backs off and retries automatically. If they persist in the
execution logs, lower `transfers` in the task's rclone configuration.
* **DataRaven defaults to `transfers: 32`, `checkers: 16`, `tpslimit: 400`** for tasks touching a
Filebase location. The same 2,098-object transfer at these settings drew no `SlowDown` response
at all.
`tpslimit` is the part that is actually a rate limit — concurrency alone is not, since 32
in-flight transfers issue requests as fast as the round trip allows. It caps the whole rclone
process at 400 transactions/second, so a transfer with Filebase on one side sends it roughly
half that and leaves headroom for the Platform API and any concurrent task. Raise it in the
task's rclone configuration if Filebase has lifted your account limit. (Plan limits still apply:
Free-tier executions run with `transfers: 1`.)
* **`s3_upload_cutoff` is workload-dependent.** Single `PutObject` is capped at **5 GB**; past that
Filebase returns `EntityTooLarge` (400). rclone's 200 MiB default cutoff sits well inside that,
and multipart parts may be 5 MiB-5 GiB with up to 10,000 parts, so every object size up to the
5 TB maximum is reachable without tuning. Raise the cutoff (`256M`-`1G`) for small-object-heavy
transfers to keep them single-PUT — one request per object instead of several — and lower it
(`64M`-`256M`) for large-object transfers, scaling `s3_chunk_size` and `s3_upload_concurrency`
rather than file-level `transfers`.
* **`s3_no_check_bucket` is on by default.** Not because `CreateBucket` is unavailable — it is
fully supported — but because the existence check is one request per run that tells you nothing
about a bucket you already named.
Orphaned parts from a multipart upload that was neither completed nor aborted **count toward
your storage quota**. If a large-object transfer is interrupted partway, the already-uploaded
parts remain — invisible in listings but billed — until explicitly aborted with
`aws s3api abort-multipart-upload` or the equivalent.
## What Transfers — and What Doesn't
Filebase stores object data and standard metadata faithfully — content type, cache control,
content disposition, user metadata, and rclone's modification-time metadata all survive a round
trip. Migrating from S3:
| S3 feature | On Filebase |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Object data & metadata | Transfers |
| Versioning | Not supported — `GetBucketVersioning` reports no status, so version history does not carry over |
| Object lock / legal hold | No object lock or WORM model |
| Object & bucket tags | `PutObject --tagging` returns `200` but drops the tags; `GetObjectTagging` then fails with **HTTP 500**. `PutBucketTagging` is rejected outright — only Filebase's own `generateBucketCid` tag is allowed |
| ACLs / bucket policies | Canned ACL headers are accepted and ignored — `GetObjectAcl` reports only the owner with `FULL_CONTROL` whatever you set. No bucket policy model, and `GetBucketPolicy` returns a bucket listing rather than an error |
| Server-side encryption config | Encryption at rest is always on and not configurable; SSE headers are accepted and ignored. `GetBucketEncryption` returns an empty configuration, so it confirms nothing |
| Storage classes / lifecycle | Single storage class — cold-tier classes are accepted and silently discarded, so an object you asked to archive stays hot at full price. Lifecycle, replication, inventory, and event notifications all read back empty |
| Archive tiers / restore | No `RestoreObject` — but no archive tier either, so nothing ever needs rehydrating |
`CopyObject` is supported with both ends on Filebase, along with range requests, conditional
headers, and pre-signed URLs across `GetObject`, `PutObject`, `HeadObject`, and multipart.
`DeleteObjects` accepts up to 1,000 keys per request.
## Importing from rclone.conf
Filebase has no `provider =` value of its own in rclone, so real configs write
`provider = Other`. The importer recognizes Filebase remotes by their endpoint hostname
(`s3.filebase.io` or the legacy `s3.filebase.com`) instead of falling back to S3 Compatible. See
the [rclone import guide](/guides/import-rclone-config).
The legacy `https://s3.filebase.com` endpoint is still accepted — enter it in the endpoint field
if your existing tooling still points there.
# Google Cloud Storage
Source: https://docs.dataraven.io/providers/google-cloud-storage
Connect Google Cloud Storage buckets as sources and destinations for transfer tasks.
Google Cloud Storage buckets, authenticated with a service account key.
## At a Glance
| | |
| ----------------- | ----------------------------------------------------- |
| **Region** | Optional |
| **Endpoint URL** | Not needed |
| **Credentials** | Service account — `service_account_json` |
| **Task defaults** | None — GCS works well with rclone's standard behavior |
## Credentials
Create a [secret](/secrets) with the **Google Cloud Storage** provider type. The auth method is
`gcs_service_account`, which requires:
| Field | Description |
| ---------------------- | ---------------------------------------- |
| `service_account_json` | The entire service account JSON key file |
The `service_account_json` field expects the **entire JSON key file** contents as a string. Copy
the full JSON output from `gcloud iam service-accounts keys create` — don't extract individual
fields.
## Location Setup
| Field | Value |
| ----------------- | ------------------------------------------------------------------------------------------- |
| **region** | Optional — GCS handles routing automatically; set it for documentation purposes if you like |
| **endpoint\_url** | Leave blank |
Example:
| Field | Example value |
| -------------- | -------------------- |
| name | `GCS Data Lake` |
| location\_type | Google Cloud Storage |
| bucket\_name | `acme-data-lake` |
## Importing from rclone.conf
GCS remotes appear in `rclone.conf` as `type = gcs` (or `type = google cloud storage`). The
importer needs the credentials **inline** via `service_account_credentials`; a remote that uses
`service_account_file` (a local file path) cannot be imported — paste the JSON inline in your
`rclone.conf` first, or create the secret manually. See the
[rclone import guide](/guides/import-rclone-config).
# Hetzner
Source: https://docs.dataraven.io/providers/hetzner
Connect Hetzner Object Storage buckets as sources and destinations for transfer tasks.
Hetzner Object Storage — S3-compatible, with the region encoded in the endpoint hostname.
## At a Glance
| | |
| ----------------- | ------------------------------------------------- |
| **Region** | Not needed (embedded in endpoint) |
| **Endpoint URL** | Required — e.g., `hel1.your-objectstorage.com` |
| **Credentials** | Access key — `access_key_id`, `secret_access_key` |
| **Task defaults** | `s3_no_check_bucket: true`, `fast_list: true` |
## Credentials
Create a [secret](/secrets) with the **Hetzner** provider type. The auth method is
`hetzner_access_key`, which requires:
| Field | Description |
| ------------------- | ------------------------- |
| `access_key_id` | Hetzner access key ID |
| `secret_access_key` | Hetzner secret access key |
Generate credentials from the Hetzner Cloud Console under **Object Storage**.
## Location Setup
| Field | Value |
| ----------------- | ------------------------------------------------------ |
| **region** | Leave blank — the endpoint hostname carries the region |
| **endpoint\_url** | Required — `.your-objectstorage.com` |
The endpoint URL includes the region (e.g., `hel1.your-objectstorage.com`,
`nbg1.your-objectstorage.com`, `fsn1.your-objectstorage.com`).
Example:
| Field | Example value |
| -------------- | ----------------------------- |
| name | `Hetzner Helsinki` |
| location\_type | Hetzner |
| bucket\_name | `acme-backups` |
| endpoint\_url | `hel1.your-objectstorage.com` |
## Importing from rclone.conf
Hetzner remotes appear in `rclone.conf` as `type = s3` with `provider = Hetzner`. See the
[rclone import guide](/guides/import-rclone-config).
# Impossible Cloud
Source: https://docs.dataraven.io/providers/impossible-cloud
Connect Impossible Cloud buckets as sources and destinations for transfer tasks.
Impossible Cloud — EU-sovereign S3-compatible object storage with six European regions and one
in New York, a single STANDARD storage class, and pay-per-use pricing with no egress or
per-request fees. Two things shape how you use it: every bucket is **pinned to the region it was
created in** and only answers on that region's endpoint (see
[below](#every-bucket-answers-on-one-endpoint)), and the fair-use policy expects monthly egress
to stay within the volume you store — which matters when Impossible Cloud is the *source* of a
transfer.
## At a Glance
| | |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Region** | Required unless an endpoint is set — the region the bucket was created in, e.g., `eu-central-2` |
| **Endpoint URL** | Optional — derived from region as `https://.storage.impossibleapi.net` |
| **Credentials** | Access key — `access_key_id`, `secret_access_key` (IAM) |
| **Task defaults** | `s3_no_check_bucket: true`, `fast_list: true`, `transfers: 32`, `checkers: 16` |
| **Notable limits** | Objects up to 50 TiB; keys up to 1,024 bytes; multipart parts at least 5 MiB; 100 buckets per account; STANDARD is the only storage class. No egress or request fees, but monthly egress is expected to stay within stored volume |
| **Provider docs** | [docs.impossiblecloud.com](https://docs.impossiblecloud.com/impossible-cloud-help/) |
## Credentials
Create a [secret](/secrets) with the **Impossible Cloud** provider type. The auth method is
`impossible_cloud_access_key`, which requires:
| Field | Description |
| ------------------- | ---------------------------------- |
| `access_key_id` | Impossible Cloud access key ID |
| `secret_access_key` | Impossible Cloud secret access key |
Generate access keys in the Impossible Cloud console under **IAM → Access keys**. The secret is
shown once, at creation. Keys are IAM credentials, not per-region: the same key pair was used
against `us-east-1`, `eu-central-2`, and `eu-west-1` in our verification runs.
A transfer makes these calls, so a scoped IAM policy needs the matching actions: `s3:ListBucket`
and `s3:GetObject` on the source; `s3:ListBucket`, `s3:PutObject`, and the multipart family
(`CreateMultipartUpload`, `UploadPart`, `UploadPartCopy`, `CompleteMultipartUpload`,
`AbortMultipartUpload`, `ListMultipartUploads`) on the destination; `s3:DeleteObject` as well
for `sync` tasks. DataRaven never calls `CreateBucket`. We verified with an unrestricted key —
a minimum-policy run is listed under [what we did not verify](#what-we-did-not-verify).
## Location Setup
| Field | Value |
| ----------------- | ------------------------------------------------------------------------------------ |
| **region** | Required unless `endpoint_url` is set — must be the region the bucket was created in |
| **endpoint\_url** | Optional — derived from region as `https://.storage.impossibleapi.net` |
Pick the bucket's region from the dropdown and the endpoint is derived automatically. An explicit
`endpoint_url` overrides the derived value and satisfies the region requirement on its own — the
region is read back out of the hostname, so an endpoint-only location behaves exactly like a
region-only one. One of the two must be set.
Region names are Impossible Cloud's own and do **not** map to AWS's:
| Region | Location |
| -------------- | ---------------------- |
| `eu-central-2` | Frankfurt, Germany |
| `eu-west-1` | Amsterdam, Netherlands |
| `eu-west-2` | London, UK |
| `eu-west-3` | Paris, France |
| `eu-east-1` | Poznań, Poland |
| `eu-north-1` | Copenhagen, Denmark |
| `us-east-1` | New York, USA |
`eu-west-1` is Amsterdam here and Ireland on AWS; `eu-central-2` is Frankfurt here and Zurich on
AWS. Read the bucket's region off the Impossible Cloud console rather than from habit.
Example:
| Field | Example value |
| -------------- | ---------------- |
| name | `Impossible EU` |
| location\_type | Impossible Cloud |
| bucket\_name | `acme-sovereign` |
| region | `eu-central-2` |
## Provider Quirks
### Every bucket answers on one endpoint
Impossible Cloud lists **every** bucket in the account from **every** regional endpoint, but a
bucket's data is only reachable on the endpoint of the region it was created in, and that region
cannot be changed afterwards. A request for a `us-east-1` bucket sent to `eu-west-1` is not
redirected — it fails:
```
400 IncorrectEndpoint: The specified bucket exists in another Region.
Direct requests to the correct endpoint.
```
DataRaven reports this at verification as **"The bucket is not served by this region's endpoint"**
(`REGION_MISMATCH`) rather than as a permissions problem. Fix the location's region; nothing about
the key is wrong.
The same code comes back for a bucket name you do not own. Bucket names are shared across every
Impossible Cloud account (`CreateBucket` on a taken name answers "The bucket namespace is shared by
all users of the system"), so a location pointed at a stranger's bucket fails with
`IncorrectEndpoint` too. The verification message says so.
### Bucket names follow the S3 rules, dots included
The rules, as `CreateBucket` itself states them:
* 3-63 characters
* Lowercase letters, numbers, dots, and hyphens
* Must begin and end with a letter or number
* No `..`, and not shaped like an IP address
* **Shared across all Impossible Cloud accounts**
Dots are allowed, so an S3 bucket named like a hostname — `backups.example.com` — carries over
unchanged. DataRaven checks the syntactic rules when you
create the location; global uniqueness surfaces at verification as described above. Uppercase is
rejected by the service rather than downcased, so the check runs on the name exactly as typed.
### Object keys: everything S3 allows, including what the docs say it rejects
Impossible Cloud's published limitations say keys may not begin with `/`, may not contain empty
path segments, may not have a segment longer than 255 bytes, and that an object cannot share its
name with a "folder" (`xxx/yyy` alongside `xxx/yyy/file`). In our end-to-end runs **none of those
were enforced**. `PutObject` accepted, stored, listed, and served back a leading-slash key, `//`,
`/./`, `/../`, a key ending in `/`, a 256-byte segment, and `conflict` next to `conflict/child.txt`
— along with `%`, `%2F`, `+`, `&`, `#`, `?`, embedded newlines and tabs, NFC/NFD Unicode twins,
case-only twins, Windows-reserved names, trailing dots and spaces, a 1,024-byte key, and CJK plus
emoji. A 1,025-byte key fails with `KeyTooLongError`, the only key rejection we found.
Two of the awkward shapes are limited by rclone rather than by Impossible Cloud, on every S3
provider alike:
* **`/./` and `/../` segments cannot be read from any S3 source.** rclone cannot form a request
for them ("failed to open source object: object not found"), so they fail on every run. Rename
them at the source.
* **`//` (an empty segment) transfers only through a recursive listing.** rclone's per-directory
listing discards the entry ("Entry doesn't belong in directory … (too short) - ignoring"), and
the recursive `ListR` path copies it. DataRaven sets `fast_list` for Impossible Cloud locations
partly for this reason — turn it off and those keys are silently skipped.
### Checksums verify normally
Single-part uploads get a real content-MD5 ETag, so rclone's post-copy hash check works and
DataRaven leaves it on. In our S3 → Impossible Cloud fixture transfer, **2,097 of 2,099 objects
landed and 2,095 verified by hash**; the other two matched on size because their *source* objects
were multipart uploads with no rclone-written MD5 to compare against. The two that did not land
were `GLACIER` and `DEEP_ARCHIVE` objects, which fail on the source side until restored.
Multipart uploads get the usual `-` ETag. rclone stores the whole-object
MD5 in `X-Amz-Meta-Md5chksum` for its own multipart uploads, so those still verify; objects
assembled by other tools are compared by size.
Impossible Cloud also stores every `x-amz-checksum-*` algorithm you send — `CRC32`, `CRC32C`,
`SHA1`, `SHA256`, and `CRC64NVME` all came back on `HeadObject` with `ChecksumType: FULL_OBJECT` —
and it **enforces `Content-MD5`**: a `PutObject` or `UploadPart` whose MD5 header does not match the
body is rejected with `BadDigest`.
That enforcement has one migration-visible consequence. An AWS S3 object stored with **SSE-KMS**
has an ETag that is not its MD5, rclone trusts the ETag and sends it as `Content-MD5`, and
Impossible Cloud answers `BadDigest` — the transfer fails for that object on every run. AWS
itself rejects the same copy the same way, so this is a property of the source object and
rclone, not of Impossible Cloud. `ignore_checksum` does not help (the header is still sent);
forcing those objects through multipart with a low `s3_upload_cutoff` does, or re-upload them at
the source without KMS.
### Standard HTTP headers do not survive — only Content-Type does
This is the most important thing to know before a migration, because nothing reports it: the
upload returns `200` and the header is simply gone. Verified with `PutObject`, `HeadObject`,
`GetObject`, a presigned GET, and `CopyObject` with `REPLACE`:
| Header | On Impossible Cloud |
| --------------------- | ------------------------------------------------------- |
| `Content-Type` | Stored and returned |
| `Cache-Control` | **Dropped** |
| `Content-Disposition` | **Dropped** |
| `Content-Language` | **Dropped** |
| `Content-Encoding` | **Dropped** |
| `Expires` | **Dropped** |
| `x-amz-meta-*` | Stored, up to the 2 KB limit (`MetadataTooLarge` above) |
| Object tags | Stored (`PutObject --tagging`, `PutObjectTagging`) |
Objects stored compressed with `Content-Encoding: gzip` arrive as raw gzip bytes served under
their `Content-Type`. Browsers and HTTP clients will not decompress them. If a bucket is served
directly to clients, expand those objects before migrating or serve them through something that
can set the header.
rclone's own modification-time metadata (`X-Amz-Meta-Mtime`) round-trips, including pre-1970 and
post-2038 timestamps, so incremental runs still skip unchanged objects correctly.
### Rejected outright rather than silently ignored
Impossible Cloud mostly says no to what it does not support, which is the behaviour you want from
a migration target: a rejected request is visible, a discarded one is not. These are all the
response to a plain `PutObject`:
| Request | Response |
| --------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| `--storage-class GLACIER` / `STANDARD_IA` / `DEEP_ARCHIVE` / … | `400 InvalidStorageClass: Only the following storage classes are currently supported: STANDARD` |
| `--acl public-read` / `authenticated-read` / `bucket-owner-full-control` | `400 AccessControlListNotSupported: The bucket does not allow ACLs` |
| `--acl private` | Accepted — this is what DataRaven sends |
| `--server-side-encryption AES256` | Stored; `HeadObject` reports `AES256` |
| `--server-side-encryption aws:kms` | `200`, but **stored as `AES256`** — the only silent substitution we found |
| `PutObjectAcl` / `GetObjectAcl` / `GetBucketPolicy` / `GetBucketLifecycleConfiguration` | `NotImplemented` |
DataRaven never sets a storage class or SSE header, so none of these affect a transfer. They do
affect what a source-side setting *means* after migration: an object that was `STANDARD_IA` on S3
is `STANDARD` here.
### Multipart, `UploadPartCopy`, and server-side copy all work
The full multipart surface is implemented, and we exercised it hard: a 5.08 GiB object assembled
server-side from **52 `UploadPartCopy` calls**, a three-part upload with non-uniform 6 + 5 + 1 MiB
parts, and an abandoned upload that `ListMultipartUploads` reports correctly, with the `prefix`
parameter honoured. Parts must be at least 5 MiB (`EntityTooSmall` otherwise), the S3 standard.
Server-side `CopyObject` works between buckets in the **same region**, so an Impossible Cloud →
Impossible Cloud task within one region copies without moving bytes through the worker — 31 test
objects copied server-side in about a second. **Across regions** the copy streams through the
worker, because rclone will not attempt server-side copies between remotes with different
endpoints: our `us-east-1` → `eu-central-2` fixture run moved all 5.5 GiB through the client.
Parts from a multipart upload that was interrupted and never completed or aborted are not
visible in listings but do occupy storage. Impossible Cloud has no lifecycle rules to expire
them (`GetBucketLifecycleConfiguration` is `NotImplemented`), so clean up with
`aws s3api abort-multipart-upload` or `rclone cleanup` after an interrupted large-object
transfer.
### Versioning and object lock
Versioning is a per-bucket switch that can be enabled, suspended, and re-enabled. On a versioned
destination:
* every overwrite creates a new version and every delete leaves a delete marker — including the
deletes a `sync` task performs;
* an unchanged object on a re-run is **skipped**, not re-written, so scheduled tasks do not pile up
identical versions;
* a transfer reads and writes **current** versions only. Version history on the source does not
carry over.
Object lock (compliance and governance retention, legal hold) is offered at bucket creation and
cannot be changed afterwards. We did not run a transfer into a locked bucket — see
[below](#what-we-did-not-verify).
## Performance and Rate Limits
Impossible Cloud publishes no request rate limit and charges nothing per request. We looked for a
ceiling and did not find one:
* **Small objects, one client, `us-east-1`:** 1,500 objects at `transfers` 16 / 32 / 64 ran at
93 / 187 / 250 objects per second; 10,000 objects at 64 ran at **357 objects/s** and at 128 at
**666 objects/s**. Zero errors and no `SlowDown`, `429`, or `503` at any setting. Listing the
resulting 24,500 objects took 3 seconds; deleting them took 71 seconds.
* **Large objects:** the fixture transfers (5.5 GiB, 2,100 objects, three regions) were bounded by
our client's uplink, not by the service. Server-side copy within a region is effectively free.
* **DataRaven defaults to `transfers: 32`, `checkers: 16`** for tasks touching an Impossible
Cloud location. That is nowhere near what the service accepted; it is a conservative starting
point that leaves headroom for the remote on the other side of the transfer, which may well
have a limit of its own. Raise it in the task's rclone configuration if the peer can take it.
(Plan limits still apply: Free-tier executions run with `transfers: 1`.) There is no
`tpslimit`, deliberately — there is no request budget to protect.
* **HTTP/1.1 only.** The endpoints do not negotiate HTTP/2, so each request owns its connection
and a server-side connection close costs at most one retryable request. No `disable_http2`
setting is needed.
* **`s3_no_check_bucket` is on by default**, and not because `CreateBucket` is unavailable — it
works with an ordinary key. With the check on, rclone would create a mistyped destination
bucket name on the spot, or collide with someone else's bucket of that name. Off, a typo fails
verification instead.
**Egress is free but not unlimited.** The fair-use policy expects your monthly egress to stay
within your stored volume — store 100 TB, download up to 100 TB a month. A one-off migration
*out* of Impossible Cloud that moves most of a bucket is within that; a recurring task that
re-reads a large bucket every day is not. Impossible Cloud's commercial team arranges exceptions
for such workloads.
## What Transfers — and What Doesn't
Impossible Cloud stores object data, `Content-Type`, user metadata, object tags, and rclone's
modification-time metadata faithfully. Migrating from S3:
| S3 feature | On Impossible Cloud |
| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Object data & metadata | Transfers — data, `Content-Type`, user metadata, mtime. `Cache-Control`, `Content-Disposition`, `Content-Encoding`, `Content-Language`, and `Expires` are dropped by the service |
| Versioning | Supported per bucket; current versions transfer, history does not |
| Object lock / legal hold | Supported by the service (compliance, governance, legal hold, set at bucket creation); retention settings are not carried by a transfer |
| Object & bucket tags | Object tags are supported by the service, but rclone does not copy tags, so they are not carried by a transfer |
| ACLs / bucket policies | No ACLs — any canned ACL other than `private` is rejected, `PutObjectAcl` is `NotImplemented`. No bucket policies (`GetBucketPolicy` is `NotImplemented`); access is IAM-only |
| Server-side encryption config | SSE-S3 (`AES256`) only, per object or as a bucket default; `aws:kms` is silently stored as `AES256`, SSE-C is not offered. DataRaven sends no SSE header, so objects follow the bucket default |
| Storage classes / lifecycle | `STANDARD` only — any other class is rejected with `InvalidStorageClass`. No lifecycle rules |
| Archive tiers / restore | No archive tier and no `RestoreObject`; nothing ever needs rehydrating. Cold objects on the *source* must be restored before they can be read |
`CopyObject` and `UploadPartCopy` are supported within a region, along with range requests and
presigned URLs (`rclone link` works). `DeleteObjects` is listed as supported by Impossible Cloud;
rclone issues one `DeleteObject` per key regardless.
## What We Did Not Verify
Every claim above comes from a run against live `us-east-1`, `eu-central-2`, and `eu-west-1`
buckets on 2026-09-04 with rclone 1.75.0. These did not get a run and are stated on Impossible
Cloud's documentation alone:
* A transfer with a **minimum-privilege IAM policy** — we used an unrestricted key.
* A transfer **into an object-locked bucket**, and what a `sync` delete does there.
* The **50 TiB** object ceiling and the maximum part count — our largest object was 5.08 GiB in
52 parts.
* Behaviour under the **fair-use egress** threshold — our egress stayed far below stored volume.
* The four regions we have no bucket in: `eu-west-2`, `eu-west-3`, `eu-east-1`, `eu-north-1`.
## Importing from rclone.conf
Impossible Cloud remotes appear in `rclone.conf` as `type = s3` with
`provider = ImpossibleCloud`. rclone's own config wizard writes the endpoint without a scheme
(`endpoint = eu-central-2.storage.impossibleapi.net`) and often no region; the importer accepts
both and recovers the region from the endpoint hostname. See the
[rclone import guide](/guides/import-rclone-config).
# MEGA S4
Source: https://docs.dataraven.io/providers/mega-s4
Connect MEGA S4 buckets as sources and destinations for transfer tasks.
MEGA's S3-compatible object storage. Buckets are **global** — readable through any regional
endpoint — and both object keys and bucket names are validated more strictly than on S3 or Azure.
## At a Glance
| | |
| ------------------ | ---------------------------------------------------------------------------------------- |
| **Region** | Required — e.g., `eu-amsterdam` |
| **Endpoint URL** | Derived from region — `https://s3..megas4.com` |
| **Credentials** | Access key — `access_key_id`, `secret_access_key` |
| **Task defaults** | `s3_no_check_bucket: true`, `fast_list: true`, `transfers: 32`, `checkers: 16` |
| **Notable limits** | \~40–50 upload requests/s per account; designed for up to \~20M objects per account |
| **Provider docs** | [MEGA S4](https://mega.io/objectstorage) · [S4 help centre](https://help.mega.io/megas4) |
`fast_list` is on by default — MEGA's own
[rclone guide](https://help.mega.io/megas4/setup-guides/rclone) recommends `--fast-list` when
scanning large directory trees: fewer listing calls, at the cost of more memory. See
[Performance and Rate Limits](#performance-and-rate-limits) for the rest of the tuning picture.
## Credentials
Create a [secret](/secrets) with the **MEGA S4** provider type. The auth method is
`mega_s4_access_key`, which requires:
| Field | Description |
| ------------------- | ------------------------- |
| `access_key_id` | MEGA S4 access key ID |
| `secret_access_key` | MEGA S4 secret access key |
Create access keys from the MEGA S4 console — see
[MEGA's S4 help centre](https://help.mega.io/megas4) for a walkthrough. S4 exposes a real
[IAM API](https://github.com/meganz/s4-specs) (users, groups, policies), so keys can be scoped
down with a bucket policy rather than granting account-wide access — worth doing for a transfer
key that only needs one bucket.
**One key works for every region**: buckets are global and reachable through any regional
endpoint, so you do not need a secret per region.
## Location Setup
| Field | Value |
| ----------------- | ------------------------------------------------------------------ |
| **region** | Required — e.g., `eu-amsterdam` |
| **endpoint\_url** | Optional — derived from region as `https://s3..megas4.com` |
Because buckets are global, the region picks the endpoint closest to your workload rather than
where the data lives. Available regions are `eu-amsterdam`, `eu-luxembourg`, `eu-paris`,
`eu-barcelona`, `ca-montreal`, `ca-vancouver`, and `ap-tokyo`.
Legacy `s3..s4.mega.io` endpoints still work — enter one in the endpoint field to
override the derived value. MEGA also operates a global endpoint, `s3.g.megas4.com` (currently
routed to Amsterdam), and publishes numbered per-datacenter hostnames such as
`s3.eu-amsterdam-1.megas4.com`; the derived `s3..megas4.com` names are stable aliases
for those, so any of the three forms can go in the endpoint field.
Example:
| Field | Example value |
| -------------- | ---------------- |
| name | `MEGA S4 EU` |
| location\_type | MEGA S4 |
| bucket\_name | `acme-transfers` |
| region | `eu-amsterdam` |
## Provider Quirks
### Object key restrictions
MEGA S4 validates object keys more strictly than S3 and Azure Blob, and rejects the request
rather than normalizing the key. A transfer carrying any of these will fail on the affected
objects mid-run:
* Consecutive forward slashes (`//`)
* `/./` or `/../` anywhere in the key, or a key equal to `..`, beginning `../`, or ending `/..`
* Keys shorter than 1 or longer than 1024 characters
* Keys ending in `/` on an object that has content
S3 and Azure Blob both permit `//`, so keys carried over from either can hit this. Check your
source keys before transferring to MEGA S4.
Checking source keys is a manual step today — a pre-flight scan that flags incompatible keys
before the transfer starts is planned.
### Bucket naming
Bucket names are also stricter than S3's:
* 3–63 characters
* Lowercase letters, numbers, dots, and hyphens only
* Must begin and end with a letter or number
* No `..`, `.-`, or `-.` sequences
* Not formatted as an IP address
* No reserved prefix (`xn--`, `sthree-`, `amzn-s3-demo-`) or suffix (`-s3alias`, `--ol-s3`,
`.mrap`, `--x-s3`, `--table-s3`)
DataRaven checks these rules when you create the location, so a non-conforming name fails
immediately with a clear message instead of surfacing later as a confusing `AccessDenied` from
the service.
### Objects uploaded through MEGA's own apps
Files placed in S4 by MEGA's own software don't store an MD5 hash in their ETag, so S3 clients
that verify checksums — rclone included — can fail integrity validation when copying those
objects **out** of S4. Objects written through the S3 API (including everything DataRaven
transfers in) are unaffected. If your source bucket holds MEGA-app-uploaded data, set
`ignore_checksum` in the task's rclone configuration — the transfer then verifies by size and
modification time instead, as MEGA's
[rclone guide](https://help.mega.io/megas4/setup-guides/rclone) recommends.
## Performance and Rate Limits
S4 rate-limits uploads at roughly **40–50 requests per second per account**, and MEGA's
[rates guidance](https://help.mega.io/megas4/setup-guides/mega-s4-rate-limits-and-performance-guidance) puts the
practical ceiling at \~42–45 objects per second regardless of concurrency. For transfer tasks
that means:
* **Many small objects hit the request cap before bandwidth does.** Larger objects and multipart
uploads make far better use of throughput; raising `transfers` past \~45 buys nothing.
* **`SlowDown` responses are throttling, not failure.** rclone backs off and retries
automatically; if they persist in the execution logs, lower `transfers` in the task's rclone
configuration.
* **DataRaven applies MEGA's suggested starting point — `transfers: 32`, `checkers: 16` — as the
default** for tasks that touch an S4 location. Override either in the task's rclone
configuration if your workload calls for it. (Plan limits still apply: Free-tier executions
run with `transfers: 1`.)
* **`s3_upload_cutoff` is workload-dependent — MEGA's guidance pulls it in opposite directions.**
Tune it for whichever shape dominates the transfer:
* *Mostly large objects (1GB+):* set the cutoff low (`64M`–`256M`) so big files always use
multipart, and scale `s3_chunk_size` / `s3_upload_concurrency` rather than file-level
parallelism.
* *Mostly small objects:* set the cutoff high (`256M`–`1G`) so files stay single PUT — one
request per object instead of several, and single-PUT objects keep their MD5 ETags.
* **`s3_no_head` trades verification for request budget.** It saves one API call per object,
which matters against the request cap — but rclone then assumes every upload succeeded exactly
as sent (size, modtime, metadata), so a truncated or corrupted upload can go undetected. Leave
it off unless the request cap is your actual bottleneck, and avoid combining it with
`ignore_checksum`: together they leave a transfer with essentially no post-upload
verification.
* **HTTP/1.1 only.** `s3..megas4.com` and the legacy `s3..s4.mega.io` hosts do
not negotiate HTTP/2 (as of 2026-09-07). Each request owns its connection, so a server-side
connection close costs at most one retryable request. No `disable_http2` setting is necessary.
* **Accounts are designed for up to \~20 million objects.** For workloads beyond that — or
bursty, object-heavy ones — MEGA asks that you contact the S4 team at
[S4@mega.io](mailto:S4@mega.io).
* **Large deletes:** MEGA recommends moderate `DeleteObjects` batches (\~100 objects) and not
running massive delete workloads alongside heavy uploads — relevant for `sync` tasks that
prune many destination objects.
## What Transfers — and What Doesn't
S4 stores object data and standard metadata, but several S3 features have no S4 equivalent (per
MEGA's [published API spec](https://github.com/meganz/s4-specs)). If you're migrating from S3
with versioning and lifecycle rules, here is what arrives:
| S3 feature | On MEGA S4 |
| ----------------------------- | ------------------------------------------------------------------------------------------------- |
| Object data & metadata | Transfers |
| Versioning | Not supported — the current version of each object transfers; version history does not carry over |
| Object lock / legal hold | Not supported — retention settings do not carry over |
| Object tags | Not supported — tags do not carry over |
| Server-side encryption config | The S3 SSE options (SSE-S3, SSE-KMS, SSE-C) do not exist on S4 |
| Storage classes | Every object lands as `STANDARD` — there are no other classes and no lifecycle rules |
If you rely on version history or object-lock retention at the source, keep the source bucket —
a transfer to S4 carries the current objects, not those controls.
## Importing from rclone.conf
MEGA S4 remotes appear in `rclone.conf` as `type = s3` with `provider = Mega`. Configs written
before rclone gained that provider value are recognized by their endpoint hostname instead
(`*.megas4.com` or the legacy `*.s4.mega.io`).
Note that rclone's `mega` **type** is the consumer MEGA cloud drive, a different product — it is
not supported and is skipped by the importer. MEGA S4 object storage always arrives as `type = s3`.
See the [rclone import guide](/guides/import-rclone-config).
# Oracle Object Storage
Source: https://docs.dataraven.io/providers/oracle-object-storage
Connect Oracle Object Storage buckets as sources and destinations for transfer tasks.
Oracle Cloud Infrastructure Object Storage, connected through its S3-compatible endpoint.
## At a Glance
| | |
| ----------------- | ------------------------------------------------- |
| **Region** | Required |
| **Endpoint URL** | Required — Oracle S3-compatible endpoint |
| **Credentials** | Access key — `access_key_id`, `secret_access_key` |
| **Task defaults** | `s3_no_check_bucket: true` |
## Credentials
Create a [secret](/secrets) with the **Oracle Object Storage** provider type. The auth method is
`oracle_object_storage_s3_access_key`, which requires:
| Field | Description |
| ------------------- | ------------------------------ |
| `access_key_id` | Customer Secret Key access key |
| `secret_access_key` | Customer Secret Key secret |
Oracle calls S3-compatible credentials **Customer Secret Keys** — generate one in the OCI console
under your user's settings.
## Location Setup
| Field | Value |
| ----------------- | ---------------------------------------- |
| **region** | Required — your OCI region |
| **endpoint\_url** | Required — Oracle S3-compatible endpoint |
Oracle Object Storage uses an S3-compatible API. Both region and endpoint URL are required; the
endpoint follows the shape `https://.compat.objectstorage..oraclecloud.com`.
Example:
| Field | Example value |
| -------------- | ----------------------------------------------------------------------- |
| name | `OCI Archive` |
| location\_type | Oracle Object Storage |
| bucket\_name | `acme-archive` |
| region | `us-ashburn-1` |
| endpoint\_url | `https://mynamespace.compat.objectstorage.us-ashburn-1.oraclecloud.com` |
## Importing from rclone.conf
The importer has no dedicated mapping for Oracle's S3-compatible remotes, so they import as
**S3 Compatible** — fully functional, since the endpoint and credentials carry over. To use the
first-class Oracle Object Storage type (and its defaults), create the secret and location
manually instead. See the [rclone import guide](/guides/import-rclone-config).
# Rabata
Source: https://docs.dataraven.io/providers/rabata
Connect Rabata Object Storage buckets as sources and destinations for transfer tasks.
Rabata Object Storage — S3-compatible, with a region-specific endpoint.
## At a Glance
| | |
| ----------------- | ------------------------------------------------- |
| **Region** | Required — e.g., `us-east-1` |
| **Endpoint URL** | Required — `s3..rabata.io` |
| **Credentials** | Access key — `access_key_id`, `secret_access_key` |
| **Task defaults** | `s3_no_check_bucket: true`, `fast_list: true` |
## Credentials
Create a [secret](/secrets) with the **Rabata** provider type. The auth method is
`rabata_access_key`, which requires:
| Field | Description |
| ------------------- | ------------------------ |
| `access_key_id` | Rabata access key ID |
| `secret_access_key` | Rabata secret access key |
Generate access keys from the Rabata console.
## Location Setup
| Field | Value |
| ----------------- | ---------------------------------- |
| **region** | Required — e.g., `us-east-1` |
| **endpoint\_url** | Required — `s3..rabata.io` |
Both region and endpoint URL are required, and the endpoint includes the region (e.g.,
`s3.us-east-1.rabata.io`).
Example:
| Field | Example value |
| -------------- | ------------------------ |
| name | `Rabata US` |
| location\_type | Rabata |
| bucket\_name | `acme-storage` |
| region | `us-east-1` |
| endpoint\_url | `s3.us-east-1.rabata.io` |
## Importing from rclone.conf
Rabata remotes appear in `rclone.conf` as `type = s3` with `provider = Rabata`. See the
[rclone import guide](/guides/import-rclone-config).
# Railway
Source: https://docs.dataraven.io/providers/railway
Connect Railway-provided S3 buckets as sources and destinations for transfer tasks.
S3-compatible buckets provisioned by [Railway](https://railway.app) alongside your services.
## At a Glance
| | |
| ----------------- | ------------------------------------------------- |
| **Region** | Depends on Railway config |
| **Endpoint URL** | Required — Railway-provided S3 endpoint |
| **Credentials** | Access key — `access_key_id`, `secret_access_key` |
| **Task defaults** | `s3_no_check_bucket: true` |
## Credentials
Create a [secret](/secrets) with the **Railway** provider type. The auth method is
`railway_access_key`, which requires:
| Field | Description |
| ------------------- | ------------------------------------------ |
| `access_key_id` | Access key from Railway's bucket variables |
| `secret_access_key` | Secret key from Railway's bucket variables |
Railway exposes the credentials as service variables on the bucket it provisions — copy them from
the bucket's **Variables** tab.
## Location Setup
| Field | Value |
| ----------------- | -------------------------------------------------------- |
| **region** | Depends on Railway config |
| **endpoint\_url** | Required — the endpoint Railway provides for your bucket |
Use the S3-compatible endpoint URL provided by Railway for your bucket.
Example:
| Field | Example value |
| -------------- | ----------------------------- |
| name | `Railway Bucket US-West` |
| location\_type | Railway |
| bucket\_name | `acme-bucket` |
| endpoint\_url | Railway-provided endpoint URL |
Replicating between Railway regions? See the
[Railway cross-region replication guide](/guides/railway-cross-region-replication).
## Importing from rclone.conf
The importer has no dedicated mapping for Railway remotes, so a hand-written `rclone.conf` entry
imports as **S3 Compatible** — fully functional, since the endpoint and credentials carry over.
To use the first-class Railway type, create the secret and location manually instead. See the
[rclone import guide](/guides/import-rclone-config).
# S3 Compatible
Source: https://docs.dataraven.io/providers/s3-compatible
Connect any S3-compatible provider not covered by a first-class integration.
The catch-all type for any S3-compatible provider without a first-class integration — MinIO,
Storj, IDrive e2, Ceph, and anything else that speaks the S3 API.
If your provider has its own page in this section, prefer that type over S3 Compatible — it
gets provider-specific endpoint handling, validation, and task defaults.
## At a Glance
| | |
| ----------------- | ------------------------------------------------- |
| **Region** | Depends on provider |
| **Endpoint URL** | Required |
| **Credentials** | Access key — `access_key_id`, `secret_access_key` |
| **Task defaults** | `s3_no_check_bucket: true` |
## Credentials
Create a [secret](/secrets) with the **S3 Compatible** provider type. The auth method is
`s3_compatible_access_key`, which requires:
| Field | Description |
| ------------------- | ---------------------------- |
| `access_key_id` | Provider's access key ID |
| `secret_access_key` | Provider's secret access key |
The key needs permission to **list, read, write, and delete** objects in the bucket — the same
operations [connection verification](/connecting-storage#connection-verification) tests.
## Location Setup
| Field | Value |
| ----------------- | -------------------------------------------------- |
| **region** | Depends on provider — set it if yours requires one |
| **endpoint\_url** | Always required |
Example:
| Field | Example value |
| -------------- | --------------------------- |
| name | `MinIO On-Prem` |
| location\_type | S3 Compatible |
| bucket\_name | `acme-minio` |
| endpoint\_url | `https://minio.example.com` |
## Provider Quirks
Many managed S3 providers don't expose `CreateBucket` permission, which surfaces as
`403 Access Denied` during transfers. S3 Compatible locations default to
`s3_no_check_bucket: true` to bypass that check.
### HTTP/2 stays on
DataRaven leaves HTTP/2 enabled for S3 Compatible locations. The one exception is
[Filebase](/providers/filebase#dataraven-puts-filebase-remotes-on-http11), which DataRaven puts on
HTTP/1.1. Filebase recycles HTTP/2 connections with a graceful `GOAWAY` that lands on every upload
in flight. Go cannot replay an upload after it writes the request body. That failure looks like
this in the execution logs:
```
cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY]
after Request.Body was written
```
If your endpoint produces this error, set `disable_http2` in the task's rclone configuration. This
option is task-wide: it puts both remotes of the task on HTTP/1.1. Use it on the tasks that touch
the affected endpoint, not as a habit.
## Importing from rclone.conf
A `type = s3` remote whose `provider =` value isn't recognized — including `provider = Other` —
imports as S3 Compatible, with its endpoint and credentials carried over. The exception:
`provider = Other` remotes whose endpoint hostname identifies
[Fil One](/providers/fil-one), [Filebase](/providers/filebase), [Tigris](/providers/tigris), or
[MEGA S4](/providers/mega-s4) are upgraded to those first-class types instead. See the
[rclone import guide](/guides/import-rclone-config).
# Tigris
Source: https://docs.dataraven.io/providers/tigris
Connect Tigris buckets as sources and destinations for transfer tasks.
Tigris — globally distributed, S3-compatible object storage with a single endpoint.
## At a Glance
| | |
| ----------------- | ------------------------------------------------- |
| **Region** | Not needed |
| **Endpoint URL** | Required — `https://t3.storage.dev` |
| **Credentials** | Access key — `access_key_id`, `secret_access_key` |
| **Task defaults** | `s3_no_check_bucket: true`, `fast_list: true` |
## Credentials
Create a [secret](/secrets) with the **Tigris** provider type. The auth method is
`tigris_access_key`, which requires:
| Field | Description |
| ------------------- | ------------------------ |
| `access_key_id` | Tigris access key ID |
| `secret_access_key` | Tigris secret access key |
Create access keys from the Tigris dashboard.
## Location Setup
| Field | Value |
| ----------------- | --------------------------------------------- |
| **region** | Leave blank — Tigris routes requests globally |
| **endpoint\_url** | Required — `https://t3.storage.dev` |
Example:
| Field | Example value |
| -------------- | ------------------------ |
| name | `Tigris Global` |
| location\_type | Tigris |
| bucket\_name | `acme-global` |
| endpoint\_url | `https://t3.storage.dev` |
## Importing from rclone.conf
Tigris has no `provider =` value of its own in rclone, so real configs write `provider = Other`.
The importer recognizes Tigris remotes by their endpoint hostname (`t3.storage.dev`) instead of
falling back to S3 Compatible. A hand-written `provider = Tigris` line is also accepted. See the
[rclone import guide](/guides/import-rclone-config).
# Wasabi
Source: https://docs.dataraven.io/providers/wasabi
Connect Wasabi buckets as sources and destinations for transfer tasks.
Wasabi hot cloud storage — S3-compatible, with a region-specific endpoint.
## At a Glance
| | |
| ----------------- | ------------------------------------------------- |
| **Region** | Required — e.g., `us-east-1`, `eu-central-1` |
| **Endpoint URL** | Required — `https://s3..wasabisys.com` |
| **Credentials** | Access key — `access_key_id`, `secret_access_key` |
| **Task defaults** | `s3_no_check_bucket: true`, `fast_list: true` |
## Credentials
Create a [secret](/secrets) with the **Wasabi** provider type. The auth method is
`wasabi_access_key`, which requires:
| Field | Description |
| ------------------- | ------------------------ |
| `access_key_id` | Wasabi access key ID |
| `secret_access_key` | Wasabi secret access key |
Create access keys from the Wasabi console under **Access Keys**.
## Location Setup
| Field | Value |
| ----------------- | ---------------------------------------------- |
| **region** | Required — e.g., `us-east-1` |
| **endpoint\_url** | Required — `https://s3..wasabisys.com` |
Both region and endpoint URL are required, and the endpoint must include the region.
Example:
| Field | Example value |
| -------------- | ------------------------------------ |
| name | `Wasabi Hot Storage` |
| location\_type | Wasabi |
| bucket\_name | `acme-hot` |
| region | `us-east-1` |
| endpoint\_url | `https://s3.us-east-1.wasabisys.com` |
## Importing from rclone.conf
Wasabi remotes appear in `rclone.conf` as `type = s3` with `provider = Wasabi`. See the
[rclone import guide](/guides/import-rclone-config).
# Quickstart
Source: https://docs.dataraven.io/quickstart
Create your first data transfer in under 5 minutes
## 1. Create an Account
Sign up at [app.dataraven.io](https://app.dataraven.io). You'll start on the **Free tier** — no
credit card required.
## 2. Connect Your Storage
Add your source and destination storage backends as **Locations**.
Enter credentials directly or connect a **Vault** (1Password, Doppler, Infisical) for
zero-knowledge credential management.
In the dashboard, go to **Locations** and click **Create**.
Select from 40+ supported cloud storage backends (S3, GCS, Azure, R2, etc.) and provide the
bucket name.
Save and Click **Test** to confirm DataRaven can reach your storage.
## 3. Create a Transfer Task
Go to **Tasks** → **Create**. Select your source and destination locations.
Set filters, bandwidth limits, and transfer behavior (sync, copy, move).
Run the task immediately or set a cron schedule for recurring transfers.
## What's Next?
Understand the DataRaven data model.
Set up zero-knowledge credential management.
Automate everything via the REST API.
Step-by-step how-tos.
# Secrets Management
Source: https://docs.dataraven.io/secrets
Store and manage cloud provider credentials for your storage locations — directly or through an external vault.
Secrets store the cloud provider credentials that DataRaven uses to access your storage locations. Every location references a secret, and DataRaven supports two storage modes to fit your security requirements.
Enter credentials directly. Stored encrypted in **AWS SSM Parameter Store** (SecureString). Simple, secure, zero configuration.
Map credential fields to references in **1Password**, **Doppler**, or **Infisical**. Credentials resolved at execution time and immediately discarded.
## Supported Providers
Each cloud storage provider requires specific credential fields. DataRaven auto-selects the auth method when you choose a provider — except Azure, which offers two options. Each provider's page covers how to obtain the credentials and any scoping quirks.
| Provider | Auth Method | Required Fields |
| ------------------------------------------------------------- | ------------------------------------- | --------------------------------------- |
| [**AWS S3**](/providers/aws-s3) | `s3_access_key` | `access_key_id`, `secret_access_key` |
| [**Azure Blob**](/providers/azure-blob) | `azure_sas_url` | `sas_url` |
| [**Azure Blob**](/providers/azure-blob) | `azure_account_key` | `account_name`, `account_key` |
| [**Google Cloud Storage**](/providers/google-cloud-storage) | `gcs_service_account` | `service_account_json` |
| [**Cloudflare R2**](/providers/cloudflare-r2) | `r2_access_key` | `access_key_id`, `secret_access_key` |
| [**Backblaze B2**](/providers/backblaze-b2) | `b2_application_key` | `application_key_id`, `application_key` |
| [**Wasabi**](/providers/wasabi) | `wasabi_access_key` | `access_key_id`, `secret_access_key` |
| [**Railway**](/providers/railway) | `railway_access_key` | `access_key_id`, `secret_access_key` |
| [**Oracle Object Storage**](/providers/oracle-object-storage) | `oracle_object_storage_s3_access_key` | `access_key_id`, `secret_access_key` |
| [**Tigris**](/providers/tigris) | `tigris_access_key` | `access_key_id`, `secret_access_key` |
| [**DigitalOcean Spaces**](/providers/digitalocean-spaces) | `digitalocean_spaces_access_key` | `access_key_id`, `secret_access_key` |
| [**Hetzner**](/providers/hetzner) | `hetzner_access_key` | `access_key_id`, `secret_access_key` |
| [**Rabata**](/providers/rabata) | `rabata_access_key` | `access_key_id`, `secret_access_key` |
| [**Fil One**](/providers/fil-one) | `filone_access_key` | `access_key_id`, `secret_access_key` |
| [**Filebase**](/providers/filebase) | `filebase_access_key` | `access_key_id`, `secret_access_key` |
| [**Impossible Cloud**](/providers/impossible-cloud) | `impossible_cloud_access_key` | `access_key_id`, `secret_access_key` |
| [**Fastly Object Storage**](/providers/fastly-object-storage) | `fastly_access_key` | `access_key_id`, `secret_access_key` |
| [**MEGA S4**](/providers/mega-s4) | `mega_s4_access_key` | `access_key_id`, `secret_access_key` |
| [**S3 Compatible**](/providers/s3-compatible) | `s3_compatible_access_key` | `access_key_id`, `secret_access_key` |
## Creating a Secret
Store credentials directly in DataRaven's encrypted vault. Best for teams that don't use an external secrets manager.
Select the cloud storage provider. The auth method is auto-selected (Azure lets you choose between SAS URL and account key).
Fill in the required credential fields for your provider. These are encrypted and stored in AWS SSM Parameter Store.
Give the secret a descriptive name (e.g., "Production AWS" or "Staging GCS") and save.
Map credential fields to references in your external vault. Requires a [vault connection](/vault-integration) to be configured first.
Choose an existing vault connection (1Password, Doppler, or Infisical). If you haven't created one yet, see the [Vault Integration guide](/vault-integration).
For each required field, provide a reference to the corresponding secret in your vault.
Give the secret a descriptive name and save. DataRaven validates that all required fields are mapped.
**Reference formats by provider:**
| Vault Provider | Format | Example |
| -------------- | ------------------------------------ | ------------------------------------ |
| 1Password | `op://VaultName/ItemName/field_name` | `op://DevOps/AWS-Prod/access_key_id` |
| Doppler | `SECRET_NAME` | `AWS_ACCESS_KEY_ID` |
| Infisical | `SECRET_NAME` | `AWS_ACCESS_KEY_ID` |
You cannot provide both `credentials` and `vault_connection_id` on the same secret. Choose one storage mode — native vault or external vault — per secret.
## Validation Rules
DataRaven validates secrets on creation and update:
* **`auth_method`** must be valid for the chosen `secret_type`
* **`field_mappings`** must cover **all** required fields for the auth method (partial mappings are rejected)
* **`credentials`** and **`vault_connection_id`** are mutually exclusive
## Updating & Rotating Secrets
Edit a secret to rotate the credentials
You have two options:
1. **Update references** — Edit the secret resource with updated `field_mappings` if your vault paths changed
2. **Update in your vault directly** — if the secret values changed but the references are the same, just update the values in 1Password/Doppler/Infisical. DataRaven resolves them at runtime, so changes take effect immediately.
You cannot switch a secret between native vault and external vault after creation. A native secret always uses `credentials`, and a BYOV secret always uses `field_mappings`.
Changing `secret_type` on an existing secret triggers validation against all linked locations to ensure compatibility.
## Secret Lifecycle
Secrets are referenced by **Locations**, which are referenced by **Tasks**. This creates a dependency chain:
Store credentials for a cloud provider using either storage mode.
When creating a storage location, select the secret that holds credentials for that provider.
Tasks reference locations. At execution time, DataRaven resolves the secret's credentials to access the storage.
A secret **cannot be deleted** while it's in use by one or more locations. You'll receive a `409 Conflict` error. Unlink or delete the dependent locations first.
The secret detail view shows all linked locations and tasks, along with creation metadata.
## Security
Credentials are **never returned** in any API response — not on read, not on list, not on update confirmation.
Native vault credentials are stored as **AWS SSM Parameter Store SecureString** parameters, encrypted with AWS KMS.
External vault credentials are resolved **in-memory** at execution time and **immediately
discarded** after the operation completes.
All secret operations — creation, updates, deletions, and runtime resolutions — are logged for audit purposes.
## Provider Tips
Provider-specific credential guidance — which console issues the keys, scoping quirks like Fil
One's region-scoped keys or Fastly's full-access requirement, and what belongs in the secret
versus the location — lives in the **Credentials** section of each
[provider's page](/connecting-storage#supported-providers).
## Tier Limits
| Plan | Secrets |
| -------- | ------- |
| **Free** | 5 |
| **Pro** | 100 |
Need more? [Upgrade your plan](https://app.dataraven.io) from the billing page.
# Audit Logs
Source: https://docs.dataraven.io/security/audit-logs
Track every action across your team with a complete, tamper-proof event trail.
DataRaven records every meaningful action that happens within your team — whether triggered by a user, an API key, or the system itself. The audit log gives you full visibility into who did what, when, and from where.
Every create, update, and delete performed through the dashboard or API.
Scheduled task executions, transfer completions, and failures — logged automatically.
IP address, user agent, and request ID captured with every user-initiated event.
## Viewing the Audit Log
The audit log is accessible in two places within the dashboard:
* **Activity Feed** — The dashboard homepage shows the 10 most recent events at a glance, so you can quickly see what's been happening.
* **Event Log** — A dedicated page at **Event Log** in the sidebar provides the full audit trail with filtering, pagination, and CSV export.
Any team member with **Viewer** role or higher can access the audit log.
## Filtering & Export
The Event Log page supports filtering by:
| Filter | Description |
| ----------------- | -------------------------------------------------------------------------------------------------------------------- |
| **Resource type** | Narrow to a specific resource — tasks, executions, secrets, locations, members, notifications, or vault connections. |
| **Actor type** | Show only events from users, the system, or API keys. |
| **Date range** | Filter events to a specific time window. |
You can **export the current view to CSV** for compliance reporting, incident investigation, or offline analysis. The export includes all fields: timestamp, event type, resource, actor, summary, IP address, user agent, and request ID.
## What Gets Recorded
Every audit log entry captures a core set of fields, plus optional context depending on who triggered the action.
### Common Fields
Every event includes:
| Field | Description |
| ----------------- | --------------------------------------------------------------------------------------------------------------- |
| **Event type** | What happened (e.g., `task_created`, `execution_completed`). |
| **Resource type** | What was affected — `task`, `execution`, `secret`, `location`, `member`, `notification`, or `vault_connection`. |
| **Resource ID** | The unique identifier of the affected resource. |
| **Resource name** | A human-readable name for the resource (where applicable). |
| **Actor type** | Who did it — `user`, `system`, or `api_key`. |
| **Actor name** | The name or email of the person or system that triggered the event. |
| **Summary** | A plain-English description of what happened. |
| **Metadata** | Structured key-value data specific to the event (see tables below). |
| **Timestamp** | When the event occurred (UTC). |
### Request Context (User Actions Only)
When an event is triggered by a user or API key through the dashboard or API, additional request context is captured:
| Field | Description |
| -------------- | -------------------------------------------------------------------------------------- |
| **IP address** | The originating IP, resolved via `cf-connecting-ip` (Cloudflare) or `x-forwarded-for`. |
| **User agent** | The browser or HTTP client used. |
| **Request ID** | A unique correlation ID for the HTTP request, useful for debugging. |
System events (scheduled executions, background workers) do not have request context since they originate from internal processes, not HTTP requests.
## Event Reference
Below is a complete reference of every event type recorded by the audit log, organized by resource.
### Task Events
Recorded when transfer tasks are created, modified, or removed.
| Event Type | Trigger | Summary Example | Metadata |
| --------------- | ------- | ------------------------------- | ----------------------------------------- |
| `task_created` | User | Created task "Daily S3 Backup" | `task_type` (copy, sync) |
| `task_updated` | User | Updated task "Daily S3 Backup" | `updated_fields` — list of changed fields |
| `task_enabled` | User | Enabled task "Daily S3 Backup" | `status: enabled` |
| `task_disabled` | User | Disabled task "Daily S3 Backup" | `status: disabled` |
| `task_archived` | User | Archived task "Daily S3 Backup" | `status: archived` |
| `task_deleted` | User | Deleted task "Daily S3 Backup" | — |
### Execution Events
Recorded across the full lifecycle of a transfer execution — from submission through completion or failure. Execution events come from both user actions and the system.
These events are initiated by a user through the dashboard or API and include full request context (IP, user agent, request ID).
| Event Type | Summary Example | Metadata |
| --------------------- | ---------------------------------------------- | ------------------------------------- |
| `execution_submitted` | Submitted execution for task "Daily S3 Backup" | `trigger` (manual, api), `is_dry_run` |
| `execution_cancelled` | Cancelled execution for task "Daily S3 Backup" | — |
These events are generated by background workers and scheduled jobs. They do **not** include request context.
| Event Type | Summary Example | Metadata |
| ----------------------------- | ---------------------------------------------- | ----------------------------------------- |
| `execution_submitted` | Submitted execution for task "Daily S3 Backup" | `trigger: scheduled`, `is_dry_run: false` |
| `execution_started` | Started execution for task "Daily S3 Backup" | — |
| `execution_completed` | Completed execution for task "Daily S3 Backup" | `stats` (transfer statistics), `error` |
| `execution_failed` | Failed execution for task "Daily S3 Backup" | `stats`, `error` (error message) |
| `execution_pending` | Pending execution for task "Daily S3 Backup" | `stats`, `error` |
| `execution_running` | Running execution for task "Daily S3 Backup" | `stats`, `error` |
| `execution_dry_run_completed` | Dry run completed for task "Daily S3 Backup" | `stats`, `error` |
Scheduled task executions are logged with `trigger: scheduled` and actor name **"Scheduled"**, making it easy to distinguish automated runs from manual ones.
### Secret Events
Recorded when cloud provider credentials are created, modified, or removed.
| Event Type | Trigger | Summary Example | Metadata |
| ---------------- | ------- | ------------------------------- | ---------------------------------------------------------------- |
| `secret_created` | User | Created secret "Production AWS" | `secret_type` (e.g., aws\_s3, azure\_blob) |
| `secret_updated` | User | Updated secret "Production AWS" | `updated_fields` — list of changed fields (credentials excluded) |
| `secret_deleted` | User | Deleted secret "Production AWS" | — |
For security, the `updated_fields` metadata for secret updates intentionally **excludes** the `credentials` field — you'll see which fields changed, but credential values are never logged.
### Location Events
Recorded when storage locations are created, verified, modified, or removed.
| Event Type | Trigger | Summary Example | Metadata |
| ------------------- | ------- | ---------------------------------- | -------------------------------------------- |
| `location_created` | User | Created location "s3-prod-bucket" | `location_type` (e.g., aws\_s3, gcs) |
| `location_updated` | User | Updated location "s3-prod-bucket" | — |
| `location_verified` | User | Verified location "s3-prod-bucket" | `verified` (true/false), `error` (if failed) |
| `location_deleted` | User | Deleted location "s3-prod-bucket" | — |
### Vault Connection Events
Recorded when external vault integrations (1Password, Doppler, Infisical) are managed.
| Event Type | Trigger | Summary Example | Metadata |
| -------------------------------- | ------- | --------------------------------------------------- | -------------------------------------------------- |
| `vault_connection_created` | User | Created vault connection "1Password Prod" | — |
| `vault_connection_tested` | User | Tested vault connection "1Password Prod" | `success`, `message`, plus provider-specific stats |
| `vault_connection_updated` | User | Updated vault connection "1Password Prod" | `updated_fields` — list of changed fields |
| `vault_connection_token_rotated` | User | Rotated token for vault connection "1Password Prod" | — |
| `vault_connection_deleted` | User | Deleted vault connection "1Password Prod" | — |
### Notification Events
Recorded when webhook or notification configurations are managed.
| Event Type | Trigger | Summary Example | Metadata |
| ---------------------- | ------- | ----------------------------------- | --------------------------------------------------------------- |
| `notification_created` | User | Created notification "Slack Alerts" | `event_type` — the execution event this notification listens to |
| `notification_updated` | User | Updated notification "Slack Alerts" | — |
| `notification_deleted` | User | Deleted notification "Slack Alerts" | — |
### Member Events
Recorded when team membership changes — invitations, joins, role changes, and removals.
| Event Type | Trigger | Summary Example | Metadata |
| --------------------- | ------- | --------------------------------------------------------------------- | -------------------------------- |
| `member_invited` | User | Invited member "[jane@example.com](mailto:jane@example.com)" | `role` (viewer, operator, admin) |
| `member_joined` | User | Member joined the team | `role` |
| `member_role_changed` | User | Changed role for member "[jane@example.com](mailto:jane@example.com)" | `new_role` |
| `member_removed` | User | Removed member "[jane@example.com](mailto:jane@example.com)" | — |
## Actor Types
Every event is attributed to an actor. DataRaven distinguishes three types:
| Actor Type | Description | Request Context |
| ----------- | -------------------------------------------------------------------------------- | ---------------------------------- |
| **User** | A team member acting through the dashboard or API. Identified by name and email. | IP address, user agent, request ID |
| **System** | An internal process such as the task scheduler or execution worker. | None — internal origin |
| **API Key** | An external integration authenticating via API key. | IP address, user agent, request ID |
## Scheduled Execution Tracking
When a task has a cron schedule configured, DataRaven's scheduler automatically submits executions at the defined intervals. These scheduled runs are fully tracked in the audit log:
* **`execution_submitted`** is logged with `trigger: scheduled` and actor name **"Scheduled"**
* **`execution_started`**, **`execution_completed`**, and **`execution_failed`** are logged as the execution progresses through its lifecycle
* Transfer statistics (files transferred, bytes moved, errors) are captured in the `stats` metadata field on completion
This means you can audit the complete history of your automated transfers — when they ran, whether they succeeded, and how much data was moved — without any manual intervention.
## Data Retention
Audit logs are retained for the lifetime of the team. When a team is deleted, all associated audit log entries are permanently removed.
## API Access
Audit logs are available via the REST API at:
```
GET /teams/{team_id}/audit-logs
```
**Query parameters:**
| Parameter | Type | Description |
| --------------- | -------- | -------------------------------------- |
| `resource_type` | string | Filter by resource type |
| `event_type` | string | Filter by event type |
| `actor_type` | string | Filter by actor type |
| `start_date` | ISO 8601 | Filter events from this date |
| `end_date` | ISO 8601 | Filter events until this date |
| `page` | integer | Page number (default: 1) |
| `limit` | integer | Items per page (default: 25, max: 250) |
Requires **Viewer** role or higher on the team.
# Permissions & Roles
Source: https://docs.dataraven.io/security/permissions
Complete reference of every API action, the minimum user role required, and the corresponding API key scope.
DataRaven uses a layered permissions model. Every request is authorized by checking two things:
1. **User Role** — hierarchical team membership role (JWT sessions)
2. **API Key Scope** — granular scope string (API key authentication)
A request succeeds when the caller meets **at least** the minimum role shown below, **or** presents an API key that includes the listed scope.
Some actions are **JWT-only** — they cannot be performed with an API key.
These are marked with a **—** in the API Key Scope column.
## Role Hierarchy
Roles are hierarchical — higher roles inherit all permissions of lower roles.
| Level | Role | Description |
| ----- | ------------ | ----------------------------------------------------------- |
| 4 | **Owner** | Full control including team deletion and ownership transfer |
| 3 | **Admin** | All operations except team deletion |
| 2 | **Operator** | Day-to-day operations — run tasks, verify connections |
| 1 | **Viewer** | Read-only access across all resources |
***
## Locations
| Action | Method | Min. Role | API Key Scope |
| --------------------- | -------- | --------- | ------------------ |
| Get provider defaults | `GET` | Viewer | `locations:read` |
| Create location | `POST` | Admin | `locations:create` |
| List locations | `GET` | Viewer | `locations:read` |
| Get location details | `GET` | Viewer | `locations:read` |
| Update location | `PATCH` | Admin | `locations:update` |
| Verify location | `POST` | Operator | `locations:verify` |
| Delete location | `DELETE` | Admin | `locations:delete` |
## Secrets
| Action | Method | Min. Role | API Key Scope |
| ------------------ | -------- | --------- | ---------------- |
| Create secret | `POST` | Admin | `secrets:create` |
| List secrets | `GET` | Viewer | `secrets:read` |
| Get secret details | `GET` | Viewer | `secrets:read` |
| Update secret | `PATCH` | Admin | `secrets:update` |
| Delete secret | `DELETE` | Admin | `secrets:delete` |
## Tasks
| Action | Method | Min. Role | API Key Scope |
| ---------------- | -------- | --------- | -------------- |
| Create task | `POST` | Operator | `tasks:create` |
| List tasks | `GET` | Viewer | `tasks:read` |
| Get task details | `GET` | Viewer | `tasks:read` |
| Update task | `PATCH` | Operator | `tasks:update` |
| Disable task | `POST` | Operator | `tasks:update` |
| Enable task | `POST` | Operator | `tasks:update` |
| Archive task | `POST` | Operator | `tasks:update` |
| Delete task | `DELETE` | Admin | `tasks:delete` |
## Executions
| Action | Method | Min. Role | API Key Scope |
| --------------------- | ------ | --------- | --------------- |
| Submit execution | `POST` | Operator | `tasks:execute` |
| Submit dry run | `POST` | Operator | `tasks:execute` |
| List executions | `GET` | Viewer | `tasks:read` |
| Get execution details | `GET` | Viewer | `tasks:read` |
| Stop execution | `POST` | Operator | `tasks:execute` |
| Download logs | `GET` | Viewer | `tasks:read` |
| Stream logs (SSE) | `GET` | Viewer | `tasks:read` |
## Vault Connections
| Action | Method | Min. Role | API Key Scope |
| ---------------------------- | -------- | --------- | -------------------------- |
| Create vault connection | `POST` | Admin | `vault_connections:create` |
| List vault connections | `GET` | Viewer | `vault_connections:read` |
| Get vault connection details | `GET` | Viewer | `vault_connections:read` |
| Test vault connection | `POST` | Operator | `vault_connections:test` |
| Update vault connection | `PATCH` | Admin | `vault_connections:update` |
| Rotate vault token | `POST` | Admin | `vault_connections:rotate` |
| Delete vault connection | `DELETE` | Admin | `vault_connections:delete` |
## Notifications
| Action | Method | Min. Role | API Key Scope |
| ------------------------ | -------- | --------- | ---------------------- |
| Create notification | `POST` | Admin | `notifications:create` |
| List notifications | `GET` | Viewer | `notifications:read` |
| Get notification details | `GET` | Viewer | `notifications:read` |
| Update notification | `PATCH` | Admin | `notifications:update` |
| Delete notification | `DELETE` | Admin | `notifications:delete` |
| Test notification | `POST` | Operator | `notifications:test` |
## Audit Logs
| Action | Method | Min. Role | API Key Scope |
| --------------- | ------ | --------- | ----------------- |
| List audit logs | `GET` | Viewer | `audit_logs:read` |
## Teams
| Action | Method | Min. Role | API Key Scope |
| ------------------ | -------- | --------- | ------------- |
| List my teams | `GET` | — | — |
| Get team details | `GET` | Viewer | `teams:read` |
| Create team | `POST` | — | — |
| Update team | `PATCH` | Admin | — |
| Delete team | `DELETE` | Owner | — |
| Update member role | `PATCH` | Admin | — |
| Remove team member | `DELETE` | Admin | — |
Team management actions (create, update, delete, member management) are **JWT-only**.
API keys cannot create or modify teams.
## Invitations
| Action | Method | Min. Role | API Key Scope |
| --------------------- | -------- | --------- | ------------- |
| List team invitations | `GET` | Admin | — |
| Create invitation | `POST` | Admin | — |
| Revoke invitation | `DELETE` | Admin | — |
| Resend invitation | `POST` | Admin | — |
| Get my invitations | `GET` | — | — |
| Accept invitation | `POST` | — | — |
| Decline invitation | `POST` | — | — |
All invitation actions are **JWT-only**. User-scoped endpoints (get/accept/decline) require only a valid session — no team role is needed.
## API Keys
| Action | Method | Min. Role | API Key Scope |
| ------------------- | -------- | --------- | ------------- |
| Create API key | `POST` | Admin | — |
| List API keys | `GET` | Viewer | — |
| Get API key details | `GET` | Viewer | — |
| Revoke API key | `POST` | Admin | — |
| Rotate API key | `POST` | Admin | — |
| Delete API key | `DELETE` | Admin | — |
API key management is **JWT-only**. You cannot use an API key to create, revoke, or rotate other API keys.
## Subscriptions & Billing
| Action | Method | Min. Role | API Key Scope |
| ------------------- | ------ | --------- | ------------- |
| Get subscription | `GET` | Viewer | — |
| Verify subscription | `POST` | Viewer | — |
| Billing portal | `POST` | Admin | — |
| Upgrade (checkout) | `POST` | Admin | — |
## Usage
| Action | Method | Min. Role | API Key Scope |
| -------------- | ------ | --------- | ------------- |
| Get team usage | `GET` | Viewer | `usage:read` |
***
## Scope Reference
API key scopes follow the `resource:action` pattern. Here is the full list of available scopes:
| Scope | Description |
| -------------------------- | ------------------------------------------ |
| `audit_logs:read` | List and view audit logs |
| `locations:create` | Create new locations |
| `locations:delete` | Delete locations |
| `locations:read` | List and view locations |
| `locations:update` | Update location properties |
| `locations:verify` | Test location connectivity |
| `notifications:create` | Create notification configurations |
| `notifications:delete` | Delete notification configurations |
| `notifications:read` | List and view notifications |
| `notifications:test` | Send test notifications |
| `notifications:update` | Update notification configurations |
| `secrets:create` | Create new secrets |
| `secrets:delete` | Delete secrets |
| `secrets:read` | List and view secret metadata |
| `secrets:update` | Update secrets and rotate credentials |
| `tasks:create` | Create new tasks |
| `tasks:delete` | Permanently delete tasks |
| `tasks:execute` | Submit, stop, and dry-run executions |
| `tasks:read` | List and view tasks and executions |
| `tasks:update` | Update, enable, disable, and archive tasks |
| `teams:read` | View team details |
| `usage:read` | View team usage analytics |
| `vault_connections:create` | Create vault connections |
| `vault_connections:delete` | Delete vault connections |
| `vault_connections:read` | List and view vault connections |
| `vault_connections:rotate` | Rotate vault access tokens |
| `vault_connections:test` | Test vault connectivity |
| `vault_connections:update` | Update vault connection properties |
# Changelog
Source: https://docs.dataraven.io/updates/changelog
What's new in DataRaven
## Sync Delete Limit
* Every sync now stops and fails if it would delete more than 1,000 objects at the destination. rclone's own default is no limit, so a source listing that came back short could empty the destination and report success
* `max_delete` and `max_delete_size` are new task options. Set `max_delete` to raise the limit, to `0` to forbid deletions, or to `-1` to remove the limit
* A run stopped by the limit reports the limit in its error message, not a generic fatal error
* Read more in [Creating Tasks](/creating-tasks)
## Impossible Cloud Improvements
* Tasks touching an Impossible Cloud location now default to `transfers: 32`, `checkers: 16`
* Location verification gives a clear message when the region does not match the bucket's region, or when the access key is wrong
* Bucket names are validated against Impossible Cloud's rules in the form
* rclone.conf import recognizes Impossible Cloud remotes that specify only an endpoint
* Read more in the [Impossible Cloud documentation](/providers/impossible-cloud)
## Filebase Support
* Filebase is now a first-class provider
* Read more in the [Filebase documentation](/providers/filebase)
## Fil One Support
* Fil One is now a first-class provider. Fil One gives S3-compatible storage on Filecoin, with no egress fees on paid plans
* Transfers that touch `eu-west-1` set `ignore_checksum` automatically. Without that flag, rclone marks every copied object as corrupt and then deletes it
* DataRaven verifies bucket names against the Fil One rules, which forbid dots
* `s3_list_version` is a new task option. DataRaven sets Fil One remotes to ListObjectsV2
* Read more in the [Fil One documentation](/providers/fil-one)
## MEGA S4 Support
* MEGA S4 is now a first-class provider, with a region dropdown covering all seven endpoints and the endpoint URL derived automatically
* Bucket names are validated against MEGA S4's rules in the form, so a name it would reject never reaches CreateBucket
* rclone.conf import recognizes MEGA S4 remotes by provider string and by endpoint, including the legacy `s4.mega.io` hostnames
* Learn more in the [MEGA S4 documentation](/providers/mega-s4).
## Four New Storage Providers
* 4 new storage providers: Fil One, Filebase, Impossible Cloud, and Fastly Object Storage
* Region dropdowns with auto-derived endpoints for providers with fixed region lists (Impossible Cloud, Fastly, Fil One)
* rclone.conf import now recognizes Fil One and Filebase remotes by endpoint (including the legacy `s3.filebase.com` endpoint)
## Passwordless Email Sign-In
* Sign in with a one-time passcode (OTP) email — replaces the previous magic-link flow
## Analytics Exclude Dry Runs by Default
* Usage analytics now exclude dry-run executions by default, so transfer counts and bytes reflect real activity
* Dry-run totals remain available when filtering specifically by dry-run status
## Onboarding UX Improvements
* Onboarding now advances only on a successful (completed or dry-run-completed) execution, not on pending or failed runs
* Role-aware onboarding: operators no longer see admin-only setup steps they can't submit
## Log Retention via Storage Lifecycle
* All users now get 90-day execution log retention
## Location Verification History
* Storage locations now keep a full verification history — see every past verification attempt with status and timestamp
* The location detail page shows current verification status alongside its history
* New API endpoints: list location verifications and fetch provider connection defaults
## Task-Level Analytics
* New Stats tab on task detail pages with task-specific analytics — summary cards, time-series charts, and metric switching without leaving the task view
## New Providers, Analytics Redesign & API Standardization
* 4 new storage providers: Tigris, DigitalOcean Spaces, Hetzner, and Rabata
* Redesigned analytics dashboard with time-series charts, top errors ranking, per-task breakdown, and flexible date range presets (MTD, 3m, 6m, 1y)
* Standardized pagination across all list API endpoints
* Usage quotas API showing current resource counts vs tier limits on the billing page
* rclone.conf import validation fixes
## rclone.conf Import & Onboarding
* rclone.conf import workflow — upload an existing config, review and select remotes, then bulk-create storage locations with credentials
* Supports S3, Azure Blob, GCS, B2, R2, Wasabi, and other S3-compatible providers
* Quick-access import link on the Getting Started dashboard
* rclone updated to v1.73.2
## Notifications, Security & UX Improvements
* API key lifecycle notifications (created, rotated, revoked, deleted) with formatted event templates
* Notification quick setup — bulk-create notification configs from a single Apprise URL
* Real-time transfer progress with live stats (transfer rate, files transferred, bytes, ETA)
* Improved task management: cron expression display with next-run time, tag-based filters, reorganized detail tabs
* Security hardening: input sanitization, safer error responses, enhanced validation
* Centralized version tracking across all services
* Streaming transfer fix for large file lists
## API Keys & Developer Platform
* Scoped API keys for programmatic access
* 28 granular permission scopes following `resource:action` pattern
* Key rotation — new secret, same key ID and scopes
* Instant revocation with full audit trail (create, rotate, revoke, delete)
* Learn more in the [API Keys documentation](/developer-platform/api-keys).
## Audit logs
* Full event trail for all team activity
* Dashboard activity feed showing recent events at a glance
* Dedicated Event Log page with filtering by resource type, actor, and date range
* Expandable rows with IP address, user agent, request ID, and event metadata
* CSV export for compliance and offline analysis
* Tracks user, system, and API key actors across tasks, executions, secrets, locations, members, notifications, and vault connections
* Learn more in the [Audit Logs documentation](/security/audit-logs).
## Initial public launch
* 40+ storage backend support
* Zero-knowledge vault integration (1Password, Doppler, Infisical)
* Team collaboration with role-based access
* Scheduled and on-demand transfer tasks
* Real-time execution logs and monitoring
# Roadmap
Source: https://docs.dataraven.io/updates/roadmap
What's coming next for DataRaven
A look at what we're building next. This roadmap reflects our current priorities — it will evolve based on user feedback and demand.
Have a feature request? [Join the Discord](https://discord.gg/VXAvSnTB2Y) or email us at [hi@dataraven.io](mailto:hi@dataraven.io).
## Up Next
Features actively being worked on or committed for the near-term horizon.
A command-line interface for managing tasks, triggering executions, and tailing logs from your terminal.
Official client libraries for Python, TypeScript, and Go — built on top of the API key system.
Trigger and monitor transfers from GitHub Actions, Airflow, Dagster, and other pipeline orchestrators using API keys.
Let AI agents orchestrate data movement across your infrastructure through the API and upcoming SDK integrations.
First-class support for AI agent integrations through skill repositories like ClawHub. Let agents manage transfers, query execution status, and orchestrate data pipelines programmatically.
Expanding beyond the current lineup. Planned additions include MinIO, IDrive e2, Storj, and many more.
Access all your DataRaven-connected storage providers through a single unified, secure, managed endpoint — built on rclone's S3 server.
Authenticate to Azure Blob Storage using Managed Identity or Service Principal credentials — no account keys or SAS URLs required.
Task-level analytics with rankings, error classification, success rates, and trend analysis. Includes per-provider egress cost estimates so you can track the real cost of your transfers across AWS, Azure, GCS, R2, and other backends.
## Exploring
Features we're evaluating based on community interest.
Enterprise single sign-on for teams that require centralized identity management.
Manage DataRaven resources — teams, tasks, locations, secrets — as infrastructure-as-code.
# Vault Integration
Source: https://docs.dataraven.io/vault-integration
Connect your secrets manager to DataRaven for zero-knowledge credential management with BYOV (Bring Your Own Vault).
DataRaven's **Bring Your Own Vault (BYOV)** architecture ensures your cloud credentials never touch
our infrastructure. Connect your existing secrets manager, map fields to vault references, and
DataRaven resolves credentials in-memory at transfer time — then immediately discards them.
BYOV is optional. DataRaven also provides **built-in secret storage** powered by AWS SSM Parameter
Store SecureString. You can store credentials securely without connecting an external vault.
**Self-hosted vaults are not supported** at this time. Only managed/cloud-hosted instances of
1Password, Doppler, and Infisical are supported.
## How It Works
Go to **Settings → Vault Connections → Add Connection**, select your provider, and provide a
scoped access token. DataRaven encrypts the token and stores it in AWS SSM Parameter Store —
it's never returned in API responses.
Create a secret that maps DataRaven field names (like `access_key_id`) to references in your
vault (like `op://DevOps/AWS-Prod/access_key_id`). Each mapping tells DataRaven where to find
the credential at runtime.
When configuring a storage location, select your vault-backed secret instead of entering
credentials directly. The location will resolve credentials from your vault every time it's
used.
At execution time, DataRaven authenticates to your vault, resolves the mapped credentials into
memory, performs the transfer operation, and immediately discards them. Nothing is cached or
persisted.
## Supported Providers
### Authentication
1Password connections use a **Service Account Token** (starts with `ops_`).
| Field | Required | Description |
| -------------- | -------- | --------------------------------- |
| `access_token` | ✅ | Service account token (`ops_...`) |
### Setup
1. Open **1Password** → **Settings** → **Developer** → **Service Accounts**
2. Click **New Service Account**
3. Give it a descriptive name (e.g., "DataRaven Production")
4. Grant **read access** to the vaults containing your cloud credentials
5. Copy the generated token — it's only shown once
No additional configuration is needed beyond the token.
### Field Mapping Format
1Password uses the `op://` URI format to reference individual fields:
```
op://vault-name/item-name/field-name
```
**Example — S3 credentials:**
```json theme={"theme":{"light":"github-light","dark":"poimandres"}}
{
"field_mappings": [
{"field_name": "access_key_id", "reference": "op://DevOps/AWS-Prod/access_key_id"},
{"field_name": "secret_access_key", "reference": "op://DevOps/AWS-Prod/secret_access_key"}
]
}
```
Use descriptive vault and item names in 1Password so your `op://` references are self-documenting.
### Authentication
Doppler connections use a **Service Token** (starts with `dp.st.`) scoped to a specific project and config.
| Field | Required | Description |
| -------------- | -------- | ------------------------------------------- |
| `access_token` | ✅ | Service token (`dp.st....`) |
| `project` | ✅ | Doppler project slug |
| `config` | ✅ | Config name (e.g., `production`, `staging`) |
### Setup
1. Open **Doppler** → select your **Project**
2. Go to **Access** → **Generate Service Token**
3. Select the **config** (environment) to scope the token to
4. Copy the generated token
Doppler tokens are scoped to a single project + config combination. If you need credentials from multiple projects, create separate vault connections.
### Field Mapping Format
Doppler uses the secret name directly — no path syntax needed:
```json theme={"theme":{"light":"github-light","dark":"poimandres"}}
{
"field_mappings": [
{"field_name": "access_key_id", "reference": "AWS_ACCESS_KEY_ID"},
{"field_name": "secret_access_key", "reference": "AWS_SECRET_ACCESS_KEY"}
]
}
```
### Authentication
Infisical connections use **Universal Auth** via a Machine Identity, requiring a client ID and client secret pair.
| Field | Required | Description |
| --------------- | -------- | ------------------------------------------- |
| `client_id` | ✅ | UUID from Machine Identity |
| `client_secret` | ✅ | Secret from Machine Identity |
| `project_id` | ✅ | Infisical project ID (UUID) |
| `environment` | ✅ | Environment slug (`dev`, `staging`, `prod`) |
| `secret_path` | ❌ | Path to secrets folder (defaults to `/`) |
### Setup
1. Open your Infisical project → **Project Settings** → **Access Control**
2. Go to **Machine Identities** → **Create**
3. Enable **Universal Auth** for the identity
4. Generate a client ID and client secret
5. Grant the identity read access to the project and environment containing your credentials
6. Copy the **project ID** from **Project Settings → General**
Infisical does not support token rotation through the API. To rotate credentials, you must **delete the vault connection and create a new one** with updated Machine Identity credentials.
### Field Mapping Format
Infisical uses the secret name directly, similar to Doppler:
```json theme={"theme":{"light":"github-light","dark":"poimandres"}}
{
"field_mappings": [
{"field_name": "access_key_id", "reference": "AWS_ACCESS_KEY_ID"},
{"field_name": "secret_access_key", "reference": "AWS_SECRET_ACCESS_KEY"}
]
}
```
## Token Management
Once a vault connection is created, you can manage it from the connection detail page.
### Test Connection
Click **Test Connection** to verify that DataRaven can authenticate with your vault. A successful
test returns:
* **1Password:** Number of accessible vaults
* **Doppler / Infisical:** Number of accessible secrets
Testing also updates the **verification status** timestamp, so you can track when a connection was
last confirmed working.
### Rotate Token
Click **Rotate Token** from the connection detail page. Enter the new token and DataRaven will replace the encrypted token in storage. All secrets using this connection will immediately use the new token — no other changes needed.
Token rotation is not available for Infisical connections. To rotate credentials:
1. Create a new Machine Identity in Infisical (or regenerate the client secret)
2. Delete the existing vault connection in DataRaven
3. Create a new vault connection with the updated credentials
4. Re-link any secrets that referenced the old connection
## Security
Vault tokens are encrypted and stored in **AWS SSM Parameter Store** SecureString parameters.
They are never stored in DataRaven's database.
Tokens and credentials are **never returned in API responses**. The API only surfaces a vault
reference ID.
Cloud credentials are resolved in-memory at execution time and **immediately discarded** after
the operation completes.
All vault operations — connections, tests, rotations, and secret resolutions — are logged for
audit purposes.
Follow the principle of **minimum privilege** when creating vault tokens. Grant read-only access
to only the specific vaults, projects, or environments that DataRaven needs.
## Tier Limits
| Plan | Vault Connections |
| -------- | ----------------- |
| **Free** | 1 |
| **Pro** | 10 |
Need more? [Upgrade your plan](https://app.dataraven.io) from the billing page.