> ## Documentation Index
> Fetch the complete documentation index at: https://docs.dataraven.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Creating & Managing Transfer Tasks
> Complete guide to DataRaven tasks — the core resource for data transfer operations
Tasks are **the core resource** in DataRaven. They are blueprints for data transfer operations
between two locations, defining source → destination mapping, transfer modes, rclone options, file
filters, and optional scheduling. Tasks can be executed on-demand or automatically on a cron
schedule.
## Transfer Modes
DataRaven supports two fundamental transfer modes, each with different behaviors and safety
implications:
**Copy** (`copy`) transfers files from source to destination without deleting anything at the
destination. This is the **safest option** for one-way backups and data distribution. - Files
are copied from source to destination - Existing files at destination are preserved unless
overwritten by newer source files - Safe for backups — no data loss risk at destination 📖
[RClone Copy Documentation](https://rclone.org/commands/rclone_copy/)
**Sync** (`sync`) makes the destination mirror the source exactly. This means files that exist
on the destination but not on the source will be **deleted**.
**Sync mode deletes files on the destination** that don't exist on the source. Use with
extreme caution. Always test with dry runs first.
* Destination becomes an exact mirror of source - Extra files at destination are permanently
deleted - Ideal for maintaining exact replicas - ⚠️ **High risk** — can cause data loss 📖
[RClone Sync Documentation](https://rclone.org/commands/rclone_sync/)
## Creating a Task
Before creating a task, ensure you have:
* At least 2 locations configured → [Connecting Storage Guide](/connecting-storage)
* Required secrets set up → [Secrets Management Guide](/secrets)
* Appropriate user role (OPERATOR or higher required)
Go to **Tasks → Create Task** in the DataRaven dashboard.
Configure the fundamental task properties:
**Required Fields:**
* **Name** (required, max 255 characters) — Human-readable task identifier
* **Task Type** — Choose `copy` or `sync` mode
* **Source Location** — Select from configured locations
* **Destination Location** — Select from configured locations (must be different from source)
**Optional Fields:**
* **Description** (max 2000 characters) — Document the task purpose
* **Source Path** — Specific path within source bucket/folder (defaults to root)
* **Destination Path** — Specific path within destination bucket/folder (defaults to root)
**Scheduling:**
* **Enable Scheduling** — Toggle scheduled execution
* **Cron Schedule** — 5-field UNIX cron expression or keywords
Source and destination locations **must be different**. This is validated by the system to prevent data corruption.
Configure performance, filtering, and transfer behavior. See [RClone
Configuration](#rclone-configuration) section below for detailed options.
Advanced options for comparison methods, transfer behavior, performance tuning, network settings, and provider-specific configurations.
## RClone Configuration
DataRaven provides extensive rclone configuration options to fine-tune transfer behavior,
performance, and filtering.
Control transfer performance and resource usage:
| Option | Type | Range | Description |
| ------------- | ------- | -------- | -------------------------------------------------- |
| `bwlimit` | string | - | Bandwidth limit with time-based scheduling support |
| `transfers` | integer | 1-64 | Number of parallel file transfers (default: 4) |
| `checkers` | integer | 1-64 | Number of parallel file checkers (default: 8) |
| `buffer_size` | string | - | In-memory buffer size per transfer |
| `max_backlog` | integer | 1-150000 | Max objects in sync backlog (default: 10000) |
**Bandwidth Limiting Examples:**
```
100M # Constant 100MB/s limit
10M:off # 10MB/s on weekdays, unlimited on weekends
Mon-00:00,512 12:00,1M # 512KB/s until noon, then 1MB/s on Mondays
Sun-20:00,off # Unlimited on Sunday after 8 PM
```
**Buffer Size Examples:**
* `16M` — 16 megabytes per transfer
* `32M` — 32 megabytes per transfer (good for high-latency connections)
Control which files are included or excluded from transfers. All filter options are nested under the
`filters` configuration key.
**Don't mix filter types!** Use either `filter` patterns OR the old-style `--include`/`--exclude`
flags, but never both together.
| Option | Type | Description |
| ---------- | ---------------- | ---------------------------------------------- |
| `filter` | array of strings | Include/exclude patterns with `+`/`-` prefixes |
| `max_age` | string | Only transfer files newer than this age |
| `min_age` | string | Only transfer files older than this age |
| `max_size` | string | Only transfer files smaller than this size |
| `min_size` | string | Only transfer files larger than this size |
**Filter Pattern Examples:**
```json theme={"theme":{"light":"github-light","dark":"poimandres"}}
{
"filters": {
"filter": [
"+ *.jpg", // Include all JPEG files
"+ *.png", // Include all PNG files
"- *.tmp", // Exclude temporary files
"+ important/**", // Include entire important directory
"- **" // Exclude everything else
]
}
}
```
**Age Filter Examples:**
* `24h` — Files from last 24 hours
* `7d` — Files from last 7 days
* `30d` — Files from last 30 days
**Size Filter Examples:**
* `100M` — Files under 100 megabytes
* `1G` — Files under 1 gigabyte
* `500K` — Files under 500 kilobytes
Configure how rclone determines if files need to be transferred:
| Option | Type | Description |
| --------------- | ------- | -------------------------------------------------- |
| `checksum` | boolean | Verify file checksums (slower but safer) |
| `size_only` | boolean | Compare files by size only (faster, less accurate) |
| `ignore_size` | boolean | Don't compare file sizes |
| `modify_window` | string | Time window for modification time comparison |
**Modify Window Examples:**
* `1s` — 1 second tolerance
* `5m` — 5 minute tolerance (useful for some cloud providers)
* `1h` — 1 hour tolerance
Use `checksum` for critical data where integrity is paramount. Use `size_only` for faster
transfers when you trust modification times aren't reliable.
Control how files are transferred and handled:
| Option | Type | Description |
| -------------------- | ------- | ------------------------------------------------- |
| `update` | boolean | Skip files that are newer on destination |
| `ignore_existing` | boolean | Skip all files that exist on destination |
| `immutable` | boolean | Fail if existing files have been modified |
| `use_server_modtime` | boolean | Use server-side modification times |
| `no_update_modtime` | boolean | Don't update modification times on destination |
| `no_traverse` | boolean | Don't scan destination directory (copy mode only) |
| `metadata` | boolean | Preserve file metadata when copying |
| `dry_run` | boolean | Preview transfer without actually moving data |
`no_traverse` can significantly speed up copy operations when you know the destination doesn't
have conflicting files.
Control file processing limits and behavior:
| Option | Type | Range | Description |
| -------------- | ------- | --------- | --------------------------------------------------------- |
| `max_depth` | integer | -1 to 100 | Directory recursion depth (-1 = unlimited, 0 = root only) |
| `max_transfer` | string | - | Total transfer size limit per execution |
**Max Transfer Examples:**
* `10G` — Stop after transferring 10 gigabytes
* `500M` — Stop after transferring 500 megabytes
* `1T` — Stop after transferring 1 terabyte
Fine-tune network behavior and connection handling:
| Option | Type | Range | Description |
| ---------------------- | ------- | ------ | -------------------------------------------------- |
| `disable_http2` | boolean | - | Disable HTTP/2 for compatibility |
| `multi_thread_streams` | integer | 0-64 | Multi-threaded download streams (default: 4) |
| `tpslimit` | float | 0-1000 | HTTP transactions per second limit (0 = unlimited) |
| `tpslimit_burst` | integer | 0-1000 | Maximum burst for transaction rate limiting |
Special options that apply to specific cloud storage providers:
| Option | Type | Default | Applies To | Description |
| -------------------- | ------- | ------- | ------------------------------------------- | -------------------------------- |
| `s3_no_check_bucket` | boolean | `true` | R2, S3\_COMPATIBLE, RAILWAY, ORACLE, WASABI | Skip bucket existence check |
| `b2_hard_delete` | boolean | `false` | B2 | Permanently delete vs hide files |
| `fast_list` | boolean | `true` | B2 | Use recursive directory listing |
Provider defaults are automatically applied based on your source and destination location types.
Destination defaults take precedence over source defaults, and your explicit configuration always
overrides defaults.
## Scheduling Tasks
DataRaven supports flexible task scheduling with cron expressions and convenient keywords.
### Cron Expressions
Tasks use **5-field UNIX cron expressions** evaluated in **UTC timezone** by the Hatchet scheduler:
```
┌─────────────── minute (0 - 59)
│ ┌───────────── hour (0 - 23)
│ │ ┌─────────── day of month (1 - 31)
│ │ │ ┌───────── month (1 - 12)
│ │ │ │ ┌─────── day of week (0 - 6) (Sunday = 0)
│ │ │ │ │
* * * * *
```
### Schedule Keywords
For common scheduling needs, use these convenient keywords:
| Keyword | Equivalent Cron | Description |
| ----------- | --------------- | ---------------------------------------- |
| `@hourly` | `0 * * * *` | Every hour at minute 0 |
| `@daily` | `0 0 * * *` | Every day at midnight UTC |
| `@weekly` | `0 0 * * 0` | Every Sunday at midnight UTC |
| `@monthly` | `0 0 1 * *` | First day of every month at midnight UTC |
| `@yearly` | `0 0 1 1 *` | January 1st at midnight UTC |
| `@annually` | `0 0 1 1 *` | Same as @yearly |
### Examples
```bash theme={"theme":{"light":"github-light","dark":"poimandres"}}
# Every 15 minutes
*/15 * * * *
# Every day at 3:30 AM UTC
30 3 * * *
# Every Monday at 9 AM UTC
0 9 * * 1
# First day of every month at 6 AM UTC
0 6 1 * *
# Every 6 hours
0 */6 * * *
```
Need help with cron expressions? Use [crontab.guru](https://crontab.guru/) for interactive cron
schedule building and validation.
**Pro Tier Required:** Task scheduling is only available on Pro tier (\$29/mo). Free tier users can
only run tasks on-demand.
## Dry Runs
Before executing a task with real data transfer, use **dry runs** to preview exactly what would
happen without actually moving any data.
### Benefits of Dry Runs
* **Preview transfers** — See which files would be copied, updated, or deleted
* **Test filters** — Verify your include/exclude patterns work correctly
* **Validate configuration** — Catch configuration errors before real execution
* **No usage impact** — Dry run data transfer doesn't count toward usage metrics
### Running Dry Runs
Click the **"Dry Run"** button on any task to execute a preview.
`bash POST /tasks/{id}/executions/dry-run `
### Dry Run Output
Dry runs provide detailed logs showing:
* Files that would be transferred
* Files that would be skipped (already up-to-date)
* Files that would be deleted (sync mode only)
* Total bytes and file counts for the operation
**Always run a dry run first** when using sync mode or complex filter patterns. It's your safety
net against unintended data loss.
## Task Execution
### Running Tasks
**Web Dashboard:** Click "Run Now" button on any task **API:** `bash POST /tasks/{id}
/executions `
When scheduling is enabled, tasks automatically run according to their cron schedule. Scheduled
runs appear in the execution history with a "scheduled" trigger type.
### Execution Tracking
Every task execution creates a detailed execution record containing:
* **Real-time logs** — Stream transfer progress and debug information
* **Transfer statistics** — Bytes transferred, files processed, error counts
* **Performance metrics** — Duration, transfer rates, checkpoint timing
* **Downloadable logs** — Full execution logs for debugging and audit
### Execution States
| State | Description |
| ----------- | ---------------------------------- |
| `PENDING` | Execution queued, waiting to start |
| `RUNNING` | Currently transferring data |
| `SUCCESS` | Completed successfully |
| `FAILED` | Completed with errors |
| `CANCELLED` | User-cancelled before completion |
For detailed information about monitoring and managing executions, see the
[Task Executions Guide](/task-executions).
## Task Lifecycle & Status Management
Tasks can be in one of three states, each with different capabilities:
**Default state** for new tasks.
* ✅ Can be executed on-demand
* ✅ Scheduled runs are active (if configured)
* ✅ Can be disabled or archived
* ✅ Full edit capabilities
**Paused state** — temporarily inactive.
* ❌ Cannot be executed
* ❌ Scheduled runs are paused
* ✅ Can be re-enabled
* ✅ Can be archived
* ✅ Configuration can be edited
**Permanent soft delete** — cannot be reactivated.
* ❌ Cannot be executed
* ❌ Cannot be re-enabled
* ❌ Cannot be edited
* ✅ Execution history is preserved
* ⚠️ **Irreversible** —
create new task if needed
### Status Transitions
* Enabled → Disabled ✅
* Disabled → Enabled ✅
* Enabled → Archived ✅
* Disabled → Archived ✅
* Archived → Enabled ❌
* Archived → Disabled ❌
* Archived → *anything* ❌
## Task Deletion vs Archiving
### Deletion Rules
Tasks can only be **hard deleted** under specific conditions:
**DELETE only works for tasks with ZERO executions.** If a task has any execution history,
deletion will fail with a 400 error — you must archive instead.
* ✅ **Can delete:** Fresh tasks that have never been executed
* ❌ **Cannot delete:** Tasks with any execution history
* 🔐 **Requires:** ADMIN role or higher
### Why This Restriction?
Execution history serves as an **immutable audit trail** for:
* Compliance requirements
* Billing calculations
* Forensic analysis
* Data governance
## Task Duplication
The DataRaven UI supports **task duplication** for creating similar tasks with slight variations:
### What Gets Copied
* All configuration settings
* RClone options
* Filter patterns
* Scheduling settings
### What Gets Reset
* Task name (you must provide a new unique name)
* Execution history (new task starts clean)
* Created/modified timestamps
### Use Cases
* Creating staging vs production versions of the same transfer
* Setting up similar tasks for different source/destination pairs
* Testing configuration variations
Duplication is perfect for creating task templates. Set up one task with complex filtering and
performance tuning, then duplicate it for different data sources.
## Tier Limits & Performance
DataRaven applies different limits based on your subscription tier. **Limits are enforced at
execution time**, not task creation time.
**Subscription:** Free
**Task Limits:**
* Maximum 2 tasks total
* On-demand execution only (no scheduling)
**Performance Limits (applied at runtime):**
* `transfers`: 1 (single-threaded transfers)
* `checkers`: 1 (single-threaded checking)
* `bwlimit`: 1G (max 1GB/s bandwidth)
**Subscription:** \$29/month
**Task Limits:**
* Maximum 50 tasks total
* Full scheduling capabilities
**Performance Limits:**
* `transfers`: up to 64 (user configurable)
* `checkers`: up to 64 (user configurable)
* `bwlimit`: up to 10G (user configurable)
Pro tier respects your configured values. If you set `transfers=8`, it stays at 8. The limits
are maximums, not forced values.
### Tier Transitions
When you change subscription tiers, the effects are **immediate**:
* **Upgrade to Pro:** Existing tasks immediately get Pro limits on next execution
* **Downgrade to Free:** Existing tasks immediately get Free limits on next execution
* **No reconfiguration needed** — DataRaven handles the transition automatically
## Security & Validation
DataRaven implements multiple security layers to protect your data and systems:
### Path Validation
* **Command injection prevention:** Source and destination paths are validated against shell
metacharacters
* **Path traversal protection:** Paths cannot contain `../` or other directory traversal patterns
* **Character restrictions:** Paths must use safe, filesystem-compatible characters
### Configuration Validation
* **Filter pattern validation:** All filter patterns must start with `+` or `-` prefixes
* **RClone config size limit:** Configuration payload limited to 50KB maximum
* **Allowed flags only:** Only explicitly defined rclone flags are accepted — extra fields are
rejected
* **Provider defaults:** Safe provider-specific defaults are injected server-side
### Data Protection
* **Payload size limits:** Request payloads are size-limited to prevent abuse
* **Rate limiting:** API endpoints are rate-limited to prevent overload
* **Audit logging:** All task operations are logged for compliance and debugging
## Required Permissions
Different task operations require different user roles:
| Operation | Required Role | Notes |
| ------------------- | ------------- | -------------------------------------- |
| View/List Tasks | VIEWER | Read-only access |
| Create Task | OPERATOR | Can create and configure |
| Update Task | OPERATOR | Can modify existing tasks |
| Run/Dry Run Task | OPERATOR | Can execute tasks |
| Enable/Disable Task | OPERATOR | Can change task status |
| Archive Task | OPERATOR | Can soft-delete tasks |
| Delete Task | ADMIN | Can hard-delete (zero executions only) |
Role inheritance applies: ADMIN users can perform all OPERATOR and VIEWER operations.
## Best Practices
* Always test with dry runs
* Use copy mode for backups
* Be extremely careful with sync mode
* Start with small file sets
* Tune transfers/checkers for your bandwidth
* Use appropriate buffer sizes
* Consider time-based bandwidth limiting
* Monitor execution performance
* Use descriptive task names
* Document complex filter patterns
* Group related tasks logically
* Archive old tasks instead of deleting
* Consider UTC timezone for cron
* Spread scheduled tasks across time
* Account for transfer duration in schedules
* Use keywords for common schedules
## Troubleshooting
### Common Issues
**Possible causes:**
* Task is disabled or archived
* Insufficient user role (need OPERATOR+)
* Source/destination locations offline
* Invalid rclone configuration
**Solutions:**
1. Check task status and enable if needed
2. Verify user permissions
3. Test location connectivity
4. Run a dry run to validate config
**Possible causes:**
* Low `transfers` or `checkers` values
* Bandwidth limiting too restrictive
* Small buffer size
* Network latency issues
**Solutions:**
1. Increase `transfers` (4-16 for most cases)
2. Increase `checkers` (8-32 for many small files)
3. Adjust `bwlimit` or remove entirely
4. Increase `buffer_size` for high-latency connections
**Possible causes:**
* Restrictive filter patterns
* Files already exist (with `ignore_existing`)
* Files newer on destination (with `update`)
* Size/age filters excluding files
**Solutions:**
1. Review filter patterns with dry run
2. Check comparison options settings
3. Verify source file timestamps
4. Test with minimal filters first
**Possible causes:**
* Files exist on destination but not source
* Filter patterns changed since last sync
* Source path configuration changed
**Solutions:**
1. Always dry run sync operations first
2. Verify source path hasn't changed
3. Check that source files still exist
4. Consider using copy mode instead
## Next Steps
* **[Task Executions Guide](/task-executions)** — Monitor and manage task runs
* **[Connecting Storage](/connecting-storage)** — Add more source/destination locations
* **[API Reference](/api-reference/tasks)** — Complete API documentation
* **[Secrets Management](/secrets)** — Secure credential storage