Creating & Managing Transfer Tasks

Tasks are the core resource in DataRaven. They are blueprints for data transfer operations between two locations, defining source → destination mapping, transfer modes, rclone options, file filters, and optional scheduling. Tasks can be executed on-demand or automatically on a cron schedule.

Transfer Modes

DataRaven supports two fundamental transfer modes, each with different behaviors and safety implications:

Copy Mode
Sync Mode

Copy (copy) transfers files from source to destination without deleting anything at the destination. This is the safest option for one-way backups and data distribution. - Files are copied from source to destination - Existing files at destination are preserved unless overwritten by newer source files - Safe for backups — no data loss risk at destination 📖 RClone Copy Documentation

Sync (sync) makes the destination mirror the source exactly. This means files that exist on the destination but not on the source will be deleted.

Sync mode deletes files on the destination that don’t exist on the source. Use with extreme caution. Always test with dry runs first.

Destination becomes an exact mirror of source - Extra files at destination are permanently deleted - Ideal for maintaining exact replicas - ⚠️ High risk — can cause data loss 📖 RClone Sync Documentation

Creating a Task

Prerequisites

Before creating a task, ensure you have:

At least 2 locations configured → Connecting Storage Guide
Required secrets set up → Secrets Management Guide
Appropriate user role (OPERATOR or higher required)

Navigate to Task Creation

Go to Tasks → Create Task in the DataRaven dashboard.

Setup Tab — Basic Configuration

Configure the fundamental task properties:Required Fields:

Name (required, max 255 characters) — Human-readable task identifier
Task Type — Choose copy or sync mode
Source Location — Select from configured locations
Destination Location — Select from configured locations (must be different from source)

Optional Fields:

Description (max 2000 characters) — Document the task purpose
Source Path — Specific path within source bucket/folder (defaults to root)
Destination Path — Specific path within destination bucket/folder (defaults to root)

Scheduling:

Enable Scheduling — Toggle scheduled execution
Cron Schedule — 5-field UNIX cron expression or keywords

Source and destination locations must be different. This is validated by the system to prevent data corruption.

Options Tab — RClone Configuration

Configure performance, filtering, and transfer behavior. See RClone Configuration section below for detailed options.

Advanced Tab — Fine-tuning

Advanced options for comparison methods, transfer behavior, performance tuning, network settings, and provider-specific configurations.

RClone Configuration

DataRaven provides extensive rclone configuration options to fine-tune transfer behavior, performance, and filtering.

Performance Options

Control transfer performance and resource usage:

Option	Type	Range	Description
`bwlimit`	string	-	Bandwidth limit with time-based scheduling support
`transfers`	integer	1-64	Number of parallel file transfers (default: 4)
`checkers`	integer	1-64	Number of parallel file checkers (default: 8)
`buffer_size`	string	-	In-memory buffer size per transfer
`max_backlog`	integer	1-150000	Max objects in sync backlog (default: 10000)

Bandwidth Limiting Examples:

100M                    # Constant 100MB/s limit
10M:off                 # 10MB/s on weekdays, unlimited on weekends
Mon-00:00,512 12:00,1M  # 512KB/s until noon, then 1MB/s on Mondays
Sun-20:00,off           # Unlimited on Sunday after 8 PM

Buffer Size Examples:

16M — 16 megabytes per transfer
32M — 32 megabytes per transfer (good for high-latency connections)

File Filters

Control which files are included or excluded from transfers. All filter options are nested under the filters configuration key.

Don’t mix filter types! Use either filter patterns OR the old-style --include/--exclude flags, but never both together.

Option	Type	Description
`filter`	array of strings	Include/exclude patterns with `+`/`-` prefixes
`max_age`	string	Only transfer files newer than this age
`min_age`	string	Only transfer files older than this age
`max_size`	string	Only transfer files smaller than this size
`min_size`	string	Only transfer files larger than this size

Filter Pattern Examples:

{
  "filters": {
    "filter": [
      "+ *.jpg", // Include all JPEG files
      "+ *.png", // Include all PNG files
      "- *.tmp", // Exclude temporary files
      "+ important/**", // Include entire important directory
      "- **" // Exclude everything else
    ]
  }
}

Age Filter Examples:

24h — Files from last 24 hours
7d — Files from last 7 days
30d — Files from last 30 days

Size Filter Examples:

100M — Files under 100 megabytes
1G — Files under 1 gigabyte
500K — Files under 500 kilobytes

Comparison Options

Configure how rclone determines if files need to be transferred:

Option	Type	Description
`checksum`	boolean	Verify file checksums (slower but safer)
`size_only`	boolean	Compare files by size only (faster, less accurate)
`ignore_size`	boolean	Don’t compare file sizes
`modify_window`	string	Time window for modification time comparison

Modify Window Examples:

1s — 1 second tolerance
5m — 5 minute tolerance (useful for some cloud providers)
1h — 1 hour tolerance

Use checksum for critical data where integrity is paramount. Use size_only for faster transfers when you trust modification times aren’t reliable.

Transfer Behavior

Control how files are transferred and handled:

Option	Type	Description
`update`	boolean	Skip files that are newer on destination
`ignore_existing`	boolean	Skip all files that exist on destination
`immutable`	boolean	Fail if existing files have been modified
`use_server_modtime`	boolean	Use server-side modification times
`no_update_modtime`	boolean	Don’t update modification times on destination
`no_traverse`	boolean	Don’t scan destination directory (copy mode only)
`metadata`	boolean	Preserve file metadata when copying
`dry_run`	boolean	Preview transfer without actually moving data

no_traverse can significantly speed up copy operations when you know the destination doesn’t have conflicting files.

File Handling

Control file processing limits and behavior:

Option	Type	Range	Description
`max_depth`	integer	-1 to 100	Directory recursion depth (-1 = unlimited, 0 = root only)
`max_transfer`	string	-	Total transfer size limit per execution

Max Transfer Examples:

10G — Stop after transferring 10 gigabytes
500M — Stop after transferring 500 megabytes
1T — Stop after transferring 1 terabyte

Network & Tuning

Fine-tune network behavior and connection handling:

Option	Type	Range	Description
`disable_http2`	boolean	-	Disable HTTP/2 for compatibility
`multi_thread_streams`	integer	0-64	Multi-threaded download streams (default: 4)
`tpslimit`	float	0-1000	HTTP transactions per second limit (0 = unlimited)
`tpslimit_burst`	integer	0-1000	Maximum burst for transaction rate limiting

Provider-Specific Options

Special options that apply to specific cloud storage providers:

Option	Type	Default	Applies To	Description
`s3_no_check_bucket`	boolean	`true`	R2, S3_COMPATIBLE, RAILWAY, ORACLE, WASABI	Skip bucket existence check
`b2_hard_delete`	boolean	`false`	B2	Permanently delete vs hide files
`fast_list`	boolean	`true`	B2	Use recursive directory listing

Provider defaults are automatically applied based on your source and destination location types. Destination defaults take precedence over source defaults, and your explicit configuration always overrides defaults.

Scheduling Tasks

DataRaven supports flexible task scheduling with cron expressions and convenient keywords.

Cron Expressions

Tasks use 5-field UNIX cron expressions evaluated in UTC timezone by the Hatchet scheduler:

┌─────────────── minute (0 - 59)
│ ┌───────────── hour (0 - 23)
│ │ ┌─────────── day of month (1 - 31)
│ │ │ ┌───────── month (1 - 12)
│ │ │ │ ┌─────── day of week (0 - 6) (Sunday = 0)
│ │ │ │ │
* * * * *

Schedule Keywords

For common scheduling needs, use these convenient keywords:

Keyword	Equivalent Cron	Description
`@hourly`	`0 * * * *`	Every hour at minute 0
`@daily`	`0 0 * * *`	Every day at midnight UTC
`@weekly`	`0 0 * * 0`	Every Sunday at midnight UTC
`@monthly`	`0 0 1 * *`	First day of every month at midnight UTC
`@yearly`	`0 0 1 1 *`	January 1st at midnight UTC
`@annually`	`0 0 1 1 *`	Same as @yearly

Examples

# Every 15 minutes
*/15 * * * *

# Every day at 3:30 AM UTC
30 3 * * *

# Every Monday at 9 AM UTC
0 9 * * 1

# First day of every month at 6 AM UTC
0 6 1 * *

# Every 6 hours
0 */6 * * *

Need help with cron expressions? Use crontab.guru for interactive cron schedule building and validation.

Pro Tier Required: Task scheduling is only available on Pro tier ($29/mo). Free tier users can only run tasks on-demand.

Dry Runs

Before executing a task with real data transfer, use dry runs to preview exactly what would happen without actually moving any data.

Benefits of Dry Runs

Preview transfers — See which files would be copied, updated, or deleted
Test filters — Verify your include/exclude patterns work correctly
Validate configuration — Catch configuration errors before real execution
No usage impact — Dry run data transfer doesn’t count toward usage metrics

Running Dry Runs

Web Dashboard
API

Click the “Dry Run” button on any task to execute a preview.

bash POST /tasks/{id}/executions/dry-run

Dry Run Output

Dry runs provide detailed logs showing:

Files that would be transferred
Files that would be skipped (already up-to-date)
Files that would be deleted (sync mode only)
Total bytes and file counts for the operation

Always run a dry run first when using sync mode or complex filter patterns. It’s your safety net against unintended data loss.

Task Execution

Running Tasks

On-Demand Execution
Scheduled Execution

Web Dashboard: Click “Run Now” button on any task API: bash POST /tasks/{id} /executions

Execution Tracking

Every task execution creates a detailed execution record containing:

Real-time logs — Stream transfer progress and debug information
Transfer statistics — Bytes transferred, files processed, error counts
Performance metrics — Duration, transfer rates, checkpoint timing
Downloadable logs — Full execution logs for debugging and audit

Execution States

State	Description
`PENDING`	Execution queued, waiting to start
`RUNNING`	Currently transferring data
`SUCCESS`	Completed successfully
`FAILED`	Completed with errors
`CANCELLED`	User-cancelled before completion

For detailed information about monitoring and managing executions, see the Task Executions Guide.

Task Lifecycle & Status Management

Tasks can be in one of three states, each with different capabilities:

Enabled
Disabled
Archived

Default state for new tasks.

✅ Can be executed on-demand
✅ Scheduled runs are active (if configured)
✅ Can be disabled or archived
✅ Full edit capabilities

Status Transitions

Valid Transitions

Enabled → Disabled ✅
Disabled → Enabled ✅
Enabled → Archived ✅
Disabled → Archived ✅

Invalid Transitions

Archived → Enabled ❌
Archived → Disabled ❌
Archived → anything ❌

Task Deletion vs Archiving

Deletion Rules

Tasks can only be hard deleted under specific conditions:

DELETE only works for tasks with ZERO executions. If a task has any execution history, deletion will fail with a 400 error — you must archive instead.

✅ Can delete: Fresh tasks that have never been executed
❌ Cannot delete: Tasks with any execution history
🔐 Requires: ADMIN role or higher

Why This Restriction?

Execution history serves as an immutable audit trail for:

Compliance requirements
Billing calculations
Forensic analysis
Data governance

Task Duplication

The DataRaven UI supports task duplication for creating similar tasks with slight variations:

What Gets Copied

All configuration settings
RClone options
Filter patterns
Scheduling settings

What Gets Reset

Task name (you must provide a new unique name)
Execution history (new task starts clean)
Created/modified timestamps

Use Cases

Creating staging vs production versions of the same transfer
Setting up similar tasks for different source/destination pairs
Testing configuration variations

Duplication is perfect for creating task templates. Set up one task with complex filtering and performance tuning, then duplicate it for different data sources.

Tier Limits & Performance

DataRaven applies different limits based on your subscription tier. Limits are enforced at execution time, not task creation time.

Free Tier
Pro Tier

Subscription: FreeTask Limits:

Maximum 2 tasks total
On-demand execution only (no scheduling) Performance Limits (applied at runtime):
transfers: 1 (single-threaded transfers)
checkers: 1 (single-threaded checking)
bwlimit: 1G (max 1GB/s bandwidth)

Subscription: $29/monthTask Limits:

Maximum 50 tasks total
Full scheduling capabilities Performance Limits:
transfers: up to 64 (user configurable)
checkers: up to 64 (user configurable)
bwlimit: up to 10G (user configurable)

Pro tier respects your configured values. If you set transfers=8, it stays at 8. The limits are maximums, not forced values.

Tier Transitions

When you change subscription tiers, the effects are immediate:

Upgrade to Pro: Existing tasks immediately get Pro limits on next execution
Downgrade to Free: Existing tasks immediately get Free limits on next execution
No reconfiguration needed — DataRaven handles the transition automatically

Security & Validation

DataRaven implements multiple security layers to protect your data and systems:

Path Validation

Command injection prevention: Source and destination paths are validated against shell metacharacters
Path traversal protection: Paths cannot contain ../ or other directory traversal patterns
Character restrictions: Paths must use safe, filesystem-compatible characters

Configuration Validation

Filter pattern validation: All filter patterns must start with + or - prefixes
RClone config size limit: Configuration payload limited to 50KB maximum
Allowed flags only: Only explicitly defined rclone flags are accepted — extra fields are rejected
Provider defaults: Safe provider-specific defaults are injected server-side

Data Protection

Payload size limits: Request payloads are size-limited to prevent abuse
Rate limiting: API endpoints are rate-limited to prevent overload
Audit logging: All task operations are logged for compliance and debugging

Required Permissions

Different task operations require different user roles:

Operation	Required Role	Notes
View/List Tasks	VIEWER	Read-only access
Create Task	OPERATOR	Can create and configure
Update Task	OPERATOR	Can modify existing tasks
Run/Dry Run Task	OPERATOR	Can execute tasks
Enable/Disable Task	OPERATOR	Can change task status
Archive Task	OPERATOR	Can soft-delete tasks
Delete Task	ADMIN	Can hard-delete (zero executions only)

Role inheritance applies: ADMIN users can perform all OPERATOR and VIEWER operations.

Best Practices

Safety First

Always test with dry runs
Use copy mode for backups
Be extremely careful with sync mode
Start with small file sets

Performance

Tune transfers/checkers for your bandwidth
Use appropriate buffer sizes
Consider time-based bandwidth limiting
Monitor execution performance

Organization

Use descriptive task names
Document complex filter patterns
Group related tasks logically
Archive old tasks instead of deleting

Scheduling

Consider UTC timezone for cron
Spread scheduled tasks across time
Account for transfer duration in schedules
Use keywords for common schedules

Troubleshooting

Common Issues

Task Won't Execute

Possible causes:

Task is disabled or archived
Insufficient user role (need OPERATOR+)
Source/destination locations offline
Invalid rclone configuration

Solutions:

Check task status and enable if needed
Verify user permissions
Test location connectivity
Run a dry run to validate config

Slow Transfer Performance

Possible causes:

Low transfers or checkers values
Bandwidth limiting too restrictive
Small buffer size
Network latency issues

Solutions:

Increase transfers (4-16 for most cases)
Increase checkers (8-32 for many small files)
Adjust bwlimit or remove entirely
Increase buffer_size for high-latency connections

Files Not Transferring

Possible causes:

Restrictive filter patterns
Files already exist (with ignore_existing)
Files newer on destination (with update)
Size/age filters excluding files

Solutions:

Review filter patterns with dry run
Check comparison options settings
Verify source file timestamps
Test with minimal filters first

Sync Mode Deleted Files

Possible causes:

Files exist on destination but not source
Filter patterns changed since last sync
Source path configuration changed

Solutions:

Always dry run sync operations first
Verify source path hasn’t changed
Check that source files still exist
Consider using copy mode instead

Next Steps

Task Executions Guide — Monitor and manage task runs
Connecting Storage — Add more source/destination locations
API Reference — Complete API documentation
Secrets Management — Secure credential storage

Getting Started

Core Features

Configuration & Integrations

Developer Platform

Security

​Transfer Modes

​Creating a Task

​RClone Configuration

​Scheduling Tasks

​Cron Expressions

​Schedule Keywords

​Examples

​Dry Runs

​Benefits of Dry Runs

​Running Dry Runs

​Dry Run Output

​Task Execution

​Running Tasks

​Execution Tracking

​Execution States

​Task Lifecycle & Status Management

​Status Transitions

Valid Transitions

Invalid Transitions

​Task Deletion vs Archiving

​Deletion Rules

​Why This Restriction?

​Task Duplication

​What Gets Copied

​What Gets Reset

​Use Cases

​Tier Limits & Performance

​Tier Transitions

​Security & Validation

​Path Validation

​Configuration Validation

​Data Protection

​Required Permissions

​Best Practices

Safety First

Performance

Organization

Scheduling

​Troubleshooting

​Common Issues

​Next Steps

Transfer Modes

Creating a Task

RClone Configuration

Scheduling Tasks

Cron Expressions

Schedule Keywords

Examples

Dry Runs

Benefits of Dry Runs

Running Dry Runs

Dry Run Output

Task Execution

Running Tasks

Execution Tracking

Execution States

Task Lifecycle & Status Management

Status Transitions

Task Deletion vs Archiving

Deletion Rules

Why This Restriction?

Task Duplication

What Gets Copied

What Gets Reset

Use Cases

Tier Limits & Performance

Tier Transitions

Security & Validation

Path Validation

Configuration Validation

Data Protection

Required Permissions

Best Practices

Troubleshooting

Common Issues

Next Steps