conversion_project/QUEUE_RETRY_FEATURE.md

Queue Retry Feature Documentation

Overview

Added smart retry logic for batch queue processing to handle temporary source availability issues. When a batch run completes with 2+ "Folder not found" errors (indicating the source is down), the system will automatically retry at regular intervals for a configurable timeout period.

New CLI Parameters

--retry-minutes (int, default: from config.xml)

Minutes to wait between retry attempts when batch fails with 2+ "Folder not found" errors.

• Default: 10 minutes (configurable in config.xml)
• Example: python main.py --paths-file paths.txt --retry-minutes 5

--retry-timeout (int, default: from config.xml)

Total minutes to keep retrying before giving up.

• Default: 60 minutes (configurable in config.xml)
• If source becomes reachable during retry window, queue immediately restarts
• Example: python main.py --paths-file paths.txt --retry-timeout 120

Configuration (config.xml)

Added new section <queue_retry> in config.xml with defaults:

<queue_retry>
  <!-- Minutes to wait between retry attempts -->
  <retry_minutes>10</retry_minutes>
  
  <!-- Total minutes to keep retrying before giving up -->
  <retry_timeout>60</retry_timeout>
</queue_retry>

How It Works

Trigger Conditions

Retry logic activates when ALL of the following are true:

1. Queue processing completes
2. Failed count > 2
3. All failures are "Folder not found" type (indicates source path is unreachable)

Retry Loop Behavior

Once activated, the system enters a retry loop that:

1. Checks source reachability every N minutes (--retry-minutes)
2. If source becomes reachable:
   • Immediately restarts batch queue processing
   • Processes all previously failed items
   • Displays detailed progress with timestamps
3. If source stays unreachable:
   • Displays countdown showing elapsed/remaining time
   • Continues checking at configured interval
4. If timeout exceeded:
   • Exits retry loop
   • Displays final summary showing total attempts

Example Output

================================================================================
✓ BATCH PROCESSING COMPLETE
  Total items processed: 33
  ✓ Succeeded: 8
  ❌ Failed: 25
================================================================================

================================================================================
⚠️  RETRY LOGIC TRIGGERED
  Failed items: 25 | Folder not found: 25
  This suggests the source location (P:\tv\Show\Season 3) may be temporarily unreachable
================================================================================

🔄 Retry Configuration:
  - Retry interval: 10 minute(s)
  - Total retry timeout: 60 minute(s)
  - Will check source availability and retry if it becomes reachable

⏳ Attempt 1: Source not yet reachable
   Elapsed: 0.0m | Remaining: 60.0m
   Waiting 10 minute(s) before next check...

⏳ Attempt 2: Source not yet reachable
   Elapsed: 10.1m | Remaining: 49.9m
   Waiting 10 minute(s) before next check...

✅ Source is now reachable! (P:\tv\Show\Season 3)
   Restarting queue processing...

================================================================================
🔃 RESTARTING BATCH QUEUE
================================================================================

📋 Found 25 item(s) to retry
[Processing continues with failed items...]

Usage Examples

Using defaults (10 min interval, 60 min timeout)

python main.py --paths-file paths.txt

Custom retry interval (5 minute checks, 120 minute timeout)

python main.py --paths-file paths.txt --retry-minutes 5 --retry-timeout 120

Disable retry by setting timeout to 0

python main.py --paths-file paths.txt --retry-timeout 0

Quick test (1 minute interval, 5 minute timeout)

python main.py --paths-file paths.txt --retry-minutes 1 --retry-timeout 5

Implementation Details

New Functions

• is_path_reachable(path: Path) -> bool: Checks if a path (network share, etc.) is accessible

Modified Components

• main.py:

  • Added import: import time
  • Added --retry-minutes and --retry-timeout CLI arguments
  • Added is_path_reachable() helper function
  • Added retry loop logic after batch queue completion
  • Detects folder-not-found failures and triggers retry logic

• config.xml:

  • Added <queue_retry> section with default values

Notes

• Retry logic only activates for queue mode (--paths-file flag)
• Single-file processing is not affected
• Retry attempts are logged with timestamps for audit trail
• Network paths (e.g., P:\tv) are checked using Path.exists() which properly handles network timeouts
• Each retry attempt shows elapsed and remaining time for transparency
• If source becomes available, all failed items are re-processed with original parameters