Crawl Operations

Monitor and manage active on-demand crawl runs in real time.

Active Runs
7
Queued Tasks
1,284
Running Tasks
23
Failed
4
New Articles
2,731
arrow_upward183 in last 5 min

Active Runs

Daily News Crawl

MANUAL
Run #92AD91
sync 0%
Sources
54
New
1,821
Duplicate
644
Failed
2
Tasks
2,943
Completed
2,486
Running
18
Waiting
437
Source State Discovered New Detail Progress
Daily Star radio_button_checked RUNNING 97 22 17/22
Prothom Alo check DONE 156 45 45/45
Dhaka Tribune schedule QUEUED - - 0/0
Weekly Tech Blog Scrape
Run #92AD92
0%
expand_more

DAILY STAR

radio_button_checked RUNNING
Started 10:42 AM (12m ago)
Last Activity 2s ago
DISCOVERY PROGRESS
check Homepage sync Politics schedule RSS
URL STATS 17/22 Processed
97 Discovered
22 New

Live Activity Log

expand_less
10:54:12 + New article discovered [Daily Star: /politics/news-123]
10:54:11 + New article discovered [Daily Star: /politics/news-122]
10:54:05 HTTP 429 — retry scheduled (Attempt 1/3) [Prothom Alo]
10:53:50 Duplicate bypassed [Daily Star: /sports/cricket]
10:53:45 Discovery task completed [Prothom Alo: /business]
10:53:12 + New article discovered [Daily Star: /sports/news-88]
Ready for Review
2,731