Datarift Bulk Resilience

loom-builderpromptmarkdown
Path: loom-builder/prompts/datarift-bulk-resilience.mdRelative: prompts/datarift-bulk-resilience.mdSubproject: loom-builderCategory: dispatch-prompts
Lines: 27Bytes: 3,072Imports: 0Exports: 0
loom-builderpromptmarkdownprompts

Content Preview

  1. # USER INTENT (verbatim, Mike — this is your success definition)
  2. "No progress bar when doing bulk URLs." AND "when trying to do huge sites 5000 plus pages — it has a tendency to error out and the data is completely unrecoverable. If we could turn the aforementioned weaknesses into strengths it would be a world class app."
  3. Success metric: (1) bulk crawls expose live progress (N of M pages, current URL, phase) consumable by the UI as a progress bar; (2) crawl state checkpoints INCREMENTALLY to D1/