From aaaaa054a7d3afe1dd3f5e7e2c143fc7714f381a Mon Sep 17 00:00:00 2001 From: lasta Date: Thu, 14 May 2026 15:04:41 +0900 Subject: [PATCH] =?UTF-8?q?Phase=201.3:=20=EC=8B=9C=EB=93=9C=20=ED=81=AC?= =?UTF-8?q?=EB=A1=A4=20=E2=80=94=20by-project=20=EC=97=94=EB=93=9C?= =?UTF-8?q?=ED=8F=AC=EC=9D=B8=ED=8A=B8=20+=20CrawlPage=20=ED=8F=BC/?= =?UTF-8?q?=ED=8F=B4=EB=A7=81/=EC=B7=A8=EC=86=8C?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 세션 영구화: UI_REBUILD_PLAN.md 신설 - 사용자 비전 5가지 흐름 + Phase 보드 + 작업 재개 가이드 - 세션이 끊겨도 이 파일 + git log로 어디까지 했는지 즉시 복원 백엔드 (crawler_platform/app/api/): - routes.py: POST /crawl-site/by-project 추가 — config_path 의존 제거 - SiteCrawlByProjectRequest: project_name 기반, DB project.config 재구성 - run_site_crawl_job이 __config_dict 메타키 인식하도록 최소 변경 - config/loader.py: project_config_from_dict() 헬퍼 추출 (DB dict ↔ ProjectConfig) 프론트엔드 API (src/lib/api/): - crawl.ts: crawlApi.startByProject/getJob/cancel + Zod 스키마 - 종료 상태 판정 헬퍼 isCrawlTerminal() TanStack Query 훅 (src/hooks/): - useCrawl.ts: useStartSiteCrawl, useCrawlJob (2초 폴링, 종료 시 자동 중단), useCancelCrawl - queryKeys에 crawl.job 키 UI 프리미티브 (src/components/ui/): - select.tsx, progress.tsx, badge.tsx (success/warning/destructive variants 포함) CrawlPage 완전 교체 (src/pages/): - 2열 레이아웃: 좌측 크롤 설정 폼 + 우측 진행 상태 - 폼: 소스 선택 (드롭다운, 비어있으면 소스 추가 안내) + 시드 URL + max_depth/max_pages + same_domain_only - react-hook-form + zod 검증 - 진행 상태: 상태 배지, 진행률 바, visited/queued/analyzed 카운터, 최근 페이지, 에러 details - 크롤 종료시 폴링 자동 중단, 완료시 "결과 검토" 버튼 - 취소 버튼 → /crawl-site/jobs/{id}/cancel - sonner 토스트 i18n: crawl.* 키 (한/영) 다음 단계: Phase 1.4 — 자율 연구 (POST /research/run by-project 마이그레이션 + ResearchPage) Co-Authored-By: Claude Opus 4.7 --- UI_REBUILD_PLAN.md | 160 ++++++ crawler_platform/app/api/routes.py | 102 +++- crawler_platform/app/config/loader.py | 5 + .../frontend/public/locales/en/common.json | 28 + .../frontend/public/locales/ko/common.json | 28 + .../web/frontend/src/components/ui/badge.tsx | 36 ++ .../frontend/src/components/ui/progress.tsx | 37 ++ .../web/frontend/src/components/ui/select.tsx | 22 + .../app/web/frontend/src/hooks/queryKeys.ts | 5 + .../app/web/frontend/src/hooks/useCrawl.ts | 44 ++ .../app/web/frontend/src/lib/api/crawl.ts | 82 +++ .../app/web/frontend/src/lib/api/index.ts | 1 + .../app/web/frontend/src/pages/CrawlPage.tsx | 487 +++++++++++++++++- 13 files changed, 1009 insertions(+), 28 deletions(-) create mode 100644 UI_REBUILD_PLAN.md create mode 100644 crawler_platform/app/web/frontend/src/components/ui/badge.tsx create mode 100644 crawler_platform/app/web/frontend/src/components/ui/progress.tsx create mode 100644 crawler_platform/app/web/frontend/src/components/ui/select.tsx create mode 100644 crawler_platform/app/web/frontend/src/hooks/useCrawl.ts create mode 100644 crawler_platform/app/web/frontend/src/lib/api/crawl.ts diff --git a/UI_REBUILD_PLAN.md b/UI_REBUILD_PLAN.md new file mode 100644 index 0000000..7fdc785 --- /dev/null +++ b/UI_REBUILD_PLAN.md @@ -0,0 +1,160 @@ +# React UI 재구축 작업 계획 + +> **목적**: 기존 vanilla JS UI를 React + TypeScript + TanStack Query + shadcn으로 재구축. +> 세션이 끊겨도 이 파일을 보고 이어서 작업할 수 있도록 단일 진실 소스(single source of truth). + +## 사용자 비전 (전체 흐름) + +1. **프로젝트 생성** — 어떤 종류의 온톨로지를 구축할지 도메인을 정하고 프로젝트별로 구분 +2. **온톨로지 기본 요소 입력** — 엔티티, 클레임 등을 직접 입력 또는 참고 사이트 URL로 자동 추출 +3. **자료수집** — 시드 URL에서 시작해 링크를 따라가며 정보 추출 + 장시간 자율 온톨로지 구축 +4. **그래프 보기/편집** — 온톨로지 관계를 그래프 맵으로 시각화하고 편집 +5. **JSON 직접 입력** — 데이터를 JSON으로 직접 넣을 수 있는 UI + +## 작업 보드 + +### ✅ 완료 + +| Phase | 내용 | 커밋 | +|---|---|---| +| Phase 0 | React + Vite + TS 환경 + 라우팅 + Redux placeholder | `1be2c7d` | +| Phase 0.5 | API 클라이언트 + TanStack Query + shadcn UI + AppShell + Dashboard 연결 | `37cad40` | +| Phase 1.1 | 프로젝트 생성 (OnboardingPage 폼 + 백엔드 `POST /projects/inline`, `GET /domains`) | `8681ac8` | +| Phase 1.2 | 참고 소스 관리 (ConfigureSourcesPage CRUD + 백엔드 `POST/DELETE /projects/{name}/sources`) | `461ebc0` | +| Phase 1.3 | 시드 크롤 (CrawlPage + 폴링 + 취소, 백엔드 `POST /crawl-site/by-project`) | (이번 커밋) | + +### 🚧 진행 중 + +(없음 — Phase 1.4 시작 전) + +### ⏳ 대기 + +| Phase | 내용 | 다음 액션 | +|---|---|---| +| **Phase 1.4** | 자율 연구 (research/run 장시간 자율 온톨로지 확장) | `POST /research/run`의 `config_path` 의존을 by-project로 마이그레이션 (Phase 1.3과 동일 패턴) | +| Phase 1.5 | 엔티티/클레임 직접 입력 (`POST /projects/{n}/entities`, `/claims` 백엔드 추가 필요) | 독립 가능 | +| Phase 2 | 그래프 시각화/편집 (Cytoscape 또는 react-flow 래퍼) | 데이터 일부 있어야 의미있음 | +| Phase 3 | JSON Import/Export (대량 데이터 직접 입력) | 독립 가능 | + +--- + +## Phase 1.3 — 시드 크롤 (CrawlPage) + +### 백엔드 (대부분 존재) +- ✅ `POST /crawl-site` — site-wide crawl 시작 (CrawlRequest 모델 확장) +- ✅ `GET /crawl-site/jobs/{id}` — 진행 상태 조회 +- ✅ `POST /crawl-site/jobs/{id}/cancel` — 작업 취소 + +### 프론트엔드 작업 항목 +- [ ] `src/lib/api/crawl.ts` — Zod 스키마 + `crawlApi.startSite/getJob/cancelJob` +- [ ] `src/hooks/useCrawl.ts` — `useStartCrawl`, `useCrawlJob`(폴링), `useCancelCrawl` +- [ ] `src/components/ui/select.tsx` — 소스 선택 드롭다운 +- [ ] `src/components/ui/progress.tsx` — 진행률 표시 바 +- [ ] `src/components/ui/badge.tsx` — 상태 배지 +- [ ] `CrawlPage` 재설계: + - 좌측: 시드 URL 입력 폼 + 소스 선택 + max_depth/max_pages + - 우측: 진행 중 작업 카드 (페이지 수, 단계, 로그) + - 작업 완료 시 결과 페이지로 이동 +- [ ] i18n locale `crawl.*` 키 추가 + +### 검증 포인트 +- [ ] 백엔드 미구동시 명확한 에러 +- [ ] 폴링 간격: 2초, 작업 종료(완료/실패/취소)시 폴링 중단 +- [ ] cancel 버튼 → 백엔드에 취소 요청 + UI 정리 + +--- + +## Phase 1.4 — 자율 연구 (Research Loop) + +### 백엔드 +- ✅ `POST /research/run` — `ResearchRunRequest` (CrawlRequest 확장 + max_depth, max_steps, max_branch, min_relevance...) +- ✅ `GET /projects/{n}/research/sessions` — 세션 이력 +- ✅ `GET /research/sessions/{job_id}` — 단일 세션 상세 + +### 프론트엔드 작업 항목 +- [ ] `src/lib/api/research.ts` + Zod 스키마 +- [ ] `src/hooks/useResearch.ts` — `useStartResearch`, `useResearchSession`, `useResearchHistory` +- [ ] Sidebar에 "자율 연구" 메뉴 추가 +- [ ] 새 페이지 `src/pages/ResearchPage.tsx`: + - 시작 폼: 시드 URL, 목표(goal) 텍스트, max_steps, min_relevance 등 + - 진행 표시: 현재 step, 누적 페이지 수, 발견 엔티티, 관련도 + - 세션 이력 사이드 패널 +- [ ] i18n locale `research.*` 키 + +--- + +## Phase 1.5 — 엔티티/클레임 직접 입력 + +### 백엔드 (신규 필요) +- [ ] `POST /projects/{n}/entities` — 단일 엔티티 직접 생성 +- [ ] `POST /projects/{n}/entities/bulk` — 다수 엔티티 일괄 입력 +- [ ] `POST /projects/{n}/claims` — 단일 클레임 직접 생성 +- [ ] `PATCH /projects/{n}/entities/{id}` — 엔티티 수정 + +### 프론트엔드 작업 항목 +- [ ] `src/lib/api/entities.ts`, `src/lib/api/claims.ts` + Zod +- [ ] `src/hooks/useEntities.ts`, `useClaims.ts` +- [ ] `src/components/ui/dialog.tsx` — 입력 다이얼로그 (Radix UI 검토) +- [ ] `src/components/ui/table.tsx` +- [ ] 새 페이지 `src/pages/OntologyEditorPage.tsx`: + - 엔티티 탭 / 클레임 탭 + - 엔티티 추가/편집 다이얼로그 (label, type, properties) + - 클레임 추가 다이얼로그 (subject/predicate/object/confidence) + - 일괄 입력 토글 (JSON 텍스트 → 파싱) +- [ ] 네비게이션: ConfigureSourcesPage에서 "직접 입력" 진입점 추가 + +--- + +## Phase 2 — 그래프 시각화/편집 + +### 백엔드 (대부분 존재) +- ✅ `GET /projects/{n}/graph/neighborhood` — 노드 주변 부분 그래프 +- ✅ `GET /projects/{n}/graph/query` — 패턴 매칭 쿼리 + +### 프론트엔드 작업 항목 +- [ ] Cytoscape 의존성 그대로 활용 (`legacy/graph.js` 패턴 참고) +- [ ] `src/components/graph/GraphView.tsx` — Cytoscape React 래퍼 + - 노드 클릭 → 인스펙터, 더블 클릭 → neighborhood 확장 +- [ ] 새 페이지 `src/pages/GraphPage.tsx`: + - 좌측: 노드 검색 / 필터 + - 중앙: 그래프 캔버스 + - 우측: 선택 노드 인스펙터 + 편집 +- [ ] 그래프 편집 mutation (노드 속성 변경, 엣지 추가/삭제) — Phase 1.5 백엔드 재사용 + +--- + +## Phase 3 — JSON Import/Export + +### 백엔드 +- [ ] `POST /projects/{n}/import/json` — JSON 일괄 import (엔티티 + 클레임 + 관계) +- [ ] `GET /projects/{n}/export/json` — 전체 온톨로지 JSON 다운로드 + +### 프론트엔드 +- [ ] 새 페이지/탭 `ImportExportPage.tsx`: + - 파일 드래그&드롭 / 텍스트 영역 붙여넣기 + - 미리보기 → 충돌 처리 (덮어쓰기/병합/스킵) + - import 진행 상태 + 결과 요약 +- [ ] Export 버튼 → JSON 다운로드 또는 클립보드 복사 + +--- + +## 작업 재개 가이드 + +세션을 처음 열거나 끊긴 후 다시 시작할 때: + +1. **이 파일을 먼저 읽기** — 현재 상태 파악 +2. **git log --oneline -10** — 최근 커밋과 작업 보드 대조 +3. **🚧 진행 중** 행의 "다음 액션"부터 시작 +4. 완료 후: + - 이 파일의 체크박스/상태/커밋 해시 업데이트 + - 같은 커밋에 이 파일도 함께 포함 + +## 컨벤션 + +- 커밋 메시지: `Phase X.Y: 한 줄 요약 — 핵심 내용` +- 백엔드 변경은 같은 커밋에 묶기 (프론트만 또는 백만 따로 분리 X) +- 모든 폼: react-hook-form + zod +- 모든 서버 통신: TanStack Query 훅을 거침 (Redux 직접 X) +- 모든 새 UI 컴포넌트: shadcn 패턴(forwardRef + cn) +- i18n 키 사용 시 fallback 문자열 같이 (`t("key", "한글 fallback")`) +- 영문/한글 locale 동시 업데이트 diff --git a/crawler_platform/app/api/routes.py b/crawler_platform/app/api/routes.py index 9ed2559..6079771 100644 --- a/crawler_platform/app/api/routes.py +++ b/crawler_platform/app/api/routes.py @@ -7,7 +7,12 @@ from fastapi import BackgroundTasks, HTTPException from pydantic import BaseModel, Field from sqlalchemy import select -from crawler_platform.app.config.loader import ProjectConfig, SourceConfig, load_project_config +from crawler_platform.app.config.loader import ( + ProjectConfig, + SourceConfig, + load_project_config, + project_config_from_dict, +) from crawler_platform.app.core.crawler.discovery import discover_links from crawler_platform.app.core.crawler.fetchers import RobotsPolicy, make_fetcher from crawler_platform.app.core.crawler.pipeline import CrawlPipeline @@ -45,6 +50,42 @@ class SiteCrawlRequest(CrawlRequest): analyze_page_types: list[str] = Field(default_factory=lambda: ["ProductPage", "BrandStoryPage", "ReviewPage"]) +class SiteCrawlByProjectRequest(BaseModel): + """Site crawl invoked against an existing project (no filesystem config_path).""" + + project_name: str + source_name: str + url: str + extractor_provider: str = "lm_studio" + extractor_model: str | None = None + extractor_base_url: str | None = "http://localhost:1234/v1" + check_robots_txt: bool = False + respect_robots_txt: bool | None = None + max_depth: int = 2 + max_pages: int = 50 + same_domain_only: bool = True + analyze_page_types: list[str] = Field( + default_factory=lambda: ["ProductPage", "BrandStoryPage", "ReviewPage"] + ) + + def to_site_crawl_request(self, config_path_placeholder: str = "") -> "SiteCrawlRequest": + """For internal handoff to existing crawl pipeline (config_path is not used).""" + return SiteCrawlRequest( + config_path=config_path_placeholder, + source_name=self.source_name, + url=self.url, + extractor_provider=self.extractor_provider, + extractor_model=self.extractor_model, + extractor_base_url=self.extractor_base_url, + check_robots_txt=self.check_robots_txt, + respect_robots_txt=self.respect_robots_txt, + max_depth=self.max_depth, + max_pages=self.max_pages, + same_domain_only=self.same_domain_only, + analyze_page_types=list(self.analyze_page_types), + ) + + class DiscoverRequest(BaseModel): config_path: str source_name: str @@ -221,9 +262,13 @@ def is_site_crawl_cancel_requested(session, job_id: int) -> bool: def run_site_crawl_job(database_url: str, job_id: int, request_data: dict[str, Any]) -> None: + inline_config = request_data.pop("__config_dict", None) request = SiteCrawlRequest(**request_data) try: - config = load_project_config(request.config_path) + if inline_config is not None: + config = project_config_from_dict(inline_config) + else: + config = load_project_config(request.config_path) apply_crawl_request_overrides(config, request) with session_scope(database_url) as session: job = session.get(models.CrawlJob, job_id) @@ -611,6 +656,59 @@ def register_routes(app, database_url: str) -> None: background_tasks.add_task(run_site_crawl_job, database_url, response["job_id"], request.model_dump()) return response + @app.post("/crawl-site/by-project") + def crawl_site_by_project( + request: SiteCrawlByProjectRequest, background_tasks: BackgroundTasks + ): + """Start a site crawl against an existing DB project (no config_path).""" + with session_scope(database_url) as session: + repo = KnowledgeRepository(session) + project = repo.get_project(request.project_name) + config_dict = project.config or {} + if not config_dict: + raise HTTPException( + status_code=400, + detail=f"Project '{request.project_name}' has no stored config", + ) + config = project_config_from_dict(config_dict) + try: + source = repo.get_source(project.id, request.source_name) + except KeyError as exc: + raise HTTPException(status_code=404, detail=str(exc)) from exc + + inner_request = request.to_site_crawl_request() + apply_crawl_request_overrides(config, inner_request) + + job = models.CrawlJob( + project_id=project.id, + source_id=source.id, + url=request.url, + status="pending", + metadata_json={ + "kind": "site_crawl", + "request": inner_request.model_dump(), + "project_name": request.project_name, + "progress": { + "seed_url": request.url, + "visited_count": 0, + "analyzed_count": 0, + "queued_count": 1, + "skipped_count": 0, + "errors": [], + "pages": [], + }, + }, + ) + session.add(job) + session.flush() + response = crawl_job_response(job) + + task_payload = {**inner_request.model_dump(), "__config_dict": config_dict} + background_tasks.add_task( + run_site_crawl_job, database_url, response["job_id"], task_payload + ) + return response + @app.get("/crawl-site/jobs/{job_id}") def crawl_site_job(job_id: int): with session_scope(database_url) as session: diff --git a/crawler_platform/app/config/loader.py b/crawler_platform/app/config/loader.py index 332d117..1c53e74 100644 --- a/crawler_platform/app/config/loader.py +++ b/crawler_platform/app/config/loader.py @@ -40,6 +40,11 @@ class ProjectConfig: def load_project_config(path: str | Path) -> ProjectConfig: config_path = Path(path) data = _load_mapping(config_path) + return project_config_from_dict(data) + + +def project_config_from_dict(data: dict[str, Any]) -> ProjectConfig: + """Build a ProjectConfig from an in-memory dict (DB row, JSON payload, etc.).""" sources = [SourceConfig(**item) for item in data.get("sources", [])] return ProjectConfig( project_name=data["project_name"], diff --git a/crawler_platform/app/web/frontend/public/locales/en/common.json b/crawler_platform/app/web/frontend/public/locales/en/common.json index 1eaa821..858ff6e 100644 --- a/crawler_platform/app/web/frontend/public/locales/en/common.json +++ b/crawler_platform/app/web/frontend/public/locales/en/common.json @@ -57,6 +57,34 @@ "deleted": "Source deleted: {{name}}", "deleteFailed": "Delete failed: {{msg}}" }, + "crawl": { + "title": "Seed Crawl", + "formTitle": "Crawl Settings", + "formDesc": "Start from a seed URL and follow links to extract information", + "source": "Source", + "pickSource": "Pick a source...", + "noSources": "No sources registered. Add a reference source first.", + "addSource": "Add Source", + "seedUrl": "Seed URL", + "maxDepth": "Max Depth", + "maxPages": "Max Pages", + "sameDomainOnly": "Same domain only", + "start": "Start Crawl", + "started": "Crawl started (job #{{id}})", + "startFailed": "Start failed: {{msg}}", + "cancel": "Cancel", + "cancelRequested": "Cancel requested", + "cancelFailed": "Cancel failed: {{msg}}", + "progressTitle": "Progress", + "idleHint": "Enter a seed URL and start the crawl", + "visited": "Visited", + "queued": "Queued", + "analyzed": "Analyzed", + "latestPage": "Latest Page", + "errorsCount": "{{count}} errors", + "doneHint": "Crawl complete. Go review the results.", + "review": "Review" + }, "common": { "retry": "Retry", "cancel": "Cancel", diff --git a/crawler_platform/app/web/frontend/public/locales/ko/common.json b/crawler_platform/app/web/frontend/public/locales/ko/common.json index e001837..8f17490 100644 --- a/crawler_platform/app/web/frontend/public/locales/ko/common.json +++ b/crawler_platform/app/web/frontend/public/locales/ko/common.json @@ -57,6 +57,34 @@ "deleted": "소스가 삭제되었습니다: {{name}}", "deleteFailed": "삭제 실패: {{msg}}" }, + "crawl": { + "title": "시드 크롤", + "formTitle": "크롤 설정", + "formDesc": "시드 URL에서 시작해 링크를 따라가며 정보를 추출합니다", + "source": "참고 소스", + "pickSource": "소스를 선택하세요...", + "noSources": "등록된 소스가 없습니다. 먼저 참고 소스를 추가하세요.", + "addSource": "소스 추가", + "seedUrl": "시드 URL", + "maxDepth": "최대 깊이", + "maxPages": "최대 페이지", + "sameDomainOnly": "동일 도메인만 따라가기", + "start": "크롤 시작", + "started": "크롤이 시작되었습니다 (job #{{id}})", + "startFailed": "시작 실패: {{msg}}", + "cancel": "취소", + "cancelRequested": "취소 요청됨", + "cancelFailed": "취소 실패: {{msg}}", + "progressTitle": "진행 상태", + "idleHint": "왼쪽에서 시드 URL을 입력하고 시작하세요", + "visited": "방문", + "queued": "대기", + "analyzed": "분석", + "latestPage": "최근 페이지", + "errorsCount": "에러 {{count}}건", + "doneHint": "크롤 완료. 결과 검토로 이동하세요.", + "review": "결과 검토" + }, "common": { "retry": "다시 시도", "cancel": "취소", diff --git a/crawler_platform/app/web/frontend/src/components/ui/badge.tsx b/crawler_platform/app/web/frontend/src/components/ui/badge.tsx new file mode 100644 index 0000000..58f67b3 --- /dev/null +++ b/crawler_platform/app/web/frontend/src/components/ui/badge.tsx @@ -0,0 +1,36 @@ +import * as React from "react"; +import { cva, type VariantProps } from "class-variance-authority"; +import { cn } from "@/lib/utils"; + +const badgeVariants = cva( + "inline-flex items-center rounded-md border px-2 py-0.5 text-xs font-semibold transition-colors focus:outline-none focus:ring-2 focus:ring-ring focus:ring-offset-2", + { + variants: { + variant: { + default: "border-transparent bg-primary text-primary-foreground", + secondary: + "border-transparent bg-secondary text-secondary-foreground", + destructive: + "border-transparent bg-destructive text-destructive-foreground", + outline: "text-foreground", + success: + "border-transparent bg-green-100 text-green-800", + warning: + "border-transparent bg-yellow-100 text-yellow-800", + }, + }, + defaultVariants: { variant: "default" }, + }, +); + +export interface BadgeProps + extends React.HTMLAttributes, + VariantProps {} + +export function Badge({ className, variant, ...props }: BadgeProps) { + return ( + + ); +} + +export { badgeVariants }; diff --git a/crawler_platform/app/web/frontend/src/components/ui/progress.tsx b/crawler_platform/app/web/frontend/src/components/ui/progress.tsx new file mode 100644 index 0000000..acaa513 --- /dev/null +++ b/crawler_platform/app/web/frontend/src/components/ui/progress.tsx @@ -0,0 +1,37 @@ +import * as React from "react"; +import { cn } from "@/lib/utils"; + +export interface ProgressProps extends React.HTMLAttributes { + value?: number; + max?: number; + indeterminate?: boolean; +} + +export const Progress = React.forwardRef( + ({ className, value = 0, max = 100, indeterminate, ...props }, ref) => { + const percent = Math.min(100, Math.max(0, (value / max) * 100)); + return ( +
+
+
+ ); + }, +); +Progress.displayName = "Progress"; diff --git a/crawler_platform/app/web/frontend/src/components/ui/select.tsx b/crawler_platform/app/web/frontend/src/components/ui/select.tsx new file mode 100644 index 0000000..d9d20c5 --- /dev/null +++ b/crawler_platform/app/web/frontend/src/components/ui/select.tsx @@ -0,0 +1,22 @@ +import * as React from "react"; +import { cn } from "@/lib/utils"; + +export type SelectProps = React.SelectHTMLAttributes; + +export const Select = React.forwardRef( + ({ className, children, ...props }, ref) => { + return ( + + ); + }, +); +Select.displayName = "Select"; diff --git a/crawler_platform/app/web/frontend/src/hooks/queryKeys.ts b/crawler_platform/app/web/frontend/src/hooks/queryKeys.ts index 5040a9f..ad8e4c6 100644 --- a/crawler_platform/app/web/frontend/src/hooks/queryKeys.ts +++ b/crawler_platform/app/web/frontend/src/hooks/queryKeys.ts @@ -19,4 +19,9 @@ export const queryKeys = { byProject: (projectName: string) => [...queryKeys.sources.all, "byProject", projectName] as const, }, + crawl: { + all: ["crawl"] as const, + job: (jobId: string) => + [...queryKeys.crawl.all, "job", jobId] as const, + }, }; diff --git a/crawler_platform/app/web/frontend/src/hooks/useCrawl.ts b/crawler_platform/app/web/frontend/src/hooks/useCrawl.ts new file mode 100644 index 0000000..748f3f1 --- /dev/null +++ b/crawler_platform/app/web/frontend/src/hooks/useCrawl.ts @@ -0,0 +1,44 @@ +import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; +import { + CrawlJob, + crawlApi, + isCrawlTerminal, + StartSiteCrawlRequest, +} from "@/lib/api/crawl"; +import { queryKeys } from "./queryKeys"; + +const POLL_INTERVAL_MS = 2000; + +export function useStartSiteCrawl() { + const queryClient = useQueryClient(); + return useMutation({ + mutationFn: (body: StartSiteCrawlRequest) => crawlApi.startByProject(body), + onSuccess: (job) => { + queryClient.setQueryData(queryKeys.crawl.job(job.job_id), job); + }, + }); +} + +export function useCrawlJob(jobId: string | null | undefined) { + return useQuery({ + queryKey: queryKeys.crawl.job(jobId ?? ""), + queryFn: () => crawlApi.getJob(jobId!), + enabled: Boolean(jobId), + refetchInterval: (query) => { + const data = query.state.data as CrawlJob | undefined; + if (!data) return POLL_INTERVAL_MS; + return isCrawlTerminal(data.status) ? false : POLL_INTERVAL_MS; + }, + refetchIntervalInBackground: false, + }); +} + +export function useCancelCrawl() { + const queryClient = useQueryClient(); + return useMutation({ + mutationFn: (jobId: string) => crawlApi.cancel(jobId), + onSuccess: (job) => { + queryClient.setQueryData(queryKeys.crawl.job(job.job_id), job); + }, + }); +} diff --git a/crawler_platform/app/web/frontend/src/lib/api/crawl.ts b/crawler_platform/app/web/frontend/src/lib/api/crawl.ts new file mode 100644 index 0000000..03da5b7 --- /dev/null +++ b/crawler_platform/app/web/frontend/src/lib/api/crawl.ts @@ -0,0 +1,82 @@ +import { z } from "zod"; +import { apiClient } from "./client"; + +const stringFromAny = z.union([z.string(), z.number()]).transform(String); + +export const crawlPageItemSchema = z + .object({ + url: z.string().optional(), + status: z.string().optional(), + page_type: z.string().optional(), + title: z.string().nullable().optional(), + error: z.string().nullable().optional(), + }) + .passthrough(); + +export const crawlProgressSchema = z + .object({ + seed_url: z.string().optional(), + visited_count: z.number().optional(), + analyzed_count: z.number().optional(), + queued_count: z.number().optional(), + skipped_count: z.number().optional(), + errors: z.array(z.string()).optional(), + pages: z.array(crawlPageItemSchema).optional(), + latest_page: crawlPageItemSchema.optional(), + }) + .passthrough(); + +export const crawlJobSchema = z.object({ + job_id: stringFromAny, + status: z.string(), + url: z.string().nullable().optional(), + error: z.string().nullable().optional(), + scheduled_at: z.string().nullable().optional(), + started_at: z.string().nullable().optional(), + finished_at: z.string().nullable().optional(), + progress: crawlProgressSchema.default({}), + request: z.record(z.string(), z.unknown()).default({}), +}); + +export type CrawlJob = z.infer; +export type CrawlProgress = z.infer; + +export interface StartSiteCrawlRequest { + project_name: string; + source_name: string; + url: string; + max_depth?: number; + max_pages?: number; + same_domain_only?: boolean; + analyze_page_types?: string[]; + extractor_provider?: string; + extractor_model?: string | null; + extractor_base_url?: string | null; + check_robots_txt?: boolean; + respect_robots_txt?: boolean | null; +} + +export const TERMINAL_CRAWL_STATUSES = new Set([ + "completed", + "failed", + "canceled", +]); + +export function isCrawlTerminal(status: string): boolean { + return TERMINAL_CRAWL_STATUSES.has(status); +} + +export const crawlApi = { + startByProject: (body: StartSiteCrawlRequest) => + apiClient.post("/crawl-site/by-project", crawlJobSchema, body), + getJob: (jobId: string) => + apiClient.get( + `/crawl-site/jobs/${encodeURIComponent(jobId)}`, + crawlJobSchema, + ), + cancel: (jobId: string) => + apiClient.post( + `/crawl-site/jobs/${encodeURIComponent(jobId)}/cancel`, + crawlJobSchema, + ), +}; diff --git a/crawler_platform/app/web/frontend/src/lib/api/index.ts b/crawler_platform/app/web/frontend/src/lib/api/index.ts index b24ccd4..458e141 100644 --- a/crawler_platform/app/web/frontend/src/lib/api/index.ts +++ b/crawler_platform/app/web/frontend/src/lib/api/index.ts @@ -3,3 +3,4 @@ export * from "./projects"; export * from "./domains"; export * from "./ontology"; export * from "./sources"; +export * from "./crawl"; diff --git a/crawler_platform/app/web/frontend/src/pages/CrawlPage.tsx b/crawler_platform/app/web/frontend/src/pages/CrawlPage.tsx index 4daeb34..2e33ff2 100644 --- a/crawler_platform/app/web/frontend/src/pages/CrawlPage.tsx +++ b/crawler_platform/app/web/frontend/src/pages/CrawlPage.tsx @@ -1,39 +1,474 @@ +import { useState } from "react"; import { useNavigate, useParams } from "react-router-dom"; import { useTranslation } from "react-i18next"; +import { useForm } from "react-hook-form"; +import { zodResolver } from "@hookform/resolvers/zod"; +import { z } from "zod"; +import { toast } from "sonner"; +import { + AlertCircle, + ArrowLeft, + ArrowRight, + Ban, + CheckCircle2, + Globe, + Loader2, + PlayCircle, + XCircle, +} from "lucide-react"; +import { + Card, + CardContent, + CardDescription, + CardHeader, + CardTitle, +} from "@/components/ui/card"; +import { Button } from "@/components/ui/button"; +import { Input } from "@/components/ui/input"; +import { Label } from "@/components/ui/label"; +import { Select } from "@/components/ui/select"; +import { Progress } from "@/components/ui/progress"; +import { Badge, BadgeProps } from "@/components/ui/badge"; +import { Skeleton } from "@/components/ui/skeleton"; +import { useProject } from "@/hooks/useProjects"; +import { + useCancelCrawl, + useCrawlJob, + useStartSiteCrawl, +} from "@/hooks/useCrawl"; +import { isCrawlTerminal } from "@/lib/api/crawl"; + +const startCrawlSchema = z.object({ + source_name: z.string().min(1, "소스를 선택하세요"), + url: z.string().url("올바른 URL 형식이 아닙니다"), + max_depth: z.number().int().min(0).max(10), + max_pages: z.number().int().min(1).max(500), + same_domain_only: z.boolean(), +}); + +type StartCrawlFormValues = z.infer; + +function statusBadgeVariant(status: string): BadgeProps["variant"] { + switch (status) { + case "completed": + return "success"; + case "failed": + return "destructive"; + case "canceled": + return "secondary"; + case "cancel_requested": + return "warning"; + case "running": + case "pending": + return "default"; + default: + return "outline"; + } +} + +function StatusBadge({ status }: { status: string }) { + return {status}; +} export default function CrawlPage() { const navigate = useNavigate(); - const { projectId } = useParams(); + const { projectId } = useParams<{ projectId: string }>(); const { t } = useTranslation(); + const projectName = projectId ?? ""; + + const { data: project, isLoading, isError, error, refetch } = useProject(projectName); + const [activeJobId, setActiveJobId] = useState(null); + const { data: job } = useCrawlJob(activeJobId); + const startCrawl = useStartSiteCrawl(); + const cancelCrawl = useCancelCrawl(); + + const { + register, + handleSubmit, + setValue, + watch, + formState: { errors, isSubmitting }, + } = useForm({ + resolver: zodResolver(startCrawlSchema), + defaultValues: { + source_name: "", + url: "", + max_depth: 2, + max_pages: 30, + same_domain_only: true, + }, + }); + + const onStart = async (values: StartCrawlFormValues) => { + try { + const created = await startCrawl.mutateAsync({ + project_name: projectName, + ...values, + }); + setActiveJobId(created.job_id); + toast.success( + t("crawl.started", "크롤이 시작되었습니다 (job #{{id}})", { + id: created.job_id, + }), + ); + } catch (e) { + toast.error( + t("crawl.startFailed", "시작 실패: {{msg}}", { + msg: (e as Error).message, + }), + ); + } + }; + + const onCancel = async () => { + if (!activeJobId) return; + try { + await cancelCrawl.mutateAsync(activeJobId); + toast.info(t("crawl.cancelRequested", "취소 요청됨")); + } catch (e) { + toast.error( + t("crawl.cancelFailed", "취소 실패: {{msg}}", { + msg: (e as Error).message, + }), + ); + } + }; + + const sources = project?.sources ?? []; + const sourceName = watch("source_name"); + const selectedSource = sources.find((s) => s.name === sourceName); + const progress = job?.progress; + const visited = progress?.visited_count ?? 0; + const queued = progress?.queued_count ?? 0; + const analyzed = progress?.analyzed_count ?? 0; + const errors_ = progress?.errors ?? []; + const max_pages = watch("max_pages"); + const progressPct = job && max_pages + ? Math.min(100, (visited / max_pages) * 100) + : 0; + const terminal = job ? isCrawlTerminal(job.status) : false; + const running = job && !terminal; return ( -
-
-

- {t("Build Ontology")} -

-

- {t("Step 3 of 4: Auto-crawl and build ontology with Phase 5 + 7")} -

- -
-

{t("Crawl pipeline will appear here")}

+
+
+ +
+

+ {t("crawl.title", "시드 크롤")} +

+ {project && ( +

+ {project.name} + + {" "} + · {project.domain} + +

+ )}
+ {terminal && job?.status === "completed" && ( + + )} +
-
- - -
+ {isError && ( + + +
+ + {(error as Error).message} +
+ + + + )} + +
+ + + {t("crawl.formTitle", "크롤 설정")} + + {t( + "crawl.formDesc", + "시드 URL에서 시작해 링크를 따라가며 정보를 추출합니다", + )} + + + +
+
+ + {isLoading ? ( + + ) : ( + + )} + {sources.length === 0 && !isLoading && ( +

+ {t( + "crawl.noSources", + "등록된 소스가 없습니다. 먼저 참고 소스를 추가하세요.", + )}{" "} + +

+ )} + {errors.source_name && ( +

+ {errors.source_name.message} +

+ )} +
+ +
+ + + {errors.url && ( +

+ {errors.url.message} +

+ )} +
+ +
+
+ + + {errors.max_depth && ( +

+ {errors.max_depth.message} +

+ )} +
+
+ + + {errors.max_pages && ( +

+ {errors.max_pages.message} +

+ )} +
+
+ + + + +
+ + + + + +
+
+ + {t("crawl.progressTitle", "진행 상태")} + + {job && ( + + job #{job.job_id} + + + )} +
+ {running && ( + + )} +
+ + + {!job && ( +
+ +

+ {t( + "crawl.idleHint", + "왼쪽에서 시드 URL을 입력하고 시작하세요", + )} +

+
+ )} + + {job && ( +
+
+
+ + {t("crawl.visited", "방문")} {visited} + {" / "} + {max_pages} + + + {t("crawl.queued", "대기")} {queued} ·{" "} + {t("crawl.analyzed", "분석")} {analyzed} + +
+ +
+ + {job.url && ( +
+
+ {t("crawl.seedUrl", "시드 URL")} +
+ + {job.url} + +
+ )} + + {progress?.latest_page && ( +
+
+ {t("crawl.latestPage", "최근 페이지")} +
+
+ {progress.latest_page.title || progress.latest_page.url} +
+ {progress.latest_page.page_type && ( + + {progress.latest_page.page_type} + + )} +
+ )} + + {job.error && ( +
+ + {job.error} +
+ )} + + {errors_.length > 0 && ( +
+ + {t("crawl.errorsCount", "에러 {{count}}건", { + count: errors_.length, + })} + +
    + {errors_.map((e, i) => ( +
  • + {e} +
    • + ))} +
+
+ )} + + {terminal && job.status === "completed" && ( +
+ + {t("crawl.doneHint", "크롤 완료. 결과 검토로 이동하세요.")} +
+ )} +
+ )} + +
);