Phase 1.3: 시드 크롤 — by-project 엔드포인트 + CrawlPage 폼/폴링/취소

세션 영구화: UI_REBUILD_PLAN.md 신설
- 사용자 비전 5가지 흐름 + Phase 보드 + 작업 재개 가이드
- 세션이 끊겨도 이 파일 + git log로 어디까지 했는지 즉시 복원

백엔드 (crawler_platform/app/api/):
- routes.py: POST /crawl-site/by-project 추가 — config_path 의존 제거
  - SiteCrawlByProjectRequest: project_name 기반, DB project.config 재구성
  - run_site_crawl_job이 __config_dict 메타키 인식하도록 최소 변경
- config/loader.py: project_config_from_dict() 헬퍼 추출 (DB dict ↔ ProjectConfig)

프론트엔드 API (src/lib/api/):
- crawl.ts: crawlApi.startByProject/getJob/cancel + Zod 스키마
  - 종료 상태 판정 헬퍼 isCrawlTerminal()

TanStack Query 훅 (src/hooks/):
- useCrawl.ts: useStartSiteCrawl, useCrawlJob (2초 폴링, 종료 시 자동 중단), useCancelCrawl
- queryKeys에 crawl.job 키

UI 프리미티브 (src/components/ui/):
- select.tsx, progress.tsx, badge.tsx (success/warning/destructive variants 포함)

CrawlPage 완전 교체 (src/pages/):
- 2열 레이아웃: 좌측 크롤 설정 폼 + 우측 진행 상태
- 폼: 소스 선택 (드롭다운, 비어있으면 소스 추가 안내) + 시드 URL + max_depth/max_pages + same_domain_only
  - react-hook-form + zod 검증
- 진행 상태: 상태 배지, 진행률 바, visited/queued/analyzed 카운터, 최근 페이지, 에러 details
- 크롤 종료시 폴링 자동 중단, 완료시 "결과 검토" 버튼
- 취소 버튼 → /crawl-site/jobs/{id}/cancel
- sonner 토스트

i18n: crawl.* 키 (한/영)

다음 단계: Phase 1.4 — 자율 연구 (POST /research/run by-project 마이그레이션 + ResearchPage)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
lasta
2026-05-14 15:04:41 +09:00
parent 461ebc062b
commit aaaaa054a7
13 changed files with 1009 additions and 28 deletions

160
UI_REBUILD_PLAN.md Normal file
View File

@@ -0,0 +1,160 @@
# React UI 재구축 작업 계획
> **목적**: 기존 vanilla JS UI를 React + TypeScript + TanStack Query + shadcn으로 재구축.
> 세션이 끊겨도 이 파일을 보고 이어서 작업할 수 있도록 단일 진실 소스(single source of truth).
## 사용자 비전 (전체 흐름)
1. **프로젝트 생성** — 어떤 종류의 온톨로지를 구축할지 도메인을 정하고 프로젝트별로 구분
2. **온톨로지 기본 요소 입력** — 엔티티, 클레임 등을 직접 입력 또는 참고 사이트 URL로 자동 추출
3. **자료수집** — 시드 URL에서 시작해 링크를 따라가며 정보 추출 + 장시간 자율 온톨로지 구축
4. **그래프 보기/편집** — 온톨로지 관계를 그래프 맵으로 시각화하고 편집
5. **JSON 직접 입력** — 데이터를 JSON으로 직접 넣을 수 있는 UI
## 작업 보드
### ✅ 완료
| Phase | 내용 | 커밋 |
|---|---|---|
| Phase 0 | React + Vite + TS 환경 + 라우팅 + Redux placeholder | `1be2c7d` |
| Phase 0.5 | API 클라이언트 + TanStack Query + shadcn UI + AppShell + Dashboard 연결 | `37cad40` |
| Phase 1.1 | 프로젝트 생성 (OnboardingPage 폼 + 백엔드 `POST /projects/inline`, `GET /domains`) | `8681ac8` |
| Phase 1.2 | 참고 소스 관리 (ConfigureSourcesPage CRUD + 백엔드 `POST/DELETE /projects/{name}/sources`) | `461ebc0` |
| Phase 1.3 | 시드 크롤 (CrawlPage + 폴링 + 취소, 백엔드 `POST /crawl-site/by-project`) | (이번 커밋) |
### 🚧 진행 중
(없음 — Phase 1.4 시작 전)
### ⏳ 대기
| Phase | 내용 | 다음 액션 |
|---|---|---|
| **Phase 1.4** | 자율 연구 (research/run 장시간 자율 온톨로지 확장) | `POST /research/run``config_path` 의존을 by-project로 마이그레이션 (Phase 1.3과 동일 패턴) |
| Phase 1.5 | 엔티티/클레임 직접 입력 (`POST /projects/{n}/entities`, `/claims` 백엔드 추가 필요) | 독립 가능 |
| Phase 2 | 그래프 시각화/편집 (Cytoscape 또는 react-flow 래퍼) | 데이터 일부 있어야 의미있음 |
| Phase 3 | JSON Import/Export (대량 데이터 직접 입력) | 독립 가능 |
---
## Phase 1.3 — 시드 크롤 (CrawlPage)
### 백엔드 (대부분 존재)
-`POST /crawl-site` — site-wide crawl 시작 (CrawlRequest 모델 확장)
-`GET /crawl-site/jobs/{id}` — 진행 상태 조회
-`POST /crawl-site/jobs/{id}/cancel` — 작업 취소
### 프론트엔드 작업 항목
- [ ] `src/lib/api/crawl.ts` — Zod 스키마 + `crawlApi.startSite/getJob/cancelJob`
- [ ] `src/hooks/useCrawl.ts``useStartCrawl`, `useCrawlJob`(폴링), `useCancelCrawl`
- [ ] `src/components/ui/select.tsx` — 소스 선택 드롭다운
- [ ] `src/components/ui/progress.tsx` — 진행률 표시 바
- [ ] `src/components/ui/badge.tsx` — 상태 배지
- [ ] `CrawlPage` 재설계:
- 좌측: 시드 URL 입력 폼 + 소스 선택 + max_depth/max_pages
- 우측: 진행 중 작업 카드 (페이지 수, 단계, 로그)
- 작업 완료 시 결과 페이지로 이동
- [ ] i18n locale `crawl.*` 키 추가
### 검증 포인트
- [ ] 백엔드 미구동시 명확한 에러
- [ ] 폴링 간격: 2초, 작업 종료(완료/실패/취소)시 폴링 중단
- [ ] cancel 버튼 → 백엔드에 취소 요청 + UI 정리
---
## Phase 1.4 — 자율 연구 (Research Loop)
### 백엔드
-`POST /research/run``ResearchRunRequest` (CrawlRequest 확장 + max_depth, max_steps, max_branch, min_relevance...)
-`GET /projects/{n}/research/sessions` — 세션 이력
-`GET /research/sessions/{job_id}` — 단일 세션 상세
### 프론트엔드 작업 항목
- [ ] `src/lib/api/research.ts` + Zod 스키마
- [ ] `src/hooks/useResearch.ts``useStartResearch`, `useResearchSession`, `useResearchHistory`
- [ ] Sidebar에 "자율 연구" 메뉴 추가
- [ ] 새 페이지 `src/pages/ResearchPage.tsx`:
- 시작 폼: 시드 URL, 목표(goal) 텍스트, max_steps, min_relevance 등
- 진행 표시: 현재 step, 누적 페이지 수, 발견 엔티티, 관련도
- 세션 이력 사이드 패널
- [ ] i18n locale `research.*`
---
## Phase 1.5 — 엔티티/클레임 직접 입력
### 백엔드 (신규 필요)
- [ ] `POST /projects/{n}/entities` — 단일 엔티티 직접 생성
- [ ] `POST /projects/{n}/entities/bulk` — 다수 엔티티 일괄 입력
- [ ] `POST /projects/{n}/claims` — 단일 클레임 직접 생성
- [ ] `PATCH /projects/{n}/entities/{id}` — 엔티티 수정
### 프론트엔드 작업 항목
- [ ] `src/lib/api/entities.ts`, `src/lib/api/claims.ts` + Zod
- [ ] `src/hooks/useEntities.ts`, `useClaims.ts`
- [ ] `src/components/ui/dialog.tsx` — 입력 다이얼로그 (Radix UI 검토)
- [ ] `src/components/ui/table.tsx`
- [ ] 새 페이지 `src/pages/OntologyEditorPage.tsx`:
- 엔티티 탭 / 클레임 탭
- 엔티티 추가/편집 다이얼로그 (label, type, properties)
- 클레임 추가 다이얼로그 (subject/predicate/object/confidence)
- 일괄 입력 토글 (JSON 텍스트 → 파싱)
- [ ] 네비게이션: ConfigureSourcesPage에서 "직접 입력" 진입점 추가
---
## Phase 2 — 그래프 시각화/편집
### 백엔드 (대부분 존재)
-`GET /projects/{n}/graph/neighborhood` — 노드 주변 부분 그래프
-`GET /projects/{n}/graph/query` — 패턴 매칭 쿼리
### 프론트엔드 작업 항목
- [ ] Cytoscape 의존성 그대로 활용 (`legacy/graph.js` 패턴 참고)
- [ ] `src/components/graph/GraphView.tsx` — Cytoscape React 래퍼
- 노드 클릭 → 인스펙터, 더블 클릭 → neighborhood 확장
- [ ] 새 페이지 `src/pages/GraphPage.tsx`:
- 좌측: 노드 검색 / 필터
- 중앙: 그래프 캔버스
- 우측: 선택 노드 인스펙터 + 편집
- [ ] 그래프 편집 mutation (노드 속성 변경, 엣지 추가/삭제) — Phase 1.5 백엔드 재사용
---
## Phase 3 — JSON Import/Export
### 백엔드
- [ ] `POST /projects/{n}/import/json` — JSON 일괄 import (엔티티 + 클레임 + 관계)
- [ ] `GET /projects/{n}/export/json` — 전체 온톨로지 JSON 다운로드
### 프론트엔드
- [ ] 새 페이지/탭 `ImportExportPage.tsx`:
- 파일 드래그&드롭 / 텍스트 영역 붙여넣기
- 미리보기 → 충돌 처리 (덮어쓰기/병합/스킵)
- import 진행 상태 + 결과 요약
- [ ] Export 버튼 → JSON 다운로드 또는 클립보드 복사
---
## 작업 재개 가이드
세션을 처음 열거나 끊긴 후 다시 시작할 때:
1. **이 파일을 먼저 읽기** — 현재 상태 파악
2. **git log --oneline -10** — 최근 커밋과 작업 보드 대조
3. **🚧 진행 중** 행의 "다음 액션"부터 시작
4. 완료 후:
- 이 파일의 체크박스/상태/커밋 해시 업데이트
- 같은 커밋에 이 파일도 함께 포함
## 컨벤션
- 커밋 메시지: `Phase X.Y: 한 줄 요약 — 핵심 내용`
- 백엔드 변경은 같은 커밋에 묶기 (프론트만 또는 백만 따로 분리 X)
- 모든 폼: react-hook-form + zod
- 모든 서버 통신: TanStack Query 훅을 거침 (Redux 직접 X)
- 모든 새 UI 컴포넌트: shadcn 패턴(forwardRef + cn)
- i18n 키 사용 시 fallback 문자열 같이 (`t("key", "한글 fallback")`)
- 영문/한글 locale 동시 업데이트

View File

@@ -7,7 +7,12 @@ from fastapi import BackgroundTasks, HTTPException
from pydantic import BaseModel, Field
from sqlalchemy import select
from crawler_platform.app.config.loader import ProjectConfig, SourceConfig, load_project_config
from crawler_platform.app.config.loader import (
ProjectConfig,
SourceConfig,
load_project_config,
project_config_from_dict,
)
from crawler_platform.app.core.crawler.discovery import discover_links
from crawler_platform.app.core.crawler.fetchers import RobotsPolicy, make_fetcher
from crawler_platform.app.core.crawler.pipeline import CrawlPipeline
@@ -45,6 +50,42 @@ class SiteCrawlRequest(CrawlRequest):
analyze_page_types: list[str] = Field(default_factory=lambda: ["ProductPage", "BrandStoryPage", "ReviewPage"])
class SiteCrawlByProjectRequest(BaseModel):
"""Site crawl invoked against an existing project (no filesystem config_path)."""
project_name: str
source_name: str
url: str
extractor_provider: str = "lm_studio"
extractor_model: str | None = None
extractor_base_url: str | None = "http://localhost:1234/v1"
check_robots_txt: bool = False
respect_robots_txt: bool | None = None
max_depth: int = 2
max_pages: int = 50
same_domain_only: bool = True
analyze_page_types: list[str] = Field(
default_factory=lambda: ["ProductPage", "BrandStoryPage", "ReviewPage"]
)
def to_site_crawl_request(self, config_path_placeholder: str = "") -> "SiteCrawlRequest":
"""For internal handoff to existing crawl pipeline (config_path is not used)."""
return SiteCrawlRequest(
config_path=config_path_placeholder,
source_name=self.source_name,
url=self.url,
extractor_provider=self.extractor_provider,
extractor_model=self.extractor_model,
extractor_base_url=self.extractor_base_url,
check_robots_txt=self.check_robots_txt,
respect_robots_txt=self.respect_robots_txt,
max_depth=self.max_depth,
max_pages=self.max_pages,
same_domain_only=self.same_domain_only,
analyze_page_types=list(self.analyze_page_types),
)
class DiscoverRequest(BaseModel):
config_path: str
source_name: str
@@ -221,9 +262,13 @@ def is_site_crawl_cancel_requested(session, job_id: int) -> bool:
def run_site_crawl_job(database_url: str, job_id: int, request_data: dict[str, Any]) -> None:
inline_config = request_data.pop("__config_dict", None)
request = SiteCrawlRequest(**request_data)
try:
config = load_project_config(request.config_path)
if inline_config is not None:
config = project_config_from_dict(inline_config)
else:
config = load_project_config(request.config_path)
apply_crawl_request_overrides(config, request)
with session_scope(database_url) as session:
job = session.get(models.CrawlJob, job_id)
@@ -611,6 +656,59 @@ def register_routes(app, database_url: str) -> None:
background_tasks.add_task(run_site_crawl_job, database_url, response["job_id"], request.model_dump())
return response
@app.post("/crawl-site/by-project")
def crawl_site_by_project(
request: SiteCrawlByProjectRequest, background_tasks: BackgroundTasks
):
"""Start a site crawl against an existing DB project (no config_path)."""
with session_scope(database_url) as session:
repo = KnowledgeRepository(session)
project = repo.get_project(request.project_name)
config_dict = project.config or {}
if not config_dict:
raise HTTPException(
status_code=400,
detail=f"Project '{request.project_name}' has no stored config",
)
config = project_config_from_dict(config_dict)
try:
source = repo.get_source(project.id, request.source_name)
except KeyError as exc:
raise HTTPException(status_code=404, detail=str(exc)) from exc
inner_request = request.to_site_crawl_request()
apply_crawl_request_overrides(config, inner_request)
job = models.CrawlJob(
project_id=project.id,
source_id=source.id,
url=request.url,
status="pending",
metadata_json={
"kind": "site_crawl",
"request": inner_request.model_dump(),
"project_name": request.project_name,
"progress": {
"seed_url": request.url,
"visited_count": 0,
"analyzed_count": 0,
"queued_count": 1,
"skipped_count": 0,
"errors": [],
"pages": [],
},
},
)
session.add(job)
session.flush()
response = crawl_job_response(job)
task_payload = {**inner_request.model_dump(), "__config_dict": config_dict}
background_tasks.add_task(
run_site_crawl_job, database_url, response["job_id"], task_payload
)
return response
@app.get("/crawl-site/jobs/{job_id}")
def crawl_site_job(job_id: int):
with session_scope(database_url) as session:

View File

@@ -40,6 +40,11 @@ class ProjectConfig:
def load_project_config(path: str | Path) -> ProjectConfig:
config_path = Path(path)
data = _load_mapping(config_path)
return project_config_from_dict(data)
def project_config_from_dict(data: dict[str, Any]) -> ProjectConfig:
"""Build a ProjectConfig from an in-memory dict (DB row, JSON payload, etc.)."""
sources = [SourceConfig(**item) for item in data.get("sources", [])]
return ProjectConfig(
project_name=data["project_name"],

View File

@@ -57,6 +57,34 @@
"deleted": "Source deleted: {{name}}",
"deleteFailed": "Delete failed: {{msg}}"
},
"crawl": {
"title": "Seed Crawl",
"formTitle": "Crawl Settings",
"formDesc": "Start from a seed URL and follow links to extract information",
"source": "Source",
"pickSource": "Pick a source...",
"noSources": "No sources registered. Add a reference source first.",
"addSource": "Add Source",
"seedUrl": "Seed URL",
"maxDepth": "Max Depth",
"maxPages": "Max Pages",
"sameDomainOnly": "Same domain only",
"start": "Start Crawl",
"started": "Crawl started (job #{{id}})",
"startFailed": "Start failed: {{msg}}",
"cancel": "Cancel",
"cancelRequested": "Cancel requested",
"cancelFailed": "Cancel failed: {{msg}}",
"progressTitle": "Progress",
"idleHint": "Enter a seed URL and start the crawl",
"visited": "Visited",
"queued": "Queued",
"analyzed": "Analyzed",
"latestPage": "Latest Page",
"errorsCount": "{{count}} errors",
"doneHint": "Crawl complete. Go review the results.",
"review": "Review"
},
"common": {
"retry": "Retry",
"cancel": "Cancel",

View File

@@ -57,6 +57,34 @@
"deleted": "소스가 삭제되었습니다: {{name}}",
"deleteFailed": "삭제 실패: {{msg}}"
},
"crawl": {
"title": "시드 크롤",
"formTitle": "크롤 설정",
"formDesc": "시드 URL에서 시작해 링크를 따라가며 정보를 추출합니다",
"source": "참고 소스",
"pickSource": "소스를 선택하세요...",
"noSources": "등록된 소스가 없습니다. 먼저 참고 소스를 추가하세요.",
"addSource": "소스 추가",
"seedUrl": "시드 URL",
"maxDepth": "최대 깊이",
"maxPages": "최대 페이지",
"sameDomainOnly": "동일 도메인만 따라가기",
"start": "크롤 시작",
"started": "크롤이 시작되었습니다 (job #{{id}})",
"startFailed": "시작 실패: {{msg}}",
"cancel": "취소",
"cancelRequested": "취소 요청됨",
"cancelFailed": "취소 실패: {{msg}}",
"progressTitle": "진행 상태",
"idleHint": "왼쪽에서 시드 URL을 입력하고 시작하세요",
"visited": "방문",
"queued": "대기",
"analyzed": "분석",
"latestPage": "최근 페이지",
"errorsCount": "에러 {{count}}건",
"doneHint": "크롤 완료. 결과 검토로 이동하세요.",
"review": "결과 검토"
},
"common": {
"retry": "다시 시도",
"cancel": "취소",

View File

@@ -0,0 +1,36 @@
import * as React from "react";
import { cva, type VariantProps } from "class-variance-authority";
import { cn } from "@/lib/utils";
const badgeVariants = cva(
"inline-flex items-center rounded-md border px-2 py-0.5 text-xs font-semibold transition-colors focus:outline-none focus:ring-2 focus:ring-ring focus:ring-offset-2",
{
variants: {
variant: {
default: "border-transparent bg-primary text-primary-foreground",
secondary:
"border-transparent bg-secondary text-secondary-foreground",
destructive:
"border-transparent bg-destructive text-destructive-foreground",
outline: "text-foreground",
success:
"border-transparent bg-green-100 text-green-800",
warning:
"border-transparent bg-yellow-100 text-yellow-800",
},
},
defaultVariants: { variant: "default" },
},
);
export interface BadgeProps
extends React.HTMLAttributes<HTMLSpanElement>,
VariantProps<typeof badgeVariants> {}
export function Badge({ className, variant, ...props }: BadgeProps) {
return (
<span className={cn(badgeVariants({ variant }), className)} {...props} />
);
}
export { badgeVariants };

View File

@@ -0,0 +1,37 @@
import * as React from "react";
import { cn } from "@/lib/utils";
export interface ProgressProps extends React.HTMLAttributes<HTMLDivElement> {
value?: number;
max?: number;
indeterminate?: boolean;
}
export const Progress = React.forwardRef<HTMLDivElement, ProgressProps>(
({ className, value = 0, max = 100, indeterminate, ...props }, ref) => {
const percent = Math.min(100, Math.max(0, (value / max) * 100));
return (
<div
ref={ref}
role="progressbar"
aria-valuemin={0}
aria-valuemax={max}
aria-valuenow={indeterminate ? undefined : value}
className={cn(
"relative h-2 w-full overflow-hidden rounded-full bg-secondary",
className,
)}
{...props}
>
<div
className={cn(
"h-full bg-primary transition-all",
indeterminate && "w-1/3 animate-pulse",
)}
style={indeterminate ? undefined : { width: `${percent}%` }}
/>
</div>
);
},
);
Progress.displayName = "Progress";

View File

@@ -0,0 +1,22 @@
import * as React from "react";
import { cn } from "@/lib/utils";
export type SelectProps = React.SelectHTMLAttributes<HTMLSelectElement>;
export const Select = React.forwardRef<HTMLSelectElement, SelectProps>(
({ className, children, ...props }, ref) => {
return (
<select
ref={ref}
className={cn(
"flex h-10 w-full rounded-md border border-input bg-background px-3 py-2 text-sm ring-offset-background focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:cursor-not-allowed disabled:opacity-50",
className,
)}
{...props}
>
{children}
</select>
);
},
);
Select.displayName = "Select";

View File

@@ -19,4 +19,9 @@ export const queryKeys = {
byProject: (projectName: string) =>
[...queryKeys.sources.all, "byProject", projectName] as const,
},
crawl: {
all: ["crawl"] as const,
job: (jobId: string) =>
[...queryKeys.crawl.all, "job", jobId] as const,
},
};

View File

@@ -0,0 +1,44 @@
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import {
CrawlJob,
crawlApi,
isCrawlTerminal,
StartSiteCrawlRequest,
} from "@/lib/api/crawl";
import { queryKeys } from "./queryKeys";
const POLL_INTERVAL_MS = 2000;
export function useStartSiteCrawl() {
const queryClient = useQueryClient();
return useMutation({
mutationFn: (body: StartSiteCrawlRequest) => crawlApi.startByProject(body),
onSuccess: (job) => {
queryClient.setQueryData(queryKeys.crawl.job(job.job_id), job);
},
});
}
export function useCrawlJob(jobId: string | null | undefined) {
return useQuery<CrawlJob>({
queryKey: queryKeys.crawl.job(jobId ?? ""),
queryFn: () => crawlApi.getJob(jobId!),
enabled: Boolean(jobId),
refetchInterval: (query) => {
const data = query.state.data as CrawlJob | undefined;
if (!data) return POLL_INTERVAL_MS;
return isCrawlTerminal(data.status) ? false : POLL_INTERVAL_MS;
},
refetchIntervalInBackground: false,
});
}
export function useCancelCrawl() {
const queryClient = useQueryClient();
return useMutation({
mutationFn: (jobId: string) => crawlApi.cancel(jobId),
onSuccess: (job) => {
queryClient.setQueryData(queryKeys.crawl.job(job.job_id), job);
},
});
}

View File

@@ -0,0 +1,82 @@
import { z } from "zod";
import { apiClient } from "./client";
const stringFromAny = z.union([z.string(), z.number()]).transform(String);
export const crawlPageItemSchema = z
.object({
url: z.string().optional(),
status: z.string().optional(),
page_type: z.string().optional(),
title: z.string().nullable().optional(),
error: z.string().nullable().optional(),
})
.passthrough();
export const crawlProgressSchema = z
.object({
seed_url: z.string().optional(),
visited_count: z.number().optional(),
analyzed_count: z.number().optional(),
queued_count: z.number().optional(),
skipped_count: z.number().optional(),
errors: z.array(z.string()).optional(),
pages: z.array(crawlPageItemSchema).optional(),
latest_page: crawlPageItemSchema.optional(),
})
.passthrough();
export const crawlJobSchema = z.object({
job_id: stringFromAny,
status: z.string(),
url: z.string().nullable().optional(),
error: z.string().nullable().optional(),
scheduled_at: z.string().nullable().optional(),
started_at: z.string().nullable().optional(),
finished_at: z.string().nullable().optional(),
progress: crawlProgressSchema.default({}),
request: z.record(z.string(), z.unknown()).default({}),
});
export type CrawlJob = z.infer<typeof crawlJobSchema>;
export type CrawlProgress = z.infer<typeof crawlProgressSchema>;
export interface StartSiteCrawlRequest {
project_name: string;
source_name: string;
url: string;
max_depth?: number;
max_pages?: number;
same_domain_only?: boolean;
analyze_page_types?: string[];
extractor_provider?: string;
extractor_model?: string | null;
extractor_base_url?: string | null;
check_robots_txt?: boolean;
respect_robots_txt?: boolean | null;
}
export const TERMINAL_CRAWL_STATUSES = new Set([
"completed",
"failed",
"canceled",
]);
export function isCrawlTerminal(status: string): boolean {
return TERMINAL_CRAWL_STATUSES.has(status);
}
export const crawlApi = {
startByProject: (body: StartSiteCrawlRequest) =>
apiClient.post("/crawl-site/by-project", crawlJobSchema, body),
getJob: (jobId: string) =>
apiClient.get(
`/crawl-site/jobs/${encodeURIComponent(jobId)}`,
crawlJobSchema,
),
cancel: (jobId: string) =>
apiClient.post(
`/crawl-site/jobs/${encodeURIComponent(jobId)}/cancel`,
crawlJobSchema,
),
};

View File

@@ -3,3 +3,4 @@ export * from "./projects";
export * from "./domains";
export * from "./ontology";
export * from "./sources";
export * from "./crawl";

View File

@@ -1,39 +1,474 @@
import { useState } from "react";
import { useNavigate, useParams } from "react-router-dom";
import { useTranslation } from "react-i18next";
import { useForm } from "react-hook-form";
import { zodResolver } from "@hookform/resolvers/zod";
import { z } from "zod";
import { toast } from "sonner";
import {
AlertCircle,
ArrowLeft,
ArrowRight,
Ban,
CheckCircle2,
Globe,
Loader2,
PlayCircle,
XCircle,
} from "lucide-react";
import {
Card,
CardContent,
CardDescription,
CardHeader,
CardTitle,
} from "@/components/ui/card";
import { Button } from "@/components/ui/button";
import { Input } from "@/components/ui/input";
import { Label } from "@/components/ui/label";
import { Select } from "@/components/ui/select";
import { Progress } from "@/components/ui/progress";
import { Badge, BadgeProps } from "@/components/ui/badge";
import { Skeleton } from "@/components/ui/skeleton";
import { useProject } from "@/hooks/useProjects";
import {
useCancelCrawl,
useCrawlJob,
useStartSiteCrawl,
} from "@/hooks/useCrawl";
import { isCrawlTerminal } from "@/lib/api/crawl";
const startCrawlSchema = z.object({
source_name: z.string().min(1, "소스를 선택하세요"),
url: z.string().url("올바른 URL 형식이 아닙니다"),
max_depth: z.number().int().min(0).max(10),
max_pages: z.number().int().min(1).max(500),
same_domain_only: z.boolean(),
});
type StartCrawlFormValues = z.infer<typeof startCrawlSchema>;
function statusBadgeVariant(status: string): BadgeProps["variant"] {
switch (status) {
case "completed":
return "success";
case "failed":
return "destructive";
case "canceled":
return "secondary";
case "cancel_requested":
return "warning";
case "running":
case "pending":
return "default";
default:
return "outline";
}
}
function StatusBadge({ status }: { status: string }) {
return <Badge variant={statusBadgeVariant(status)}>{status}</Badge>;
}
export default function CrawlPage() {
const navigate = useNavigate();
const { projectId } = useParams();
const { projectId } = useParams<{ projectId: string }>();
const { t } = useTranslation();
const projectName = projectId ?? "";
const { data: project, isLoading, isError, error, refetch } = useProject(projectName);
const [activeJobId, setActiveJobId] = useState<string | null>(null);
const { data: job } = useCrawlJob(activeJobId);
const startCrawl = useStartSiteCrawl();
const cancelCrawl = useCancelCrawl();
const {
register,
handleSubmit,
setValue,
watch,
formState: { errors, isSubmitting },
} = useForm<StartCrawlFormValues>({
resolver: zodResolver(startCrawlSchema),
defaultValues: {
source_name: "",
url: "",
max_depth: 2,
max_pages: 30,
same_domain_only: true,
},
});
const onStart = async (values: StartCrawlFormValues) => {
try {
const created = await startCrawl.mutateAsync({
project_name: projectName,
...values,
});
setActiveJobId(created.job_id);
toast.success(
t("crawl.started", "크롤이 시작되었습니다 (job #{{id}})", {
id: created.job_id,
}),
);
} catch (e) {
toast.error(
t("crawl.startFailed", "시작 실패: {{msg}}", {
msg: (e as Error).message,
}),
);
}
};
const onCancel = async () => {
if (!activeJobId) return;
try {
await cancelCrawl.mutateAsync(activeJobId);
toast.info(t("crawl.cancelRequested", "취소 요청됨"));
} catch (e) {
toast.error(
t("crawl.cancelFailed", "취소 실패: {{msg}}", {
msg: (e as Error).message,
}),
);
}
};
const sources = project?.sources ?? [];
const sourceName = watch("source_name");
const selectedSource = sources.find((s) => s.name === sourceName);
const progress = job?.progress;
const visited = progress?.visited_count ?? 0;
const queued = progress?.queued_count ?? 0;
const analyzed = progress?.analyzed_count ?? 0;
const errors_ = progress?.errors ?? [];
const max_pages = watch("max_pages");
const progressPct = job && max_pages
? Math.min(100, (visited / max_pages) * 100)
: 0;
const terminal = job ? isCrawlTerminal(job.status) : false;
const running = job && !terminal;
return (
<div className="min-h-screen bg-gray-50 p-4">
<div className="max-w-4xl mx-auto">
<h1 className="text-3xl font-bold text-gray-900 mb-2">
{t("Build Ontology")}
</h1>
<p className="text-gray-600 mb-6">
{t("Step 3 of 4: Auto-crawl and build ontology with Phase 5 + 7")}
</p>
<div className="bg-white rounded-lg shadow p-6">
<p className="text-gray-500">{t("Crawl pipeline will appear here")}</p>
<div className="mx-auto max-w-6xl px-6 py-10">
<div className="mb-6 flex items-center gap-3">
<Button
variant="ghost"
size="icon"
onClick={() => navigate(`/sources/${projectName}`)}
aria-label={t("common.back", "이전")}
>
<ArrowLeft className="h-4 w-4" />
</Button>
<div className="flex-1">
<h1 className="text-2xl font-bold tracking-tight">
{t("crawl.title", "시드 크롤")}
</h1>
{project && (
<p className="text-sm text-muted-foreground">
{project.name}
<span className="capitalize text-muted-foreground/70">
{" "}
· {project.domain}
</span>
</p>
)}
</div>
{terminal && job?.status === "completed" && (
<Button onClick={() => navigate(`/review/${projectName}`)}>
{t("crawl.review", "결과 검토")}
<ArrowRight className="h-4 w-4" />
</Button>
)}
</div>
<div className="flex gap-4 mt-6">
<button
onClick={() => navigate(`/sources/${projectId}`)}
className="flex-1 px-4 py-2 border border-gray-300 rounded-lg text-gray-700 hover:bg-gray-50 transition"
>
{t("Back")}
</button>
<button
onClick={() => navigate(`/review/${projectId}`)}
className="flex-1 px-4 py-2 bg-blue-600 text-white rounded-lg hover:bg-blue-700 transition"
>
{t("Review Results")}
</button>
</div>
{isError && (
<Card className="mb-6 border-destructive">
<CardContent className="flex items-center justify-between gap-3 py-4">
<div className="flex items-center gap-2 text-sm text-destructive">
<AlertCircle className="h-4 w-4" />
<span>{(error as Error).message}</span>
</div>
<Button
variant="outline"
size="sm"
type="button"
onClick={() => refetch()}
>
{t("common.retry", "다시 시도")}
</Button>
</CardContent>
</Card>
)}
<div className="grid gap-6 lg:grid-cols-[420px_1fr]">
<Card>
<CardHeader>
<CardTitle>{t("crawl.formTitle", "크롤 설정")}</CardTitle>
<CardDescription>
{t(
"crawl.formDesc",
"시드 URL에서 시작해 링크를 따라가며 정보를 추출합니다",
)}
</CardDescription>
</CardHeader>
<CardContent>
<form
onSubmit={handleSubmit(onStart)}
className="space-y-4"
noValidate
>
<div className="space-y-1.5">
<Label htmlFor="source_name">
{t("crawl.source", "참고 소스")}
</Label>
{isLoading ? (
<Skeleton className="h-10" />
) : (
<Select
id="source_name"
{...register("source_name")}
onChange={(e) =>
setValue("source_name", e.target.value, {
shouldValidate: true,
})
}
>
<option value="">
{t("crawl.pickSource", "소스를 선택하세요...")}
</option>
{sources.map((s) => (
<option key={s.id} value={s.name}>
{s.name} ({s.type})
</option>
))}
</Select>
)}
{sources.length === 0 && !isLoading && (
<p className="text-xs text-muted-foreground">
{t(
"crawl.noSources",
"등록된 소스가 없습니다. 먼저 참고 소스를 추가하세요.",
)}{" "}
<button
type="button"
className="text-primary underline"
onClick={() => navigate(`/sources/${projectName}`)}
>
{t("crawl.addSource", "소스 추가")}
</button>
</p>
)}
{errors.source_name && (
<p className="text-xs text-destructive">
{errors.source_name.message}
</p>
)}
</div>
<div className="space-y-1.5">
<Label htmlFor="url">{t("crawl.seedUrl", "시드 URL")}</Label>
<Input
id="url"
placeholder={
selectedSource?.base_url ?? "https://example.com/start"
}
{...register("url")}
/>
{errors.url && (
<p className="text-xs text-destructive">
{errors.url.message}
</p>
)}
</div>
<div className="grid grid-cols-2 gap-3">
<div className="space-y-1.5">
<Label htmlFor="max_depth">
{t("crawl.maxDepth", "최대 깊이")}
</Label>
<Input
id="max_depth"
type="number"
min={0}
max={10}
{...register("max_depth", { valueAsNumber: true })}
/>
{errors.max_depth && (
<p className="text-xs text-destructive">
{errors.max_depth.message}
</p>
)}
</div>
<div className="space-y-1.5">
<Label htmlFor="max_pages">
{t("crawl.maxPages", "최대 페이지")}
</Label>
<Input
id="max_pages"
type="number"
min={1}
max={500}
{...register("max_pages", { valueAsNumber: true })}
/>
{errors.max_pages && (
<p className="text-xs text-destructive">
{errors.max_pages.message}
</p>
)}
</div>
</div>
<label className="flex items-center gap-2 text-sm">
<input
type="checkbox"
className="h-4 w-4 rounded border-input"
{...register("same_domain_only")}
/>
<span>
{t("crawl.sameDomainOnly", "동일 도메인만 따라가기")}
</span>
</label>
<Button
type="submit"
className="w-full"
disabled={isSubmitting || startCrawl.isPending || Boolean(running)}
>
{startCrawl.isPending ? (
<Loader2 className="h-4 w-4 animate-spin" />
) : (
<PlayCircle className="h-4 w-4" />
)}
{t("crawl.start", "크롤 시작")}
</Button>
</form>
</CardContent>
</Card>
<Card>
<CardHeader>
<div className="flex items-center justify-between">
<div>
<CardTitle>
{t("crawl.progressTitle", "진행 상태")}
</CardTitle>
{job && (
<CardDescription className="flex items-center gap-2">
<span>job #{job.job_id}</span>
<StatusBadge status={job.status} />
</CardDescription>
)}
</div>
{running && (
<Button
variant="outline"
size="sm"
onClick={onCancel}
disabled={cancelCrawl.isPending}
>
<Ban className="h-4 w-4" />
{t("crawl.cancel", "취소")}
</Button>
)}
</div>
</CardHeader>
<CardContent>
{!job && (
<div className="flex flex-col items-center gap-3 py-12 text-center text-muted-foreground">
<Globe className="h-12 w-12 opacity-40" />
<p>
{t(
"crawl.idleHint",
"왼쪽에서 시드 URL을 입력하고 시작하세요",
)}
</p>
</div>
)}
{job && (
<div className="space-y-4">
<div>
<div className="mb-1 flex items-center justify-between text-sm">
<span className="text-muted-foreground">
{t("crawl.visited", "방문")} {visited}
{" / "}
{max_pages}
</span>
<span className="text-muted-foreground">
{t("crawl.queued", "대기")} {queued} ·{" "}
{t("crawl.analyzed", "분석")} {analyzed}
</span>
</div>
<Progress value={progressPct} indeterminate={running && visited === 0} />
</div>
{job.url && (
<div className="rounded-md bg-secondary/30 px-3 py-2 text-xs">
<div className="text-muted-foreground">
{t("crawl.seedUrl", "시드 URL")}
</div>
<a
href={job.url}
target="_blank"
rel="noopener noreferrer"
className="break-all text-foreground hover:underline"
>
{job.url}
</a>
</div>
)}
{progress?.latest_page && (
<div className="rounded-md border bg-background px-3 py-2 text-xs">
<div className="mb-1 text-muted-foreground">
{t("crawl.latestPage", "최근 페이지")}
</div>
<div className="truncate font-medium">
{progress.latest_page.title || progress.latest_page.url}
</div>
{progress.latest_page.page_type && (
<Badge variant="outline" className="mt-1">
{progress.latest_page.page_type}
</Badge>
)}
</div>
)}
{job.error && (
<div className="flex items-start gap-2 rounded-md border border-destructive bg-destructive/5 px-3 py-2 text-sm text-destructive">
<XCircle className="mt-0.5 h-4 w-4 flex-shrink-0" />
<span>{job.error}</span>
</div>
)}
{errors_.length > 0 && (
<details className="rounded-md border bg-background">
<summary className="cursor-pointer px-3 py-2 text-sm font-medium">
{t("crawl.errorsCount", "에러 {{count}}건", {
count: errors_.length,
})}
</summary>
<ul className="max-h-40 overflow-y-auto px-4 py-2 text-xs text-muted-foreground">
{errors_.map((e, i) => (
<li key={i} className="border-b py-1 last:border-0">
{e}
</li>
))}
</ul>
</details>
)}
{terminal && job.status === "completed" && (
<div className="flex items-center gap-2 text-sm text-green-700">
<CheckCircle2 className="h-4 w-4" />
{t("crawl.doneHint", "크롤 완료. 결과 검토로 이동하세요.")}
</div>
)}
</div>
)}
</CardContent>
</Card>
</div>
</div>
);