참고소스 수정본

참고/firecrawl-main/.github/CODEOWNERS

@@ -0,0 +1,120 @@
+# Firecrawl Ownership Chart
+
+# api
+
+## V2 API
+/apps/api/src/routes/v2.ts                      @mogery @nickscamara
+
+### /scrape
+/apps/api/src/controllers/v2/scrape*            @mogery
+
+### /crawl
+/apps/api/src/controllers/v2/crawl*             @mogery
+
+### /batch/scrape
+/apps/api/src/controllers/v2/batch-scrape*      @mogery 
+
+### /extract
+/apps/api/src/controllers/v2/extract*           @nickscamara
+
+### /map
+/apps/api/src/controllers/v2/map*               @nickscamara @mogery
+
+### /search
+/apps/api/src/controllers/v2/search*            @nickscamara
+
+### Input Validation/Zod
+/apps/api/src/controllers/v2/types.ts           @mogery
+
+## V1 API
+/apps/api/src/routes/v1.ts                      @mogery @nickscamara
+
+### /scrape
+/apps/api/src/controllers/v1/scrape*            @mogery
+
+### /crawl
+/apps/api/src/controllers/v1/crawl*             @mogery
+
+### /batch/scrape
+/apps/api/src/controllers/v1/batch-scrape*      @mogery 
+
+### /extract
+/apps/api/src/controllers/v1/extract*           @nickscamara
+/apps/api/src/lib/extract/*                     @nickscamara
+/apps/api/src/lib/generic-ai.ts                 @mogery @nickscamara # (AI SDK)
+
+### /map
+/apps/api/src/controllers/v1/map*               @nickscamara
+/apps/api/src/lib/map-cosine.ts                 @nickscamara
+
+### /search
+/apps/api/src/controllers/v1/search*            @nickscamara
+/apps/api/src/search/*                          @nickscamara
+
+### /llmstxt
+/apps/api/src/controllers/v1/generate-llmstxt*  @nickscamara
+/apps/api/src/lib/generate-llmstxt/*            @nickscamara
+
+### /deep-research
+/apps/api/src/controllers/v1/deep-research*     @nickscamara
+/apps/api/src/lib/deep-research/*               @nickscamara
+
+### Input Validation/Zod
+/apps/api/src/controllers/v1/types.ts           @mogery
+
+## V0 API, deprecated
+/apps/api/src/controllers/v0/*                  @mogery @nickscamara
+/apps/api/src/routes/v0.ts                      @mogery @nickscamara
+
+# Worker
+
+## scrapeURL
+/apps/api/src/scraper/scrapeURL/*               @mogery
+
+### crawler
+/apps/api/src/lib/crawl-redis*                  @mogery
+
+### remnants of WebScraper/WebCrawler
+/apps/api/src/scraper/WebScraper/*              @mogery @nickscamara
+
+## concurrency limits
+/apps/api/src/lib/concurrency-limit.ts          @mogery @nickscamara
+
+## BullMQ-related code
+/apps/api/src/services/queue-worker.ts          @mogery @nickscamara
+/apps/api/src/services/worker/scrape-worker.ts  @mogery
+/apps/api/src/main/runWebScraper.ts             @mogery @nickscamara
+/apps/api/src/services/queue*                   @mogery @nickscamara
+/apps/api/src/lib/job-priority.ts               @nickscamara @mogery
+
+# Shared Libraries
+/apps/api/sharedLibs/go-html-to-md/*            @tomkosm
+/apps/api/src/lib/html-to-markdown.ts           @tomkosm
+/apps/api/sharedLibs/html-transformer/*         @mogery
+/apps/api/src/lib/html-transformer.ts           @mogery
+
+# playwright-serice-ts
+/apps/playwright-service-ts/*                   @mogery
+
+# self-hosting
+/docker-compose.yaml                            @mogery
+/SELF_HOST.md                                   @mogery
+
+# SDKs
+/apps/python-sdk/*                              @rafaelsideguide @nickscamara
+/apps/js-sdk/*                                  @mogery @nickscamara
+/apps/rust-sdk/*                                @mogery
+/apps/go-sdk/*                                  @rafaelsideguide
+
+# CI/CD and GitHub Workflows
+/.github/*                                      @mogery @rafaelsideguide
+
+# Tests
+/apps/api/src/__tests__/snips/*                 @mogery
+
+# Branding
+**/branding*                                    @abimaelmartell
+**/branding-script*                             @abimaelmartell
+
+# Examples
+/examples/*                                     @ericciarla @nickscamara

참고/firecrawl-main/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,36 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: "[Bug] "
+labels: bug
+assignees: ''
+
+---
+
+**Describe the Bug**
+Provide a clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the issue:
+1. Configure the environment or settings with '...'
+2. Run the command '...'
+3. Observe the error or unexpected output at '...'
+4. Log output/error message
+
+**Expected Behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots or copies of the command line output to help explain the issue.
+
+**Environment (please complete the following information):**
+- OS: [e.g. macOS, Linux, Windows]
+- Deployment Type: [Cloud (firecrawl.dev) / Self-hosted]
+- Firecrawl Version: [e.g. 1.2.3]
+- Node.js Version: [e.g. 14.x]
+
+**Logs**
+If applicable, include detailed logs to help understand the problem.
+
+**Additional Context**
+Add any other context about the problem here, such as configuration specifics, network conditions, data volumes, etc.

참고/firecrawl-main/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,26 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: "[Feat] "
+labels: ''
+assignees: ''
+
+---
+
+**Problem Description**
+Describe the issue you're experiencing that has prompted this feature request. For example, "I find it difficult when..."
+
+**Proposed Feature**
+Provide a clear and concise description of the feature you would like implemented.
+
+**Alternatives Considered**
+Discuss any alternative solutions or features you've considered. Why were these alternatives not suitable?
+
+**Implementation Suggestions**
+If you have ideas on how the feature could be implemented, share them here. This could include technical details, API changes, or interaction mechanisms.
+
+**Use Case**
+Explain how this feature would be used and what benefits it would bring. Include specific examples to illustrate how this would improve functionality or user experience.
+
+**Additional Context**
+Add any other context such as comparisons with similar features in other products, or links to prototypes or mockups.

참고/firecrawl-main/.github/ISSUE_TEMPLATE/self_host_issue.md

@@ -0,0 +1,40 @@
+---
+name: Self-host issue
+about: Report an issue with self-hosting Firecrawl
+title: "[Self-Host] "
+labels: self-host
+assignees: ''
+
+---
+
+**Describe the Issue**
+Provide a clear and concise description of the self-hosting issue you're experiencing.
+
+**To Reproduce**
+Steps to reproduce the issue:
+1. Configure the environment or settings with '...'
+2. Run the command '...'
+3. Observe the error or unexpected output at '...'
+4. Log output/error message
+
+**Expected Behavior**
+A clear and concise description of what you expected to happen when self-hosting.
+
+**Screenshots**
+If applicable, add screenshots or copies of the command line output to help explain the self-hosting issue.
+
+**Environment (please complete the following information):**
+- OS: [e.g. macOS, Linux, Windows]
+- Firecrawl Version: [e.g. 1.2.3]
+- Node.js Version: [e.g. 14.x]
+- Docker Version (if applicable): [e.g. 20.10.14]
+- Database Type and Version: [e.g. PostgreSQL 13.4]
+
+**Logs**
+If applicable, include detailed logs to help understand the self-hosting problem.
+
+**Configuration**
+Provide relevant parts of your configuration files (with sensitive information redacted).
+
+**Additional Context**
+Add any other context about the self-hosting issue here, such as specific infrastructure details, network setup, or any modifications made to the original Firecrawl setup.

참고/firecrawl-main/.github/archive/js-sdk.yml

@@ -0,0 +1,53 @@
+name: Run JavaScript SDK E2E Tests
+  
+on: []
+
+env:
+  ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+  BULL_AUTH_KEY: ${{ secrets.BULL_AUTH_KEY }}
+  FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
+  HOST: ${{ secrets.HOST }}
+  LLAMAPARSE_API_KEY: ${{ secrets.LLAMAPARSE_API_KEY }}
+  NUM_WORKERS_PER_QUEUE: ${{ secrets.NUM_WORKERS_PER_QUEUE }}
+  OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+  PLAYWRIGHT_MICROSERVICE_URL: ${{ secrets.PLAYWRIGHT_MICROSERVICE_URL }}
+  PORT: ${{ secrets.PORT }}
+  REDIS_URL: ${{ secrets.REDIS_URL }}
+  SUPABASE_ANON_TOKEN: ${{ secrets.SUPABASE_ANON_TOKEN }}
+  SUPABASE_SERVICE_TOKEN: ${{ secrets.SUPABASE_SERVICE_TOKEN }}
+  SUPABASE_URL: ${{ secrets.SUPABASE_URL }}
+  TEST_API_KEY: ${{ secrets.TEST_API_KEY }}
+  HDX_NODE_BETA_MODE: 1
+
+jobs:  
+  build:  
+    runs-on: blacksmith-4vcpu-ubuntu-2404  
+    services:
+      redis:
+        image: redis
+        ports:
+          - 6379:6379
+  
+    steps:
+      - uses: actions/checkout@v3
+      - name: Set up Node.js
+        uses: actions/setup-node@v3
+        with:
+          node-version: "20"
+      - name: Install pnpm
+        run: npm install -g pnpm
+      - name: Install dependencies for API
+        run: pnpm install
+        working-directory: ./apps/api
+      - name: Start the application
+        run: npm start &
+        working-directory: ./apps/api
+      - name: Start workers
+        run: npm run workers &
+        working-directory: ./apps/api
+      - name: Install dependencies for JavaScript SDK
+        run: pnpm install
+        working-directory: ./apps/js-sdk/firecrawl
+      - name: Run E2E tests for JavaScript SDK
+        run: npm run test
+        working-directory: ./apps/js-sdk/firecrawl

참고/firecrawl-main/.github/archive/publish-rust-sdk.yml

@@ -0,0 +1,42 @@
+name: Publish Rust SDK 
+
+on: []
+
+env:
+  CRATES_IO_TOKEN: ${{ secrets.CRATES_IO_TOKEN }}
+
+jobs:
+  build-and-publish:
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v3
+
+      - name: Set up Rust
+        uses: actions-rs/toolchain@v1
+        with:
+          toolchain: stable
+          default: true
+          profile: minimal
+
+      - name: Install dependencies
+        run: cargo build --release
+
+      - name: Run version check script
+        id: version_check_script
+        run: |
+          VERSION_INCREMENTED=$(cargo search --limit 1 my_crate_name | grep my_crate_name)
+          echo "VERSION_INCREMENTED=$VERSION_INCREMENTED" >> $GITHUB_ENV
+
+      - name: Build the package
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        run: cargo package
+        working-directory: ./apps/rust-sdk
+
+      - name: Publish to crates.io
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        env:
+          CARGO_REG_TOKEN: ${{ secrets.CRATES_IO_TOKEN }}
+        run: cargo publish
+        working-directory: ./apps/rust-sdk

참고/firecrawl-main/.github/archive/python-sdk.yml

@@ -0,0 +1,65 @@
+name: Run Python SDK E2E Tests
+  
+on: []
+
+env:
+  ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+  BULL_AUTH_KEY: ${{ secrets.BULL_AUTH_KEY }}
+  FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
+  HOST: ${{ secrets.HOST }}
+  LLAMAPARSE_API_KEY: ${{ secrets.LLAMAPARSE_API_KEY }}
+  NUM_WORKERS_PER_QUEUE: ${{ secrets.NUM_WORKERS_PER_QUEUE }}
+  OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+  PLAYWRIGHT_MICROSERVICE_URL: ${{ secrets.PLAYWRIGHT_MICROSERVICE_URL }}
+  PORT: ${{ secrets.PORT }}
+  REDIS_URL: ${{ secrets.REDIS_URL }}
+  SUPABASE_ANON_TOKEN: ${{ secrets.SUPABASE_ANON_TOKEN }}
+  SUPABASE_SERVICE_TOKEN: ${{ secrets.SUPABASE_SERVICE_TOKEN }}
+  SUPABASE_URL: ${{ secrets.SUPABASE_URL }}
+  TEST_API_KEY: ${{ secrets.TEST_API_KEY }}
+  HDX_NODE_BETA_MODE: 1
+
+jobs:  
+  build:  
+    runs-on: blacksmith-4vcpu-ubuntu-2404  
+    strategy:  
+      matrix:  
+        python-version: ["3.10"]
+    services:
+      redis:
+        image: redis
+        ports:
+          - 6379:6379
+  
+    steps:
+      - uses: actions/checkout@v3
+      - name: Set up Node.js
+        uses: actions/setup-node@v3
+        with:
+          node-version: "20"
+      - name: Install pnpm
+        run: npm install -g pnpm
+      - name: Install dependencies for API
+        run: pnpm install
+        working-directory: ./apps/api
+      - name: Start the application
+        run: npm start &
+        working-directory: ./apps/api
+        id: start_app
+      - name: Start workers
+        run: npm run workers &
+        working-directory: ./apps/api
+        id: start_workers
+      - name: Set up Python ${{ matrix.python-version }}  
+        uses: actions/setup-python@v4  
+        with:  
+          python-version: ${{ matrix.python-version }}
+      - name: Install Python dependencies  
+        run: |  
+          python -m pip install --upgrade pip  
+          pip install -r requirements.txt
+        working-directory: ./apps/python-sdk
+      - name: Run E2E tests for Python SDK  
+        run: |  
+          python -m pytest firecrawl/__tests__/e2e/v2
+        working-directory: ./apps/python-sdk

참고/firecrawl-main/.github/archive/rust-sdk.yml

@@ -0,0 +1,56 @@
+name: Run Rust SDK E2E Tests
+
+on: []
+
+env:
+  ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+  BULL_AUTH_KEY: ${{ secrets.BULL_AUTH_KEY }}
+  FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
+  HOST: ${{ secrets.HOST }}
+  LLAMAPARSE_API_KEY: ${{ secrets.LLAMAPARSE_API_KEY }}
+  NUM_WORKERS_PER_QUEUE: ${{ secrets.NUM_WORKERS_PER_QUEUE }}
+  OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+  PLAYWRIGHT_MICROSERVICE_URL: ${{ secrets.PLAYWRIGHT_MICROSERVICE_URL }}
+  PORT: ${{ secrets.PORT }}
+  REDIS_URL: ${{ secrets.REDIS_URL }}
+  SUPABASE_ANON_TOKEN: ${{ secrets.SUPABASE_ANON_TOKEN }}
+  SUPABASE_SERVICE_TOKEN: ${{ secrets.SUPABASE_SERVICE_TOKEN }}
+  SUPABASE_URL: ${{ secrets.SUPABASE_URL }}
+  TEST_API_KEY: ${{ secrets.TEST_API_KEY }}
+  HDX_NODE_BETA_MODE: 1
+
+
+jobs:
+  build:
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+    services:
+      redis:
+        image: redis
+        ports:
+          - 6379:6379
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v3
+      - name: Install pnpm
+        run: npm install -g pnpm
+      - name: Install dependencies for API
+        run: pnpm install
+        working-directory: ./apps/api
+      - name: Start the application
+        run: npm start &
+        working-directory: ./apps/api
+        id: start_app
+      - name: Start workers
+        run: npm run workers &
+        working-directory: ./apps/api
+        id: start_workers
+      - name: Set up Rust
+        uses: actions/setup-rust@v1
+        with:
+          rust-version: stable
+      - name: Try the lib build
+        working-directory: ./apps/rust-sdk
+        run: cargo build
+      - name: Run E2E tests for Rust SDK
+        run: cargo test --test e2e_with_auth

참고/firecrawl-main/.github/dependabot.yml

@@ -0,0 +1,47 @@
+version: 2
+updates:
+  # playwright-service
+  - package-ecosystem: "pip"
+    directory: "/apps/playwright-service"
+    schedule:
+      interval: "weekly"
+    open-pull-requests-limit: 0  # Disable version updates
+    commit-message:
+      prefix: "apps/playwright-service"
+      include: "scope"
+
+  # python-sdk
+  - package-ecosystem: "pip"
+    directory: "/apps/python-sdk"
+    schedule:
+      interval: "weekly"
+    open-pull-requests-limit: 0  # Disable version updates
+    commit-message:
+      prefix: "apps/python-sdk"
+      include: "scope"
+
+  # api
+  - package-ecosystem: "npm"
+    directory: "/apps/api"
+    schedule:
+      interval: "weekly"
+    open-pull-requests-limit: 0  # Disable version updates
+    commit-message:
+      prefix: "apps/api"
+      include: "scope"
+
+  # test-suite
+  - package-ecosystem: "npm"
+    directory: "/apps/test-suite"
+    schedule:
+      interval: "weekly"
+    open-pull-requests-limit: 0  # Disable version updates
+    commit-message:
+      prefix: "apps/test-suite"
+      include: "scope"
+
+  # GitHub Actions
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "weekly"

참고/firecrawl-main/.github/scripts/audit-ci-vuln-scan.mjs

@@ -0,0 +1,597 @@
+#!/usr/bin/env node
+
+import { spawnSync } from "node:child_process";
+import { appendFileSync, mkdirSync, writeFileSync } from "node:fs";
+import path from "node:path";
+import process from "node:process";
+
+const AUDITS = [
+  { name: "API", appPath: "apps/api", outputName: "api" },
+  {
+    name: "Playwright Service",
+    appPath: "apps/playwright-service-ts",
+    outputName: "playwright-service",
+  },
+  { name: "JavaScript SDK", appPath: "apps/js-sdk", outputName: "js-sdk" },
+  {
+    name: "JavaScript SDK Firecrawl",
+    appPath: "apps/js-sdk/firecrawl",
+    outputName: "js-sdk-firecrawl",
+  },
+  {
+    name: "Test Suite",
+    appPath: "apps/test-suite",
+    outputName: "test-suite",
+  },
+  {
+    name: "Ingestion UI",
+    appPath: "apps/ui/ingestion-ui",
+    outputName: "ingestion-ui",
+  },
+  { name: "Test Site", appPath: "apps/test-site", outputName: "test-site" },
+];
+
+const GHSA_REGEX = /GHSA-[0-9a-z]{4}-[0-9a-z]{4}-[0-9a-z]{4}/gi;
+const MARKER_REGEX = /<!--\s*audit-ci-vuln-keys:\s*(\[[\s\S]*?\])\s*-->/g;
+const OUTPUT_DIR = process.env.AUDIT_REMEDIATION_OUTPUT_DIR || "/tmp/audit-remediation";
+const GITHUB_API_URL = process.env.GITHUB_API_URL || "https://api.github.com";
+const GITHUB_TOKEN = process.env.GH_TOKEN || process.env.GITHUB_TOKEN || "";
+const GITHUB_REPOSITORY = process.env.GITHUB_REPOSITORY || "";
+
+function run(command, args) {
+  return spawnSync(command, args, {
+    encoding: "utf8",
+    maxBuffer: 1024 * 1024 * 50,
+  });
+}
+
+function auditCommand(appPath, extraArgs = []) {
+  return [
+    "dlx",
+    "audit-ci@^7",
+    "--directory",
+    appPath,
+    "--config",
+    `${appPath}/audit-ci.jsonc`,
+    ...extraArgs,
+  ];
+}
+
+function commandForDisplay(appPath) {
+  return `pnpm dlx audit-ci@^7 --directory ${appPath} --config ${appPath}/audit-ci.jsonc`;
+}
+
+function combinedOutput(result) {
+  return `${result.stdout || ""}${result.stderr || ""}`;
+}
+
+function unique(values) {
+  return [...new Set(values.filter(Boolean))];
+}
+
+function advisoryIdsFrom(value) {
+  if (value === undefined || value === null) {
+    return [];
+  }
+
+  return unique(String(value).match(GHSA_REGEX) || []).map((id) => id.toUpperCase());
+}
+
+function advisoryIdsFromObject(value) {
+  return advisoryIdsFrom(JSON.stringify(value));
+}
+
+function normalizeKey(appPath, advisoryId, packageName) {
+  return `${appPath}|${advisoryId.toUpperCase()}|${packageName || "unknown"}`;
+}
+
+function pushFinding(findings, finding) {
+  if (!finding.advisoryId) {
+    return;
+  }
+
+  const packageName = finding.packageName || "unknown";
+  const key = normalizeKey(finding.appPath, finding.advisoryId, packageName);
+  const existing = findings.find((item) => item.key === key);
+
+  if (existing) {
+    existing.paths = unique([...existing.paths, ...(finding.paths || [])]);
+    existing.severities = unique([...existing.severities, ...(finding.severities || [])]);
+    existing.urls = unique([...existing.urls, ...(finding.urls || [])]);
+    existing.titles = unique([...existing.titles, ...(finding.titles || [])]);
+    return;
+  }
+
+  findings.push({
+    key,
+    appName: finding.appName,
+    appPath: finding.appPath,
+    advisoryId: finding.advisoryId.toUpperCase(),
+    packageName,
+    paths: unique(finding.paths || []),
+    severities: unique(finding.severities || []),
+    urls: unique(finding.urls || []),
+    titles: unique(finding.titles || []),
+  });
+}
+
+function extractJson(output) {
+  const trimmed = output.trim();
+
+  for (const [startToken, endToken] of [
+    ["{", "}"],
+    ["[", "]"],
+  ]) {
+    const start = trimmed.indexOf(startToken);
+    const end = trimmed.lastIndexOf(endToken);
+    if (start !== -1 && end > start) {
+      try {
+        return JSON.parse(trimmed.slice(start, end + 1));
+      } catch {
+        // Try the next shape before falling back to text parsing.
+      }
+    }
+  }
+
+  return null;
+}
+
+function collectFromVulnerabilities(report, audit) {
+  const findings = [];
+  const vulnerabilities = report?.vulnerabilities;
+
+  if (!vulnerabilities || typeof vulnerabilities !== "object") {
+    return findings;
+  }
+
+  for (const [packageName, vulnerability] of Object.entries(vulnerabilities)) {
+    const via = Array.isArray(vulnerability?.via) ? vulnerability.via : [];
+    const paths = unique([
+      ...(Array.isArray(vulnerability?.nodes) ? vulnerability.nodes : []),
+      ...(Array.isArray(vulnerability?.effects) ? vulnerability.effects : []),
+    ]);
+
+    for (const viaEntry of via) {
+      if (typeof viaEntry === "string") {
+        continue;
+      }
+
+      const advisoryIds = advisoryIdsFromObject(viaEntry);
+      for (const advisoryId of advisoryIds) {
+        pushFinding(findings, {
+          appName: audit.name,
+          appPath: audit.appPath,
+          advisoryId,
+          packageName: viaEntry.name || packageName,
+          paths,
+          severities: [viaEntry.severity || vulnerability.severity],
+          urls: [viaEntry.url],
+          titles: [viaEntry.title],
+        });
+      }
+    }
+  }
+
+  return findings;
+}
+
+function collectFromAdvisories(report, audit) {
+  const findings = [];
+  const advisories = report?.advisories;
+
+  if (!advisories || typeof advisories !== "object") {
+    return findings;
+  }
+
+  for (const advisory of Object.values(advisories)) {
+    const advisoryIds = unique([
+      ...advisoryIdsFrom(advisory.github_advisory_id),
+      ...advisoryIdsFrom(advisory.url),
+      ...advisoryIdsFrom(advisory.title),
+      ...advisoryIdsFromObject(advisory),
+    ]);
+    const paths = [];
+
+    if (Array.isArray(advisory.findings)) {
+      for (const finding of advisory.findings) {
+        paths.push(...(Array.isArray(finding.paths) ? finding.paths : []));
+      }
+    }
+
+    for (const advisoryId of advisoryIds) {
+      pushFinding(findings, {
+        appName: audit.name,
+        appPath: audit.appPath,
+        advisoryId,
+        packageName: advisory.module_name || advisory.name || "unknown",
+        paths,
+        severities: [advisory.severity],
+        urls: [advisory.url],
+        titles: [advisory.title],
+      });
+    }
+  }
+
+  return findings;
+}
+
+function collectFallbackFromText(output, audit) {
+  const findings = [];
+  const advisoryIds = advisoryIdsFrom(output);
+
+  for (const advisoryId of advisoryIds) {
+    pushFinding(findings, {
+      appName: audit.name,
+      appPath: audit.appPath,
+      advisoryId,
+      packageName: "unknown",
+      paths: [],
+      severities: [],
+      urls: [],
+      titles: [],
+    });
+  }
+
+  return findings;
+}
+
+function collectFindings(report, output, audit) {
+  const structuredFindings = [
+    ...collectFromVulnerabilities(report, audit),
+    ...collectFromAdvisories(report, audit),
+  ];
+  const structuredAdvisoryIds = new Set(
+    structuredFindings.map((finding) => `${finding.appPath}|${finding.advisoryId}`),
+  );
+  const fallbackFindings = collectFallbackFromText(output, audit).filter(
+    (finding) => !structuredAdvisoryIds.has(`${finding.appPath}|${finding.advisoryId}`),
+  );
+
+  return [...structuredFindings, ...fallbackFindings];
+}
+
+async function fetchJson(url) {
+  const response = await fetch(url, {
+    headers: {
+      Accept: "application/vnd.github+json",
+      Authorization: `Bearer ${GITHUB_TOKEN}`,
+      "X-GitHub-Api-Version": "2022-11-28",
+      "User-Agent": "audit-ci-vuln-scan",
+    },
+  });
+
+  if (!response.ok) {
+    throw new Error(`GitHub API request failed: ${response.status} ${response.statusText}`);
+  }
+
+  return response.json();
+}
+
+async function listOpenPullRequests() {
+  if (!GITHUB_TOKEN || !GITHUB_REPOSITORY) {
+    return [];
+  }
+
+  const pulls = [];
+  for (let page = 1; ; page += 1) {
+    const url = `${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/pulls?state=open&per_page=100&page=${page}`;
+    const batch = await fetchJson(url);
+    pulls.push(...batch);
+
+    if (batch.length < 100) {
+      break;
+    }
+  }
+
+  return pulls;
+}
+
+function extractCoveredKeys(pulls) {
+  const covered = new Map();
+
+  for (const pull of pulls) {
+    const body = pull.body || "";
+    const markers = body.matchAll(MARKER_REGEX);
+
+    for (const marker of markers) {
+      try {
+        const keys = JSON.parse(marker[1]);
+        if (!Array.isArray(keys)) {
+          continue;
+        }
+
+        for (const key of keys) {
+          if (typeof key !== "string") {
+            continue;
+          }
+
+          const existing = covered.get(key) || [];
+          existing.push({
+            number: pull.number,
+            title: pull.title,
+            url: pull.html_url,
+          });
+          covered.set(key, existing);
+        }
+      } catch {
+        // Ignore malformed markers rather than treating them as coverage.
+      }
+    }
+  }
+
+  return covered;
+}
+
+function buildMarker(keys) {
+  return `<!-- audit-ci-vuln-keys: ${JSON.stringify(keys)} -->`;
+}
+
+function writeGithubOutput(name, value) {
+  if (!process.env.GITHUB_OUTPUT) {
+    return;
+  }
+
+  if (String(value).includes("\n")) {
+    const delimiter = `EOF_${name}_${Date.now()}_${Math.random().toString(16).slice(2)}`;
+    appendFileSync(process.env.GITHUB_OUTPUT, `${name}<<${delimiter}\n${value}\n${delimiter}\n`);
+  } else {
+    appendFileSync(process.env.GITHUB_OUTPUT, `${name}=${value}\n`);
+  }
+}
+
+function buildPrompt({ uncoveredFindings, coveredFindings, marker, commands }) {
+  const uncoveredJson = JSON.stringify(uncoveredFindings, null, 2);
+  const coveredJson = JSON.stringify(
+    coveredFindings.map((finding) => ({
+      key: finding.key,
+      appPath: finding.appPath,
+      advisoryId: finding.advisoryId,
+      packageName: finding.packageName,
+      coveredBy: finding.coveredBy,
+    })),
+    null,
+    2,
+  );
+
+  return `PNPM Audit Failures Fix
+
+You are fixing CI security audit failures in this monorepo and must mirror this repo's workflow exactly.
+
+Source of truth:
+- Workflow file: \`.github/workflows/npm-audit.yml\`
+- Job: \`audit\`
+- Failure reporter step: \`Report audit failures\`
+- Reproduce using the same \`audit-ci\` commands/flags used in that workflow.
+
+Current uncovered audit failures on main:
+- Patch only the vulnerability keys listed in this section.
+- Do not spend effort on vulnerabilities listed as already covered by open PRs.
+- Add this hidden marker inside the PR body's \`## Summary\` section:
+  ${marker}
+
+Uncovered vulnerability records:
+
+\`\`\`json
+${uncoveredJson}
+\`\`\`
+
+Already covered by open PR markers:
+
+\`\`\`json
+${coveredJson}
+\`\`\`
+
+CI-equivalent audit commands from \`.github/workflows/npm-audit.yml\`:
+${commands.map((command) => `- \`${command}\``).join("\n")}
+
+Decision policy (strict order):
+1) Upgrade higher-level direct dependencies first (non-breaking only: patch/minor).
+   - Goal: eliminate vulnerable transitive \`vite\` via parent upgrades (e.g. \`astro\` or app-level deps).
+   - No major upgrades unless explicitly approved.
+2) If still failing, add minimal targeted \`pnpm overrides\` / \`resolutions\` for vulnerable transitives.
+3) If neither upgrades nor overrides can reach a non-vulnerable version:
+   - Mark as \`BLOCKED\`
+   - Document exact blocking dependency chain and required follow-up (likely major upgrade or upstream fix).
+4) If advisory is not practically exploitable for this repo:
+   - Provide evidence-based impact assessment (reachability, prod vs dev/build-only, exploit preconditions),
+   - Then propose a temporary ignore with: reason, expiry date, owner, and tracking issue.
+
+Override safety rules (important):
+- Never use unbounded replacement ranges that can cross into a new major.
+- For major-constrained selector overrides, the replacement must preserve the same major ceiling.
+- Example of BAD override:
+  - \`\"vite@>=6.0.0 <7.0.0\": \">=6.4.2\"\`  (can resolve to Vite 7/8)
+- Example of GOOD override:
+  - \`\"vite@>=6.0.0 <7.0.0\": \">=6.4.2 <7.0.0\"\`
+- If deterministic pin is preferred, use exact patched version:
+  - \`\"vite@>=6.0.0 <7.0.0\": \"6.4.2\"\`
+- In the final explanation, explicitly state why the chosen override cannot drift to a higher major.
+
+Mandatory local verification (must match workflow commands):
+- Run exactly these commands locally (same tool/flags/targets as CI):
+  - \`pnpm dlx audit-ci@^7 --directory apps/ui/ingestion-ui --config apps/ui/ingestion-ui/audit-ci.jsonc\`
+  - \`pnpm dlx audit-ci@^7 --directory apps/test-site --config apps/test-site/audit-ci.jsonc\`
+- If broader validation is needed, also run the other audit commands defined in \`.github/workflows/npm-audit.yml\`.
+- Do not claim success unless these CI-equivalent local commands pass (or a documented temporary ignore/blocked path is approved).
+
+PR requirements:
+- Create a PR when done.
+- PR body must contain only a **Summary** section (no **Test plan** section).
+- Do not include any co-author lines/footers.
+- Summary must be organized by app in this monorepo.
+- Include the hidden \`audit-ci-vuln-keys\` marker exactly once inside the **Summary** section.
+- For each app, list each package update and explicitly map it to the advisory/advisories it addresses.
+- For each resolved package version used to fix the advisory, include its release date in the PR summary.
+- If an override is used, include selector and replacement, and note the major-bound guarantee.
+- If you update an SDK package, bump that SDK's package version as part of the same PR; otherwise the publish workflow will not publish the SDK changes.
+
+Output format (exact):
+## Findings
+- Advisory: <GHSA>
+- Severity: <...>
+- Affected path(s): <...>
+- Fixed by: <direct dep upgrade | override | blocked | temp ignore>
+- Rationale: <short>
+
+## Changes Made
+- <file>: <change>
+- <file>: <change>
+
+## Local Verification (CI-equivalent)
+- Workflow reference: \`.github/workflows/npm-audit.yml\`
+- Commands run locally (exact):
+  - \`<command>\`
+- Results:
+  - \`<pass/fail + key output>\`
+
+## Decision Log
+- Step 1 (direct upgrades): <success/fail + what tried>
+- Step 2 (overrides): <success/fail + what tried>
+- Step 3 (blocked?): <yes/no + reason>
+- Step 4 (temp ignore needed?): <yes/no + evidence>
+
+## PR Summary Draft (by app)
+- \`<app path/name>\`
+  - \`<package update>\` -> \`<GHSA(s)>\` (resolved version: \`<x.y.z>\`, release date: \`<YYYY-MM-DD>\`)
+  - \`<override selector> => <replacement>\` -> \`<GHSA(s)>\` (major-bound: \`<explain bound>\`, resolved version: \`<x.y.z>\`, release date: \`<YYYY-MM-DD>\`)
+
+## Risk / Follow-up
+- Runtime impact: <affected/not affected/uncertain>
+- If ignored: expires <date>, tracked in <ticket>, owner <team/person>
+- Next recommended action: <one line>
+`;
+}
+
+function writeStepSummary({ findings, coveredFindings, uncoveredFindings, outputDir }) {
+  if (!process.env.GITHUB_STEP_SUMMARY) {
+    return;
+  }
+
+  const lines = [
+    "## Audit CI Vulnerability Scan",
+    "",
+    `- Current vulnerabilities on default branch: ${findings.length}`,
+    `- Covered by open PR markers: ${coveredFindings.length}`,
+    `- Uncovered vulnerabilities for Claude: ${uncoveredFindings.length}`,
+    `- Output directory: \`${outputDir}\``,
+    "",
+  ];
+
+  if (uncoveredFindings.length > 0) {
+    lines.push("### Uncovered", "");
+    for (const finding of uncoveredFindings) {
+      lines.push(`- \`${finding.key}\``);
+    }
+  }
+
+  appendFileSync(process.env.GITHUB_STEP_SUMMARY, `${lines.join("\n")}\n`);
+}
+
+async function main() {
+  mkdirSync(OUTPUT_DIR, { recursive: true });
+
+  const findings = [];
+  const auditResults = [];
+
+  for (const audit of AUDITS) {
+    const textResult = run("pnpm", auditCommand(audit.appPath));
+    const textOutput = combinedOutput(textResult);
+    const textOutputPath = path.join(OUTPUT_DIR, `${audit.outputName}.txt`);
+
+    writeFileSync(textOutputPath, textOutput);
+    process.stdout.write(textOutput);
+
+    const failed = textResult.status !== 0;
+    auditResults.push({
+      ...audit,
+      failed,
+      command: commandForDisplay(audit.appPath),
+      textOutputPath,
+    });
+
+    if (!failed) {
+      continue;
+    }
+
+    const jsonResult = run("pnpm", auditCommand(audit.appPath, ["--output-format", "json"]));
+    const jsonOutput = combinedOutput(jsonResult);
+    const jsonOutputPath = path.join(OUTPUT_DIR, `${audit.outputName}.json`);
+    writeFileSync(jsonOutputPath, jsonOutput);
+
+    const report = extractJson(jsonOutput);
+    const appFindings = collectFindings(report, `${textOutput}\n${jsonOutput}`, audit);
+
+    if (appFindings.length === 0) {
+      throw new Error(
+        `${audit.name} audit failed, but no GHSA advisory IDs could be parsed from audit-ci output.`,
+      );
+    }
+
+    for (const finding of appFindings) {
+      pushFinding(findings, finding);
+    }
+  }
+
+  const pulls = await listOpenPullRequests();
+  const coveredKeys = extractCoveredKeys(pulls);
+  const coveredFindings = [];
+  const uncoveredFindings = [];
+
+  for (const finding of findings) {
+    const coveredBy = coveredKeys.get(finding.key);
+    if (coveredBy) {
+      coveredFindings.push({ ...finding, coveredBy });
+    } else {
+      uncoveredFindings.push(finding);
+    }
+  }
+
+  const uncoveredKeys = uncoveredFindings.map((finding) => finding.key);
+  const marker = buildMarker(uncoveredKeys);
+  const prompt = buildPrompt({
+    uncoveredFindings,
+    coveredFindings,
+    marker,
+    commands: AUDITS.map((audit) => commandForDisplay(audit.appPath)),
+  });
+
+  const findingsPath = path.join(OUTPUT_DIR, "findings.json");
+  const uncoveredPath = path.join(OUTPUT_DIR, "uncovered-findings.json");
+  const coveredPath = path.join(OUTPUT_DIR, "covered-findings.json");
+  const auditResultsPath = path.join(OUTPUT_DIR, "audit-results.json");
+  const promptPath = path.join(OUTPUT_DIR, "claude-prompt.md");
+
+  writeFileSync(findingsPath, `${JSON.stringify(findings, null, 2)}\n`);
+  writeFileSync(uncoveredPath, `${JSON.stringify(uncoveredFindings, null, 2)}\n`);
+  writeFileSync(coveredPath, `${JSON.stringify(coveredFindings, null, 2)}\n`);
+  writeFileSync(auditResultsPath, `${JSON.stringify(auditResults, null, 2)}\n`);
+  writeFileSync(promptPath, prompt);
+
+  writeGithubOutput("has_uncovered", uncoveredFindings.length > 0 ? "true" : "false");
+  writeGithubOutput("prompt", prompt);
+  writeGithubOutput("prompt_file", promptPath);
+  writeGithubOutput("marker", marker);
+  writeGithubOutput("vuln_keys_json", JSON.stringify(uncoveredKeys));
+  writeGithubOutput("uncovered_findings_json", JSON.stringify(uncoveredFindings));
+  writeGithubOutput("uncovered_count", String(uncoveredFindings.length));
+  writeGithubOutput("covered_count", String(coveredFindings.length));
+  writeGithubOutput("findings_count", String(findings.length));
+
+  writeStepSummary({
+    findings,
+    coveredFindings,
+    uncoveredFindings,
+    outputDir: OUTPUT_DIR,
+  });
+
+  if (uncoveredFindings.length === 0) {
+    console.log("No uncovered audit-ci vulnerabilities found on the default branch.");
+  } else {
+    console.log(`Found ${uncoveredFindings.length} uncovered audit-ci vulnerability key(s).`);
+  }
+}
+
+main().catch((error) => {
+  console.error(error);
+  process.exit(1);
+});

참고/firecrawl-main/.github/scripts/check_version_has_incremented.py

@@ -0,0 +1,252 @@
+"""
+checks local versions against published versions.
+
+# Usage:
+
+python .github/scripts/check_version_has_incremented.py js ./apps/js-sdk/firecrawl @mendable/firecrawl-js
+Local version: 0.0.22
+Published version: 0.0.21
+true
+
+python .github/scripts/check_version_has_incremented.py python ./apps/python-sdk/firecrawl firecrawl-py
+Local version: 0.0.11
+Published version: 0.0.11
+false
+
+python .github/scripts/check_version_has_incremented.py java ./apps/java-sdk com.firecrawl:firecrawl-java
+Local version: 1.0.0
+Published version: 0.0.0  (0.0.0 means not yet published on Maven Central)
+true
+
+python .github/scripts/check_version_has_incremented.py ruby ./apps/ruby-sdk firecrawl-sdk
+Local version: 1.0.0
+Published version: 0.0.0  (0.0.0 means not yet published on RubyGems)
+true
+
+python .github/scripts/check_version_has_incremented.py dotnet ./apps/dot-net-sdk/Firecrawl firecrawl-sdk
+Local version: 1.0.0
+Published version: 0.0.0  (0.0.0 means not yet published on NuGet)
+true
+python .github/scripts/check_version_has_incremented.py php ./apps/php-sdk firecrawl/firecrawl-sdk
+Local version: 1.0.0
+Published version: 0.0.0  (0.0.0 means not yet published on Packagist)
+true
+"""
+import json
+import os
+import re
+import sys
+from pathlib import Path
+
+import requests
+from packaging.version import Version
+from packaging.version import parse as parse_version
+
+
+def get_python_version(file_path: str) -> str:
+    """Extract version string from Python file."""
+    version_file = Path(file_path).read_text()
+    version_match = re.search(r"^__version__ = ['\"]([^'\"]*)['\"]", version_file, re.M)
+    if version_match:
+        return version_match.group(1).strip()
+    raise RuntimeError("Unable to find version string.")
+
+def get_pypi_version(package_name: str) -> str:
+    """Get latest version of Python package from PyPI."""
+    response = requests.get(f"https://pypi.org/pypi/{package_name}/json")
+    version = response.json()['info']['version']
+    return version.strip()
+
+def get_js_version(file_path: str) -> str:
+    """Extract version string from package.json."""
+    with open(file_path, 'r') as file:
+        package_json = json.load(file)
+    if 'version' in package_json:
+        return package_json['version'].strip()
+    raise RuntimeError("Unable to find version string in package.json.")
+
+def get_npm_version(package_name: str) -> str:
+    """Get latest version of JavaScript package from npm."""
+    response = requests.get(f"https://registry.npmjs.org/{package_name}/latest")
+    version = response.json()['version']
+    return version.strip()
+
+def get_gradle_version(file_path: str) -> str:
+    """Extract version string from build.gradle.kts."""
+    build_file = Path(file_path).read_text()
+    version_match = re.search(r'^version\s*=\s*["\']([^"\']*)["\']', build_file, re.M)
+    if version_match:
+        return version_match.group(1).strip()
+    raise RuntimeError("Unable to find version string in build.gradle.kts.")
+
+def get_ruby_version(file_path: str) -> str:
+    """Extract version string from Ruby version file (lib/firecrawl/version.rb)."""
+    version_file = Path(file_path).read_text()
+    version_match = re.search(r'VERSION\s*=\s*["\']([^"\']*)["\']', version_file, re.M)
+    if version_match:
+        return version_match.group(1).strip()
+    raise RuntimeError("Unable to find version string in Ruby version file.")
+
+def get_rubygems_version(package_name: str) -> str:
+    """Get latest version of Ruby gem from RubyGems.org."""
+    response = requests.get(f"https://rubygems.org/api/v1/versions/{package_name}/latest.json")
+    if response.status_code == 404:
+        return "0.0.0"
+    version = response.json()['version']
+    if version == "unknown":
+        return "0.0.0"
+    return version.strip()
+
+def get_maven_central_version(package_name: str) -> str:
+    """Get latest version of Java package from Maven Central. package_name should be groupId:artifactId."""
+    group_id, artifact_id = package_name.split(":")
+    group_path = group_id.replace(".", "/")
+    url = f"https://repo1.maven.org/maven2/{group_path}/{artifact_id}/maven-metadata.xml"
+    response = requests.get(url)
+    if response.status_code == 404:
+        return "0.0.0"
+    response.raise_for_status()
+    version_match = re.search(r"<release>(.*?)</release>", response.text)
+    if not version_match:
+        version_match = re.search(r"<latest>(.*?)</latest>", response.text)
+    if version_match:
+        return version_match.group(1).strip()
+    return "0.0.0"
+
+def get_csproj_version(file_path: str) -> str:
+    """Extract version string from .csproj file."""
+    csproj_file = Path(file_path).read_text()
+    version_match = re.search(r'<Version>(.*?)</Version>', csproj_file)
+    if version_match:
+        return version_match.group(1).strip()
+    raise RuntimeError("Unable to find version string in .csproj file.")
+
+def get_nuget_version(package_name: str) -> str:
+    """Get latest version of .NET package from NuGet."""
+    url = f"https://api.nuget.org/v3-flatcontainer/{package_name.lower()}/index.json"
+    response = requests.get(url)
+    if response.status_code == 404:
+        return "0.0.0"
+    response.raise_for_status()
+    versions = response.json().get('versions', [])
+    if versions:
+        return versions[-1].strip()
+    return "0.0.0"
+
+def get_php_version(file_path: str) -> str:
+    """Extract version string from PHP Version.php file."""
+    version_file = Path(file_path).read_text()
+    version_match = re.search(r"SDK_VERSION\s*=\s*['\"]([^'\"]*)['\"]", version_file)
+    if version_match:
+        return version_match.group(1).strip()
+    raise RuntimeError("Unable to find SDK_VERSION string in Version.php.")
+
+def get_packagist_version(package_name: str) -> str:
+    """Get latest version of PHP package from Packagist. package_name should be vendor/package."""
+    url = f"https://packagist.org/packages/{package_name}.json"
+    response = requests.get(url)
+    if response.status_code == 404:
+        return "0.0.0"
+    response.raise_for_status()
+    data = response.json()
+    package_data = data.get("package", {})
+    versions = package_data.get("versions", {})
+    # Filter out dev versions and find highest stable version
+    stable_versions = []
+    for v in versions:
+        normalized = v.lstrip("v")
+        if "dev" in v or not re.match(r"^\d", normalized):
+            continue
+        try:
+            parsed = parse_version(normalized)
+        except Exception:
+            continue
+        if not parsed.is_prerelease:
+            stable_versions.append(normalized)
+    if not stable_versions:
+        return "0.0.0"
+    stable_versions.sort(key=lambda x: parse_version(x), reverse=True)
+    return stable_versions[0]
+
+def get_rust_version(file_path: str) -> str:
+    """Extract version string from Cargo.toml."""
+    import toml
+    cargo_toml = toml.load(file_path)
+    if 'package' in cargo_toml and 'version' in cargo_toml['package']:
+        return cargo_toml['package']['version'].strip()
+    raise RuntimeError("Unable to find version string in Cargo.toml.")
+
+def get_crates_version(package_name: str) -> str:
+    """Get latest version of Rust package from crates.io."""
+    response = requests.get(
+        f"https://crates.io/api/v1/crates/{package_name}",
+        headers={"User-Agent": "firecrawl-version-check"}
+    )
+    if response.status_code == 404:
+        return "0.0.0"
+    response.raise_for_status()
+    version = response.json()['crate']['newest_version']
+    return version.strip()
+
+def is_version_incremented(local_version: str, published_version: str) -> bool:
+    """Compare local and published versions."""
+    local_version_parsed: Version = parse_version(local_version)
+    published_version_parsed: Version = parse_version(published_version)
+    return local_version_parsed > published_version_parsed
+
+if __name__ == "__main__":
+    package_type = sys.argv[1]
+    package_path = sys.argv[2]
+    package_name = sys.argv[3]
+
+    if package_type == "python":
+        # Get current version from __init__.py
+        current_version = get_python_version(os.path.join(package_path, '__init__.py'))
+        # Get published version from PyPI
+        published_version = get_pypi_version(package_name)
+    elif package_type == "js":
+        # Get current version from package.json
+        current_version = get_js_version(os.path.join(package_path, 'package.json'))
+        # Get published version from npm
+        published_version = get_npm_version(package_name)
+    elif package_type == "java":
+        # Get current version from build.gradle.kts
+        current_version = get_gradle_version(os.path.join(package_path, 'build.gradle.kts'))
+        # Get published version from Maven Central
+        published_version = get_maven_central_version(package_name)
+    elif package_type == "ruby":
+        # Get current version from lib/firecrawl/version.rb
+        current_version = get_ruby_version(os.path.join(package_path, 'lib', 'firecrawl', 'version.rb'))
+        # Get published version from RubyGems
+        published_version = get_rubygems_version(package_name)
+    elif package_type == "dotnet":
+        # Get current version from .csproj file — look for any .csproj in the directory
+        csproj_files = list(Path(package_path).glob('*.csproj'))
+        if not csproj_files:
+            raise RuntimeError(f"No .csproj file found in {package_path}")
+        current_version = get_csproj_version(str(csproj_files[0]))
+        # Get published version from NuGet
+        published_version = get_nuget_version(package_name)
+    elif package_type == "php":
+        # Get current version from src/Version.php
+        current_version = get_php_version(os.path.join(package_path, 'src', 'Version.php'))
+        # Get published version from Packagist
+        published_version = get_packagist_version(package_name)
+    elif package_type == "rust":
+        # Get current version from Cargo.toml
+        current_version = get_rust_version(os.path.join(package_path, 'Cargo.toml'))
+        # Get published version from crates.io
+        published_version = get_crates_version(package_name)
+
+    else:
+        raise ValueError("Invalid package type. Use 'python', 'js', 'java', 'ruby', 'rust', 'dotnet', or 'php'.")
+
+    # Print versions for debugging
+    # print(f"Local version: {current_version}")
+    # print(f"Published version: {published_version}")
+
+    # Compare versions and print result
+    if is_version_incremented(current_version, published_version):
+        print("true")
+    else:
+        print("false")

참고/firecrawl-main/.github/scripts/eval_run.py

@@ -0,0 +1,38 @@
+import requests
+import argparse
+import sys
+import os
+
+def main():
+    parser = argparse.ArgumentParser(description='Run evaluation benchmark')
+    parser.add_argument('--label', required=True, help='Label for the evaluation run')
+    parser.add_argument('--api-url', required=True, help='API URL')
+    parser.add_argument('--api-key', required=True, help='API key')
+    parser.add_argument('--experiment-id', required=True, help='Experiment ID')
+
+    args = parser.parse_args()
+
+    try:
+        response = requests.post(
+            f"{args.api_url}/run",
+            json={
+                "experiment_id": args.experiment_id,
+                "api_key": args.api_key,
+                "label": args.label
+            },
+            headers={
+                "Content-Type": "application/json"
+            }
+        )
+        
+        response.raise_for_status()
+        
+        print("Evaluation run started successfully")
+        print(f"Response: {response.json()}")
+        
+    except requests.exceptions.RequestException as e:
+        print(f"Error running evaluation: {str(e)}", file=sys.stderr)
+        sys.exit(1)
+
+if __name__ == "__main__":
+    main()

참고/firecrawl-main/.github/scripts/requirements.txt

@@ -0,0 +1,3 @@
+requests
+packaging
+toml

참고/firecrawl-main/.github/workflows/deploy-go-service.yaml

@@ -0,0 +1,40 @@
+name: Deploy Go Service to GHCR
+
+env:
+  DOTNET_VERSION: "6.0.x"
+
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - apps/go-html-to-md-service/**
+      - .github/workflows/deploy-go-service.yaml
+  workflow_dispatch:
+
+jobs:
+  push-app-image:
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+    defaults:
+      run:
+        working-directory: "./apps/go-html-to-md-service"
+    steps:
+      - name: "Checkout GitHub Action"
+        uses: actions/checkout@main
+
+      - name: Setup Blacksmith Builder
+        uses: useblacksmith/setup-docker-builder@v1
+
+      - name: "Login to GitHub Container Registry"
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: ${{github.actor}}
+          password: ${{secrets.GITHUB_TOKEN}}
+
+      - name: "Build and Push Image"
+        uses: useblacksmith/build-push-action@v2
+        with:
+          context: ./apps/go-html-to-md-service
+          push: true
+          tags: ghcr.io/firecrawl/go-html-to-md-service:latest

참고/firecrawl-main/.github/workflows/deploy-image-staging.yml

@@ -0,0 +1,29 @@
+name: STAGING Deploy Images to GHCR
+
+env:
+  DOTNET_VERSION: '6.0.x'
+
+on:
+  workflow_dispatch:
+
+jobs:
+  push-app-image:
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+    defaults:
+      run:
+        working-directory: './apps/api'
+    steps:
+      - name: 'Checkout GitHub Action'
+        uses: actions/checkout@main
+
+      - name: 'Login to GitHub Container Registry'
+        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
+        with:
+          registry: ghcr.io
+          username: ${{github.actor}}
+          password: ${{secrets.GITHUB_TOKEN}}
+
+      - name: 'Build Inventory Image'
+        run: |
+          docker build . --tag ghcr.io/firecrawl/firecrawl-staging:latest
+          docker push ghcr.io/firecrawl/firecrawl-staging:latest

참고/firecrawl-main/.github/workflows/deploy-image.yml

@@ -0,0 +1,81 @@
+name: Deploy Images to GHCR
+
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - apps/api/**
+      - .github/workflows/deploy-image.yml
+  workflow_dispatch:
+
+jobs:
+  build:
+    runs-on: ${{ matrix.runner }}
+    strategy:
+      matrix:
+        include:
+          - platform: linux/amd64
+            runner: blacksmith-4vcpu-ubuntu-2404
+          - platform: linux/arm64
+            runner: blacksmith-4vcpu-ubuntu-2404-arm
+    defaults:
+      run:
+        working-directory: './apps/api'
+    steps:
+      - name: 'Checkout GitHub Action'
+        uses: actions/checkout@main
+
+      - name: Setup Blacksmith Builder
+        uses: useblacksmith/setup-docker-builder@ef12d5b165b596e3aa44ea8198d8fde563eab402 # v1
+
+      - name: 'Login to GitHub Container Registry'
+        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
+        with:
+          registry: ghcr.io
+          username: ${{github.actor}}
+          password: ${{secrets.GITHUB_TOKEN}}
+
+      - name: Lowercase Repo Owner
+        run: |
+          echo "REPO_OWNER=${GITHUB_REPOSITORY_OWNER,,}" >>${GITHUB_ENV}
+        env:
+          GITHUB_REPOSITORY_OWNER: '${{ github.repository_owner }}'
+
+      - name: Extract platform suffix
+        run: |
+          platform=${{ matrix.platform }}
+          echo "PLATFORM_SUFFIX=${platform//\//-}" >> $GITHUB_ENV
+
+      - name: 'Build and Push Image'
+        uses: useblacksmith/build-push-action@30c71162f16ea2c27c3e21523255d209b8b538c1 # v2
+        with:
+          context: ./apps/api
+          push: true
+          tags: ghcr.io/${{ env.REPO_OWNER }}/firecrawl:${{ env.PLATFORM_SUFFIX }}
+          platforms: ${{ matrix.platform }}
+          provenance: false
+
+  manifest:
+    runs-on: blacksmith-2vcpu-ubuntu-2404
+    needs: build
+    steps:
+      - name: 'Login to GitHub Container Registry'
+        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
+        with:
+          registry: ghcr.io
+          username: ${{github.actor}}
+          password: ${{secrets.GITHUB_TOKEN}}
+
+      - name: Lowercase Repo Owner
+        run: |
+          echo "REPO_OWNER=${GITHUB_REPOSITORY_OWNER,,}" >>${GITHUB_ENV}
+        env:
+          GITHUB_REPOSITORY_OWNER: '${{ github.repository_owner }}'
+
+      - name: 'Create and Push Multi-Arch Manifest'
+        run: |
+          docker manifest create ghcr.io/${{ env.REPO_OWNER }}/firecrawl:latest \
+            ghcr.io/${{ env.REPO_OWNER }}/firecrawl:linux-amd64 \
+            ghcr.io/${{ env.REPO_OWNER }}/firecrawl:linux-arm64
+          docker manifest push ghcr.io/${{ env.REPO_OWNER }}/firecrawl:latest

참고/firecrawl-main/.github/workflows/deploy-nuq-postgres.yml

@@ -0,0 +1,31 @@
+name: Deploy NuQ Postgres to GHCR
+
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - apps/nuq-postgres/**
+  workflow_dispatch:
+
+jobs:
+  push-app-image:
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+    defaults:
+      run:
+        working-directory: './apps/nuq-postgres'
+    steps:
+      - name: 'Checkout GitHub Action'
+        uses: actions/checkout@main
+
+      - name: 'Login to GitHub Container Registry'
+        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
+        with:
+          registry: ghcr.io
+          username: ${{github.actor}}
+          password: ${{secrets.GITHUB_TOKEN}}
+
+      - name: 'Build NuQ Postgres Image'
+        run: |
+          docker build . --tag ghcr.io/firecrawl/nuq-postgres:latest
+          docker push ghcr.io/firecrawl/nuq-postgres:latest

참고/firecrawl-main/.github/workflows/deploy-playwright.yml

@@ -0,0 +1,81 @@
+name: Deploy Playwright to GHCR
+
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - apps/playwright-service-ts/**
+      - .github/workflows/deploy-playwright.yml
+  workflow_dispatch:
+
+jobs:
+  build:
+    runs-on: ${{ matrix.runner }}
+    strategy:
+      matrix:
+        include:
+          - platform: linux/amd64
+            runner: blacksmith-4vcpu-ubuntu-2404
+          - platform: linux/arm64
+            runner: blacksmith-4vcpu-ubuntu-2404-arm
+    defaults:
+      run:
+        working-directory: './apps/playwright-service-ts'
+    steps:
+      - name: 'Checkout GitHub Action'
+        uses: actions/checkout@main
+
+      - name: Setup Blacksmith Builder
+        uses: useblacksmith/setup-docker-builder@ef12d5b165b596e3aa44ea8198d8fde563eab402 # v1
+
+      - name: 'Login to GitHub Container Registry'
+        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
+        with:
+          registry: ghcr.io
+          username: ${{github.actor}}
+          password: ${{secrets.GITHUB_TOKEN}}
+
+      - name: Lowercase Repo Owner
+        run: |
+          echo "REPO_OWNER=${GITHUB_REPOSITORY_OWNER,,}" >>${GITHUB_ENV}
+        env:
+          GITHUB_REPOSITORY_OWNER: '${{ github.repository_owner }}'
+
+      - name: Extract platform suffix
+        run: |
+          platform=${{ matrix.platform }}
+          echo "PLATFORM_SUFFIX=${platform//\//-}" >> $GITHUB_ENV
+
+      - name: 'Build and Push Image'
+        uses: useblacksmith/build-push-action@30c71162f16ea2c27c3e21523255d209b8b538c1 # v2
+        with:
+          context: ./apps/playwright-service-ts
+          push: true
+          tags: ghcr.io/${{ env.REPO_OWNER }}/playwright-service:${{ env.PLATFORM_SUFFIX }}
+          platforms: ${{ matrix.platform }}
+          provenance: false
+
+  manifest:
+    runs-on: blacksmith-2vcpu-ubuntu-2404
+    needs: build
+    steps:
+      - name: 'Login to GitHub Container Registry'
+        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
+        with:
+          registry: ghcr.io
+          username: ${{github.actor}}
+          password: ${{secrets.GITHUB_TOKEN}}
+
+      - name: Lowercase Repo Owner
+        run: |
+          echo "REPO_OWNER=${GITHUB_REPOSITORY_OWNER,,}" >>${GITHUB_ENV}
+        env:
+          GITHUB_REPOSITORY_OWNER: '${{ github.repository_owner }}'
+
+      - name: 'Create and Push Multi-Arch Manifest'
+        run: |
+          docker manifest create ghcr.io/${{ env.REPO_OWNER }}/playwright-service:latest \
+            ghcr.io/${{ env.REPO_OWNER }}/playwright-service:linux-amd64 \
+            ghcr.io/${{ env.REPO_OWNER }}/playwright-service:linux-arm64
+          docker manifest push ghcr.io/${{ env.REPO_OWNER }}/playwright-service:latest

참고/firecrawl-main/.github/workflows/deploy-redis.yml

@@ -0,0 +1,34 @@
+name: Deploy Redis to GHCR
+
+env:
+  DOTNET_VERSION: '6.0.x'
+
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - apps/redis/**
+  workflow_dispatch:
+
+jobs:
+      push-app-image:
+        runs-on: blacksmith-4vcpu-ubuntu-2404
+        defaults:
+          run:
+            working-directory: './apps/redis'
+        steps:
+          - name: 'Checkout GitHub Action'
+            uses: actions/checkout@main
+
+          - name: 'Login to GitHub Container Registry'
+            uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
+            with:
+              registry: ghcr.io
+              username: ${{github.actor}}
+              password: ${{secrets.GITHUB_TOKEN}}
+
+          - name: 'Build Inventory Image'
+            run: |
+              docker build . --tag ghcr.io/firecrawl/firecrawl-redis:latest
+              docker push ghcr.io/firecrawl/firecrawl-redis:latest

참고/firecrawl-main/.github/workflows/eval-prod.yml

@@ -0,0 +1,36 @@
+name: Run Eval Benchmark Prod
+
+env:
+  EVAL_API_URL: ${{ secrets.EVAL_API_URL }}
+  EVAL_API_KEY: ${{ secrets.EVAL_API_KEY }}
+  EVAL_EXPERIMENT_ID: ${{ secrets.EVAL_BENCHMARK_EXPERIMENT_ID }}
+
+on:
+  workflow_run:
+    workflows: ["Deploy Images to GHCR"]
+    types:
+      - completed
+    branches:
+      - main
+  workflow_dispatch:
+
+jobs:
+  run-eval-benchmark-prod:
+        runs-on: blacksmith-2vcpu-ubuntu-2404
+        if: ${{ github.event.workflow_run.conclusion == 'success' || github.event_name == 'workflow_dispatch' }}
+        steps:
+          - name: Checkout repository
+            uses: actions/checkout@v5
+
+          - name: 'Install dependencies'
+            run: |
+              python -m pip install --upgrade pip
+              pip install requests
+
+          # make sure the image is deployed before running the eval benchmark
+          - name: Wait for 2 minutes
+            run: sleep 120
+
+          - name: 'Run Eval Benchmark Prod'
+            run: |
+              python .github/scripts/eval_run.py --label prod.${{ github.sha }} --api-url ${{ env.EVAL_API_URL }} --api-key ${{ env.EVAL_API_KEY }} --experiment-id ${{ env.EVAL_EXPERIMENT_ID }}

참고/firecrawl-main/.github/workflows/ghcr-clean.yml

@@ -0,0 +1,18 @@
+name: Cleanup Untagged Images
+
+on:
+  workflow_dispatch:
+
+jobs:
+  delete-untagged-images:
+    name: Delete Untagged Images
+    runs-on: blacksmith-2vcpu-ubuntu-2404
+    steps:
+      - uses: bots-house/ghcr-delete-image-action@3827559c68cb4dcdf54d813ea9853be6d468d3a4 # v1.1.0
+        with:
+          owner: firecrawl
+          name: firecrawl
+          # NOTE: using Personal Access Token
+          token: ${{secrets.GITHUB_TOKEN}}
+          # Keep latest N untagged images
+          untagged-keep-latest: 5

참고/firecrawl-main/.github/workflows/npm-audit-claude-remediation.yml

@@ -0,0 +1,106 @@
+name: Audit NPM Claude Remediation
+
+on:
+  workflow_run:
+    workflows: ["Audit NPM Packages"]
+    types:
+      - completed
+  workflow_dispatch:
+
+permissions:
+  actions: read
+  contents: write
+  id-token: write
+  issues: write
+  pull-requests: write
+
+concurrency:
+  group: audit-npm-claude-remediation-${{ github.event.repository.default_branch }}
+  cancel-in-progress: false
+
+jobs:
+  remediate:
+    if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'failure' }}
+    runs-on: blacksmith-2vcpu-ubuntu-2404
+    timeout-minutes: 60
+
+    steps:
+      - name: Checkout default branch
+        uses: actions/checkout@v5
+        with:
+          ref: ${{ github.event.repository.default_branch }}
+
+      - name: Install pnpm
+        uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v4
+        with:
+          version: 10
+
+      - name: Scan default branch audit failures
+        id: scan
+        env:
+          GH_TOKEN: ${{ github.token }}
+          GITHUB_TOKEN: ${{ github.token }}
+        run: node .github/scripts/audit-ci-vuln-scan.mjs
+
+      - name: Skip when all vulnerabilities are covered
+        if: ${{ steps.scan.outputs.has_uncovered != 'true' }}
+        run: |
+          echo "No uncovered audit-ci vulnerabilities on ${GITHUB_REF_NAME}; skipping Claude."
+
+      - name: Run Claude remediation
+        id: claude
+        if: ${{ steps.scan.outputs.has_uncovered == 'true' }}
+        uses: anthropics/claude-code-action@2cc1ac1331eac7a6a96d716dd204dd2888d0fcd2 # v1
+        with:
+          anthropic_api_key: ${{ secrets.NPM_AUDIT_CLAUDE_ANTHROPIC_API_KEY }}
+          base_branch: ${{ github.event.repository.default_branch }}
+          branch_prefix: claude/audit-ci/
+          prompt: ${{ steps.scan.outputs.prompt }}
+          claude_args: |
+            --max-turns 40
+            --allowedTools "Read,Edit,MultiEdit,Write,Glob,Grep,LS,WebFetch,WebSearch,TodoWrite,Bash(pnpm:*),Bash(npm:*),Bash(node:*),Bash(git:*),Bash(gh:*),Bash(jq:*),Bash(rg:*),Bash(curl:*),Bash(date:*)"
+
+      - name: Ensure remediation PR marker
+        if: ${{ steps.scan.outputs.has_uncovered == 'true' && steps.claude.outputs.branch_name != '' }}
+        env:
+          GH_TOKEN: ${{ github.token }}
+          BRANCH_NAME: ${{ steps.claude.outputs.branch_name }}
+          MARKER: ${{ steps.scan.outputs.marker }}
+        run: |
+          set -euo pipefail
+
+          PR_NUMBER="$(gh pr list --head "$BRANCH_NAME" --state open --json number --jq '.[0].number // empty')"
+          if [ -z "$PR_NUMBER" ]; then
+            echo "::warning::Claude did not leave an open PR for branch ${BRANCH_NAME}; marker could not be enforced."
+            exit 0
+          fi
+
+          gh label create audit-ci-remediation \
+            --description "Automated audit-ci vulnerability remediation" \
+            --color "B60205" \
+            2>/dev/null || true
+          gh pr edit "$PR_NUMBER" --add-label audit-ci-remediation
+
+          BODY_FILE="$(mktemp)"
+          UPDATED_BODY_FILE="$(mktemp)"
+          gh pr view "$PR_NUMBER" --json body --jq '.body // ""' > "$BODY_FILE"
+
+          node - "$BODY_FILE" "$UPDATED_BODY_FILE" "$MARKER" <<'NODE'
+          const { readFileSync, writeFileSync } = require("node:fs");
+
+          const [, , inputPath, outputPath, marker] = process.argv;
+          const markerRegex = /<!--\s*audit-ci-vuln-keys:\s*\[[\s\S]*?\]\s*-->/;
+          let body = readFileSync(inputPath, "utf8");
+
+          if (!markerRegex.test(body)) {
+            if (/^## Summary\b.*$/m.test(body)) {
+              body = body.replace(/^## Summary\b.*$/m, (heading) => `${heading}\n\n${marker}`);
+            } else {
+              body = `## Summary\n\n${marker}\n\n${body.trim()}\n`;
+            }
+          }
+
+          writeFileSync(outputPath, body);
+          NODE
+
+          gh pr edit "$PR_NUMBER" --body-file "$UPDATED_BODY_FILE"

참고/firecrawl-main/.github/workflows/npm-audit.yml

@@ -0,0 +1,112 @@
+name: Audit NPM Packages
+
+on:
+  pull_request:
+    branches:
+      - main
+
+jobs:
+  audit:
+    runs-on: blacksmith-2vcpu-ubuntu-2404
+    steps:
+      - uses: actions/checkout@v5
+      - name: Install pnpm
+        uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v4
+        with:
+          version: 10
+      
+      - name: Create audit output directory
+        run: mkdir -p /tmp/audit-outputs
+
+      - name: Audit API Packages
+        id: audit-api
+        continue-on-error: true
+        run: |
+          set -o pipefail
+          pnpm dlx audit-ci@^7 --directory apps/api --config apps/api/audit-ci.jsonc 2>&1 | tee /tmp/audit-outputs/api.txt
+
+      - name: Audit Playwright Service Packages
+        id: audit-playwright-service
+        continue-on-error: true
+        run: |
+          set -o pipefail
+          pnpm dlx audit-ci@^7 --directory apps/playwright-service-ts --config apps/playwright-service-ts/audit-ci.jsonc 2>&1 | tee /tmp/audit-outputs/playwright-service.txt
+
+      - name: Audit JavaScript SDK Packages
+        id: audit-js-sdk
+        continue-on-error: true
+        run: |
+          set -o pipefail
+          pnpm dlx audit-ci@^7 --directory apps/js-sdk --config apps/js-sdk/audit-ci.jsonc 2>&1 | tee /tmp/audit-outputs/js-sdk.txt
+
+      - name: Audit JavaScript SDK Firecrawl Packages
+        id: audit-js-sdk-firecrawl
+        continue-on-error: true
+        run: |
+          set -o pipefail
+          pnpm dlx audit-ci@^7 --directory apps/js-sdk/firecrawl --config apps/js-sdk/firecrawl/audit-ci.jsonc 2>&1 | tee /tmp/audit-outputs/js-sdk-firecrawl.txt
+
+      - name: Audit Test Suite Packages
+        id: audit-test-suite
+        continue-on-error: true
+        run: |
+          set -o pipefail
+          pnpm dlx audit-ci@^7 --directory apps/test-suite --config apps/test-suite/audit-ci.jsonc 2>&1 | tee /tmp/audit-outputs/test-suite.txt
+
+      - name: Audit Ingestion UI Packages
+        id: audit-ingestion-ui
+        continue-on-error: true
+        run: |
+          set -o pipefail
+          pnpm dlx audit-ci@^7 --directory apps/ui/ingestion-ui --config apps/ui/ingestion-ui/audit-ci.jsonc 2>&1 | tee /tmp/audit-outputs/ingestion-ui.txt
+
+      - name: Audit Test Site Packages
+        id: audit-test-site
+        continue-on-error: true
+        run: |
+          set -o pipefail
+          pnpm dlx audit-ci@^7 --directory apps/test-site --config apps/test-site/audit-ci.jsonc 2>&1 | tee /tmp/audit-outputs/test-site.txt
+
+      - name: Report audit failures
+        if: always()
+        run: |
+          declare -A AUDIT_FILES=(
+            ["API"]="api.txt"
+            ["Playwright Service"]="playwright-service.txt"
+            ["JavaScript SDK"]="js-sdk.txt"
+            ["JavaScript SDK Firecrawl"]="js-sdk-firecrawl.txt"
+            ["Test Suite"]="test-suite.txt"
+            ["Ingestion UI"]="ingestion-ui.txt"
+            ["Test Site"]="test-site.txt"
+          )
+
+          declare -A AUDIT_OUTCOMES=(
+            ["API"]="${{ steps.audit-api.outcome }}"
+            ["Playwright Service"]="${{ steps.audit-playwright-service.outcome }}"
+            ["JavaScript SDK"]="${{ steps.audit-js-sdk.outcome }}"
+            ["JavaScript SDK Firecrawl"]="${{ steps.audit-js-sdk-firecrawl.outcome }}"
+            ["Test Suite"]="${{ steps.audit-test-suite.outcome }}"
+            ["Ingestion UI"]="${{ steps.audit-ingestion-ui.outcome }}"
+            ["Test Site"]="${{ steps.audit-test-site.outcome }}"
+          )
+
+          FAILED=false
+          for name in "API" "Playwright Service" "JavaScript SDK" "JavaScript SDK Firecrawl" "Test Suite" "Ingestion UI" "Test Site"; do
+            if [ "${AUDIT_OUTCOMES[$name]}" == "failure" ]; then
+              FAILED=true
+              echo ""
+              echo "=========================================="
+              echo "❌ $name audit failed"
+              echo "=========================================="
+              if [ -f "/tmp/audit-outputs/${AUDIT_FILES[$name]}" ]; then
+                # Extract only the summary (from "Found vulnerable advisory paths:" to end)
+                sed -n '/Found vulnerable advisory paths:/,$p' "/tmp/audit-outputs/${AUDIT_FILES[$name]}" | sed 's/\x1b\[[0-9;]*m//g'
+              fi
+            fi
+          done
+
+          if [ "$FAILED" == "true" ]; then
+            exit 1
+          else
+            echo "✅ All audits passed"
+          fi

참고/firecrawl-main/.github/workflows/publish-dotnet-sdk.yml

@@ -0,0 +1,59 @@
+name: Publish .NET SDK
+
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - main
+    paths:
+      - 'apps/dot-net-sdk/**'
+
+jobs:
+  publish:
+    name: Publish to NuGet
+    runs-on: blacksmith-2vcpu-ubuntu-2404
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v5
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.x'
+
+      - name: Install version check dependencies
+        run: pip install requests packaging
+
+      - name: Run version check script
+        id: version_check
+        run: |
+          VERSION_INCREMENTED=$(python .github/scripts/check_version_has_incremented.py dotnet ./apps/dot-net-sdk/Firecrawl firecrawl-sdk)
+          echo "VERSION_INCREMENTED=$VERSION_INCREMENTED" >> $GITHUB_ENV
+
+      - name: Set up .NET 8
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        uses: actions/setup-dotnet@v4
+        with:
+          dotnet-version: '8.0.x'
+
+      - name: Restore dependencies
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        working-directory: ./apps/dot-net-sdk
+        run: dotnet restore
+
+      - name: Build
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        working-directory: ./apps/dot-net-sdk
+        run: dotnet build --no-restore --configuration Release
+
+      - name: Pack
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        working-directory: ./apps/dot-net-sdk/Firecrawl
+        run: dotnet pack --no-build --configuration Release --output ../nupkgs
+
+      - name: Publish to NuGet
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        working-directory: ./apps/dot-net-sdk
+        env:
+          NUGET_API_KEY: ${{ secrets.NUGET_API_KEY }}
+        run: dotnet nuget push ./nupkgs/*.nupkg --api-key $NUGET_API_KEY --source https://api.nuget.org/v3/index.json --skip-duplicate

참고/firecrawl-main/.github/workflows/publish-elixir-sdk.yml

@@ -0,0 +1,150 @@
+name: Publish Elixir SDK
+
+on:
+  schedule:
+    # Run daily at 06:00 UTC
+    - cron: '0 6 * * *'
+  push:
+    branches:
+      - main
+    paths:
+      - 'apps/elixir-sdk/**'
+  workflow_dispatch:
+
+defaults:
+  run:
+    working-directory: ./apps/elixir-sdk
+
+jobs:
+  regenerate:
+    if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
+    runs-on: blacksmith-2vcpu-ubuntu-2404
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v5
+
+      - name: Set up Elixir
+        uses: erlef/setup-beam@ee09b1e59bb240681c382eb1f0abc6a04af72764 # v1
+        with:
+          elixir-version: '1.17'
+          otp-version: '27'
+
+      - name: Install dependencies
+        run: mix deps.get
+
+      - name: Regenerate from OpenAPI spec
+        run: mix run generate.exs
+
+      - name: Check for changes
+        id: check_changes
+        run: |
+          if git diff --quiet HEAD -- . && [ -z "$(git ls-files --others --exclude-standard .)" ]; then
+            echo "changed=false" >> "$GITHUB_OUTPUT"
+          else
+            echo "changed=true" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Create PR with regenerated code
+        if: steps.check_changes.outputs.changed == 'true'
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          cd "$GITHUB_WORKSPACE"
+
+          VERSION=$(grep '@version' apps/elixir-sdk/mix.exs | head -1 | sed 's/.*"\(.*\)".*/\1/')
+          if [ -z "$VERSION" ]; then
+            echo "::error::Could not parse version from mix.exs"
+            exit 1
+          fi
+          BRANCH="chore/elixir-sdk-regen-v${VERSION}"
+
+          # Check if a PR already exists for this version.
+          # If gh fails, abort rather than risk deleting a live PR branch.
+          if ! PR_JSON=$(gh pr list --head "$BRANCH" --state open --json number 2>&1); then
+            echo "::error::Failed to check for existing PRs: $PR_JSON"
+            exit 1
+          fi
+          EXISTING_PR=$(echo "$PR_JSON" | jq -r '.[0].number // empty')
+          if [ -n "$EXISTING_PR" ]; then
+            echo "PR #${EXISTING_PR} already exists for ${BRANCH} — skipping."
+            exit 0
+          fi
+
+          git config user.name "github-actions[bot]"
+          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+
+          # Delete stale remote branch if it exists (no open PR at this point)
+          git push origin --delete "$BRANCH" 2>/dev/null || true
+
+          git checkout -b "$BRANCH"
+          git add apps/elixir-sdk/
+          git commit -m "chore(elixir-sdk): regenerate from OpenAPI spec (v${VERSION})"
+          git push -u origin "$BRANCH"
+
+          PR_BODY=$(cat <<EOF
+          Auto-generated PR to update the Elixir SDK from the latest OpenAPI spec.
+
+          ## Changes
+          - Regenerated \`lib/firecrawl.ex\` from the v2 OpenAPI spec
+          - Version bumped to \`${VERSION}\`
+
+          This PR was created automatically by the \`publish-elixir-sdk\` workflow.
+          EOF
+          )
+
+          gh pr create \
+            --title "chore(elixir-sdk): regenerate from OpenAPI spec (v${VERSION})" \
+            --body "$PR_BODY" \
+            --head "$BRANCH" \
+            --base main
+
+  publish:
+    if: github.event_name == 'push'
+    runs-on: blacksmith-2vcpu-ubuntu-2404
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v5
+
+      - name: Set up Elixir
+        uses: erlef/setup-beam@ee09b1e59bb240681c382eb1f0abc6a04af72764 # v1
+        with:
+          elixir-version: '1.17'
+          otp-version: '27'
+
+      - name: Install dependencies
+        run: mix deps.get
+
+      - name: Check if version needs publishing
+        id: version_info
+        run: |
+          LOCAL_VERSION=$(grep '@version' mix.exs | head -1 | sed 's/.*"\(.*\)".*/\1/')
+          if [ -z "$LOCAL_VERSION" ]; then
+            echo "::error::Could not parse local version from mix.exs"
+            exit 1
+          fi
+          echo "local_version=$LOCAL_VERSION" >> "$GITHUB_OUTPUT"
+
+          # Check published version on hex.pm
+          HTTP_STATUS=$(curl -s -o /tmp/hex_response.json -w "%{http_code}" "https://hex.pm/api/packages/firecrawl")
+          if [ "$HTTP_STATUS" = "200" ]; then
+            PUBLISHED_VERSION=$(python3 -c "import sys,json; releases=json.load(open('/tmp/hex_response.json'))['releases']; print(releases[0]['version'] if releases else '0.0.0')")
+          else
+            PUBLISHED_VERSION="0.0.0"
+          fi
+          echo "published_version=$PUBLISHED_VERSION" >> "$GITHUB_OUTPUT"
+
+          # Compare versions using simple tuple comparison
+          python3 -c "
+          local = tuple(map(int, '$LOCAL_VERSION'.split('.')))
+          pub = tuple(map(int, '$PUBLISHED_VERSION'.split('.')))
+          print('should_publish=true' if local > pub else 'should_publish=false')
+          " >> "$GITHUB_OUTPUT"
+
+      - name: Publish to hex.pm
+        if: steps.version_info.outputs.should_publish == 'true'
+        env:
+          HEX_API_KEY: ${{ secrets.HEX_API_KEY }}
+        run: |
+          mix hex.publish --yes

참고/firecrawl-main/.github/workflows/publish-go-sdk.yml

@@ -0,0 +1,94 @@
+name: Publish Go SDK
+
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - main
+    paths:
+      - 'apps/go-sdk/**'
+      - '.github/workflows/publish-go-sdk.yml'
+
+# Go modules are "published" by pushing a git tag. For modules living in a
+# monorepo subdirectory, the tag MUST be prefixed with the subdir path —
+# e.g. apps/go-sdk/v1.0.0 — for the Go module proxy to resolve it.
+#
+# This workflow reads Version from apps/go-sdk/version.go, checks whether the
+# corresponding tag already exists, and if not creates+pushes it, then warms
+# the proxy.golang.org cache to trigger pkg.go.dev indexing.
+jobs:
+  publish:
+    name: Tag and index on pkg.go.dev
+    runs-on: blacksmith-2vcpu-ubuntu-2404
+    permissions:
+      contents: write
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Set up Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: "1.23"
+          cache-dependency-path: apps/go-sdk/go.sum
+
+      - name: Build and vet
+        working-directory: ./apps/go-sdk
+        run: |
+          go build ./...
+          go vet ./...
+
+      - name: Read SDK version
+        id: version
+        working-directory: ./apps/go-sdk
+        run: |
+          VERSION=$(grep -E '^const Version = ' version.go | sed -E 's/.*"([^"]+)".*/\1/')
+          if [ -z "$VERSION" ]; then
+            echo "Failed to parse Version from apps/go-sdk/version.go" >&2
+            exit 1
+          fi
+          TAG="apps/go-sdk/v${VERSION}"
+          echo "version=${VERSION}" >> "$GITHUB_OUTPUT"
+          echo "tag=${TAG}" >> "$GITHUB_OUTPUT"
+          echo "Resolved version: ${VERSION} (tag: ${TAG})"
+
+      - name: Check if tag already exists
+        id: tag_check
+        run: |
+          TAG="${{ steps.version.outputs.tag }}"
+          if git rev-parse -q --verify "refs/tags/${TAG}" >/dev/null; then
+            echo "exists=true" >> "$GITHUB_OUTPUT"
+            echo "Tag ${TAG} already exists — skipping publish."
+          else
+            echo "exists=false" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Create and push tag
+        if: steps.tag_check.outputs.exists == 'false'
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          TAG="${{ steps.version.outputs.tag }}"
+          git config user.name "github-actions[bot]"
+          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+          git tag -a "${TAG}" -m "Release Go SDK ${TAG}"
+          git push origin "${TAG}"
+
+      - name: Warm Go module proxy (triggers pkg.go.dev indexing)
+        if: steps.tag_check.outputs.exists == 'false'
+        run: |
+          MODULE="github.com/firecrawl/firecrawl/apps/go-sdk"
+          VERSION="v${{ steps.version.outputs.version }}"
+          # Force a module-proxy fetch; pkg.go.dev discovers modules from proxy.golang.org.
+          # Retry a few times because the tag may take a moment to be visible to the proxy.
+          for i in 1 2 3 4 5; do
+            if curl -fsSL "https://proxy.golang.org/${MODULE}/@v/${VERSION}.info" -o /dev/null; then
+              echo "Module ${MODULE}@${VERSION} fetched from proxy.golang.org"
+              exit 0
+            fi
+            echo "Attempt $i: proxy not yet aware of ${VERSION}; sleeping 30s"
+            sleep 30
+          done
+          echo "::warning::proxy.golang.org did not resolve ${MODULE}@${VERSION} yet; pkg.go.dev will pick it up on the next fetch."

참고/firecrawl-main/.github/workflows/publish-java-sdk.yml

@@ -0,0 +1,69 @@
+name: Publish Java SDK
+
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - main
+    paths:
+      - 'apps/java-sdk/**'
+
+jobs:
+  publish:
+    name: Publish to Maven Central
+    runs-on: blacksmith-2vcpu-ubuntu-2404
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v5
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.x'
+
+      - name: Install version check dependencies
+        run: pip install requests packaging
+
+      - name: Run version check script
+        id: version_check
+        run: |
+          VERSION_INCREMENTED=$(python .github/scripts/check_version_has_incremented.py java ./apps/java-sdk com.firecrawl:firecrawl-java)
+          echo "VERSION_INCREMENTED=$VERSION_INCREMENTED" >> $GITHUB_ENV
+
+      - name: Set up JDK 11
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        uses: actions/setup-java@v4
+        with:
+          distribution: temurin
+          java-version: '11'
+
+      - name: Cache Gradle packages
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        uses: actions/cache@v4
+        with:
+          path: |
+            ~/.gradle/caches
+            ~/.gradle/wrapper
+          key: ${{ runner.os }}-gradle-${{ hashFiles('apps/java-sdk/**/*.gradle.kts', 'apps/java-sdk/gradle/wrapper/gradle-wrapper.properties') }}
+          restore-keys: |
+            ${{ runner.os }}-gradle-
+
+      - name: Grant execute permission for gradlew
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        working-directory: ./apps/java-sdk
+        run: chmod +x gradlew
+
+      - name: Build
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        working-directory: ./apps/java-sdk
+        run: ./gradlew build -x test
+
+      - name: Publish to Maven Central
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        working-directory: ./apps/java-sdk
+        env:
+          ORG_GRADLE_PROJECT_mavenCentralUsername: ${{ secrets.MAVEN_USERNAME }}
+          ORG_GRADLE_PROJECT_mavenCentralPassword: ${{ secrets.MAVEN_PASSWORD }}
+          ORG_GRADLE_PROJECT_signingInMemoryKey: ${{ secrets.GPG_SIGNING_KEY }}
+          ORG_GRADLE_PROJECT_signingInMemoryKeyPassword: ${{ secrets.GPG_SIGNING_PASSWORD }}
+        run: ./gradlew publishAndReleaseToMavenCentral --no-configuration-cache

참고/firecrawl-main/.github/workflows/publish-js-sdk.yml

@@ -0,0 +1,43 @@
+name: Publish JS SDK
+
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - apps/js-sdk/firecrawl/package.json
+
+env:
+  TEST_API_KEY: ${{ secrets.TEST_API_KEY }}
+
+jobs:
+  publish:
+    name: Publish
+    runs-on: blacksmith-2vcpu-ubuntu-2404
+    steps:
+      - uses: actions/checkout@v5
+      - name: Install pnpm
+        uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v4
+        with:
+          version: 10
+      - name: Set up Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: "20"
+          cache: "pnpm"
+          cache-dependency-path: './apps/js-sdk/firecrawl/pnpm-lock.yaml'
+      - name: Authenticate
+        run: echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > ~/.npmrc
+        env:
+          NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
+      - name: Publish
+        run: |
+          pnpm install
+          pnpm run build
+          pnpm publish --access public --no-git-checks
+          sed -i 's/"name": "@mendable\/firecrawl-js"/"name": "@mendable\/firecrawl"/g' package.json
+          pnpm publish --access public --no-git-checks
+          sed -i 's/"name": "@mendable\/firecrawl"/"name": "firecrawl"/g' package.json
+          sed -i '/"firecrawl":/d' package.json
+          pnpm publish --access public --no-git-checks
+        working-directory: ./apps/js-sdk/firecrawl

참고/firecrawl-main/.github/workflows/publish-php-sdk.yml

@@ -0,0 +1,70 @@
+name: Publish PHP SDK
+
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - main
+    paths:
+      - 'apps/php-sdk/**'
+
+jobs:
+  publish:
+    name: Publish to Packagist
+    runs-on: blacksmith-2vcpu-ubuntu-2404
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v5
+        with:
+          fetch-depth: 0
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.x'
+
+      - name: Install version check dependencies
+        run: pip install requests packaging
+
+      - name: Run version check script
+        id: version_check
+        run: |
+          VERSION_INCREMENTED=$(python .github/scripts/check_version_has_incremented.py php ./apps/php-sdk firecrawl/firecrawl-sdk)
+          echo "VERSION_INCREMENTED=$VERSION_INCREMENTED" >> $GITHUB_ENV
+
+      - name: Set up PHP
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        uses: shivammathur/setup-php@v2
+        with:
+          php-version: '8.3'
+          tools: composer:v2
+
+      - name: Install dependencies
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        working-directory: ./apps/php-sdk
+        run: composer install --no-dev --prefer-dist --no-interaction
+
+      - name: Validate composer.json
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        working-directory: ./apps/php-sdk
+        run: composer validate --strict
+
+      - name: Set up SSH for firecrawl-php
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        uses: webfactory/ssh-agent@v0.9.0
+        with:
+          ssh-private-key: ${{ secrets.PHP_SDK_DEPLOY_KEY }}
+
+      - name: Subtree split and push to firecrawl-php
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        run: |
+          git subtree split --prefix=apps/php-sdk -b php-sdk-split
+          git remote add php-sdk git@github.com:firecrawl/firecrawl-php.git
+          git push php-sdk php-sdk-split:main --force
+
+      - name: Notify Packagist
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        run: |
+          curl -sfS -X POST "https://packagist.org/api/update-package?username=${{ secrets.PACKAGIST_USERNAME }}&apiToken=${{ secrets.PACKAGIST_TOKEN }}" \
+            -H "Content-Type: application/json" \
+            -d '{"repository":{"url":"https://github.com/firecrawl/firecrawl-php"}}'

참고/firecrawl-main/.github/workflows/publish-python-sdk.yml

@@ -0,0 +1,51 @@
+name: Publish Python SDK 
+
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - 'apps/python-sdk/**'
+
+env:
+  PYPI_USERNAME: ${{ secrets.PYPI_USERNAME }}
+  PYPI_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
+
+jobs:
+  build-and-publish:
+    runs-on: blacksmith-2vcpu-ubuntu-2404
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v5
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.x'
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install setuptools wheel twine build requests packaging
+
+      - name: Run version check script
+        id: version_check_script
+        run: |
+          VERSION_INCREMENTED=$(python .github/scripts/check_version_has_incremented.py python ./apps/python-sdk/firecrawl firecrawl-py)
+          echo "VERSION_INCREMENTED=$VERSION_INCREMENTED" >> $GITHUB_ENV
+
+      - name: Build and publish to PyPI
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        env:
+          TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
+          TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
+        run: |
+          python -m build
+          twine upload dist/*
+          sed -i 's/name = "firecrawl-py"/name = "firecrawl"/g' pyproject.toml
+          sed -i 's/name="firecrawl-py"/name="firecrawl"/g' setup.py
+          python -m build
+          twine upload dist/*
+        working-directory: ./apps/python-sdk
+

참고/firecrawl-main/.github/workflows/publish-ruby-sdk.yml

@@ -0,0 +1,50 @@
+name: Publish Ruby SDK
+
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - main
+    paths:
+      - 'apps/ruby-sdk/**'
+
+jobs:
+  publish:
+    name: Publish to RubyGems
+    runs-on: blacksmith-2vcpu-ubuntu-2404
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v5
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.x'
+
+      - name: Install version check dependencies
+        run: pip install requests packaging
+
+      - name: Run version check script
+        id: version_check
+        run: |
+          VERSION_INCREMENTED=$(python .github/scripts/check_version_has_incremented.py ruby ./apps/ruby-sdk firecrawl-sdk)
+          echo "VERSION_INCREMENTED=$VERSION_INCREMENTED" >> $GITHUB_ENV
+
+      - name: Set up Ruby
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: '3.3'
+          working-directory: ./apps/ruby-sdk
+
+      - name: Build gem
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        working-directory: ./apps/ruby-sdk
+        run: gem build firecrawl-sdk.gemspec
+
+      - name: Publish to RubyGems
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        working-directory: ./apps/ruby-sdk
+        env:
+          GEM_HOST_API_KEY: ${{ secrets.RUBYGEMS_API_KEY }}
+        run: gem push firecrawl-*.gem

참고/firecrawl-main/.github/workflows/publish-rust-sdk.yml

@@ -0,0 +1,64 @@
+name: Publish Rust SDK
+
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - main
+    paths:
+      - 'apps/rust-sdk/**'
+
+jobs:
+  publish:
+    name: Publish to crates.io
+    runs-on: blacksmith-2vcpu-ubuntu-2404
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.x'
+
+      - name: Install version check dependencies
+        run: pip install requests packaging toml
+
+      - name: Run version check script
+        id: version_check
+        run: |
+          VERSION_INCREMENTED=$(python .github/scripts/check_version_has_incremented.py rust ./apps/rust-sdk firecrawl)
+          echo "VERSION_INCREMENTED=$VERSION_INCREMENTED" >> $GITHUB_ENV
+
+      - name: Install Rust toolchain
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        uses: dtolnay/rust-toolchain@631a55b12751854ce901bb631d5902ceb48146f7 # stable
+
+      - name: Cache cargo registry
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        uses: actions/cache@v4
+        with:
+          path: |
+            ~/.cargo/registry
+            ~/.cargo/git
+            apps/rust-sdk/target
+          key: ${{ runner.os }}-cargo-${{ hashFiles('apps/rust-sdk/Cargo.lock') }}
+          restore-keys: |
+            ${{ runner.os }}-cargo-
+
+      - name: Build
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        working-directory: ./apps/rust-sdk
+        run: cargo build --release
+
+      - name: Run tests
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        working-directory: ./apps/rust-sdk
+        run: cargo test --lib
+
+      - name: Publish to crates.io
+        if: ${{ env.VERSION_INCREMENTED == 'true' }}
+        working-directory: ./apps/rust-sdk
+        env:
+          CARGO_REGISTRY_TOKEN: ${{ secrets.CRATES_IO_TOKEN }}
+        run: cargo publish --allow-dirty

참고/firecrawl-main/.github/workflows/scrape-evals.yml

@@ -0,0 +1,131 @@
+name: Scrape Evals
+on:
+  issue_comment:
+    types: [created]
+  pull_request:
+    types: [opened]
+
+jobs:
+  quality-eval:
+    if: |
+      (github.event_name == 'issue_comment' &&
+       github.event.issue.pull_request != null &&
+       (contains(github.event.comment.body, '#scrape-quality-eval') ||
+        contains(github.event.issue.title, '#scrape-quality-eval'))) ||
+      (github.event_name == 'pull_request' &&
+       (contains(github.event.pull_request.body, '#scrape-quality-eval') ||
+        contains(github.event.pull_request.title, '#scrape-quality-eval')))
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check org membership
+        id: check-member
+        uses: actions/github-script@v7
+        with:
+          github-token: ${{ secrets.SCRAPE_EVALS_PAT }}
+          script: |
+            try {
+              await github.rest.orgs.checkMembershipForUser({
+                org: 'firecrawl',
+                username: context.actor,
+              });
+              return true;
+            } catch {
+              return false;
+            }
+
+      - name: Dispatch quality eval
+        if: steps.check-member.outputs.result == 'true'
+        uses: actions/github-script@v7
+        with:
+          github-token: ${{ secrets.SCRAPE_EVALS_PAT }}
+          script: |
+            let prNumber, prBranch, commitSha, headRepo;
+            if (context.eventName === 'pull_request') {
+              prNumber = context.payload.pull_request.number;
+              prBranch = context.payload.pull_request.head.ref;
+              commitSha = context.payload.pull_request.head.sha;
+              headRepo = context.payload.pull_request.head.repo.full_name;
+            } else {
+              const pr = await github.rest.pulls.get({
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                pull_number: context.issue.number,
+              });
+              prNumber = pr.data.number;
+              prBranch = pr.data.head.ref;
+              commitSha = pr.data.head.sha;
+              headRepo = pr.data.head.repo.full_name;
+            }
+            await github.rest.repos.createDispatchEvent({
+              owner: 'firecrawl',
+              repo: 'scrape-evals',
+              event_type: 'firecrawl-pr-eval',
+              client_payload: {
+                pr_number: prNumber,
+                pr_branch: prBranch,
+                commit_sha: commitSha,
+                repo: headRepo,
+              },
+            });
+
+  ocr-eval:
+    if: |
+      (github.event_name == 'issue_comment' &&
+       github.event.issue.pull_request != null &&
+       (contains(github.event.comment.body, '#scrape-ocr-eval') ||
+        contains(github.event.issue.title, '#scrape-ocr-eval'))) ||
+      (github.event_name == 'pull_request' &&
+       (contains(github.event.pull_request.body, '#scrape-ocr-eval') ||
+        contains(github.event.pull_request.title, '#scrape-ocr-eval')))
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check org membership
+        id: check-member
+        uses: actions/github-script@v7
+        with:
+          github-token: ${{ secrets.SCRAPE_EVALS_PAT }}
+          script: |
+            try {
+              await github.rest.orgs.checkMembershipForUser({
+                org: 'firecrawl',
+                username: context.actor,
+              });
+              return true;
+            } catch {
+              return false;
+            }
+
+      - name: Dispatch OCR eval
+        if: steps.check-member.outputs.result == 'true'
+        uses: actions/github-script@v7
+        with:
+          github-token: ${{ secrets.SCRAPE_EVALS_PAT }}
+          script: |
+            let prNumber, prBranch, commitSha, headRepo;
+            if (context.eventName === 'pull_request') {
+              prNumber = context.payload.pull_request.number;
+              prBranch = context.payload.pull_request.head.ref;
+              commitSha = context.payload.pull_request.head.sha;
+              headRepo = context.payload.pull_request.head.repo.full_name;
+            } else {
+              const pr = await github.rest.pulls.get({
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                pull_number: context.issue.number,
+              });
+              prNumber = pr.data.number;
+              prBranch = pr.data.head.ref;
+              commitSha = pr.data.head.sha;
+              headRepo = pr.data.head.repo.full_name;
+            }
+            await github.rest.repos.createDispatchEvent({
+              owner: 'firecrawl',
+              repo: 'scrape-evals',
+              event_type: 'firecrawl-ocr-eval',
+              client_payload: {
+                pr_number: prNumber,
+                pr_branch: prBranch,
+                commit_sha: commitSha,
+                repo: headRepo,
+              },
+            });

참고/firecrawl-main/.github/workflows/test-dotnet-sdk.yml

@@ -0,0 +1,52 @@
+name: .NET SDK Test Suite
+
+on:
+  pull_request:
+    branches:
+      - main
+    paths:
+      - apps/dot-net-sdk/**
+      - .github/workflows/test-dotnet-sdk.yml
+  push:
+    branches:
+      - main
+    paths:
+      - apps/dot-net-sdk/**
+  workflow_dispatch:
+
+jobs:
+  build-and-test:
+    name: Build and Test
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+    if: >-
+      github.event_name == 'workflow_dispatch' ||
+      github.event_name == 'pull_request' ||
+      (github.event_name == 'push' && github.ref == 'refs/heads/main')
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up .NET 8
+        uses: actions/setup-dotnet@v4
+        with:
+          dotnet-version: "8.0.x"
+
+      - name: Restore dependencies
+        working-directory: ./apps/dot-net-sdk
+        run: dotnet restore
+
+      - name: Build
+        working-directory: ./apps/dot-net-sdk
+        run: dotnet build --no-restore --configuration Release
+
+      - name: Run unit tests
+        working-directory: ./apps/dot-net-sdk
+        run: dotnet test --no-build --configuration Release --logger "trx;LogFileName=test-results.trx" --results-directory ./TestResults
+
+      - name: Publish test report
+        if: always() && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.fork == false)
+        uses: dorny/test-reporter@3eeb9fc888e82e8be2fb356bbeec2750231672bc # v1
+        with:
+          name: .NET SDK Test Report
+          path: apps/dot-net-sdk/TestResults/**/*.trx
+          reporter: dotnet-trx
+          fail-on-error: true

참고/firecrawl-main/.github/workflows/test-go-html-to-md-service.yml

@@ -0,0 +1,44 @@
+name: Go HTML-to-Markdown Service Check
+
+on:
+  pull_request:
+    branches:
+      - main
+    paths:
+      - apps/go-html-to-md-service/**
+      - .github/workflows/test-go-html-to-md-service.yml
+  push:
+    branches:
+      - main
+    paths:
+      - apps/go-html-to-md-service/**
+  workflow_dispatch:
+
+jobs:
+  build-vet-test:
+    name: Build, Vet and Test
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+    if: >-
+      github.event_name == 'workflow_dispatch' ||
+      github.event_name == 'pull_request' ||
+      (github.event_name == 'push' && github.ref == 'refs/heads/main')
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: "1.23"
+          cache-dependency-path: apps/go-html-to-md-service/go.sum
+
+      - name: Build
+        working-directory: ./apps/go-html-to-md-service
+        run: go build ./...
+
+      - name: Vet
+        working-directory: ./apps/go-html-to-md-service
+        run: go vet ./...
+
+      - name: Test
+        working-directory: ./apps/go-html-to-md-service
+        run: go test ./...

참고/firecrawl-main/.github/workflows/test-go-sdk.yml

@@ -0,0 +1,40 @@
+name: Go SDK Check
+
+on:
+  pull_request:
+    branches:
+      - main
+    paths:
+      - apps/go-sdk/**
+      - .github/workflows/test-go-sdk.yml
+  push:
+    branches:
+      - main
+    paths:
+      - apps/go-sdk/**
+  workflow_dispatch:
+
+jobs:
+  build-and-vet:
+    name: Build and Vet
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+    if: >-
+      github.event_name == 'workflow_dispatch' ||
+      github.event_name == 'pull_request' ||
+      (github.event_name == 'push' && github.ref == 'refs/heads/main')
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: "1.23"
+          cache-dependency-path: apps/go-sdk/go.sum
+
+      - name: Build
+        working-directory: ./apps/go-sdk
+        run: go build ./...
+
+      - name: Vet
+        working-directory: ./apps/go-sdk
+        run: go vet ./...

참고/firecrawl-main/.github/workflows/test-java-sdk.yml

@@ -0,0 +1,70 @@
+name: Java SDK Test Suite
+
+on:
+  pull_request:
+    branches:
+      - main
+    paths:
+      - apps/java-sdk/**
+      - .github/workflows/test-java-sdk.yml
+  push:
+    branches:
+      - main
+    paths:
+      - apps/java-sdk/**
+  workflow_dispatch:
+
+jobs:
+  build-and-test:
+    name: Build and Test
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+    if: >-
+      github.event_name == 'workflow_dispatch' ||
+      github.event_name == 'pull_request' ||
+      (github.event_name == 'push' && github.ref == 'refs/heads/main')
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up JDK 11
+        uses: actions/setup-java@v4
+        with:
+          distribution: temurin
+          java-version: "11"
+
+      - name: Cache Gradle packages
+        uses: actions/cache@v4
+        with:
+          path: |
+            ~/.gradle/caches
+            ~/.gradle/wrapper
+          key: ${{ runner.os }}-gradle-${{ hashFiles('apps/java-sdk/**/*.gradle.kts', 'apps/java-sdk/gradle/wrapper/gradle-wrapper.properties') }}
+          restore-keys: |
+            ${{ runner.os }}-gradle-
+
+      - name: Grant execute permission for gradlew
+        working-directory: ./apps/java-sdk
+        run: chmod +x gradlew
+
+      - name: Build
+        working-directory: ./apps/java-sdk
+        run: ./gradlew build -x test
+
+      - name: Run unit tests
+        working-directory: ./apps/java-sdk
+        run: ./gradlew test
+
+      - name: Run E2E tests
+        if: env.FIRECRAWL_API_KEY != ''
+        working-directory: ./apps/java-sdk
+        env:
+          FIRECRAWL_API_KEY: ${{ secrets.FIRECRAWL_API_KEY }}
+        run: ./gradlew test
+
+      - name: Publish test report
+        if: always() && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.fork == false)
+        uses: dorny/test-reporter@3eeb9fc888e82e8be2fb356bbeec2750231672bc # v1
+        with:
+          name: Java SDK Test Report
+          path: apps/java-sdk/build/test-results/test/*.xml
+          reporter: java-junit
+          fail-on-error: true

참고/firecrawl-main/.github/workflows/test-js-sdk.yml

@@ -0,0 +1,44 @@
+name: JS SDK Test Suite
+
+on:
+  pull_request:
+    branches:
+      - main
+    paths:
+      - apps/js-sdk/firecrawl/**
+
+env:
+  IDMUX_URL: ${{ secrets.IDMUX_URL }}
+
+jobs:
+  test:
+    name: Run tests
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+    steps:
+      - uses: actions/checkout@v5
+      - name: Install pnpm
+        uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v4
+        with:
+          version: 10
+      - name: Set up Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: "20"
+          cache: "pnpm"
+          cache-dependency-path: './apps/js-sdk/firecrawl/pnpm-lock.yaml'
+      - name: Tailscale
+        uses: tailscale/github-action@306e68a486fd2350f2bfc3b19fcd143891a4a2d8 # v4
+        with:
+          oauth-client-id: ${{ secrets.TS_OAUTH_CLIENT_ID }}
+          oauth-secret: ${{ secrets.TS_OAUTH_SECRET }}
+          tags: tag:ci
+          use-cache: 'true'
+      - name: Install dependencies
+        run: pnpm install
+        working-directory: ./apps/js-sdk/firecrawl
+      - name: Build
+        run: pnpm run build
+        working-directory: ./apps/js-sdk/firecrawl
+      - name: Run tests
+        run: pnpm run test
+        working-directory: ./apps/js-sdk/firecrawl

참고/firecrawl-main/.github/workflows/test-php-sdk.yml

@@ -0,0 +1,71 @@
+name: PHP SDK Test Suite
+
+on:
+  pull_request:
+    branches:
+      - main
+    paths:
+      - apps/php-sdk/**
+      - .github/workflows/test-php-sdk.yml
+  push:
+    branches:
+      - main
+    paths:
+      - apps/php-sdk/**
+  workflow_dispatch:
+
+jobs:
+  build-and-test:
+    name: Build and Test
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+    env:
+      HAS_API_KEY: ${{ secrets.FIRECRAWL_API_KEY != '' }}
+    if: >-
+      github.event_name == 'workflow_dispatch' ||
+      github.event_name == 'pull_request' ||
+      (github.event_name == 'push' && github.ref == 'refs/heads/main')
+    steps:
+      - uses: actions/checkout@v5
+
+      - name: Set up PHP
+        uses: shivammathur/setup-php@v2
+        with:
+          php-version: '8.3'
+          tools: composer:v2
+          coverage: none
+
+      - name: Get Composer cache directory
+        id: composer-cache
+        working-directory: ./apps/php-sdk
+        run: echo "dir=$(composer config cache-files-dir)" >> $GITHUB_OUTPUT
+
+      - name: Cache Composer dependencies
+        uses: actions/cache@v4
+        with:
+          path: ${{ steps.composer-cache.outputs.dir }}
+          key: ${{ runner.os }}-composer-${{ hashFiles('apps/php-sdk/composer.json') }}
+          restore-keys: |
+            ${{ runner.os }}-composer-
+
+      - name: Install dependencies
+        working-directory: ./apps/php-sdk
+        run: composer install --prefer-dist --no-interaction
+
+      - name: Validate composer.json
+        working-directory: ./apps/php-sdk
+        run: composer validate --strict
+
+      - name: Run PHPStan
+        working-directory: ./apps/php-sdk
+        run: vendor/bin/phpstan analyse --no-progress
+
+      - name: Run unit tests
+        working-directory: ./apps/php-sdk
+        run: vendor/bin/pest --ci
+
+      - name: Run E2E tests
+        if: env.HAS_API_KEY == 'true'
+        working-directory: ./apps/php-sdk
+        env:
+          FIRECRAWL_API_KEY: ${{ secrets.FIRECRAWL_API_KEY }}
+        run: vendor/bin/pest --ci --group=e2e

참고/firecrawl-main/.github/workflows/test-ruby-sdk.yml

@@ -0,0 +1,41 @@
+name: Ruby SDK Test Suite
+
+on:
+  pull_request:
+    branches:
+      - main
+    paths:
+      - apps/ruby-sdk/**
+      - .github/workflows/test-ruby-sdk.yml
+  push:
+    branches:
+      - main
+    paths:
+      - apps/ruby-sdk/**
+  workflow_dispatch:
+
+jobs:
+  build-and-test:
+    name: Build and Test
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+    if: >-
+      github.event_name == 'workflow_dispatch' ||
+      github.event_name == 'pull_request' ||
+      (github.event_name == 'push' && github.ref == 'refs/heads/main')
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Ruby
+        uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: "3.3"
+          bundler-cache: true
+          working-directory: ./apps/ruby-sdk
+
+      - name: Install dependencies
+        working-directory: ./apps/ruby-sdk
+        run: bundle install
+
+      - name: Run unit tests
+        working-directory: ./apps/ruby-sdk
+        run: bundle exec rake test

참고/firecrawl-main/.github/workflows/test-rust-sdk.yml

@@ -0,0 +1,51 @@
+name: Rust SDK Test Suite
+
+on:
+  pull_request:
+    branches:
+      - main
+    paths:
+      - apps/rust-sdk/**
+
+jobs:
+  test:
+    name: Build and Test
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install Rust toolchain
+        uses: dtolnay/rust-toolchain@631a55b12751854ce901bb631d5902ceb48146f7 # stable
+        with:
+          components: clippy
+
+      - name: Cache cargo registry
+        uses: actions/cache@v4
+        with:
+          path: |
+            ~/.cargo/registry
+            ~/.cargo/git
+            apps/rust-sdk/target
+          key: ${{ runner.os }}-cargo-${{ hashFiles('apps/rust-sdk/Cargo.lock') }}
+          restore-keys: |
+            ${{ runner.os }}-cargo-
+
+      - name: Check formatting
+        working-directory: ./apps/rust-sdk
+        run: cargo fmt --check
+
+      - name: Build
+        working-directory: ./apps/rust-sdk
+        run: cargo build --all-targets
+
+      - name: Run clippy
+        working-directory: ./apps/rust-sdk
+        run: cargo clippy --all-targets -- -D warnings -A clippy::needless_borrows_for_generic_args -A clippy::redundant_closure -A clippy::large_enum_variant -A clippy::extra_unused_lifetimes
+
+      - name: Run unit tests (mocked)
+        working-directory: ./apps/rust-sdk
+        run: cargo test --lib
+
+      - name: Build examples
+        working-directory: ./apps/rust-sdk
+        run: cargo build --examples

참고/firecrawl-main/.github/workflows/test-server.yml

@@ -0,0 +1,538 @@
+name: Server Test Suite
+
+on:
+  pull_request:
+    branches:
+      - main
+
+concurrency:
+  group: ci=${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  self-host:
+    name: Self-hosted environment tests
+    strategy:
+      fail-fast: false
+      matrix:
+        engine: ['playwright', 'fetch'] # unsure if we need both of these
+        proxy: ['proxy', 'no-proxy'] # proxy / no-proxy run different tests, keep both
+        search: ['searxng'] # disabled google for now, should be fine with just Searxng
+        ai: ['openai'] # AI only should be fine, as it simply adds tests, if non-AI fails, AI will fail.
+
+        # legacy matrix:
+        # engine: ["playwright", "fetch"]
+        # proxy: ["proxy", "no-proxy"]
+        # search: ["searxng", "google"]
+        # ai: ["openai", "no-ai"]
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+    services:
+      redis:
+        image: redis
+        ports:
+          - 6379:6379
+      rabbitmq:
+        image: rabbitmq
+        ports:
+          - 5672:5672
+    env:
+      HOST: 0.0.0.0
+      TEST_SUITE_SELF_HOSTED: true
+      TEST_SUITE_WEBSITE: http://127.0.0.1:4321
+      OPENAI_API_KEY: ${{ matrix.ai == 'openai' && secrets.OPENAI_API_KEY || '' }}
+      GOOGLE_GENERATIVE_AI_API_KEY: ${{ secrets.GOOGLE_GENERATIVE_AI_API_KEY }}
+      SEARXNG_ENDPOINT: ${{ matrix.search == 'searxng' && 'http://localhost:3434' || '' }}
+      PLAYWRIGHT_MICROSERVICE_URL: ${{ matrix.engine == 'playwright' && 'http://localhost:3003/scrape' || '' }}
+      PROXY_SERVER: ${{ matrix.proxy == 'proxy' && secrets.PROXY_SERVER || '' }}
+      PROXY_USERNAME: ${{ matrix.proxy == 'proxy' && secrets.PROXY_USERNAME || '' }}
+      PROXY_PASSWORD: ${{ matrix.proxy == 'proxy' && secrets.PROXY_PASSWORD || '' }}
+      NUQ_DATABASE_URL: postgres://postgres:postgres@localhost:5432/postgres
+      NUQ_RABBITMQ_URL: amqp://localhost:5672
+      USE_GO_MARKDOWN_PARSER: true
+      ALLOW_LOCAL_WEBHOOKS: true
+      FIRECRAWL_LOG_TO_FILE: true
+    steps:
+      - uses: actions/checkout@v5
+
+      - uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v4
+        with:
+          version: 10
+
+      - run: pnpm config set store-dir ~/.pnpm-store
+      - run: mkdir -p ~/.pnpm-store
+
+      - uses: actions/setup-node@v6
+        with:
+          node-version: 22
+          cache: pnpm
+          cache-dependency-path: |
+            apps/api/pnpm-lock.yaml
+            apps/test-site/pnpm-lock.yaml
+            apps/playwright-service-ts/pnpm-lock.yaml
+
+      - run: pnpm fetch
+        working-directory: apps/api
+      - run: pnpm fetch
+        working-directory: apps/test-site
+      - run: pnpm fetch
+        if: matrix.engine == 'playwright'
+        working-directory: apps/playwright-service-ts
+
+      - name: Restore native lib
+        id: napi_restore
+        uses: actions/cache/restore@v4
+        with:
+          path: |
+            apps/api/native/*.node
+            apps/api/native/index.js
+            apps/api/native/index.d.ts
+          # note: this key is not ideal, need to find a better solution
+          key: ${{ runner.os }}-napi-${{ hashFiles('apps/api/native/Cargo.toml', 'apps/api/native/package.json', 'apps/api/native/src/**') }}
+
+      - name: Build native lib
+        if: steps.napi_restore.outputs.cache-hit != 'true'
+        run: pnpm install
+        working-directory: apps/api/native
+
+      - name: Cache native lib
+        if: steps.napi_restore.outputs.cache-hit != 'true'
+        uses: actions/cache/save@v4
+        with:
+          path: |
+            apps/api/native/*.node
+            apps/api/native/index.js
+            apps/api/native/index.d.ts
+          key: ${{ steps.napi_restore.outputs.cache-primary-key }}
+
+      - uses: actions/setup-go@v6
+        with:
+          go-version: 1.24
+          cache-dependency-path: apps/api/sharedLibs/go-html-to-md/go.sum
+
+      - name: Verify go.sum is up to date (sharedLibs/go-html-to-md)
+        run: |
+          cd apps/api/sharedLibs/go-html-to-md
+          go mod tidy
+          git diff --exit-code go.mod go.sum || (echo "go.mod/go.sum is out of sync -- run 'go mod tidy' and commit" && exit 1)
+
+      - name: Verify go.sum is up to date (go-html-to-md-service)
+        run: |
+          cd apps/go-html-to-md-service
+          go mod tidy
+          git diff --exit-code go.mod go.sum || (echo "go.mod/go.sum is out of sync -- run 'go mod tidy' and commit" && exit 1)
+
+      - name: Restore Go lib
+        id: golib_restore
+        uses: actions/cache/restore@v4
+        with:
+          path: apps/api/sharedLibs/go-html-to-md/libhtml-to-markdown.so
+          key: ${{ runner.os }}-golib-${{ hashFiles('apps/api/sharedLibs/go-html-to-md/go.sum', 'apps/api/sharedLibs/go-html-to-md/*.go') }}
+
+      - name: Build go-html-to-md
+        if: steps.golib_restore.outputs.cache-hit != 'true'
+        run: |
+          cd apps/api/sharedLibs/go-html-to-md
+          go build -o libhtml-to-markdown.so -buildmode=c-shared html-to-markdown.go
+
+      - name: Cache Go lib
+        if: steps.golib_restore.outputs.cache-hit != 'true'
+        uses: actions/cache/save@v4
+        with:
+          path: apps/api/sharedLibs/go-html-to-md/libhtml-to-markdown.so
+          key: ${{ steps.golib_restore.outputs.cache-primary-key }}
+
+      - name: Restore Playwright cache
+        if: matrix.engine == 'playwright'
+        id: pw_cache
+        uses: actions/cache/restore@v4
+        with:
+          path: ~/.cache/ms-playwright
+          key: ${{ runner.os }}-${{ runner.arch }}-pw-chromium-${{ hashFiles('apps/playwright-service-ts/pnpm-lock.yaml', 'apps/playwright-service-ts/package.json') }}
+          restore-keys: |
+            ${{ runner.os }}-${{ runner.arch }}-pw-chromium-
+
+      - name: Install Playwright node modules
+        if: matrix.engine == 'playwright'
+        run: pnpm install --frozen-lockfile
+        working-directory: ./apps/playwright-service-ts
+
+      - name: Install Playwright system deps
+        if: matrix.engine == 'playwright'
+        run: pnpm exec playwright install-deps chromium
+        working-directory: ./apps/playwright-service-ts
+
+      - name: Install Playwright Chromium
+        if: matrix.engine == 'playwright' && steps.pw_cache.outputs.cache-hit != 'true'
+        run: pnpm exec playwright install chromium
+        working-directory: ./apps/playwright-service-ts
+
+      - name: Cache Playwright
+        if: matrix.engine == 'playwright' && steps.pw_cache.outputs.cache-hit != 'true'
+        uses: actions/cache/save@v4
+        with:
+          path: ~/.cache/ms-playwright
+          key: ${{ steps.pw_cache.outputs.cache-primary-key }}
+
+      - name: Set up SearXNG
+        if: matrix.search == 'searxng'
+        run: |
+          mkdir searxng
+
+          echo "use_default_settings: true
+          search:
+            formats: [html, json, csv]
+          server:
+            secret_key: 'fcsecret'" > searxng/settings.yml
+
+          docker run -d -p 3434:8080 -v "${PWD}/searxng:/etc/searxng" --name searxng searxng/searxng
+          pnpx wait-on tcp:3434 -t 30s
+        working-directory: ./
+
+      - name: Install API dependencies
+        run: pnpm install --frozen-lockfile --ignore-scripts
+        working-directory: apps/api
+        env:
+          npm_config_ignore_scripts: 'true'
+
+      - name: Install test site dependencies
+        run: pnpm install --frozen-lockfile
+        working-directory: apps/test-site
+
+      - name: Build + serve test site
+        run: |
+          pnpm build
+          pnpm preview --port 4321 --strictPort --host 127.0.0.1 > test-site.log 2>&1 &
+          pnpx wait-on tcp:4321 -t 20s
+        working-directory: apps/test-site
+
+      - name: Start playwright
+        if: matrix.engine == 'playwright'
+        run: |
+          pnpm run dev > playwright.log 2>&1 &
+          pnpx wait-on tcp:3003 -t 15s
+        working-directory: ./apps/playwright-service-ts
+        env:
+          PORT: 3003
+
+      - name: Run Docker Postgres
+        run: |
+          docker build -t firecrawl/nuq-postgres:latest ./apps/nuq-postgres
+          docker run -d -p 5432:5432 -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=postgres --name postgres firecrawl/nuq-postgres:latest
+
+      - name: Run tests
+        run: pnpm harness pnpm test:snips
+        working-directory: apps/api
+        env:
+          npm_config_ignore_scripts: 'true' # required currently to prevent re-building cached native lib
+
+      - name: Publish test report
+        if: always()
+        shell: python3 {0}
+        env:
+          JUNIT: apps/api/test-results/junit.xml
+          MATRIX_LABEL: "${{ matrix.engine }}, ${{ matrix.proxy }}, ${{ matrix.search }}, ${{ matrix.ai }}"
+        run: |
+          import os, sys, xml.etree.ElementTree as ET
+
+          junit = os.environ["JUNIT"]
+          label = os.environ["MATRIX_LABEL"]
+          summary = os.environ["GITHUB_STEP_SUMMARY"]
+
+          if not os.path.isfile(junit):
+              print(f"::warning::No junit.xml found — tests may have crashed before producing a report.")
+              with open(summary, "a") as f:
+                  f.write(f"## ⚠️ No test report\njunit.xml was not generated. Tests likely crashed before completing.\n")
+              sys.exit(0)
+
+          try:
+              tree = ET.parse(junit)
+              root = tree.getroot()
+          except ET.ParseError as e:
+              print(f"::warning::junit.xml is malformed: {e}")
+              with open(summary, "a") as f:
+                  f.write(f"## ⚠️ Malformed test report\njunit.xml could not be parsed (tests may have crashed mid-run): `{e}`\n")
+              sys.exit(1)
+
+          # Get total time from root element
+          top = root if root.tag == "testsuites" else root
+          time_s = top.get("time", "?")
+
+          failed = []
+          passed = []
+          skipped = []
+          for tc in tree.iter("testcase"):
+              name = tc.get("name", "unknown")
+              suite = tc.get("classname", "")
+              t = tc.get("time", "?")
+              if tc.find("skipped") is not None:
+                  skipped.append((suite, name, t))
+                  continue
+              fail = tc.find("failure")
+              if fail is None:
+                  fail = tc.find("error")
+              if fail is not None:
+                  msg = (fail.get("message") or fail.text or "")[:200].replace("\n", " ")
+                  failed.append((suite, name, t, msg))
+              else:
+                  passed.append((suite, name, t))
+
+          # Derive counts from parsed testcases (root attributes can be unreliable)
+          tests = len(failed) + len(passed) + len(skipped)
+          failures = len(failed)
+
+          def esc(s):
+              return s.replace("|", "\\|")
+
+          def log(line):
+              print(line)
+
+          def md(line, f):
+              f.write(line + "\n")
+
+          # Print plain text to step log (failures at the end so they're visible on auto-scroll)
+          log(f"Test Report ({label}): {tests} tests, {failures} failures, {time_s}s")
+          log("")
+          log("All tests:")
+          for suite, name, t in passed:
+              log(f"  ✅ PASS  {suite} > {name} ({t}s)")
+          for suite, name, t in skipped:
+              log(f"  ⏭️ SKIP  {suite} > {name} ({t}s)")
+          for suite, name, t, _msg in failed:
+              log(f"  ❌ FAIL  {suite} > {name} ({t}s)")
+          if failed:
+              log("")
+              log("Failed tests:")
+              for suite, name, t, msg in failed:
+                  log(f"  ❌ {suite} > {name} ({t}s)")
+                  if msg:
+                      log(f"     {msg}")
+
+          # Write markdown tables to step summary
+          with open(summary, "a") as f:
+              status = "❌" if failures > 0 else "✅"
+              md(f"## {status} Test Report ({label})", f)
+              md("| 🧪 Tests | ❌ Failures | ⏭️ Skipped | ⏱️ Time |", f)
+              md("|----------|------------|-----------|---------|", f)
+              md(f"| {tests} | {failures} | {len(skipped)} | {time_s}s |", f)
+              md("", f)
+
+              if failed:
+                  md("### ❌ Failed tests", f)
+                  md("", f)
+                  md("| Suite | Test | Time | Message |", f)
+                  md("|-------|------|------|---------|", f)
+                  for suite, name, t, msg in failed:
+                      md(f"| `{esc(suite)}` | **{esc(name)}** | {t}s | {esc(msg)} |", f)
+                  md("", f)
+
+              md("<details>", f)
+              md(f"<summary>📋 All tests ({len(passed)} passed, {len(failed)} failed, {len(skipped)} skipped)</summary>", f)
+              md("", f)
+              md("| Status | Suite | Test | Time |", f)
+              md("|--------|-------|------|------|", f)
+              for suite, name, t, _msg in failed:
+                  md(f"| ❌ | `{esc(suite)}` | {esc(name)} | {t}s |", f)
+              for suite, name, t in passed:
+                  md(f"| ✅ | `{esc(suite)}` | {esc(name)} | {t}s |", f)
+              for suite, name, t in skipped:
+                  md(f"| ⏭️ | `{esc(suite)}` | {esc(name)} | {t}s |", f)
+              md("", f)
+              md("</details>", f)
+
+          if failures > 0:
+              print(f"::error::{failures} failure(s)")
+              sys.exit(1)
+
+      - name: Copy log files
+        if: always()
+        run: |
+          mkdir -p logs
+          cp ./apps/api/firecrawl-app.log logs/firecrawl.log || true
+          cp ./apps/api/firecrawl-worker.log logs/firecrawl-worker.log || true
+
+      - name: Copy SearXNG logs
+        if: always() && matrix.search == 'searxng'
+        run: docker logs searxng > logs/searxng.log && docker kill searxng
+
+      - name: Copy Playwright logs
+        if: always() && matrix.engine == 'playwright'
+        run: cp ./apps/playwright-service-ts/playwright.log logs/playwright.log
+
+      - name: Zip logs
+        if: always()
+        run: |
+          cd logs
+          zip -r logs.zip ./*
+
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: Logs (kubernetes, ${{ matrix.ai }}, ${{ matrix.search }}, ${{ matrix.engine }}, ${{ matrix.proxy }})
+          path: logs/logs.zip
+
+  # temp disabled
+  # prod-test:
+  #   name: Production environment tests
+  #   runs-on: big-runner
+  #   services:
+  #     redis:
+  #       image: redis
+  #       ports:
+  #         - 6379:6379
+  #     rabbitmq:
+  #       image: rabbitmq
+  #       ports:
+  #         - 5672:5672
+  #   env:
+  #     BULL_AUTH_KEY: ${{ secrets.BULL_AUTH_KEY }}
+  #     OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+  #     REDIS_URL: ${{ secrets.REDIS_URL }}
+  #     SUPABASE_ANON_TOKEN: ${{ secrets.SUPABASE_ANON_TOKEN }}
+  #     SUPABASE_SERVICE_TOKEN: ${{ secrets.SUPABASE_SERVICE_TOKEN }}
+  #     SUPABASE_URL: ${{ secrets.SUPABASE_URL }}
+  #     SUPABASE_REPLICA_URL: ${{ secrets.SUPABASE_REPLICA_URL }}
+  #     INDEX_SUPABASE_SERVICE_TOKEN: ${{ secrets.INDEX_SUPABASE_SERVICE_TOKEN }}
+  #     INDEX_SUPABASE_ANON_TOKEN: ${{ secrets.INDEX_SUPABASE_ANON_TOKEN }}
+  #     INDEX_SUPABASE_URL: ${{ secrets.INDEX_SUPABASE_URL }}
+  #     TEST_API_KEY: ${{ secrets.TEST_API_KEY }}
+  #     TEST_TEAM_ID: ${{ secrets.TEST_TEAM_ID }}
+  #     TEST_API_KEY_CONCURRENCY: ${{ secrets.TEST_API_KEY_CONCURRENCY }}
+  #     TEST_TEAM_ID_CONCURRENCY: ${{ secrets.TEST_TEAM_ID_CONCURRENCY }}
+  #     TEST_API_KEY_ZDR: ${{ secrets.TEST_API_KEY_ZDR }}
+  #     TEST_TEAM_ID_ZDR: ${{ secrets.TEST_TEAM_ID_ZDR }}
+  #     FIRE_ENGINE_BETA_URL: ${{ secrets.FIRE_ENGINE_BETA_URL }}
+  #     FIRE_ENGINE_STAGING_URL: ${{ secrets.FIRE_ENGINE_STAGING_URL }}
+  #     USE_DB_AUTHENTICATION: true
+  #     ENV: ${{ secrets.ENV }}
+  #     RUNPOD_MU_POD_ID: ${{ secrets.RUNPOD_MU_POD_ID }}
+  #     RUNPOD_MUV2_POD_ID: ${{ secrets.RUNPOD_MUV2_POD_ID }}
+  #     RUNPOD_MU_API_KEY: ${{ secrets.RUNPOD_MU_API_KEY }}
+  #     GCS_CREDENTIALS: ${{ secrets.GCS_CREDENTIALS }}
+  #     GCS_BUCKET_NAME: ${{ secrets.GCS_BUCKET_NAME }}
+  #     GCS_INDEX_BUCKET_NAME: ${{ secrets.GCS_INDEX_BUCKET_NAME }}
+  #     GCS_MEDIA_BUCKET_NAME: ${{ secrets.GCS_MEDIA_BUCKET_NAME }}
+  #     GOOGLE_GENERATIVE_AI_API_KEY: ${{ secrets.GOOGLE_GENERATIVE_AI_API_KEY }}
+  #     GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}
+  #     ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+  #     VERTEX_CREDENTIALS: ${{ secrets.VERTEX_CREDENTIALS }}
+  #     USE_GO_MARKDOWN_PARSER: true
+  #     SENTRY_ENVIRONMENT: dev
+  #     IDMUX_URL: ${{ secrets.IDMUX_URL }}
+  #     LOG_ENCRYPTION_KEY: ${{ secrets.LOG_ENCRYPTION_KEY }}
+  #     TEST_SUITE_WEBSITE: ${{ secrets.TEST_SUITE_WEBSITE }}
+  #     NUQ_DATABASE_URL: postgres://postgres:postgres@localhost:5432/postgres
+  #     NUQ_RABBITMQ_URL: amqp://localhost:5672
+  #     HOST: 0.0.0.0
+  #   steps:
+  #     - uses: actions/checkout@v5
+
+  #     - name: Tailscale
+  #       uses: tailscale/github-action@v4
+  #       with:
+  #         oauth-client-id: ${{ secrets.TS_OAUTH_CLIENT_ID }}
+  #         oauth-secret: ${{ secrets.TS_OAUTH_SECRET }}
+  #         tags: tag:ci
+  #         use-cache: 'true'
+
+  #     - uses: pnpm/action-setup@v4
+  #       with:
+  #         version: 10
+
+  #     - run: pnpm config set store-dir ~/.pnpm-store
+  #     - run: mkdir -p ~/.pnpm-store
+
+  #     - uses: actions/setup-node@v6
+  #       with:
+  #         node-version: 22
+  #         cache: pnpm
+  #         cache-dependency-path: |
+  #           apps/api/pnpm-lock.yaml
+
+  #     - run: pnpm fetch
+  #       working-directory: apps/api
+
+  #     - name: Restore native lib
+  #       id: napi_restore
+  #       uses: actions/cache/restore@v4
+  #       with:
+  #         path: |
+  #           apps/api/native/*.node
+  #           apps/api/native/index.js
+  #           apps/api/native/index.d.ts
+  #         # note: this key is not ideal, need to find a better solution
+  #         key: ${{ runner.os }}-napi-${{ hashFiles('apps/api/native/Cargo.toml', 'apps/api/native/package.json', 'apps/api/native/src/**') }}
+
+  #     - name: Build native lib
+  #       if: steps.napi_restore.outputs.cache-hit != 'true'
+  #       run: pnpm install
+  #       working-directory: apps/api/native
+
+  #     - name: Cache native lib
+  #       if: steps.napi_restore.outputs.cache-hit != 'true'
+  #       uses: actions/cache/save@v4
+  #       with:
+  #         path: |
+  #           apps/api/native/*.node
+  #           apps/api/native/index.js
+  #           apps/api/native/index.d.ts
+  #         key: ${{ steps.napi_restore.outputs.cache-primary-key }}
+
+  #     - uses: actions/setup-go@v6
+  #       with:
+  #         go-version: 1.24
+  #         cache-dependency-path: apps/api/sharedLibs/go-html-to-md/go.sum
+
+  #     - name: Restore Go lib
+  #       id: golib_restore
+  #       uses: actions/cache/restore@v4
+  #       with:
+  #         path: apps/api/sharedLibs/go-html-to-md/libhtml-to-markdown.so
+  #         key: ${{ runner.os }}-golib-${{ hashFiles('apps/api/sharedLibs/go-html-to-md/go.sum', 'apps/api/sharedLibs/go-html-to-md/*.go') }}
+
+  #     - name: Build go-html-to-md
+  #       if: steps.golib_restore.outputs.cache-hit != 'true'
+  #       run: |
+  #         cd apps/api/sharedLibs/go-html-to-md
+  #         go mod tidy
+  #         go build -o libhtml-to-markdown.so -buildmode=c-shared html-to-markdown.go
+
+  #     - name: Cache Go lib
+  #       if: steps.golib_restore.outputs.cache-hit != 'true'
+  #       uses: actions/cache/save@v4
+  #       with:
+  #         path: apps/api/sharedLibs/go-html-to-md/libhtml-to-markdown.so
+  #         key: ${{ steps.golib_restore.outputs.cache-primary-key }}
+
+  #     - name: Install API dependencies
+  #       run: pnpm install --frozen-lockfile --ignore-scripts
+  #       working-directory: apps/api
+  #       env:
+  #         npm_config_ignore_scripts: "true"
+
+  #     - name: Run Docker Postgres
+  #       run: |
+  #         docker build -t firecrawl/nuq-postgres:latest ./apps/nuq-postgres
+  #         docker run -d -p 5432:5432 -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=postgres --name postgres firecrawl/nuq-postgres:latest
+
+  #     - name: Run tests
+  #       run: pnpm harness pnpm test:snips
+  #       working-directory: apps/api
+  #       env:
+  #         npm_config_ignore_scripts: "true" # required currently to prevent re-building cached native lib
+
+  #     - name: Create logs directory
+  #       if: always()
+  #       run: |
+  #         mkdir -p logs
+  #         cp ./apps/api/firecrawl.log logs/firecrawl.log
+  #         cd logs
+  #         zip -r logs.zip ./*
+  #         echo "${{ secrets.LOG_ENCRYPTION_KEY }}" | gpg --batch --yes --passphrase-fd 0 -c logs.zip
+  #         rm logs.zip
+
+  #     - uses: actions/upload-artifact@v4
+  #       if: always()
+  #       with:
+  #         name: Encrypted Logs
+  #         path: logs/logs.zip.gpg
+  #         retention-days: 5

참고/firecrawl-main/.github/workflows/validate-lockfiles.yml

@@ -0,0 +1,23 @@
+name: Validate Lockfiles
+
+on:
+  pull_request:
+    branches:
+      - main
+    paths:
+      - "apps/test-suite/package.json"
+      - "apps/test-suite/pnpm-lock.yaml"
+
+jobs:
+  validate:
+    runs-on: blacksmith-2vcpu-ubuntu-2404
+    steps:
+      - uses: actions/checkout@v5
+      - name: Install pnpm
+        uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v4
+        with:
+          version: 10
+
+      - name: Validate test-suite lockfile
+        run: pnpm install --frozen-lockfile
+        working-directory: apps/test-suite