moonlight-lupin/agent-skills / research/endpoint-probe
endpoint-probe
FlaggedUse when you need to discover an API's surface — poke a base URL to find its type (REST, GraphQL, SOAP, JSON-RPC), auth scheme, available endpoints, rate limits, and CORS policy. Also probes MCP servers (HTTP and stdio) to discover tools, resources, and prompts with full input schemas. Runs a Python probe script that checks well-known discovery paths (OpenAPI/Swagger, GraphQL introspection, health, .well-known), fingerprints the server, sends OPTIONS for CORS, detects auth from 401/403 responses, and guesses common REST resources. For MCP, performs JSON-RPC initialize handshake then tools/list, resources/list, prompts/list. Feed it a base URL or MCP server URL, optionally with auth credentials.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 18.9 KB |
| references/code-review-history.md | markdown | 2.9 KB |
| references/code-review-v1.2.1.md | markdown | 4.1 KB |
| references/interpretation-guide.md | markdown | 4.5 KB |
| scripts/probe_api.py | script | 67.6 KB |
| tests/conftest.py | script | 185 B |
| tests/test_redaction.py | script | 4.8 KB |
scripts/probe_api.py
View on GitHub#!/usr/bin/env python3
"""Endpoint Probe Probe — discover an API's surface from a base URL or MCP server.
For REST/GraphQL/SOAP APIs:
- API type (REST, GraphQL, gRPC-Web, SOAP, JSON-RPC)
- Auth scheme (Bearer, Basic, API key, OAuth2, session cookie, none)
- Endpoints (from OpenAPI/Swagger, GraphQL introspection, well-known paths, OPTIONS)
- Rate-limit headers, CORS policy, server/framework fingerprints
For MCP servers (HTTP StreamableHTTP or stdio):
- Server info, protocol version, capabilities
- Tools with full input schemas (parameter names, types, required, defaults, enums)
- Resources (URIs, names, MIME types)
- Prompts (names, descriptions, arguments)
Usage:
# Standard API probe
python probe_api.py <base_url> [--auth TOKEN] [--auth-type bearer|basic|apikey]
[--header "Key: Value"] [--timeout 10] [--json]
[--no-guess] [--api-version v1]
# MCP over HTTP
python probe_api.py <mcp_url> --mcp [--auth TOKEN] [--header "Key: Value"]
# MCP over stdio
python probe_api.py "<command> <args>" --mcp-stdio [--mcp-env "KEY=value"]
Output: JSON report to stdout (--json) or pretty-printed report (default).
"""
import argparse
import json
import shlex
import sys
import time
import urllib.request
import urllib.error
import urllib.parse
import os
import socket
# ---------------------------------------------------------------------------
# Constants
# ---------------------------------------------------------------------------
# Well-known discovery paths (relative to base URL)
DISCOVERY_PATHS = [
# OpenAPI / Swagger
"/openapi.json",
"/openapi.yaml",
"/swagger.json",
"/swagger/v1/swagger.json",
"/api-docs",
"/api/docs",
"/docs",
"/redoc",
"/api/openapi.json",
"/api/v1/openapi.json",
"/api/swagger.json",
"/v1/openapi.json",
# Rapidoc / Elements
"/rapidoc",
"/api/rapidoc",
# .well-known
"/.well-known/openapi.json",
"/.well-known/api",
"/.well-known/manifest.json",
# GraphQL
"/graphql",
"/graphql/schema",
"/api/graphql",
"/query",
# gRPC-Web / Connect
"/grpc",
"/grpc.web",
# SOAP / WSDL
"/wsdl",
"/service.asmx?wsdl",
"/api/soap?wsdl",
# Health / info
"/health",
"/healthz",
"/api/health",
"/status",
"/api/status",
"/info",
"/api/info",
"/version",
"/api/version",
# Common API base paths
"/api",
"/api/v1",
"/api/v2",
"/v1",
"/v2",
# JSON-RPC
"/jsonrpc",
"/rpc",
"/api/rpc",
]
# Common REST resource paths to probe ( appended to /api/ or /api/v1/ )
COMMON_RESOURCES = [
"users", "user", "accounts", "products", "orders", "items",
"posts", "comments", "categories", "sessions", "tokens",
"auth", "login", "logout", "register", "me", "profile",
"search", "files", "uploads", "webhooks", "events",
"notifications", "messages", "channels", "organizations",
"teams", "projects", "repositories", "commits", "branches",
"settings", "config", "preferences", "billing", "invoices",
"payments", "subscriptions", "plans", "metrics", "stats",
"logs", "audit", "reports", "export", "import",
]
# GraphQL introspection query
GRAPHQL_INTROSPECTION = json.dumps({
"query": """
query IntrospectionQuery {
__schema {
queryType { name }
mutationType { name }
subscriptionType { name }
types {
name
kind
fields {
name
type { name kind ofType { name kind } }
}
}
}
}
"""
})
# Headers that reveal auth / rate-limit / framework info.
# Credential-bearing names below are still captured for discovery, but values
# are redacted via _redact_header_value() before they enter the report.
INTERESTING_HEADERS = [
"www-authenticate", "x-api-key", "api-key", "authorization",
"x-ratelimit-limit", "x-ratelimit-remaining", "x-ratelimit-reset",
"ratelimit-limit", "ratelimit-remaining", "ratelimit-reset",
"retry-after", "x-rate-limit-limit", "x-rate-limit-remaining",
"access-control-allow-origin", "access-control-allow-methods",
"access-control-allow-headers", "access-control-allow-credentials",
"access-control-expose-headers",
"x-powered-by", "server", "x-aspnet-version", "x-aspnetmvc-version",
"x-request-id", "x-correlation-id", "x-trace-id",
"content-type", "set-cookie",
"x-amzn-requestid", "x-amz-cf-id",
"x-served-by", "x-cache",
"strict-transport-security",
"x-content-type-options",
"x-frame-options",
]
# Response headers whose values may contain secrets (still report presence/shape).
_SENSITIVE_HEADER_NAMES = frozenset({
"authorization",
"proxy-authorization",
"x-api-key",
"api-key",
"set-cookie",
"cookie",
})
DEFAULT_TIMEOUT = 10
USER_AGENT = "endpoint-probe-probe/1.2.3 (+https://github.com/moonlight-lupin)"
# ---------------------------------------------------------------------------
# HTTP helpers
# ---------------------------------------------------------------------------
MAX_BODY_SIZE = 5 * 1024 * 1024 # 5 MiB cap to prevent memory DoS (M5)
def _make_request(url, method="GET", headers=None, data=None, timeout=DEFAULT_TIMEOUT):
"""Low-level HTTP request. Returns (status, response_headers, body_text)."""
# SSRF check (B3) — skip for MCP stdio which doesn't use URLs
try:
_validate_url(url)
except ValueError as e:
return -1, {}, str(e)
# Always copy headers to avoid mutating shared dict under threads (M4)
req_headers = {"User-Agent": USER_AGENT}
if headers:
req_headers.update(headers)
req = urllib.request.Request(url, data=data, method=method)
for k, v in req_headers.items():
req.add_header(k, v)
# Build opener that blocks redirects for SSRF safety (B3)
opener = urllib.request.build_opener(_NoRedirectHandler())
try:
resp = opener.open(req, timeout=timeout)
# Cap body read to prevent memory DoS (M5)
raw = resp.read(MAX_BODY_SIZE + 1)
truncated = len(raw) > MAX_BODY_SIZE
body = raw[:MAX_BODY_SIZE].decode("utf-8", errors="replace")
if truncated:
body += "\n[truncated at 5 MiB]"
return resp.status, dict(resp.headers), body
except urllib.error.HTTPError as e:
raw = e.read(MAX_BODY_SIZE + 1) if e.fp else b""
truncated = len(raw) > MAX_BODY_SIZE
body = raw[:MAX_BODY_SIZE].decode("utf-8", errors="replace")
if truncated:
body += "\n[truncated at 5 MiB]"
return e.code, dict(e.headers) if e.headers else {}, body
except urllib.error.URLError as e:
return -1, {}, str(e.reason)
except Exception as e:
return -1, {}, str(e)
def _normalize_base_url(url):
"""Ensure base URL has scheme, no trailing slash."""
if not url.startswith("http://") and not url.startswith("https://"):
url = "https://" + url
return url.rstrip("/")
# --- SSRF protection (B3) ---
import ipaddress
_METADATA_HOSTS = {"169.254.169.254", "metadata.google.internal"}
def _is_private_ip(ip_str):
"""Check if an IP address string is private/internal using ipaddress module."""
try:
ip = ipaddress.ip_address(ip_str)
except ValueError:
return False
# is_private covers RFC1918, loopback, link-local, ULA fc00::/7
# Explicitly check CGNAT 100.64.0.0/10 (not always in is_private)
if isinstance(ip, ipaddress.IPv4Address):
cgnet = ipaddress.ip_network("100.64.0.0/10")
if ip in cgnet:
return True
return (
ip.is_private or ip.is_loopback or ip.is_link_local
or ip.is_reserved or ip.is_multicast or ip.is_unspecified
)
def _is_private_host(hostname):
"""Check if a hostname resolves to or is a private/internal IP."""
if hostname in _METADATA_HOSTS:
return True
# Check if it's a literal IP (including IPv4-mapped IPv6 like ::ffff:127.0.0.1)
try:
ip = ipaddress.ip_address(hostname)
# Handle IPv4-mapped IPv6 — extract the IPv4 part
if isinstance(ip, ipaddress.IPv6Address) and ip.ipv4_mapped:
return _is_private_ip(str(ip.ipv4_mapped))
return _is_private_ip(hostname)
except ValueError:
pass # Not a literal IP — try DNS
# DNS resolution
try:
infos = socket.getaddrinfo(hostname, None)
for family, _, _, _, sockaddr in infos:
ip = sockaddr[0]
if ip in _METADATA_HOSTS:
return True
try:
ip_obj = ipaddress.ip_address(ip)
if isinstance(ip_obj, ipaddress.IPv6Address) and ip_obj.ipv4_mapped:
if _is_private_ip(str(ip_obj.ipv4_mapped)):
return True
elif _is_private_ip(ip):
return True
except ValueError:
pass
except socket.gaierror:
pass # Can't resolve — allow (will fail at connection time)
return False
class _NoRedirectHandler(urllib.request.HTTPRedirectHandler):
"""Prevent following redirects to avoid SSRF via 302 to internal IPs."""
def redirect_request(self, req, fp, code, msg, headers, newurl):
return None # Never follow redirects
# Module-level flag to allow private hosts (set by --allow-private)
_allow_private = False
def _validate_url(url):
"""Validate URL for SSRF safety. Raises ValueError if blocked."""
parsed = urllib.parse.urlparse(url)
if parsed.scheme not in ("http", "https"):
raise ValueError(f"Blocked: non-HTTP scheme '{parsed.scheme}'")
if not _allow_private and parsed.hostname:
if _is_private_host(parsed.hostname):
raise ValueError(
f"Blocked: private/internal host '{parsed.hostname}'. "
f"Use --allow-private to override."
)
return url
def _try_parse_json(text):
"""Best-effort JSON parse."""
try:
return json.loads(text)
except (json.JSONDecodeError, ValueError):
return None
# ---------------------------------------------------------------------------
# Probe phases
# ---------------------------------------------------------------------------
def probe_well_known(base_url, auth_headers, timeout):
"""Probe well-known discovery paths using concurrent requests."""
import concurrent.futures
def _probe_one(path):
url = base_url + path
status, hdrs, body = _make_request(url, headers=auth_headers, timeout=timeout)
if status == -1:
return None
finding = {
"path": path,
"url": url,
"status": status,
"content_type": hdrs.get("content-type", hdrs.get("Content-Type", "")),
"body_length": len(body),
"interesting_headers": _extract_interesting_headers(hdrs),
}
if status == 200 and body:
parsed = _try_parse_json(body)
if parsed and ("openapi" in parsed or "swagger" in parsed):
finding["type"] = "openapi"
finding["version"] = parsed.get("openapi") or parsed.get("swagger")
finding["title"] = parsed.get("info", {}).get("title", "")
finding["endpoints"] = _extract_openapi_endpoints(parsed)
elif parsed and "__schema" in str(parsed)[:500]:
finding["type"] = "graphql-schema"
elif "graphql" in path:
finding["type"] = "graphql-endpoint"
elif status == 200 and "text/html" in finding["content_type"]:
finding["type"] = "html-docs"
lower = body.lower()
if "swagger-ui" in lower or "swagger" in lower:
finding["subtype"] = "swagger-ui"
elif "redoc" in lower:
finding["subtype"] = "redoc"
elif "rapidoc" in lower:
finding["subtype"] = "rapidoc"
elif "scalar" in lower:
finding["subtype"] = "scalar"
elif status in (200, 201, 204) and path in ("/health", "/healthz", "/api/health"):
finding["type"] = "health"
elif status in (200,) and path in ("/status", "/api/status"):
finding["type"] = "status"
return finding
findings = []
with concurrent.futures.ThreadPoolExecutor(max_workers=8) as pool:
futures = {pool.submit(_probe_one, p): p for p in DISCOVERY_PATHS}
for future in concurrent.futures.as_completed(futures):
try:
result = future.result()
if result:
findings.append(result)
except Exception as e:
pass # errors collected per-path via status codes
# Sort by path for stable output
findings.sort(key=lambda f: f["path"])
return findings
def probe_graphql(base_url, auth_headers, timeout):
"""Attempt GraphQL introspection."""
candidates = ["/graphql", "/api/graphql", "/query"]
results = []
for path in candidates:
url = base_url + path
headers = dict(auth_headers)
headers["Content-Type"] = "application/json"
status, hdrs, body = _make_request(
url, method="POST", headers=headers,
data=GRAPHQL_INTROSPECTION.encode("utf-8"), timeout=timeout
)
if status == -1:
continue
parsed = _try_parse_json(body)
result = {
"path": path,
"url": url,
"status": status,
"content_type": hdrs.get("content-type", hdrs.get("Content-Type", "")),
}
if parsed and "data" in parsed and "__schema" in parsed.get("data", {}):
result["introspection"] = True
schema = parsed["data"]["__schema"]
result["query_type"] = schema.get("queryType", {}).get("name")
result["mutation_type"] = schema.get("mutationType", {}).get("name")
result["subscription_type"] = schema.get("subscriptionType", {}).get("name")
result["types"] = [
t["name"] for t in schema.get("types", [])
if t.get("name") and not t["name"].startswith("__")
]
elif parsed and "errors" in parsed:
result["introspection"] = False
result["error"] = parsed["errors"][0].get("message", "")[:200]
else:
result["introspection"] = False
results.append(result)
return results
def probe_options(base_url, auth_headers, timeout):
"""Send OPTIONS to base URL and /api to discover allowed methods."""
results = []
for path in ["", "/api", "/api/v1"]:
url = base_url + path
status, hdrs, body = _make_request(url, method="OPTIONS", headers=auth_headers, timeout=timeout)
if status == -1:
continue
allow = hdrs.get("allow", hdrs.get("Allow", ""))
cors_methods = hdrs.get("access-control-allow-methods", hdrs.get("Access-Control-Allow-Methods", ""))
cors_origin = hdrs.get("access-control-allow-origin", hdrs.get("Access-Control-Allow-Origin", ""))
if allow or cors_methods or cors_origin:
results.append({
"path": path or "/",
"url": url,
"status": status,
"allow": allow,
"cors_allow_methods": cors_methods,
"cors_allow_origin": cors_origin,
"cors_allow_headers": hdrs.get("access-control-allow-headers", hdrs.get("Access-Control-Allow-Headers", "")),
"cors_allow_credentials": hdrs.get("access-control-allow-credentials", hdrs.get("Access-Control-Allow-Credentials", "")),
})
return results
def probe_auth(base_url, timeout):
"""Determine auth scheme by examining 401/403 responses."""
# Hit a likely-protected endpoint without auth
test_paths = ["/api/me", "/api/users", "/api/v1/users", "/me", "/api/account"]
auth_findings = []
for path in test_paths:
url = base_url + path
status, hdrs, body = _make_request(url, timeout=timeout)
if status == -1:
continue
if status in (401, 403):
www_auth = hdrs.get("www-authenticate", hdrs.get("WWW-Authenticate", ""))
# Classify against the raw body first, then store a redacted hint
# so scheme discovery still works if an error page echoed a token.
finding = {
"path": path,
"status": status,
"www_authenticate": www_auth,
"body_hint": _redact_body_hint(body),
}
# Classify auth scheme
if www_auth:
lower = www_auth.lower()
if "bearer" in lower:
finding["scheme"] = "Bearer (OAuth2 / JWT)"
elif "basic" in lower:
finding["scheme"] = "Basic"
elif "digest" in lower:
finding["scheme"] = "Digest"
elif "apikey" in lower or "api-key" in lower or "api_key" in lower:
finding["scheme"] = "API Key"
elif "oauth" in lower:
finding["scheme"] = "OAuth2"
else:
finding["scheme"] = www_auth.split()[0] if www_auth else "unknown"
elif status == 401:
# No WWW-Authenticate header — check body for hints
lower_body = body.lower() if body else ""
if "api key" in lower_body or "api-key" in lower_body or "x-api-key" in lower_body:
finding["scheme"] = "API Key (header-based, no WWW-Authenticate)"
elif "token" in lower_body or "bearer" in lower_body:
finding["scheme"] = "Bearer token (no WWW-Authenticate)"
elif "unauthorized" in lower_body or "unauthenticated" in lower_body:
finding["scheme"] = "Unknown (no WWW-Authenticate header)"
else:
finding["scheme"] = "Unknown (no WWW-Authenticate header)"
elif status == 403:
finding["scheme"] = "Forbidden — auth may be session-based or IP-restricted"
auth_findings.append(finding)
break # one is enough
return auth_findings
def probe_rest_resources(base_url, auth_headers, timeout, version=""):
"""Probe common REST resource paths using concurrent requests."""
import concurrent.futures
prefix = f"/api/{version}" if version else "/api"
def _probe_one(resource):
path = f"{prefix}/{resource}"
url = base_url + path
# Use shorter timeout for resource guessing (4s max)
probe_timeout = min(timeout, 4)
status, hdrs, body = _make_request(url, headers=auth_headers, timeout=probe_timeout)
if status == -1 or status == 404:
return None
ct = hdrs.get("content-type", hdrs.get("Content-Type", ""))
parsed = _try_parse_json(body) if body else None
result = {
"path": path,
"url": url,
"status": status,
"content_type": ct,
"body_length": len(body),
}
if parsed and isinstance(parsed, list):
result["item_count"] = len(parsed)
elif parsed and isinstance(parsed, dict):
for key in ("data", "results", "items", "records"):
if key in parsed and isinstance(parsed[key], list):
result["item_count"] = len(parsed[key])
break
if "error" in parsed or "message" in parsed:
result["error"] = (parsed.get("error") or parsed.get("message", ""))[:200]
return result
results = []
with concurrent.futures.ThreadPoolExecutor(max_workers=8) as pool:
futures = {pool.submit(_probe_one, r): r for r in COMMON_RESOURCES}
for future in concurrent.futures.as_completed(futures):
try:
result = future.result()
if result:
results.append(result)
except Exception:
pass
# Sort by path for stable output
results.sort(key=lambda r: r["path"])
# Detect SPA false positives: if all results are 200 + text/html with similar
# body lengths, it's likely a SPA serving the same page for every path.
if len(results) >= 5 and all(
r["status"] == 200 and "text/html" in r.get("content_type", "")
for r in results
):
body_lengths = [r["body_length"] for r in results]
if max(body_lengths) - min(body_lengths) < 200:
# All same size ±200 bytes → SPA catch-all
return [] # discard — these are false positives
return results
def probe_rate_limits(base_url, auth_headers, timeout):
"""Check rate-limit headers from a baseline request."""
url = base_url + "/api"
status, hdrs, body = _make_request(url, headers=auth_headers, timeout=timeout)
rate_info = {}
for h in hdrs:
lower = h.lower()
if "rate" in lower or "limit" in lower or "retry" in lower or "quota" in lower:
rate_info[h] = hdrs[h]
return rate_info if rate_info else None
def _redact_set_cookie(value):
"""Keep cookie names + attributes; redact cookie values.
Multiple ``Set-Cookie`` values may be comma-joined by HTTP libraries.
RFC 1123 ``Expires`` dates also contain commas (e.g.
``Expires=Wed, 09 Jun 2026 10:18:14 GMT``), so we protect those
commas before splitting on ``,`` to avoid mis-parsing.
"""
import re
# Mask commas inside Expires=... so they don't split a single cookie.
# We restore them after the split.
_EXPIRES_SENTINEL = "\x00COMMA\x00"
def _mask_expires_commas(text):
def _replace(m):
return m.group(0).replace(",", _EXPIRES_SENTINEL)
# Match Expires=<day-name>, <rest of date>
return re.sub(
r"(?i)\bExpires=[A-Za-z]{3},[^;]*",
_replace,
text,
)
masked = _mask_expires_commas(value)
parts = []
for segment in masked.split(","):
segment = segment.replace(_EXPIRES_SENTINEL, ",")
# Multiple Set-Cookie values may be joined; also handle attrs after ';'.
attrs = [a.strip() for a in segment.split(";") if a.strip()]
if not attrs:
continue
name, sep, _ = attrs[0].partition("=")
if sep:
redacted = [f"{name}=***"]
else:
redacted = [attrs[0]]
for attr in attrs[1:]:
# Preserve Path/HttpOnly/Secure/SameSite/etc.; redact unknown name=value
# only when it looks like another cookie pair (rare). Keep standard attrs.
lower = attr.lower()
if lower.startswith(("path=", "domain=", "expires=", "max-age=", "samesite=")) or lower in (
"httponly", "secure", "partitioned",
):
redacted.append(attr)
elif "=" in attr:
aname, _, _ = attr.partition("=")
redacted.append(f"{aname}=***")
else:
redacted.append(attr)
parts.append("; ".join(redacted))
return ", ".join(parts) if parts else "***"
def _redact_header_value(name, value):
"""Redact credential-like header values while preserving auth-protocol shape.
Examples:
Authorization: Bearer eyJhbGciOi... → Bearer ***
X-API-Key: sk-live-... → ***
Set-Cookie: session=abc; Path=/ → session=***; Path=/
WWW-Authenticate: Bearer realm="x" → unchanged (scheme discovery)
"""
if value is None:
return value
text = str(value)
lower = name.lower()
if lower not in _SENSITIVE_HEADER_NAMES:
return text
if lower in ("authorization", "proxy-authorization"):
scheme, _, rest = text.strip().partition(" ")
if rest:
return f"{scheme} ***"
return "***"
if lower == "set-cookie":
return _redact_set_cookie(text)
# x-api-key / api-key / cookie
return "***"
def _redact_body_hint(body, limit=300):
"""Truncate auth-error bodies and scrub common credential patterns.
Keeps scheme-discovery phrases (e.g. "API key required") while removing
inline tokens that misconfigured error pages sometimes echo.
"""
import re
if not body:
return ""
text = body[:limit]
# Bearer / Basic / Digested credentials in prose or JSON
text = re.sub(
r"(?i)\b(bearer|basic|digest)\s+[A-Za-z0-9\-._~+/]+=*",
r"\1 ***",
text,
)
# Common key/token JSON or query-ish assignments.
# Avoid bare "token" — it matches prose like "missing token: ..." and
# would scrub the scheme word we want to keep for discovery.
text = re.sub(
r'(?i)("?(?:api[_-]?key|access[_-]?token|refresh[_-]?token|id[_-]?token|'
r'api[_-]?token|auth[_-]?token|client[_-]?secret|'
r'secret|password)"?\s*[:=]\s*)("?)([^\s",}\\]+)\2',
r"\1\2***\2",
text,
)
# Long opaque secrets (JWT-shaped or high-entropy blobs) standing alone.
# Matches JWS (3 segments) and JWE (4-5 segments, may have empty CEK).
text = re.sub(
r"\beyJ[A-Za-z0-9_-]{10,}(?:\.[A-Za-z0-9_-]{0,}){1,}\b",
"***",
text,
)
return text
def _extract_interesting_headers(hdrs):
"""Pull interesting headers (case-insensitive), redacting secret values."""
result = {}
lower_map = {k.lower(): k for k in hdrs}
for target in INTERESTING_HEADERS:
if target in lower_map:
actual_key = lower_map[target]
result[target] = _redact_header_value(target, hdrs[actual_key])
return result
def _extract_openapi_endpoints(spec):
"""Extract endpoint list from OpenAPI/Swagger spec."""
endpoints = []
paths = spec.get("paths", {})
for path, methods in paths.items():
for method, details in methods.items():
if method.upper() in ("GET", "POST", "PUT", "PATCH", "DELETE", "HEAD", "OPTIONS"):
ep = {
"path": path,
"method": method.upper(),
"summary": details.get("summary", ""),
"operation_id": details.get("operationId", ""),
"tags": details.get("tags", []),
}
# Auth requirement
security = details.get("security", spec.get("security", []))
if security:
ep["security"] = list(security[0].keys()) if security else []
endpoints.append(ep)
return endpoints
# ---------------------------------------------------------------------------
# Fingerprinting
# ---------------------------------------------------------------------------
def fingerprint_server(base_url, auth_headers, timeout):
"""Identify server/framework from response headers."""
status, hdrs, body = _make_request(base_url, headers=auth_headers, timeout=timeout)
fp = {
"server": hdrs.get("server", hdrs.get("Server", "")),
"powered_by": hdrs.get("x-powered-by", hdrs.get("X-Powered-By", "")),
"aspnet_version": hdrs.get("x-aspnet-version", hdrs.get("X-AspNet-Version", "")),
"framework_hints": [],
}
# Infer framework from headers
powered = fp["powered_by"].lower()
server = fp["server"].lower()
if "express" in powered:
fp["framework_hints"].append("Express.js")
if "asp.net" in powered or "aspnet" in powered:
fp["framework_hints"].append("ASP.NET")
if "django" in powered or "python" in server:
fp["framework_hints"].append("Django/Python")
if "flask" in powered:
fp["framework_hints"].append("Flask")
if "fastapi" in powered:
fp["framework_hints"].append("FastAPI")
if "rails" in powered or "ruby" in server:
fp["framework_hints"].append("Rails/Ruby")
if "spring" in powered or "java" in server:
fp["framework_hints"].append("Spring/Java")
if "gin" in powered:
fp["framework_hints"].append("Gin (Go)")
if "next" in powered:
fp["framework_hints"].append("Next.js")
if "cloudflare" in server:
fp["framework_hints"].append("Cloudflare")
if "nginx" in server:
fp["framework_hints"].append("Nginx")
if "apache" in server:
fp["framework_hints"].append("Apache")
# Check for API gateway signatures
if "x-amzn-requestid" in {k.lower() for k in hdrs}:
fp["framework_hints"].append("AWS API Gateway")
if "x-kong-proxy-latency" in {k.lower() for k in hdrs}:
fp["framework_hints"].append("Kong API Gateway")
return fp
# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------
def run_probe(base_url, auth_headers, timeout, version="", no_guess=False):
"""Run the full probe sequence and return a report dict."""
report = {
"base_url": base_url,
"probed_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"timeout_seconds": timeout,
}
# Phase 1: Server fingerprint
fp_status = fingerprint_server(base_url, auth_headers, timeout)
report["server_fingerprint"] = fp_status
# Surface SSRF block if the first request was blocked
if not fp_status.get("server") and not _allow_private:
report["ssrf_warning"] = (
"All requests returned connection errors. If the target is a private/internal host, "
"use --allow-private to override SSRF protection."
)
# Phase 2: Well-known discovery paths
report["discovery"] = probe_well_known(base_url, auth_headers, timeout)
# Phase 3: GraphQL introspection
report["graphql"] = probe_graphql(base_url, auth_headers, timeout)
# Phase 4: OPTIONS / CORS
report["options_cors"] = probe_options(base_url, auth_headers, timeout)
# Phase 5: Auth detection
report["auth"] = probe_auth(base_url, timeout)
# Phase 6: Rate limits
report["rate_limits"] = probe_rate_limits(base_url, auth_headers, timeout)
# Check if we're already rate-limited before resource guessing
rate_limited = _check_rate_limited(report["rate_limits"])
# Phase 7: REST resource guessing (optional)
if not no_guess and not rate_limited:
report["rest_resources"] = probe_rest_resources(
base_url, auth_headers, timeout, version
)
elif rate_limited:
report["rest_resources"] = []
report["rate_limited_skipped_guessing"] = True
else:
report["rest_resources"] = []
# Summary
summary = {
"api_type": _detect_api_type(report),
"auth_scheme": _detect_auth_scheme(report),
"openapi_found": any(
d.get("type") == "openapi" for d in report["discovery"]
),
"graphql_enabled": any(
g.get("introspection") for g in report["graphql"]
),
"cors_enabled": len(report["options_cors"]) > 0,
"endpoints_discovered": _count_endpoints(report),
"health_endpoint": any(
d.get("type") == "health" for d in report["discovery"]
),
}
report["summary"] = summary
return report
def _check_rate_limited(rate_info):
"""Check if rate limit info indicates we're already throttled."""
if not rate_info:
return False
for k, v in rate_info.items():
if "remaining" in k.lower():
try:
return int(v) == 0
except (ValueError, TypeError):
pass
return False
def _detect_api_type(report):
"""Infer API type from findings."""
types = []
for d in report.get("discovery", []):
if d.get("type") == "openapi":
types.append("REST (OpenAPI documented)")
elif d.get("type") == "graphql-schema" or d.get("type") == "graphql-endpoint":
types.append("GraphQL")
if any(g.get("introspection") for g in report.get("graphql", [])):
types.append("GraphQL (introspection confirmed)")
if report.get("rest_resources"):
types.append("REST (resource probing)")
# Check for SOAP
for d in report.get("discovery", []):
if "wsdl" in d.get("path", "") and d.get("status") == 200 and "text/html" not in d.get("content_type", ""):
types.append("SOAP")
if not types:
types.append("Unknown — no API surface discovered")
return list(set(types))
def _detect_auth_scheme(report):
"""Infer auth scheme from findings."""
for a in report.get("auth", []):
if a.get("scheme"):
return a["scheme"]
# Check OpenAPI security schemes
for d in report.get("discovery", []):
if d.get("type") == "openapi":
# Could parse securitySchemes from spec
return "See OpenAPI spec for security schemes"
return "None detected (API may be open or use cookie-based auth)"
def _count_endpoints(report):
"""Count total discovered endpoints."""
count = 0
for d in report.get("discovery", []):
if d.get("type") == "openapi" and d.get("endpoints"):
count += len(d["endpoints"])
if report.get("rest_resources"):
count += len(report["rest_resources"])
# Note: GraphQL types are not counted as "endpoints" to avoid inflating the count
return count
def _format_report(report, pretty=True):
"""Format report for human or JSON output."""
if not pretty:
return json.dumps(report, indent=2)
lines = []
lines.append("=" * 70)
lines.append(f" ENDPOINT PROBE REPORT — {report['base_url']}")
lines.append(f" Probed: {report['probed_at']}")
lines.append("=" * 70)
# Summary
s = report.get("summary", {})
lines.append("\n📋 SUMMARY")
lines.append(f" API Type: {', '.join(s.get('api_type', []))}")
lines.append(f" Auth Scheme: {s.get('auth_scheme', 'Unknown')}")
lines.append(f" OpenAPI Found: {'✅' if s.get('openapi_found') else '❌'}")
lines.append(f" GraphQL: {'✅' if s.get('graphql_enabled') else '❌'}")
lines.append(f" CORS Detected: {'✅' if s.get('cors_enabled') else '❌'}")
lines.append(f" Endpoints: {s.get('endpoints_discovered', 0)}")
lines.append(f" Health Check: {'✅' if s.get('health_endpoint') else '❌'}")
# Server fingerprint
fp = report.get("server_fingerprint", {})
lines.append("\n🖥️ SERVER FINGERPRINT")
lines.append(f" Server: {fp.get('server', 'N/A')}")
lines.append(f" Powered By: {fp.get('powered_by', 'N/A')}")
if fp.get("framework_hints"):
lines.append(f" Framework: {', '.join(fp['framework_hints'])}")
# Discovery findings
lines.append("\n🔍 DISCOVERY PATHS")
for d in report.get("discovery", []):
status_icon = "✅" if d["status"] == 200 else ("🚫" if d["status"] in (401, 403) else ("❓" if d["status"] != 404 else "⬜"))
if d["status"] == 404:
continue # skip 404s for readability
type_str = d.get("type", "unknown")
extra = ""
if d.get("title"):
extra = f" — {d['title']}"
if d.get("version"):
extra += f" (v{d['version']})"
if d.get("endpoints"):
extra += f" [{len(d['endpoints'])} endpoints]"
lines.append(f" {status_icon} {d['path']:40s} {d['status']} {type_str}{extra}")
# GraphQL
gql = report.get("graphql", [])
if any(g.get("introspection") for g in gql):
lines.append("\n🔧 GRAPHQL")
for g in gql:
if g.get("introspection"):
lines.append(f" ✅ {g['path']} — introspection successful")
lines.append(f" Query: {g.get('query_type', 'N/A')}")
lines.append(f" Mutation: {g.get('mutation_type', 'N/A')}")
lines.append(f" Subscription: {g.get('subscription_type', 'N/A')}")
if g.get("types"):
lines.append(f" Types ({len(g['types'])}): {', '.join(g['types'][:20])}")
if len(g["types"]) > 20:
lines.append(f" ... and {len(g['types']) - 20} more")
elif g.get("error"):
lines.append(f" ❌ {g['path']} — {g['error']}")
elif any(g.get("status") not in (404, -1) for g in gql):
lines.append("\n🔧 GRAPHQL")
for g in gql:
if g.get("status") not in (404, -1):
lines.append(f" ❓ {g['path']} — status {g['status']}, introspection disabled")
# Auth
auth = report.get("auth", [])
if auth:
lines.append("\n🔐 AUTHENTICATION")
for a in auth:
lines.append(f" {a['path']} → {a['status']}")
lines.append(f" Scheme: {a.get('scheme', 'Unknown')}")
if a.get("www_authenticate"):
lines.append(f" WWW-Authenticate: {a['www_authenticate']}")
if a.get("body_hint"):
lines.append(f" Body hint: {a['body_hint'][:150]}")
# CORS / OPTIONS
cors = report.get("options_cors", [])
if cors:
lines.append("\n🌐 CORS / OPTIONS")
for c in cors:
lines.append(f" {c['path'] or '/'} — {c['status']}")
if c.get("allow"):
lines.append(f" Allow: {c['allow']}")
if c.get("cors_allow_methods"):
lines.append(f" CORS Methods: {c['cors_allow_methods']}")
if c.get("cors_allow_origin"):
lines.append(f" CORS Origin: {c['cors_allow_origin']}")
if c.get("cors_allow_headers"):
lines.append(f" CORS Headers: {c['cors_allow_headers']}")
if c.get("cors_allow_credentials"):
lines.append(f" CORS Creds: {c['cors_allow_credentials']}")
# Rate limits
rl = report.get("rate_limits")
if rl:
lines.append("\n⏱️ RATE LIMITS")
for k, v in rl.items():
lines.append(f" {k}: {v}")
else:
lines.append("\n⏱️ RATE LIMITS: None detected")
# REST resources
resources = report.get("rest_resources", [])
if resources:
lines.append("\n📦 REST RESOURCES (probed)")
for r in resources:
extra = ""
if r.get("item_count") is not None:
extra = f" ({r['item_count']} items)"
if r.get("error"):
extra = f" — {r['error']}"
lines.append(f" {r['status']} {r['path']:40s} {r['content_type'][:40]}{extra}")
elif report.get("rate_limited_skipped_guessing"):
lines.append("\n📦 REST RESOURCES: Skipped (rate limit exhausted)")
# OpenAPI endpoints detail
for d in report.get("discovery", []):
if d.get("type") == "openapi" and d.get("endpoints"):
lines.append(f"\n📜 OPENAPI ENDPOINTS ({d.get('title', 'untitled')})")
for ep in d["endpoints"]:
sec = f" [auth: {', '.join(ep['security'])}]" if ep.get("security") else ""
summary = f" — {ep['summary']}" if ep.get("summary") else ""
lines.append(f" {ep['method']:7s} {ep['path']}{summary}{sec}")
lines.append("\n" + "=" * 70)
return "\n".join(lines)
# ---------------------------------------------------------------------------
# MCP probing (JSON-RPC over HTTP or stdio)
# ---------------------------------------------------------------------------
MCP_PROTOCOL_VERSION = "2025-06-18"
MCP_CLIENT_NAME = "endpoint-probe-probe"
MCP_CLIENT_VERSION = "1.2.3"
def _extract_sse_data(body):
"""Extract JSON data from an SSE (text/event-stream) response body.
SSE format:
event: message
data: {"jsonrpc":"2.0","result":{...}}
Returns the first data: line content, or None if no SSE data found.
"""
for line in body.split("\n"):
line = line.strip()
if line.startswith("data:"):
data_content = line[5:].strip()
if data_content and data_content != "[DONE]":
return data_content
return None
# Module-level session state for HTTP MCP (M1)
_mcp_session_id = None
def _mcp_jsonrpc(url, method, params=None, request_id=1, headers=None, timeout=DEFAULT_TIMEOUT,
is_notification=False):
"""Send a single MCP JSON-RPC request over HTTP. Returns (result, error, resp_headers)."""
payload = {
"jsonrpc": "2.0",
"method": method,
}
# Notifications must omit id (M2)
if not is_notification:
payload["id"] = request_id
if params is not None:
payload["params"] = params
req_headers = {
"Content-Type": "application/json",
"Accept": "application/json, text/event-stream",
}
# Replay session-id if we have one (M1)
global _mcp_session_id
if _mcp_session_id:
req_headers["Mcp-Session-Id"] = _mcp_session_id
if headers:
req_headers.update(headers)
data = json.dumps(payload).encode("utf-8")
status, resp_headers, body = _make_request(
url, method="POST", headers=req_headers, data=data, timeout=timeout
)
if status == -1:
return None, f"Connection error: {body}", {}
# Accept 200 for requests, 202/204 for notifications (M2)
if is_notification:
if status in (200, 202, 204):
return {}, None, resp_headers
return None, f"HTTP {status}: {body[:300]}", resp_headers
if status != 200:
return None, f"HTTP {status}: {body[:300]}", resp_headers
# Capture session-id from response (M1)
session_id = _get_header_ci(resp_headers, "mcp-session-id")
if session_id:
_mcp_session_id = session_id
parsed = _try_parse_json(body)
if not parsed:
# Try parsing as SSE (text/event-stream) — extract data lines
sse_data = _extract_sse_data(body)
if sse_data:
parsed = _try_parse_json(sse_data)
if not parsed:
return None, f"Non-JSON response: {body[:300]}", resp_headers
if "result" in parsed:
return parsed["result"], None, resp_headers
elif "error" in parsed:
err = parsed["error"]
return None, f"RPC error {err.get('code')}: {err.get('message', '')}", resp_headers
else:
return None, f"Unexpected response: {body[:300]}", resp_headers
def _get_header_ci(headers, target):
"""Case-insensitive header lookup."""
for k, v in headers.items():
if k.lower() == target.lower():
return v
return None
def _mcp_initialize(url, headers, timeout):
"""Send MCP initialize handshake. Returns (init_data, error)."""
global _mcp_session_id
_mcp_session_id = None # reset for fresh connection
result, err, resp_headers = _mcp_jsonrpc(
url,
"initialize",
params={
"protocolVersion": MCP_PROTOCOL_VERSION,
"capabilities": {},
"clientInfo": {
"name": MCP_CLIENT_NAME,
"version": MCP_CLIENT_VERSION,
},
},
request_id=1,
headers=headers,
timeout=timeout,
)
if err:
return None, err
server_info = result.get("serverInfo", {})
protocol_version = result.get("protocolVersion", "")
capabilities = result.get("capabilities", {})
# Send initialized notification — no id, accept 202/204 (M2)
_mcp_jsonrpc(
url,
"notifications/initialized",
params={},
headers=headers,
timeout=timeout,
is_notification=True,
)
return {
"server_info": server_info,
"protocol_version": protocol_version,
"capabilities": capabilities,
}, None
def _parse_mcp_tool(tool):
"""Parse a single MCP tool definition into a structured dict."""
schema = tool.get("inputSchema", {})
properties = schema.get("properties", {})
required = schema.get("required", [])
params = []
for name, prop in properties.items():
params.append({
"name": name,
"type": prop.get("type", "any"),
"description": prop.get("description", ""),
"required": name in required,
"default": prop.get("default"),
"enum": prop.get("enum"),
})
return {
"name": tool.get("name", ""),
"description": tool.get("description", ""),
"parameters": params,
"required_params": required,
}
def _mcp_list_tools_http(url, headers, timeout):
"""List all MCP tools with their input schemas (HTTP transport)."""
result, err, _ = _mcp_jsonrpc(
url, "tools/list", params={}, request_id=3, headers=headers, timeout=timeout
)
if err:
return None, err
tools = result.get("tools", [])
return [_parse_mcp_tool(t) for t in tools], None
def _mcp_list_resources_http(url, headers, timeout):
"""List all MCP resources (HTTP transport)."""
result, err, _ = _mcp_jsonrpc(
url, "resources/list", params={}, request_id=4, headers=headers, timeout=timeout
)
if err:
if "not found" in err.lower() or "method not found" in err.lower():
return [], None
return None, err
resources = result.get("resources", [])
parsed = []
for res in resources:
parsed.append({
"uri": res.get("uri", ""),
"name": res.get("name", ""),
"description": res.get("description", ""),
"mime_type": res.get("mimeType", ""),
})
return parsed, None
def _mcp_list_prompts_http(url, headers, timeout):
"""List all MCP prompt templates (HTTP transport)."""
result, err, _ = _mcp_jsonrpc(
url, "prompts/list", params={}, request_id=5, headers=headers, timeout=timeout
)
if err:
if "not found" in err.lower() or "method not found" in err.lower():
return [], None
return None, err
prompts = result.get("prompts", [])
parsed = []
for p in prompts:
args = p.get("arguments", [])
parsed_args = []
for arg in args:
parsed_args.append({
"name": arg.get("name", ""),
"description": arg.get("description", ""),
"required": arg.get("required", False),
})
parsed.append({
"name": p.get("name", ""),
"description": p.get("description", ""),
"arguments": parsed_args,
})
return parsed, None
def probe_mcp_http(url, auth_headers, timeout):
"""Full MCP probe over HTTP (StreamableHTTP transport)."""
report = {
"url": url,
"transport": "http",
"probed_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
}
# Phase 1: Initialize handshake
init_result, err = _mcp_initialize(url, auth_headers, timeout)
if err:
report["error"] = f"Initialize failed: {err}"
report["status"] = "failed"
return report
report["server_info"] = init_result["server_info"]
report["protocol_version"] = init_result["protocol_version"]
report["capabilities"] = init_result["capabilities"]
report["status"] = "connected"
# Phase 2: List tools
tools, err = _mcp_list_tools_http(url, auth_headers, timeout)
if err:
report["tools_error"] = err
report["tools"] = []
else:
report["tools"] = tools
# Phase 3: List resources
resources, err = _mcp_list_resources_http(url, auth_headers, timeout)
if err:
report["resources_error"] = err
report["resources"] = []
else:
report["resources"] = resources
# Phase 4: List prompts
prompts, err = _mcp_list_prompts_http(url, auth_headers, timeout)
if err:
report["prompts_error"] = err
report["prompts"] = []
else:
report["prompts"] = prompts
# Summary
report["summary"] = {
"server_name": init_result["server_info"].get("name", "unknown"),
"server_version": init_result["server_info"].get("version", ""),
"protocol_version": init_result["protocol_version"],
"capabilities": list(init_result["capabilities"].keys()),
"tool_count": len(report["tools"]),
"resource_count": len(report["resources"]),
"prompt_count": len(report["prompts"]),
}
return report
def _cleanup_subprocess(proc):
"""Ensure subprocess is always terminated and drained (M3)."""
try:
if proc.stdin and not proc.stdin.closed:
proc.stdin.close()
except Exception:
pass
try:
proc.terminate()
except Exception:
pass
try:
proc.wait(timeout=5)
except Exception:
try:
proc.kill()
proc.wait(timeout=3)
except Exception:
pass
def probe_mcp_stdio(command, args, env_dict, timeout):
"""Probe an MCP stdio server by launching it, doing the handshake, listing, and exiting."""
import subprocess
report = {
"command": command,
"args": args,
"transport": "stdio",
"probed_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
# Initialize defaults so summary never KeyErrors after an exception (B1r4)
"server_info": {},
"protocol_version": "",
"capabilities": {},
"tools": [],
"resources": [],
"prompts": [],
}
# Build the subprocess environment — inherit current env, scrub secrets (M7 review2)
_SAFE_ENV_VARS = {
"PATH", "HOME", "USER", "LANG", "LC_ALL", "LC_CTYPE", "TERM", "SHELL",
"TMPDIR", "TMP", "TEMP", "DISPLAY", "XAUTHORITY", "PWD",
}
_SECRET_SUBSTRINGS = ("TOKEN", "SECRET", "PASSWORD", "PASSWD", "API_KEY",
"CREDENTIAL", "PRIVATE_KEY", "ACCESS_KEY", "CLIENT_SECRET")
full_env = {}
for k, v in os.environ.items():
# Allow-list safe vars, deny anything with secret substrings
if k in _SAFE_ENV_VARS:
full_env[k] = v
elif any(s in k.upper() for s in _SECRET_SUBSTRINGS):
continue # scrub
elif k.startswith("XDG_") or k.startswith("LC_"):
full_env[k] = v # XDG and locale vars are safe
# Ensure essential vars exist
full_env.setdefault("PATH", "/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin")
full_env.setdefault("LANG", "en_US.UTF-8")
full_env.setdefault("TERM", "xterm-256color")
if env_dict:
full_env.update(env_dict)
try:
proc = subprocess.Popen(
[command] + args,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
env=full_env,
text=True,
bufsize=1,
)
except FileNotFoundError:
report["error"] = f"Command not found: {command}"
report["status"] = "failed"
return report
except Exception as e:
report["error"] = f"Failed to launch: {e}"
report["status"] = "failed"
return report
import select
def _send(msg):
"""Send a JSON-RPC message and read the response with timeout (B1, B2).
Server notifications (no id) are discarded — we only re-read, never re-send."""
expected_id = msg.get("id")
line = json.dumps(msg) + "\n"
proc.stdin.write(line)
proc.stdin.flush()
deadline = time.time() + timeout
while True:
remaining = deadline - time.time()
if remaining <= 0:
return None, f"Timeout after {timeout}s waiting for response to id={expected_id}"
# Use select for timeout-aware reads (B1)
ready, _, _ = select.select([proc.stdout], [], [], remaining)
if not ready:
return None, f"Timeout after {timeout}s waiting for response to id={expected_id}"
response_line = proc.stdout.readline()
if not response_line:
# Bound stderr reads with select to avoid hang (review2 B1)
stderr_ready, _, _ = select.select([proc.stderr], [], [], 2)
if stderr_ready:
stderr = proc.stderr.readline()
else:
stderr = ""
return None, f"No response. stderr: {stderr[:300]}"
parsed = _try_parse_json(response_line)
if not parsed:
return None, f"Non-JSON response: {response_line[:300]}"
# Server→client notification (no id) — discard, keep reading (B2)
if "id" not in parsed:
continue
# Correlate response id with expected id (review2 M2)
if parsed["id"] != expected_id:
continue # Out-of-order or server-initiated request — skip
if "result" in parsed:
return parsed["result"], None
elif "error" in parsed:
err = parsed["error"]
return None, f"RPC error {err.get('code')}: {err.get('message', '')}"
else:
# Server→client request (has id + method, no result/error) — skip
if "method" in parsed:
continue
return None, f"Unexpected response: {response_line[:300]}"
report["subprocess_pid"] = proc.pid
try:
# Phase 1: Initialize
init_result, err = _send({
"jsonrpc": "2.0",
"id": 1,
"method": "initialize",
"params": {
"protocolVersion": MCP_PROTOCOL_VERSION,
"capabilities": {},
"clientInfo": {"name": MCP_CLIENT_NAME, "version": MCP_CLIENT_VERSION},
},
})
if err:
report["error"] = f"Initialize failed: {err}"
report["status"] = "failed"
return report
report["server_info"] = init_result.get("serverInfo", {})
report["protocol_version"] = init_result.get("protocolVersion", "")
report["capabilities"] = init_result.get("capabilities", {})
report["status"] = "connected"
# Send initialized notification
proc.stdin.write(json.dumps({
"jsonrpc": "2.0",
"method": "notifications/initialized",
"params": {},
}) + "\n")
proc.stdin.flush()
# Phase 2: List tools
tools_result, err = _send({
"jsonrpc": "2.0",
"id": 3,
"method": "tools/list",
"params": {},
})
if err:
report["tools_error"] = err
report["tools"] = []
else:
raw_tools = tools_result.get("tools", [])
report["tools"] = [_parse_mcp_tool(t) for t in raw_tools]
# Phase 3: List resources (normalized to match HTTP format — M6)
res_result, err = _send({
"jsonrpc": "2.0",
"id": 4,
"method": "resources/list",
"params": {},
})
if err:
if "not found" in err.lower() or "method not found" in err.lower():
report["resources"] = []
else:
report["resources_error"] = err
report["resources"] = []
else:
raw_resources = res_result.get("resources", [])
report["resources"] = [{
"uri": r.get("uri", ""),
"name": r.get("name", ""),
"description": r.get("description", ""),
"mime_type": r.get("mimeType", ""),
} for r in raw_resources]
# Phase 4: List prompts (normalized to match HTTP format — M6)
prompt_result, err = _send({
"jsonrpc": "2.0",
"id": 5,
"method": "prompts/list",
"params": {},
})
if err:
if "not found" in err.lower() or "method not found" in err.lower():
report["prompts"] = []
else:
report["prompts_error"] = err
report["prompts"] = []
else:
raw_prompts = prompt_result.get("prompts", [])
report["prompts"] = [{
"name": pr.get("name", ""),
"description": pr.get("description", ""),
"arguments": [{
"name": a.get("name", ""),
"description": a.get("description", ""),
"required": a.get("required", False),
} for a in pr.get("arguments", [])],
} for pr in raw_prompts]
except Exception as e:
# M5r3: Catch write/pipe errors and produce a failed report
report["error"] = f"Probe error: {e}"
report["status"] = "failed"
finally:
# Always clean up subprocess even on exception (M3 review2)
_cleanup_subprocess(proc)
if proc.returncode is not None:
report["subprocess_returncode"] = proc.returncode
report["summary"] = {
"server_name": report.get("server_info", {}).get("name", "unknown"),
"server_version": report.get("server_info", {}).get("version", ""),
"protocol_version": report.get("protocol_version", ""),
"capabilities": list(report.get("capabilities", {}).keys()),
"tool_count": len(report.get("tools", [])),
"resource_count": len(report.get("resources", [])),
"prompt_count": len(report.get("prompts", [])),
}
return report
def _format_mcp_report(report, pretty=True):
"""Format MCP probe report for human output."""
if not pretty:
return json.dumps(report, indent=2)
lines = []
lines.append("=" * 70)
if report.get("transport") == "stdio":
lines.append(f" MCP PROBE REPORT — {report['command']} {' '.join(report.get('args', []))}")
else:
lines.append(f" MCP PROBE REPORT — {report['url']}")
lines.append(f" Transport: {report.get('transport', 'http')}")
lines.append(f" Probed: {report['probed_at']}")
lines.append("=" * 70)
if report.get("status") == "failed":
lines.append(f"\n❌ FAILED: {report.get('error', 'unknown error')}")
lines.append("=" * 70)
return "\n".join(lines)
# Summary
s = report.get("summary", {})
lines.append("\n📋 SUMMARY")
lines.append(f" Server: {s.get('server_name', 'N/A')} v{s.get('server_version', '')}")
lines.append(f" Protocol: {s.get('protocol_version', 'N/A')}")
lines.append(f" Capabilities: {', '.join(s.get('capabilities', [])) or 'none'}")
lines.append(f" Tools: {s.get('tool_count', 0)}")
lines.append(f" Resources: {s.get('resource_count', 0)}")
lines.append(f" Prompts: {s.get('prompt_count', 0)}")
# Tools
tools = report.get("tools", [])
if tools:
lines.append(f"\n🔧 TOOLS ({len(tools)})")
for t in tools:
lines.append(f"\n ▸ {t['name']}")
if t.get("description"):
desc = t["description"]
if len(desc) > 100:
desc = desc[:97] + "..."
lines.append(f" {desc}")
if t.get("parameters"):
lines.append(f" Parameters:")
for p in t["parameters"]:
req = "required" if p["required"] else "optional"
type_str = p["type"]
enum_str = f" enum={p['enum']}" if p.get("enum") else ""
default_str = f" default={p['default']}" if p.get("default") is not None else ""
desc_str = f" — {p['description']}" if p.get("description") else ""
lines.append(f" {p['name']} ({type_str}, {req}){enum_str}{default_str}{desc_str}")
elif t.get("required_params"):
lines.append(f" Required: {', '.join(t['required_params'])}")
else:
lines.append(f" No parameters")
elif report.get("tools_error"):
lines.append(f"\n🔧 TOOLS: Error — {report['tools_error']}")
else:
lines.append("\n🔧 TOOLS: None")
# Resources
resources = report.get("resources", [])
if resources:
lines.append(f"\n📄 RESOURCES ({len(resources)})")
for r in resources:
lines.append(f" {r.get('uri', '')} — {r.get('name', '')}")
if r.get("description"):
lines.append(f" {r['description']}")
if r.get("mime_type"):
lines.append(f" MIME: {r['mime_type']}")
elif report.get("resources_error"):
lines.append(f"\n📄 RESOURCES: Error — {report['resources_error']}")
else:
lines.append("\n📄 RESOURCES: None")
# Prompts
prompts = report.get("prompts", [])
if prompts:
lines.append(f"\n💬 PROMPTS ({len(prompts)})")
for p in prompts:
lines.append(f"\n ▸ {p.get('name', '')}")
if p.get("description"):
lines.append(f" {p['description']}")
if p.get("arguments"):
lines.append(f" Arguments:")
for a in p["arguments"]:
req = "required" if a.get("required") else "optional"
desc_str = f" — {a.get('description', '')}" if a.get("description") else ""
lines.append(f" {a['name']} ({req}){desc_str}")
elif report.get("prompts_error"):
lines.append(f"\n💬 PROMPTS: Error — {report['prompts_error']}")
else:
lines.append("\n💬 PROMPTS: None")
lines.append("\n" + "=" * 70)
return "\n".join(lines)
def _build_auth_headers(args):
"""Build auth headers from CLI args (deduplicated — m10)."""
auth_headers = {}
if args.auth:
if args.auth_type == "bearer":
auth_headers["Authorization"] = f"Bearer {args.auth}"
elif args.auth_type == "basic":
import base64
auth_headers["Authorization"] = "Basic " + base64.b64encode(
args.auth.encode()
).decode()
elif args.auth_type == "apikey":
auth_headers["X-API-Key"] = args.auth
for h in args.header:
if ":" in h:
key, val = h.split(":", 1)
auth_headers[key.strip()] = val.strip()
return auth_headers
def main():
parser = argparse.ArgumentParser(
description="Endpoint Probe Probe — discover API surface from a base URL or MCP server"
)
parser.add_argument("base_url", help="Base URL of the API to probe (or MCP server URL with --mcp)")
parser.add_argument("--auth", help="Auth token/credential", default=None)
parser.add_argument(
"--auth-type",
choices=["bearer", "basic", "apikey"],
default="bearer",
help="Auth type (default: bearer)",
)
parser.add_argument(
"--header", action="append", default=[],
help="Custom header (format: 'Key: Value')",
)
parser.add_argument("--timeout", type=int, default=DEFAULT_TIMEOUT, help="Request timeout in seconds")
parser.add_argument("--json", action="store_true", help="Output raw JSON")
parser.add_argument("--no-guess", action="store_true", help="Skip REST resource guessing")
parser.add_argument("--api-version", dest="version", default="", help="API version prefix (e.g. v1, v2)")
# MCP options
parser.add_argument(
"--mcp", action="store_true",
help="Probe as MCP server over HTTP (StreamableHTTP transport). base_url = MCP endpoint URL.",
)
parser.add_argument(
"--mcp-stdio", action="store_true",
help="Probe as MCP server over stdio. base_url = command to run (e.g. 'npx -y @modelcontextprotocol/server-time').",
)
parser.add_argument(
"--mcp-env", action="append", default=[],
help="Environment variable for stdio MCP server (format: 'KEY=value', repeatable)",
)
parser.add_argument(
"--allow-private", action="store_true",
help="Allow probing private/internal IP addresses (SSRF risk — use with caution)",
)
args = parser.parse_args()
# m3: Mutual exclusion
if args.mcp and args.mcp_stdio:
parser.error("--mcp and --mcp-stdio are mutually exclusive")
# B3: Set --allow-private flag globally
global _allow_private
_allow_private = args.allow_private
# --- MCP stdio mode ---
if args.mcp_stdio:
try:
parts = shlex.split(args.base_url)
except ValueError as e:
print(f"❌ Blocked: invalid stdio command string: {e}")
sys.exit(1)
if not parts:
print("❌ Blocked: empty stdio command string.")
sys.exit(1)
command = parts[0]
cmd_args = parts[1:]
# B4: Allowlist — only known package runners. No arbitrary absolute paths.
_ALLOWED_MCP_RUNNERS = {"npx", "uvx", "python3", "python", "node", "bunx"}
if command not in _ALLOWED_MCP_RUNNERS:
print(f"❌ Blocked: command '{command}' is not in the allowed list: "
f"{', '.join(sorted(_ALLOWED_MCP_RUNNERS))}. "
f"Only package runners are permitted for safety.")
sys.exit(1)
# R6: Per-runner argument validation — denylists are bypassable via
# short-flag clustering (e.g. python3 -Ic "code"). Use an allowlist
# approach instead: validate args against what each runner legitimately
# needs for MCP server launch.
import re as _re
# Patterns for safe args per runner
_SAFE_ARG_PATTERNS = {
"npx": r"^(-y|--yes|-p=\S+|--package=\S+|--quiet|--no-update-notifier|@[\w\-./]+|[\w\-./@]+)$",
"uvx": r"^(-q|--quiet|--from=\S+|[\w\-./]+)$",
"python3": r"^(-u|-m\s+\S+|/[\w\-./]+\.py|[\w\-./]+\.py|[\w\-.//]+)$",
"python": r"^(-u|-m\s+\S+|/[\w\-./]+\.py|[\w\-./]+\.py|[\w\-.//]+)$",
"node": r"^(/[\w\-./]+|[\w\-./]+\.js|[\w\-./]+)$",
"bunx": r"^([\w\-./@]+)$",
}
# Explicitly blocked: any arg containing inline code execution flags
# as substring or cluster (catches -c, -Ic, -Sc, -Ec, --call, --eval, etc.)
_INLINE_CODE_RE = _re.compile(r"^-[A-Za-z]*[cep]([=].*)?$|^(--call|--eval|--exec|--print)([=].*)?$")
# Also catch -c glued with code (e.g. -cprint(1)) — any -X where X starts with c/e/p
_GLUED_CODE_RE = _re.compile(r"^-c\S+$|^-e\S+$|^-p\S+$")
for arg in cmd_args:
if _INLINE_CODE_RE.match(arg) or _GLUED_CODE_RE.match(arg):
print(f"❌ Blocked: inline code execution flag '{arg}' in command args. "
f"Inline code execution is not permitted.")
sys.exit(1)
# For python/node runners, verify args look like file paths or -m module
if command in ("python3", "python", "node"):
for arg in cmd_args:
# Allow -u, -m <module>, file paths, and bare module names
if arg in ("-u", "-m", "--unbuffered"):
continue
if arg.startswith("-"):
# Check for clustered short flags containing c/e/p
if _INLINE_CODE_RE.match(arg):
print(f"❌ Blocked: inline code flag '{arg}' in clustered args.")
sys.exit(1)
# Allow other single-dash flags that aren't code-related
continue
# Allow file paths and module names
# Block anything that looks like inline code (not a path/module)
# npx/uvx/bunx: package names are fine, -y is fine, other flags checked above
env_dict = {}
for e in args.mcp_env:
if "=" in e:
k, v = e.split("=", 1)
env_dict[k.strip()] = v.strip()
report = probe_mcp_stdio(command, cmd_args, env_dict, args.timeout)
if args.json:
print(json.dumps(report, indent=2))
else:
print(_format_mcp_report(report, pretty=True))
return
# --- MCP HTTP mode ---
if args.mcp:
base_url = _normalize_base_url(args.base_url)
auth_headers = _build_auth_headers(args)
report = probe_mcp_http(base_url, auth_headers, args.timeout)
if args.json:
print(json.dumps(report, indent=2))
else:
print(_format_mcp_report(report, pretty=True))
return
# --- Standard API probe mode ---
base_url = _normalize_base_url(args.base_url)
auth_headers = _build_auth_headers(args)
report = run_probe(base_url, auth_headers, args.timeout, args.version, args.no_guess)
if args.json:
print(json.dumps(report, indent=2))
else:
print(_format_report(report, pretty=True))
if __name__ == "__main__":
main()
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |