# Website moderation

Send the URL of a public web page and CalmLens loads it in a browser, extracts the text and images, and moderates each one. The page becomes a `website` asset, and what it contains becomes child assets. You get one decision for the page and a report for every piece of extracted content.

## Submit a page

Upload the page with a [remote URL upload](/docs/guides/moderating-assets#remote-url-upload). A URL that returns `text/html` is stored as a `website` asset.

```bash
curl -X POST "https://api.calmlens.com/v1/projects/your-project-id/assets?queue=true" \
  -H "X-Api-Key: your-api-key" \
  -H "Content-Type: application/json" \
  --data '{
  "name": "landing-page",
  "url": "https://example.com"
}'
```

The URL must use HTTP or HTTPS and be reachable from the public internet. Private IP addresses, localhost names and URL credentials are rejected with [`400 asset/invalid-remote-url`](/docs/api/assets#post-asset-error-asset-invalid-remote-url), a URL that answers with an error status is refused with [`422 asset/remote-url-unavailable`](/docs/api/assets#post-asset-error-asset-remote-url-unavailable), and the same check applies to every request the page makes while it loads. Loading and moderating a page takes longer than a single image. Use `queue=true` to get an asset ID straight away, with `processing.status` `processing`, then read the decision from an [`asset.approved`](/docs/api/events#event-asset-approved) or [`asset.rejected`](/docs/api/events#event-asset-rejected) webhook. See [Queue mode](/docs/guides/moderating-assets).

## What CalmLens extracts

CalmLens opens the page in a browser, waits for the document to load and the network to go quiet (for up to ten seconds, so pages that poll or stream are still read), and reads the rendered result. Pages that build their content in JavaScript are therefore moderated as visitors see them. Loading gives up after 30 seconds.

| Extracted | Stored as |
| --- | --- |
| The rendered HTML | The `website` asset's content |
| A screenshot of the first screen (1280 × 720) | The asset's preview image |
| All visible text, with scripts and styles removed | A `plaintext-document` child named after the page title |
| Images the page loaded, and inline or embedded SVG graphics | `image` children |

At most 50 images are moderated per page. SVG graphics are taken first, then images the page requested, in the order they loaded. Images that cannot be downloaded, are not images, or are over 10 MB are skipped and don't fail the page.

Children inherit the page's [processing options](/docs/guides/moderating-assets#processing-features), folder and visibility, so a feature such as OCR that you enable on the upload also runs on the page's images. They are moderated with the project's policy like any other upload.

## Reading the result

The `website` asset has no report of its own. Its decision comes from its children, and one rejected child rejects the page, and a failed child sets its `processing.status` to `failed`.

1. Read the page's `moderation.status` for the overall decision.
2. List its children, found by `parentId`, to see which text or image caused it. See [Child assets](/docs/guides/asset-lifecycle#child-assets).
3. Read each child's `moderation.report` as described in [Reading moderation reports](/docs/guides/moderation-reports).

The page's content is the HTML as it rendered when it was submitted. Later changes to the live page aren't picked up; submit the URL again to moderate it again.

## Limits and coverage

- Only the rendered first screen is captured as a screenshot. It is a preview and is not moderated on its own; the images in the page are.
- Text inside images is read only when OCR is enabled for the upload.
- A page that returns an HTTP error, or shows a bot-protection check instead of its content, ends with `processing.status` `failed` and no children, rather than being approved. Content behind a login or paywall is moderated as the page CalmLens receives. See [Crawler](#crawler).
- Pages with more than 50 images have the rest left unmoderated. Split large galleries, or submit the images directly.
- Links are not followed. Submit each page you want moderated.

## Restricting domains and pace

Both settings are in [Project settings](/portal/settings#project-settings) in the portal, and apply to the whole project.

- [Accepted domains](/portal/settings#remote-url-domains): choose *Only the domains I list* or *Any domain except the ones I list*, then list one domain per line, such as `example.com`. A domain also covers its subdomains. The rule applies to every file and website submitted by URL, to each redirect the download follows, and to every page the browser is sent to while loading a website. Images and scripts a page pulls in from other domains are not restricted. A refused URL returns [`403 upload/remote-url-not-allowed`](/docs/api/assets#post-asset-error-upload-remote-url-not-allowed), and a website that redirects to a refused domain ends with `processing.status` `failed` and no children.
- [Website submissions per minute](/portal/settings#website-submission-rate): every website is loaded in a full browser, so each project can submit up to 60 a minute. You can lower that number, not raise it. Over the limit, the request returns [`429 upload/website-rate-limit-exceeded`](/docs/api/assets#post-asset-error-upload-website-rate-limit-exceeded) with a `Retry-After` header giving the seconds until the oldest submission in the last minute ages out. Only websites count, not images or other files submitted by URL. Retrying a failed website is refused while the project is at its limit, but a retry doesn't count as a submission.

## Crawler

CalmLens identifies itself on every request it makes for a submitted page, both the first download and the browser that renders it:

```text
Mozilla/5.0 (compatible; CalmLensBot/1.0; +https://calmlens.com/docs/guides/websites#crawler)
```

Match on the `CalmLensBot` token. The version number can change.

- **Bot protection:** sites behind a firewall or bot manager may block CalmLens, and a blocked page ends with `processing.status` `failed`. To let it through, add an allow rule for requests whose user agent contains `CalmLensBot`. In Cloudflare, that is a custom rule with the action Skip, or a Verified Bots exception.
- **No fixed IP addresses:** pages are loaded from a shared cloud network, so there is no IP range to allow. Allow by user agent.
- **Anyone can send that header.** Allow it only for pages that are safe to show publicly, and never to bypass authentication.
- **What it loads:** only the URL that was submitted, the files that page requests, and the images on it. It doesn't follow links or sign in.

Your own users' links are usually on sites you don't control, so some will be blocked. Treat `processing.status` `failed` on a website asset as "couldn't be checked", and decide whether to show the link without a decision.