Website moderation

Send the URL of a public web page and CalmLens loads it in a browser, extracts the text and images, and moderates each one. The page becomes a website asset, and what it contains becomes child assets. You get one decision for the page and a report for every piece of extracted content.

Website moderationPreview

Submit a page

Upload the page with a remote URL upload. A URL that returns text/html is stored as a website asset.

const response = await client.assets.create({
  "name": "landing-page",
  "url": "https://example.com"
});

The URL must use HTTP or HTTPS and be reachable from the public internet. While the page loads, the browser also refuses its requests to private addresses.

Submitted URLResult
A private IP address, a localhost name or URL credentialsRejected with 400 remote-url/invalid
Answers with an error statusRefused with 422 remote-url/unavailable

Loading and moderating a page takes longer than a single image. Use wait=false to get an asset ID straight away, with classification.status processing, then read the decision from a webhook:

EventSent when
asset.allowed, asset.limited, asset.age-restricted, asset.blocked, asset.purgedThe page has a decision.
asset.failedThe page can't be processed.

See Waiting for the decision.

What CalmLens extracts

CalmLens opens the page in a desktop browser 1280 pixels wide, waits for the document to load and the network to go quiet (for up to ten seconds, so pages that poll or stream are still read), then scrolls through the page so that lazily loaded images and content load, and reads the rendered result. Pages that build their content in JavaScript are therefore moderated as visitors see them. Loading gives up after 30 seconds.

ExtractedStored as
The rendered HTMLThe website asset's content
Screenshots of the whole page, in tiles 1280 pixels wide and up to 1600 pixels tallimage children. The first tile is also the asset's rendered image and preview (content.rendered and content.preview)
All visible text, with scripts and styles removedA plaintext-document child named after the page title
Images the page loaded, inline (data: and blob:) images, the other images a <picture> offers, and inline or embedded SVG graphicsimage children

The screenshot children cover everything visible on the page, including CSS backgrounds, canvas drawings and video frames. Images are moderated as the browser received them, with the page's cookies and referrer, so an image a site only serves to its own pages is still checked. An image that appears more than once is moderated once, and tracking pixels (2 × 2 or smaller) are left out.

A page is moderated whole or not at all. It ends with classification.status failed and no children, rather than being moderated in part, when:

The pageclassification.failureReason
Needs more than 10 screenshot tiles: for a page 1280 pixels wide, taller than 16,000 pixels, including pages that keep loading more as you scrolltoo-many-items
Shows more than 50 images besides its screenshots, or more than 40 MiB of images in alltoo-many-items
Shows an image over 10 MiBfile-too-large
Has an image that failed to load, returned an error or wasn't an imageprocessing-error

Split long pages and large galleries, or submit their images directly.

Children inherit the page's processing options, folder and visibility, so a feature such as OCR that you enable on the upload also runs on the page's images. They are moderated with the same policy as the page.

Reading the result

The website asset has no report of its own. Its status comes from its children, as described in Child assets: a blocked child blocks the page, and a child that failed processing fails it.

  1. Read the page's classification.moderation.status for the overall decision.
  2. List its children, found by parentId, to see which text or image caused it.
  3. Read each child's classification.moderation as described in Reading classification results.

The page's content is the HTML as it rendered when it was submitted. Later changes to the live page aren't picked up; submit the URL again to moderate it again.

Limits and coverage

  • Image moderation considers readable text directly. Optional OCR extracts that text onto each image asset and adds a separate text moderation pass.
  • If the first download succeeds but the browser then gets an HTTP error instead of the page's content, the page ends with classification.status failed and no children, rather than being allowed, as a page blocked by bot protection does. Content behind a login or paywall is moderated as the page CalmLens receives. See Crawler.
  • The page is moderated as it rendered for one visitor, in a desktop browser with the default light colour scheme. Content a site shows only on other screen sizes, to signed-in visitors or after an interaction such as a click is not captured, apart from the other images a <picture> offers.
  • A site can still tell some automated browsers apart and show them a different page from what visitors see. Treat a website decision as covering the page CalmLens received when it was submitted.
  • Links are not followed. Submit each page you want moderated.

Restricting domains and pace

Both settings are in Project settings in the portal, and apply to the whole project.

  • Accepted domains: allow or block submissions from specified domains, such as example.com. A domain also covers its subdomains. The rule applies to every file and website submitted by URL, to each redirect the download follows, and to every page the browser is sent to while loading a website. The rule decides which pages are loaded, not what a page shows: the browser still loads images and scripts a page pulls in from refused domains, and those images are moderated as children like any other. A refused URL returns 403 remote-url/not-allowed, and a website that redirects to a refused domain ends with classification.status failed and no children.
  • Website submissions per minute: every website is loaded in a full browser, so each project can submit up to 60 a minute. You can lower that number, not raise it. Over the limit, the request returns 429 upload/website-rate-limit-reached with a Retry-After header giving the seconds until the oldest submission in the last minute ages out. Only websites count, not images or other files submitted by URL. Retrying a failed website is refused while the project is at its limit, but a retry doesn't count as a submission.

Crawler

CalmLens identifies itself on the first download of a submitted page, which checks that the URL is reachable, and on every request link safety makes:

Mozilla/5.0 (compatible; CalmLensBot/1.0; +https://calmlens.com/docs/guides/websites#crawler)

Match on the CalmLensBot token. The version number can change.

The browser that renders the page, and the images and files it loads, use the user agent of an ordinary desktop Chrome instead. A site that recognised CalmLens could show it a harmless page and its visitors something else, so the browser doesn't announce itself.

  • Bot protection: sites behind a firewall or bot manager may block CalmLens, and a blocked page ends with classification.status failed. The first download can be allowed by its CalmLensBot user agent; the browser can't be told apart from other Chrome visitors by its user agent, so a page that only lets known crawlers through can't be moderated.
  • No fixed IP addresses: pages are loaded from a shared cloud network, so there is no IP range to allow.
  • Anyone can send that header. Allow it only for pages that are safe to show publicly, and never to bypass authentication.
  • What it loads: only the URL that was submitted, the files that page requests, and the images on it. It doesn't follow links or sign in.
  • Link safety: when a policy acts on links, CalmLens also requests each link in an asset, and each address it redirects to, with the same user agent. These requests read only the redirect or the start of the page; they don't load the page's files or run its scripts. See Link safety.

Your own users' links are usually on sites you don't control, so some will be blocked. Treat classification.status failed on a website asset as "couldn't be checked", and decide whether to show the link without a decision.

Last updated

Signing in

Signing out

Cookies on CalmLens

We use essential cookies for sign-in and security. With your permission, functional cookies remember your preferences and Google Analytics measures website use and signups. Optional ad measurement cookies link Google ad clicks to signups. Personalised advertising stays off. Manage cookies