Skip to main content
warning

This is a beta feature. If you would like to participate in the beta, please contact your account manager or support.

Getting started with custom metrics

Extend Lumar with your custom metrics and unlock the full power of extracting data from the web.

Using TypeScript/JavaScript you can create custom metrics that extract data from your pages. You can use Puppeteer to access the DOM and extract data or use the provided API to extract data from the DOM.

Custom metrics can be grouped into custom metric containers. Each custom metric container can contain multiple custom metrics.

Create a new projectโ€‹

Generate a fresh TypeScript custom metric container project and register it with the API.

bootstrap into new directory
$ npx @deepcrawl/oreo@latest metric bootstrap my-metrics/

replace my-metrics/ with your directory name or use . for current directory.

bootstrap into current directory
$ npx @deepcrawl/oreo@latest metric bootstrap .

When bootstraping a new container you will be asked to provide a name. This name will be used to identify the container in the API and it needs to be globally unique.

You will also have a choice between DOM or Puppeteer container. Using Puppeteer requires the project in Lumar to be run with JS rendering enabled.

In case you want to extract metrics from images or style sheets, you can enable that as well.

You can also provide all of these options via CLI without getting prompted. Refer to our CLI docs. for all the available arguments.

providing all options via CLI
$ npx @deepcrawl/oreo@latest metric bootstrap examples/bootstrap-example3 --name "MyGloballyUniqueMetricContainerName" --inputType=Puppeteer --resourceTypes=Document --description "Example metric extraction container"

After bootstrapping your container project, you will need to install dependencies with your package manager (npm, yarn, pnpm or other).

Change to the directory where you have bootstrapped your container project and run the following command.

$ npm install

Writing metrics extraction codeโ€‹

Open the container project in your favorite editor. You will find a src/index.ts file with a sample metrics extraction script.

export interface IMetrics extends MetricScriptBasicOutput {
url: string;
}

export const handler: MetricScriptHandler<IMetrics, IPuppeteerRequestContainerInput> = async (input, _context) => {
return {
url: input.page.url(),
};
};

Your container needs to explicitly specify its return types. This is done by defining an interface that extends the MetricScriptBasicOutput interface, and referencing that interface name in .oreorc.json file under metricsTypeName key. (Make sure it's exported from the entrypoint file.)

{
"id": "XXX",
"handler": "handler",
"entrypoint": "src/index.ts",
"metricsTypeName": "IMetrics"
}

Your container can export one or more metrics. Each metric needs to have a unique name and a type.

Releasing new version of CustomMetricContainerโ€‹

Once you are happy with your container and want to release a new version of it, you need to build and upload it.

npm run build
npm run upload

At this point you have a container that is published and ready to be linked with a project.

Adding custom metrics to a Lumar projectโ€‹

Running a metric link command without any arguments will fetch your default account's projects and CustomMetricContainer ID from .oreorc.json.

$ npm run oreo metric link
linking to multiple projects at the same time
$ npm run oreo metric link -- --projectIds 123456,12345,123454

Fetching metrics via Single Page Requesterโ€‹

Before commiting to a full crawl, you can test your custom metrics using the Single Page Requester.

$ npm run oreo project request-custom-metrics -- --projectId 123456 --url http://example.com/

This will output a table with your custom metrics.

Fetching the metrics from the crawlโ€‹

Running a crawlโ€‹

Once a CustomMetricContainer is linked to a project, the next time a crawl is run it will inherit the container and extract custom metrics.

You can start the crawl as you would usually would via the UI, or start crawling from the CLI.

$ npm run oreo crawl create -- --projectId 123456

Fetching the metricsโ€‹

Once the crawl finishes, you can access custom metrics through Graph-API Explorer or using Analyze UI.

query FetchCustomMetrics($reportInput: GetReportInputType!) {
getReport(input: $reportInput) {
crawlUrls(first: 100) {
nodes {
url
customMetrics
}
}
}
}

Try in explorer

Advanced topicsโ€‹

Native dependenciesโ€‹

You can use native dependencies in your custom metrics by including them in the externalPackages array in the .oreorc.json file. You also need to have them in your package.json dependencies so we can install correct version.

{
"externalPackages": ["sharp"]
}

Secrets in custom metric containersโ€‹

Sometimes there is a need to pass in a secret into your Container, or other variables which are unique to a project. (For example OPENAI_APIKEY.) You can set secrets for your CustomMetricContainer which will be accessible via environment variables.

const openaiApiKey = process.env["OPENAI_APIKEY"];

You can set the secret from the CLI

npm run oreo metric secret set -- --name OPENAI_APIKEY --projectId 123456 --value "mySecretKey"

or using GraphQL API directly

mutation setCustomMetricContainerProjectSecret(
$input: SetCustomMetricContainerProjectSecretInput!
) {
setCustomMetricContainerProjectSecret(input: $input) {
customMetricContainerProjectSecret {
name
}
}
}

Try in explorer

info

Secrets are set on the project level. So if you have multiple projects using the same container, each project will need to have the secrets set.

CI/CD integrationโ€‹

You can integrate your custom metric container with your CI/CD pipeline. For example, you can use GitHub Actions to build and upload your container. For this you will need to login to the CLI programmatically without user interaction with a Lumar ACCOUNT_ID, API_KEY_ID and API_KEY_SECRET.

To create API_KEY_ID and API_KEY_SECRET you can use the CLI command locally or do so via Lumar Accounts app where you can also find ACCOUNT_ID.

npm run oreo user-key create

Once you have all secrets you can use them in your CI/CD worflow file.

npm run oreo login -- --id ${{ secrets.API_KEY_ID }} --secret ${{ secrets.API_KEY_SECRET }} --accountId ${{ secrets.ACCOUNT_ID }}
npm run build
npm run upload

Programmatic accessโ€‹

If you would like to run your custom metric container programmatically, you can do so using @deepcrawl/oreo-api-sdk package.

npm install @deepcrawl/oreo-api-sdk

Example Jest usageโ€‹

import { SinglePageRequesterClient } from "@deepcrawl/oreo-api-sdk";
import { IMetrics } from "..";

const creds = { userKeyId: process.env.LUMAR_ID, userKeySecret: process.env.LUMAR_SECRET };

jest.setTimeout(60000);

const projectId = 123456;
let client: SinglePageRequesterClient;

beforeAll(async () => {
client = await SinglePageRequesterClient.create(creds);
});

describe("errors", () => {
it("blog", async () => {
const url = "https://www.example.com/blog/";
const { customMetrics } = await client.requestProjectCustomMetrics<IMetrics>(projectId, url);
expect(customMetrics["99"].hasFailure).toBe(false);
expect(customMetrics["99"].failuresCount).toBe(0);
});

it("careers", async () => {
const url = "https://www.example.com/about/careers/";
const { customMetrics } = await client.requestProjectCustomMetrics<IMetrics>(projectId, url);
expect(customMetrics["99"].hasFailure).toBe(true);
expect(customMetrics["99"].failuresCount).toBe(64);
});
});

Container failuresโ€‹

If your container fails to extract metrics and returns an error, the information is stored as a separate metric containerExecutionFailures. Failing containers will not stop the crawl.

Supported types for filteringโ€‹

Even though custom metric containers can extract and store almost any data type, not all of them will be queryable via the API in our UI.

Supported filterable types are:

  • boolean
  • number
  • number[]
  • string
  • string[]

Automatic __count metrics for arraysโ€‹

If your metric returns an array, we will automatically generate a metric that counts the number of elements in the array. This metric will have the same name as the original metric with __count suffix.

Universal containerโ€‹

Project created from running bootstrap command will have a specific type either for DOM or Puppeteer specified, but you can create a universal container that can handle both, using input.inputType, input.resourceType to narrow down type during extraction.

export interface IMetrics extends MetricScriptBasicOutput {
isImage: boolean;
wasJsRenderingEnabled: boolean;
}

export const myHandler: MetricScriptHandler<IMetrics> = input => {
if (input.resourceType === "document") {
if (input.inputType === "dom") {
// do extractions without puppeteer
return {
wasJsRenderingEnabled: false,
};
} else if (input.inputType === "puppeteer") {
// do extractions with puppeteer
return {
wasJsRenderingEnabled: true,
};
}
} else if (input.resourceType === "image") {
// do extractions for images
return {
isImage: true,
wasJsRenderingEnabled: input.inputType === "puppeteer",
};
}
};