Skip to main content
warning

This is a beta feature. If you would like to participate in the beta, please contact your account manager or support.

Getting started with custom metrics

Extend Lumar with your custom metrics and unlock the full power of extracting data from the web.

Using TypeScript/JavaScript you can create custom metrics that extract data from your pages. You can use Puppeteer to access the DOM and extract data or use the provided API to extract data from the DOM.

Custom metrics can be grouped into custom metric containers. Each custom metric container can contain multiple custom metrics.

Create a new project​

Generate a fresh TypeScript custom metric container project and register it with the API.

npx @deepcrawl/oreo@latest metric bootstrap my-metrics/

When bootstraping a new container you will be asked to provide a name. This name will be used to identify the container in the API and it needs to be globally unique.

You will also have a choice between DOM or Puppeteer container. Using Puppeteer requires the project in Lumar to be run with JS rendering enabled.

In case you want to extract metrics from images or style sheets, you can enable that as well.

You can also provide all of these options via CLI without getting prompted. Refer to our CLI docs. for all the available arguments.

npx @deepcrawl/oreo@latest metric bootstrap examples/bootstrap-example3 --name "MyGloballyUniqueMetricContainerName" --inputType=Puppeteer --resourceTypes=Document --description "Example metric extraction container"

After bootstrapping your container project, you will need to install dependencies with your package manager (npm, yarn, pnpm or other).

Change to the directory where you have bootstrapped your container project and run the following command.

npm install

Writing metrics extraction code​

Open the container project in your favorite editor. You will find a src/index.ts file with a sample metrics extraction script.

export interface IMetrics extends MetricScriptBasicOutput {
  url: string;
}

export const handler: MetricScriptHandler<IMetrics, IPuppeteerRequestContainerInput> = async (input, _context) => {
  return {
    url: input.page.url(),
  };
};

Your container needs to explicitly specify its return types. This is done by defining an interface that extends the MetricScriptBasicOutput interface, and referencing that interface name in .oreorc.json file under metricsTypeName key. (Make sure it's exported from the entrypoint file.)

{
  "id": "XXX",
  "handler": "handler",
  "entrypoint": "src/index.ts",
  "metricsTypeName": "IMetrics"
}

Your container can export one or more metrics. Each metric needs to have a unique name and a type.

Releasing new version of CustomMetricContainer​

Once you are happy with your container and want to release a new version of it, you need to build and upload it.

npm run build
npm run upload

At this point you have a container that is published and ready to be linked with a project.

Adding custom metrics to a Lumar project​

Running a metric link command without any arguments will fetch your default account's projects and CustomMetricContainer ID from .oreorc.json.

npm run oreo metric link
linking to multiple projects at the same time
npm run oreo metric link -- --projectIds 123456,12345,123454

Fetching metrics via Single Page Requester​

Before commiting to a full crawl, you can test your custom metrics using the Single Page Requester.

npm run oreo project request-custom-metrics -- --projectId 123456 --url http://example.com/

This will output a table with your custom metrics.

Fetching the metrics from the crawl​

Running a crawl​

Once a CustomMetricContainer is linked to a project, the next time a crawl is run it will inherit the container and extract custom metrics.

You can start the crawl as you would usually would via the UI, or start crawling from the CLI.

npm run oreo crawl create -- --projectId 123456

Fetching the metrics​

Once the crawl finishes, you can access custom metrics through Graph-API Explorer or using Analyze UI.

query FetchCustomMetrics($reportInput: GetReportInputType!) {
  getReport(input: $reportInput) {
    crawlUrls(first: 100) {
      nodes {
        url
        customMetrics
      }
    }
  }
}

Try in explorer

Advanced topics​

Providing extra metadata for custom metrics for UI​

You can provide display names for your metrics in the .oreorc.json file. This will make it easier to understand what the metric represents in the UI. You can also provide names for auto generated __count metrics.

{
  "metricsMetadata": {
    "pageTitle": {
      "title": "Page Title",
      "description": "The title of the page.",
      "type": "string"
    },
    "myObjects": {
      "title": "My Object With Specific Order",
      "description": "My Object With Specific Order",
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "aString": {
            "title": "String Metric",
            "type": "string"
          },
          "cBoolean": {
            "title": "Boolean Metric",
            "description": "You can also provide description for specific properties in object arrays.",
            "type": "boolean"
          },
          "bNumber": {
            "title": "Number Metric",
            "type": "number"
          },
          "dateString": {
            "title": "My Date Field",
            "type": "string",
            "format": "date-time"
          }
        }
      }
    },
    "extract": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "title": "Extracts",
      "description": "The extracted values"
    },
    "pageSize": {
      "title": "Page Size",
      "description": "Page size in bytes.",
      "type": "number",
      "format": "bytes"
    },
    "myFloat": {
      "type": "number"
    },
    "myInt": {
      "type": "number",
      "format": "integer"
    },
    "extract__count": {
      "title": "Extracts Count",
      "description": "Count of extracted values (auto-generated)"
    }
  }
}

You can also use JSDoc to provide metadata for your metrics​

export interface MyMetrics extends MetricScriptBasicOutput {
  /**
   * @title Page Title
   * @description The title of the page.
   */
  pageTitle: string;
  /**
   * Order of properties will be kept in the UI.
   *
   * @title My Object With Specific Order
   * @description My Object With Specific Order
   */
  myObjects: Array<{
    /**
     * @title String Metric
     */
    aString: string;
    /**
     * @title Boolean Metric
     * @description You can also provide description for specific properties in object arrays.
     */
    cBoolean?: boolean;
    /**
     * @title Number Metric
     */
    bNumber: number;
    /**
     * @title My Date Field
     * @format date-time
     */
    dateString: string;
  }>;
  url: string;
  extract: string[];
  /**
   * @title Page Size
   * @description Page size in bytes.
   * @format bytes
   */
  pageSize: number;
  myFloat: number;
  /**
   * @format integer
   */
  myInt: number;
}

Native dependencies​

You can use native dependencies in your custom metrics by including them in the externalPackages array in the .oreorc.json file. You also need to have them in your package.json dependencies so we can install correct version.

{
  "externalPackages": ["sharp"]
}

Secrets in custom metric containers​

Sometimes there is a need to pass in a secret into your Container, or other variables which are unique to a project. (For example OPENAI_APIKEY.) You can set secrets for your CustomMetricContainer which will be accessible via environment variables.

const openaiApiKey = process.env["OPENAI_APIKEY"];

You can set the secret from the CLI

npm run oreo metric secret set -- --name OPENAI_APIKEY --projectId 123456 --value "mySecretKey"

or using GraphQL API directly

mutation setCustomMetricContainerProjectSecret(
  $input: SetCustomMetricContainerProjectSecretInput!
) {
  setCustomMetricContainerProjectSecret(input: $input) {
    customMetricContainerProjectSecret {
      name
    }
  }
}

Try in explorer

info

Secrets are set on the project level. So if you have multiple projects using the same container, each project will need to have the secrets set.

CI/CD integration​

You can integrate your custom metric container with your CI/CD pipeline. For example, you can use GitHub Actions to build and upload your container. For this you will need to login to the CLI programmatically without user interaction with a Lumar ACCOUNT_ID, API_KEY_ID and API_KEY_SECRET.

To create API_KEY_ID and API_KEY_SECRET you can use the CLI command locally or do so via Lumar Accounts app where you can also find ACCOUNT_ID.

npm run oreo user-key create

Once you have all secrets you can use them in your CI/CD worflow file.

npm run oreo login -- --id ${{ secrets.API_KEY_ID }} --secret ${{ secrets.API_KEY_SECRET }} --accountId ${{ secrets.ACCOUNT_ID }}
npm run build
npm run upload

Programmatic access​

If you would like to run your custom metric container programmatically, you can do so using @deepcrawl/oreo-api-sdk package.

npm install @deepcrawl/oreo-api-sdk

Example Jest usage​

import { SinglePageRequesterClient } from "@deepcrawl/oreo-api-sdk";
import { IMetrics } from "..";

const creds = { userKeyId: process.env.LUMAR_ID, userKeySecret: process.env.LUMAR_SECRET };

jest.setTimeout(60000);

const projectId = 123456;
let client: SinglePageRequesterClient;

beforeAll(async () => {
  client = await SinglePageRequesterClient.create(creds);
});

describe("errors", () => {
  it("blog", async () => {
    const url = "https://www.example.com/blog/";
    const { customMetrics } = await client.requestProjectCustomMetrics<IMetrics>(projectId, url);
    expect(customMetrics["99"].hasFailure).toBe(false);
    expect(customMetrics["99"].failuresCount).toBe(0);
  });

  it("careers", async () => {
    const url = "https://www.example.com/about/careers/";
    const { customMetrics } = await client.requestProjectCustomMetrics<IMetrics>(projectId, url);
    expect(customMetrics["99"].hasFailure).toBe(true);
    expect(customMetrics["99"].failuresCount).toBe(64);
  });
});

Container failures​

If your container fails to extract metrics and returns an error, the information is stored as a separate metric containerExecutionFailures. Failing containers will not stop the crawl.

Supported types for filtering​

Even though custom metric containers can extract and store almost any data type, not all of them will be queryable via the API in our UI.

Supported filterable types are:

  • boolean
  • number
  • number[]
  • string
  • string[]

Automatic __count metrics for arrays​

If your metric returns an array, we will automatically generate a metric that counts the number of elements in the array. This metric will have the same name as the original metric with __count suffix.

Universal container​

Project created from running bootstrap command will have a specific type either for DOM or Puppeteer specified, but you can create a universal container that can handle both, using input.inputType, input.resourceType to narrow down type during extraction.

export interface IMetrics extends MetricScriptBasicOutput {
  isImage: boolean;
  wasJsRenderingEnabled: boolean;
}

export const myHandler: MetricScriptHandler<IMetrics> = input => {
  if (input.resourceType === "document") {
    if (input.inputType === "dom") {
      // do extractions without puppeteer
      return {
        wasJsRenderingEnabled: false,
      };
    } else if (input.inputType === "puppeteer") {
      // do extractions with puppeteer
      return {
        wasJsRenderingEnabled: true,
      };
    }
  } else if (input.resourceType === "image") {
    // do extractions for images
    return {
      isImage: true,
      wasJsRenderingEnabled: input.inputType === "puppeteer",
    };
  }
};