Index a custom data source

Bring an unsupported source into Glean with the Indexing API — documents, permissions, and people — and see it live in search. This recipe seeds the Acme corpus every other recipe searches.

~1 hrIntermediate

Problem

Most of what your company knows lives somewhere Glean doesn't crawl out of the box — an internal wiki, a service catalog, a bespoke database. The Indexing API brings any of it in, with real per-document permissions from day one, so search results respect who's allowed to see what.

Architecture

Acme corpusJSON documents + ACLs

Indexing SDKbulk index + permissions

Glean indexsearchable in minutes

Prerequisites

Admin access to create a custom datasource and Indexing API token

Python 3.10+ (uses glean-indexing-sdk)

Steps

1

Scaffold the project

connector.py resolves the corpus via a relative path two levels up (Path(__file__).parent.parent.parent / "acme-corpus") — scaffold both directories preserving that same nesting, not flattened into one directory.

mkdir -p index-custom-source/recipes/index-custom-source index-custom-source/acme-corpus
npx tiged --mode=git gleanwork/glean-cookbook/recipes/index-custom-source index-custom-source/recipes/index-custom-source
npx tiged --mode=git gleanwork/glean-cookbook/acme-corpus index-custom-source/acme-corpus

2

Install dependencies

cd index-custom-source/recipes/index-custom-source && pip install -r requirements.txt

3

Set credentials

Export GLEAN_INDEXING_API_TOKEN and GLEAN_SERVER_URL — this recipe has no .env.example, it reads directly from the environment (see indexing-token in cookbook-conventions).

4

Run it

Registers the custom datasource and bulk-indexes documents/people with real per-document permissions — never allow-all.

python seed.py

5

Verify

Search "Who owns the payments-service catalog entry?" as yourself and as a restricted test user — the restricted user must not see Acme-HR-only documents.

Take it further

Add incremental sync by implementing get_data(since=...) and calling index_data(mode=IndexingMode.INCREMENTAL).

Index structured entities (people, assets) alongside documents with BasePeopleConnector.

Watch Datasource Debugging to confirm documents and permissions landed as expected.

Scaffold starter code

Copies a prompt your AI assistant can build from.

At a glance

SurfacesIndexing API, Connector SDK

StatusProduction pattern

Time~1 hr

Required scopes

Indexing API token (datasource-scoped)

Code assets

Custom connector indexing the Acme corpus with per-document permissionsThe seed dataset itself