Skip to content

Configuring search

Every command should be ran as the pleroma user from it's home directory. For example if you are superuser, you would have to wrap the command in su pleroma -s $SHELL -lc "$COMMAND".

From source note about MIX_ENV

The mix command should be prefixed with the name of environment your Pleroma server is running in, usually it's MIX_ENV=prod

To use built-in search that has no external dependencies, set the search module to Pleroma.Activity:

config :pleroma, Pleroma.Search, module: Pleroma.Search.DatabaseSearch

While it has no external dependencies, it has problems with performance and relevancy.

QdrantSearch

This uses the vector search engine Qdrant to search the posts in a vector space. This needs a way to generate embeddings and uses the OpenAI API. This is implemented by several project besides OpenAI itself, including the python-based fastembed-server found in supplemental/search/fastembed-api.

The default settings will support a setup where both the fastembed server and Qdrant run on the same system as pleroma. To use it, set the search provider and run the fastembed server, see the README in supplemental/search/fastembed-api:

config :pleroma, Pleroma.Search, module: Pleroma.Search.QdrantSearch

Then, start the Qdrant server, see here for instructions.

You will also need to create the Qdrant index once by running mix pleroma.search.indexer create_index. Running mix pleroma.search.indexer index will retroactively index the last 100_000 activities.

Indexing and model options

To see the available configuration options, check out the QdrantSearch section in config/config.exs.

The default indexing option work for the default model (snowflake-arctic-embed-xs). To optimize for a low memory footprint, adjust the index configuration as described in the Qdrant docs. See also this blog post that goes into detail.

Different embedding models will need different vector size settings. You can see a list of the models supported by the fastembed server here, including their vector dimensions. These vector dimensions need to be set in the qdrant_index_configuration.

E.g, If you want to use sentence-transformers/all-MiniLM-L6-v2 as a model, you will not need to adjust things, because it and snowflake-arctic-embed-xs are both 384 dimensional models. If you want to use snowflake/snowflake-arctic-embed-l, you will need to adjust the size parameter in the qdrant_index_configuration to 1024, as it has a dimension of 1024.

When using a different model, you will need do drop the index and recreate it (mix pleroma.search.indexer drop_index and mix pleroma.search.indexer create_index), as the different embeddings are not compatible with each other.

Meilisearch

Note that it's quite a bit more memory hungry than PostgreSQL (around 4-5G for ~1.2 million posts while idle and up to 7G while indexing initially). The disk usage for this additional index is also around 4 gigabytes. Like RUM indexes, it offers considerably higher performance and ordering by timestamp in a reasonable amount of time. Additionally, the search results seem to be more accurate.

Due to high memory usage, it may be best to set it up on a different machine, if running pleroma on a low-resource computer, and use private key authentication to secure the remote search instance.

To use meilisearch, set the search module to Pleroma.Search.Meilisearch:

config :pleroma, Pleroma.Search, module: Pleroma.Search.Meilisearch

You then need to set the address of the meilisearch instance, and optionally the private key for authentication. You might also want to change the initial_indexing_chunk_size to be smaller if you're server is not very powerful, but not higher than 100_000, because meilisearch will refuse to process it if it's too big. However, in general you want this to be as big as possible, because meilisearch indexes faster when it can process many posts in a single batch.

config :pleroma, Pleroma.Search.Meilisearch, url: "http://127.0.0.1:7700/", private_key: "private key", initial_indexing_chunk_size: 100_000

Information about setting up meilisearch can be found in the official documentation. You probably want to start it with MEILI_NO_ANALYTICS=true environment variable to disable analytics. At least version 0.25.0 is required, but you are strongly advised to use at least 0.26.0, as it introduces the --enable-auto-batching option which drastically improves performance. Without this option, the search is hardly usable on a somewhat big instance.

Private key authentication (optional)

To set the private key, use the MEILI_MASTER_KEY environment variable when starting. After setting the master key, you have to get the private key, which is actually used for authentication.

./bin/pleroma_ctl search.meilisearch show-keys <your master key here>

mix pleroma.search.meilisearch show-keys <your master key here>

You will see a "Default Admin API Key", this is the key you actually put into your configuration file.

Initial indexing

After setting up the configuration, you'll want to index all of your already existing posts. Only public/unlisted posts are indexed. You'll only have to do it one time, but it might take a while, depending on the amount of posts your instance has seen. This is also a fairly RAM consuming process for meilisearch, and it will take a lot of RAM when running if you have a lot of posts (seems to be around 5G for ~1.2 million posts while idle and up to 7G while indexing initially, but your experience may be different).

The sequence of actions is as follows:

  1. First, change the configuration to use Pleroma.Search.Meilisearch as the search backend
  2. Restart your instance, at this point it can be used while the search indexing is running, though search won't return anything
  3. Start the initial indexing process (as described below with index), and wait until the task says it sent everything from the database to index
  4. Wait until everything is actually indexed (by checking with stats as described below), at this point you don't have to do anything, just wait a while.

To start the initial indexing, run the index command:

./bin/pleroma_ctl search.meilisearch index

mix pleroma.search.meilisearch index

This will show you the total amount of posts to index, and then show you the amount of posts indexed currently, until the numbers eventually become the same. The posts are indexed in big batches and meilisearch will take some time to actually index them, even after you have inserted all the posts into it. Depending on the amount of posts, this may be as long as several hours. To get information about the status of indexing and how many posts have actually been indexed, use the stats command:

./bin/pleroma_ctl search.meilisearch stats

mix pleroma.search.meilisearch stats

Clearing the index

In case you need to clear the index (for example, to re-index from scratch, if that needs to happen for some reason), you can use the clear command:

./bin/pleroma_ctl search.meilisearch clear

mix pleroma.search.meilisearch clear

This will clear all the posts from the search index. Note, that deleted posts are also removed from index by the instance itself, so there is no need to actually clear the whole index, unless you want all of it gone. That said, the index does not hold any information that cannot be re-created from the database, it should also generally be a lot smaller than the size of your database. Still, the size depends on the amount of text in posts.

ParadeDB

ParadeDB is a Postgres extension that provides BM25 full-text search. In Pleroma, it can be used as an external search backend by pointing Pleroma at a separate Postgres instance with the pg_search extension installed. Pleroma's integration suite tests pg_search 0.24.3 on PostgreSQL 15.

Pleroma will maintain a small pleroma_search_documents table in that database (via the existing search indexing queue) and run search queries against it. The indexed text includes status content, content warnings, and attachment descriptions.

Configuration

Set the search module to Pleroma.Search.ParadeDB and configure the ParadeDB database URL. These settings must be in static/runtime config (or PARADEDB_DATABASE_URL) so the dedicated ParadeDB Repo is started with the application. They cannot be migrated to or changed through ConfigDB.

config :pleroma, Pleroma.Search, module: Pleroma.Search.ParadeDB

config :pleroma, Pleroma.Search.ParadeDB, url: System.get_env("PARADEDB_DATABASE_URL"), table: "pleroma_search_documents", fuzzy_distance: 0

config :pleroma, Pleroma.Search.ParadeDB.Repo, pool_size: 2, prepare: :unnamed

When connecting over an untrusted network, enable certificate verification in the Repo configuration. For example:

config :pleroma, Pleroma.Search.ParadeDB.Repo, ssl: [ verify: :verify_peer, cacertfile: "/etc/pleroma/paradedb-ca.pem", server_name_indication: ~c"paradedb.example.com" ]

Initial rollout

The pg_search extension must already be available in the database. Creating the extension normally requires an administrative database role. The Pleroma runtime role requires CONNECT on the database, USAGE on the table's schema and the pdb schema, and SELECT, INSERT, UPDATE, and DELETE on the generated table. For example, after creating the table as its owner:

GRANT CONNECT ON DATABASE paradedb TO pleroma_search;
GRANT USAGE ON SCHEMA public, pdb TO pleroma_search;
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLE pleroma_search_documents TO pleroma_search;

Use this rollout order to prevent jobs from being handled by the previous backend:

  1. Install pg_search 0.24.3 on PostgreSQL 15 or another deliberately tested combination.
  2. Stop Pleroma so no search indexing jobs are running.
  3. Put the ParadeDB backend in static configuration.
  4. Create the table and BM25 index in a one-shot offline process using PARADEDB_DATABASE_URL with the table owner/setup credential.
  5. Apply the runtime grants shown above, set the service's PARADEDB_DATABASE_URL to the restricted runtime credential, and start Pleroma.
  6. Start the initial backfill and monitor the search_indexing queue.

Create the table and BM25 index once with:

PARADEDB_DATABASE_URL='OWNER_URL' PLEROMA_CTL_RPC_DISABLED=true \
  ./bin/pleroma_ctl search.indexer create_index

PARADEDB_DATABASE_URL='OWNER_URL' mix pleroma.search.indexer create_index

The task currently prints an error without returning a failing process status. Verify that it prints Index created before continuing.

Backfill

Run an initial indexing pass to enqueue existing posts:

./bin/pleroma_ctl search.indexer index

PARADEDB_DATABASE_URL='RUNTIME_URL' mix pleroma.search.indexer index

The command enqueues up to 100,000 Create activities by default. It uses keyset pagination, so new activities arriving during the backfill do not shift the remaining pages. The defaults and accepted options are:

  • --limit: maximum activities to enqueue in this run; defaults to 100000 and must be positive
  • --step: activities fetched from Pleroma's database per page; defaults to 1000 and must be positive
  • --chunk: jobs inserted into Oban per batch; defaults to 100 and must be positive
  • --before: resume before the checkpoint activity ID printed by an earlier run; the cursor is exclusive

The checkpoint is the last activity enqueued, not confirmation that its job succeeded. Repeat the command with its last --before checkpoint only when it reports that more activities remain. Inspect failed or discarded Oban jobs as well as waiting jobs before declaring the backfill complete.

The same operation can be run from an attached console while Pleroma is online:

Pleroma.Search.Backfill.run(limit: 10_000)

The returned next_cursor can be supplied as before when exhausted is false. The attached API also accepts limit: :infinity, but this can create an unbounded number of Oban rows; bounded runs are safer on large instances. An on_page callback may be supplied for progress reporting.

Note: Like the other external search backends, only public/unlisted Notes are indexed.

Set fuzzy_distance to 1 or 2 to allow typo-tolerant matching (2 is the maximum ParadeDB supports).

Indexing runs via the Oban search_indexing queue. Its stock configuration starts paused so the search health monitor can control it; verify that the queue is running before backfilling. A manually managed queue can be enabled with search_indexing: 10 in the Oban queues configuration instead of a paused: true entry. Monitor that queue until the backfill jobs have drained before evaluating search completeness.

Rebuilding

Stop Pleroma, or otherwise ensure no search indexing jobs are running, before rebuilding. While ParadeDB is still the configured backend, run drop_index, then create_index with the table owner/setup credential. Reapply runtime grants after recreating the table, restart Pleroma, and begin a new backfill. The DDL tasks print failures without returning a failing process status, so verify the exact Index dropped and Index created success messages. Dropping the table removes only derived search data; Pleroma activities remain in the primary database.

Rolling back

To retain the ParadeDB table for a later retry, configure Pleroma.Search.DatabaseSearch (or the previous external backend) in static config and restart Pleroma.

Before switching backends, inspect the search_indexing queue. Drain or cancel pending backfill jobs, or explicitly accept that they will run against the newly configured backend after restart.

To remove the table during rollback, first stop Pleroma and run drop_index with the table owner/setup credential while ParadeDB is still configured:

PARADEDB_DATABASE_URL='OWNER_URL' PLEROMA_CTL_RPC_DISABLED=true \
  ./bin/pleroma_ctl search.indexer drop_index

PARADEDB_DATABASE_URL='OWNER_URL' mix pleroma.search.indexer drop_index

Verify that the task prints Index dropped. Then change the backend in static config and restart. The ParadeDB environment variable may be removed after removing the ParadeDB stanza or switching to a backend that does not start its Repo. After switching away from ParadeDB, the generic drop_index task targets the newly configured backend and will not remove the ParadeDB table; remove it directly with the database owner instead.