Skip to main content

Self Hosting

Logflare can be self-hosted. As of now, only a single machine setup is supported.

Two different backends are supported:

  • BigQuery
  • PostgreSQL (experimental)

Docker-compose is the recommended way to manage single node deployments.

Limitations

Inviting team users and other team-related functionality is currently not supported, as Logflare self-hosted is currently intended for single-user experience only.

All browser authentication will be disabled when in single-tenant mode.

Configuration

Common Configuration

Env VarTypeDescription
LOGFLARE_SINGLE_TENANTBoolean, defaults to falseIf enabled, a singular user will be seeded. All browser usage will default to the user.
LOGFLARE_API_KEYstring, defaults to nilIf set, this API Key can be used for interacting with the Logflare API. API key will be automatically generated if not set.
LOGFLARE_SUPABASE_MODEBoolean, defaults to falseA special mode for Logflare, where Supabase-specific resources will be seeded. Intended for Suapbase self-hosted usage.
PHX_HTTP_PORTInteger, defaults to 4000Allows configuration of the HTTP server port.
DB_SCHEMAString, defaults to nilAllows configuration of the database schema to scope Logflare operations.
LOGFLARE_LOG_LEVELString, defaults to info.
Options: error,warn, info
Allows runtime configuration of log level.
LOGFLARE_NODE_HOSTstring, defaults to 127.0.0.1Sets node host on startup, which affects the node name logflare@<host>
LOGFLARE_LOGGER_METADATA_CLUSTERstring, defaults to nilSets global logging metadata for the cluster name. Useful for filtering logs by cluster name.
LOGFLARE_PUBSUB_POOL_SIZEInteger, defaults to 10Sets the number of Phoenix.PubSub.PG2 partitions to be created. Should be configured to the number of cores of your server for optimal multi-node performance.

BigQuery Backend Configuration

Env VarTypeDescription
GOOGLE_PROJECT_IDstring, requiredSpecifies the GCP project to use.
GOOGLE_PROJECT_NUMBERstring, requiredSpecifies the GCP project to use.
GOOGLE_DATASET_ID_APPENDstring, defaults to _prodThis allows customization of the dataset created in BigQuery.

PostgreSQL Backend Configuration

Env VarTypeDescription
POSTGRES_BACKEND_URLstring, requiredPostgreSQL connection string, for connecting to the database. User must have sufficient permssions to manage the schema.
POSTGRES_BACKEND_SCHEMAstring, optional, defaults to publicSpecifies the database schema to scope all operations.

BigQuery Setup

Pre-requisites

You will need a Google Cloud project with billing enabled in order to proceed.

The requirements for server startup are as follows after creating the project:

  • Project ID
  • Project number
  • A service account key

Setting up BigQuery Service Account

To ensure that you have sufficient permissions to insert into your Google Cloud BigQuery, ensure that you have created a service account with either:

  • BigQuery Admin role; or
  • The following permissions:
    • bigquery.datasets.create
    • bigquery.datasets.get
    • bigquery.datasets.getIamPolicy
    • bigquery.datasets.update
    • bigquery.jobs.create
    • bigquery.routines.create
    • bigquery.routines.update
    • bigquery.tables.create
    • bigquery.tables.delete
    • bigquery.tables.get
    • bigquery.tables.getData
    • bigquery.tables.update
    • bigquery.tables.updateData

We recommend setting the BigQuery Admin role, as it simplifies permissions setup.

Obtaining the BigQuery Service Account Key

In order for Logflare to connect sources to their relevant BigQuery tables, we would need to have a service account key that can sign the JWTs needed to authenticate with the Google Cloud APIs.

To obtain the BigQuery service account key after creating it, navigate to IAM > Service Accounts in the web console and click on the "Manage Keys" action option.

Service Account List Manage Keys Action

Thereafter, click on "Add Key" to create a new key. The key will be in a JSON format. Store this key securely on your host machine.

Add Key Button

You can also obtain the key via the gcloud cli by following the official documentation.

Deployment with Docker Compose

Using docker compose is the recommended method for self-hosting.

  1. Create the docker-compose.yml
services:
db:
image: postgres:13.4-alpine
environment:
POSTGRES_PASSWORD: postgres
POSTGRES_USER: postgres
POSTGRES_DATABASE: logflare_docker
ports:
- "5432:5432"
volumes:
- ./priv/wal.sql:/docker-entrypoint-initdb.d/wal.sql
- pg-data:/var/lib/postgresql/data
logflare:
image: supabase/logflare:1.0.1
ports:
- "4000:4000"
hostname: 127.0.0.1
environment:
- DB_DATABASE=logflare_docker
- DB_HOSTNAME=db
- DB_PORT=5432
- DB_PASSWORD=postgres
- DB_USERNAME=postgres
- LOGFLARE_SINGLE_TENANT=true
- LOGFLARE_API_KEY=my-cool-api-key

# Required for BigQuery backend
- GOOGLE_DATASET_ID_APPEND=_your_env
- GOOGLE_PROJECT_ID=logflare-docker-example
- GOOGLE_PROJECT_NUMBER=123123123213

# Required for Postgres backend
- POSTGRES_BACKEND_URL=postgresql://user:pass@host:port/db
- POSTGRES_BACKEND_SCHEMA=my_schema
volumes:
- type: bind
source: ${PWD}/.env
target: /tmp/.secrets.env
read_only: true
- type: bind
source: ${PWD}/gcloud.json
target: /opt/app/rel/logflare/bin/gcloud.json
read_only: true
depends_on:
- db
  1. Using the Service Account key that you had obtained under the pre-requisites section, move and rename the JSON file to gcloud.json in your working directory.

The directory structure should be as follows:

\
|- gcloud.json
|- docker-compose.yml
  1. Run docker-compose up -d and visit http://localhost:4000

Using an .env file

You can optionally use a .env file to manage your environemnts. You can base the file contents on this reference file

note

You cannot have comments in the env file as we load it at startup via xargs.

# ... the rest is the same
volumes:
# add in this bind bound. If you have a different name or location, update the source
- type: bind
source: ${PWD}/.env
target: /tmp/.secrets.env
read_only: true

The directory structure will now be as follows:

\
|- gcloud.json
|- .env
|- docker-compose.yml

Protecting the User Interface

When self-hosting, it is advised to protect the user interface with minimally basic HTTP authentication. How this is achieved is left to the self-hoster.

When applying such authentication rules, we recommend requiring all routes to be authenticated except for the following paths, as illustrated using glob patterns:

/api/**/*
/logs/**/*

the /logs path is for legacy reasons and is mostly for compatibility with older Logflare libraries.