Skip to main content

Configuration

You can make any configuration changes you need for your deepstream setup in the config.yml file or in the options object passed to the deepstream constructor when using the node API. Depending on the installation sources, the configuration can be found in

  • in the folder conf in the Node.js module directory if you installed with npm, or
  • in the folder conf after unpacking the standalone server binary.
info

The default server configuration can be found on the source code here

Macros

Before we start diving into config, lets look at the current macros we have in order to make our lives a little bit easier:

  • ${ENVIRONMENT_VARIABLE}

This is the usual replace environment variable into my config before running, useful in around 100% of deployment usecases.

  • file(relative/path)

This allows the server to binary to point to the file relative to your config file. This is pretty useful for global and binary installs.

  • fileLoad(relative/path)

This is more of a utility for us plugin developers, it automatically loads up the data and parses it (if json or yml). This means node developers can now just deal with writing everything as parsed data and leave loading / parsing and errors to deepstream startup.

General Configuration

In this section you can change general settings for each server in a cluster.

# general configuration with default values
serverName: UUID
showLogo: true
dependencyInitializationTimeout: 2000
exitOnFatalError: false
logLevel: INFO
#libDir: ../lib

serverName

Every server in a cluster of servers needs a unique name. You can add your own or set it to UUID to let deepstream auto-generate a unique ID.

Default: UUID

When starting, a server can show the deepstream logo. This setting is best left enabled.

Default: true

logLevel

The logLevel to use across the application. Sets at what level and above the server should log messages. Valid levels are DEBUG, INFO, WARN, ERROR, and OFF.

This value can be overwritten in the logger plugin options.

Default: INFO

dependencyInitializationTimeout

Sets how long deepstream will wait for dependencies to initialize.

Default:2000

exitOnFatalError

Exit if a fatal error occurs

Default: false

libDir

The directory where all the plugins reside, this is used in standalone binaries

Default: None, it assumes all plugins are installed via npm

Features configuration

This disables specific feature in deepstream, which is a more performant way than disabling via permissions and is also how telemetry figures out what features are enabled.

enabledFeatures:
record: true
event: true
rpc: true
presence: true

RPC Configuration

ackTimeout

Sets how long deepstream will wait for a RPC provider to acknowledge receipt of a request.

Default:1000

responseTimeout

Sets how long deepstream will wait for RPCs to complete before sending a response timeout error.

Default:10000

provideRequestorName

Send the rpc requestor userId by default to every rpc provider: { requestorName: string }

Default:false

provideRequestorData

Send the rpc requestor client data to every rpc provider: { requestorData: object }

Default:false

Record Configuration

cacheRetrievalTimeout

Sets how long deepstream will wait when retrieving values from the cache.

Default:1000

storageRetrievalTimeout

Sets how long deepstream will wait when retrieving values from the database.

Default:2000

storageExclusionPrefixes

A list of prefixes that, when a record starts with one of the prefixes the records data won't be stored in the db

Default: []

storageHotPathPrefixes

A list of prefixes that designate a record for direct writes to storage. When a correctly permissioned matching record is updated via setData(), it will be written directly to the cache and storage without a record transition. This can be up to twice as fast as updating a normal record using setData()

Default: []

Listening

shuffleProviders

Try finding a provider randomly rather than by the order they subscribed to.

Default: true

responseTimeout

The amount of time to wait for a provider to acknowledge or reject a listen request

Default: 500

rematchInterval

The amount of time before trying to reattempt finding matches for subscriptions. This is not a cheap operation so it's recommended to raise keep this at minutes rather then second intervals if you are experiencing heavy loads

Default: 30000

matchCooldown

The amount of time a server will refuse to retry finding a subscriber after a previously failed attempt. This is used to avoid servers constantly trying to find a match without a cooldown period

Default: 10000

Connection Endpoint Configuration

Deepstream (v5 and later) runs all websocket / HTTP services in one server to which they hook to provide their own functionality. This means you can run all the services at the same time on the same port if you want (for example text, binary and JSON). The idea is that EVERYTHING (minus MQTT) runs on port 6020, which makes deployments much easier.

httpServer:
type: default
options:
# url path for http health-checks, GET requests to this path will return 200 if deepstream is alive
healthCheckPath: /health-check
# -- CORS --
# if disabled, only requests with an 'Origin' header matching one specified under 'origins'
# below will be permitted and the 'Access-Control-Allow-Credentials' response header will be
# enabled
allowAllOrigins: true
# a list of allowed origins
origins:
- 'https://example.com'
# maximum allowed size of a POST request body, in bytes, defaults to 1 MB
maxMessageSize: 1048576
# Headers to copy over from websocket
headers:
- user-agent
ssl:
key: fileLoad(/path/to/sslKey)
cert: fileLoad(/path/to/sslCert)
ca: fileLoad(/path/to/caAuth)

OR

httpServer:
type: uws
options:
# url path for http health-checks, GET requests to this path will return 200 if deepstream is alive
healthCheckPath: /health-check
# -- CORS --
# if disabled, only requests with an 'Origin' header matching one specified under 'origins'
# below will be permitted and the 'Access-Control-Allow-Credentials' response header will be
# enabled
allowAllOrigins: true
# a list of allowed origins
origins:
- 'https://example.com'
# maximum allowed size of a POST request body, in bytes, defaults to 1 MB
maxMessageSize: 1048576
# Maximum length of allowed backpressure per socket when publishing or sending messages.
# Slow receivers with too high backpressure will be skipped until they catch up or timeout. Defaults to 1024 * 1024.
maxBackpressure: 1024*1024
# Headers to copy over from websocket
headers:
- user-agent
# Options required to create an ssl app
ssl:
key: file(ssl/key.pem)
cert: file(ssl/cert.pem)
## dhParams: ...
## passphrase: ...

The server can be configured with custom connection endpoints. You can supply as many as you want, each as any individual list entry below the connectionEndpoints key:

# Connection Endpoint Configuration
connectionEndpoints:
- type: ws-binary
options:
# url path websocket connections connect to
urlPath: /deepstream
# the amount of milliseconds between each ping/heartbeat message
heartbeatInterval: 30000
# the amount of milliseconds that writes to sockets are buffered
outgoingBufferTimeout: 0
# the maximum amount of bytes to buffer before flushing, stops the client from large enough packages
# to block its responsiveness
maxBufferByteSize: 100000

# Security
# should the server log invalid auth data, defaults to false
logInvalidAuthData: false
# amount of time a connection can remain open while not being logged in
unauthenticatedClientTimeout: 180000
# invalid login attempts before the connection is cut
maxAuthAttempts: 3
# maximum allowed size of an individual message in bytes
maxMessageSize: 1048576

- type: ws-text
options:
# url path websocket connections connect to
urlPath: /deepstream-v3
# the amount of milliseconds between each ping/heartbeat message
heartbeatInterval: 30000
# the amount of milliseconds that writes to sockets are buffered
outgoingBufferTimeout: 0
# the maximum amount of bytes to buffer before flushing, stops the client from large enough packages
# to block its responsiveness
maxBufferByteSize: 100000

# Security
# should the server log invalid auth data, defaults to false
logInvalidAuthData: false
# amount of time a connection can remain open while not being logged in
unauthenticatedClientTimeout: 180000
# invalid login attempts before the connection is cut
maxAuthAttempts: 3
# maximum allowed size of an individual message in bytes
maxMessageSize: 1048576

- type: ws-json
options:
# url path websocket connections connect to
urlPath: /deepstream-json
# the amount of milliseconds between each ping/heartbeat message
heartbeatInterval: 30000
# the amount of milliseconds that writes to sockets are buffered
outgoingBufferTimeout: 0
# the maximum amount of bytes to buffer before flushing, stops the client from large enough packages
# to block its responsiveness
maxBufferByteSize: 100000

# Security
# should the server log invalid auth data, defaults to false
logInvalidAuthData: false
# amount of time a connection can remain open while not being logged in
unauthenticatedClientTimeout: 180000
# invalid login attempts before the connection is cut
maxAuthAttempts: 3
# maximum allowed size of an individual message in bytes
maxMessageSize: 1048576

Websockets

The websocket endpoints share the following options:

urlPath

Sets which URL path Websocket connections should connect to.

Default: /deepstream

heartbeatInterval

The number of milliseconds between each ping/heartbeat message.

Default: 30000

unauthenticatedClientTimeout

The amount of time a connection can remain open while not being logged in.

Default: 180000

maxAuthAttempts

Invalid login attempts before the connection is cut.

Default: 3

logInvalidAuthData

Controls whether logs should contain the cleartext usernames and passwords of invalid login attempts.

Default: false

maxMessageSize

Sets the maximum message size allowed to be sent to the server (in bytes).

Default: 1048576

outgoingBufferTimeout

The amount of milliseconds that secondary writes to sockets are buffered. This means writes that are not realtime critical, which currently are either ACKs or non critical ERROR messages.

Default: 0

http

  - type: http
options:
# allow 'authData' parameter in POST requests, if disabled only token and OPEN auth is
# possible
allowAuthData: true
# enable the authentication endpoint for requesting userData and/or tokens
# note: a custom authentication handler is required for token generation
enableAuthEndpoint: false
# path for authentication requests
authPath: /api/auth
# path for POST requests
postPath: /api
# path for GET requests
getPath: /api
# should the server log invalid auth data, defaults to false
logInvalidAuthData: false
# http request timeout in milliseconds, defaults to 20000
requestTimeout: 20000

mqtt

  - type: mqtt
options:
# port for the mqtt server
port: 1883
# host for the mqtt server
host: 0.0.0.0
# timeout for idle devices
idleTimeout: 60000

Take a look at this blog post on the older docs for some mqtt usage examples

Custom Plugin Configuration

You can extend deepstream with plugins for connectors to other services, these are for logging, storage engines, caching layers, and message systems. To enable a plugin, uncomment the relevant category key underneath the plugins key. Each plugin type has a path or name, and a set of options.

# Plugin example using redis
plugins:
myCustomPlugin:
path: ./my-custom-plugin
options:
host: localhost
port: 5672

path

Set a path to a JavaScript file, node module or a folder with an index.js file which exports a constructor.

name

If you are using any of the official deepstream connectors, add the name of what the plugin connects to here, for example redis.

note

You can set path or name, but not both.

options

Under this key, add sub key/value pairs to set the configuration options that are passed to the plugin. Each plugin should mention what configuration options you can set.

Logger

deepstream uses by default a logger which prints out to stdout (errors and warnings to stderr). You can set these options for the default logger by using the same configuration style for the plugins:

logger:
type: default
options:
colors: true

colors

Sets whether the server's logs should output in color. This will look great in a console, but will leave color markers in log files if you redirect the output into a file.

Default: true

logLevel

If defined overwrites the general configuration logLevel value.

Authentication

In this section you can configure the authentication types the server uses.

You set the authentication type as a subkey in the auth key. The available authentication options are none, file, storage and http, each of them having different options which are described in the tutorials on Auth None, file-based authentication, storage-based authentication, HTTP authentication, and JWT authentication respectively.

You can set multiple authentication types simultaneously and the incoming connection will be validated against each of them until one succeeds or all fail. Authentication strategies will be queried in the same order they are declared on the configuration file.

#Authentication
auth:
- type: none

Default: none

Permissioning

In this section you can configure the permissioning. The key used for this section is permission and you can create your own custom permission handler or use a configuration file. To use the former method, select the option type: none and override the permissionHandler with the aid of the NodeJS server API. To use the latter method, set type: config and modify the option values below.

# Permissioning example with default values for config-based permissioning
permission:
type: config
options:
path: ./permissions.yml
maxRuleIterations: 3
cacheEvacuationInterval: 60000

path

Set the path to the file that declares permissions. This option is mandatory with type: config. The file can be in JSON, JavaScript, or YAML format. By default, deepstream ships with a permissions.yml permitting every action.

maxRuleIterations

The deepstream permissions model allows for some complex nested actions and queries. To prevent a performance hit you can limit the nesting level with this option.

Default: 3

cacheEvacuationInterval

The results of permission checks are cached to improve performance. Use this setting to change the time interval (in milliseconds) that the cache is regenerated.

Default: 60000

Storage and Cache

Storage and Cache plugins can be configured as follows, please look at storage and cache connectors for in depth tutorials how to configure them.

cache:
name: redis
options:
host: localhost
port: 6379

storage:
name: postgres
options:
user: postgres
database: postgres
password: mysecretpassword
host: 'localhost'
port: 5432

Default: in memory cache and no storage.

Monitoring

Set monitoring options. You can set multiple monitoring types simultaneously, combined with custom monitoring plugins.

monitoring:
- type: none

# # Allows monitoring stats to be requested via HTTP, useful for polling agents
# # such as LogStash
# - type: http
# options:
# url: /monitoring
# allowOpenPermissions: false
# headerKey: deepstream-password2
# headerValue: deepstream-secret

# # Logs monitoring stats, useful for kibana where you can visualize meta data
# - type: log
# options:
# logInterval: 30000
# monitoringKey: DEEPSTREAM_MONITORING

# # Custonm monitoring plugin example
# - path: ./custom-monitoring-plugin
# options:
# monitoringKey: CUSTOM_DEEPSTREAM_MONITORING

Default: none

Clusternode

Configuration options for deepstream clusters. Deepstream comes with two possible cluster modes: no cluster and vertical cluster in order to use all CPU cores. See here in order to use redis as cluster connector in multiple machine deployments.

clusterNode:
type: default
#name: vertical

Default: none

Lock registry

The lock registry is responsible for maintaing a single source of truth within the cluster, used mainly for issuing cluster wide locks when an operation that stretches over multiple nodes are required. For example, distributed listening requires a leader to drive the nodes in sequence, so issuing a lock prevents multiple nodes from assuming the lead.

Check the default server configuration for details, source code here

Cluster registry

This class maintains a list of all nodes that are currently present within the cluster. It provides status messages on a predefined interval and keeps track of incoming status messages.

Check the default server configuration for details, source code here

Cluster States

This class provides a generic mechanism that allows to maintain a distributed state amongst the nodes of a cluster. The state is an array of unique strings in arbitrary order. Whenever a string is added by any node within the cluster for the first time, an 'add' event is emitted. Whenever its removed by the last node within the cluster, a 'remove' event is emitted.

Check the default server configuration for details, source code here

Custom Plugins

Deepstream can be extended with any plugin via the plugin API. Custom plugins can be declared as follows:

plugins:
custom:
path: '...'