Skip to main content

Configuration

You can make any configuration changes you need for your deepstream setup in the config.yml file or in the options object passed to the deepstream constructor when using the node API. Depending on the installation sources, the configuration can be found in

  • in the folder conf in the Node.js module directory if you installed with npm, or
  • in the folder conf after unpacking the standalone server binary.
info

The default server configuration can be found on the source code here

Macros​

Before we start diving into config, lets look at the current macros we have in order to make our lives a little bit easier:

  • ${ENVIRONMENT_VARIABLE}

This is the usual replace environment variable into my config before running, useful in around 100% of deployment usecases.

  • file(relative/path)

This allows the server to binary to point to the file relative to your config file. This is pretty useful for global and binary installs.

  • fileLoad(relative/path)

This is more of a utility for us plugin developers, it automatically loads up the data and parses it (if json or yml). This means node developers can now just deal with writing everything as parsed data and leave loading / parsing and errors to deepstream startup.

General Configuration​

In this section you can change general settings for each server in a cluster.

# general configuration with default values
serverName: UUID
showLogo: true
dependencyInitializationTimeout: 2000
exitOnFatalError: false
logLevel: INFO
#libDir: ../lib

serverName​

Every server in a cluster of servers needs a unique name. You can add your own or set it to UUID to let deepstream auto-generate a unique ID.

Default: UUID

When starting, a server can show the deepstream logo. This setting is best left enabled.

Default: true

logLevel​

The logLevel to use across the application. Sets at what level and above the server should log messages. Valid levels are DEBUG, INFO, WARN, ERROR, and OFF.

This value can be overwritten in the logger plugin options.

Default: INFO

dependencyInitializationTimeout​

Sets how long deepstream will wait for dependencies to initialize.

Default:2000

exitOnFatalError​

Exit if a fatal error occurs

Default: false

libDir​

The directory where all the plugins reside, this is used in standalone binaries

Default: None, it assumes all plugins are installed via npm

Features configuration​

This disables specific feature in deepstream, which is a more performant way than disabling via permissions and is also how telemetry figures out what features are enabled.

enabledFeatures:
  record: true
  event: true
  rpc: true
  presence: true

RPC Configuration​

ackTimeout​

Sets how long deepstream will wait for a RPC provider to acknowledge receipt of a request.

Default:1000

responseTimeout​

Sets how long deepstream will wait for RPCs to complete before sending a response timeout error.

Default:10000

provideRequestorName​

Send the rpc requestor userId by default to every rpc provider: { requestorName: string }

Default:false

provideRequestorData​

Send the rpc requestor client data to every rpc provider: { requestorData: object }

Default:false

Record Configuration​

cacheRetrievalTimeout​

Sets how long deepstream will wait when retrieving values from the cache.

Default:1000

storageRetrievalTimeout​

Sets how long deepstream will wait when retrieving values from the database.

Default:2000

storageExclusionPrefixes​

A list of prefixes that, when a record starts with one of the prefixes the records data won't be stored in the db

Default: []

storageHotPathPrefixes​

A list of prefixes that designate a record for direct writes to storage. When a correctly permissioned matching record is updated via setData(), it will be written directly to the cache and storage without a record transition. This can be up to twice as fast as updating a normal record using setData()

Default: []

Listening​

shuffleProviders​

Try finding a provider randomly rather than by the order they subscribed to.

Default: true

responseTimeout​

The amount of time to wait for a provider to acknowledge or reject a listen request

Default: 500

rematchInterval​

The amount of time before trying to reattempt finding matches for subscriptions. This is not a cheap operation so it's recommended to raise keep this at minutes rather then second intervals if you are experiencing heavy loads

Default: 30000

matchCooldown​

The amount of time a server will refuse to retry finding a subscriber after a previously failed attempt. This is used to avoid servers constantly trying to find a match without a cooldown period

Default: 10000

Connection Endpoint Configuration​

Deepstream (v5 and later) runs all websocket / HTTP services in one server to which they hook to provide their own functionality. This means you can run all the services at the same time on the same port if you want (for example text, binary and JSON). The idea is that EVERYTHING (minus MQTT) runs on port 6020, which makes deployments much easier.

httpServer:
  type: default
  options:
    # url path for http health-checks, GET requests to this path will return 200 if deepstream is alive
    healthCheckPath: /health-check
    # -- CORS --
    # if disabled, only requests with an 'Origin' header matching one specified under 'origins'
    # below will be permitted and the 'Access-Control-Allow-Credentials' response header will be
    # enabled
    allowAllOrigins: true
    # a list of allowed origins
    origins:
      - 'https://example.com'
    # maximum allowed size of a POST request body, in bytes, defaults to 1 MB
    maxMessageSize: 1048576
    # Headers to copy over from websocket
    headers:
      - user-agent
    ssl:
      key: fileLoad(/path/to/sslKey)
      cert: fileLoad(/path/to/sslCert)
      ca: fileLoad(/path/to/caAuth)

OR

httpServer:
  type: uws
  options:
    # url path for http health-checks, GET requests to this path will return 200 if deepstream is alive
    healthCheckPath: /health-check
    # -- CORS --
    # if disabled, only requests with an 'Origin' header matching one specified under 'origins'
    # below will be permitted and the 'Access-Control-Allow-Credentials' response header will be
    # enabled
    allowAllOrigins: true
    # a list of allowed origins
    origins:
      - 'https://example.com'
    # maximum allowed size of a POST request body, in bytes, defaults to 1 MB
    maxMessageSize: 1048576
    # Maximum length of allowed backpressure per socket when publishing or sending messages.
    # Slow receivers with too high backpressure will be skipped until they catch up or timeout. Defaults to 1024 * 1024.
    maxBackpressure: 1024*1024
    # Headers to copy over from websocket
    headers:
      - user-agent
    # Options required to create an ssl app
    ssl:
      key: file(ssl/key.pem)
      cert: file(ssl/cert.pem)
    ##  dhParams: ...
    ##  passphrase: ...

The server can be configured with custom connection endpoints. You can supply as many as you want, each as any individual list entry below the connectionEndpoints key:

# Connection Endpoint Configuration
connectionEndpoints:
  - type: ws-binary
    options:
      # url path websocket connections connect to
      urlPath: /deepstream
      # the amount of milliseconds between each ping/heartbeat message
      heartbeatInterval: 30000
      # the amount of milliseconds that writes to sockets are buffered
      outgoingBufferTimeout: 0
      # the maximum amount of bytes to buffer before flushing, stops the client from large enough packages
      # to block its responsiveness
      maxBufferByteSize: 100000

      # Security
      # should the server log invalid auth data, defaults to false
      logInvalidAuthData: false
      # amount of time a connection can remain open while not being logged in
      unauthenticatedClientTimeout: 180000
      # invalid login attempts before the connection is cut
      maxAuthAttempts: 3
      # maximum allowed size of an individual message in bytes
      maxMessageSize: 1048576

  - type: ws-text
    options:
      # url path websocket connections connect to
      urlPath: /deepstream-v3
      # the amount of milliseconds between each ping/heartbeat message
      heartbeatInterval: 30000
      # the amount of milliseconds that writes to sockets are buffered
      outgoingBufferTimeout: 0
      # the maximum amount of bytes to buffer before flushing, stops the client from large enough packages
      # to block its responsiveness
      maxBufferByteSize: 100000

      # Security
      # should the server log invalid auth data, defaults to false
      logInvalidAuthData: false
      # amount of time a connection can remain open while not being logged in
      unauthenticatedClientTimeout: 180000
      # invalid login attempts before the connection is cut
      maxAuthAttempts: 3
      # maximum allowed size of an individual message in bytes
      maxMessageSize: 1048576

  - type: ws-json
    options:
      # url path websocket connections connect to
      urlPath: /deepstream-json
      # the amount of milliseconds between each ping/heartbeat message
      heartbeatInterval: 30000
      # the amount of milliseconds that writes to sockets are buffered
      outgoingBufferTimeout: 0
      # the maximum amount of bytes to buffer before flushing, stops the client from large enough packages
      # to block its responsiveness
      maxBufferByteSize: 100000

      # Security
      # should the server log invalid auth data, defaults to false
      logInvalidAuthData: false
      # amount of time a connection can remain open while not being logged in
      unauthenticatedClientTimeout: 180000
      # invalid login attempts before the connection is cut
      maxAuthAttempts: 3
      # maximum allowed size of an individual message in bytes
      maxMessageSize: 1048576

Websockets​

The websocket endpoints share the following options:

urlPath​

Sets which URL path Websocket connections should connect to.

Default: /deepstream

heartbeatInterval​

The number of milliseconds between each ping/heartbeat message.

Default: 30000

unauthenticatedClientTimeout​

The amount of time a connection can remain open while not being logged in.

Default: 180000

maxAuthAttempts​

Invalid login attempts before the connection is cut.

Default: 3

logInvalidAuthData​

Controls whether logs should contain the cleartext usernames and passwords of invalid login attempts.

Default: false

maxMessageSize​

Sets the maximum message size allowed to be sent to the server (in bytes).

Default: 1048576

outgoingBufferTimeout​

The amount of milliseconds that secondary writes to sockets are buffered. This means writes that are not realtime critical, which currently are either ACKs or non critical ERROR messages.

Default: 0

http​

  - type: http
    options:
      # allow 'authData' parameter in POST requests, if disabled only token and OPEN auth is
      # possible
      allowAuthData: true
      # enable the authentication endpoint for requesting userData and/or tokens
      # note: a custom authentication handler is required for token generation
      enableAuthEndpoint: false
      # path for authentication requests
      authPath: /api/auth
      # path for POST requests
      postPath: /api
      # path for GET requests
      getPath: /api
      # should the server log invalid auth data, defaults to false
      logInvalidAuthData: false
      # http request timeout in milliseconds, defaults to 20000
      requestTimeout: 20000

mqtt​

  - type: mqtt
    options:
        # port for the mqtt server
        port: 1883
        # host for the mqtt server
        host: 0.0.0.0
        # timeout for idle devices
        idleTimeout: 60000

Take a look at this blog post on the older docs for some mqtt usage examples

Custom Plugin Configuration​

You can extend deepstream with plugins for connectors to other services, these are for logging, storage engines, caching layers, and message systems. To enable a plugin, uncomment the relevant category key underneath the plugins key. Each plugin type has a path or name, and a set of options.

# Plugin example using redis
plugins:
  myCustomPlugin:
    path: ./my-custom-plugin
    options:
       host: localhost
       port: 5672

path​

Set a path to a JavaScript file, node module or a folder with an index.js file which exports a constructor.

name​

If you are using any of the official deepstream connectors, add the name of what the plugin connects to here, for example redis.

note

You can set path or name, but not both.

options​

Under this key, add sub key/value pairs to set the configuration options that are passed to the plugin. Each plugin should mention what configuration options you can set.

Logger​

deepstream uses by default a logger which prints out to stdout (errors and warnings to stderr). You can set these options for the default logger by using the same configuration style for the plugins:

logger:
  type: default
  options:
    colors: true

colors​

Sets whether the server's logs should output in color. This will look great in a console, but will leave color markers in log files if you redirect the output into a file.

Default: true

logLevel​

If defined overwrites the general configuration logLevel value.

Authentication​

In this section you can configure the authentication types the server uses.

You set the authentication type as a subkey in the auth key. The available authentication options are none, file, storage and http, each of them having different options which are described in the tutorials on Auth None, file-based authentication, storage-based authentication, HTTP authentication, and JWT authentication respectively.

You can set multiple authentication types simultaneously and the incoming connection will be validated against each of them until one succeeds or all fail. Authentication strategies will be queried in the same order they are declared on the configuration file.

#Authentication
auth:
  - type: none

Default: none

Permissioning​

In this section you can configure the permissioning. The key used for this section is permission and you can create your own custom permission handler or use a configuration file. To use the former method, select the option type: none and override the permissionHandler with the aid of the NodeJS server API. To use the latter method, set type: config and modify the option values below.

# Permissioning example with default values for config-based permissioning
permission:
  type: config
  options:
    path: ./permissions.yml
    maxRuleIterations: 3
    cacheEvacuationInterval: 60000

path​

Set the path to the file that declares permissions. This option is mandatory with type: config. The file can be in JSON, JavaScript, or YAML format. By default, deepstream ships with a permissions.yml permitting every action.

maxRuleIterations​

The deepstream permissions model allows for some complex nested actions and queries. To prevent a performance hit you can limit the nesting level with this option.

Default: 3

cacheEvacuationInterval​

The results of permission checks are cached to improve performance. Use this setting to change the time interval (in milliseconds) that the cache is regenerated.

Default: 60000

Storage and Cache​

Storage and Cache plugins can be configured as follows, please look at storage and cache connectors for in depth tutorials how to configure them.

cache:
  name: redis
  options:
    host: localhost
    port: 6379

storage:
  name: postgres
  options:
    user: postgres
    database: postgres
    password: mysecretpassword
    host: 'localhost'
    port: 5432

Default: in memory cache and no storage.

Monitoring​

Set monitoring options. You can set multiple monitoring types simultaneously, combined with custom monitoring plugins.

monitoring:
  - type: none

  # # Allows monitoring stats to be requested via HTTP, useful for polling agents
  # # such as LogStash
  # - type: http
  #   options:
  #     url: /monitoring
  #     allowOpenPermissions: false
  #     headerKey: deepstream-password2
  #     headerValue: deepstream-secret

  # # Logs monitoring stats, useful for kibana where you can visualize meta data
  # - type: log
  #   options:
  #     logInterval: 30000
  #     monitoringKey: DEEPSTREAM_MONITORING

  # # Custonm monitoring plugin example
  # - path: ./custom-monitoring-plugin
  #   options:
  #     monitoringKey: CUSTOM_DEEPSTREAM_MONITORING

Default: none

Clusternode​

Configuration options for deepstream clusters. Deepstream comes with two possible cluster modes: no cluster and vertical cluster in order to use all CPU cores. See here in order to use redis as cluster connector in multiple machine deployments.

clusterNode:
  type: default
  #name: vertical

Default: none

Lock registry​

The lock registry is responsible for maintaing a single source of truth within the cluster, used mainly for issuing cluster wide locks when an operation that stretches over multiple nodes are required. For example, distributed listening requires a leader to drive the nodes in sequence, so issuing a lock prevents multiple nodes from assuming the lead.

Check the default server configuration for details, source code here

Cluster registry​

This class maintains a list of all nodes that are currently present within the cluster. It provides status messages on a predefined interval and keeps track of incoming status messages.

Check the default server configuration for details, source code here

Cluster States​

This class provides a generic mechanism that allows to maintain a distributed state amongst the nodes of a cluster. The state is an array of unique strings in arbitrary order. Whenever a string is added by any node within the cluster for the first time, an 'add' event is emitted. Whenever its removed by the last node within the cluster, a 'remove' event is emitted.

Check the default server configuration for details, source code here

Custom Plugins​

Deepstream can be extended with any plugin via the plugin API. Custom plugins can be declared as follows:

plugins:
  custom:
    path: '...'