3. Metrics Database

3.1. Network Metrics Database Tooling

Highly Performant Timescale Metric Database

3.1.1. Two Database Flavors:

Note

• Standard

  • ontp-tsdb [ base timescale 2.6.0-pg13 (Postgres 13) ]

• Machine Learning (Apache MadLib, Tensorflow 2, pyTorch, scikit-learn), pythonpl[2,3]u database support

  • ontp-tsdb-ml [ base timescale 2.3.0 (Postgres 12.7) - 12.7 for MadLib support ]

NOTE For production use ssl keys!!

Before starting create valid ssl keys for the db communication for all necessary nodes SSL Certificate HowTo

3.1.2. Launch With Docker

Note

• docker repository

  • Standard https://hub.docker.com/repository/docker/gbontpdev/ontp-tsdb

  • ML https://hub.docker.com/repository/docker/gbontpdev/ontp-tsdb-ml

Docker command example for launching a small database install [<20 clients < 10Mbs].

docker container run --network host -p 192.168.1.20:5432:5432
 --name ontp-tsdb --shm-size=3G --env TIMESCALEDB_TELEMETRY=off
 -e TS_TUNE_MEMORY=6GB -e TS_TUNE_NUM_CPUS=4
 -e POSTGRES_USER_FILE=/opt/local/netplus/netmetrics-db/secrets/postgres-user
 -e POSTGRES_DB_FILE=/opt/local/netplus/netmetrics-db/secrets/postgres-db
 -e POSTGRES_PASSWORD_FILE=/opt/local/netplus/netmetrics-db/secrets/postgres-passwd
 -v $PWD/secrets/.phrase.log:/opt/local/ontp-tsdb/secrets/.phrase.log:ro
 --mount type=bind,source=/opt/local/netmetrics/db-data,target=/opt/local/db-data-dir/netmetrics/,bind-propagation=private ontp-tsdb
 -cshared_buffers=5GB
 -cmax_locks_per_transaction=500
 -cmax_connections=500
 -ceffective_cache_size=5GB
 -cmaintenance_work_mem=2GB
 -ccheckpoint_completion_target=0.9
 -cwal_buffers=16MB
 -cdefault_statistics_target=100
 -crandom_page_cost=1.1
 -ceffective_io_concurrency=200
 -cwork_mem=1GB
 -cmin_wal_size=2GB
 -cmax_wal_size=4GB
 -cmax_worker_processes=24
 -cmax_parallel_workers_per_gather=4
 -cmax_parallel_workers=8
 -cmax_parallel_maintenance_workers=4
 -ctimescaledb.max_background_workers=16

3.1.3. Deployment Options

Single Deployment	Replicated Reader/Writer Database Config
1 metric database for many metric decoders.	Many metric databases for many metric decoders

Note

• We recommend the load balanced approach so that your metric database is not a single point of failure. Where you run multiple ontp-tsdb instances to fit the size of your env.

3.1.4. ontp-tsdb Example Cluster Configuration Start

ontp-tsdb Cluster Database How To.

1. Start access node

ANODE_SSL_CRT=certs/e8a48653851e28c69d0506508fb27fc5.crt \
ANODE_SSL_KEY=certs/e8a48653851e28c69d0506508fb27fc5.key \
START_ACCESS_NODE=1 CONTAINER_NAME=ontp-tsdb ./start_cluster_access_node.sh

2. Start data nodes ..

DNODE_SSL_CRT=certs/ontp_tsdb-dnode-dn1.crt DNODE_SSL_KEY=certs/ontp_tsdb-dnode-dn1.key\
RUN_CONTAINER_NAME=ontp-tsdb-dnode1 \
START_DATA_NODE=1 CONTAINER_NAME=ontp-tsdb ./start_cluster_data_node.sh
..
DNODE_SSL_CRT=certs/ontp_tsdb-dnode-dn2.crt DNODE_SSL_KEY=certs/ontp_tsdb-dnode-dn2.key\
RUN_CONTAINER_NAME=ontp-tsdb-dnode2 \
START_DATA_NODE=1 CONTAINER_NAME=ontp-tsdb ./start_cluster_data_node.sh
..
DNODE_SSL_CRT=certs/ontp_tsdb-dnode-dn3.crt DNODE_SSL_KEY=certs/ontp_tsdb-dnode-dn3.key\
RUN_CONTAINER_NAME=ontp-tsdb-dnode3 \
START_DATA_NODE=1 CONTAINER_NAME=ontp-tsdb ./start_cluster_data_node.sh
..
..
..

3. Add data nodes to access nodes

sudo docker exec -it ontp-tsdb-anode load_ontp \
     --postgres-con "postgres,postgres,tsLogin77,192.168.1.20,5432"\
     -a dn1:192.168.1.21,dn2:192.168.1.22,dn3:192.168.2.23

4. Apply tables

sudo docker exec -it ontp-tsdb-anode1 load_ontp -n cluster \
            -P "postgres,postgres,tsLogin77,192.168.1.20,5432"
#test if tables propagated. sudo docker container exec 2cbb99fc7367 psql -U postgres -c "\dt;"

5. Apply/Register license See Applying License Section Below

sudo docker exec -it ontp-tsdb-anode1 config-auth \
  --postgres-con  "postgres,postgres,tsLogin77,192.168.1.20,5432" \
   --newlicense --update-org \
   -p "-----BEGIN PGP MESSAGE-----

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxx=
=xxxu
-----END PGP MESSAGE-----
"

• DB Node is now ready for connections from ontp-mbus with valid keys ontp-mbus will relay any ontp-wire information from an agent with valid keys

3.1.5. Start Access Node Script - Please Modify for your env

#!/bin/bash
#########################################################################
# Script to Start our timecale db instances
#########################################################################
# Notes:
#  1 Time:
#     Handle time/timezone in the container
#     time is important to be able to sync events
#          -v /etc/timezone:/etc/timezone:ro \
#          -v /etc/localtime:/etc/localtime:ro \
#     And the docker host should be running a valid ntp configuration
#     -- OR docker run ... hwclock -s
#########################################################################


RUN_CONTAINER_NAME=${RUN_CONTAINER_NAME:-"ontp-tsdb-anode"}
CONTAINER_NAME=${CONTAINER_NAME:-"ontp-tsdb"}
VERSION=${VERSION:-"2.0.1"}
#
# local file system path where the database will be stored
SOURCE_DIR=${SOURCE_DIR:-"/opt/local/netmetrics/db-data-ahost"}
# map SOURCE_DIR to a directory within the container
DEST_DIR=${DEST_DIR:-"/opt/local/ontp-tsdb/data"}
# where we place files locally within the container
DIST_DIR=/opt/local/ontp-tsdb
# default postgres server port
LOCAL_PORT=5432
# our local host ip
HOST_IP=${HOST_IP:-"192.168.1.20"}
#
POSTGRES_HOST_AUTH_METHOD=${POSTGRES_HOST_AUTH_METHOD:-"scram-sha-256"}

#-e POSTGRES_USER_FILE=${DIST_DIR}/secrets/postgres-user \
#-e POSTGRES_DB_FILE=${DIST_DIR}/secrets/postgres-db \
#-e POSTGRES_PASSWORD_FILE=${DIST_DIR}/secrets/postgres-passwd \
#-e PGHOST=127.0.0.1 \

# Notes:
#   Options:
#     DB_CERT_AUTH=yes -- to modify the pg_hba file will hostssl arg
#   Cluster configuration:
#       enable_partitionwise_aggregate=on the "access node"
#       jit=off on the "access node"
#       statement_timeout=off|disabled on the "data nodes" can be set on the "access node"
#


START_ACCESS_NODE=${START_ACCESS_NODE:-""}
#
OPTIONAL_DB_ARGS=${OPTIONAL_DB_ARGS:-""}
NODE_ARGS=""


# CA crt file
ONTP_CA_CRT=${ONTP_CA_CRT:-"certs/ontp_tsdb-ca.crt"}

# access node ssl crt/key
ANODE_SSL_CRT=${ANODE_SSL_CRT:-"certs/ontp_tsdb-access_node.crt"}
ANODE_SSL_KEY=${ANODE_SSL_KEY:-"certs/ontp_tsdb-access_node.key"}

#
ACCESS_NODE_ARGS=\
"-v $PWD/${ANODE_SSL_CRT}:/var/lib/postgresql/server.crt:ro\
 -v $PWD/${ANODE_SSL_KEY}:/var/lib/postgresql/server.key:ro\
 -v $PWD/secrets/.phrase.log:/opt/local/ontp-tsdb/secrets/.phrase.log:ro \
 -e DB_CERT_AUTH=yes\
 -e POSTGRES_COMM_PASS=z9dbaFe9xlbeR\
 ${CONTAINER_NAME}\
 -cmax_prepared_transactions=500\
 -cstatement_timeout=600\
 -cenable_partitionwise_aggregate=on\
 -cjit=off\
"


ARG_SET=0
# append optional args if any
if [ ! -z "${OPTIONAL_DB_ARGS:-}" ]; then
     ONTP_ARGS="${ONTP_ARGS} ${OPTIONAL_DB_ARGS}"
fi


if [ ! -z "${START_ACCESS_NODE:-}" ]; then
     echo "Starting 'cluster access' node ontp/${CONTAINER_NAME} instance"
     NODE_ARGS=${ACCESS_NODE_ARGS}
     ARG_SET=1
fi

# -e POSTGRES_HOST_AUTH_METHOD=${POSTGRES_HOST_AUTH_METHOD}\
#
ONTP_ARGS=${ONTP_ARGS:-"--network host -p ${HOST_IP}:${LOCAL_PORT}:5432 \
 --name ${RUN_CONTAINER_NAME}\
 --shm-size=3G\
 --env TIMESCALEDB_TELEMETRY=off\
 -e LANG=en_US.UTF-8\
 -e LC_CTYPE=en_US.UTF-8\
 -e LANGUAGE=en_US.UTF-8\
 -e TS_TUNE_MEMORY=6GB\
 -e TS_TUNE_NUM_CPUS=4\
 -e POSTGRESQL_CONF_DIR=/opt/local/ontp-tsdb/data/\
 -e POSTGRES_USER=postgres\
 -e PGHOST=${HOST_IP}\
 -e PGHOSTADDR=${HOST_IP}\
 -e PGUSER=postgres\
 -e PGDATABASE=postgres\
 -e POSTGRES_DB=postgres\
 -e POSTGRES_PASSWORD=tsLogin77\
 -e DB_HOST_IP=${HOST_IP}\
 -v /etc/localtime:/etc/localtime:ro \
 --mount type=bind,source=${SOURCE_DIR},target=${DEST_DIR},bind-propagation=private \
 ${NODE_ARGS}\
 -cssl=on\
 -cssl_cert_file=/var/lib/postgresql/server.crt\
 -cssl_key_file=/var/lib/postgresql/server.key\
 -cshared_buffers=5GB\
 -cmax_locks_per_transaction=500\
 -cmax_connections=500\
 -ceffective_cache_size=5GB\
 -cmaintenance_work_mem=2GB\
 -ccheckpoint_completion_target=0.9\
 -cwal_buffers=16MB\
 -cdefault_statistics_target=100\
 -crandom_page_cost=1.1\
 -ceffective_io_concurrency=200\
 -cwork_mem=1GB\
 -cmin_wal_size=2GB\
 -cmax_wal_size=4GB\
 -cmax_worker_processes=24\
 -cmax_parallel_workers_per_gather=4\
 -cmax_parallel_workers=8\
 -cmax_parallel_maintenance_workers=4\
 -ctimescaledb.max_background_workers=16\
"}

if [ ${ARG_SET}==1 ]; then
     echo "Args: '$ONTP_ARGS'"
     sudo docker container run ${ONTP_ARGS}
fi
exit 0

3.1.6. Start Data Node Script - Please Modify for your env

#!/bin/bash
#########################################################################
# Script to Start our timecale db instances
#########################################################################
# Notes:
#  1 Time:
#     Handle time/timezone in the container
#     time is important to be able to sync events
#          -v /etc/timezone:/etc/timezone:ro \
#          -v /etc/localtime:/etc/localtime:ro \
#     And the docker host should be running a valid ntp configuration
#     -- OR docker run ... hwclock -s
#########################################################################


RUN_CONTAINER_NAME=${RUN_CONTAINER_NAME:-"ontp-tsdb-dnode1"}
CONTAINER_NAME=${CONTAINER_NAME:-"ontp-tsdb"}
VERSION=${VERSION:-"2.0.1"}
# local file system path where the database will be stored
SOURCE_DIR=${SOURCE_DIR:-"/opt/local/netmetrics/db-data-dhost1"}
# map SOURCE_DIR to a directory within the container
DEST_DIR=${DEST_DIR:-"/opt/local/ontp-tsdb/data"}
# where we place files locally within the container
DIST_DIR=/opt/local/ontp-tsdb
# default postgres server port
LOCAL_PORT=5432
# our local host ip
HOST_IP=${HOST_IP:-"192.168.1.21"}
#
POSTGRES_HOST_AUTH_METHOD=${POSTGRES_HOST_AUTH_METHOD:-"scram-sha-256"}

#-e POSTGRES_USER_FILE=${DIST_DIR}/secrets/postgres-user \
#-e POSTGRES_DB_FILE=${DIST_DIR}/secrets/postgres-db \
#-e POSTGRES_PASSWORD_FILE=${DIST_DIR}/secrets/postgres-passwd \
#-e PGHOST=127.0.0.1 \

# Notes:
#   Options:
#     DB_CERT_AUTH=yes -- to modify the pg_hba file will hostssl arg
#   Cluster configuration:
#       enable_partitionwise_aggregate=on the "access node"
#       jit=off on the "access node"
#       statement_timeout=off|disabled on the "data nodes" can be set on the "access node"
#


START_DATA_NODE=${START_DATA_NODE:-""}
#
OPTIONAL_DB_ARGS=${OPTIONAL_DB_ARGS:-""}
NODE_ARGS=""

# args to start the ontp timescale db container
# -v \"/etc/localtime:/etc/localtime:ro\" \

# CA crt file
ONTP_CA_CRT=${ONTP_CA_CRT:-"certs/ontp_tsdb-ca.crt"}

# data node ssl crt/key
DNODE_SSL_CRT=${DNODE_SSL_CRT:-"certs/ontp_tsdb-dnode-dbcontainer1.crt"}
DNODE_SSL_KEY=${DNODE_SSL_KEY:-"certs/ontp_tsdb-dnode-dbcontainer1.key"}

# data node additional args
DATA_NODE_ARGS=\
"-v $PWD/${DNODE_SSL_CRT}:/var/lib/postgresql/server.crt:ro\
 -v $PWD/${DNODE_SSL_KEY}:/var/lib/postgresql/server.key:ro\
 -v $PWD/${ONTP_CA_CRT}:/var/lib/postgresql/root.crt\
 -v $PWD/secrets/.phrase.log:/opt/local/ontp-tsdb/secrets/.phrase.log:ro \
 -e DB_CERT_AUTH=yes\
 -e POSTGRES_COMM_PASS=z9dbaFe9xlbeR\
 ${CONTAINER_NAME}\
 -cmax_prepared_transactions=500\
 -cjit=on\
"


ARG_SET=0
# append optional args if any
if [ ! -z "${OPTIONAL_DB_ARGS:-}" ]; then
     ONTP_ARGS="${ONTP_ARGS} ${OPTIONAL_DB_ARGS}"
fi

if [ ! -z "${START_DATA_NODE:-}" ]; then
     echo "Starting 'data' node ontp/${CONTAINER_NAME} instance"
     NODE_ARGS=${DATA_NODE_ARGS}
     ARG_SET=1
fi

# -e POSTGRES_HOST_AUTH_METHOD=${POSTGRES_HOST_AUTH_METHOD}\
#
ONTP_ARGS=${ONTP_ARGS:-"--network host -p ${HOST_IP}:${LOCAL_PORT}:5432 \
 --name ${RUN_CONTAINER_NAME}\
 --shm-size=3G\
 --env TIMESCALEDB_TELEMETRY=off\
 -e LANG=en_US.UTF-8\
 -e LC_CTYPE=en_US.UTF-8\
 -e LANGUAGE=en_US.UTF-8\
 -e TS_TUNE_MEMORY=6GB\
 -e TS_TUNE_NUM_CPUS=4\
 -e POSTGRESQL_CONF_DIR=/opt/local/ontp-tsdb/data/\
 -e POSTGRES_USER=postgres\
 -e PGHOST=${HOST_IP}\
 -e PGHOSTADDR=${HOST_IP}\
 -e PGUSER=postgres\
 -e PGDATABASE=postgres\
 -e POSTGRES_DB=postgres\
 -e POSTGRES_PASSWORD=tsLogin77\
 -e DB_HOST_IP=${HOST_IP}\
 -v /etc/localtime:/etc/localtime:ro \
 --mount type=bind,source=${SOURCE_DIR},target=${DEST_DIR},bind-propagation=private \
 ${NODE_ARGS}\
 -cssl=on\
 -cssl_ca_file=/var/lib/postgresql/root.crt\
 -cssl_cert_file=/var/lib/postgresql/server.crt\
 -cssl_key_file=/var/lib/postgresql/server.key\
 -cshared_buffers=5GB\
 -cmax_locks_per_transaction=500\
 -cmax_connections=500\
 -ceffective_cache_size=5GB\
 -cmaintenance_work_mem=2GB\
 -ccheckpoint_completion_target=0.9\
 -cwal_buffers=16MB\
 -cdefault_statistics_target=100\
 -crandom_page_cost=1.1\
 -ceffective_io_concurrency=200\
 -cwork_mem=1GB\
 -cmin_wal_size=2GB\
 -cmax_wal_size=4GB\
 -cmax_worker_processes=24\
 -cmax_parallel_workers_per_gather=4\
 -cmax_parallel_workers=8\
 -cmax_parallel_maintenance_workers=4\
 -ctimescaledb.max_background_workers=16\
"}

#
if [ ${ARG_SET}==1 ]; then
     echo "Args: '$ONTP_ARGS'"
     sudo docker container run ${ONTP_ARGS}
     exit $?
fi
exit 1

3.1.7. ontp-tsdb Example Single Node Configuration Start Script

ontp-tsdb Single Instance Database How To.

1. Start the node

NODE_SSL_CRT=certs/e8a48653851e28c69d0506508fb27fc5.crt \
NODE_SSL_KEY=certs/e8a48653851e28c69d0506508fb27fc5.key \
START_NODE=1 RUN_CONTAINER_NAME=ontp-tsdb-snode CONTAINER_NAME=ontp-tsdb ./start_single_node.sh

2. Apply/Register license See Applying License Section Below

sudo docker exec -it ontp-tsdb-anode1 config-auth \
  --postgres-con  "postgres,postgres,tsLogin77,192.168.1.20,5432" \
   --newlicense --update-org \
   -p "-----BEGIN PGP MESSAGE-----
.....
.....
.....
-----END PGP MESSAGE-----
"

• DB Node is now ready for connections from ontp-mbus with valid keys ontp-mbus will relay any ontp-wire information from an agent with valid keys

Start Single Node Database Script - Please Modify for your env

#!/bin/bash
#########################################################################
# Script to Start our timecale db instances
#########################################################################
# Notes:
#  1 Time:
#     Handle time/timezone in the container
#     time is important to be able to sync events
#          -v /etc/timezone:/etc/timezone:ro \
#          -v /etc/localtime:/etc/localtime:ro \
#     And the docker host should be running a valid ntp configuration
#     -- OR docker run ... hwclock -s
#########################################################################


RUN_CONTAINER_NAME=${RUN_CONTAINER_NAME:-"ontp-tsdb-snode"}
CONTAINER_NAME=${CONTAINER_NAME:-"ontp-tsdb"}
# local file system path where the database will be stored
SOURCE_DIR=${SOURCE_DIR:-"/opt/local/netmetrics/db-single"}
# map SOURCE_DIR to a directory within the container
DEST_DIR=${DEST_DIR:-"/opt/local/ontp-tsdb/data"}
# where we place files locally within the container
DIST_DIR=/opt/local/ontp-tsdb
# default postgres server port
LOCAL_PORT=5432
# our local host ip
HOST_IP=${HOST_IP:-"192.168.1.20"}
VERSION=${VERSION:-"3.0.1"}
#
#POSTGRES_HOST_AUTH_METHOD=${POSTGES_HOST_AUTH_METHOD:-"scram-sha-256"}

#-e POSTGRES_USER_FILE=${DIST_DIR}/secrets/postgres-user \
#-e POSTGRES_DB_FILE=${DIST_DIR}/secrets/postgres-db \
#-e POSTGRES_PASSWORD_FILE=${DIST_DIR}/secrets/postgres-passwd \
#-e PGHOST=127.0.0.1 \

# Notes:
#   Options:
#     DB_CERT_AUTH=yes -- to modify the pg_hba file will hostssl arg
#   Cluster configuration:
#       enable_partitionwise_aggregate=on the "access node"
#       jit=off on the "access node"
#       statement_timeout=off|disabled on the "data nodes" can be set on the "access node"
#


START_ACCESS_NODE=${START_ACCESS_NODE:-""}
START_NODE=${START_NODE:-""}
#
OPTIONAL_DB_ARGS=${OPTIONAL_DB_ARGS:-""}


# access node ssl crt/key
NODE_SSL_CRT=${NODE_SSL_CRT:-"certs/ontp_tsdb-node.crt"}
NODE_SSL_KEY=${NODE_SSL_KEY:-"certs/ontp_tsdb-node.key"}


NODE_ARGS=\
"-v ${PWD}/${NODE_SSL_CRT}:/var/lib/postgresql/server.crt:ro\
 -v ${PWD}/${NODE_SSL_KEY}:/var/lib/postgresql/server.key:ro\
 -v $PWD/secrets/.phrase.log:/opt/local/ontp-tsdb/secrets/.phrase.log:ro \
 -e DB_CERT_AUTH=yes\
 ${CONTAINER_NAME}\
"

# args to start the ontp timescale db container
# -v /etc/timezone:/etc/timezone:ro\
# -e POSTGRES_HOST_AUTH_METHOD=${POSTGRES_HOST_AUTH_METHOD}\
ONTP_ARGS=${ONTP_ARGS:-"--network host -p ${HOST_IP}:${LOCAL_PORT}:5432 \
 --name ${RUN_CONTAINER_NAME}\
 --shm-size=3G\
 --env TIMESCALEDB_TELEMETRY=off\
 -e LANG=en_US.UTF-8\
 -e LC_CTYPE=en_US.UTF-8\
 -e LANGUAGE=en_US.UTF-8\
 -e TS_TUNE_MEMORY=6GB\
 -e TS_TUNE_NUM_CPUS=4\
 -e POSTGRESQL_CONF_DIR=/opt/local/ontp-tsdb/data/\
 -e POSTGRES_USER=postgres\
 -e PGHOST=${HOST_IP}\
 -e PGHOSTADDR=${HOST_IP}\
 -e PGUSER=postgres\
 -e PGDATABASE=postgres\
 -e POSTGRES_DB=postgres\
 -e POSTGRES_PASSWORD=tsLogin77\
 -e ONTP_SINGLE_INSTACE_DB=1\
 -e DB_HOST_IP=${HOST_IP}\
 -v /etc/localtime:/etc/localtime:ro\
 --mount type=bind,source=${SOURCE_DIR},target=${DEST_DIR},bind-propagation=private\
 ${NODE_ARGS}\
 -cssl=on\
 -cssl_cert_file=/var/lib/postgresql/server.crt\
 -cssl_key_file=/var/lib/postgresql/server.key\
 -cshared_buffers=5GB\
 -cmax_locks_per_transaction=500\
 -cmax_connections=500\
 -ceffective_cache_size=5GB\
 -cmaintenance_work_mem=2GB\
 -ccheckpoint_completion_target=0.9\
 -cwal_buffers=16MB\
 -cdefault_statistics_target=100\
 -crandom_page_cost=1.1\
 -ceffective_io_concurrency=200\
 -cwork_mem=1GB\
 -cmin_wal_size=2GB\
 -cmax_wal_size=4GB\
 -cmax_worker_processes=24\
 -cmax_parallel_workers_per_gather=4\
 -cmax_parallel_workers=8\
 -cmax_parallel_maintenance_workers=4\
 -ctimescaledb.max_background_workers=16\
"}


# append optional args if any
if [ ! -z "${OPTIONAL_DB_ARGS:-}" ]; then
     ONTP_ARGS="${ONTP_ARGS} ${OPTIONAL_DB_ARGS}"
fi


if [ ! -z "${START_NODE:-}" ]; then
     echo "Starting 'single access' node ontp/${CONTAINER_NAME} instance"
     ONTP_ARGS="${ONTP_ARGS} ${SSL_CERT_ARGS}"
     echo "Args: '$ONTP_ARGS'"
     sudo docker container run ${ONTP_ARGS}
     exit 0
fi


exit 0

3.1.8. ontp-tsdb-ml Example Cluster Configuration Start

ontp-tsdb-ml Cluster Database How To.

1. Start access node

ANODE_SSL_CRT=certs/e8a48653851e28c69d0506508fb27fc5.crt \
ANODE_SSL_KEY=certs/e8a48653851e28c69d0506508fb27fc5.key \
START_ACCESS_NODE=1 CONTAINER_NAME=ontp-tsdb ./start_cluster_access_node.sh

2. Start data nodes ..

DNODE_SSL_CRT=certs/ontp_tsdb-dnode-dn1.crt DNODE_SSL_KEY=certs/ontp_tsdb-dnode-dn1.key\
RUN_CONTAINER_NAME=ontp-tsdb-dnode1 \
START_DATA_NODE=1 CONTAINER_NAME=ontp-tsdb ./start_cluster_data_node.sh
..
     DNODE_SSL_CRT=certs/ontp_tsdb-dnode-dn2.crt DNODE_SSL_KEY=certs/ontp_tsdb-dnode-dn2.key\
RUN_CONTAINER_NAME=ontp-tsdb-dnode2 \
START_DATA_NODE=1 CONTAINER_NAME=ontp-tsdb ./start_cluster_data_node.sh
..
     DNODE_SSL_CRT=certs/ontp_tsdb-dnode-dn3.crt DNODE_SSL_KEY=certs/ontp_tsdb-dnode-dn3.key\
RUN_CONTAINER_NAME=ontp-tsdb-dnode3 \
START_DATA_NODE=1 CONTAINER_NAME=ontp-tsdb ./start_cluster_data_node.sh
..
..
..

3. Load the necessary extensions

# load the ontp extensions for the access node
sudo docker exec -it ontp-tsdb-anode load_ontp  -E \
  --postgres-con "postgres,postgres,tsLogin77,192.168.1.20,5432" \

# load the ontp extensions for each dnode
sudo docker exec -it ontp-tsdb-dnode1 load_ontp -E \
     --postgres-con "postgres,postgres,tsLogin77,192.168.1.21,5432"
  ...
  ...

4. Map cluster data node to access node

sudo docker exec -it ontp-tsdb-anode1 load_ontp -a 'dn1:192.168.1.21,dn2:192.168.1.22' \
                 -P "ontp_netmetrics,postgres,tsLogin77,192.168.1.20,5432"

4. Access node only! - add the db objects

# access node only - add the db objects
sudo docker exec -it ontp-tsdb-anode1 load_ontp -n cluster \
            -P "ontp_netmetrics,postgres,tsLogin77,192.168.1.20,5432"

5. Apply/Register license See Applying License Section Above

3.1.9. ontp-tsdb-ml Example Single Node Configuration Start Script

ontp-tsdb-ml Single Instance Database How To.

1 Start Node

START_ACCESS_NODE=anode1 \
ANODE_SSL_CRT=certs/e8a48653851e28c69d0506508fb27fc5.crt \
ANODE_SSL_KEY=certs/e8a48653851e28c69d0506508fb27fc5.key START_ACCESS_NODE=1 \
CONTAINER_NAME=ontp-tsdb-ml ./run_tools/start_ontp_node.sh

2 Ontp db initilization [TODO map above and below into 1 call]

sudo docker exec -it ontp-tsdb-ml load_ontp -E \
     --postgres-con "ontp_netmetrics,postgres,tsLogin77,192.168.1.20,5432"

3 Add db objects

sudo docker exec -it ontp-tsdb-ml load_ontp -n standalone\
     --postgres-con "postgres,postgres,tsLogin77,192.168.1.20,5432"

4. Apply/Register license See Applying License Section Above

• 3.1.9.1. Machine Learning Example
  • 3.1.9.1.1. Network Metrics Database ML
    • 3.1.9.1.1.1. ML SQL Arima Example
    • 3.1.9.1.1.2. ML SQL Arima Decision Tree Example
    • 3.1.9.1.1.3. ML SQL Test Tensorflow
    • 3.1.9.1.1.4. ML PYthon Example Code
• 3.1.9.2. Metrics Database ontp-tsdb-ml
  • 3.1.9.2.1. Enable Default Machine Learning Code / Testing
    • 3.1.9.2.1.1. Machine Learning Notes:
    • 3.1.9.2.1.2. ML Functions
    • 3.1.9.2.1.3. ARIMA payload forecast example
    • 3.1.9.2.1.4. Test Tensorflow
    • 3.1.9.2.1.5. Decision Tree For TCP Flag Data
    • 3.1.9.2.1.6. Random Forrest For TCP Flag Data 1
    • 3.1.9.2.1.7. Random Forrest For TCP Flag Data 2