BPM Quick Start Guide

Follow

Quickstart Guide

The following tutorials give you a quick overview of BPM. We will install the BPM CLI and use it to configure and run Blockchain nodes.

The BPM CLI serves two main purposes:

  1. Manage packages. A BPM package contains the functionality to launch a specific Blockchain node. There is typically one package per supported Blockchain client.
  2. Manage nodes. A BPM node is a combination of the configuration, data, runtime as well as monitoring agents of a particular Blockchain node. Nodes can be started and stopped.

Note: While it is possible to run multiple Blockchain nodes at the same time we typically recommend to only have one node started at any time. There are multiple reasons for this but the main ones are as follows:

  • Blockchain nodes are resource intensive and running more than one can easily overwhelm the server.
  • A lot of Blockchain protocols use the same common default ports which causes conflicts if more than one is run at the same time. While it is possible to change those ports, care has to be taken to avoid using the same ones.

System Requirements

  • OSX or Linux operating system
  • Docker needs to be installed and running for most packages
  • Wget running on the operating system.
    • Ubuntu installation: sudo apt install wget
    • OSX with Homebrewbrew install wget

Prerequisites

If you ran a development version of BPM, please make sure to wipe any remnants of that before continuing:

  1. Delete ~/.bpm
  2. Remove any BPM related docker containers.
  3. Remove any BPM related docker networks.

Installing the BPM CLI

Note: In this section, you will need to replace values for placeholder found between < >:

Using wget you will download the binary for your OS. Then you will copy it to the correct bin directory path and set it to an executable binary.

wget <BPM_URL>
sudo cp bpm-<VERSION>-<OS>-amd64 /usr/local/bin/bpm
sudo chmod 755 /usr/local/bin/bpm

Tutorial 1: Managing packages

In this first tutorial you will learn how to install and manage BPM packages. To get an overview over what is possible, let's start with listing all commands available for managing packages:

bpm packages --help

The very first time you run bpm it needs to initialize it's directories, you should see the following output:

Looks like bpm isn't initialized correctly in "/Users/blockdaemon/.bpm", do you want to do that now? (y/N): y

Just type y to start the BPM initialization procedure.

You should see the following output:

Manage packages

Usage:
  bpm packages [command]

Available Commands:
  info        Show information about a package
  install     Installs or upgrades a package to a specific version or latest if no version is specified
  list        List installed packages
  search      Search available packages
  uninstall   Uninstall a package. Data and configuration will not be removed.

Flags:
  -h, --help   help for packages

Global Flags:
      --base-dir string           The directory plugins and configuration are stored (default "~/.bpm/")
      --debug                     Enable debug output
      --package-registry string   The package registry provides packages to install (default "https://dev.registry.blockdaemon.com")
  -y, --yes                       Automatic yes to prompts; assume "yes" as answer to all prompts and run non-interactively

Use "bpm packages [command] --help" for more information about a command.

The first step is usually to search for a package to install. To list all available packages, just run the search command without a search parameter:

bpm packages search

You can also search for specific protocol packages by providing a search parameter (e.g. parity):

bpm packages search parity

After you have found a suitable packages, install it:

bpm packages install parity

This can take a few minutes while it is downloading the package.

To check which packages have already beeen installed, run the list command:

bpm packages list

You should see similar output:

  NAME  | INSTALLED VERSION | RECOMMENDED VERSION
+--------+-------------------+---------------------+
  parity | 1.0.0             | 1.0.0

This tells us that the parity package is installed using the latest recommended version.

To show more information about a package, use the info command:

bpm packages info parity

Finally, if a package isn't needed anymore, it can be uninstalled:

bpm packages uninstall parity

Tutorial 2: Starting a Polkadot testnet node

First, let's list all commands available for managing nodes:

bpm nodes --help

You should see the following output:

Manage blockchain nodes

Usage:
  bpm nodes [command]

Available Commands:
  configure   Configure a new blockchain node
  remove      Remove blockchain node data and configuration
  show        Print a resource to stdout
  start       Start a blockchain node
  status      Display statuses of configured nodes
  stop        Stops a running blockchain node
  test        Tests a running blockchain node

Flags:
  -h, --help   help for nodes

Global Flags:
      --base-dir string           The directory plugins and configuration are stored (default "~/.bpm/")
      --debug                     Enable debug output
      --package-registry string   The package registry provides packages to install (default "https://dev.registry.blockdaemon.com")
  -y, --yes                       Automatic yes to prompts; assume "yes" as answer to all prompts and run non-interactively

Use "bpm nodes [command] --help" for more information about a command.

Installing the polkadot package

Before diving into the actual node management we need to install the polkadot package:

bpm packages install polkadot

Configuring the node

Run the configure command with --help to see the available parameters:

bpm nodes configure polkadot --help

There are two parameters that are specific to the polkadot package:

      --subtype watcher        The type of node. Must be either watcher or `validator` (default "watcher")
      --validator-key string   The key used for a validator (required if subtype = validator)

For now we will leave the defaults which creates a configuration for a non-validating polkadot node. Run the configure command to create it now:

bpm nodes configure polkadot

You should see an output similar to this (but with different IDs):

No credentials found in "/Users/jonas/.bpm/beats", skipping configuration of Blockdaemon monitoring. Please configure your own monitoring in the node configuration files.

Nothing to do here, skipping create-secrets
Writing file '/Users/jonas/.bpm/nodes/bob3rhstp9lpfnasuff0/configs/filebeat.yml'
Writing file '/Users/jonas/.bpm/nodes/bob3rhstp9lpfnasuff0/configs/polkadot.dockercmd'
Writing file '/Users/jonas/.bpm/nodes/bob3rhstp9lpfnasuff0/configs/polkadotbeat.yml'

Node with id "bob3rhstp9lpfnasuff0" has been initialized.

To change the configuration, modify the files here:
    /Users/jonas/.bpm/nodes/bob3rhstp9lpfnasuff0/configs
To start the node, run:
    bpm nodes start bob3rhstp9lpfnasuff0
To see the status of configured nodes, run:
    bpm nodes status

Node status

You can verify the status of all nodes by running the status command:

bpm nodes status

Starting and stopping the node

To start the node, run the node start command. Replace <node-id> with the ID outputed by the configure or status command:

bpm nodes start <node-id>

You should get an output similar to:

Creating network 'bpm-bob3rhstp9lpfnasuff0-polkadot'
Network 'bpm-bob3rhstp9lpfnasuff0-polkadot' already exists, skipping creation
Network 'bpm-bob3rhstp9lpfnasuff0-polkadot' already exists, skipping creation
Creating container 'bpm-bob3rhstp9lpfnasuff0-polkadot'
Starting container 'bpm-bob3rhstp9lpfnasuff0-polkadot'
Creating container 'bpm-bob3rhstp9lpfnasuff0-polkadotbeat'
Starting container 'bpm-bob3rhstp9lpfnasuff0-polkadotbeat'
Creating container 'bpm-bob3rhstp9lpfnasuff0-filebeat'
Starting container 'bpm-bob3rhstp9lpfnasuff0-filebeat'
The node "bob3rhstp9lpfnasuff0" has been started.

This shows BPM creating a dedicated Docker network for this node as well as starting the Docker containers for Parity and the monitoring agents.

Verify the node status by running:

bpm nodes status

The node should now show up as running, similar to below:

        NODE ID        | PACKAGE | STATUS   | SECRETS
+----------------------+---------+----------+---------+
  bob3rhstp9lpfnasuff0 | running | polkadot |       0

To stop it temporarily, run:

bpm nodes stop <node-id>

Removing a node

When removing a node you need to consider the following. A node consists of:

  1. Secrets (e.g. accounts, passwords, private keys)
  2. Node configuration
  3. Runtime (e.g. Docker networks and containers)
  4. Data (typically the parts of the Blockchain that have already been synced)

Depending on the use-case it may be desirable to remove all or only parts of the node. For example:

  • In order to re-configure a node one might only want to remove the configuration but leave the data intact to avoid having to re-sync the Blockchain
  • If the node crashed due to an unexpected error it can make sense to remove the runtime and start it again but keep the configuration and data
  • If something went wrong during the initial sync it can help to remove the data and then start the node again to start syncing from scratch

To support the above use-cases plus others we have allowed parameters/flags to be used with our remove command.

You can view all the available parameters/flags by running the following BPM command:

bpm nodes remove --help

Which should return:

Remove blockchain node data and configuration

Usage:
  bpm nodes remove <id> [flags]

Flags:
      --all       Remove all data, configuration files and node information
      --config    Remove all configuration files but keep data and node information
      --data      Remove all data but keep configuration files and node information
  -h, --help      help for remove
      --runtime   Remove all runtimes but keep configuration files and node information

Global Flags:
      --base-dir string           The directory plugins and configuration are stored (default "~/.bpm/")
      --debug                     Enable debug output
      --package-registry string   The package registry provides packages to install (default "https://dev.registry.blockdaemon.com")
  -y, --yes                       Automatic yes to prompts; assume "yes" as answer to all prompts and run non-interactively

For now, let's remove the whole node:

nodes remove --all <node-id>

Warning: By removing the whole node you will also remove the secrets directory. It's always advisable to backup the secrets directory to a safe place before doing anything with the node.

Tutorial 3: Connect to an existing permissioned Parity network

Installing the parity package

The first step is to install the parity package:

bpm packages install parity

Configuring the node

The Parity package is built specifically to run a node for a permissioned Blockchain network. It takes two parameters:

  1. A comma separated list of enode URLs that can act as bootnodes. These will serve as a list of peers that a new BPM Parity node will connect to.
  2. chain specification that contains the configuration for the permissioned Blockchain network.

Note: If you've used Blockdaemon to launch your Blockchain network you can find both parameters on your Blockdaemon dashboard. Otherwise please ask the administrator of the permissioned Blockchain network for parameters.

Let's start configuring a node using the configure command:

bpm nodes configure parity --bootnodes "<enode_URLs>" --chain-spec <chain-spec_filepath>

Depending on your specific parameters this could look something like:

bpm nodes configure parity --bootnodes "enode://8b1dfdfb03fbf40b7094aefa3e46a67b6e97cd06f28196413e76963935900b3c60bfc2c2ad7b14eec31245f7788cbf87cc8ca2352d138a736c081d7243e43163@33.236.162.172:30303,enode://aa32c91c962fad2cea3d6fac8a3ee9a86732e49423c78237a6827903ebba9d7b6a151d6dc14c2ce44663f140942afadd365290624f86c11c5a3adf4991c8ded7@19.199.180.203:30303" --chain-spec ~/Downloads/chain.json

You should see an output similar to this (but with different IDs):

generating accounts...
generating accounts...
Copying chain spec file to '/Users/blockdaemon/.bpm/nodes/bnn4trbo6him2ii9uca0/configs/chain.json'
Copying signer account file to '/Users/blockdaemon/.bpm/nodes/bnn4trbo6him2ii9uca0/configs/keys/DemoPoA/signer.json'
Copying signer password file file to '/Users/blockdaemon/.bpm/nodes/bnn4trbo6him2ii9uca0/configs/password.secret'
Writing file '/Users/blockdaemon/.bpm/nodes/bnn4trbo6him2ii9uca0/configs/parity.dockercmd'
Writing file '/Users/blockdaemon/.bpm/nodes/bnn4trbo6him2ii9uca0/configs/filebeat.yml'

Node with id "bnn4trbo6him2ii9uca0" has been initialized.

To change the configuration, modify the files here:
    /Users/blockdaemon/.bpm/nodes/bnn4trbo6him2ii9uca0/configs
To start the node, run:
    bpm nodes start bnn4trbo6him2ii9uca0
To see the status of configured nodes, run:
    bpm nodes status

Note: The configure step is deliberately separated from the start step. This enables users to manually change the configuration before starting the node. Feel free to do so by editing the files in the configs directory but please be advised that it's easy to break things! If you are not going to make any changes you can move onto the next section.

To view the configuration of a node, run the show config command. Replace <node-id> with the ID outputed by the configure command:

bpm nodes show config <node-id>

Similarly it is possible to show the configure parameters for the node using the show node command:

bpm nodes show node <node-id>

This concludes the node configuration steps using BPM. The node is now configured but not launched.

Starting, stopping and removing the node is covered in detail in Tutorial 2.

To summarize, use the following commands to manage the node:

bpm nodes start <node-id>
bpm nodes stop <node-id>
bpm nodes remove --all <node-id>

Usage

BPM CLI comes with a built-in help functionality for all commands. You can view this information by adding --help to any command.

Examples:

bpm --help
bpm packages --help
bpm nodes --help
bpm nodes configure --help

The intention of this Usage guide is to provide additional information that is not already covered by --help on any given command. For examples on how to use BPM end-to-end please consult the Quickstart tutorials.

Terms and concepts

bpm : The Blockchain Package Manager is a combination of tools to simplify configuring and running Blockchain nodes in an enterprise environment.

bpm-cli : The bpm-cli is the main user interface with which users will manage packages and nodes. It can be invoked by running bpm <command>.

bpm-sdk : The bpm-sdk is a software library written in Go that makes it easy to develop new packages.

bpr : The Blockchain Package Registry is a server component that allows to search for packages and versions of packages. The main interface to interact with bpr are the bpm packages commands within the bpm-cli.

package : A BPM package contains the functionality to launch a specific Blockchain node. This typically involves creating secrets, configuration files and starting the Blockchain client as well as monitoring agents. Packages can be managed via the bpm packages commands.

node : A BPM node is a combination of the configuration, data, runtime as well as monitoring agents of a particular Blockchain node. Nodes can be started and stopped. Nodes can be managed via the bpm nodes commands.

monitoring : A key component of BPM is that all nodes are deployed with monitoring agents. Monitoring data can be sent to any Logstash instance or to the proprietary Blockdaemon monitoring system.

node-id : Each BPM node gets assigned a unique ID that is used with the bpm nodes commands. (Example: bnn4gnjo6him0f09h07g)

Upgrades

Even considering the general speed of changes in information technology, Blockchain is an outlier because it changes even faster. Some Blockchains release new versions weekly and it is important to stay up-to-date for security reasons, to use the latest features as well as to be able to connect to the Blockchain. Not upgrading carries the risk of eventually being left behind or connecting to a fork of the Blockchain.

The BPM CLI regularly checks if it is running with the latest version. If you see a message like:

bpm version "0.8.0" is available. Please upgrade as soon as possible!

It is advised to install this newer version of the BPM CLI as soon as possible. Please see the Quickstart tutorials for installation instructions.

Similarly the BPM CLI will check if a particular package is up-to-date before configuring a new node. Since it is so crucial to always use the latest version of a Blockchain client, BPM will not allow configuring of new nodes with outdated packages unless specifically instructed by the user.

Error: A new version of package "parity" is available. Please install using "bpm install parity" or skip this check using "--skip-upgrade-check".

As described in the output one can always add --skip-upgrade-check to run an older version of a package. Do so at your own risk!

To upgrade a package, just install it again:

bpm packages install <package>

For development purposes it is possible to install a specific version using:

bpm packages install <package> <version>

To view a list of installed packages with their version and the recommended version to use, run:

bpm packages list

Directories

Note: BPM doesn't dictate where a Blockchain node should be run or how it is being configured. At the moment most packages create configuration files and will start Docker containers. In the future there may be packages that, for example, create Kubernetes secrets and configuration instead of configuration and secrets directories.

By default BPM writes it's configuration to ~/.bpm. This default can be changed by supplying a --base-dir parameter to all commands.

~/.bpm/manifest.json : Contains version information and details about the installed plugins. You should never need to edit this file manually.

~/.bpm/plugins : Contains binaries for the installed packages. While it is possible to run these binaries directly for development purposes it is recommended to use the bpm command as the main interface for end-users.

~/.bpm/nodes/<node-id>/node.json : Contains configuration parameters and meta-information about an existing node.

~/.bpm/nodes/<node-id>/configs/* : Contains configuration files for a particular node. Feel free to manually edit these files before launching a node if you need special configuration but also be aware that this may break things.

~/.bpm/nodes/<node-id>/secrets/* : Contains private keys, accounts and passwords for a particular node. Always keep a secure backup of this directory. To supply your own secrets, you can overwrite the existing files in this directory before re-configuring a node.

Warning: Both secrets and configs can contain sensitive information that should not be shared with unauthorized people. The main difference between the two is that configs can always be re-generated as long as secrets is available and the initial configuration parameters are known.

Setting defaults for the BPM CLI

To avoid having to set parameters with every single bpm call, it is possible to definee them using environment variables.

Instead of specifying --base-dir ~/.other-bpm one can set an environment variable like this:

export BPM_BASE_DIR=~/.other-bpm

To enable debug mode all the time set this environment variable:

export BPM_DEBUG=true

Automation

BPM sometimes asks for user confirmation. To use it in an automated non-interactive environment use the --yes parameter to always answer yes when prompted. Be careful and use only if you know what you are doing!

Troubleshooting

  • If somethign goes wrong, the first step should usually be to run the command again using the --debug parameter. This will print additional debug information which can aid in troubleshooting the issue.

  • If a bpm nodes start command fails there is a good chance that something went wrong with the docker containers. Use the typical docker commands like docker ps or docker logs to see if all containers are running and whether there are any errors in the logs.

  • Temporary docker issues can sometimes be resolved by removing all containers (bpm nodes remove --runtime <node-id>) followed by starting them again bpm nodes start <node-id>.

Please report any issues to Github or contact support@blockdaemon.com.

 

 

Design Goals

We embrace decentralization

A BPM managed node should be run anywhere by anyone. Never should it be possible for a "service provider" (hint: we are one!) to change or modify a BPM node without the administrators explicit consent.

This has certain implications:

  • BPM will not provide a way to manage the nodes "from outside"
  • Monitoring data can flow from the node to a central point but not vice-versa.
  • BPM will not auto-upgrade nodes without a users explicit consent.

One interface to manage all blockchains while keeping most flexilibity

Packages abstract away the details of a Blockchain client and only expose a clearly defined interface.

Blockchain is a bleeding-edge technology that constantly changes. While a clearly defined interface is desirable most of the time we want to enable users to edit configuration files easily and run one-off tasks where necessary for highly specialized use-cases. This is achieved by a two-step approach that creates a default configuration first, optionally let's the user manually edit this configuration and only then does the node get started.

Simple by default; Flexible when necessary

Blockchain protocols/clients vary greatly; BPM must be flexible enough to accompany them all.

This is achieved by:

  • Keeping the bpm-cli itself as simple as possible and putting most of the actual logic in the packages allows to change nearly everything a package can do.
  • Depending on the level of customization necessary, the BPM SDK provides different entry points. For exapmle there are pre-defined functions to create a Docker based package but one can always supply their own customized functions.
  • The package interface is clearly defined. If the BPM SDK proves to be too limiting for a specific task it is always possible to write a package from scratch using an programming language.

 

Run nodes everywhere

BPM is not designed with a paritcular deployment target in mind. While currently most Blockchain deploy to Docker it is possible to write packages that run directly on bare-metal or deploy to Kuberneters. This is achieved by putting most logic into the package or BPM SDK.

Fast development cycle for new packages

Packages can be developed by launching them directly (not advised for end-users!) and individual steps can be re-run easily (see idempotency below). This allows to quickly iterate during the development of packages.

Allowing developers (as well as users!) to edit configuration files manually helps with quickly testing out new ideas and configurations.

 

  • Make sure to return correct error codes
  • If a interactive prompt is used, provide a way to also use it non-interactively (--yes)
  • Make use of idempotency to allow the same step to be re-run without accidentally overwriting

An operation is idempotent if the result of performing it once is exactly the same as the result of performing it repeatedly without any intervening actions. [1]

 

Further reading

Blockdaemon BPM docs

0 out of 0 found this helpful

Comments

0 comments

Please sign in to leave a comment.