The following tutorials give you a quick overview of BPM. We will install the BPM CLI and use it to configure and run Blockchain nodes.
The BPM CLI serves two main purposes:
packages. A BPM
packagecontains the functionality to launch a specific Blockchain node. There is typically one package per supported Blockchain client.
nodes. A BPM
nodeis a combination of the configuration, data, runtime as well as monitoring agents of a particular Blockchain node. Nodes can be started and stopped.
Note: While it is possible to run multiple Blockchain nodes at the same time we typically recommend to only have one node started at any time. There are multiple reasons for this but the main ones are as follows:
- Blockchain nodes are resource intensive and running more than one can easily overwhelm the server.
- A lot of Blockchain protocols use the same common default ports which causes conflicts if more than one is run at the same time. While it is possible to change those ports, care has to be taken to avoid using the same ones.
If you ran a development version of BPM, please make sure to wipe any remnants of that before continuing:
- Remove any BPM related docker containers.
- Remove any BPM related docker networks.
Installing the BPM CLI
Note: In this section, you will need to replace values for placeholder found between
linuxdepending on your operating system.
<VERSION>with the BPM version, e.g.
Using wget you will download the binary for your OS. Then you will copy it to the correct bin directory path and set it to an executable binary.
wget <BPM_URL> sudo cp bpm-<VERSION>-<OS>-amd64 /usr/local/bin/bpm sudo chmod 755 /usr/local/bin/bpm
Tutorial 1: Managing packages
In this first tutorial you will learn how to install and manage BPM packages. To get an overview over what is possible, let's start with listing all commands available for managing packages:
bpm packages --help
The very first time you run
bpm it needs to initialize it's directories, you should see the following output:
Looks like bpm isn't initialized correctly in "/Users/blockdaemon/.bpm", do you want to do that now? (y/N): y
y to start the BPM initialization procedure.
You should see the following output:
Manage packages Usage: bpm packages [command] Available Commands: info Show information about a package install Installs or upgrades a package to a specific version or latest if no version is specified list List installed packages search Search available packages uninstall Uninstall a package. Data and configuration will not be removed. Flags: -h, --help help for packages Global Flags: --base-dir string The directory plugins and configuration are stored (default "~/.bpm/") --debug Enable debug output --package-registry string The package registry provides packages to install (default "https://dev.registry.blockdaemon.com") -y, --yes Automatic yes to prompts; assume "yes" as answer to all prompts and run non-interactively Use "bpm packages [command] --help" for more information about a command.
The first step is usually to search for a package to install. To list all available packages, just run the
search command without a search parameter:
bpm packages search
You can also search for specific protocol packages by providing a search parameter (e.g. parity):
bpm packages search parity
After you have found a suitable packages, install it:
bpm packages install parity
This can take a few minutes while it is downloading the package.
To check which packages have already beeen installed, run the
bpm packages list
You should see similar output:
NAME | INSTALLED VERSION | RECOMMENDED VERSION +--------+-------------------+---------------------+ parity | 1.0.0 | 1.0.0
This tells us that the parity package is installed using the latest recommended version.
To show more information about a package, use the
bpm packages info parity
Finally, if a package isn't needed anymore, it can be uninstalled:
bpm packages uninstall parity
Tutorial 2: Starting a Polkadot testnet node
First, let's list all commands available for managing nodes:
bpm nodes --help
You should see the following output:
Manage blockchain nodes Usage: bpm nodes [command] Available Commands: configure Configure a new blockchain node remove Remove blockchain node data and configuration show Print a resource to stdout start Start a blockchain node status Display statuses of configured nodes stop Stops a running blockchain node test Tests a running blockchain node Flags: -h, --help help for nodes Global Flags: --base-dir string The directory plugins and configuration are stored (default "~/.bpm/") --debug Enable debug output --package-registry string The package registry provides packages to install (default "https://dev.registry.blockdaemon.com") -y, --yes Automatic yes to prompts; assume "yes" as answer to all prompts and run non-interactively Use "bpm nodes [command] --help" for more information about a command.
Installing the polkadot package
Before diving into the actual node management we need to install the polkadot package:
bpm packages install polkadot
Configuring the node
Run the configure command with
--help to see the available parameters:
bpm nodes configure polkadot --help
There are two parameters that are specific to the polkadot package:
--subtype watcher The type of node. Must be either watcher or `validator` (default "watcher") --validator-key string The key used for a validator (required if subtype = validator)
For now we will leave the defaults which creates a configuration for a non-validating polkadot node. Run the configure command to create it now:
bpm nodes configure polkadot
You should see an output similar to this (but with different IDs):
No credentials found in "/Users/jonas/.bpm/beats", skipping configuration of Blockdaemon monitoring. Please configure your own monitoring in the node configuration files. Nothing to do here, skipping create-secrets Writing file '/Users/jonas/.bpm/nodes/bob3rhstp9lpfnasuff0/configs/filebeat.yml' Writing file '/Users/jonas/.bpm/nodes/bob3rhstp9lpfnasuff0/configs/polkadot.dockercmd' Writing file '/Users/jonas/.bpm/nodes/bob3rhstp9lpfnasuff0/configs/polkadotbeat.yml' Node with id "bob3rhstp9lpfnasuff0" has been initialized. To change the configuration, modify the files here: /Users/jonas/.bpm/nodes/bob3rhstp9lpfnasuff0/configs To start the node, run: bpm nodes start bob3rhstp9lpfnasuff0 To see the status of configured nodes, run: bpm nodes status
You can verify the status of all nodes by running the
bpm nodes status
Starting and stopping the node
To start the node, run the
node start command. Replace
<node-id> with the ID outputed by the
bpm nodes start <node-id>
You should get an output similar to:
Creating network 'bpm-bob3rhstp9lpfnasuff0-polkadot' Network 'bpm-bob3rhstp9lpfnasuff0-polkadot' already exists, skipping creation Network 'bpm-bob3rhstp9lpfnasuff0-polkadot' already exists, skipping creation Creating container 'bpm-bob3rhstp9lpfnasuff0-polkadot' Starting container 'bpm-bob3rhstp9lpfnasuff0-polkadot' Creating container 'bpm-bob3rhstp9lpfnasuff0-polkadotbeat' Starting container 'bpm-bob3rhstp9lpfnasuff0-polkadotbeat' Creating container 'bpm-bob3rhstp9lpfnasuff0-filebeat' Starting container 'bpm-bob3rhstp9lpfnasuff0-filebeat' The node "bob3rhstp9lpfnasuff0" has been started.
This shows BPM creating a dedicated Docker network for this node as well as starting the Docker containers for Parity and the monitoring agents.
Verify the node status by running:
bpm nodes status
The node should now show up as
running, similar to below:
NODE ID | PACKAGE | STATUS | SECRETS +----------------------+---------+----------+---------+ bob3rhstp9lpfnasuff0 | running | polkadot | 0
To stop it temporarily, run:
bpm nodes stop <node-id>
Removing a node
When removing a node you need to consider the following. A node consists of:
- Secrets (e.g. accounts, passwords, private keys)
- Node configuration
- Runtime (e.g. Docker networks and containers)
- Data (typically the parts of the Blockchain that have already been synced)
Depending on the use-case it may be desirable to remove all or only parts of the node. For example:
- In order to re-configure a node one might only want to remove the configuration but leave the data intact to avoid having to re-sync the Blockchain
- If the node crashed due to an unexpected error it can make sense to remove the runtime and start it again but keep the configuration and data
- If something went wrong during the initial sync it can help to remove the data and then start the node again to start syncing from scratch
To support the above use-cases plus others we have allowed parameters/flags to be used with our
You can view all the available parameters/flags by running the following BPM command:
bpm nodes remove --help
Which should return:
Remove blockchain node data and configuration Usage: bpm nodes remove <id> [flags] Flags: --all Remove all data, configuration files and node information --config Remove all configuration files but keep data and node information --data Remove all data but keep configuration files and node information -h, --help help for remove --runtime Remove all runtimes but keep configuration files and node information Global Flags: --base-dir string The directory plugins and configuration are stored (default "~/.bpm/") --debug Enable debug output --package-registry string The package registry provides packages to install (default "https://dev.registry.blockdaemon.com") -y, --yes Automatic yes to prompts; assume "yes" as answer to all prompts and run non-interactively
For now, let's remove the whole node:
nodes remove --all <node-id>
Warning: By removing the whole node you will also remove the
secrets directory. It's always advisable to backup the
secrets directory to a safe place before doing anything with the node.
Tutorial 3: Connect to an existing permissioned Parity network
Installing the parity package
The first step is to install the parity package:
bpm packages install parity
Configuring the node
The Parity package is built specifically to run a node for a permissioned Blockchain network. It takes two parameters:
- A comma separated list of enode URLs that can act as bootnodes. These will serve as a list of peers that a new BPM Parity node will connect to.
- A chain specification that contains the configuration for the permissioned Blockchain network.
Note: If you've used Blockdaemon to launch your Blockchain network you can find both parameters on your Blockdaemon dashboard. Otherwise please ask the administrator of the permissioned Blockchain network for parameters.
Let's start configuring a node using the
bpm nodes configure parity --bootnodes "<enode_URLs>" --chain-spec <chain-spec_filepath>
Depending on your specific parameters this could look something like:
bpm nodes configure parity --bootnodes "enode://firstname.lastname@example.org:30303,enode://email@example.com:30303" --chain-spec ~/Downloads/chain.json
You should see an output similar to this (but with different IDs):
generating accounts... generating accounts... Copying chain spec file to '/Users/blockdaemon/.bpm/nodes/bnn4trbo6him2ii9uca0/configs/chain.json' Copying signer account file to '/Users/blockdaemon/.bpm/nodes/bnn4trbo6him2ii9uca0/configs/keys/DemoPoA/signer.json' Copying signer password file file to '/Users/blockdaemon/.bpm/nodes/bnn4trbo6him2ii9uca0/configs/password.secret' Writing file '/Users/blockdaemon/.bpm/nodes/bnn4trbo6him2ii9uca0/configs/parity.dockercmd' Writing file '/Users/blockdaemon/.bpm/nodes/bnn4trbo6him2ii9uca0/configs/filebeat.yml' Node with id "bnn4trbo6him2ii9uca0" has been initialized. To change the configuration, modify the files here: /Users/blockdaemon/.bpm/nodes/bnn4trbo6him2ii9uca0/configs To start the node, run: bpm nodes start bnn4trbo6him2ii9uca0 To see the status of configured nodes, run: bpm nodes status
configure step is deliberately separated from the
start step. This enables users to manually change the configuration before starting the node. Feel free to do so by editing the files in the
configs directory but please be advised that it's easy to break things! If you are not going to make any changes you can move onto the next section.
To view the configuration of a node, run the
show config command. Replace
<node-id> with the ID outputed by the
bpm nodes show config <node-id>
Similarly it is possible to show the
configure parameters for the node using the
show node command:
bpm nodes show node <node-id>
This concludes the node configuration steps using BPM. The node is now configured but not launched.
Starting, stopping and removing the node is covered in detail in Tutorial 2.
To summarize, use the following commands to manage the node:
bpm nodes start <node-id> bpm nodes stop <node-id> bpm nodes remove --all <node-id>
BPM CLI comes with a built-in help functionality for all commands. You can view this information by adding
--help to any command.
bpm --help bpm packages --help bpm nodes --help bpm nodes configure --help
The intention of this Usage guide is to provide additional information that is not already covered by
--help on any given command. For examples on how to use BPM end-to-end please consult the Quickstart tutorials.
Terms and concepts
bpm : The Blockchain Package Manager is a combination of tools to simplify configuring and running Blockchain nodes in an enterprise environment.
bpm-cli : The bpm-cli is the main user interface with which users will manage packages and nodes. It can be invoked by running
bpm-sdk : The bpm-sdk is a software library written in Go that makes it easy to develop new packages.
bpr : The Blockchain Package Registry is a server component that allows to search for packages and versions of packages. The main interface to interact with bpr are the
bpm packages commands within the bpm-cli.
package : A BPM package contains the functionality to launch a specific Blockchain node. This typically involves creating secrets, configuration files and starting the Blockchain client as well as monitoring agents. Packages can be managed via the
bpm packages commands.
node : A BPM node is a combination of the configuration, data, runtime as well as monitoring agents of a particular Blockchain node. Nodes can be started and stopped. Nodes can be managed via the
bpm nodes commands.
monitoring : A key component of BPM is that all nodes are deployed with monitoring agents. Monitoring data can be sent to any Logstash instance or to the proprietary Blockdaemon monitoring system.
node-id : Each BPM node gets assigned a unique ID that is used with the
bpm nodes commands. (Example:
Even considering the general speed of changes in information technology, Blockchain is an outlier because it changes even faster. Some Blockchains release new versions weekly and it is important to stay up-to-date for security reasons, to use the latest features as well as to be able to connect to the Blockchain. Not upgrading carries the risk of eventually being left behind or connecting to a fork of the Blockchain.
The BPM CLI regularly checks if it is running with the latest version. If you see a message like:
bpm version "0.8.0" is available. Please upgrade as soon as possible!
It is advised to install this newer version of the BPM CLI as soon as possible. Please see the Quickstart tutorials for installation instructions.
Similarly the BPM CLI will check if a particular package is up-to-date before configuring a new node. Since it is so crucial to always use the latest version of a Blockchain client, BPM will not allow configuring of new nodes with outdated packages unless specifically instructed by the user.
Error: A new version of package "parity" is available. Please install using "bpm install parity" or skip this check using "--skip-upgrade-check".
As described in the output one can always add
--skip-upgrade-check to run an older version of a package. Do so at your own risk!
To upgrade a package, just install it again:
bpm packages install <package>
For development purposes it is possible to install a specific version using:
bpm packages install <package> <version>
To view a list of installed packages with their version and the recommended version to use, run:
bpm packages list
Note: BPM doesn't dictate where a Blockchain node should be run or how it is being configured. At the moment most packages create configuration files and will start Docker containers. In the future there may be packages that, for example, create Kubernetes secrets and configuration instead of configuration and secrets directories.
By default BPM writes it's configuration to
~/.bpm. This default can be changed by supplying a
--base-dir parameter to all commands.
~/.bpm/manifest.json : Contains version information and details about the installed plugins. You should never need to edit this file manually.
~/.bpm/plugins : Contains binaries for the installed packages. While it is possible to run these binaries directly for development purposes it is recommended to use the
bpm command as the main interface for end-users.
~/.bpm/nodes/<node-id>/node.json : Contains configuration parameters and meta-information about an existing node.
~/.bpm/nodes/<node-id>/configs/* : Contains configuration files for a particular node. Feel free to manually edit these files before launching a node if you need special configuration but also be aware that this may break things.
~/.bpm/nodes/<node-id>/secrets/* : Contains private keys, accounts and passwords for a particular node. Always keep a secure backup of this directory. To supply your own secrets, you can overwrite the existing files in this directory before re-configuring a node.
configs can contain sensitive information that should not be shared with unauthorized people. The main difference between the two is that
configs can always be re-generated as long as
secrets is available and the initial configuration parameters are known.
Setting defaults for the BPM CLI
To avoid having to set parameters with every single
bpm call, it is possible to definee them using environment variables.
Instead of specifying
--base-dir ~/.other-bpm one can set an environment variable like this:
To enable debug mode all the time set this environment variable:
BPM sometimes asks for user confirmation. To use it in an automated non-interactive environment use the
--yes parameter to always answer yes when prompted. Be careful and use only if you know what you are doing!
If somethign goes wrong, the first step should usually be to run the command again using the
--debugparameter. This will print additional debug information which can aid in troubleshooting the issue.
bpm nodes startcommand fails there is a good chance that something went wrong with the docker containers. Use the typical docker commands like
docker logsto see if all containers are running and whether there are any errors in the logs.
Temporary docker issues can sometimes be resolved by removing all containers (
bpm nodes remove --runtime <node-id>) followed by starting them again
bpm nodes start <node-id>.
We embrace decentralization
A BPM managed node should be run anywhere by anyone. Never should it be possible for a "service provider" (hint: we are one!) to change or modify a BPM node without the administrators explicit consent.
This has certain implications:
- BPM will not provide a way to manage the nodes "from outside"
- Monitoring data can flow from the node to a central point but not vice-versa.
- BPM will not auto-upgrade nodes without a users explicit consent.
One interface to manage all blockchains while keeping most flexilibity
Packages abstract away the details of a Blockchain client and only expose a clearly defined interface.
Blockchain is a bleeding-edge technology that constantly changes. While a clearly defined interface is desirable most of the time we want to enable users to edit configuration files easily and run one-off tasks where necessary for highly specialized use-cases. This is achieved by a two-step approach that creates a default configuration first, optionally let's the user manually edit this configuration and only then does the node get started.
Simple by default; Flexible when necessary
Blockchain protocols/clients vary greatly; BPM must be flexible enough to accompany them all.
This is achieved by:
- Keeping the
bpm-cliitself as simple as possible and putting most of the actual logic in the packages allows to change nearly everything a package can do.
- Depending on the level of customization necessary, the BPM SDK provides different entry points. For exapmle there are pre-defined functions to create a Docker based package but one can always supply their own customized functions.
- The package interface is clearly defined. If the BPM SDK proves to be too limiting for a specific task it is always possible to write a package from scratch using an programming language.
Run nodes everywhere
BPM is not designed with a paritcular deployment target in mind. While currently most Blockchain deploy to Docker it is possible to write packages that run directly on bare-metal or deploy to Kuberneters. This is achieved by putting most logic into the package or BPM SDK.
Fast development cycle for new packages
Packages can be developed by launching them directly (not advised for end-users!) and individual steps can be re-run easily (see idempotency below). This allows to quickly iterate during the development of packages.
Allowing developers (as well as users!) to edit configuration files manually helps with quickly testing out new ideas and configurations.
- Make sure to return correct error codes
- If a interactive prompt is used, provide a way to also use it non-interactively (
- Make use of idempotency to allow the same step to be re-run without accidentally overwriting
An operation is idempotent if the result of performing it once is exactly the same as the result of performing it repeatedly without any intervening actions.