www.openlinksw.com
docs.openlinksw.com

Book Home

Contents
Preface

Installation Guide

Virtuoso on All platforms Commmon Specifics
Virtuoso for Windows
Installing the Virtuoso Universal Server on Unix-like OS (AIX, HP-UX, Linux, Solaris, etc.)
Virtuoso for Mac OS X
OpenLink License Management
Virtuoso ADO.Net Data Grid Form Application
Using Visual Studio 2008 to Build an Entity Frameworks based Windows Form Application
Using Visual Studio 2008 to Build an ADO.NET Data Services based Application
Windows Form Application for accessing Virtuoso RDF data via SPASQL using the Virtuoso ADO.Net Provider
Creating a Web Browser Application to Access RDF Data Using The Virtuoso ADO.Net Provider
Creating a Silverlight Application to consume the service
Creating A Simple .NET RIA Services Application To Display Data From Virtuoso
Creating a .Net RIA Services Application That Will Update Virtuoso Data
Cluster Installation and Configuration
Virtuoso Elastic Cluster Installation & Configuration -- Version 7.x Virtuoso default Cluster Installation and Configuration Backup and Restore Cluster Configuration HTTP Service Configuration on Subordinate Nodes of a Virtuoso Cluster Troubleshooting Tips

2.14. Cluster Installation and Configuration

2.14.1. Virtuoso Elastic Cluster Installation & Configuration -- Version 7.x

These sections apply to Virtuoso as of version 7.x .

2.14.1.1. What

Virtuoso 7 and later versions can be configured in Elastic scale out cluster mode. The data is sharded in a large number of self-contained partitions. These partitions are divided among a number of database server processes and can migrate between them.


2.14.1.2. Why

An Elastic cluster can be dynamically resized with new partitions or shards dynamically added to the same or new hardware resources as the need to increase the size of the scale cluster or relocation of partitions is required.


2.14.1.3. How

This documentation details the steps for the installation and configuration of a Virtuoso Elastic Cluster on Unix:

  1. Step 1: Perform Virtuoso 7 Unix installation;
  2. Step 2: Determine Elastic Cluster size;
  3. Step 3: Enable Elastic Cluster mode;
  4. Step 4: Start Elastic Cluster;
  5. Step 5: Splitting Cluster nodes across different machines.

2.14.1.4. Virtuoso 7 Unix installation

Step 1: Perform Virtuoso 7 Unix installation

$ tar xvf virtuoso-universal-server-7.x.tar
x install.sh
x universal-server.taz
$ sh install.sh

- Extracting Virtuoso Universal Server v7.x

- Creating default environment settings

- Creating default database settings
Configuring: database
Creating directory $VIRTUOSO_HOME/database/backup
Installing new virtuoso.ini in $VIRTUOSO_HOME/database
Installing new php.ini in $VIRTUOSO_HOME/database
Creating symlink to $VIRTUOSO_HOME/bin/virtuoso-iodbc-t

- Registering ODBC drivers

- Registering .NET provider for Mono

- Installing VAD packages in database (this can take some time)

- Checking where license should be stored

- Starting OpenLink License Manager

- Checking for initial Virtuoso license

- Starting Virtuoso server instance

- Finalizing installation


This concludes the first part of the installation.


- Attempting to start browser

Please start a browser manually and open the following URL to finalize
the installation process:

     http://cname:8890/install/


Installation completed
$

2.14.1.5. Determine Elastic Cluster size

Step 2: Determine Elastic Cluster size

  1. Determine how many nodes you want to start the elastic cluster with: 2, 3, 4, 5, 6, 7, 8 etc. Note: the number of the nodes must be >= 2
  2. Setup Virtuoso environment and stop default database:
    $ . ./virtuoso-environment.sh
    $ virtuoso-stop.sh
    Shutting down Virtuoso instance in [database]
    $
    
  3. Optionally remove the default database such that is does not get started with the cluster:
    rm  -rf database
    
  4. Full list of supported options for the script "virtuoso-mkcluster.sh":
      -cluster-size       Number of nodes in cluster
      -cluster-node       Node number in a cluster
      -cluster-port       Base portnumer for cluster
      -cluster-ipc-port   Base portnumer for cluster IPC
      -cluster_size       Size of the cluster
      -virtuoso_home      Virtuoso home path
    
  5. Run the virtuoso-mkcluster.sh script to create the cluster, note the default number of nodes is 4, but this can be changed by setting the environment variable CLUSTER_SIZE to the required number of nodes to be created before running the script:
    $ virtuoso-mkcluster.sh
    Configuring cluster node: 1/4
    
    Creating directory $VIRTUOSO_HOME/cluster_01
    Creating directory $VIRTUOSO_HOME/cluster_01/backup
    Creating directory $VIRTUOSO_HOME/cluster_01/logs
    Installing new virtuoso.ini in $VIRTUOSO_HOME/cluster_01
    Installing new php.ini in $VIRTUOSO_HOME/cluster_01
    Creating symlink to $VIRTUOSO_HOME/bin/virtuoso-iodbc-t
    Configuring cluster node: 2/4
    
    Creating directory $VIRTUOSO_HOME/cluster_02
    Creating directory $VIRTUOSO_HOME/cluster_02/backup
    Installing new virtuoso.ini in $VIRTUOSO_HOME/cluster_02
    Removing unneeded sections from virtuoso.ini
    Creating symlink to $VIRTUOSO_HOME/bin/virtuoso-iodbc-t
    Configuring cluster node: 3/4
    
    Creating directory $VIRTUOSO_HOME/cluster_03
    Creating directory $VIRTUOSO_HOME/cluster_03/backup
    Installing new virtuoso.ini in $VIRTUOSO_HOME/cluster_03
    Removing unneeded sections from virtuoso.ini
    Creating symlink to $VIRTUOSO_HOME/bin/virtuoso-iodbc-t
    Configuring cluster node: 4/4
    
    Creating directory $VIRTUOSO_HOME/cluster_04
    Creating directory $VIRTUOSO_HOME/cluster_04/backup
    Installing new virtuoso.ini in $VIRTUOSO_HOME/cluster_04
    Removing unneeded sections from virtuoso.ini
    Creating symlink to $VIRTUOSO_HOME/bin/virtuoso-iodbc-t
    $
    
  6. In each cluster_XX directory the following the cluster.ini files are created for cluster internal communication:
    $ more cluster_01/cluster.ini
    [Cluster]
    Threads             = 20
    Master              = Host1
    ThisHost            = Host1
    ReqBatchSize        = 10000
    BatchesPerRPC       = 4
    BatchBufferBytes    = 20000
    LocalOnly           = 2
    MaxKeepAlivesMissed = 1000
    Host1               = localhost:22201
    Host2               = localhost:22202
    Host3               = localhost:22203
    Host4               = localhost:22204
    MaxHosts            = 5
    $ more cluster_02/cluster.ini
    [Cluster]
    Threads             = 20
    Master              = Host1
    ThisHost            = Host2
    ReqBatchSize        = 10000
    BatchesPerRPC       = 4
    BatchBufferBytes    = 20000
    LocalOnly           = 2
    MaxKeepAlivesMissed = 1000
    Host1               = localhost:22201
    Host2               = localhost:22202
    Host3               = localhost:22203
    Host4               = localhost:22204
    MaxHosts            = 5
    $ more cluster_03/cluster.ini
    [Cluster]
    Threads             = 20
    Master              = Host1
    ThisHost            = Host3
    ReqBatchSize        = 10000
    BatchesPerRPC       = 4
    BatchBufferBytes    = 20000
    LocalOnly           = 2
    MaxKeepAlivesMissed = 1000
    Host1               = localhost:22201
    Host2               = localhost:22202
    Host3               = localhost:22203
    Host4               = localhost:22204
    MaxHosts            = 5
    $ more cluster_04/cluster.ini
    [Cluster]
    Threads             = 20
    Master              = Host1
    ThisHost            = Host4
    ReqBatchSize        = 10000
    BatchesPerRPC       = 4
    BatchBufferBytes    = 20000
    LocalOnly           = 2
    MaxKeepAlivesMissed = 1000
    Host1               = localhost:22201
    Host2               = localhost:22202
    Host3               = localhost:22203
    Host4               = localhost:22204
    MaxHosts            = 5
    $
    

2.14.1.6. Enable Elastic Cluster mode

Step 3: Enable Elastic Cluster mode

  1. The cluster.ini files need to be reconfigured as detailed below for elastic cluster and file slicing/sharding to be enabled.
  2. A common file called cluster.global.ini can be created and placed in the home directory of the Virtuoso installation:
    [Cluster]
    Threads             = 300
    ReqBatchSize        = 10000
    BatchesPerRPC       = 4
    BatchBufferBytes    = 20000
    LocalOnly           = 2
    MaxKeepAlivesMissed = 100
    RDFLoadBytes        = 52428800
    Host1               = localhost:22201
    Host2               = localhost:22202
    Host3               = localhost:22203
    Host4               = localhost:22204
    
  3. This file can then be sym-linked to each of the cluster_XX directories of the cluster, and its content will be merged with the cluster.ini in the respective directory of each node when starting the cluster.
    ln -s cluster.global.ini cluster_01/cluster.global.ini
    ln -s cluster.global.ini cluster_02/cluster.global.ini
    ln -s cluster.global.ini cluster_03/cluster.global.ini
    ln -s cluster.global.ini cluster_04/cluster.global.ini
    
  4. Edit each of the the cluster.ini files in the cluster_XX directories as follows to enable, elastic cluster mode:
    $ more cluster_01/cluster.ini
    [Cluster]
    Master              = Host1
    ThisHost            = Host1
    
    [ELASTIC]
    Segment1 = 4G, database.db = q1
    Slices = 6
    MaxSlices = 2048
    
    $ more cluster_02/cluster.ini
    
    [Cluster]
    Master              = Host1
    ThisHost            = Host2
    
    [ELASTIC]
    Segment1 = 4G, database.db = q1
    Slices = 6
    MaxSlices = 2048
    
    $ more cluster_03/cluster.ini
    
    [Cluster]
    Master              = Host1
    ThisHost            = Host3
    
    [ELASTIC]
    Segment1 = 4G, database.db = q1
    Slices = 6
    MaxSlices = 2048
    
    $ more cluster_04/cluster.ini
    
    [Cluster]
    Master              = Host1
    ThisHost            = Host4
    
    [ELASTIC]
    Segment1 = 4G, database.db = q1
    Slices = 6
    MaxSlices = 2048
    
  5. The [ELASTIC] section in the cluster.ini files above, enables the elastic cluster mode, where multiple segments and stripes as detailed in the standard Virtuoso documentation database striping.
  6. The MaxSlices parameter above sets the the maximum number of physical slices in the cluster.

2.14.1.7. Start Elastic Cluster

Step 4: Start Elastic Cluster

  1. Start the Elastic cluster using the standard virtuoso-start.sh script run from the home directory of the Virtuoso installation,which automatically detects the number for nodes to be started:
    
    $ virtuoso-start.sh
    Starting Virtuoso instance in [cluster_01]
    Starting Virtuoso instance in [cluster_02]
    Starting Virtuoso instance in [cluster_03]
    Starting Virtuoso instance in [cluster_04]
    $
    
  2. The default SQL port of the master node is 12201, as indicated in the virtuoso.ini file of the cluster_01 directory, and can then be used for connecting to the newly created cluster and check its status to ensure all nodes are online:
    $ isql 12201
    Connected to OpenLink Virtuoso
    Driver: 07.10.3211 OpenLink Virtuoso ODBC Driver
    OpenLink Interactive SQL (Virtuoso), version 0.9849b.
    Type HELP; for help and EXIT; to exit.
    SQL> status('cluster_d');
    REPORT
    VARCHAR
    _______________________________________________________________________________
    
    Cluster: No samples, Please refresh
    
    1 Rows. -- 22 msec.
    SQL> status('cluster_d');
    REPORT
    VARCHAR
    _______________________________________________________________________________
    
    Cluster 4 nodes, 2 s. 5 m/s 0 KB/s  5% cpu 0%  read 0% clw threads 1r 0w 0i buffers 349144 2 d 0 w 0 pfs
    cl 1: 2 m/s 0 KB/s  0% cpu 0%  read 0% clw threads 1r 0w 0i buffers 45391 2 d 0 w 0 pfs
    cl 2: 0 m/s 0 KB/s  0% cpu 0%  read 0% clw threads 0r 0w 0i buffers 43367 0 d 0 w 0 pfs
    cl 3: 0 m/s 0 KB/s  0% cpu 0%  read 0% clw threads 0r 0w 0i buffers 50129 0 d 0 w 0 pfs
    cl 4: 0 m/s 0 KB/s  0% cpu 0%  read 0% clw threads 0r 0w 0i buffers 39383 0 d 0 w 0 pfs
    
    5 Rows. -- 22 msec.
    SQL>
    SQL> cl_ping(1,500, 1000);
    
    Done. -- 7 msec.
    SQL> cl_ping(2,500, 1000);
    
    Done. -- 52 msec.
    SQL> cl_ping(3,500, 1000);
    
    Done. -- 60 msec.
    SQL> cl_ping(4,500, 1000);
    
    Done. -- 51 msec.
    SQL>
    
  3. The database.log of the master node can be checked to verify the Elastic cluster mode has been enable on server startup, which will contain the entry PL LOG: Elastic cluster setup:
    $ more cluster_01/database.log
    
                    Thu Oct 09 2014
    05:11:49 { Loading plugin 1: Type `plain', file `wikiv' in `../hosting'
    05:11:49   WikiV version 0.6 from OpenLink Software
    05:11:49   Support functions for WikiV collaboration tool
    05:11:49   SUCCESS plugin 1: loaded from ../hosting/wikiv.so }
    05:11:49 { Loading plugin 2: Type `plain', file `mediawiki' in `../hosting'
    05:11:49   MediaWiki version 0.1 from OpenLink Software
    05:11:49   Support functions for MediaWiki collaboration tool
    05:11:49   SUCCESS plugin 2: loaded from ../hosting/mediawiki.so }
    05:11:49 { Loading plugin 3: Type `plain', file `creolewiki' in `../hosting'
    05:11:49   CreoleWiki version 0.1 from OpenLink Software
    05:11:49   Support functions for CreoleWiki collaboration tool
    05:11:49   SUCCESS plugin 3: loaded from ../hosting/creolewiki.so }
    05:11:49 { Loading plugin 4: Type `plain', file `im' in `../hosting'
    05:11:49   IM version 0.61 from OpenLink Software
    05:11:49   Support functions for Image Magick 6.8.1
    05:11:49   SUCCESS plugin 4: loaded from ../hosting/im.so }
    05:11:49 { Loading plugin 5: Type `plain', file `wbxml2' in `../hosting'
    05:11:49   WBXML2 version 0.9 from OpenLink Software
    05:11:49   Support functions for WBXML2 0.9.2 Library
    05:11:49   SUCCESS plugin 5: loaded from ../hosting/wbxml2.so }
    05:11:49 { Loading plugin 6: Type `attach', file `libphp5.so' in `../hosting'
    05:11:49   SUCCESS plugin 6: loaded from ../hosting/libphp5.so }
    05:11:49 { Loading plugin 7: Type `Hosting', file `hosting_php.so' in `../hosting'
    05:11:49   Hosting version 3208 from OpenLink Software
    05:11:49   PHP engine version 5.3.21
    05:11:49   SUCCESS plugin 7: loaded from ../hosting/hosting_php.so }
    05:11:49 { Loading plugin 8: Type `plain', file `qrcode' in `../hosting'
    05:11:49   QRcode version 0.1 from OpenLink Software
    05:11:49   Support functions for ISO/IEC 18004:2006, using QR Code encoder (C) 2006 Kentaro Fukuchi <fukichi@megaui.net>
    05:11:49   SUCCESS plugin 8: loaded from ../hosting/qrcode.so }
    05:11:49 OpenLink Virtuoso Universal Server
    05:11:49 Version 07.10.3211-pthreads for Linux as of Oct  6 2014
    05:11:49 uses parts of OpenSSL, PCRE, Html Tidy
    05:11:51 SQL Optimizer enabled (max 1000 layouts)
    05:11:52 Compiler unit is timed at 0.000403 msec
    05:12:03 Checkpoint started
    05:12:04 Roll forward started
    05:12:04 Roll forward complete
    05:12:07 PL LOG: Elastic cluster setup
    05:12:08 Checkpoint started
    05:12:09 Checkpoint finished, log reused
    05:12:11 Checkpoint started
    05:12:12 Checkpoint finished, log reused
    05:12:12 PL LOG: new clustered database:Init of RDF
    05:12:23 Checkpoint started
    05:12:25 Checkpoint finished, log reused
    05:12:50 PL LOG: Installing Virtuoso Conductor version 1.00.8727 (DAV)
    05:12:51 Checkpoint started
    05:12:53 Checkpoint finished, log reused
    05:13:23 Checkpoint started
    05:13:25 Checkpoint finished, log reused
    05:13:26 Checkpoint started
    05:13:26 Checkpoint finished, log reused
    05:13:28 HTTP/WebDAV server online at 8890
    05:13:28 Server online at 12201 (pid 15211)
    05:13:29 ZeroConfig registration CLUSTER (MASALA)
    
  4. The cluster node directories can also be checked, where the database slice/shard files i.e. database.db.X, database.db.Y, database.db.Z can be seen:
    $ ls -ltr cluster_01
    total 2322804
    drwxr-xr-x. 2 virtuoso virtuoso      4096 Oct  9 04:26 backup
    drwxr-xr-x. 2 virtuoso virtuoso      4096 Oct  9 04:26 logs
    -rwxr-xr-x. 1 virtuoso virtuoso     70607 Oct  9 04:26 php.ini
    lrwxrwxrwx. 1 virtuoso virtuoso        24 Oct  9 04:26 virtuoso -> ..//bin/virtuoso-iodbc-t
    -rw-r--r--. 1 virtuoso virtuoso         0 Oct  9 04:26 database.pxa
    -rwxr-xr-x. 1 virtuoso virtuoso      6594 Oct  9 04:33 virtuoso.ini
    -rw-r--r--. 1 virtuoso virtuoso       137 Oct  9 05:03 cluster.ini
    lrwxrwxrwx. 1 virtuoso virtuoso        21 Oct  9 05:04 cluster.global.ini -> ../cluster.global.ini
    drwxr-xr-x. 2 virtuoso virtuoso    131072 Oct  9 05:37 dump
    -rw-r--r--. 1 virtuoso virtuoso     32915 Oct  9 06:55 database.2pc
    -rw-r--r--. 1 virtuoso virtuoso        25 Oct 10 02:24 database.map
    -rw-r--r--. 1 virtuoso virtuoso  56623104 Oct 10 03:23 database-temp.db
    -rw-r--r--. 1 virtuoso virtuoso 824180736 Oct 10 05:10 database.db.16
    -rw-r--r--. 1 virtuoso virtuoso 723517440 Oct 10 05:10 database.db.8
    -rw-r--r--. 1 virtuoso virtuoso 740294656 Oct 10 05:10 database.db.0
    -rw-r--r--. 1 virtuoso virtuoso  33554432 Oct 10 05:10 database.db
    -rw-r--r--. 1 virtuoso virtuoso         0 Oct 10 05:10 database.trx
    -rw-r--r--. 1 virtuoso virtuoso     34234 Oct 10 05:10 database.log
    $
    

2.14.1.8. Splitting Cluster nodes across different machines

Step 5: Splitting Cluster nodes across different machines

  1. To split the node across across physical machines for better scale out performance, scalability and growth, simply perform a parallel Virtuoso installation on the additional physical machines and move the cluster nodes required to the designated machine. Example, for the default 4 node cluster to be split across two identical machines it would make sense to split 2 nodes across each machine, thus you would move say the cluster_03 and cluster_04 directory nodes to the new machine (removing them from the original). The cluster.global.ini file on each node would then need to be updated to set the HostXX parameters to point to the new locations for nodes 03 and 04:
    $ more cluster.global.ini
    
    [Cluster]
    Threads             = 300
    ReqBatchSize        = 10000
    BatchesPerRPC       = 4
    BatchBufferBytes    = 20000
    LocalOnly           = 2
    MaxKeepAlivesMissed = 100
    RDFLoadBytes        = 52428800
    Host1               = hostname1:22201
    Host2               = hostname1:22202
    Host3               = hostname2:22203
    Host4               = hostname2:22204
    
    Machine 1 (hostname1)
    
    $ more cluster_01/cluster.ini
    [Cluster]
    Master              = Host1
    ThisHost            = Host1
    
    [ELASTIC]
    Segment1 = 4G, database.db = q1
    Slices = 12
    MaxSlices = 2048
    
    $ more cluster_02/cluster.ini
    
    [Cluster]
    Master              = Host1
    ThisHost            = Host2
    
    [ELASTIC]
    Segment1 = 4G, database.db = q1
    Slices = 12
    MaxSlices = 2048
    
    Machine 2 (hostname2)
    
    $ more cluster_03/cluster.ini
    
    [Cluster]
    Master              = Host1
    ThisHost            = Host3
    
    [ELASTIC]
    Segment1 = 4G, database.db = q1
    Slices = 12
    MaxSlices = 2048
    
    $ more cluster_04/cluster.ini
    
    [Cluster]
    Master              = Host1
    ThisHost            = Host4
    
    [ELASTIC]
    Segment1 = 4G, database.db = q1
    Slices = 12
    MaxSlices = 2048
    
  2. Note assuming the same number of CPU threads on each machine i.e. 24, then the Slices param can be doubled to 12 for each node in cluster.ini, as above.
  3. The cluster nodes can then be started on each machine, with 2 nodes being started on each in this case to form the cluster:
    Machine 1 (hostname1)
    
    $ virtuoso-start.sh
    Starting Virtuoso instance in [cluster_01]
    Starting Virtuoso instance in [cluster_02]
    $
    
    Machine 2 (hostname2)
    
    $ virtuoso-start.sh
    Starting Virtuoso instance in [cluster_03]
    Starting Virtuoso instance in [cluster_04]
    $
    
  4. To stop the cluster use the standard virtuoso-stop.sh script on each machine, which automatically detects the number for nodes to be stopped:
    Machine 1 (hostname1)
    
    $ virtuoso-stop.sh
    Stopping Virtuoso instance in [cluster_01]
    Stopping Virtuoso instance in [cluster_02]
    
    Machine 2 (hostname2)
    
    $ virtuoso-stop.sh
    Stopping Virtuoso instance in [cluster_03]
    Stopping Virtuoso instance in [cluster_04]
    $
    
  5. The cl_exec('shutdown') command can also be run from isql on any node of the cluster to shutdown all nodes at once:
    $ isql 12201
    Connected to OpenLink Virtuoso
    Driver: 07.10.3211 OpenLink Virtuoso ODBC Driver
    OpenLink Interactive SQL (Virtuoso), version 0.9849b.
    Type HELP; for help and EXIT; to exit.
    SQL> cl_exec ('checkpoint');
    
    Done. -- 2487 msec.
    SQL> cl_exec ('shutdown');
    
    *** Error 08S01: VD CL065: Lost connection to server
    at line 2 of Top-Level:
    cl_exec ('shutdown')
    $
    


2.14.2. Virtuoso default Cluster Installation and Configuration

These sections apply to Virtuoso as of version 6.x and higher.

  1. Run the Virtuoso 6 Unix ( MacOsX) installer to perform a default installation:
    $ tar xvf virtuoso-universal-server-6.1.tar
    x install.sh
    x universal-server.taz
    $ sh install.sh
    
    - Extracting Virtuoso Universal Server v6.1
    
    - Creating default environment settings
    
    - Creating default database settings
    Configuring: database
    Creating directory $VIRTUOSO_HOME/database/backup
    Installing new virtuoso.ini in $VIRTUOSO_HOME/database
    Installing new php.ini in $VIRTUOSO_HOME/database
    Creating symlink to $VIRTUOSO_HOME/bin/virtuoso-iodbc-t
    
    - Registering ODBC drivers
    
    - Registering .NET provider for Mono
    
    - Installing VAD packages in database (this can take some time)
    
    - Checking where license should be stored
    
    - Starting OpenLink License Manager
    
    - Checking for initial Virtuoso license
    
    - Starting Virtuoso server instance
    
    - Finalizing installation
    
    
    This concludes the first part of the installation.
    
    
    - Attempting to start browser
    
    Please start a browser manually and open the following URL to finalize
    the installation process:
    
         http://cname:8890/install/
    
    
    Installation completed
    $
    
  2. Determine how many nodes you want to start the cluster with: 2, 3, 4, 5, 6, 7, 8 etc. Note: the number of the nodes should be >= 2
  3. Setup Virtuoso environment and stop default database:
    $ . ./virtuoso-environment.sh
    $ virtuoso-stop.sh
    Shutting down Virtuoso instance in [database]
    $
    
  4. Optionally remove the default database such that is does not get started with the cluster:
    rm  -rf database
    
  5. Full list of supported options for the script "virtuoso-mkcluster.sh":
      -cluster-size       Number of nodes in cluster
      -cluster-node       Node number in a cluster
      -cluster-port       Base portnumer for cluster
      -cluster-ipc-port   Base portnumer for cluster IPC
      -cluster_size       Size of the cluster
      -virtuoso_home      Virtuoso home path
    
  6. Run the virtuoso-mkcluster.sh script to create the cluster, note the default number of nodes is 4, but this can be changed by setting the environment variable CLUSTER_SIZE to the required number of nodes to be created before running the script:
    $ virtuoso-mkcluster.sh
    Configuring cluster node: 1/4
    
    Creating directory $VIRTUOSO_HOME/cluster_01
    Creating directory $VIRTUOSO_HOME/cluster_01/backup
    Creating directory $VIRTUOSO_HOME/cluster_01/logs
    Installing new virtuoso.ini in $VIRTUOSO_HOME/cluster_01
    Installing new php.ini in $VIRTUOSO_HOME/cluster_01
    Creating symlink to $VIRTUOSO_HOME/bin/virtuoso-iodbc-t
    Configuring cluster node: 2/4
    
    Creating directory $VIRTUOSO_HOME/cluster_02
    Creating directory $VIRTUOSO_HOME/cluster_02/backup
    Installing new virtuoso.ini in $VIRTUOSO_HOME/cluster_02
    Removing unneeded sections from virtuoso.ini
    Creating symlink to $VIRTUOSO_HOME/bin/virtuoso-iodbc-t
    Configuring cluster node: 3/4
    
    Creating directory $VIRTUOSO_HOME/cluster_03
    Creating directory $VIRTUOSO_HOME/cluster_03/backup
    Installing new virtuoso.ini in $VIRTUOSO_HOME/cluster_03
    Removing unneeded sections from virtuoso.ini
    Creating symlink to $VIRTUOSO_HOME/bin/virtuoso-iodbc-t
    Configuring cluster node: 4/4
    
    Creating directory $VIRTUOSO_HOME/cluster_04
    Creating directory $VIRTUOSO_HOME/cluster_04/backup
    Installing new virtuoso.ini in $VIRTUOSO_HOME/cluster_04
    Removing unneeded sections from virtuoso.ini
    Creating symlink to $VIRTUOSO_HOME/bin/virtuoso-iodbc-t
    $
    
  7. For each of the cluster_XX directories created edit the cluster.ini file and set the HostXX parameter to the hostname:portno to be used by the cluster nodes for internal communication.
    $ more cluster_01/cluster.ini
    [Cluster]
    Threads             = 20
    Master              = Host1
    ThisHost            = Host1
    ReqBatchSize        = 10000
    BatchesPerRPC       = 4
    BatchBufferBytes    = 20000
    LocalOnly           = 2
    MaxKeepAlivesMissed = 1000
    Host1               = hostname:22201
    Host2               = hostname:22202
    Host3               = hostname:22203
    Host4               = hostname:22204
    MaxHosts            = 5
    $ more cluster_02/cluster.ini
    [Cluster]
    Threads             = 20
    Master              = Host1
    ThisHost            = Host2
    ReqBatchSize        = 10000
    BatchesPerRPC       = 4
    BatchBufferBytes    = 20000
    LocalOnly           = 2
    MaxKeepAlivesMissed = 1000
    Host1               = hostname:22201
    Host2               = hostname:22202
    Host3               = hostname:22203
    Host4               = hostname:22204
    MaxHosts            = 5
    $ more cluster_03/cluster.ini
    [Cluster]
    Threads             = 20
    Master              = Host1
    ThisHost            = Host3
    ReqBatchSize        = 10000
    BatchesPerRPC       = 4
    BatchBufferBytes    = 20000
    LocalOnly           = 2
    MaxKeepAlivesMissed = 1000
    Host1               = hostname:22201
    Host2               = hostname:22202
    Host3               = hostname:22203
    Host4               = hostname:22204
    MaxHosts            = 5
    $ more cluster_04/cluster.ini
    [Cluster]
    Threads             = 20
    Master              = Host1
    ThisHost            = Host4
    ReqBatchSize        = 10000
    BatchesPerRPC       = 4
    BatchBufferBytes    = 20000
    LocalOnly           = 2
    MaxKeepAlivesMissed = 1000
    Host1               = hostname:22201
    Host2               = hostname:22202
    Host3               = hostname:22203
    Host4               = hostname:22204
    MaxHosts            = 5
    $
    
  8. Start the cluster using the standard virtuoso-start.sh script which automatically detects the number for nodes to be started:
    $ virtuoso-start.sh
    Starting Virtuoso instance in [cluster_01]
    Starting Virtuoso instance in [cluster_02]
    Starting Virtuoso instance in [cluster_03]
    Starting Virtuoso instance in [cluster_04]
    $
    
  9. The default SQL port of the master node is 12201, as indicated in the virtuoso.ini file of the cluster_01 directory, and can then be used for connecting to the newly created cluster and check its status to ensure all nodes are online:
    $ isql 12201
    Connected to OpenLink Virtuoso
    Driver: 06.01.3127 OpenLink Virtuoso ODBC Driver
    OpenLink Interactive SQL (Virtuoso), version 0.9849b.
    Type HELP; for help and EXIT; to exit.
    SQL> status ('cluster');
    REPORT
    VARCHAR
    _______________________________________________________________________________
    
    Cluster 4 nodes, 4 s. 1 m/s 0 KB/s  0% cpu 0%  read 0% clw threads 1r 0w 0i buffers 2981 0 d 0 w 0 pfs
    
    1 Rows. -- 7 msec.
    SQL> cl_ping(1,500, 1000);
    
    Done. -- 7 msec.
    SQL> cl_ping(2,500, 1000);
    
    Done. -- 52 msec.
    SQL> cl_ping(3,500, 1000);
    
    Done. -- 60 msec.
    SQL> cl_ping(4,500, 1000);
    
    Done. -- 51 msec.
    SQL>
    
  10. To split the node across across physical machines for better performance and scalability, simply perform a parallel Virtuoso installation on the additional physical machines and move the cluster nodes required to the designated machine. Example, for the default 4 node cluster to be split across two identical machines it would make sense to split 2 nodes across each machine, thus you would move say the cluster_03 and cluster_04 directory nodes to the new machine (removing them from the original). The cluster.ini file on each node would then need to be updated to set the HostXX parameters to point to the new locations for nodes 03 and 04:
    Machine 1 (hostname1)
    
    $ more cluster_01/cluster.ini
    [Cluster]
    Threads             = 20
    Master              = Host1
    ThisHost            = Host1
    ReqBatchSize        = 10000
    BatchesPerRPC       = 4
    BatchBufferBytes    = 20000
    LocalOnly           = 2
    MaxKeepAlivesMissed = 1000
    Host1               = hostname1:22201
    Host2               = hostname1:22202
    Host3               = hostname2:22203
    Host4               = hostname2:22204
    MaxHosts            = 5
    $ more cluster_02/cluster.ini
    [Cluster]
    Threads             = 20
    Master              = Host1
    ThisHost            = Host2
    ReqBatchSize        = 10000
    BatchesPerRPC       = 4
    BatchBufferBytes    = 20000
    LocalOnly           = 2
    MaxKeepAlivesMissed = 1000
    Host1               = hostname1:22201
    Host2               = hostname1:22202
    Host3               = hostname2:22203
    Host4               = hostname2:22204
    MaxHosts            = 5
    $
    
    Machine 2 (hostname2)
    
    $ more cluster_03/cluster.ini
    [Cluster]
    Threads             = 20
    Master              = Host1
    ThisHost            = Host3
    ReqBatchSize        = 10000
    BatchesPerRPC       = 4
    BatchBufferBytes    = 20000
    LocalOnly           = 2
    MaxKeepAlivesMissed = 1000
    Host1               = hostname1:22201
    Host2               = hostname1:22202
    Host3               = hostname2:22203
    Host4               = hostname2:22204
    MaxHosts            = 5
    $ more cluster_04/cluster.ini
    [Cluster]
    Threads             = 20
    Master              = Host1
    ThisHost            = Host4
    ReqBatchSize        = 10000
    BatchesPerRPC       = 4
    BatchBufferBytes    = 20000
    LocalOnly           = 2
    MaxKeepAlivesMissed = 1000
    Host1               = hostname1:22201
    Host2               = hostname1:22202
    Host3               = hostname2:22203
    Host4               = hostname2:22204
    MaxHosts            = 5
    $
    
  11. The nodes can then be started on each machine, with 2 nodes being started on each in this case to form the cluster:
    
    Machine 1 (hostname1)
    
    $ virtuoso-start.sh
    Starting Virtuoso instance in [cluster_01]
    Starting Virtuoso instance in [cluster_02]
    $
    
    Machine 2 (hostname2)
    
    $ virtuoso-start.sh
    Starting Virtuoso instance in [cluster_03]
    Starting Virtuoso instance in [cluster_04]
    $
    
  12. To stop the cluster use the standard virtuoso-stop.sh script which automatically detects the number for nodes to be stopped:
    $ virtuoso-stop.sh
    Stopping Virtuoso instance in [cluster_01]
    Stopping Virtuoso instance in [cluster_02]
    Stopping Virtuoso instance in [cluster_03]
    Stopping Virtuoso instance in [cluster_04]
    $
    

2.14.3. Backup and Restore

Backup

There are 2 ways to backup Virtuoso Cluster DB:

  1. Backup every node using back_online

    For example, for every cluster in its backup folder execute:

    SQL> backup_online('dbp',10000000,0,vector('backup'));
    
    Done. -- 272 msec.
    
  2. Make backup of all nodes at once using cl_exec. For example, execute:
    SQL> cl_exec ('backup_online (''dbp'', 10000000, 0,  vector (''backup''))');
    
    Done. -- 573 msec.
    

Restore from Backup

After backup is done for every node in its backup folder, in order to restore, execute from the ..\bin folder the virtuoso-restore.sh:

# . ./virtuoso-restore.sh all dbp

where the second parameter is the restore prefix.


2.14.4. Cluster Configuration

  1. Set "FAST_START=1" in the viruoso-start.sh file and then run:
    viruoso-start.sh
    

    or

  2. Execute the following line:
    # /etc/init.d/virtuoso start
    Starting OpenLink Virtuoso:                                [  OK  ]
    
  3. In order to check the nodes, connect to port 12201 using the ISQL tool:
    isql 12201
    
  4. To check the cluster status, execute the following command:
    SQL> status('cluster');
    REPORT
    VARCHAR
    _______________________________________________________________________________
    
    Cluster 4 nodes, 293 s. 0 m/s 0 KB/s  0% cpu 0%  read 0% clw threads 1r 0w 0i buffers 1781 0 d 0 w 0 pfs
    
    1 Rows. -- 4 msec.
    

2.14.5. HTTP Service Configuration on Subordinate Nodes of a Virtuoso Cluster

This section applies to Virtuoso as of version 6.x and higher.

2.14.5.1. What

This documentation details how to configure the Subordinate (also called Slave) Nodes of a Virtuoso Elastic Cluster to service HTTP clients.


2.14.5.2. Why

By default, only the Primary (also called Master) instance of a Virtuoso Elastic Cluster is configured to provide HTTP services.

The Subordinate (also called Slave) nodes of the cluster may also be configured to provide HTTP services, enabling load balancing by spreading HTTP requests across the cluster's nodes.


2.14.5.3. How

This documentation details the steps for the installation and configuration of a Virtuoso Elastic Cluster on Unix:

  1. Step 1: Set up each instance as a HTTP Server;
  2. Step 2: Install and configure HTTP services on each instance;
  3. Step 3: Configure load balancing.

2.14.5.4. Set up each instance as an HTTP Server

Step 1: Set up each instance as a HTTP Server

Each node can be configured to provide HTTP services as follows:

  1. Copy the [HTTP Server] section from the Primary instance's configuration file (by default, virtuoso.ini) to the configuration file of each Subordinate instance:
    [HTTPServer]
    ServerPort                  = 8890
    ServerRoot                  = ../vsp
    DavRoot                     = DAV
    EnabledDavVSP               = 0
    HTTPProxyEnabled            = 0
    TempASPXDir                 = 0
    DefaultMailServer           = localhost:25
    MaxClientConnections        = 5
    MaxKeepAlives               = 10
    KeepAliveTimeout            = 10
    MaxCachedProxyConnections   = 10
    ProxyConnectionCacheTimeout = 15
    HTTPThreadSize              = 280000
    HttpPrintWarningsInOutput   = 0
    Charset                     = UTF-8
    ;HTTPLogFile                 = logs/http.log
    MaintenancePage             = atomic.html
    EnabledGzipContent          = 1
    
  2. Edit the ServerPort parameter to make it unique on the machine hosting this instance; i.e., if a subordinate instance is running on same physical node as the primary instance, then the subordinate's HTTP port must to be changed (from 8890, for instance) to a unique port (e.g., 8891).
  3. Install the Virtuoso Conductor to enable HTTP Administration of the instance being configured. Note: if the subordinate instance is not on the same machine as the primary instance, then the vad directory may also need to be copied from the primary instance to the subordinate instance.:
    SQL> vad_install ('../vad/conductor_dav.vad', 0);
    SQL_STATE  SQL_MESSAGE
    VARCHAR  VARCHAR
    _______________________________________________________________________________
    
    00000    No errors detected
    00000    Installation of "Virtuoso Conductor" is complete.
    00000    Now making a final checkpoint.
    00000    Final checkpoint is made.
    00000    SUCCESS
    
    
    6 Rows. -- 10263 msec.
    SQL>
    

2.14.5.5. Install and configure HTTP services on each instance

Step 2: Install and configure HTTP services on each instance

Any HTTP services required on the subordinate instance will need to specifically installed or configured on that physical node. For example, the Virtuoso default SPARQL endpoint (/sparql) may be configured by:

  1. Log in into the Virtuoso Conductor http://hostname:port/conductor :
    Configure SPARQL Endpoint: log in

    Figure: 2.14.5.5.1.1. Configure SPARQL Endpoint: log in
  2. Go the the Web Application Server -> Virtual Domains & Directories tab:
    Configure SPARQL Endpoint: Virtual Domains and Directories

    Figure: 2.14.5.5.1.2. Configure SPARQL Endpoint: Virtual Domains and Directories
  3. Select the New Directory Action for the Default Web Site HTTP host:
    Configure SPARQL Endpoint: new directory

    Figure: 2.14.5.5.1.3. Configure SPARQL Endpoint: new directory
  4. Select the Type radio button and SPARQL access point item from the drop down list box:
    Configure SPARQL Endpoint: set type SPARQL

    Figure: 2.14.5.5.1.4. Configure SPARQL Endpoint: set type SPARQL
  5. Click "Next".
  6. Enter /sparql as the Path param in the Virtual Directory Information section and click Save Changes:
    Configure SPARQL Endpoint: set /sparql virtual directory

    Figure: 2.14.5.5.1.5. Configure SPARQL Endpoint: set /sparql virtual directory
  7. The SPARQL endpoint will not be accessible on http://hostname:port/sparql the the newly configured slave nodes:
    Configure SPARQL Endpoint: SPARQL Endpoint

    Figure: 2.14.5.5.1.6. Configure SPARQL Endpoint: SPARQL Endpoint
  8. Further details on SPARQL endpoint configuration can be found in Service Endpoint documentation section.
  9. Typical Virtuoso server log output from a slave node when started, showing the HTTP server running on port 8890, being:
    20:12:49 OpenLink Virtuoso Universal Server
    20:12:49 Version 07.10.3209-pthreads for Linux as of Apr 26 2014
    20:12:49 uses parts of OpenSSL, PCRE, Html Tidy
    20:12:49 Registered to OpenLink Virtuoso (Internal Use)
    20:12:49 Personal Edition license for 500 connections
    20:12:49 Issued by OpenLink Software
    20:12:49 This license will expire on Sun May 17 06:18:35 2015 GMT
    20:12:49 Enabled Cluster Extension
    20:12:49 Enabled Column Store Extension
    20:12:57 Database version 3126
    20:12:57 SQL Optimizer enabled (max 1000 layouts)
    20:12:58 Compiler unit is timed at 0.000208 msec
    20:12:58 Roll forward started
    20:12:58 Roll forward complete
    20:12:59 Checkpoint started
    20:12:59 Checkpoint finished, log reused
    20:12:59 HTTP/WebDAV server online at 8890
    20:12:59 Server online at 12202 (pid 15969)
    

2.14.5.6. Configure load balancing

Step 3: Configure load balancing

A reverse-proxy service (like Nginx or Apache) can then be configured such that requests are proxied across as any or all nodes of the cluster, to provide the desired load balancing.


2.14.5.7. Additional Information



2.14.6. Troubleshooting Tips

If an operation seems to hang, see the output of:

status ()

Check for the presence of the following conditions:


See Also:

Setting up and operating Virtuoso on a cluster.

Virtuoso Cluster Programming

Virtuoso Cluster Fault Tolerance.