TakTuk

From Grid5000
Jump to navigation Jump to search

During the course of an experiment on G5k, one often wants to run programs or scripts on all the reserved nodes simultaneously. For instance, to send data to remote workers. This could be achieved sequentially with a loop construct in a script, but this method does not scale well. That's why a launcher for parallel commands named TakTuk has been installed on each G5k site.

What is TakTuk

TakTuk is a parallele launcher.

To speed up the remote execution of the same command on a large number of nodes, TakTuk both

  • uses a deployment tree to initiate several connections to remote peers at the same time
  • initiates locally a few connections in parallel (the deployment window)

The balance beetween the two mechanism is ensured by an adaptive work-stealing algorithm combined with an on-line evaluation of the local machine load (see A guide to TakTuk use by Guillaume Huard for more details).

Taktuk schema.png

Therefore, for taktuk to operate correctly, peers should be able to connect to each other. By default, TakTuk uses ssh connections. So, to check if Taktuk execution can work, please follow instructions on this page: Test key authentication. There is also a practical example for setting up taktuk in the Advanced OAR tutorial.

External Links

Using TakTuk in Grid'5000

Using TakTuk with Grid'5000's default environment

Prerequisites

You own a reservation for some machines:

Terminal.png users@frontale.site.grid5000.fr:
oarsub -I -l nodes=4

You are logged on one of the reserved nodes:

Terminal.png users@node-15.site.grid5000.fr:

Connections can be made without password from this node and between nodes using your account.

Using Taktuk to launch scripts

To execute scripts simultaneously on all the nodes:

Terminal.png users@node-15.site.grid5000.fr:
taktuk -c "oarsh" -f $OAR_FILE_NODES broadcast exec [ /home/site/user/testScript.sh ]

Note: do not forget to type the characters [ and ] and separate them from their content (using spaces for instance).

Note: taktuk uses classic ssh by default. It's possible to do without the -c "oarsh" but in this case you'll have to type oarsub -I -l nodes=4 -t allow_classic_ssh (see : https://www.grid5000.fr/mediawiki/index.php/Getting_Started#Reserving_resources_with_OAR:_the_basics )

By default, TakTuk displays a lot of informations and part of it might not be necessary. Basically, you can choose exactly what is displayed by TakTuk (type of information and format of the displayed messages) using the output-template option. For instance, the following options suppress the connector and status streams and format the output of commands to use the host name followed by the content of the output:

-o connector -o status -o output='"$host: $line\n"'
Example

To check if time clocks are synchronized on all the nodes (Result formated correctly)

Terminal.png users@node-15.site.grid5000.fr:
taktuk -c "oarsh" -o connector -o status -o output='"$host: $line\n"' -f <( uniq $OAR_FILE_NODES ) broadcast exec [ date ]
Output
node-101.bordeaux.grid5000.fr: Fri Mar 23 09:38:46 CET 2007
node-23.bordeaux.grid5000.fr: Fri Mar 23 09:38:46 CET 2007
node-24.bordeaux.grid5000.fr: Fri Mar 23 09:38:46 CET 2007
node-67.bordeaux.grid5000.fr: Fri Mar 23 09:38:46 CET 2007
node-97.bordeaux.grid5000.fr: Fri Mar 23 09:38:46 CET 2007
To find CPUs installed on nodes
Terminal.png user@node-15.site.grid5000.fr:
taktuk -c "oarsh" -o connector -o status -o output='"$host: $line\n"' -f <( uniq $OAR_FILE_NODES ) broadcast exec [ "cat /proc/cpuinfo | grep 'model name'" ] | uniq
Output
node-101.bordeaux.grid5000.fr: model name       :                   Intel(R) Xeon(TM) CPU 3.00GHz
node-23.bordeaux.grid5000.fr: model name        : AMD Opteron(tm) Processor 248
node-24.bordeaux.grid5000.fr: model name        : AMD Opteron(tm) Processor 248
node-67.bordeaux.grid5000.fr: model name        :                   Intel(R) Xeon(TM) CPU 3.00GHz
node-97.bordeaux.grid5000.fr: model name        :                   Intel(R) Xeon(TM) CPU 3.00GHz

Note: instead of using the option -c "oarsh" in each command, we could type the following command :

Terminal.png users@node-15.site.grid5000.fr:
export TAKTUK_CONNECTOR=oarsh

and then taktuk would know that it needs to use oarsh as a connector.

Using TakTuk with your own environment but without TakTuk installation

Prerequisites

Start by making a reservation:

Terminal.png users@frontale.site.grid5000.fr:
oarsub -I -l nodes=4 -t deploy

Deploy your environment:

Terminal.png users@frontale.site.grid5000.fr:
kadeploy3 -f $OAR_FILE_NODES -e debian9-x64-base -k

Using Taktuk to launch programs

Using the -s option, you can use taktuk to deploy on remote hosts that don't have taktuk installed.

Terminal.png users@frontale.site.grid5000.fr:
taktuk -s -o connector -o status -o output='"$host: $line\n"' -f <( uniq $OAR_FILE_NODES ) broadcast exec [ "apt-get update" ]

To execute an apt-get as root for system update, you have to use the following option: "-l root" before any machine specification.

Terminal.png users@frontale.site.grid5000.fr:
taktuk -l root -s -o connector -o status -o output='"$host: $line\n"' -f <( uniq $OAR_FILE_NODES ) broadcast exec [ "apt-get update" ]

Some extra examples

  • To execute a simple command like date on 1 node:
taktuk -m node-5.site.grid5000.fr broadcast exec [ date ]
  • Same execution but on more than 1 node:
taktuk -m node-5.site.grid5000.fr -m node-10.site.grid5000.fr broadcast exec [ date ]
  • Execute command on nodes given by OAR after reservation
taktuk -f <( uniq $OAR_FILE_NODES ) broadcast exec [ date ]
  • Self propagation of TakTuk if it is not available everywhere (works only if you can login from nodes to nodes without typing your password):
taktuk -s -f <( uniq $OAR_FILE_NODES ) broadcast exec [ date ]
  • Connect to nodes with a different login, root for instance:
taktuk -l root -f <( uniq $OAR_FILE_NODES ) broadcast exec [ date ]
  • In order to test latencies between nodes using taktuk:

As a user from the frontend, run:

taktuk -c "ssh -p 6667" -s -o 'state=(($line == 4)?($init_date - $start_date)." s de connection de $host a $peer (dont ".($init_date-$reply_date)." du a TakTuk)\n":"")' -f allnodes.txt b e [ true ] | tee taktuk.log

where taktuk version is at least 3.3 and allnodes.txt contains the list of the nodes.

Then doing a grep connection taktuk.log | sort -n gives:

0.220579862594604 s de connection de azur-25.sophia.grid5000.fr a sol-12.sophia.grid5000.fr (dont 0.047921895980835 du a TakTuk)
0.220623016357422 s de connection de azur-17.sophia.grid5000.fr a sol-36.sophia.grid5000.fr (dont 0.0476739406585693 du a TakTuk)
0.222663879394531 s de connection de azur-1.sophia.grid5000.fr a sol-24.sophia.grid5000.fr (dont 0.0479059219360352 du a TakTuk)
0.225576162338257 s de connection de azur-30.sophia.grid5000.fr a sol-27.sophia.grid5000.fr (dont 0.0478250980377197 du a TakTuk)
0.226027965545654 s de connection de azur-16.sophia.grid5000.fr a sol-31.sophia.grid5000.fr (dont 0.0470089912414551 du a TakTuk)
...