Fed4FIRE: Difference between revisions

From Grid5000
Jump to navigation Jump to search
(18 intermediate revisions by 2 users not shown)
Line 2: Line 2:
This page provides specific information about Grid'5000 for Fed4FIRE users.
This page provides specific information about Grid'5000 for Fed4FIRE users.


== Current Status (March 2020) ==
== Current Status (April 2020) ==
The Grid'5000 Aggregate Manager (AM) will soon be added to Fed4FIRE's jFed suite. Although integration is not complete yet this will allow users to perform basic tasks using only  Fed4FIRE-standard APIs.
The Grid'5000 Aggregate Manager (AM) will soon be added to Fed4FIRE's jFed suite. Although integration is not complete yet this will allow users to perform basic tasks using only  Fed4FIRE-standard APIs.


* The Grid'5000 Aggregate Manager (am.grid5000.fr) advertises Grid'5000 resources
* The Grid'5000 Aggregate Manager (am.grid5000.fr) advertises Grid'5000 resources
* Fed4FIRE users can allocate and provision Grid'5000 resources from the Aggregate Manager. Using the AMv3 API.
* Fed4FIRE users can allocate and provision Grid'5000 resources from the Aggregate Manager, using the AMv3 API.
* New Grid'5000 accounts are automatically created by the AM for new users for 1 month. After which Fed4FIRE users will need to contact Grid'5000 support staff to validate and extend their account.
* New Grid'5000 accounts are automatically created by the AM for new users for 1 month. After which Fed4FIRE users will need to complete their account to regain access (see [[Fed4FIRE#Extending_a_valid_or_expired_account|Extending a valid or expired account]]).
* Fed4FIRE users can login to Grid'5000 frontends and to provisioned resources via [[SSH]] using their Fed4FIRE certificate private key.
* Fed4FIRE users can login to Grid'5000 frontends and to provisioned resources via [[SSH]] using their Fed4FIRE certificate private key.
* Network-level function, such as internal and external vlans, are not yet available through the AM and require connected to Grid'5000 tools. See [[KaVLAN]]
* Network-level functions, such as internal and external vlans, are not yet available through the AM and require using the Grid'5000 tools. See [[KaVLAN]]
* Network-level interconnection using dedicated links with other Fed4FIRE testbed is functional. See [[Fed4FIRE_VLAN_Stitching]]. Network interconnection over the public Internet is functional (Grid'5000 nodes can access the public internet).
* Network-level interconnection using dedicated links with other Fed4FIRE testbed is functional. See [[Fed4FIRE_VLAN_Stitching]]. Network interconnection over the public Internet is functional (Grid'5000 nodes can access the public internet).
== Technical detail ==
* Aggregate Manager
** Address: am.grid5000.fr
** Port: 443
** Component managers:
***    <nowiki>urn:publicid:IDN+am.grid5000.fr+authority+am</nowiki>
***    <nowiki>urn:publicid:IDN+am.grid5000.fr:<site>+authority+am</nowiki>


== Grid'5000 Accounts ==
== Grid'5000 Accounts ==


Access to any Grid'5000 resources requires a Grid'5000 account.  
Access to any Grid'5000 resources requires a Grid'5000 account.  
Grid'5000 users who already have an account can link it to their Fed4Fire identity from their [https://api.grid5000.fr/ui/account account management page]:   
 
=== Linking Fed4FIRE identity to existing Grid'5000 accounts ===
Grid'5000 users who already have an account can link it to their Fed4FIRE identity from their [https://api.grid5000.fr/ui/account account management page]:   
* go to the '''External identifiers''' and press the '''Add new identifier''' button,
* go to the '''External identifiers''' and press the '''Add new identifier''' button,
* select '''Fed4FIRE''' as External engine and your Fed4FIRE URN as External identifier.
* select '''Fed4FIRE''' as External engine and your Fed4FIRE URN as External identifier.
The Fed4FIRE URN can be found in jFed tools once logged in, or by parsing the Fed4FIRE user certificate using openssl.
The Fed4FIRE URN can be found in jFed tools once logged in, or by parsing the Fed4FIRE user certificate using openssl.
Please not that for preexisting Grid'5000 account the AM will not add new ssh keys to your account. 
Users should feel free to [[Fed4FIRE#Adding_your_certificate_key_as_SSH_keys|add their certificate key as an ssh key to their Grid'5000 account]] or [[Fed4FIRE#Adding_your_ssh_key_to_jFed| add their ssh key to jFed experimenter]].


=== Fed4FIRE Users ===
=== Fed4FIRE Users ===
Fed4FIRE users without a existing Grid'5000, or that fail to link their existing Grid'5000 account, will have a new one created for them the first time they allocate resources. These new accounts are be valid for a single month.
Fed4FIRE users without a existing Grid'5000, or that fail to link their existing Grid'5000 account, will have a new one created for them the first time they allocate resources. These new accounts are be valid for a single month.
Three emails will inform you of you account's expiry and retierment:
Three emails will inform you of you account's expiry:
* one week before the account's expiry
* one week before the account's expiry
* on the day of the account's expiry
* on the day of the account's expiry
* on the day of the account's closing, one week after account retirement.
* on the day of the account's is retired, 1 week after expiry
 
=== Extending a valid or expired account Grid'5000 account, created automatically for a Fed4FIRE user ===
Users are welcomed to request an account extension. To do so:


=== Extending a valid or expired account ===
# Go to [[Special:G5KChangePassword|Grid'5000 password reset page]] to create a password for your account.  
Users are welcome to request an account extension. To do so will need to [[Special:G5KChangePassword|create a password for your Grid'5000 account]]. Please not that (re)setting your password requires you to input the email associated with your account, which will be the one provided by the Fed4FIRE federation and not your institutional email.
#: Please note that (re)setting your password requires you to input the email associated with your account, which will be the one provided by the Fed4FIRE federation and not your institutional email.
# Login to the [https://api.grid5000.fr/ui/account account management page].
# Complete your information :
#: Use the '''Action''' buttons to '''Edit''' your Account and Affiliation
#:- In the account section you will be asked to provide your name and, if you so wish, your institutional email address.
#:- In the affiliation section you will be asked about your work and employer, as well as your intended usage for Grid'5000
# Request access by going to the '''Groups''' tab and using the ''Join a new group'' button.
#: Join the open-access group.
#: Your request will be checked by the group manager based on your account and affiliation information, so fill them as correctly.
#: INRIA members can try to join a group relevant to their research teams instead of open-access.
#: Do NOT request an extension within the Fed4FIRE access group. It will not be granted.


Once you have set a password for you can access you [https://api.grid5000.fr/ui/account account management page]. From there users should be able to request an account prolongation from the actions menu on the account page.
=== Extending a closed account, created automatically for a Fed4FIRE user ===
Account that have been expired for more than one week are retired automatically.


=== Extending a closed account ===
Retired accounts can not be accessed from the [https://api.grid5000.fr/ui/account account management interface] and need to be reopened by Grid'5000 staff.  
One week after a automatically created account is expired it will become expired. Expired accounts can only be reopened by Grid'5000 staff, and users willing to do so should send an email to support-staff@lists.grid5000.fr. This email should provide the following information:
* your Grid'5000 account name
* your fed4fire email (to which Grid'5000 sent all previous emails) and your institutional email.
* your institutionnal affiliation:
** employer/reasearch institution
** department/laboratory
** team
* a paragraph with your research topic
* a paragraph or 2 (100 words) with your intended usage for Grid'5000
* an expiration date for your account
* acceptance of Grid'5000's Usage Policy


We will attempt to find your name on the web site of your team, laboratory or research institution to check the data sent. You can help us by sending the url of the relevant page.
To reopen a closed account you will need to mail the support staff at ''support-staff @ lists.grid5000.fr''.


== Contact information ==
== Contact information ==
Line 66: Line 81:
By default accounts created though through tools such as jFed, the ssh key of the account is derived from the user's federation certificate. To connect users must provide the certificate to the ssh client as the identity file.
By default accounts created though through tools such as jFed, the ssh key of the account is derived from the user's federation certificate. To connect users must provide the certificate to the ssh client as the identity file.


Users can add additional keys using the Grid'5000 [[Fed4FIRE#Grid.275000_Account_Management|account management interface]]. These keys will be used to connect to access gateways and nodes.  
Users can add additional keys using the Grid'5000 [[Fed4FIRE#Grid.275000_Account_Management|account management interface]]. These keys will be used to connect to access gateways and nodes.
 
=== Adding your certificate key as SSH keys ===
 
By default jFed tools will try to connect to nodes using your user certificate key. For this reason the Grid'5000 AM will add this ssh key to all new account it creates, and will update your key every-time your certificate changes.
Users who use a pre-existing Grid'5000 account do not benefit from this feature by default. And can instead opt to add their usual ssh-key to jFed.
 
Users wanting to use their certficate key with ssh can use <code>ssh-keygen -y -f <path/to/certificate></code> to derive an ssh public key from their PEM certificate.
Users wanting to benefit from the AM's automatic key update feature should append the <code>encoded by users-api-ror from rsa cert</code> comment that the end of their key line.
The final result should look like : <code>ssh-rsa AAAAB3Nyc2EADAQzaC1ABAAAA[...]+sw== encoded by users-api-ror from rsa cert</code>
 
=== Adding your ssh key to jFed ===
By default jFed tools will try to connect to nodes using your user certificate key. Users can if they so wish add an other ssh key to try during ssh connections.
Ssh keys are added in <code>Preference > SSH Authentication</code> section.
 
However at the time of writing jFed Experimenter does not recognize the latest openssh prive key format, starting with <code>-----BEGIN OPENSSH PRIVATE KEY-----</code>.
If you have such a key you can work around the problem by:
# making a copy of your private key
# using <code>ssh-keygen -p -m PEM -f </path/to/key/copy></code>
#* the command will prompt you for a new password, you are free to reuse the same password or leave the field blank for no password
# the copy should not have the key in the PEM format, starting with <code>-----BEGIN RSA PRIVATE KEY-----</code>
The copy will now work with jFed.


=== Grid'5000 Account Management ===
=== Grid'5000 Account Management ===
Line 83: Line 119:
Even if several persons are going to collaborate on the same experiment, we strongly prefer that each person uses its own account, for traceability purposes. It is possible to share scripts etc using standard Unix mechanisms (directory permissions), or using an external Git service (which are accessible from Grid'5000 nodes).
Even if several persons are going to collaborate on the same experiment, we strongly prefer that each person uses its own account, for traceability purposes. It is possible to share scripts etc using standard Unix mechanisms (directory permissions), or using an external Git service (which are accessible from Grid'5000 nodes).


=== Fed4FIRE expriment sharing and SSH key injection. ===
=== Fed4FIRE experiment sharing and SSH key injection ===
It is possible to share experiments using the corresponding options in jFed. It is also possible to add SSH keys to nodes at provisioning time using the ̀ geni_users` option. However the Grid'5000 has multiple caveats:
It is possible to share experiments using the corresponding options in jFed. It is also possible to add SSH keys to nodes at provisioning time using the ̀ geni_users` option. However the Grid'5000 has multiple caveats:
* The new ssh keys are only installed on the provisioned node and not on the ssh access gateways. Only keys registered with your account before can be used on the access gateway. Keys registered with your account are always loaded into provisioned nodes.
* The new ssh keys are only installed on the provisioned node and not on the ssh access gateways. Only keys registered with your account before can be used on the access gateway. Keys registered with your account are always loaded into provisioned nodes.

Revision as of 13:06, 23 June 2020

This page provides specific information about Grid'5000 for Fed4FIRE users.

Current Status (April 2020)

The Grid'5000 Aggregate Manager (AM) will soon be added to Fed4FIRE's jFed suite. Although integration is not complete yet this will allow users to perform basic tasks using only Fed4FIRE-standard APIs.

  • The Grid'5000 Aggregate Manager (am.grid5000.fr) advertises Grid'5000 resources
  • Fed4FIRE users can allocate and provision Grid'5000 resources from the Aggregate Manager, using the AMv3 API.
  • New Grid'5000 accounts are automatically created by the AM for new users for 1 month. After which Fed4FIRE users will need to complete their account to regain access (see Extending a valid or expired account).
  • Fed4FIRE users can login to Grid'5000 frontends and to provisioned resources via SSH using their Fed4FIRE certificate private key.
  • Network-level functions, such as internal and external vlans, are not yet available through the AM and require using the Grid'5000 tools. See KaVLAN
  • Network-level interconnection using dedicated links with other Fed4FIRE testbed is functional. See Fed4FIRE_VLAN_Stitching. Network interconnection over the public Internet is functional (Grid'5000 nodes can access the public internet).

Technical detail

  • Aggregate Manager
    • Address: am.grid5000.fr
    • Port: 443
    • Component managers:
      • urn:publicid:IDN+am.grid5000.fr+authority+am
      • urn:publicid:IDN+am.grid5000.fr:<site>+authority+am

Grid'5000 Accounts

Access to any Grid'5000 resources requires a Grid'5000 account.

Linking Fed4FIRE identity to existing Grid'5000 accounts

Grid'5000 users who already have an account can link it to their Fed4FIRE identity from their account management page:

  • go to the External identifiers and press the Add new identifier button,
  • select Fed4FIRE as External engine and your Fed4FIRE URN as External identifier.

The Fed4FIRE URN can be found in jFed tools once logged in, or by parsing the Fed4FIRE user certificate using openssl. Please not that for preexisting Grid'5000 account the AM will not add new ssh keys to your account. Users should feel free to add their certificate key as an ssh key to their Grid'5000 account or add their ssh key to jFed experimenter.

Fed4FIRE Users

Fed4FIRE users without a existing Grid'5000, or that fail to link their existing Grid'5000 account, will have a new one created for them the first time they allocate resources. These new accounts are be valid for a single month. Three emails will inform you of you account's expiry:

  • one week before the account's expiry
  • on the day of the account's expiry
  • on the day of the account's is retired, 1 week after expiry

Extending a valid or expired account Grid'5000 account, created automatically for a Fed4FIRE user

Users are welcomed to request an account extension. To do so:

  1. Go to Grid'5000 password reset page to create a password for your account.
    Please note that (re)setting your password requires you to input the email associated with your account, which will be the one provided by the Fed4FIRE federation and not your institutional email.
  2. Login to the account management page.
  3. Complete your information :
    Use the Action buttons to Edit your Account and Affiliation
    - In the account section you will be asked to provide your name and, if you so wish, your institutional email address.
    - In the affiliation section you will be asked about your work and employer, as well as your intended usage for Grid'5000
  4. Request access by going to the Groups tab and using the Join a new group button.
    Join the open-access group.
    Your request will be checked by the group manager based on your account and affiliation information, so fill them as correctly.
    INRIA members can try to join a group relevant to their research teams instead of open-access.
    Do NOT request an extension within the Fed4FIRE access group. It will not be granted.

Extending a closed account, created automatically for a Fed4FIRE user

Account that have been expired for more than one week are retired automatically.

Retired accounts can not be accessed from the account management interface and need to be reopened by Grid'5000 staff.

To reopen a closed account you will need to mail the support staff at support-staff @ lists.grid5000.fr.

Contact information

  • Fed4FIRE contact points for Grid'5000:
    • Lucas Nussbaum (lucas.nussbaum@loria.fr)
    • David Margery (david.margery@inria.fr)
    • Luke Bertot (luke.bertot@inria.fr)
  • Grid'5000 support staff: see the Support page

FAQ

Limits for the duration of an experiment?

If experiment means project, there is no limit. Accounts are created with a short-term expiration date (one month or two months depending on the process used for account creation) but can be extended at will.

If experiment means resources reservation, the limits are described in the Grid'5000 Usage Policy. The philosophy behind the Usage Policy is that users should be able to find some resources to prepare experiments during the day, and then reserve resources in advance to do large-scale experiments during nights and week-ends. So the effective limits are 10 hours during the day (9h-19h), 14 hours during nights (19h-9h), and 62 hours during week-ends (Friday 19h -> Monday 9h). Users are therefore strongly encouraged to automate the setup of their experiments (using scripts or tools such as Ansible). If an experiment requires a longer reservation, a special request can be made, as described in the Grid'5000 Usage Policy.

Accessing your Grid'5000 homedirs

Grid'5000 provides home directories on every site of the testbed with ssh access. This access requires connecting through ssh gateways as described on this page.

SSH keys

By default accounts created though through tools such as jFed, the ssh key of the account is derived from the user's federation certificate. To connect users must provide the certificate to the ssh client as the identity file.

Users can add additional keys using the Grid'5000 account management interface. These keys will be used to connect to access gateways and nodes.

Adding your certificate key as SSH keys

By default jFed tools will try to connect to nodes using your user certificate key. For this reason the Grid'5000 AM will add this ssh key to all new account it creates, and will update your key every-time your certificate changes. Users who use a pre-existing Grid'5000 account do not benefit from this feature by default. And can instead opt to add their usual ssh-key to jFed.

Users wanting to use their certficate key with ssh can use ssh-keygen -y -f <path/to/certificate> to derive an ssh public key from their PEM certificate. Users wanting to benefit from the AM's automatic key update feature should append the encoded by users-api-ror from rsa cert comment that the end of their key line. The final result should look like : ssh-rsa AAAAB3Nyc2EADAQzaC1ABAAAA[...]+sw== encoded by users-api-ror from rsa cert

Adding your ssh key to jFed

By default jFed tools will try to connect to nodes using your user certificate key. Users can if they so wish add an other ssh key to try during ssh connections. Ssh keys are added in Preference > SSH Authentication section.

However at the time of writing jFed Experimenter does not recognize the latest openssh prive key format, starting with -----BEGIN OPENSSH PRIVATE KEY-----. If you have such a key you can work around the problem by:

  1. making a copy of your private key
  2. using ssh-keygen -p -m PEM -f </path/to/key/copy>
    • the command will prompt you for a new password, you are free to reuse the same password or leave the field blank for no password
  3. the copy should not have the key in the PEM format, starting with -----BEGIN RSA PRIVATE KEY-----

The copy will now work with jFed.

Grid'5000 Account Management

Grid'5000 keeps user accounts linked to your Fed4FIRE identity. These account will be automatically generated when you first attempt a node allocation for a duration of 1 month. To access Grid'5000's account management interface you will first need to set a Grid'5000 password.

  • Resetting your password:
  • Accessing your account
    • Go to UMS
    • From here you can:
      • Add new ssh keys to your account.
      • Update your affiliation information
      • Request account extentions

Sharing one user account per experiment?

Even if several persons are going to collaborate on the same experiment, we strongly prefer that each person uses its own account, for traceability purposes. It is possible to share scripts etc using standard Unix mechanisms (directory permissions), or using an external Git service (which are accessible from Grid'5000 nodes).

Fed4FIRE experiment sharing and SSH key injection

It is possible to share experiments using the corresponding options in jFed. It is also possible to add SSH keys to nodes at provisioning time using the ̀ geni_users` option. However the Grid'5000 has multiple caveats:

  • The new ssh keys are only installed on the provisioned node and not on the ssh access gateways. Only keys registered with your account before can be used on the access gateway. Keys registered with your account are always loaded into provisioned nodes.
  • To grant access to other users they will need a Grid'5000 account to connect to the access gateway. Like with all other Fed4FIRE users this account can be created by connecting to the Aggregate Manager using the Allocate or Describe calls.
  • If you grant another user access to one of the nodes you have allocated, they will gain Read/Write access to your Grid'5000 homedir for the duration of the experiment.

Public IP Address for Grid'5000 nodes?

Grid'5000 nodes are on a private network. Interconnection to the Internet is achieved to a NAT, using a 10 Gbps link to RENATER (the french NREN).

We are in the process of:

  • Adding public IPv6 addresses to nodes
  • Adding a configurable firewall to allow reaching Grid'5000 nodes from the Internet using IPv6
  • Extending this to a set of IPv4 addresses (probably doing NAT from the public IPv4 address to the internal IPv4 addresses)

However, this is still work in progress.