Principal Investigator Guide

This document is a guide for new PlanetLab Principal Investigators who have just joined or are considering joining PlanetLab. This guide outlines the special responsibilities that PIs have and the procedures that PIs should follow to manage their site's participation.

1. Overview

As a Principal Investigator, you have several roles and responsibilities:

  • Oversight. You are responsible for overseeing all slices that you create on behalf of the users at your site. Because you may create slices across any of the nodes in the PlanetLab network, you may be required to respond to incidents at other sites caused by the behavior of one of your slices.

    For example, if one of your slices (accidentally or deliberately) sends malformed packets from a node at Princeton to a large number of hosts and a complaint is filed, you (not the PI at Princeton) are responsible for responding to and resolving the complaint. The PI and Technical Contact at Princeton may choose to help you, but ultimately, you are responsible for the behavior of your slices wherever they run.

    Consequently, you should ensure that you, as well as all of the users at your site, understand, agree to, and abide by the PlanetLab Acceptable Use Policy (AUP). If you abide by the AUP, you minimize the risk to both yourself and the PlanetLab network.

  • Account management. You are responsible for account management at your site. You are the only person at your site who can:

    Before enabling a user account or assigning a slice to a user, ensure that the user reads the User's Guide first and understands his or her responsibilities.

  • Node management. You are also responsible for the physical maintenance of the nodes at your site. The PlanetLab Consortium Membership Agreement that you signed, requires you to provide and maintain at least two nodes that meet PlanetLab minimum hardware and availability requirements. The staff at PlanetLab Operations remotely installs and manages the software on your nodes. There is no need for you to purchase or maintain any operating system software for the nodes, you only need to provide working hardware, connectivity, and routine maintenance.

    You may choose to appoint a Technical Contact to assist you. Both you and your appointed Technical Contact should read the Technical Contact's Guide before attempting to bring up a node.

    Your nodes should be promptly serviced or replaced when required (or requested by PlanetLab Support). Ensure that each node is provided with continuous and unfiltered Internet connectivity, and that your local site administrators understand that your nodes are participating in the PlanetLab network. Consult the PlanetLab Hosting Requirements document for more information.

2. Account Management

Your most common task as a PI will be to approve the use of PlanetLab resources by your users. To grant access to PlanetLab nodes, create a slice and assign users to that slice. You should create one slice per project or experiment at your site.

Users may be assigned to more than one slice at a time. All users assigned to a slice may add and delete nodes from that slice. Users login to nodes by specifying the slice name as the login name; thus all users of a slice login to the same account on each node. Each user should generate and use their own SSH key to login, however. See the User's Guide for more information about how to login to nodes.

Because all of the users assigned to a slice have virtual root access to all the nodes they assign to the slice, as a PI, you must exert careful oversight over their use of the slice. Granting and revoking privileges to users at your site is your responsibility.

2.1. Creating an account

Once your site has filed the appropriate paperwork and has been approved to join PlanetLab, submit a connect request which will establish your site and account in the database. This request must be approved by PlanetLab Operations; wait until you receive an e-mail indicating that your request has been approved, before continuing.

Make sure that you list yourself as the Principal Investigator of your site. If you do not want to appoint a separate Technical Contact to handle the day-to-day maintenance of your nodes, also list yourself as the Technical Contact for your site.

PlanetLab Support will enable your site and account within a day. After you receive an e-mail from PlanetLab Support, you may login and begin to enable user accounts and create slices.

Under certain circumstances, multiple PIs may exist within a single site. Additional PIs should create their own accounts. If a user with an existing account needs to be upgraded to PI status, have him or her contact PlanetLab Support.

2.2. Enabling users

You must enable the accounts of users at your site before they may login. Click on the name of the user to be enabled and click Enable this account to enable the account. Ensure that the users at your site understand that you, rather than PlanetLab Support, are responsible for enabling their accounts.

If a user leaves your site or no longer requires access to PlanetLab, disable his or her account by using the Manage Users page on the website.

2.3. Creating slices

You may create and delete slices at any time. To create a slice, visit the Manage Slice page. Slice names must begin with the designated prefix for your site, must contain only alphanumeric low ASCII characters, and must be less than 32 characters in length.

Academic or other educational institutions may create up to 10 slices. Commercial institutions may create up to 2 slices. Your slice limit may be adjusted depending on your site's resource contributions and other factors. If you require more slices than you have been allocated, contact PlanetLab Support.

Be aware that when you delete a slice, all processes and data associated with that slice will be permanently deleted. Before deleting a slice, notify all of the users assigned to the slice and ensure that any data that they would like to keep, are retrieved before the slice is deleted.

2.4. Assigning users to slices

After creating a slice, you may assign users to it. Ensure that the users who will use the slice have registered for accounts and that you have enabled their accounts (see above). Visit the Manage Slices page, select the slice to assign users to, and select the set of users that should be granted access to the slice.

Note that deleting a user from a slice does not cause slice data or process state to be affected on any node. Deleting a user simply disables SSH logins to the slice by that user. It may take up to an hour for any changes to be propagated to active nodes.

2.5. Assigning nodes to slices

Once you have assigned users to a slice, you or any user of the slice may assign nodes to it by using the Manage Nodes form. If you intend to run a slice on all available nodes, think carefully about you will manage and monitor it before you deploy it, and visit the Manage Nodes form often to add new nodes. To automate this task, consider using the programmatic interface to the slice database instead of the website.

Be aware that when you delete a node from a slice, all processes and data associated with that slice will be permanently deleted from that node. Before removing a node from a slice, notify all of the users assigned to the slice and ensure that any data that they would like to keep, are retrieved before the node is removed from the slice.

3. Node Management

The Operations staff at PlanetLab Central remotely maintains and administers your nodes. There is usually no need for you to become involved in day-to-day maintenance, although you may occasionally be asked to reboot or repair your nodes.

Because your site owns the nodes and their connectivity, however, you can exert some control over their operation. In particular, you may restrict the outbound bandwidth of your nodes, login to a restricted administrative account on the nodes, and switch the nodes into an administrative Diagnose/Safeboot state.

3.1. Setting bandwidth limits

By default, your PlanetLab nodes are not bandwidth limited. You may update the outbound bandwidth limits of each of your nodes by visiting the Bandwidth Limits page on the website. The bandwidth limits cap the average transmission rate of the nodes.

3.2. Logging in as site_admin

You may login to your own nodes remotely via SSH as the user site_admin. Before attempting to do so, ensure that you have already uploaded your public SSH key to the website and waited at least an hour for the key to propagate to your nodes. See the User's Guide for more information about creating and uploading an SSH key. To login to one of your nodes, specify site_admin as the login name, the path to your private key file (e.g. ~/.ssh/identity), and the node to login to (e.g. planetlab-1.cs.princeton.edu):

ssh -l site_admin -i ~/.ssh/identity planetlab-1.cs.princeton.edu

The site_admin account is not isolated in a VServer as all other accounts are. The site_admin user may execute a variety of administrative programs in /usr/sbin via sudo:

# See all processes (including those running in VServers)
sudo /usr/sbin/vps ax

# See /var/log/messages
sudo NOT CURRENTLY SUPPORTED. Contact support@planet-lab.org if you need this functionality.
# See all network traffic
sudo /usr/sbin/tcpdump
# Shutdown (halt) the node in 5 minutes
sudo /sbin/shutdown -h 5 "Scheduled maintenance, please save your work and logout"
# Set my password for console login
sudo /usr/bin/passwd site_admin

# Disable my password for console login
sudo /usr/bin/passwd -d site_admin

The last two commands may be especially useful if your nodes are connected to monitors and keyboards, are located in a secured environment, and you wish to be able to login to them on their consoles. To login to a node on the console, specify site_admin as the user and the password you set with /usr/bin/passwd as in the example above.

3.3. Stopping or rebooting nodes

Before powering off or rebooting a node for scheduled maintenance, please try to notify all users of the node first by e-mailing the PlanetLab Users mailing list. To power off a node, login to the node as the user site_admin and execute:

sudo /sbin/shutdown -h 5 "Scheduled maintenance, please save your work and logout"

To reboot a node, execute:

sudo /sbin/shutdown -r 5 "Scheduled maintenance, please save your work and logout"

Wait for the node to power off or reboot by itself. Wait at least 15 minutes before deciding to manually power off or reboot a node.

As a last resort, if you cannot login to the node via SSH or the console, simply power off (or power cycle) the machine. If you do not have physical access to the nodes but have connected them to a remotely accessible power control unit (PCU), follow the instructions that came with your PCU for removing or cycling power to the outlets.

3.4. Diagnose/Safeboot mode

In an emergency situation, such as hardware failure or a verified security compromise, you may reboot your nodes into a secure administrative Safeboot state. Slices will not run in this state, and only staff at PlanetLab Operations will be able to access the nodes. Unless a hard drive has failed, slice data (or forensic evidence of compromise) will not be lost.

To place a node into safeboot mode, visit the Node Control page on the website and click Safeboot (or Diagnose). Login to the node as the user site_admin and execute:

sudo /sbin/shutdown -r 5 "Emergency, please save your work and logout"

If you cannot login to the node via SSH or the console, you may send to the node a specially formed ICMP packet called the PlanetLab Ping Of Death (POD) that will immediately reboot it. Click Stop to attempt this action. You may only Stop nodes that are in Boot state, so if you placed the node into Safeboot (or Diagnose) state, put it back into Boot state before clicking Stop.

4. Incident Response

One of the main purposes of PlanetLab is to enable research into new Internet technology. Frequently, researchers will deploy technologies on PlanetLab that use the Internet in new ways. As a result, PlanetLab network traffic is often viewed as anomalous by many automated Intrusion Detection Systems (IDSs) and may trigger alerts. If an alert is interpreted as a security threat by a system administrator or IDS, a complaint may be lodged against your site or other PlanetLab hosting sites.

While the vast majority of such complaints can be quickly resolved by explaining PlanetLab's purpose, some complaints may be an indication that an experiment is out of control or, rarely, that a slice has been compromised. All complaints should be taken seriously. When you receive a complaint about a security incident involving one of your PlanetLab nodes, determine the IP addresses of all hosts involved, and the time when the incident occurred. You may have to ask the person who reported the incident for this information. If they cannot or will not give you this information, there is little that can be done to investigate the incident, and the incident can be resolved if your local site administrators agree.

Once you have determined the IP addresses of all hosts involved, you may be able to identify which slice sent the traffic and ask the researchers in charge of the slice to deal with the complainant directly. Use the search form on the home page of the PlanetLab node(s) involved in the incident (for example, http://planetlab-1.cs.princeton.edu/) and input as much information as you can about the incident (the hosts and ports involved and a time frame). If you are able to determine which slice was responsible for the traffic, contact the researchers in charge of it for an explanation, and ask that they respond to the complainant directly and copy you and PlanetLab Support on all correspondence.

To contact the PI, Technical Contact, or users of a particular slice, such as princeton_test1, send e-mail to the following aliases, replacing princeton_test1 with the name of the slice and princeton with the "short name" of the site:

  • princeton_test1@slices.planet-lab.org

    The users of the princeton_test1 slice, and their PI.

  • pi-princeton@sites.planet-lab.org

    The PI(s) at Princeton.

  • tech-princeton@sites.planet-lab.org

    The Technical Contact(s) at Princeton.

If you are forced to take immediate action by your local site administrators, or because you suspect that one of your nodes has been compromised, reboot the affected node(s) into their secure Safeboot (or Diagnose) modes by following the instructions in Section 3.4, “Safeboot/Diagnose Mode”. Do not disconnect your nodes from the network or turn them off; as explained in Section 3.4, “Safeboot/Diagnose Mode”, any offending network traffic will stop once the nodes reboot into Safeboot mode. Contact PlanetLab Support immediately after rebooting your nodes to assist in resolving the complaint.

4.1. Common problem reports

It is not uncommon for normal PlanetLab network traffic to be misinterpreted by IDSs as an attack or a symptom of a compromised machine. For example, PlanetLab Support often receives complaints that a PlanetLab node is a Windows-based machine that has been infected by a virus. Of course, such a claim is misguided, since PlanetLab nodes are Linux-based, but you should still take such complaints seriously, investigate the source of the traffic, and attempt to resolve them.

Other common problem reports, and suggestions for resolving them, are listed below.

4.1.1. Excessive traffic

Some experiments generate large amounts of network traffic, sometimes for sustained periods of time, and may trigger alarms at your site. You may cap the outbound network bandwidth of your PlanetLab nodes by following the instructions in Section 3.1, “Setting bandwidth limits”. If you suspect that an experiment is out of control, or your site cannot handle the traffic load even with bandwidth limits in place, contact PlanetLab Support. Otherwise, try to explain to local site administrators the nature of PlanetLab, its requirements, and that heavy network usage is to be expected.

4.1.2. Port scans

Network mapping experiments running on PlanetLab often probe remote hosts, triggering "port scan" alerts. Sometimes, problems with probing experiments manifest themselves locally:

  • Probes often contact a large number of unique destination addresses and may overload the caches on your local routers.

  • Bursts of incoming ICMP errors (e.g., TTL Exceeded or Port Unreachable) may be received in response to locally generated probes.

Established network mapping services running on PlanetLab such as Scriptroute go to great lengths to minimize the chance of triggering a port scan alert. If you receive a port scan alert and cannot placate the complainant, contact PlanetLab Support to have the complainant's subnet added to a blacklist.

4.1.3. Open proxy alerts and objectionable content complaints

Several "semi-open" proxies run on PlanetLab that test new content distribution technologies. Two of the more successful proxies currently running are CoDeeN and CoralCDN.

Many system administrators lodge complaints against PlanetLab proxies because they notice large amounts of traffic from PlanetLab to their site, and are either unaware that users at their own site are requesting the traffic through PlanetLab proxies, or do not know that the proxies are research projects that are actively managed. Most of the time, a simple explanation of what a proxy is, and how PlanetLab proxies are not entirely "open", suffices to resolve this type of complaint.

You may receive complaints about objectionable or illegal content being served from proxies running on your PlanetLab nodes. Because most proxies running on PlanetLab are caching proxies, the content is not being served from your nodes, it is being served from a third-party server somewhere else on the Internet, to a user who has voluntarily requested it. Nevertheless, if you or your local site administrators feel pressured to withdraw from the PlanetLab Consortium agreement because of objectionable or illegal content passing through your nodes, contact PlanetLab Support to have the incident reviewed.

5. PlanetLab Security

Because of its distributed nature, PlanetLab was designed from the ground up to be secure. If you are concerned about the security of your nodes or the network, or wonder how your nodes are securely managed from a remote location, read on.

5.1. Secure boot

Every PlanetLab node boots from the BootCD rather than the hard disk. Because the CD is immutable, the boot environment is secure and immune to root kits. The first step in the boot process is to contact PlanetLab Central through a secure, authenticated channel for the appropriate task to run, based on the boot state of the node. If the node has been placed into its secure Safeboot (or Diagnose) mode, it will not initialize further, and a secure console will be opened that can only be accessed from PlanetLab Central by PlanetLab Operations staff.

5.2. Minimal software

To minimize potential security vulnerabilities, PlanetLab nodes are installed with only the minimum amount of server software necessary to manage and run virtual servers. Only an SSH server runs in the base system; FTP, TELNET, and SMTP servers do not (and, in fact, are prohibited from running). All other services run inside isolated virtual servers, described below.

5.3. Isolated virtual servers

All management services and PlanetLab experiments run inside isolated virtual servers (also called slivers of a slice or VServers), that are restricted from performing most administrative operations. The resource consumption of every virtual server is actively managed. Even if a virtual server is compromised or goes out of control, the rest of the virtual machines and the base management software may still run.

5.4. Active management

PlanetLab Operations staff works full-time to ensure the security and integrity of the network by applying security patches as necessary, and by constantly auditing usage of the network. All staff members have worked in industry and are well-qualified to provide support and analyze security incidents.

Document Revision History

Revision 1.0

02.19.2005

Initial draft.

Mark L. Huang

Revision 1.1

02.21.2005

Add Section 4, “Incident Response” and Section 5, “PlanetLab Security”.

Mark L. Huang

Revision 2.0

02.02.2006

Update for PlanetLab 4.0 release.

Marc E. Fiuczynski

 Revision 2.1 12.01.2008 Updated with new boot state names  Stephen Soltesz