Table of contents

Overview

The Boston University Scientific Computing Facilities (SCF) consist of a collection of high-performance computers, high-speed networks, and advanced visualization facilities. These facilities are managed by the Research Computing Services (RCS) group of Information Services & Technology (IS&T) in close consultation with the Research Computing Governance Committee, the BU Center for Computational Science, and the Rafik B. Hariri Institute for Computing and Computational Science & Engineering. The current computing system is the Shared Computing Cluster (SCC).

Now that you have an SCF account, your BU login and Kerberos password give you access to the SCC. New users to the system should consult our Getting Started on the SCC introductory page which will walk you through the basics of connecting to and using the SCC from a Windows, Macintosh, or Linux machine. We have also created a PDF document entitled Getting Started Reference Sheet for the SCC which we recommend you print, read, and keep handy as a reference, particular if you are not familiar with Linux systems, but it will likely be useful for everyone. We recommend you make use of our web materials, including tutorials and consulting services available through Research Computing staff on the use of all these facilities. Please contact us at help@scc.bu.edu if you have additional questions.

General conditions of use

Your use of these machines is governed by Boston University’s Conditions of Use and Policy on Computing Ethics. By using your account on the SCF and other University computers, you are agreeing to the terms and conditions set forth therein, as well as the usage policies described below.

Passwords and access

When you first get an account, you will receive instructions on accessing your account using your BU login and Kerberos password. External users only will be sent instructions on setting a BU login and Kerberos password. Use your Kerberos password to access the computing systems and web materials.

You may optionally, if you have one, also use your local SCF Linux (non-Kerberos) password to log in to the SCC. If you ever wish to change your SCF password, you should log in to scc1.bu.edu and run the command passwd. If you know your BU Kerberos password, you can also set/change your SCF password here. Otherwise, if you forget your SCF password, you should send email to scfacct@bu.edu. Your default shell is the Bash shell but you can change it here.

News and announcements

We will periodically make important announcements regarding usage policies, software and hardware upgrades, downtime, etc. It is important that you read these messages on a regular basis and we provide several methods for you to do so.

All messages are posted to the system message board, which can be viewed by running the command msgs. By default this command will be included in your .login startup file. If you modify this file, we suggest that you continue to include this command.

We also have an SCC Updates page which we will keep updated with information on the status of the system and any planned downtime.

We will also regularly send individual messages to you regarding your account status and usage. These will be sent via email to your SCF account (your_login_name@scc.bu.edu). If you do not regularly read email on this system, you should have your mail forwarded to a machine where you do. For new accounts for internal BU users, we will by default have created a .forward file in your home directory forwarding your SCC email to your_login_name@bu.edu but you can edit this as you wish.

Getting information and help

We rely heavily on the web to provide information about our facilities. Here are some pages we think you will find particularly useful.

If you are experiencing system problems, please send email to help@scc.bu.edu.

For more information or help in using or porting applications to our systems, please see our Scientific Programming Consulting page or contact one of our Scientific Programmers, Kadin Tseng (kadin@bu.edu) or Yann Tambouret (yannpaul@bu.edu).

If you have questions regarding your computer account or resource allocations, please send email to scfacct@bu.edu.

Allocations and Accounting

We account for all batch usage by users on the SCC. It is the responsibility of the Principal Investigator to monitor his/her project’s usage and to request an appropriate allocation of processor time, expressed in “service units” (SUs). Information on accounts and allocations, as well as forms to request resources may be found on the Accounts & Project Maintenance pages.

Allocations

Although there are no monetary charges for use of the shared services, all projects are given a specific annual usage allocation, in terms of “service units” (SUs). This allocation must be renewed (and adjusted if appropriate) annually. Different compute nodes have a different SU charge rate based on clock speed and other performance-related factors. These rates and other information about the nodes is detailed in our Technical Summary. Note that by “CPU” we are referring to a single “core” or processor.

As indicated in the Technical Summary, some of the SCC nodes have limited access due to being part of the Buy-in Program. Not all SCF users can fully utilize these systems. For those users with special access to these systems, the SU charge is 0.0 for these systems only.

On the SCC, you will be charged for the amount of wall clock time you use each processor for so if you request 10 processors and your job runs for 6 hours on a set of nodes with an SU charge rate of 2.6, you will be charged 156 SUs (10 processors * 6 hours * 2.6 SU charge rate).

Projects which exceed their allocation will be prohibited from running additional batch jobs. It is the responsibility of the project’s Principal Investigator to monitor his project’s usage and to request an appropriate allocation of time/SUs. Information on accounts and allocations may also be found on the Accounts & Project Maintenance pages.

Reporting

Each month we send Principal Investigators (and project Administrative Contacts where applicable) a summary of usage and remaining allocations for their projects; this report also provides Project Disk Space allocation and usage information. Individual researchers are sent a summary of their own usage for all the projects with which they are associated. Individuals may also review the details of their recent usage, Project Disk Space usage, and monthly summary information using the password-protected web pages which may be found under the SCF Resource Monitoring page.

In addition to emailed reports and individual usage web pages, we have developed a utility called “acctool” to help you keep track of your CPU usage. Type “acctool -help” on any of the machines to get more information. An example command to show a particular user’s usage for all of January 2014 in minimal detail but including Project SU Balance is “acctool -host all -detail -1 -user kadin -balance 1/01/14 1/31/14“.

Project accounting

All usage accounting is based on projects. For researchers who only belong to one project, the fact that the accounting is project-based will be inconsequential. However, researchers who are associated with multiple projects must pay special attention to assure that their usage is properly attributed to the correct project. The procedures for doing this are described below.

Each account on the system has been assigned a default project. This is the project which will be charged if no further actions are taken. Projects are implemented as UNIX groups on all systems. The command “groups” shows all of your projects. The first one listed is your default project. To change your default project, go to our Resource Monitoring page and click the link which begins “Individuals can see” (you will need your BU login ID and Kerberos password). You should then complete and submit the appropriate web form, and your default project will be changed the next time the system configuration files are updated, generally overnight. Batch jobs will be accounted to your default project unless you use the -P project_name flag for qsub; this flag is required for people who are members of any BUMC projects.

Usage policies and batch

The following “login nodes” are available for interactive work: scc1.bu.edu and scc2.bu.edu (and geo.bu.edu and scc4.bu.edu for certain groups of users). All other nodes are reserved for batch use only.

SCC Batch System

The batch system on the SCC is the Open Grid Scheduler (OGS), which is an open source version of the Sun Grid Engine.

Jobs running on the login nodes are limited to 15 minutes of CPU time and a small number of processors. Jobs on the batch nodes have a default wall clock limit of 12 hours but this can be increased, depending on the type of job. Single processor (serial) and omp (multiple processors all on one node) jobs can run for 720 hours, MPI jobs for 120 hours, and jobs using GPUs are limited to 48 hours. Use the qsub option -l h_rt=HH:MM:SS to ask for a higher limit. An individual user is also only allowed to have 256 processors simultaneously in the run state; this does not affect job submission. These limitations apply to the shared SCC resources. Limitations on Buy-In nodes are defined by their owners.

Hardware configuration

The Boston University Shared Computing Cluster (SCC) is the first Boston University high performance computing resource located off of the Boston University campus. The SCC is located in Holyoke, MA, site of the Massachusetts Green High Performance Computing Center (MGHPCC) where energy is plentiful, clean, and inexpensive. The SCC is a Linux Cluster running Centos 6.3; it is continually expanding and currently has over 3600 cores, 232 GPUs, and over a petabyte of storage for research data. The cluster went into production use in June, 2013. The login machines are scc1.bu.edu, scc2.bu.edu, and for certain user groups geo.bu.edu and scc4.bu.edu.

Please see our Technical Summary page for more information on the configurations of all the machines that are part of the SCF.

Storage/Disk Space

You have a home directory on the SCC with 10 GB of space. All home directories are backed up nightly and protected by Snapshots. If you accidentally remove a file, you should see if there is a Snapshot of it in the .snapshots subdirectory. If you need help with this, you can contact us. Please specify exactly what files you deleted, what machine and file system those files were on, and at what time you deleted them.

Projects are also granted Project Disk Space for the exclusive use of the project group. Projects are granted both backed-up space and not-backed-up space; both are protected by Snapshots. The default amounts of space are 50GB on each partition but project leaders can request more. Details are available on our Project Disk Space page.

The /scratch file systems are available for people who need a large amount of storage for a short period of time. Files are automatically purged after 30 days. If there is a critical shortage of scratch space, it may be necessary to purge files which are less than 30 days old. Files which have been “touched” but not modified will be treated as old and removed immediately. The scratch partitions are NOT BACKED UP and do not have Snapshots.

Each machine has its own /scratch partition, but these partitions can be accessed from all SCC machines by using the full path (/net/scc-aa1/scratch/ for example).

We also encourage users with large amounts of data that does not need to be online at all times to take advantage of the IS&T Data Archiving Service.

Software

Information regarding the software available on the systems can be found on our Software Packages page.

Note that for some software packages, you may need to add specific directories to your execution path or correctly set particular environment variables in order to use them. This is usually done by modifying your .login, .bashrc, or .cshrc file. Please refer to the documentation on the web page referenced above for the specific details. Packages which use Modules will generally handle this automatically.