Transferring Data
Copying Files to/from the HBSGrid
Copying Files to/from the HBSGrid
Copying files between computers is made easy via terminal and GUI. But before you start the copying process please pause a moment and consider the following:
- What type of data do you have in terms of classification level? Level 3 data must not be stored on user devices or portable media unless the device or media is encrypted. Level 4+ data must not be stored on user computing devices (i.e. laptops, smartphones, tablets), but may be stored on external encrypted portable storage media. Data designated as level 3 and higher may only be used, stored or processed on servers or services that comply with Harvard data protection requirements. For details on data classification please visit the Harvard University Information Security Policy page.
- Remember that home folders are not sharable, and typically contain files that pertain to your work only. This also includes the default location for configuration or preference files, and work that is considered 'in development'.
- Project folders are shared locations, and ideally used for collaborative project spaces. Please see our recommendation below on organizing project spaces. NOTE: Only those persons who are listed on the data usage agreement or IRB proposal should have access to these directories. This is the responsibility of the primary HBS faculty member.
- Do not dump the contents of your laptop or desktop in your home or project spaces as a backup. Please consider solutions like Dropbox, CrashPlan, or BackBlaze; or the free Code42 backup service from HBS IT if the functionality of a backup is needed (HBS members only).
- HBS IT regularly scans the research storage areas for viruses as a part of the Harvard University security policy. Normally viruses are quarantined or removed via antivirus software on HBS-issued laptops and desktops, preventing them from reaching the research storage. But personal devices may not have similar protections. If you should receive a message from HBS IT or RCS about virus-containing files in your home or project spaces, please remove these as soon as possible to ensure the safety and integrity of research data across the entire computing grid and research storage.
Below is a brief guide on how to copy files.
Transferring Data when using NoMachine
NoMachine has a cool feature: one can "attach" one's local disk as if it were a network volume on the HBSGrid cluster. Although probably too slow for computing at HBS on files at home, this feature is an easy way to make a disk, directory, or set of files available for transfer or synchronization. Please see NoMachine's Accessing files on local or remote disk drives, and be sure to mark your disk as Private. (See this page for NoMachine client Settings Panel options) Note: if working with L3 or L4 data, ensure that any data movement operations are compliant with your data safety level.
Copying Files Using Terminal
SSH terminal protocols such as SFTP, SCP, and rsync allow you to copy files with simple unix commands.
scp
Simply issue the scp
command and make sure that the order of the files is always source before destination.
scp [username@server:] [location of file] [destination of file]
or
scp [location of file] [username@server:][destination of file]
For details please refer to Accessing Project Spaces & Folder via Terminal
rsync
Rsync is known for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination.
The basic syntax is:
rsync [username@server:]SOURCE [username@server:]DESTINATION
where
SOURCE
and DESTINATION
are filesystem paths on local servers. Include the [username@server:]
specification only if the source or destination are on a remote server.
For more details please refer to Accessing Project Spaces & Folder via Terminal
Copying Files Using a GUI
Another convenient way to copy files is by directly connecting to your home folder or project space. Common ways to achieve this include mapping a network drive (PC) or mounting a shared volume (Mac), and using a file transfer program such as SecureFX, Filezilla, or Cyberduck.
Mapped Drive (PC)
Map a network drive to your home folder or project spaces in Windows Explorer to easily access and copy over files.
For details please refer to this page: Accessing Project Spaces & Folders via GUI
Mapped Volume (Mac)
Connect to your home folder or project spaces in Mac OS with Finder "Connect to Server..." to conveniently access and copy over files.
For details please refer to this page: Accessing Project Spaces & Folders via GUI
SecureFX
Windows SecureFX GUI file transfer client implements the SFTP protocol for fast and simple file transfers.
Please see this document for using the Windows SecureFX GUI file transfer client.
FileZilla
FileZilla is a cross-platform GUI file transfer client.
Please see this document for using the cross-platform FileZilla GUI file transfer client.
Cyberduck
Cyberduck is a Mac GUI file transfer client.
Please see this document for using the Mac Cyberduck GUI file transfer client.