Distributing data sets from NFS

Many CSAIL researchers have large collections of data stored in NFS. Sometimes it's desirable to publish some or a part of these data sets on the Web, either to the general public or to a limited set of users. This can be done by means of the data.csail.mit.edu Web server, which is the only supported Web server that has access to CSAIL NFS.

Features

  • Supports both plain HTTP and HTTPS
  • Can serve any file from NFS so long as it is readable by the server
  • Runs nginx, a modern, high-performance Web server, with a load-balancing proxy (also nginx) in front
  • Supports time-limited, per-user links (on a per-directory basis)
    • Other access controls are available but should be kept to a minimum as TIG must implement all changes
  • Custom host names and redirects will not be supported

How to use

There is no default mapping from NFS space to the data server in the way that there is for AFS and the "regular" Web servers. Instead, each NFS directory you wish to publish will be assigned a separate URL on the data server, which will be of the form http[s]://data.csail.mit.edu/somename/. To request a mapping, send in a help ticket and include:
  1. The full path of the directory you want to publish in CSAIL NFS space
  2. The short name you want it published under (somename in the example above)
  3. Whether you want to use time-limited links

About time-limited links

Time-limited links use a weak form of authentication to ensure that the link you give out cannot be used after its expiration date. The version of this that we support is specific to the directory where the target files are located, but not to the individual file, and includes a user parameter so that you can give out different links to different users. The shell script make-expiring-url can be used to create URLs for this mechanism; you must supply a secret string that we will generate for you when the directory is configured on the Web server. Other configurations are possible, depending on your needs, however these will not work with the make-expiring-url script and this mechanism is not a defense against intentional attacks.

-- GarrettWollman - 28 Apr 2015
Topic revision: 28 Apr 2015, GarrettWollman
 

MIT Computer Science and Artificial Intelligence Laboratory

 

  • About CSAIL
  • Research
  • News + Events
  • Resources
  • People

This site is powered by Foswiki MIT: Massachusetts Institute of Technology