Data at Scale

CPSC 538W // Winter 2016
Andrew Warfield (andy@cs.ubc.ca)
Monday/Wednesday 9:00-10:30am, ICCS 206

One of the most significant changes to take place in computing over the past decade has been the incorporation of large quantities of data within the applications that we use on a daily basis, and the applications that we, as computer sciences, build and maintain in our careers. The move to work with data at scale is often a subtle and implicit aspect of systems that we use without thinking about. For example, the rendering of most web pages on the internet triggers one or more complex and low-latency ad auctions, in which multiple advertisers bid real dollars to place content as part of the rendered page. Similarly, using mobile devices to solicit driving directions through Waze or a ride through Lyft depend critically on complex and near real time geoinformatics systems that must consider geography, traffic, and other factors, and then must project these properties over the travel time of the trip to be taken.

In this course, we will survey applications of large-scale data analysis, and the associated systems-level challenges and opportunities that accompany them. As a systems course, we will take particular focus on the practical and mechanistic properties of real-world systems with a mind to understanding what makes them hard to build, whether they are efficient, and what the challenges are in achieving scale.

Our selection and coverage of material will be based on three broad questions:

• What are the applications? What interesting application use cases are driving new technologies for data-intensive applications? What new application areas are being enabled through the emergence of tools and infrastructure that allow developers to work with data in new ways or at large scale?
• How are software tools evolving to serve these applications? What are teh state-of-the-art tools being used to work with data at scale today? What are the dominant challenges that are currently being explored by the research community?
• How will evolutions to datacenter hardware impact the way these systems are built? What changes do we expect in terms of the relative performance and latencies of memories and networks? Will computational elements become more specialized? Does the density and cost of emerging hardware change the way we approach teh design and deployment of these systems?

Course Organization

The course is organized as 11 core weeks, each with a specific topic. We will read three papers per week, two on Monday and one on Wednesday. Each paper will be presented by one of the students in the class and then discussed as a group. We will spend roughly the second half of Wednesday’s class discussing the three papers as a set, and generalizing out to the week’s topic area.

You will be expected to write a program committee-style review of each paper in advance of the class in which it is discussed.

You will also be expected to complete a self-directed project for the course. This may be done individually or as a group, and will account for 50% or your course grade. The remaining 50% will be divided evenly between the paper presentations that you do (25%) and your reviews and participation in class discussion(25%).

Things to read and watch for background and context. This is a bit of a random list, mostly intended as interesting motivation and background.

• Flash Boys by Michael Lewis
• The Tail at scale, Jeff Dean (Google)

Classes:

Part 1: frameworks and approaches to working with data.

Part 2: Infrastructure support.