New Assistant Professor Serena Wang studies evaluation of AI systems — and how it affects society
UBC Department of Computer Science Assistant Professor Serena Wang combines social science and computer science to improve evaluation of AI systems
When Serena Wang walked into her first computer science systems class as an undergraduate at Harvard University, she never imagined that a decade later, her professor — Dr. Margo Seltzer — would become her colleague in the same department at UBC.
As an undergraduate, Dr. Wang explored research projects in astronomy and biology, but none of them felt like a good fit — until she found computer science.
“With computer science, there was both the math side and the social side,” she says. “I felt that the field could do a lot of good for people if applied thoughtfully.”
She completed her undergraduate thesis project with Dr. Seltzer, learning the basics of research and studying how machine learning can be used to improve operating systems design.
After graduation, Dr. Wang knew that she liked research but she also wanted to explore the industry side of computer science. She landed a role as a software engineer at Google Research. There, she enjoyed learning about how companies work and how a piece of technology can go from an idea to a product for billions of users, but she found herself gravitating more towards the research side of her work. She discovered that she particularly loved writing papers.
“I felt like papers had a staying power,” says Dr. Wang. “Software can be deprecated, but a paper will live in the archive forever. Maybe nobody will look at it, or maybe the work might inspire somebody in the future.”
The pull towards academia led her to The University of California, Berkeley for her PhD where she started her research focusing on machine learning optimization and fairness with Professor Michael I. Jordan, one of the field’s leading figures. During that time, the computer science field was just starting to have discussions about biases in machine learning.
“There were many discussions of social aspects that intersected with the technical aspects of machine learning that really resonated with me,” she says.
She initially worked on optimization algorithms for enforcing notions of fairness and implementing them into machine learning models, but halfway through her PhD, she had a realization that made her pause. She started to question what she was working on.
“I started asking, ‘Do we even really have a good definition of fairness in the first place?’” she says. “How are we turning these ambiguous, subjective notions of fairness into mathematical definitions, and does that even make sense? How do we deal with this sort of ambiguity as computer scientists and as a society?”
It turned out that if she wanted to answer these questions, she needed to look beyond computer science. Her quest to understand the objectives that should be optimized in machine learning pushed her towards the social science field and led her to studying their tools and methods. From economics to education to healthcare, she read books upon books in the social sciences about ways in which metrics and objectives have been used or misused in the past in different fields.
“Whoever designs the evaluations affects the behaviour of the developers of the models, which in turn affects the behaviour of the models that get released. These models might end up with undesirable behaviours, so we need to make sure that we get it right at the level of evaluation.” - Dr. Serena Wang
She became particularly interested in the concept of Goodhart’s Law, which states that “when a measure becomes a target, it ceases to be a good measure.” Because of this, it’s not obvious that more measurement is always better: if a hospital aimed to have lower mortality rates for high-risk procedures, patient outcomes may actually become worse if the hospitals decide to take on fewer high-risk patients to reach their goals.
Along these lines, Dr. Wang started to focus on evaluation in AI systems — how should computer scientists evaluate fairness, safety, and capabilities in such a way that improves social welfare?
“Whoever designs the evaluations affects the behaviour of the developers of the models, which in turn affects the behaviour of the models that get released,” she says. “These models might end up with undesirable behaviours, so we need to make sure that we get it right at the level of evaluation.”
As a postdoctoral fellow at Harvard University, she dove deeper into learning new theoretical tools to approach these problems, including social choice theory, with Professor Ariel Procaccia. Now, she plans to continue studying the AI evaluation ecosystem as an Assistant Professor at UBC Computer Science.
She’s particularly excited about joining UBC because of how the research community actively embraced her interdisciplinary work. She’s also thrilled to be working alongside Dr. Seltzer again.
“Margo is an excellent mentor, and being a faculty in the same department was a huge draw for me,” says Dr. Wang.
By combining methods in theoretical computer science, statistics and economic theory to study evaluation of AI systems, Dr. Wang aims to influence how researchers think about how evaluations are designed.
“I hope to take ideas from the social sciences and inspire other computer science researchers to think differently about the problem they’re working on and think outside the box in the way that they design algorithms.”