ICML 2018: An AI party in our own backyard

Keynotes

The keynotes highlighted a number of bigger themes that were featured throughout the conference, from deep learning security and energy efficiency to fairness and transparency. Here are a couple of the ones we found particularly interesting:

• Dawn Song: AI and Security: Lessons, Challenges and Future Directions, pt1, pt2. Dawn discussed the security implications that deep learning systems must consider before being put into production. She discussed many different types of attacks, from those that compromise the integrity or confidentiality of the model to those that misuse the AI for a different and unintended purpose. She closed by discussing how a pipeline that includes data encryption, program verification and differentially private results can prevent these types of attacks.
• Max Welling: Intelligence per Kilowatthour, pt1, pt2. Max presented a talk that ventured from statistical physics to information theory and Bayesian theory, and finally back to thermodynamics, all in order to discuss the energy costs and perceived economic value of deep neural networks. He emphasized the fact that overparameterization aids optimization but leads to an energetic trade-off when evaluating. He offered a number of solutions for how to optimize these networks to be energy efficient via pruning and compression techniques.
• Josh Tenenbaum: Building Machines That Learn and Think Like People, second half of this (the first half is Joyce Chai). Josh gave an overview of research that highlighted the difference between how children actually learn about the physical world and how deep learning models make predictions. It was a talk that spanned all the way from probabilistic programming to video game physics engines in order to show that human learning is much more than sophisticated pattern recognition.
• Test of Time Award: The “Test of Time Award” is given to the most influential ICML paper from ten years ago. This year it was given to Ronan Collobert and Jason Weston for their work, “A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning,” where they introduced several key contributions for modern NLP, such as using neural networks for learning semantically meaningful word vectors, using CNNs for NLP, and using attention mechanisms for word selections. Take a look at the ICML presentation by the winners or read the interview with them.

Workshops

The main technical program of the conference was followed by smaller, more niched, and somewhat more informal workshops. There were an astounding 67 different workshops to choose from, ranging from more theoretical topics such as “Geometry in Machine Learning” and “Theoretical Foundations and Applications of Deep Generative Models” to very practical ones such as the “Joint Workshop on Multimedia for Cooking and Eating Activities and Multimedia Assisted Dietary Management” and “Artificial Intelligence for Wildlife Conservation Workshop.” A complete list of workshops can be found here. Below are accounts on a few of the workshops we attended.

• The 2018 Joint Workshop on Machine Learning for Music. As deep learning overtakes much of music informatics, the increasingly convincing results in singing voice synthesis are truly inspiring. A highlight of the music workshop was the focus on ethical implications of enabling a future where a person’s voice can be automatically imitated, and we are excited that the research community strives for sustainable AI. There was a clear signal from presenters that AI is not meant to relieve artists of expression but rather to enhance the human experience. Put simply, just as the electric guitar or the digital audio workstation, deep learning methods will add new colors to an already vibrant palette. How to make these colors accessible was a recurring theme throughout the workshop, and we are super excited to help get deep learning models into the hands of musicians.
• Theoretical Foundations and Applications of Deep Generative Models. Generative models is one of the hottest topics in machine learning these days, and several of the sessions in the workshop were jam-packed. A couple of the main questions that researchers were focusing on here are related to model convergence and better understanding of what is actually learned.
• AutoML. One of the workshops that we were the most excited about was on AutoML. In AutoML were using machine learning techniques to automate the process of building and training machine learning models, thereby cutting out humans from the loop almost entirely. Zoubin Ghahramani, Chief Scientist at Uber, gave a good overview about the topic, including automating data preprocessing, automating report generation and Uber’s own framework for probabilistic programming Pyro, among other things. Lars Sjösund really liked “AlphaD3M: Machine Learning Pipeline Synthesis,” where the process of finding a good machine learning pipeline was framed as a one-player game. While having similar performance to already existing frameworks such as TPOT, it was an order of magnitude faster.
• Fairness, Accountability and Transparency in Machine Learning. On the last day of this year's ICML was a dedicated workshop for fairness, accountability and transparency that included a number of presentations on methods for enforcing mathematical fairness, removing bias from data and handling other difficult challenges in eliminating discrimination by AI. There are many known frameworks and methodologies to help remove bias from the data and machine learning algorithms used, like ignoring sensitive attributes, applying counterfactual decision-making and focusing on equality of opportunity, just to name a few. The workshop focused on specific methods for evaluating automated decision-making tools and finding ways to build AI systems without inadvertently encoding and perpetuating societal biases.

Debates

A set of panel discussions were held on Sunday where panelists were given the opportunity to present their views about topics such as scientific rigor, security in machine learning systems and algorithmic fairness and limitations with deep learning systems. The quality of the debates varied substantially, where some discussions were reduced to uncontroversial tautologies and panelists agreed with one another. However, there were a number of highlights and noteworthy takeaways:

• How important is reproducibility in machine learning? It is clear that many papers today are difficult to reproduce given problems such as lack of details in the article, inaccessible data and unavailability of computational resources.
• The success of deep learning in computer vision is clear to everyone. However, as deep learning systems are being deployed in self-driving cars, can we trust the current level of performance in a secure way? Do we need to perform more research into protecting against adversarial attacks, or should we proceed to deploy systems given that they statistically are already safer than humans?
• Two interesting concepts regarding security which were discussed was if AI systems should be studied “in vitro” (offline, outside of real-world) or “in vivo” (online, find weaknesses while deployed in real-world). It would be best to find and learn about weaknesses in the real world, but for some tasks that may be difficult for security reasons.
• The fairness debate was rather strange since everyone seemed to agree with each other and there was little to debate. However, it seemed clear that the problem of bias in data and fairness in algorithms needs to be resolved not only with technical means but also from social and political aspects.
• The limitations of deep learning debate was by far the most interesting, with senior experts on the panel. To summarize the discussion briefly, it is clear that deep learning has significantly advanced our ability to build intelligent systems. However, it would be naive to believe that nothing else is needed to achieve general intelligence 50 years down the road.

Papers

With over 5,000 participants, the conference more than doubled in size compared to last year. The total number of submissions exceeded 2,400, with approximately 600 papers accepted. Deep learning was the most popular topic among submissions, as expected.

A few papers that caught our attention and are now at the top of our reading lists:

• Conditional Neural Processes. A paper from DeepMind where they show the benefits of combining Gaussian processes with neural networks.
• AlphaD3M. A paper from New York University describing an automatic machine learning pipeline generator (see AutoML above).
• Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). A paper from Google showing how to introspect image classification models with human-friendly concepts.
• Essentially No Barriers in Neural Network Energy Landscape. A paper from Heidelberg University that studies the energy barriers present in neural network energy surfaces.
• TherML: Thermodynamics of Machine Learning. A paper from Google that hints at some interesting parallels between thermal physics and representation learning.
• Delayed impact of Fair Machine Learning. A paper from UC Berkeley showing that fairness criteria may actually be detrimental to those groups that they set out to protect.

Booths

A big part of the conference are the booths and poster sessions. Poster sessions are held every evening to give attendees an opportunity to read overviews of the topics and sessions they couldn’t attend, or the chance to ask follow-up questions to the paper authors. The booths, however, are arguably the best part of the conference (besides the learnings you got from the sessions, of course). You are surely already aware of the concept of swag (the free stuff we all get), and here you really don’t need to be shy! Companies who host booths are eager to tell you more about their company, products and career opportunities. As an incentive for your time, they give away fun or useful stuff such as stickers, socks, t-shirts or other creative marketing items. It's fun to explore the various swag offerings from different companies, much like a treasure hunt.

At the Peltarion booth, ICML attendees could play the emotion detection game "Party Pi", get a tour of the Peltarion Platform as well as get their hands on a copy of our book The essential AI handbook for leaders.

There were however many other great booths ⁠— here are some of them which we thought stood out:

• Wadhwani AI was a standout among the booths. The nonprofit was founded in India as recently as February 2018 with the goal of doing AI for social good, has US$30 million funding and a goal to recruit between 30 and 50 researchers within the next two to three years. On top of being really friendly and discussing their interesting projects, they also handed out delicious samosas!
• The city of Montreal has taken on the AI revolution and used ICML as a recruiting opportunity to share job postings for local companies alongside fresh maple syrup popsicle giveaways.
• NVIDIA brought along a +1, their bot Maryam, to host the booth. Maryam is an autonomous robot using technology housed in her orange belly to map an environment, dodge unexpected obstacles and navigate different routes. People had a lot of fun trying to spot Maryam in the halls,  and when they succeeded, she rewarded them with candy and a photo opportunity.

Parties

If you haven’t been to a conference like ICML before, you may be unaware of how many other parallel events are going on besides the talks and workshops. After a long day of talks, it is really hard to absorb anything else and thankfully there are plenty of parties to attend. We enjoyed good food and refreshing drinks at our own party at Morfar Ginko, one of the coziest courtyard restaurants in Södermalm.

We also attended a number of other parties:

• Google AI hosted their ICML party at Tak (Swedish for roof), one of my (Lars) favorite bars in Stockholm. Lots of first-class researchers and an amazing view of the city made it a great party. Half of the guests were quite distracted though by the exciting World Cup game between England and Croatia that was being streamed one rooftop away.
• On Thursday, Stockholm AI and Google Cloud for Startups hosted a party where the local community and ICML participants got a chance to mingle. The event was beautifully located on the island of Skeppsholmen in the middle of Stockholm, within a stone's throw from the modern arts museum, Moderna Museet. The event began with presentations by Timothy Lillicrap (DeepMind), Vincent Dumoulin (Google Brain), and Anna Huang (Google Magenta). The talks were followed by plenty of food and drinks in classical conference party manner.
• Quantum Black hosted an evening with a panel discussion about the challenges of algorithmic fairness and cautioned that despite the rapid evolution of new definitions of fairness, all of these suffer from shortcomings that can easily lead to serious adverse consequences when used as an objective. The discussion ignited some interesting conversations and questions around the complexity and potential side effects of introducing mathematical fairness into machine learning  algorithms in different fields like finance, pharmaceutics and insurance. All the conversations were well capped with delicious Swedish “bibimbap.”
• To conclude the exciting AI-concentrated ICML week, all attendees were invited to the federated AI meetings joint reception in Skansen. What could be more appropriate than visiting the oldest open-air museum in the world right in the middle of Stockholm. Attendees enjoyed walking through the gardens, getting a quick glimpse of Swedish town life in the 19th century, enjoying fresh food and drinks and experiencing the true spirit of Swedish folk dancing. The combination of atmosphere, amazing views over Stockholm and the local scenery was a blast, and the researchers, engineers and other attendees were overwhelmed with the opportunity to learn some new dance steps.