The work in Artificial Intelligence (AI) has been a cornerstone of this decade — from computers that have been able to beat the top professionals at Go (a game previously thought to be completely impenetrable by computers), to playing some of the class Atari 2600 games to robots that can nimbly navigate rocky forest terrain.

A computer playing Go!

It’s becoming more and more pervasive in our day to day lives, as our own personal devices and computers are becoming smarter, more adept at helping us with our day-to-day tasks, and taking care of much of the noise that we previously had to deal with.

Meet our new robot overlords!

However, it turns out that most forms of artificial intelligence are actually pretty simplistic; for anyone afraid of actual robotic overlords consuming our planet — fear not. In fact, this blog post aims at highlighting some shortcomings of the state of the art AI systems, and how people are trying to overcome these. We’ll introduce some broad ideas in this post, and explore some specific results that scientists have found in later posts.

Right now, a lot of artificial intelligence systems, such as those that try to understand text (like in blog posts or essays), or those that try to understand images and visual stimuli have a very hard time keeping track of what they’ve seen in the past, or where to focus their processing power. If these systems could do either of these tasks, their power would increase vastly. This blog post will about these two things explicitly: attention and memory.


Think about you walking down a street. Your visual field, or everything that your eyes can see, has an incredible amount of detail and information packed in there. It’s almost overwhelming. Moreover, we can switch our focus between any number of objects, whether it’s someone else walking toward us, or a car zooming away, or even the drifting clouds in the sky. Seamlessly, we can focus our attention on certain things and disregard the rest of what we’re seeing, and even follow objects and their trajectories:

A computer “looking” at the various digits in order. [1]

This notion of attention is important because if we can endow computational systems with this capacity, we can potentially do much better at tasks like image classification (by focusing on a particular part of the image), text understanding (by learning where to search in a text to find an answer), and many more. There is a lot of exciting work in this area, and it’s looking like this year will introduce a lot of new advances!


Another critical part of our intelligence that we would like manifested in AI is the idea of memory. We store previous interactions, conversations, and information in our brain and we’re able to retrieve it quickly, whether it’s for an examination or a skill. In fact, our brain is so good at this information extraction and retrieval, and it seems like we can store not just factual information but also other structured representations, like a mental map of our surroundings when we’re driving.

NTMs: one way that computers can try to access and leverage memory! [2]

This idea is also important, especially when we ask computers to solve tasks that require looking back at previous inputs to figure out the answer right now. On first thought, this problem (and the one above), may not seem very difficult for a particular use case. However, the idea is that we want our methods to be general and work for various modalities of information.

The question of how to generally store information in a useful way is an open one, and how to properly retrieve this information and use it to solve a current task is also an unsolved problem. Computer scientists and researchers have been working  all over the world on this problem.


The ideas of memory and attention are quite closely linked — one can use attention to selectively focus on particular memories, and if the memories are stored in a meaningful, semantic manner, then it can augment AI in many ways and bring it much closer to the coveted human-level AI.

That’s it for this week! Stay tuned for a lot more exciting posts about machine learning and artificial intelligence, where we’ll actually explore and discuss some of these ideas in more depth.

— Surya