Image recognition has been around for decades, but usually when we communicate with AI, our preferred medium is words, whether spoken or written. Some apps and tools are changing that, including Quickdraw, an experimental game where a neural network tries to guess what you’re drawing. Although it may seem like simple fun, this game is not only an experimental project, but it also is a useful tool for understanding the relationship between symbols and language.
How Quickdraw Unintentionally Explores Semiotics
In Quickdraw, you are given a prompt and 20 seconds to draw what is usually a common item or concept. The neural network sometimes guesses it within seconds, but if you don’t envision the item similarly to the people who’ve drawn it in the app in the past, the network may not be able to guess what it is at all. Probably the most fascinating feature is that after you’ve drawn a set of six images, the game allows you to view past images that the neural network used to connect the image and word, as well as what the neural network was comparing your drawing to when it guessed correctly.
These doodles are typically what we might think of as symbols, while we tend to think of words as something completely different, but this game provides an example of how closely an image and a word can be linked. Words themselves are symbols—An alphabet is a group of symbols that can be arranged to represent certain sounds, which we have collectively decided represents a concept, whether it be concrete or abstract.
Semiotics, also called semiology, is the study of symbols and broader concepts such as signification, which is the idea that one thing stands in for something else. Semiotics can often sound very academic, theoretical, and detached from real-world and practical experiences, but the reality is that we use semiotic concepts every day, not just when using direct visual representations like the doodles in Quickdraw, but every time we speak or write something down. Natural language processing often focuses only on the practice of language rather than the theory behind it, and understanding deeper linguistic theories such as semiotics could help us improve NLP processes and really understand what we are analyzing when looking at qualitative data.
How is a word a symbol?
When we write down a word, it is just a couple of markings on paper, but we don’t think of that reality because we have assigned it a meaning. The word “Dog” as I have just typed it out is a collection of pixels on a screen, but the shape of those pixels has a predetermined meaning, causing you to think of a dog in your mind when you read those words.
These ideas come into play in the famous painting, The Treachery of Images by Belgian artist René Magritte, created in 1929. The image displays a picture of a pipe along with the words "this is not a pipe." If we aren’t thinking about language in an abstract way, this picture is quite confusing, because it clearly displays a pipe, but the text says that it does not. The answer is that it does not display a literal pipe, but a representation, or symbol, of a pipe. This artwork makes us stop and really consider what we do every day when we speak or see a painting. Our minds are incredibly complex and have developed complex shortcuts so we can understand each other.
Semiotics
In semiotics, the pipe is called a signifier, and a real, physical pipe would be the signified. The word is an arbitrary: it only represents what it represents because we have all agreed to it. This kind of thinking can help us understand linguistic shifts that change the meaning of words. A great example of this is the word “tablet.” Does it mean an ancient block of clay, perhaps containing one of the earliest samples of modern writing, or a high-tech, touch-screen device? At some point, we decided that this word could refer to both, and at that point the context became more important than ever to understanding what a tablet is.
Quickdraw
On Quickdraw, one of the reasons the neural network may struggle to identify a simple doodle is because the player envisions the item differently from previous players, so the neural network can’t match the two symbols. In the following screenshot, we can clearly tell that the reason why the neural network didn’t recognize the previous doodle of a baseball was because the player didn’t draw the usual stitch pattern that is often seen on baseballs. In the future, the neural network may be better able to identify the baseball even without this iconic pattern—Its understanding of the symbol is widening, just like words take on new meanings.
The Future of Symbols in Textual Analysis
With the rise of visual media and improved image recognition, it is possible that in the future qualitative analysis processes could become better at understanding the meaning of images included within a text. For example, currently in NLP although emojis are sometimes deleted, analysts can also have them transformed into words to preserve meaning. It is possible that these same processes could someday be applied to doodles like those in Quickdraw, capturing a broader range of nuance. While it may feel like those symbols are not “really” text, the line between text and doodles is less defined than we may realize. As every day analysts create new innovations, it is possible that the future will bring textual analysis that is almost unrecognizable from what we have today in its breadth and variation, including both words and images alike to produce an even wider range of analysis than what we already witness.
Comentarios