Technology
Jason Grow for Al Jazeera America
Editor's Note: This is the fourth installment of The Decoder, a column that's part of the Living with Data series exploring how our online data is tracked, collected and used. Do you have questions about how your personal data is being used? Curious to learn more about your daily encounters with algorithms? Email The Decoder at thedecoder@aljazeera.net or submit your question via the form here. Screen shots and links are helpful clues!
Jean Yang went on a big trip through Europe this summer, from Edinburgh, Scotland, to Dubrovnik, Croatia, to Oslo, Norway, and back. Like a good tourist, she took pictures on her phone, an Android, throughout her trip. When she returned home, she found a surprise package in her Google+ notifications: a neatly collated, summarized, annotated digital scrapbook titled “Trip.”
Jean shared the album with me with this message:
This is equally cool and creepy: Google made this scrapbook of my June travels based on a random selection of photos I took — and also its knowledge of where I was. It’s kind of nice to have this information organized automatically, but this is really trusting them with a lot of information. It would be funny if they took quotes from emails I sent during this time and put in quotes relevant to the places. “Oslo is so expensive! My second dinner of wonton soup cost 68 kroner.” I’m curious how they decide what to include. — Jean Yang, Cambridge, Massachusetts
When I spoke with Jean later, she was surprised in part because she didn’t know this feature existed. She was also a little taken aback by all the location information included, given that she hadn’t been using her roaming phone plan or data while abroad.
So how did Google pull this together? And why did it leave Jean with mixed feelings?
We looked into the program. Google introduced this scrapbooking feature in May, just before Jean’s trip. The company calls it Stories: “Your best photos are automatically chosen and arranged in a fun timeline to show the highlights of your trip or event.” There’s an example scrapbook here.
Jean guessed that most of her location information was sourced from when she connected to Wi-Fi. She wondered if some of the details were coming from her travel details records in her email, like the way flights show up in Google Now. She noted that Android phones are “very smart,” since they start to see regular patterns and deduce where home and work locations are over time. She suspects that when you change your patterns and locations, it triggers the attention of Google’s algorithmic curators.
Her instincts were right. Google draws on a lot of data points to try to make sense of the photos you took. They use geotagging from photos, location information from Google Now or Maps and GPS data. They also identify locations using machine vision to match key landmarks. Even though Jean’s phone was mostly off grid for the trip, Google was able to pull together enough information about the photos to organize it in a location timeline.
Google suggests that Stories works best when all the defaults are in place: your photos are geotagged, you back up images to Google+, let Auto Awesome work it’s secret-sauce magic and take a whole lot of pictures.
This isn’t the first time Jean noticed Google doing funny things with her photos. She mentioned she has seen Google animate sets of images when she takes a lot of pictures in a row. She described one that involved falling snow to match the scenery. When she takes 10 images in a row, she’s not trying to create a GIF; she’s just trying to get things to look right.
Auto Awesome — falling snowflakes and all — is on by default for Android devices. It’s Google’s automated image and video editing algorithm. It’s pretty cool, but it can be pretty finicky. Google teases, “You might get copies of them made into things like short animations or wide-screen panoramas.” Google then notifies users once a picture is made “awesome.” The images are private until you decide to share them.
Apple has similarly started to organize photos in iOS devices by location and time stamp as Collections and Moments and offering smart editing tools. And it’s not so different from what Facebook introduced with Look Back, which summarizes important anniversaries.
For Jean, the auto-magically curated packages feels different from a helpfully organized, glanceable summary. It feels a little more like when your mom cleans your room for you. And in fact, Google likens sifting through your 437 photos to taking care of your piles of dirty laundry after you return home.
Jean admitted the feature was useful, if a little creepy. She likes to take notes about the places she visits on her travels, especially to share recommendations with friends later. When Google captures those automatically, it could be pretty useful.
Google wanted to develop a personal news feed of moments from all the data it has about our activities, and it ended up building a “personal biographer” into every smartphone. Google, of course, has a lot of data about what we’re doing, and it wanted to turn it into something useful by solving a specific problem — in this case, taking a lot of pictures and not doing much else with them. Stories fills a hole in apps that help document adventures. Alexis Madrigal offers an interesting look into technical challenge of turning data into stories like this.
By Google’s count, Jean’s trip consisted of 24 days and 117 “moments.” But how does Google determine which images are the best? And why did it choose an image with the most faces and therefore the most friends as the cover image for Jean’s “Trip?”
Auto Awesome elevates “best” photos based on exposure settings, cutting out duplicates and prioritizing clarity and sharpness. It also features people and landmarks.
Auto Awesome also takes multiple pictures and captures everyone smiling, essentially doing the Photoshopping for you. Sure, it may be great for capturing a group shot, but did that moment ever really happen? Reminds me of some deep philosophical questions about the nature of photography…
Of course, Stories’ scrapbooking algorithm isn’t the be-all, end-all of photo editing and curation. You can go in and edit a Story to add captions or change locations. You can also add pictures from other cameras.
So what’s in it for Google? Jean found this album in her Google+ notifications, so it might be another way to keep us coming back for more. For now, it’s just a summary of your trip, but we have to remember Google is at heart an advertising company. How soon before the sponsored restaurant details show up in the Stories that you share with friends?
Do you have questions about how your personal data is being used? Curious to learn more about your daily encounters with algorithms? Email The Decoder at thedecoder@aljazeera.net or submit your question via the form here. Screen shots and links are helpful clues!
Error
Sorry, your comment was not saved due to a technical problem. Please try again later or using a different browser.