How Your Smartphone Is Listening to You

The smartphones most of us carry around with us contain just about every sensor imaginable. They know where we are; they have cameras to see us with and microphones to hear us; they have biometric sensors to learn our fingerprints and map the unique contours of our faces. All of these are connected to a compact but powerful microcomputer, upon which is installed any number of third-party applications, some of which know things like our bank account numbers and balances, our complete address book, our password managers, our exercise and diet apps with our heart rates and blood pressure, our prescription drug histories, and our email histories. And all of them have access to the Internet, potentially transmitting any or all of that data to anyone in the world. Potentially. Obviously there are many security roadblocks in the way attempting to keep the lid on much of this data, but still, it is not paranoid to wonder about all the myriad necessary holes in the dam through which data does flow. This is exacerbated by our phones' tendencies to show us targeted ads that indicate websites know much more about us than they evidently should. In recent years it has become almost a de facto concern: Are our smartphones listening to our conversations to learn our interests and show us targeted ads?

Facebook is the poster child for this behavior. Facebook, many people believe, is the app that uses your phone's microphone to eavesdrop on your conversations. However it collects information on our interests, it has a lot of it. But they're certainly not alone. Google probably collects even more data than Facebook does, and virtually every other major Internet site collects it as well. The basic mechanism is that when you view any web page, the site owner — and the owners of any sites with any code, images, widgets, ads, or other assets on that page — can recognize who you are, where you're currently located, the content of the page you're looking at and how you got to it; and that's all added to the giant databases out there that already know things like who your friends are and your basic habits like where you work and what you do and what you've bought and what people you've been around recently. All this lets websites advertise to you, with ads optimized for where you are now, the interests of people you might buy gifts for, things that are liked by people with profiles similar to yours, what kinds of news and political articles get you fired up and make you likely to share, and a million other things. Such optimized ads drive engagement and purchases, and that's what makes this nice free Internet we all enjoy able to exist, and to make it extremely profitable.

If you go visit your cousin in another state, and your cousin is into antique cars, you can expect to see ads for stuff pertaining to antique cars. There was no need for your phone to listen in on your conversations about crank-starting his 1907 Cadillac Model M; Facebook could tell that you traveled to a location where antique car stuff was frequently searched on the Internet, and made a reasonable conclusion that it's an interest of yours.

And after your left, when your cousin's birthday comes around, his friends and family all get deluged with ads about antique car stuff. It's not because their phones heard them say "Hey, Ezra's birthday's coming up, you know the guy who's into antique cars," it's simply because they're all Facebook friends and Facebook knows his birthday.

You might go to a restaurant that has a big fancy absinthe fountain, fancy enough that you and your date likely commented on it. Tomorrow's absinthe ad on Facebook doesn't mean your phone eavesdropped; it means you went to a place where people searched for information about absinthe incrementally more than they do at other places.

Here's a specific example of how companies like Facebook achieve this. At Skeptoid, we will sometimes have events or special fundraisers, and we might promote those things on Facebook where we have a large following. We want to know how well those campaigns are doing; if we pay Facebook $100 to promote our event, we want to know if we make that back from the visitors who see our ad on Facebook and come to Skeptoid.com to donate. To get this data, Facebook needs us to place a tiny snippet of code on the landing page of the website — and that code detecting each web visitor is how they know who everyone is, where they are, what web page they're looking at. But it's not just Facebook doing this.

Anytime you're on a web article that has share buttons or Like buttons, like a Twitter button, a Facebook button — or LinkedIn, Tumblr, Pinterest, Snapchat, Telegram, a thousand others — each of those companies has a snippet of code on that site doing essentially the same thing. Free tools like Google Analytics exist not to magnanimously give webmasters free statistics about visitors to their website, they exist to get webmasters to place a snippet of code from Google on every page of their site, giving Google all of this exact same information about every visitor. Amazon's affiliate program pays commissions to website owners if they place a little ad for an Amazon product on their site. It's worth it for them to pay this commission because a thousand people visit that website without buying anything — and the Amazon code underlying that affiliate ad gives Amazon all this same information about every visitor to that page. There is virtually no useful website that you can visit that is not recognizing you and remembering where you are and what you're doing right now, and adding that to the massive cloud of data about your online behavior.

Companies like Google are happy to pay website owners to show ads, and merchants like Amazon or eBay are happy to make affiliate payouts to website owners — because those commissions they pay are a drop in the bucket compared to the revenue they make from targeted advertising. YouTube is delighted to spend money on bandwidth showing you cat videos all day long, because they're collecting data on every visitor and every video they watch. If you think Facebook is somehow unique in their efforts to serve targeted ads, well, let's just say "your information is less than complete."

In fact, Google has tracking on about three times as many web pages as Facebook. "So," you might conclude, "I just won't use Google anymore. I'll use DuckDuckGo," or some other search engine that does no tracking. So what. That only hides your web searches from Google. It doesn't hide any of the web pages you visit or the things you buy; and it makes no difference whether you have a Gmail account or a Google account of any kind. Due to the depth of its penetration into websites alone, Google knows about something like three quarters of all credit card transactions — like that unusual thing your friend bought, and you had a conversation about, and you saw an ad for and decided the phone was eavesdropping — forgetting that Google and Facebook and everyone else all already knew that you're friends.

You may also decide to close your Facebook account. So what. As long as you still visit pages that have Like or Share buttons, or invisible marketing campaign tracking code, Facebook is still getting just as much data on you as ever. And again, it's not just Facebook. It's everyone.

One thing I will assure you is that the number of ways Facebook can track you that I found out about in my research was a lot more than I was able to edit down into a Skeptoid-length show. And something I would bet a billion dollars on is that the number of ways they track you that I was able to find out about is a tiny fraction of what they can actually do. Getting your phone to listen 24 hours a day and decipher who's talking about what would be a computationally intense process that would drain your phone's battery in half an hour, and it couldn't possibly be less necessary to them. They have so many easier ways to learn so much more, so much more reliably. But just because that's true doesn't prove that it's not happening.

So, let's look instead for actual proof that it's either happening or not. First, let's be clear on one point. When we say "your phone is listening to your conversations," we need to understand what's meant by that. Modern phones all have a basic, and very important, security feature: and that's that you must explicitly give permission to any application to allow it to access your phone's microphone. And there is a very important distinction to understand about this. You know how you can click in a text field, to post an update to social media, for example. Then you can click the microphone button to use speech to text, allowing you to speak your update so you don't have to type it. This is a function of your phone's operating system, just like the keyboard. It's not the social media app that heard you speak; it's only the operating system. It performed the speech to text conversion and then sent the text to the social media app. It is not necessary to give permission to the social media app to access your microphone to use speech to text. So the ability to use speech to text in an app does not mean you've given it microphone access, and if you haven't, there is no Earthly way that app can overhear your conversations.

Lots of journalists have interviewed former Facebook employees (including former executives) about this question, and they've all said that Facebook doesn't do that. Mark Zuckerberg has been hauled in front of Congress a number of times and has always said under oath that Facebook doesn't do it. But they could all be lying; a monolithic conspiracy of thousands of engineers with never a whistleblower. So researchers have done tests to find out. In one such study by computer security firm Wandera, an iPhone and an Android with similar configurations had all the app permissions enabled and then were placed in a room for three days, and exposed to a 30-minute loop of pet food ads. Control phones were placed in a quiet room for three days. No pet food ads showed up in either phone's Facebook apps afterward, but the crucial finding was that battery usage was the same on both sets of phones, meaning no extra processing of audio was being done on the test phones; and data usage by both sets of phones was the same, meaning the test phones weren't uploading any voice recordings to be processed elsewhere. It was one test, but it was pretty conclusive.

Nobody at Facebook wants to go to jail, yet they publish a statement that says "Facebook does not use your phone's microphone to inform ads or to change what you see in News Feed." If that were untrue, the Federal Trade Commission would be hauling people off in paddy wagons like it was St. Patrick's Day.

It's true that nearly everyone has an anecdote about seeing an ad that they're absolutely certain couldn't have come from anywhere else but eavesdropping. You spoke once in your life about alpaca undercoat brushes and then saw an ad for them? It's likely a few of these are coincidences, but in most cases, something made you talk about alpaca undercoat brushes. Did you see alpacas on a TV show? Keep in mind Hulu and Netflix are part of this game too. I spent the whole week I was researching this episode speaking "alpaca undercoat brushes" at my Facebook app, and told nobody; still no ads for anything like that.

So, let's come to a conclusion; the data and the circumstantial evidence all support only one. Facebook and most other major Internet service providers are absolutely all spying on you, via many, many methods; but these do not include the least efficient of all imaginable means: unauthorized and illegal listening through your phone's microphone.

By Brian Dunning
Follow @BrianDunning