Fudong Chen
Abstract
This article attempts to answer the following questions: whether and how the recommendation system can recommend a topic-related content that has never appeared in the system. To figure out the question, the article gives a brief description of the recommendation system and concludes that without relative data, the system cannot recommend relative content. Then the article focuses on the external data of the app and deblackbox the digital fingerprint to show that it is possible to improve the recommendation system by tracking users and sharing data. Finally the article discusses the data privacy and expressed some concerns.
Introduction
The recommendation system of various apps based on machine learning and algorithms brings us a lot of convenience. Shopping apps recommend products we need, video apps recommend videos that attract us, and search engines guess what we want to search. The prediction or interference of our needs through machine learning and various algorithms is actually well understood, since they are based on the behavioral data created by ourselves. For example, I bought a science fiction novel, and the shopping app recommends other science fiction novels; another example, if I click on a cat-related video, the app will recommend more cat videos. Admittedly, based on the different recommendation system algorithm, there are different recommendation strategies, but most of the strategies are explainable and understandable from a human perspective. But in daily life, we may meet the following situations: we discussed a topic with friends (maybe on other apps, or even in reality). And the topic has never been discussed or searched on an app. but after a while, the topic-related advertisements or videos are recommended on the app. This coincidence naturally makes us question: can the recommendation system recommend a topic-related content that has never appeared in the system, or is our mobile app monitoring us all the time and extracting key words for recommendation? This article will try to answer this question. First, explain the composition and data sources of the recommendation system in general. Then, starting from data sources, explain how large Internet companies build user portraits in multiple dimensions, debalckbox the method of tracking users. And finally discuss the impact of mobile phone fingerprints (digital fingerprints) on data privacy.
How does the recommendation system works?
Before discussing whether the recommendation system can make recommendations as accurate as monitoring，we need to briefly describe how the recommendation system works. Simply put, the recommendation system is divided into three aspects: data, algorithm, and architecture. The data provides information and is the input of the recommendation system. The data contains user and content attributes information and user behavior and preference information, such as clicking on a certain type of video, purchasing a certain type of goods, etc. The algorithm provides the logic for processing the data, that is, how to process the data to get the desired output. Take the most commonly used algorithm in the recommendation system, Collaborative Filtering algorithm, as an example. Collaborative Filtering is based an assumption: if A and B have similar historical annotation patterns or behavior habits in some content, then they will have similar interests in content. It generally uses the nearest neighbor algorithm to calculate the distance between users by using the user’s historical preference information, and then uses the weighted product reviews of other user which is the nearest neighbor to predict the target user’s preference for a specific product. The system recommends products or content to target users based on the result. The architecture specifies how data flows and processes. It specifies the process of how data travels from the client to the storage unit (database) and then back to the client.
In other words, the recommendation system categorizes raw data and forms user portraits, attaches model tags or labels (ie patterns) to each user, and then recommends content based on various algorithms, such as the Collaborative Filter just mentioned.
Fig1, data processing
As fig1 shown above, the original data contains four aspects:
User data refers to the user’s personal information, such as gender, age, registration time, mobile phone model, etc.
Content data refers to the content provided by the app. Foe example, content data of shopping apps such as Taobao and Amazon are related with products and product reviews. Content data of video apps such as Tik Tok and Netflix are related with videos and video reviews.
User behavior logs refer to what the user did on the app, such as what videos they searched for, what videos they shared, or what product they purchased.
External data is data given by other apps. A single app can only collect a certain aspect of the user’s preferences data. For example, a video app can only describe what type of content user prefers in the video field. But if we integrate other different types of app data, the user’s data dimension will be greatly enriched.
The fact labels are cleaned based on the original data, including dynamic and static portraits:
Static portrait refers to the attributes of the user which are independent of the product scene, such as age and gender. Such information is relatively stable.
Dynamic portrait refers to the user’s behavior data on the app, and explicit (the behavior clearly expressed by the user) includes likes, sharing, etc. It is worth mentioning that if it is a comment, it is necessary to use NLP to determine whether the user is positive, negative or neutral. The implicit ones (the user does not clearly express their preferences) include the duration time the user watch video, clicks, etc.
Model labels are obtained through weighted calculation and cluster analysis through fact labels, which means weight for each dimension, and then calculate, and the users will be classified (cluster analysis) depended on the calculation.
In short, the recommendation system processes the data layer by layer by using various models and algorithms, and then returns the corresponding recommendation results. But in any case, the recommendation system cannot give recommendation results out of nothing. It needs to input various data, process the data according to algorithms designed by humans, and return the results according to certain logic. Therefore, for a single app, if we have not discussed the topic on the app (that is, there is no corresponding data for the recommendation system), it is reasonable that the app will not return the recommendation results of the related topic.
However, it can be seen in fig1 that the data source is not limited to the app itself. If there is corresponding external data, the recommendation system have the ability to recommend the content corresponding to the external data. In fact, technically speaking, large Internet companies such as Google, Alibaba and ByteDance, etc., usually have multiple apps in different fields, which can share user data and expand user portraits’ dimensions through user account information and digital fingerprints. Take Alibaba as an example. Ali’s apps include map, health, payment, video platform and even weibo, a social platform, so Ali’s portrait of Chinese users can cover many dimensions. It is worth mentioning that for different apps with common accounts, it is reasonable to directly match the account with the database. However, some Ali-owned apps, such as AutoNavi Maps, do not require users to log in to their accounts. Does Ali have a way to track this kind of users? The answer is yes. For users who use the app without logging in to a personal account, the app can identify or track users by the fingerprint of the smartphone.
How to track users?
Existing tracking mechanisms are usually based on either tagging or fingerprinting (Klein & Pinkas, n.d.). Tracking here are similar to the word recognize or identify mentioned above.
The typical tagging method is cookies. Cookies are data stored on the user’s local terminal. It is a small piece of text information sent by the server to the client browser and stored locally on the client as a basis for the server to identify the user’s identity status. Their main use is to remember helpful things like your account login info, or what items were in your online shopping cart (Cover Your Tracks, n.d.). But now, whether PC browser or mobile phone, there are many users who choose to delete or hide cookies, which leads to the poor effect of using cookies to identify users.
The typical fingerprint technology is Browser fingerprint technology. it is a concept proposed by Ecjersley in 2010 (Eckersley, n.d.), which means when a user uses a browser to access the server, the server get browser feature identification, canvas feature value, some hardware Information and system information, and generates a unique strings for the browser used by the user through a specific fingerprint generation algorithm. The accuracy of user identification technology based on browser fingerprints depends on the identification ability of browser fingerprints, and the identification ability of browser fingerprints depends on its degree of uncertainty. The higher the uncertainty, the higher its uniqueness. , The stronger the identification ability. For example, whether sharing cookies is a measurement of fingerprints. Some people are willing to share and the others are not. So if we know the measurement whether sharing the cookies, we can make sure which one the user belongs to. And if we have more measurements, users will be more likely to identify. From the source of measurements acquisition, they are divided into HTTP headers and JavaScript. HTTP headers means when connecting to a server, browsers send the user-agent, the desired language for a webpage, the type of encoding supported by the browser, among other headers. JavaScript is a programming language used to develop web pages. The server can obtain device information through JavaScript commands. For example, obtain the User-Agent through navigator.userAgent, and use commend of Intl.DateTimeFormat().resolvedOptions().timeZone to obtain the time zone. The following figure shows my fingerprint information on the website AmIUnique:
Figure 2, some measurements of fingerprints, source: https://amiunique.org/fp
All the measurements in Figure 2 are to find out the uniqueness of the user. It is worth mentioning the measurement of Canvas and WebGL. When drawing a 2D picture or 3D picture on different operating systems and different browsers including PC and mobile phone, the generated image content is actually not exactly the same, even if it looks the same to our eyes. So by extracting the picture information of Canvas and WebGL, we can uniquely identify and track the user.
Deblackbox the digital fingerprints
In the above, we talked about the measurements of browser fingerprints. In fact, digital fingerprints of mobile phone and browser fingerprints have many similarities, especially those related to JavaScript. Although different algorithms use different measurements to track mobile phones, these digital fingerprints models all follow a generic methodology which is shown below:
Figure 3, Generic methodology of digital fingerprints, source: (Baldini & Steri, 2017)
Meanwhile, we can also deblackbox digital fingerprint following the fingerprint recognition process of the browser.
Figure 4, Browser fingerprint recognition process
Looking at the two pictures together, digital fingerprint recognition is composed of 3 entities, namely the mobile phone on the client side (refers to Browser), the apps on the server side (refers to Website), and the database (SQL). In fact, for fingerprints of mobile phones, in addition to the above-mentioned measurements similar to browser fingerprints, such as device information, user configuration, etc., there are also many measurements about mobile phone components (hardware). But all the data needs to be digitized before proceeding to the next step. Therefore, for apps, digital information that can be directly obtained is usually used for identification.
When the user enters the app, the identification process of digital fingerprints begins. After the users access, App will send files such as html, css and JavaScript to the client, and usually the fingerprint collection script will be sent to the user together. The fingerprint collection script is defined by the app developer. For simple features, they can be obtained directly through API. For example, the user agent can directly use the userAgent property of the navigator object to obtain, and the screen resolution can be obtained through the width and height properties of the Screen object. The client (here means the phone) will send the fingerprint digital information to the app according to the script command. Note that because JavaScript and html do not require permission to run, users cannot perceive this process. The digital information will then be sent to the database, and be matched by Instance based algorithm and machine learning algorithm in the database. Instance based algorithm is often used in static fingerprint, which means the collected fingerprint feature values are converted into string form and spliced, and the spliced string is transformed into a fixed-length number through a hash algorithm. So if the number matches one of the instance in the database, then the user is identified. However, due to the frequency of feature value changes, the tracking time of static fingerprints for users is often very short. Most of the time, the company will use dynamic fingerprint and matches it by deep learning. In simple terms, the dynamic fingerprint compares each feature value of the fingerprint and sets a threshold. When the similarity of fingerprint to be matched and a fingerprint in the database reach the threshold, then confirm that the two match, otherwise insert the fingerprint to be matched into the database as a new one. There are many methods to generate threshold, such as statistical analysis methods, distance algorithms, random forest algorithms, LSTM algorithms and so on.
Back to the original question, when the user portraits of people are enriched, the portraits will not only include behavioral data, but also interpersonal relationship data and the data about relationship between you devices (PC, phone and so on) and accounts. For example, if you shared a shopping link to a friend a long time ago, your user portrait and your friend’s user portrait will be considered relevant, so when you discuss a topic with your friend, your friend may have left data on the topic online. The recommendation system based on the relationship between you and your friends, as well as other data such as location, coexisting in a local area network, etc. It is reasonable that after discussing the topic, the recommendation system will recommend the relevant content to your friend and also recommend it to you at the same time.
Discussion of data privacy and sharing personal data
Whether it is the opt-out privacy policy in the United States or the principle of informed consent represented by Europe, I think the key to data privacy lies in informed and optional. Like cookies that record user data, sharing cookies lets users enjoy convenience on the website and get a better experience; not sharing cookies will not lead to be unable to use the main services of the website. More importantly, the user has the right to choose whether to share cookies or not. Even if the website does not provide the option of not obtaining cookies, users can manually cancel sharing cookies through browser settings. But the emergence of digital fingerprints broke the principle of informed and optional. Now whether it is a website or an app, whether it is a PC or a mobile phone, companies can collect digital fingerprint information to identify users without the user’s perception. Secondly, for data share, the app usually provides a privacy policy statement before use. No matter which type of app, the topic of data sharing will be mentioned that the consent of data privacy statement means that the company is allowed to share the data in the company and its affiliates. If the user rejects the statement of privacy policy, he will not be able to use the services of the entire app. This actually deduces the user’s choice. Additionally, Some companies can let users turn off Ad personalization by themselves. Apps of Google and Ali all have this option. But this option does not guarantee that these companies will not collect your data. For example, app of Taobao clearly states: Service logs, device-related information, and device location information when you use the app will all be used for personalized recommendations. You can make decision independently on recommended content by turning off personalized recommendations (in my view, instead of refusing to be collected information). Take another example, Google’s privacy policy update of June 2015 indicates that they use “technologies to identify your browser or device (Privacy Policy – Privacy & Terms – Google, n.d.)” In fact, according to an interview with Bytedance employees, the above-mentioned information is classified as level-2 information, which means we cannot find a specific person in reality followed this kind of information. But it contains information like Consumer behavior, geographic location, browsing history and it can point to a specific account but not directly pointing to the owner of the account. After special approval, this kind of information can be shared with related company or different departments in the same company. In other words, the data we generate in an app and the user portraits generated therefrom may be used and analyzed by other apps of the same company. In addition, the combination of data sharing and tracking user technique also makes the app’s permission acquisition policy useless. For example, even if I forbid the shopping app to obtain the current location permission, it can still get the desired data through the map app. Take another example, the content I posted on social media can also be learned and analyzed by other apps, even if I do not log in to other apps with a social media account. In fact, when a user logs in to Tik Tok for the first time, which is also called cold start, the user may still be recommended to the accounts of classmates or friends he know in reality before he generates first bit of behavioral data in Tik Tok. This is brought about by track technique and data sharing. In addition, although the permission acquisition situation of apps is transparent, and the sensitive permission will needs to be confirmed by the user every time when it is used (for example, for the acquisition of microphone permission, app permission needs to ask for user consent, and a second confirmation is required when using microphone permission. This is also the reason why I think it is temporarily impossible to use mobile phone to monitor keywords for advertising recommendations), some mobile phone components that are considered not sensitive and do not require permission to use may also be used to violate privacy. According to Zheng, there is technology to eavesdrop part of the voice information of the mobile phone speaker through the accelerometer, a motion sensors of mobile phone (Zheng et al., n.d.).
Conclusion
In the article, we ask a question based on a daily phenomenon: whether the mobile app has the ability to make recommendations as accurate as monitoring. First of all, we introduce the basic composition and operation of the recommendation system, and concluded that the recommendation system cannot give recommendation results out of nothing. It needs to input various data and process the data according to the algorithm designed by humans. The result should be relative with the input data. From the perspective of data sources, we deblackbox the process of digital fingerprint and believe that the data sharing of apps in different fields and the user tracking technique can enrich user portraits and make accurate recommendations. Finally, the article expresses the concerns about the impact of digital fingerprint on data privacy, and considers that data privacy in the mobile phone field needs more research and corresponding restrictive measures.
References
Baldini, G., & Steri, G. (2017). A Survey of Techniques for the Identification of Mobile Phones Using the Physical Fingerprints of the Built-In Components. 19(3), 29.
Eckersley, P. (n.d.). How Unique Is Your Web Browser? 19.
Klein, A., & Pinkas, B. (n.d.). DNS Cache-Based User Tracking. 15.
Laperdrix, P., Rudametkin, W., & Baudry, B. (n.d.). Beauty and the Beast: Diverting modern web browsers to build unique browser fingerprints. 18.
Privacy Policy – Privacy & Terms – Google. (n.d.). Retrieved May 13, 2021, from https://www.google.com/policies/privacy/archive/20150501-20150605/
Zheng, T., Zhang, X., Qin, Z., Li, B., Liu, X., & Ren, K. (n.d.). Learning-based Practical Smartphone Eavesdropping with Built-in Accelerometer. 18.
Cover Your Tracks. (n.d.). Retrieved May 13, 2021, from https://coveryourtracks.eff.org/learn
Anand, S. A., & Saxena, N. (n.d.). Speechless: Analyzing the Threat to Speech Privacy from Smartphone Motion Sensors. 18.
FP-STALKER: Tracking Browser Fingerprint Evolutions. (n.d.). 14.
Das, A., Borisov, N., & Chou, E. (n.d.). Every Move You Make: Exploring Practical Issues in Smartphone Motion Sensor Fingerprinting and Countermeasures. 21.
Hauk, C. (2021, January 14). Browser Fingerprinting: What Is It and What Should You Do About It? Pixel Privacy. https://pixelprivacy.com/resources/browser-fingerprinting/