Author Archives: Annaliese Blank

The Real Siri: Past, Present, and Future.

Annaliese Blank

Professor Irvine

CCTP: 607-01

3 May 2019

The Real Siri Unpacked: Past, Present, and Future

Abstract

In today’s pro-tech world, virtual assistants are becoming highly prevalent for at home technology. One assistant in particular, Siri, has transformed the structure and capability of cell phone services for Apple Products and acts as a third party to all of your needs at the touch of a button. The rise and reputation of this technology has inspired further investigation for this research topic. Based off this, a research question to be looked into further would be, ‘Does this type of technology, Siri in particular, pose a serious threat to your personal information, privacy, and overall data ethics’? Some main points to hypothesize would be Siri is a virtual assistant that is always listening to its ‘owner’ or ‘master voice’ and Siri technology is invasive for personal data ownership, usage, and distribution to Apple INC.

One main approach to understanding and answering this question would be to use a socio-historic analysis of Siri in terms of how it evolved, what was the initial goal and purpose, pro’s and con’s to its advancements, and where the technology is headed in terms of re-modeling and further implications for the future. For the analysis, being inclusive as possible, I aim to implement current journal articles on this technology, websites on the history, community blogs for outside opinions, and previous personal blog post insight from our course on virtual assistant technology in general. Having content to unpack from various platforms will be informative and exclusively educational towards de-black-boxing Siri and the future Siri is creating for the world. 

Introduction

In response to my research question, a common reply would be: why does this matter or why should we care? The answer to this question is simple: your privacy is entirely at risk when virtual assistants are being used on a regular basis. For example, if a person is using Siri within their apple device, it acts as a virtual contract that is able to listen and access information about you which is then stored by Apple to benefit their brand and grant ownership to your data. As most would agree, if more research was conducted on the purpose, function, ethics, and effects of Siri, I hypothesize there would be an increased desire for further legal revisions or privacy regulations for the future.

In this paper, I address the origins of Siri, how it was refined, projected in the media commercials, the mechanics to perform its operations, algorithmic functions, cloud connectivity, natural language processing, understanding the connection to the deep neural network, and voice control wake word controversy as the technology continually updates in newer versions. All of these features were dense to de-black-box but more can be understood when we combine all of these components in a socio-technical lens with the combination of its history and how it all came together. Future implications are necessary if we do not want virtual assistants to slowly start governing and controlling our lives in terms of day to day tasks. From the literature I gathered, it is evident that other researchers would agree with my claims and provide support that this technology would indeed pose a serious threat to our personal data, usage, and listening when we are not fully aware of it. It poses an unethical standpoint on its purpose and integration in our lives. Further legal revisions and implications would need to be implemented if we chose to not let artificial intelligence replace human discourse and interface. 

 

Literature Review

When it comes to Apple products, most people would assume Apple masterminds are behind patenting their technology and features independently. But for Siri, this was actually not the case. The idea of Siri technology dates back to the early 1980’s and was fully implemented in Apple near 2003, and wasn’t fully installed until their recent iPhone 6S model which was released in 2013-2014 (Cult of Mac, 2018, pg. 1). During this time, the 80’s were all about technology, innovation, and industrial work. In the 1980’s it was ironically mentioned as a neat feature they would eventually like to acquire or wish they were able to (Cult of Mac, 2018, pg. 1). Another unknown fact was when this idea became rolling, Steve Jobs put most of his final efforts into the Siri feature before his passing, which Apple holds most-high in his honor. Some say this was his final gift in really improving the way cell phones were intended to be used. 

As technology was advancing, so was our military and national security. Near 2003, The Defense Advanced Research Projects Agency, formally known as DARPA, “began working on an AI assistant project that would help military commanders deal with the overwhelming amount of data they received on a daily basis” (Cult of Mac, 2018, pg. 1). In order for this pro-type idea to see if it was possible, the DARPA reached out the “Stanford Research Institute for further input and testimonial research” (Cult of Mac, 2018, pg. 1). The SRI decided to jump onboard with their proposal and test further ideas to see what else this platform would be able to handle. This is where the classic ‘Siri’ name is rooted from. Many people do not know this is how and where the technology originated. “The SRI decided to create a spin-off called Siri, a phonetic version of their company name, and was launched in the app store in 2010.” (Cult of Mac, 2018, pg. 1). The most interesting part to this history of Siri was it was developed as its own separate app and unattached to anything else. It was designed originally to tackle multiple things such as, “Order taxis through Taxi Magic (which was the original form of Uber in 2010), pull concert data from StubHub, movie reviews from Rotten Tomatoes, or restaurant reviews from Yelp” (Cult of Mac, 2018, pg. 1).

As impressive as this prototype was, Apple decided to buy out the app and partner with the SRI for a “200 million-dollar deal” (Cult of Mac, 2018, pg. 1). that would soon change the game for Apple iPhones forever. According to Apple INC AI specialist, Scott Forstall, “Apple decided to partner with the same groups SRI did, including Wolfram Alpha, Wikipedia, Yelp, Weather Apps, Yahoo, NASDAQ, Dow, and local news stations for traffic and current time zones, and many more. Scott makes it very clear that Apple wanted this feature to be as accurate, fast, and hands-free as possible, also with a virtual voice that was friendly, respectful, and reliable” (Cult of Mac, 2018, pg. 1). In doing this, it pro-creates a positive Apple user experience and makes the user feel confident in the phone’s capability. This was their initial goal and they prioritized making this the most innovative phone technology to date. 

During this acquisition process, Apple’s first version of Siri once purchased was unable to originally speak back to the user. It was originally used to provide answers through the other brands it partnered with to provide quick help or feedback on a question. In order to improve this, what’s called as Natural Language Processing, or NLP, was implemented in the newer version of Siri to allow the verbal connection to the technology and have Siri fully understand the words, connotation, voice pitch, voice que’s, and pronunciation of what the user is saying or asking itself. In other words, Siri will be able to understand what the user is saying, and voice the correct response back in the correct style, language, and structure. This was the ideal model for the iPhone 4S that was able to perform Speech Recognition and received a large amount of media attention when it was released. Further Speech Recognition ideology will be explained later below when the mechanics are unpacked. 

The first version of Siri in the iPhone 4S provided sample questions in their first commercial showing real world and hands-free scenarios asking tasks to be done such as: 

‘Can you reschedule my meeting to 5pm, What is the weather in New York looking like for this weekend, Call Mom, Text back I will be there soon, What time is it in Paris, What is the currency for 60 Euros in dollars, etc’. Types of questions like these require the applications they have partnered with to provide the fastest and most accurate responses in a matter of seconds, such as Wolfram Alpha, Yelp, Wiki, and many more. The reaction to this new feature was so positive and popular, Apple then created this feature accessible on their other devices such as their computer, Mac and now iPad technology (Cult of Mac, 2018, pg. 1). Siri is the fast, accessible medium between the user and all of the other apps it has partnered with to be the first platform necessary to answer these types of questions or tasks. 

As this 200 million-dollar deal has progressed, many capabilities to this component have now been researched further in terms of its’ NLP processing, data collection, and personalization functions in newer iPhones today. As every year passes, Apple is known for bettering each device they produce by making faster and smarter improvements in their mechanics and AI functions in their products. For their iPhones, this is priority since their phone models have dominated the cell phone industry since the first Siri feature. In recent development, the progression of Siri has now become gendered, more accurate, and geographically diverse. 

As Siri continued to progress in each newer model of the iPhone, the voice responding back to the user eventually became more noticeably female rather than male. This is very controversial for virtual assistants in general since they are artificial it can be difficult to create a voice that isn’t one gender or the other. This feature did receive some backlash in terms of a female voice in earlier models, but Apple has now created a feature to Siri that doesn’t have to be a She; it can be a He or an It. The problem with this dates simply back to earlier stages of 1950’s gender roles and norms when women were “Ready to answer serious inquiries and deflect ridiculous ones. Though they lack bodies, they embody what we think of when we picture a personal assistant: a component, efficient, and reliable woman. She gets you to meetings on time with reminders and directions, serves up a reading material for commute, and delivers relevant information on the way, like weather and traffic, etc.” (The Real Reason, 2018, pg.1). Apple released a statement in 2013 now saying “both options are available for voice preference” (The Real Reason, 2018, pg.1). Small changes like this allow Apple to create room for improvement to present Siri as the best virtual assistant on the market that is not only smart technology but is customizable per each user. Such categories would be voice volume, gender voice, accent, and notification preferences. 

As if Siri cannot be any more of a personal experience and technology, Apple’s newer feature of cloud computing and cloud capability has transformed Siri even further than before. The feature of Cloud Computing was originally released in 2011 (Apple Privacy, 2019, pg.1). Apple’s Press Release Statement thus follows, “Apple today introduced iCloud as a breakthrough set of free new cloud services that work seamlessly with applications on your iPhone, iPad, iPod touch, Mac, or PC to automatically and wirelessly store your content in iCloud and automatically and wirelessly push it to all your devices. When anything changes on one of your devices, all of your devices are wirelessly instantly” (Apple Privacy, 2019, pg.1).

Some operations of the cloud include, “cloud computing, cloud storage, cloud backups, or access to photos, documents, files, contacts, reminders, music, etc.” (Apple Privacy, 2019, pg.1). In terms to Siri, it is able to perform NLP through cloud computing and virtually store your data the more you use the device. As stated in the press release, the key here is virtual, wireless, automatic service which is easily accessible via the Siri component. Some tasks to ask Siri on newer models of the iPhone would be, “Siri, can you save my email in the cloud, can you add my song to my playlist, or can you save these documents in my work folder, Send this to the cloud, etc.” 

A common question in response to this would be: How does the architecture of NLP and Cloud Computing work? In order for Siri to be used correctly, it first must be used from your device, have wireless connection to use other platforms, and access to the cloud feature in order to store your data to process the important information get to know the user better. When this happens, your data becomes personalized, which is then stored away virtually. The cloud component is needed for your user profile to be understood and processed. 

The next question would be: How does Siri actually listen to the user in order to then be able to function with the Cloud? What are the mechanics that make all of this possible? The answer to this question is complex, but it can be de-black-boxed. From a historic standpoint, Apple improved this technology by allowing it to perform speech recognition using speech patterns and sound waves which can be computed and understood through NLP and Cloud computing and then sent back to the user. 

There is a constant signal sent back and forth in order for Siri to hear, understand, save, and respond to you. All of this is done in seconds; sometimes milliseconds, depending on the complexity of your request. As mentioned previously, a hands-free experience is priority, so when we de-black-box cloud computing, the required mechanics for Siri include the ability to perform text-to-speech and speech-to-text recognition and access to the DNN. All of this is done through layers of the deep neural network which is explained in the next step below.

 

From the design standpoint, there are many designs and layers to Siri that must be understood. Once you have asked Siri a question with the button, or asked “Hey Siri”, there are signals being sent via cloud computing and the deep neural network that record your questions and determine the correct answer, which is then recorded in text, and presented back to the user by voice. To make this as clear and simple as possible, according to Apple’s own Siri Team site, they said, “The ‘Hey Siri’ feature allows users to invoke Siri hands-free. A very small speech recognizer runs all the time and listens for just those two words. When it detects “Hey Siri”, the rest of Siri parses the following speech as a command or query. The “Hey Siri” detector uses a Deep Neural Network (DNN) to convert the acoustic pattern of your voice at each instant into a probability distribution over speech sounds. It then uses a temporal integration process to compute a confidence score that the phrase you uttered was “Hey Siri”. If the score is high enough, Siri wakes up” (Apple Hey Siri, 2019, pg.1). Here is a picture below to visualize the layers and further understand where the speech waves travel.

For newer updates of virtual assistants to fulfill the hands-free experience, the wake word is required. For Siri, as mentioned before, it is now in newer models known as, ‘Hey Siri’. Siri must be turned on in the settings of your phone in order to be always listening and awaiting your attempt to wake it with the wake word. To address the second part of this question, the mechanics that make all of this understood and possible happens within what’s called, Speech Synthesis. This is a very interesting layer to Siri. Speech synthesis is the layer that is able to then understand and voice back to the user the proper response once the initial question was heard, understood, and processed through NLP and Cloud Computing (Apple Siri Voices, 2019, pg.1). 

According to Apple’s Siri Team site, they say, “Starting in iOS 10 and continuing with new features in iOS 11, we base Siri voices on deep learning. The resulting voices are the more natural, smoother, and allow Siri’s personality to shine though” (Apple Hey Siri, 2019, pg.1). In the picture below, provides a clear representation of how text-to-speech synthesis looks, and operates. Starting from the left, text is used as the input, text analysis occurs, then followed by the prosody model, which deals with rhythm, and then signal processing begins with unit selection and wave from concatenation, which deals with the sequence of chain of code deliverable back in speech form. To be clear, this is where the predictive feature can be further explained once it goes through each of these units in the model (Apple Hey Siri, 2019, pg.1).

 

This is great example of how the user is able to speak to Siri, and how Siri is able to respond to the user and get to know them through this process of machine learning and deep neural networks with NLP discourse. All of this virtual assistant process is entirely possible with help from NLP, the deep neural network, speech recognition ability, machine learning and algorithmic implementation, and speech synthesis, and many more complex features.  (insert other pic here)

Literature Continued: Pro’s and Con’s 

With the general socio-technical history, mechanics, and layers to Siri understood, this is where my original research question initially began since Siri is posed as complex and positive technology, but one must question what are the negatives within the positives? With progress in any form of technology, there are always some drawbacks since nothing is deemed perfect. The entire process of Siri is non-visible and a lot is going on that most users are not aware of. With this, it is important to lay out the framework for unpacking the pro’s vs. the con’s to Siri. Another question to also address would be, what other things can Siri do? 

Some pro’s to Siri technology would be, “Siri can act as a personal scribe, she can write texts for you, post to your social media accounts, solve complex math equations, finding emails, and converting measurements, even Morse code” (UK Norton, 2018, pg.1). Other things include, “booking your evening out for you with certain apps, like food apps, or Yelp for food reviews. Siri can also be used for Open-Table and automatically book your reservation” (UK Norton, 2018, pg.1). As mentioned before, in the newer software updates such as iOS 8 or iOS 9, the “Hey Siri feature must be turned on and you can accomplish any task with Siri” if you start your interaction with Hey Siri as the wake word (UK Norton, 2018, pg.1). 

Some con’s to Siri technology would be, “Siri has listening problems, is always listening to you if turned on, or if your Wi-Fi dies, Siri dies with it” (UK Norton, 2018, pg.1). When we say listening problems, sometimes if your question is too complex, Siri might not be able to fully understand you or the answer you need. If the pitch, tone, or acoustics of your ‘master voice’ are off, it can also be difficult for Siri to hear you properly or register that it is still your voice. Some common replies Siri can re-iterate back to you would be, “I’m sorry, I don’t understand you, I don’t know how to respond to that, or Can you repeat your question?” (UK Norton, 2018, pg.1). In terms of Wi-Fi connectivity, which is highly important in order for Siri to operate, this connection gives Siri the power to access the sub-platforms that are already installed within her mechanics. Without Wi-Fi, Siri isn’t able to access the Apple Server to store and collect your data or reach the networks needed in order to answer your question. When connection lessens or worsens, it becomes increasingly difficult for a speedy-accurate answer to be delivered to you normally. 

Throughout the course of the semester with Professor Irvine, some quick pro’s and con’s I personally have gathered with this technology would be: Pro’s: quick virtual help, hands-free, audio enabled, customizable, personable, free, accurate, useful when needed, and proficient. Con’s: Risk to your privacy, data is owned and accessible by Apple, the microphone is always listening to some degree in order to respond to the initial wake word, Hey Siri, and for all people especially very private people, this poses a threat to ethics of data usage, storage, and collection. All of this combined results in the next big question, Is Siri functionality ethical or unethical and does this put our privacy at risk? 

To further answer this, in the Stucke and Ezrachi 2017 study, they discuss, “The digital assistant with the users’ trust and consent will likely become the key gateway to the internet. Because of personalization and customization, consumers will likely relinquish other less personal and useful interfaces and increasingly rely on digital assistants to anticipate and fulfill their needs. They transform and improve the lives of consumers yet come at a cost” (Stucke and Ezrachi, 2017, pg. 1243). They found these types of assistants, especially Siri, follow a learning-by-doing model, and this is where the voice recognition and NLP happens that gets personally stored to each user profile. The more it is used, the more it learns about you (Stucke and Ezrachi, 2017, pg. 1249). They also say, the more someone uses Siri, the more it is able to predict the type of apps it needs to answer you, and the more it can start to personalize your data and formulate search bias (Stucke and Ezrachi, 2017, pg. 1242). Their concluding argument suggested it is nearly impossible to create an organic algorithm and not have a super-personalized experience that isn’t virtually stored and owned by the company (Stucke and Ezrachi, 2017, pg. 1296).

In a similar article, the Hoy 2018 study discusses what virtual assistants are and how they can pose a threat to privacy and need immense future regulation if they ever were to be used for other things than just your cell phone. He argues this because they already have so much access and ability to own your data, it would be extremely vast to think about the complexity and ability of Siri in a real-life large internet hosted setting. Hoy says, “Currently available voice assistant products from Apple, Amazon, Google, and Microsoft allow users to ask questions and issue commands to computers in natural language. There are many possible future uses of this technology, from home automation to translation to companionship and support for the elderly. However, there are also several problems with the currently available voice assistant products. Privacy and security controls will need to be improved before voice assistants should be used for anything that requires confidentiality” (Hoy, 2018, pg. 1). 

In relation to these studies, consumers then want to know, Does Siri actually always listen to you and what can be done about this? As impressive as this feature is in the new iOS 8 software update, where a user can say, ‘Hey Siri’ in a hands-free conversation, what they don’t know is Siri to some degree is always listening and is fully listens once woken.

As this product has improved, a recent conversation with Apple CEO, Tim Cook, took place between the House of Representatives and their legal team. The house wanted to know more of what’s really going on in their updates with Siri, the user location to pin point data, and the listening feature of what she is collecting and is this potentially harmful or against their policy (Sophos, 2018, pg.1). Tim Cook responded and said, “We are not Google or Facebook. The customer is not our product, and our business model does not depend on collecting vast amounts of personally identifiable information to enrich targeted profiles marketed to advertising” (Sophos, 2018, pg.1). To back this up further, Apple’s own director of Federal Government Affairs chimed in and wrote a formal letter that says, “We believe privacy is a fundamental human right and purposely design our products and services to minimize our collection of consumer data. When we do collect data, we’re transparent about it and work to disassociate it from the user” (Apple Response Letter, 2019, pg.1). 

An interesting feature in Apple’s response says, “The iPhone doesn’t listen to consumers, except to recognize the clear, unambiguous audio trigger ‘Hey Siri’. The on device speech recognizer runs in a short buffer and doesn’t record audio or send audio to the Siri app if ‘Hey Siri’ isn’t recognized” (Apple Response Letter, 2019, pg.1). All of this is good information to know and have the official update on, but this listening feature with Siri will always be a controversial pro and con to the technology. Even though the company is telling the world they are not creating technology designed to always listen to you, they in some other ways are saying it has the ability to always listen, especially when ‘woken’, but this isn’t so settling. Many reporters and consumers would argue big data companies, especially Apple, have the power to do this, but in terms of legal or privacy policy, they would never fully disclose to the general public that this technology is always listening. 

Even though Apple claims they nor Siri do not always listen to you, the answer to this question is still up for debate. In recent reports, other news sites would argue the opposite. In a recent USA Today article, they say, “With iOS 8, Apple introduced the ‘Hey Siri’ wake phrase, so you can summon Siri without even touching your iPhone. If you turn this feature on, this means your iPhone’s mic is always listening, waiting for the phrase, ‘Hey Siri’ to occur (USA Today, 2017, pg.1). 

In response to these reports such as this one, and similar ones, Apple claims the Siri microphone does not start to listen to you until the wake word is used, but it wouldn’t take much math involved to understand that in order for that to happen or be pronounced true, the device has to be listening to some degree in order for it to fully wake and then proceed with processing your information and provide an answer to your request. Whether the company owns up to this feature or not, either way, this poses a threat to privacy and one’s personal data with Siri. 

Looking ahead towards the future, one might then ask, Where is Siri going and what implications are needed for the future, if any? This is the most-dense component to my entire initial research question because Siri is already able to do so much, what more does she need to do? When thinking ahead for the next decade and beyond this is a mind-blowing thought process to experience. In order for the privacy threat controversy to disappear, there must definitely be more regulations reinforced in terms of more listening rights, protocol, access to full disclosure of how Siri listening fully works and is visually understood by all users who decide to turn it on. 

Another regulation to reinforce for the future would be that we cannot let virtual assistants control too much of our lives. I suggest this strongly, but looking in terms of the current projection of Siri on the Apple Siri website, it is evident that she will be running the world in other ways than just our phones. On their site they say, “Now you can control your smart appliances in your home, check their status, or even do many things at once—just using your voice. In the Home App, you can create a page ‘I’m Home’ that opens the garage, unlocks the front door, and turns on the lights” (Apple Siri, 2019, pg.1). Some common questions we can ask it to do inside our homes would be: ‘Did I close the garage, Show me the driveway camera, or tell it to redirect your smart TV remote to record a show for you when you’re not home’. (Apple Siri, 2019, pg.1). 

As if At-Home assistance like this isn’t overwhelming enough, Siri is now accessible within smart cars and newer models of cars across all brands. You can also ask questions related to your car such as, ‘Did I close my door, Where did I park, What song is this, Play 92.5 FM Radio, Answer phone call, etc’. all at the power of your voice, hands-free inside your moving vehicle (Apple Siri, 2019, pg.1). This feature is gaining a lot of speed and is now easily used and accessible to enhance not just your phone experience, but at-home, on-the-go, music, or even car related experiences. 

According to VOX Media, the future for voice assistants is looking extremely bright in terms of running the ways in which we use technology professionally, socially, economically, industrially, and personally. They pulled some statistics that say, “There are 90.1 million people who own and use smart phone technology, 77.1 million people who use it inside cars, and 45.7 million people who use it on speakers” (Vox Media, 2019, pg.1). These statics also can suggest that the future of virtual assistants, especially Siri, will become the new face of voice automated technology, and for the other categories previously mentioned from Apple. 

Personal Interpretation

As a frequent user of Siri and Apple consumer, this topic sparked many interests in my participation with their products and I wanted to learn more about Siri and how it all actually works. The most controversial part to this entire works was the listening section where Apple’s privacy statements were up for debate in recent news. From what I have gathered in further research, it’s still hazy to argue that Apple is not fully always listening to you, because most people would argue the opposite despite what they continue to legally market and disclose in their statements. 

It is increasingly difficult to know the truth regarding this matter, but what I have gathered in this AI course this semester and the research done to answer my initial research question, I could argue my hypothesis as pending true due to the fact that the technology is listening to the user to some degree in order to process the wake word. Understandable in terms of legal issues, Apple would never fully disclose this performance, but from understanding the NLP architecture and algorithmic cloud computing features, I would confidently stand on the always listening side to this argument due to the fact in the privacy statements they never even fully and clearly shut down the possibility of that being possible. 

Conclusion

Virtual Assistants are technology that is designed to enhance our lives. Siri in particular is a highly skilled AI virtual assistant that can act as a large key component to our inquiries, questions, or requests in order to achieve a certain task. In nature, the main goal of Siri is genius, and extremely convenient. However, as it continually progresses, we can find and see the leverage it is slowly gaining on our lives, not just our phones. The purpose of this progression is to keep users relying on this technology and Apple products. In this, it reinforces Siri has the answers we deeply desire, but AI and Siri in particular is taking a route that could be going too far in replacing human actions in human life.

As exciting as the future looks, all of this overlapping control and capability for all areas of our lives such as our phones, homes, cars, businesses models, software, and more is agreeably innovative, but extremely inconclusive and terrifying at the same exact time. These technologies are a privilege and we must use them to our own degree when necessary but not let it overpower the meaning of life. No artificial technology is better than real authentic life choices and actions.

Works Cited

Apple Introduces iCloud. (2019, April 05). Retrieved May 5, 2019.

Apple Response to July 9 Letter. 2019.  Retrieved May 5, 2019, from SCRIBD

Daniel Jurafsky and James H. Martin, Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, 2nd ed. (Upper Saddle River, N.J: Prentice Hall, 2008).

Deep Learning for Siri’s Voice: On-device Deep Mixture Density Networks for Hybrid Unit Selection Synthesis – Apple. Retrieved May 5, 2019, from Apple Siri Voices.

Hey Siri: An On-device DNN-powered Voice Trigger for Apple’s Personal Assistant – Apple. Retrieved May 5, 2019, from Hey Siri.

Hey Siri: The Pros and Cons of Voice Commands. 2018. Retrieved May 5, 2019, from UK Norton Blog.

Komando, K. (2017, September 29). How to stop your devices from listening to (and saving) what you say. Retrieved May 5, 2019, from USA TODAY

Matthew B. Hoy (2018) Alexa, Siri, Cortana, and More: An Introduction to Voice Assistants, Medical Reference Services Quarterly, 37:1, 81-88, DOI: 1080/02763869.2018.1404391

Molla, R. (2019, January 15). The future of voice assistants like Alexa and Siri isn’t just in homes – it’s in cars. Retrieved May 5, 2019, from VOX media.

Siri. (Retrieved May 5, 2019, from Apple Siri.

Stucke, M. E.; Ezrachi, A. (2017). How digital assistants can harm our economy, privacy, and democracy. Berkeley Technology Law Journal, 32(3), 1239-1300.

The Real Reason Voice Assistants Are Female (and Why it Matters). (2018, January 29). Retrieved May 5, 2019.

Today in Apple history: Siri debuts on iPhone 4s. Cult of Mac. (2018, October 04). Retrieved May 5, 2019.

De-blackbox Past, Present, and Future Of Alexa

Annaliese Blank

Zachary Omer

Beiyuan Gu

Amazon is one of the biggest global commodity based companies that is running the world. One of their most important technologies they have marketed is Alexa. Alexa is known as their own patented product that is their very own virtual assistant technology. “She” was first made in 2014 and has been refined in various versions in now 2018-2019. She is designed to be a virtual assistant technology in your own home to continually listen to its “master” and be of any form of assistance to their needs or inquiries. Alexa requires the internet and relies on verbal speech to be used in order to “wake the technology with the wake word” which then records virtually what was asked and records down all of your speech patterns in order for better speech recognition and performance.

The purpose of Alexa was solely to enhance the smaller or larger tasks of our lives whether it be answering a question, complex or not, texting a message to anyone in your contact list, or looking up the easiest recipe in your kitchen to be as at home sue-chef. All of these tasks are done virtually and answer is produced within on average less than 5 seconds. The data we give to Alexa is virtually coded, understood, and stored al within milliseconds and in this algorithm, the most accurate answer is then produced, usually without most general awareness of how this was made so quickly. No task too small is Alexa’s motto for our group!

Some quotes we wanted to pull from some research we did:

  • You control Alexa with your voice. Alexa streams audio to the cloud when you interact with Alexa. Amazon processes and retains your Alexa Interactions, such as your voice inputs, music playlists, and your Alexa to-do and shopping lists, in the cloud to provide, personalize, and improve our services” (Amazon Help, pg.1).
  • “Voice interaction, music playback, makingto-do lists, setting alarms, streamingpodcasts, and playingaudiobooks, in addition to providing weather, traffic and other real-time information. It can also control severalsmart devices, acting as ahome automationhub” (Wikipedia, pg.1).

To analyze this technology further, we wanted to understand the technology and de-black box its body parts and see more visual aids on where the voice recognition process occurs and be able to understand how the actual root of the machinery works. The socio-technical components are thus followed:

We will break down the components more in the presentation. Some quick parts are the light ring, volume ring, and 7 piece microphone array to detect record and listen to your voice when we speak directly to Alexa. This is where she will start to recognize your master voice and virtually store the conversation in the cloud.The whole process to this allows Alexa to begin forming a better way of knowing you and keeping track of your personal usage and data. In doing this, it sets her apart from other competing virtual assistant technologies.

Some pro’s and con’s are thus followed:

PRO’S

  • Efficiency
  • Low Maintenance
  • Timeless
  • Non-tedious
  • Quick Help
  • Accuracy
  • Proficiency
  • Cost Effective

CON’S

  • Privacy risks and costs
  • Data is shared and owned
  • Always listening
  • Agreeing to sell your data to Amazon
  • Ethical or unethical?

With the negative aspects of this technology in mind, Alexa herself has received a lot of backlash over the years in terms of this biggest question: DOES ALEXA POSE A THREAT TO YOUR PERSONAL PRIVACY AND DATA THAT IS SHARED AND STORED, AND OWNED AND USED BY AMAZON WITHOUT OUR PERMISSION OR FULL KNOWLEDGE?

Some current Privacy Control Updates and Thoughts:

  • While Amazon Echo’s microphones are always listening, speech recognition is performed locally by the device until the wake word has been detected, at which point the subsequent voice command is forwarded to Amazon’s servers for processing. In addition, Amazon Echo is equipped with a physical button to mute the microphones.
  • Companion mobile apps and websites enable users to review and delete prior voice interactions with the device should they feel uncomfortable or not want Amazon to keep particular voice recordings on their servers.

With this in mind, it becomes increasingly difficult for users to believe this though, because the counter argument would be : DOES ALEXA ALWAYS LISTEN IN ORDER TO ACQUIRE THE WAKE WORD? This is where the threat to privacy and personal data control lies.

Some questions we asked ourselves was, What is actually being recorded? How will this collected information be used and to who? If so, how will it be protected? Will it be used for targeted advertising?

When thinking more about this invasion of privacy, we found an example case to expand on this further:

Case 1: In January, 2017 in Dallas, Texas, when a six-year-old girl asked her family’s new Amazon Echo “can you play dollhouse with me and get me a dollhouse?” The device readily complied, ordering a KidKraft Sparkle mansion dollhouse, in addition to “four pounds of sugar cookies.” The parents quickly realized what had happened and have since added a code for purchases. They have also donated the dollhouse a local children’s hospital.

The story could have stopped there, had it not ended up on a local morning show on San Diego’s CW6 News. At the end of the story, Anchor Jim Patton remarked: “I love the little girl, saying ‘Alexa ordered me a dollhouse,’” According to CW6 News, Echo owners who were watching the broadcast found that the remark triggered orders on their own devices.

Case 2: May 25, 2018, a woman in Portland, Oregon found out that her family’s home digital assistant, Amazon’s Alexa, had recorded a conversation between her and her husband without their permission or awareness, and sent the audio recording to a random person on their contacts list.

With all of this said, we wanted then make some concluding thoughts on the future to Alexa and where it would be headed. According to the SLU project, at SLU they are the first University to bring Amazon Alexa embedded devices, managed by Alexa for Business purposes in the residence halls and on campus apartments. This is a great example of empowering education with better technology for the future. SLU has installed more than 2,300 ECHO devices that are also great campus helpers to inform students on campus information and updates.

When we mean business, this is the future. Thousands of national and international companies use this type of virtual assistant technology in their own algorithmic work and company structure. Here is a picture below of a business model and how it contributes to the work environment:

 

In light of all of this, we gained a better perspective for this respective technology and how she is changing the business and social world, one model and revision at a time. She is going nowhere; we look forward to seeing more virtual assistant technology unfold in the future and see how much more they will able to do and alter in our every day lives.

Citations:

 

Alexa for Business Overview, Retrieved from https://aws.amazon.com/alexaforbusiness/

Alexa Privacy and Data Handling Overview, Retrieved from https://d1.awsstatic.com/product-marketing/A4B/White%20Paper%20-%20Alexa%20Privacy%20and%20Data%20Handling%20Overview.pdfAlexa Terms of Use. Updated 11/27/2018, Retrieved from https://www.amazon.com/gp/help/customer/display.html?nodeId=201809740

Alexa, Echo Devices, and Your Privacy (FAQs), Retrieved from https://www.amazon.com/gp/help/customer/display.html/?nodeId=GVP69FUJ48X9DK8V?ref_=ods_aucc_dp_farfield_rd

D’Angelo, M. (2018, December 26). Alexa for Business: What Small to Medium Businesses Need to Know. Business News Daily. Retrieved from https://www.businessnewsdaily.com/10426-amazon-alexa-business-details.html

History of Amazon Echo, Retrieved from https://en.wikipedia.org/wiki/Amazon_Echo#History

Horcher, G. (2018, May 25). Woman says her Amazon device recorded private conversation, sent it out to random contact. KIRO 7 News. Retrieved from https://www.kiro7.com/news/local/woman-says-her-amazon-device-recorded-private-conversation-sent-it-out-to-random-contact/755507974

Lau, J., Zimmerman, B., & Schaub, F. (2018). Alexa, are you listening? Proceedings of the ACM on Human-Computer Interaction, 2(CSCW), 1-31. doi:10.1145/3274371

Molla, R. (2019, January 15). The future of voice assistants like Alexa and Siri isn’t just in homes — it’s in cars. Retrieved from Recode website: https://www.recode.net/2019/1/15/18182465/voice-assistant-alexa-siri-home-car-future

Saint Louis University. (2018, August). SLU Installing Amazon Alexa-Enabled Devices in Every Student Living Space on Campus. Retrieved from SLU Alexa Project web page: https://www.slu.edu/news/2018/august/slu-alexa-project.php

 

 

Final Thoughts: Siri and Machine Learning

As the semester begins to close, I am reflecting on some topics that interest me

MACHINE LEARNING AND SIRI

In the Machine Learning piece, Martin and Jurafsky, helped me gather some fundamentals on my inquiries. When we de-black box this, we can see there is no perfect way to translate something, especially how I mentioned before that the “perfect” translation doesn’t exist in all of the same locations, since not all language is “universal”. They say, “Technical texts are not translated in the same way as literary texts. A specific text concerns a world that is remote from the world of the reader in the target language. The translator has to choose between saying close to the original text or making use of paraphrasing to ensure comprehension. The tone and style of the text are highly subjective” (Machine Learning, pg. 19). This got me thinking, How, can we trust machine translation or google translate so much if it is impossible to gain 100% accuracy? Where does this trust reside?

( week 7 )

From the Wikipedia article this week, Siri was defined as, “the virtual assistant for Apple systems that uses voice queries and natural language user interface to answer questions, make recommendations, and perform actions through requests of the internet. The software adapts to the user’s individual language usages, searches, preferences, etc. The results are personalized” (Wikipedia, Siri, pg. 1). This helped me pieces together what it really is doing and how it’s a personal experience and technology. This made me raise the question of now that I know this and some basic layers to how the voice recognition works, what else can we do with siri? 

I wanted to look for other outside material to help answer my question, since its pretty standard that siri is just basically a voice over platform that we use to speak into in order to receive some type of mediated response or aid to an inquiry. According to writer Todd Haselton, for CNBC, with the competing rise to Google Assistant, we can now talk to Google Assistant, THROUGH your Siri. This new technology is basically Google’s own version of Siri, which is somewhat funny that you can use the competition to open up the other app. When you do this, you must APPROVE the ability to pair Siri with your downloaded google assistant, assuring the same voice over controls. When you’re ready, you say, “Hey Siri, OK GOOGLE” (Haselton, pg.1). This allows the Siri to approve your voice and open up the Google assistant instead.

( week 8 )

________________________

For this week, I wanted to add more readings. One article that stood out in particular was The Verge piece that discussed the ethical side to AI a bit further. They discuss a more European way to enhance the ethical and privacy issues that they face with AI and made a list of 7 things that they feel is the most important and needs work. In their list they say, “human agency and oversight, more safety, securing privacy and data, better transparency for humans to explain the algorithms chosen and implemented, more diversity and fairness, sustainable technology that will promote well-being rather than changing it, and more accountable on restoring privacy and further updates to the technology. (The Verge, pg.1).

For me I found this article really important in relation to my class connections that I have made over the semester. I think the most important role in AI that I have learned in the privacy, and code of ethics that must be implemented in these technologies that have not only given rise to their advancements but their disadvantages too, which of course can be a huge issue.

All of this is helpful in guiding me towards preparing for my final project. I met with Dr. Irvine and we helped narrow down my interests in this issue and how it plays a role in Apple products, like Siri. I took the most interest in our virtual assistant and privacy week work. I think this would be a great topic and issue to look into and gather a socio-technical-historical approach to the technology and incorporate the development, ownership, production, and implementation of Siri, and the future of where it’s going. I appreciate Siri and its ability to aid so much in our every-day lives. In connection to this reading, this would be a great example and issue to pull in my literature review and see what other pieces of literature I can find on this issue and other key features to Siri, from then until now.

A question I have developed is with recent advancements, research, and reports on AI ethics and privacy, where do the boundaries get drawn when we look ahead towards the future? Where do the limitations start and end? What will the future of AI look like with these advancements in place? I look forward to unpacking more answers and insight to these questions in the remainder of the course and final paper.

 

https://www.theverge.com/2019/4/8/18300149/eu-artificial-intelligence-ai-ethical-guidelines-recommendations

 

Haselton, T. (2018, November 21). You can now talk to Google Assistant through Siri on your iPhone here’s how. Retrieved March 12, 2019.

 

Daniel Jurafsky and James H. Martin, Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, 2nd ed. (Upper Saddle River, N.J: Prentice Hall, 2008).

big data purpose

Annaliese Blank

When I think about big data, the first thing that comes to mind would be the internet, or some form of a collection of user data from a global perspective that is intertwined with each other that gets recorded as one big unit of data exchange. In the Ubiquity piece, they explain the terminology and a brief history of this term. They say, “: the expansion of the internet into billions of computing devices, and the digitization of almost everything. The internet gives us access to vast amounts of data. Digitization creates digital representations for many things once thought to be beyond the reach of computing technology. The result is an explosion of innovation of network-based big data applications and the automation of cognitive tasks. This revolution is introducing what Brynjolfsson and McAfee call the “Second Machine Age.” This symposium will examine this revolution from a number of angles.” (ACM Ubiquity, pg.1).

Within the past twenty years and more, none of this was possible until recently. Internet expansion has opened up doors of opportunity for the future of big data. This is extremely important because this transition into the tech era requires the tools and the components to make universal connectivity possible. The transfer of data packets to hold vast amounts of information and code and have it be sent wirelessly and instantaneously would be another great description to big data and its ideal purpose. The big takeaway here would be this is the revolution in the power of digitization.

This revolution is the foundation to the new functions and operations for society, politics, education, policy, government, and science. For digital data and data science, not only does big data capabilities allow computing power able to handle volumes of data, but for data science and education, this aids the process of “data analysis, research, manual and automated search capabilities, and machine learning functions and modeling” (Ubiquity Big data, pg. 1). This is changing the way in which we learn information, search and record data, send data, analyze data, and compute and translate data for everyday or personal use. Big data has changed the world.

I wanted to pull in another outside source after this. The company and brand Statistical Analysis System, SAS, empowers the function and support to big data analysis. According to SAS, they define big data as, “Big data analytics examines large amounts of data to uncover hidden patterns, correlations and other insights. With today’s technology, it’s possible to analyze your data and get answers from it almost immediately – an effort that’s slower and less efficient with more traditional business intelligence solutions” (SAS, Big data, pg.1). The main benefits to big data are speed, efficiency, and innovation. It has influenced the business world in terms of business communications and analytics that provides the efficiency tools and proficient environment for advanced stability and recognition. The concept of big data allows the “competitive edge” that big companies need (SAS, big data, pg.1). According to SAS, the importance to big data lies in: “cost reduction, faster better decision making, and new products and services” (SAS, big data, pg. 1-2). The power of big data is taking the world by storm and will be unstoppable with continuous efforts and changes to its mechanics and process functions.

 

https://ubiquity.acm.org/article.cfm?id=3158335

https://ubiquity.acm.org/article.cfm?id=3158352

https://www.sas.com/en_us/insights/analytics/big-data-analytics.html

Jeffrey Johnson, Peter Denning, et al., Big Data, Digitization, and Social Change (Opening Statement), Ubiquity 2017, (December 2017).

Cloud AI

Annaliese Blank

When we unpack the AI Cloud computing the realization here is most technology and big companies use Cloud software. According to Wikipedia, they say that Amazon has the largest public cloud and the internet itself acts as a cloud service. Some virtual assistants I wanted to unpack this week are Alexa, Google-Home, and Siri. Another thing I wanted to understand was how these technologies connect to the cloud in order to operate.

Virtual assistants are a different type of learning machine that always stay on and are always actively listening to you. They look for patterns in your voice and try to make predictions and connections that best identify you as their master operator. Whenever you ask it something, it must run your voice recognition patterns in order to make sure it is you speaking to them. They are friendly devices to other voices other than your own, but in general preference they must do this in order to keep track of what their owner says, wants, or needs to know by asking it a question or task. When you speak to it, the machine sends your recording to the cloud server to record what you say and they predict the best response possible. How do they do this?  

For Amazon Alexa, she mainly operates on the cloud. She is designed for more simple tasks and is able to answer quick or dense questions in a matter of seconds. She uses the cloud to store your questions and save your data. She gets to know you better by doing this.

For google products, they operate on a similar basis but in their own google cloud. Google performs links to your information by creating your own ‘google record’, and by doing this when we create google accounts we are agreeing to fully trust all of our information via their privacy services. Google home does exactly that. When you need certain answers to information and you have already researched your questions via phone or computer on google, Google home already has that saved under your profile which will help lead to an even faster answer. You can remove your search history on the product, or permanently from their Google cloud account on you.

Apple Siri is similar to these but uses their own apple cloud software. They do record the information you search or ask, but they don’t store it to your own personal length. Infact they use it for their own bettering of their company. They like to see who identifies what way and what they mainly search for and how they get the best results. They even say in their privacy statement on their cloud services, “We use personal information to help create, develop, operate, and improve our products, services, content, and advertising” (Apple Privacy, pg.1).

The cloud service is a superior advancement to AI. Its’ abilities are endless. When we unpack these forms of AI, it is important that we understand how the cloud works and what is actually being store privately in the cloud. Keeping information secure and private is the goal, but after learning bits and pieces of what each of these do it is vital that we are careful about what we search and what we say.

https://en.wikipedia.org/wiki/Cloud_computing

https://cybercity.nyc/blog/virtual-assistants-how-do-they-handle-your-privacy/

https://www.apple.com/legal/privacy/en-ww/

https://cloud.google.com/products/ai/

 

Algorithms Unpacked

Annaliese Blank

As AI continues to progress, it is important that we unpack the issues attached with it. According to Google, AI should be, “socially beneficial, avoid creating or reinforcing unfair bias, be built and tested for safety, accountable for people, incorporate privacy design principles, uphold high standards of scientific excellence, and be made available for uses that include factors of scale, uniqueness, and nature of Google’s involvement” (AI at Google, pg. 1).

For me, some issues that AI typically faces is privacy and ethical issues. When we think about ethics, are the algorithms in AI actually ethical? I formulate this question because machine learning algorithms are extremely smart and predictive in their mechanics, and at times they learn as they go and form ways to predict or reenact human answers or thoughts to something. If we deblackbox this, how can we know that AI is actually ethical if we are not always aware of what’s going on or how they re-adjust to new changes in their performances? In some ways it’s almost like a secret upgrade that we cannot see. It also makes us question how do we determine what is ethical in a system if the AI is going to tell us something different? This is tricky because AI is supposed to AID human behavior, not REPLACE it. “An algorithm cannot be modified to avoid its focus and consequences” (Ethical Character of Algorithms, pg.1).

When I think about privacy, I think AI is used in a way to personalize your experience in its usage. It is going to try to understand your likes and dislikes and grow accustomed to how you operate on your own technology. This is extremely controversial in conserving privacy and non-disclosing personal data to outside sources. Big data tech companies like Facebook and Google do every day. It happens under our nose and we are not aware of it because of these issues and loopholes in AI. “Using personalized algorithms increases polarization and provides incentives for clickbait” (Ethical Character of Algorithms, pg. 1). Other continuing negative effects would be biased or un-truthful spread of misinformation which AI seems to generate and this is affecting democracy for society. AI has a lot of controversy in how it controls our lives and government and it truly is a great tool but these are the negative effects we all must be more aware of.

Something that truly did scare me this week was the Youtube video showing how AI can do voice over changes if done properly and correctly. In the video, President Obama is speaking but not the words we are hearing. The male in the background is using voice over control technology to change Obama’s statement and make it falsified. In doing this, which happens a lot on platforms like Youtube, this as I said before, really does contribute to misinformation and creates more backlash than intended. Some myths about AI would be: its non-biased, not every company needs AI algorithms, or that it works in the same way your brain does. Which all of these sound so true, but in reality it’s just not true. They in almost all cases can be biased, especially in their setup, companies should be fools if they did not implement AI technology, and it is motivated to act similar to a brain but never fully intended to replace one, just to improve on one or be predictive about it for research purposes only. (Information Management, pg. 1). There should always be room for improvement, but never replacement. 

https://www.youtube.com/watch?v=gLoI9hAX9dw

https://www.information-management.com/list/debunking-five-common-misconceptions-about-ai

https://ai.shorensteincenter.org/ideas/2019/1/14/the-ethical-character-of-algorithmsand-what-it-means-for-fairness-the-character-of-decision-making-and-the-future-of-news-yak6m

https://www.lyrn.ai/2018/12/26/a-style-based-generator-architecture-for-generative-adversarial-networks/

Mark MacCarthy, “The Ethical Character of Algorithms—and What It Means for Fairness, the Character of Decision-Making, and the Future of News,” The Ethical Machine (blog), March 15, 2019.

Siri: A Deep Neutral Network

Annaliese Blank

Layers of Design Speech Activated services:

  • Virtual assistant services are compiled into other technologies such as Amazon echo, alexa, siri, google home, google assistant, and many more.
  • Digital speech recognition technology is the foundation for these technologies

For this week, I wanted to unpack the Siri technology to the best of my ability. In the apple machine learning piece, this site was able to inform me of these layers. The main way to speak to siri is to state the conversation by saying “hey siri”. Once you are able to speak to Siri, the neutral network is activated to then listen to your voice and then confirms that you needed Siri’s attention. The parts of Siri operate within the cloud.

The pattern recognition then takes less than .01 seconds to break apart your command into a set of frames, which is sent to the deep neutral network. This neutral network is made up these 6 layers that are very important into understanding and breaking down your speech pattern.

Neural Network –> trigger –> softmax –> bias units –> sigmoidal –> streaming vectors —> final input window.

 

From the Wikipedia article this week, Siri was defined as, “the virtual assistant for Apple systems that uses voice queries and natural language user interface to answer questions, make recommendations, and perform actions through requests of the internet. The software adapts to the user’s individual language usages, searches, preferences, etc. The results are personalized” (Wikipedia, Siri, pg. 1). This helped me pieces together what it really is doing and how it’s a personal experience and technology. This made me raise the question of now that I know this and some basic layers to how the voice recognition works, what else can we do with siri?

I wanted to look for other outside material to help answer my question, since its pretty standard that siri is just basically a voice over platform that we use to speak into in order to receive some type of mediated response or aid to an inquiry. According to writer Todd Haselton, for CNBC, with the competing rise to Google Assistant, we can now talk to Google Assistant, THROUGH your Siri. This new technology is basically Google’s own version of Siri, which is somewhat funny that you can use the competition to open up the other app. When you do this, you must APPROVE the ability to pair Siri with your downloaded google assistant, assuring the same voice over controls. When you’re ready, you say, “Hey Siri, OK GOOGLE” (Haselton, pg.1). This allows the Siri to approve your voice and open up the Google assistant instead.

I then found another outside source. I think this youtube video I found is a great summary of showing how this voice pattern process looks like and how it works with the examples they provide. She can do list creations, answer your questions, look up anything for you, and so much more. More details in the link below. I enjoyed the new information I learned this week. I am still fuzzy on some of the verbiage that explains the “bits exchange” going on and how the inputs of answers are produced by Siri once the layers and decoding is complete.

Apple Machine Learning Journal (1/9, April 2018): “Personalized ‘Hey Siri’.

Apple’s Patent Application for “An intelligent automated assistant system” (US Patent Office, 2011).

Haselton, T. (2018, November 21). You can now talk to Google Assistant through Siri on your iPhone here’s how. Retrieved March 12, 2019.

https://en.wikipedia.org/wiki/Virtual_assistant

https://www.youtube.com/watch?v=VsO_aKlxses

https://www.cnbc.com/2018/11/21/how-to-use-google-assistant-through-siri-on-iphone.html

Ambiguity vs. Predictability in ML and Google

Annaliese Blank

Some key themes for this week would be grammar and online translation of language. My goal for this week was to unpack this more and see how machine translation and google translate work. I use google translate all the time and I wanted to see its operations since I use the Spanish translations a lot for my travels. I have been to many parts of Mexico and Argentina where I specifically used Google translate to begin a foundation before staying with my previous host families. I took Spanish from third grade forward and even during high school, google translate definitely peaked at my school. It was like the perfect solution to so many problems when other websites or textbooks just couldn’t get the job done enough. The key word here is enough, Google translate when we unpack this does so much more than a simple translation, it does a grammar check and conversational check and makes sure that the current translation is the correct verbal translation, depending on what region you’re in since some areas don’t use the same versions of Spanish.

To further this, I really enjoyed the Machine Learning piece. I especially wanted to make connections here on machine learning. All of this really got me thinking about translation. A question I’d like to raise is, what exactly is translation and how can we understand the process, perhaps through other technology than google, like machine translation? What is the criteria?

In the Machine Learning piece, Martin and Jurafsky, helped me gather some fundamentals on my inquiries. When we de-black box this, we can see there is no perfect way to translate something, especially how I mentioned before that the “perfect” translation doesn’t exist in all of the same locations, since not all language is “universal”. They say, “Technical texts are not translated in the same way as literary texts. A specific text concerns a world that is remote from the world of the reader in the target language. The translator has to choose between saying close to the original text or making use of paraphrasing to ensure comprehension. The tone and style of the text are highly subjective” (Machine Learning, pg. 19). This got me thinking, How, can we trust machine translation or google translate so much if it is impossible to gain 100% accuracy? Where does this trust reside?

 Some other important areas I found really interesting were the discussions of morphology and syntax. Morphology deals with the structures of the words and syntax designs the sentence. For computing, or machine translation this is really hard to do because there is one thing they mentioned the most was AMBIGUITY, high amounts of uncertainty. From what I have read and gathered this seems to be still a main limitation to online translations and could potentially still be a problem for google translate in the future, this problem isn’t fixable. How does ambiguity exist if predictability prevails?

After watching these videos and crash course sites, I feel I have gathered a better understanding of what google translate does and how language can be modified in machine learning and coding. Coding in its own way is its own language.

And finally to describe the levels of a technology for this week, I wanted to continue the topic of google translate more. In the last google video they say, in the tap to translate feature, there is a button where you can translate a message sent to you, and say in voice-over or type out a message in English, and have them send it back in the designated language. It can be any phrase or character count up to 5000 characters. The translation options run as simple as English, Spanish, French, Italian, Russian, Chinese, Arabic. Etc. Once the translation is complete, you can send it, save it, or drag it wherever you need it onto a different app. This is most definitely the least complex NPL and is very easy to manage. What happens here is any of these languages are translated based off their original entry in English and then re-configured into the pre-set language translation of their preference. For AI, this is a game changer because the translations become predictable if the machine has already learned the appropriate inputs. This really helped me understand how this translation process works. Having pre-set features of things like syntax too go hand-in-hand for producing the best translated result. All of this now has led to Google’s most current upgrade, which is “translate as you type” where I mentioned before the pre-set features of the system allow this predictability to be heightened and makes the translation much easier for anyone.

I wanted to take this a step further and I looked up how google translate works in other ways, such as handwriting translation, creating your own verbal phrases, slow down pro-nunciations of certain phrases, and connects your style of languge from facebook messenger on to the translation tab. More details can be found here; https://www.express.co.uk/life-style/science-technology/976492/Google-Translate-how-to-use-Google-Translate-how-accurate

I feel I have gathered a better sense of machine learning and google translate in relation to AI, but I still feel stuck on artificial vs. natural systems.

Data Structure Crash Course:

  • Arrays- values stored in memory
  • Indexes
  • Strings- arrays of characters
  • Null characters
  • Matrix – array of arrays (3 total)

Machine Learning & Artificial Intelligence:

  • Algorithms give computers the ability to learn from the data and allow the ability to give and make decisions
  • Input layer, Output layer, Neuron Layer

How Google Translate Works:

  • language translations- word by word
  • curated data base to help translate pairs
  • tokens- smallest units of language
  • grammar- defines ordering of tokens
  • syntax analysis- does the structure look correct?
  • Semantic analysis- meaning, does this sentence make sense in context?
  • Neural network – component that learns to solve problems and allows the network to learn patterns and data
  • This helps with the translation process
  • Encoder – Decoder architecture – the pathway to insert vectors that carry out translations

Daniel Jurafsky and James H. Martin, Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, 2nd ed. (Upper Saddle River, N.J: Prentice Hall, 2008).

https://en.wikipedia.org/wiki/Google_Neural_Machine_Translation

https://translate.google.com/intl/en/about/

https://www.youtube.com/watch?v=AIpXjFwVdIE

https://www.youtube.com/watch?v=DuDz6B4cqVc

https://www.youtube.com/watch?v=z-EtmaFJieY

 

https://learningenglish.voanews.com/a/eight-things-you-didnt-know-google-translate-could-do/4097446.html

 

Unicode and Emoji, Bitmoji Systems

Annaliese Blank

For this week, I wanted to take more time to unpack these readings and really understand “data” in all of its capacity. Apologies on the late post. I needed more time to digest this.

To start off, I began with the Irvine reading. My main goal was to address what is data? We are so lucky to call him our professor because he specifically laid out what it’s means to define data and how its constructed in various terms.

A question that came to mind was does big data only store unstructured data? Or would it be a combination of both structured and unstructured data? This question came to mind for me when I think about cloud software and how that’s a very organized space that functions based off unstructured data since it can store pretty much anything. But, inside the cloud, it’s easy to find whatever you need since it is stored properly based on what it’s made of and how its labeled. I’m curious to know if the cloud operates in the same way, by using structured and unstructured data in order to virtually store it for you. This idea came to me when I was thinking about the representation aspect to this topic.

A great way to think about data and define it would be through Unicode. Ever since items like the emoji or bitmoji were released I’ve always wondered how they operate. These are forms of communication that we all use in our day to day messages without really realizing what we’re sending. He says, “what we see on our screens is a software ‘projection’ of the bytecode, interpreted in whatever selected font style for ‘rendering’ the pixel patterns on screens” (Irvine, pg. 4). When I also think about these relational databases that he mentions later, I think of Excel spreadsheets, where different entries can be labeled and organized into specific tables. I was still a bit confused on the difference between this and NoSQL in terms of “container structures”.

Another way that I can think about this data and its meaning would be the section where Unicode emoji is mentioned. It says, “Emoji’s are pictographs, pictorial symbols, that are typically presented in a colorful form and use inline in text.” (Unicode Emoji, pg.1). I then looked at the Unicode Emoji Data Files. In here lies several documents that explain and display the data and codes that produce and send the emoji’s that we are able to see on our phones.

After reading through these, I would gather my own synopsis of data is based on structured or unstructured inputs, categorical or numerical, that are designed with sending purposes for collection or presentation, and the way in which it happens is through these bits and bytes and forms of Unicode that allow us to see something easier, vs. not knowing how it actually got there or what it’s entity truly is in code form.

Another interesting thing I wanted to look at was Amazon’s RDS. It says this is able to store data through a fast cloud performing base called “Aurora, Maria, Oracle, and SQL Servers. It focuses on management, security, fixes, and global access to other databases instantaneously. Its high speed functionality allows it to store and improve consumer, company, and product data, faster on its own, leaving little work for consumers to worry about. This to me Is a bit extreme considering how we all use Amazon but don’t know the background on really what we’re buying and how that data is stored and used for their company through their RDS. This type of data management I think should be more exposed to the public eye that way some reinforcement of better protection of data can occur.

An outside source to help me make further connections was this YouTube tutorial that addresses ASCII and Unicode. It reviews the binary codes and how letters turn into binary numbers. This process helps us understand how to code or decode a letter or phrase on our screen. This is similar to the emoji. Each emoji has its own code and this video does a great job of explaining how the process works that makes it all happen and visible on our screens!

I look forward to next week to further this and my ideas more.

 

Irvine, “Introduction to Data Concepts and Database Systems.”

Unicode Emoji. (n.d.). Retrieved February 20, 2019, from https://www.unicode.org/emoji/

https://aws.amazon.com/rds/?nc2=h_m1(video)

http://mentalfloss.com/article/66338/how-are-emoji-made(article)

https://www.youtube.com/watch?v=5aJKKgSEUnY(extra video)

Symbols and Signals

Annaliese Blank

For this week, my goal was to understand information from multiple angles. From the Gleick reading, he focused on the theory of information and how it composes multiple layers. Some questions he asked were: Can machines think and what tasks were mechanical?This got me thinking about what’s actually being produced such as a word, image, or website, but also automatic or pre-determined, such as algorithm improvements. He says, “the justification lies in the fact that the human memory is necessarily limited…Humans solve problems with intuition, imagination, flashes of insight – arguably non mechanical calculation…” (pg. 15). This got me thinking about how absolute certainty plays a required role in machine computing which is able to include all preceding decimals for information. This makes me question, is information in a computer a tool or is it a machine within its own mechanics? Is information just based of gatherings and collections of signals?

Transitioning to the Irvine piece, I really enjoyed learning about the designs of information interfaces and how important the “signal” role is. I am still falling short on the signal transmission theory and the verbiage associated with it. It was difficult to unpack the question at hand, but I’m hoping I achieved a general sense of how to de-black box this.

The main features of signal transmission theory of information would be the digital design of “information” that is structured as “units” of “preserved structures” which use electricity via bits and bytes to extract certain patterns that signal an internal message that gets completed. The signal code transmission model is not a description of meaning because it’s not meant to describe “meanings”. It is designed as single units that are “point-to-point” models that display how “information” passes through a channel. There are data types, signs, tokens, data types, etc. that are involved in this encoding and decoding process. (Irvine, pg. 13-20).

Information theory model is essential for everything electronic and digital because digitized data, information, or tasks do not get performed without being instructed a certain way. This type of model helps ensure certainty with numbers, codes, or data that is transmitted electronically and received electronically, whereas it lacks the verbiage to further explain what each subset performs in terms of semantic meanings, or the specific uses and meanings of other systems. This could also be due to the information theory model is designed to DISPLAY how something is achieved electronically in the simplest way.

It doesn’t necessarily apply to the type of model that would fully explain sign and symbol systems, because it is laid out to explain how something gets done, vs. something of a “symbol” and that typically would not change within the model. Whereas, explaining a sign or a symbol in this type of language wouldn’t fully apply because the information model is based of E-signals that are transmitted and received, and in other cases symbols and signs systems are not.

To expand on this more with more easy to understand vocabulary, I wanted to pull in an outside source through video help me understand this better. When in doubt, YouTube it out! I found a great video that describes how speakers create sound waves, and how signals are transmitted. The user uses physical drawings to reproduce real life signals and explains how “Information” travels wirelessly. I recommend this video to the rest of the class to help us gather more simple terminology of this process.

James Gleick, The Information: A History, a Theory, a Flood. (New York, NY: Pantheon, 2011).
Excerpts from Introduction and Chap. 7.

Martin Irvine, “Introduction to the Technical Theory of Information” 2019.

https://www.youtube.com/watch?v=Ax7dYaRiY6o

(MIT VIDEO)