I also changed the word “fair & lovely” to more neutral phrases such as “your product” and “this product.” The output result changed. However, from a human analytical standpoint the message — and its sentiment — are the same.

At this point, I wouldn’t use either of these tools for this specific case! You tell me if the sentence “Unilever- cancel fair & lovely -sign the petition!” is joyous! 🤦🏻‍♀️

However, let’s say our hypothetical developer still thinks that there are benefits for using these tools.

In that case, we’d need to take into account the following criteria. I have to say this list is very preliminary and by no means comprehensive. But at least it gives you a sense of what to look for if you, as a developer, decide to use these tools.

Registration: privacy policies and terms of service

When you want to sign up for a developer account, always read the complete Terms of Service (ToS) and privacy policy. Crucially, this is different from a company website’s terms and policies.

In particular, be vigilant for information about how the data you provide as input is going to be handled. There is a service called Polisis that helps you to compare policies from different service providers (it’s not perfect, but it is still helpful).

Read the developer’s privacy policy and product-specific policy to understand what data from you (as an account holder) the company collects, how they protect them? Do they encrypt the data at rest and in transit? Is the data they collect personally identifiable? Do they define what they mean by metadata? Do they collect the data you provide as an input to the service? Do they retain it? For how long? Do they keep the log files?

Here’s a comparison between the policies of the two services. (For the rest of this post, my comparisons between the details of IBM Tone Analyzer and ParallelDots will appear in gray text boxes like the one below, featuring summaries of what I found in their posted policies and documentation).

IBM Tone Analyzer: When you want to create a developer account, IBM points you to their general When you want to create a developer account, IBM points you to their general privacy policy that lists everything from website visit to using cloud services. There are some vague statement such as: • "IBM may also share your personal information with selected partners to help us provide you ..." Who are their partners though?! • Or "We will not retain personal information longer than necessary to fulfill the purposes for which it is processed." What is "longer than necessary?" If you want specific information about data collection and retention via Tone Analyzer API go to the product document page . Some relevant information includes: • "Request logging is disabled for the Tone Analyzer service.the service does not log or retain data from requests and responses." • The service "processes but does not store users' data. Users of the Tone Analyzer service do not need to take any action to identify, protect, or delete their data for this service." ParallelDots: The The website says that ParallelDots “protects your data and follow the GDPR compliance guidelines to the last word.” But it doesn’t go further? Which data? Metadata or developers’ information or users’ data? • ParallelDots' ParallelDots' ToS says "you may not access the services for purposes of monitoring their availability, performance or functionality, or for any other benchmarking or competitive purposes" This is bizarre to me, does that mean I broke their Tos?!

Documentation

If a company has already provided documentation such as Model Card for Model Reporting for that specific API, read it before using the service. If not, good luck on finding such important information! Look at the API’s documents and dig in for information about API security, background research. papers, training data, architecture and algorithms, evaluation metrics, and recommended use and not-to-use cases.

🔐 API Security. During the past few years, there have been numerous examples of data breaches via the use of insecure APIs. It’s an API provider’s responsibility to be able to detect security vulnerabilities, identify suspicious requests, provide encrypted traffic, and traffic monitoring methods. Make sure an API provider has already put these security practices in place — and read more about APIs security here.

Here’s another comparison, this time comparing API security:

IBM Tone Analyzer: • IBM suggests developers use IBM suggests developers use IBM Cloud Activity Tracker with LogDN to monitor the activity of an IBM Cloud account and investigate abnormal activity. • The service also requires a strong password and sends you a verification code to confirm your developer account. ParallelDots: There is no information about API security in the API document page. However they There is no information about API security in the API document page.However they mentioned that they only provide encrypted access to premium content. • For registration, developers are not required to set a strong password, however, ParallelDots sends you a verification email to confirm your account.

🌏 Accurate and Precise… but for who? In the example of “Fair & Lovely,” language plays an important role. English-only tweets don’t provide an accurate understanding of discussions around the product because it’s not one restricted to a single language.

Check the API to see if it support other languages. If so, what is the accuracy rate for different languages? Service providers often say they support multiple languages, but don’t provide broken down details about accuracy and other evaluation metrics for each language. Dig in the API document, background research pages, and try to find metrics for different sub-categories.

In our case, here’s what I found:

IBM Tone Analyzer: The company listed : The company listed 11 supported languages . However, there is no breakdown information about accuracy or other evaluation metrics based on different languages. ParallelDots: The company listed : The company listed 14 supported languages . However, there is no information about accuracy or other evaluation metrics based on different languages.

❗️Suggested (Not) Use Cases. Companies also provide guidance about the suggested use of their services, but sometimes the use case can be dangerous or unethical. Companies need to be transparent about the cases in which developers should not use their services.

IBM Tone Analyzer: Tone Analyzer use cases according to : Tone Analyzer use cases according to the document page include predicting customer satisfaction in support forums; predicting customer satisfaction in Twitter responses; predicting online dating matches; Predicting TED Talk applause. There is no indication about Not-to-Use Cases. ParallelDots: There are two : There are two suggested use cases : “target[ing] detractors to improve service to them” and “brand-watching.” There is no indication about Not-to-Use Cases.

⚖️ Fairness Practices. In the past couple of years, researchers and practitioners have raised awareness about discriminatory outcomes of machine learning systems. They’ve provided numerous toolkits to help companies assess the human rights implications of their tools and be transparent about potential social risks. I keep track of different initiatives, papers, toolkits here.

But how many companies provide that information for their specific ML APIs?

IBM Tone Analyzer: IBM provides information about background research, data collection process (twitter data), and data annotation method. However there is no mention of potential discrminatory outcomes and no breakdown information about demographics and measurement based on different sub-groups (language, gender, age, etc.) • Fun fact: IBM Research is one of the pioneers in providing fairness and explanability toolkits (check out Fun fact: IBM Research is one of the pioneers in providing fairness and explanability toolkits (check out IBM 360 ). They also proposed using FactSheets for every ML model to show the origin of training datasets, model specifications, and use cases. But when it comes to their own model, you rarely find such information on their product pages! This reminded me of the great piece of poem from Nizami , basically meaning first fix your own flaws before being too critical of others: عیب کسان منگر و احسان خویش دیده فرو کن بگریبان خویش ParallelDots: I found no information about fairness practices.

🛠 Maintenance and Updates

IBM Tone Analyzer: The company frequently update the service and provides information about the The company frequently update the service and provides information about the updates . However, in some updates there are generic sentecs including "The service was also updated for internal changes and improvements." What are those internal changes and improvements? ParallelDots: I couldn't find information about updates and maintenance.

💬 Developers Community. Communities of developers(via Slack Workspace, Stack Overflow, GitHub, etc) help share feedback, interact with themselves and service providers, and raise issues around privacy, security, fairness, explainability about a certain product and in a specific domain.

Source: Medium

Powered by NewsAPI.org