「详细教程」基于Algorithmia工具将NLP模型部署为API-FlyAI

「详细教程」基于Algorithmia工具将NLP模型部署为API

自然语言处理

最后更新 2020-12-16 15:23 阅读 176

最后更新 2020-12-16 15:23

阅读 176

自然语言处理

##FlyAI文献翻译

**英文原文：**[：How to Deploy your NLP Model to Production as an API with Algorithmia](https://www.freecodecamp.org/news/deploy-ml-model-to-production-as-api/)

> **标签** ：自然语言处理

Did you know that 90% of machine learning models never actually make it into production?

This means that the topic of machine learning deployment is rarely discussed when people learn machine learning. As a result, many AI practitioners know how to create useful ML models, but they find it difficult to deploy them into production.

Needless to say, machine learning deployment is one of the more important skills you should have if you're going to work with ML models.

>你知道90%的机器学习模型从未真正投入生产吗？

>这意味着，当人们学习机器学习时，很少讨论机器学习部署的话题。因此，许多AI从业者知道如何创建有用的ML模型，但他们发现很难将它们部署到生产中。

>毋庸置疑，如果你要使用ML模型，机器学习部署是你应该掌握的比较重要的技能之一。

![](https://static.leiphone.com/uploads/new/sns/article/202011/1605686942268720.jpeg)

Model deployment is the process of integrating your model into an existing production environment. The model will receive input and predict an output for decision making for a specific use case.

>模型部署是将您的模型集成到现有生产环境的过程。该模型将接收输入并预测输出，以便为特定的用例进行决策。

For example, a model can be deployed in an e-commerce site and it can predict if a review about a specific product is positive or negative.

>例如，可以在电子商务网站中部署一个模型，它可以预测关于特定产品的评论是正面还是负面。

Only when a model is fully integrated with the business systems, we can extract real value from its predictions. - Christopher Samiullah

>只有当一个模型与业务系统完全集成时，我们才能从它的预测中提取真正的价值。- 克里斯托弗-萨米乌拉

There are different ways you can deploy your machine learning model into production. But in today's article, you will learn how to deploy your NLP model into production as an API with Algorithmia.

>你可以通过不同的方式将你的机器学习模型部署到生产中。但在今天的文章中，你将学习如何通过Algorithmia将你的NLP模型作为API部署到生产中。

In this article, you will learn:

How to create an NLP model that detects spam SMS text messages

How to use Algorithmia, a MLOps platform.

How to deploy your model into the Algorithmia platform

How to use your deployed NLP model in any Python application.

>在这篇文章中，你将学习

>- 如何创建一个检测垃圾短信的NLP模型？
- 如何使用Algorithmia这个MLOps平台。
- 如何将你的模型部署到Algorithmia平台上。
- 如何在任何Python应用中使用你部署的NLP模型。

Our first step is to create a machine learning model that can detect spam SMS text messages. So let't get started!

>我们的第一步是创建一个可以检测垃圾短信的机器学习模型。那么让我们开始吧!

###How to Build the ML Model

First, we need to build our model. Here are the steps you should follow to do that.

>###如何建立ML模型

>首先，我们需要建立我们的模型，下面是步骤。

###Import Python Packages

We first import all the importance python packages that we will use to load the data, preprocess the data, and create a text classification model.

>###导入Python包

>我们首先导入所有的重要性python包，我们将用于加载数据，预处理数据，并创建文本分类模型。

```
# import important modules
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from string import punctuation

# sklearn modules
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn.svm import SVC
from sklearn.metrics import (
    accuracy_score,
    classification_report,
    plot_confusion_matrix,
    f1_score,
    roc_auc_score,
)
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.model_selection import cross_val_score, RandomizedSearchCV
# text preprocessing modules
from nltk.tokenize import word_tokenize
from cleantext import clean
import nltk
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer 
import re #regular expression
from wordcloud import WordCloud, STOPWORDS
# Download dependency
for dependency in (
    "brown",
    "names",
    "wordnet",
    "averaged_perceptron_tagger",
    "universal_tagset",
    "stopwords"):
    nltk.download(dependency)#nltk.download('stopwords')import warnings
warnings.filterwarnings("ignore")# seedingnp.random.seed(123)\
```
###Load the Spam Dataset

Then we load the spam dataset from the data directory, like this:

>###加载垃圾邮件数据集

>然后我们从数据目录中加载垃圾邮件数据集，像这样。

```
# load data
data = pd.read_csv("../data/spam.tsv", sep="\t")
```
Let's look at the first five rows of the dataset.

>让我们看看数据集的第五行。

```
# show top five rows
data.head()
```
![](https://static.leiphone.com/uploads/new/sns/article/202011/1605687141465668.png)

The dataset has four columns, but we will focus only on the message and label columns.

Let’s check out the shape of the dataset:

>数据集有四列，但我们只关注消息和标签列。

>让我们来看看数据集的形状。

```
# check the shape
data.shape
```
output: (5572, 4)

We have 5572 rows and 4 columns.

>(5572, 4) (5572, 4)

>我们有5572行，4列

###How to Handle Missing Values

Sometimes data can have missing values. We can use the isnull() method from pandas to check if our dataset has any missing values.

>###如何处理缺失的值

>有时数据可能会有缺失值。我们可以使用pandas的isnull()方法来检查我们的数据集是否有缺失值。

![](https://static.leiphone.com/uploads/new/sns/article/202011/1605687205139675.png)

The output shows that our dataset does not have any missing values.

>输出结果显示，我们的数据集没有任何缺失值。

###How to Evaluate Class Distribution

We can use the value_counts() method from the pandas package to evaluate the class distribution form our dataset.

>###如何评估类分布

>我们可以使用pandas包中的value_counts()方法来评估数据集的类分布。

```
# evalute class distribution
data["label"].value_counts()
```
![](https://static.leiphone.com/uploads/new/sns/article/202011/1605687228457250.png)

In this dataset we have more legitimate messages (ham) than spam messages.

>在这个数据集中，我们的合法信息（ham）比垃圾信息多。

Exploratory Data Analysis

This is very important steps in creating your machine learning project. It helps you understand more about your dataset.

In this step we are going to find frequent words that are used in both legitimate and spam messages.

>###探索性数据分析

>这是在创建你的机器学习项目中非常重要的步骤。它可以帮助你更多地了解你的数据集。

>在这一步中，我们要找到合法和垃圾信息中经常使用的词。

```
# collect words from the datasetdef collect_words(data, label):
    collected_words = " "

# iterate through the csv file
    for val in data.message[data["label"] == label]:

# typecaste each val to string
        val = str(val)

# split the value
        tokens = val.split()

# Converts each token into lowercase
        for i in range(len(tokens)):
            tokens[i] = tokens[i].lower()

for words in tokens:
            collected_words = collected_words + words + " "

return collected_words
	```

The above function called collect_words() will collect all words from the dataset according to their labels (ham or spam).

Then we can visualize frequent words by using the wordcloud Python package. We will start with messages labeled as ham (legitimate).

>上面的函数名为collect_words()，会根据数据集的标签(合法信息或垃圾邮件)收集数据集中的所有单词。

>然后，我们可以通过使用wordcloud的Python包来可视化频繁的单词。我们将从标记为合法信息开始。

```
# visualize ham labeled sms
cloud_stopwords = set(STOPWORDS)
ham_words = collect_words(data, label="ham")
print("Total words {}".format(len(ham_words)))
wordcloud = WordCloud(
    width=1000,
    height=1000,
    background_color="white",
    stopwords=cloud_stopwords,
    min_font_size=10,).generate(ham_words)
# plot the WordCloud imageplt.figure(figsize=(15, 8), facecolor=None)
plt.imshow(wordcloud)
plt.axis("off")
plt.tight_layout(pad=0)
plt.show()
```
总词数：349132

![](https://static.leiphone.com/uploads/new/sns/article/202011/1605687298207614.png)

As you can see in the legit messages, the most frequent words are will, gt, now, ok, call, want, got, and so on.

Now we can visualize the most frequent words in messages labeled as spam.

>在合法信息中可以看到，最常出现的词是will、gt、now、ok、call、want、got等。

>现在，我们可以直观地看到标有垃圾邮件的消息中最常出现的词。

```
# visualize spam labeled sms
cloud_stopwords = set(STOPWORDS)
spam_words = collect_words(data, label="spam")
print("Total words {}".format(len(spam_words)))
wordcloud = WordCloud(
    width=1000,
    height=1000,
    background_color="white",
    stopwords=cloud_stopwords,
    min_font_size=10,).generate(spam_words)
# plot the WordCloud imageplt.figure(figsize=(10, 8), facecolor=None)
plt.imshow(wordcloud)
plt.axis("off")
plt.tight_layout(pad=0)
plt.show()
```

总词数：104304

![](https://static.leiphone.com/uploads/new/sns/article/202011/1605687386560106.png)

In the above figure it shows that the most frequent words are those like call, claim, free, txt, mobile, reply, offer and so on.

>在上图中可以看出，出现频率最高的词是像call、claim、free、txt、mobile、reply、offer等。

###How to Process the Data

After exploring and analyzing the dataset, the next step is to preprocess the dataset into the right format before creating our machine learning model.

We first replace the ham and spam classes with numerical values. The ham class will be labeled as 0 and spam class will be labeled as 1.

>###如何处理数据

>在探索和分析数据集之后，下一步就是在创建我们的机器学习模型之前，将数据集预处理成正确的格式。

>我们首先用数值替换合法类和垃圾邮件类。合法类将被标记为0，垃圾邮件类将被标记为1。

```
 replace ham to 0 and spam to 1
new_data = data.replace({"ham": 0, "spam": 1})
new_data.head()
```

![](https://static.leiphone.com/uploads/new/sns/article/202011/1605687419229847.png)

The messages in this dataset contain a lot of unnecessary words and characters that we don't need when creating machine learning models.

We will clean the messages by removing stopwords, numbers, and punctuation. Then we will change words into lower case, and finally convert each word into its base form by using the lemmatization process in the NLTK package.

The text_cleaning() function will handle all necessary steps to clean our dataset.

>这个数据集中的消息包含了很多不必要的单词和字符，我们在创建机器学习模型的时候不需要。

>我们将通过删除停止符、数字和标点符号来清理消息。然后我们将把单词改成小写，最后通过使用NLTK包中的词法化过程把每个单词转换成其基本形式。

>text_cleaning()函数将处理清理数据集的所有必要步骤。

```
stop_words =  stopwords.words('english')

def text_cleaning(text, remove_stop_words=True, lemmatize_words=True):
    # Clean the text, with the option to remove stop_words and to lemmatize word

# Clean the text
    text = re.sub(r"[^A-Za-z0-9]", " ", text)
    text = re.sub(r"\'s", " ", text)
    text = re.sub(r"n't", " not ", text)
    text = re.sub(r"I'm", "I am", text)
    text = re.sub(r"ur", " your ", text)
    text = re.sub(r" nd "," and ",text)
    text = re.sub(r"\'d", " would ", text)
    text = re.sub(r"\'ll", " will ", text)
    text = re.sub(r" tkts "," tickets ",text)
    text = re.sub(r" c "," can ",text)
    text = re.sub(r" e g ", " eg ", text)
    text =  re.sub(r'http\S+',' link ', text)
    text = re.sub(r'\b\d+(?:\.\d+)?\s+', '', text) # remove numbers
    text = re.sub(r" u "," you ",text)
    text = text.lower()  # set in lowercase 
        
    # Remove punctuation from text
    text = ''.join([c for c in text if c not in punctuation])
    
    # Optionally, remove stop words
    if remove_stop_words:
        text = text.split()
        text = [w for w in text if not w in stop_words]
        text = " ".join(text)
    
    # Optionally, shorten words to their stems
    if lemmatize_words:
        text = text.split()
        lemmatizer = WordNetLemmatizer() 
        lemmatized_words = [lemmatizer.lemmatize(word) for word in text]
        text = " ".join(lemmatized_words)
    
    # Return a list of words
    return(text)
	```

Now we can clean our dataset by using the text_cleaning() function.

>现在我们可以通过使用text_cleaning()函数来清理我们的数据集。

```
#clean the dataset 
new_data["clean_message"] = new_data["message"].apply(text_cleaning)
```
We then split our dataset into train and test data. The test size is 15% of the entire dataset.

>然后我们将数据集分为训练数据和测试数据。测试大小为整个数据集的15%。

```
# split data into train and testX_train, X_test, y_train, y_test = train_test_split(
    new_data["clean_message"],
    new_data["label"],
    test_size=0.15,
    random_state=0,
    shuffle=True,
    stratify=data["label"],)
	```

The CountVectorizer method from scikit-learn will help us transform our cleaned dataset into numerical values. The method converts a collection of text documents to a matrix of token counts.

>scikit-learn中的CountVectorizer方法将帮助我们将清理后的数据集转化为数值。该方法将文本文档集合转换为标记计数的矩阵。

```
# Transform text data 
vectorizer = CountVectorizer(lowercase=False)
vectorizer.fit(X_train)
#transform train data 
X_train_trans = vectorizer.transform(X_train)
#transform test data
X_text_trans = vectorizer.transform(X_test)
```

###How to Actually Create Our Model

We will train the Multinomial Naive Bayes algorithm to classify if a message is legitimate or spam. This is one of the most common algorithms used for text classification.

>###如何真正创建我们的模型

>我们将训练Multinomial Naive Bayes算法来分类信息是合法的还是垃圾邮件。这是最常用的文本分类算法之一。

```
# Create a classifier
spam_classifier = MultinomialNB()
```
Then we train our classifier by using cross validation to avoid overfitting.

>然后我们通过使用交叉验证来训练我们的分类器，以避免过拟合。

```
# Train the model with cross validation
scores = cross_val_score(spam_classifier,X_train_trans,y_train,cv=10,verbose=3,n_jobs=-1)
```
![](https://static.leiphone.com/uploads/new/sns/article/202011/1605687508348764.png)

Let's see the mean score

```
# find the mean of the all scores
scores.mean()
```
output: 0.9767713936539371

The mean of the scores is around 97.68%. Our model performs well, but we can improve its performance by optimizing its hyperparameter values with the Randomized Search method from scikit-learn.

>输出：0.9767713936539371

>分数的平均值约为97.68%。我们的模型表现良好，但我们可以通过scikit-learn的随机搜索方法优化其超参数值来提高其性能。

```
# fine turning model parameters
distribution = {"alpha": [1, 0.1, 0.01, 0.001, 0.0001, 0, 0.2, 0.3]}grid = RandomizedSearchCV(
    spam_classifier,
    param_distributions=distribution,
    n_jobs=-1,
    cv=10,
    n_iter=20,
    random_state=42,
    return_train_score=True,
    verbose=2,)
	```

We will optimize the alpha hyparameter from our model to get the best value that will increase our model's performance.

>我们将优化模型中的α超参数，以获得最佳值，从而提高模型的性能。

```
# training with randomized search
grid.fit(X_train_trans, y_train)
```

![](https://static.leiphone.com/uploads/new/sns/article/202011/1605687549263020.png)

要显示超参数优化结果：

```
# summarize the results of the random parameter search
print(grid.best_score_)
print(grid.best_estimator_)
print(grid.best_params_)
```
0.9767713936539371
MultinomialNB(alpha=1)
{'alpha': 1}

The best score is the same as the previous one. Now let's test our model with the test data.

>0.9767713936539371
MultinomialNB(alpha=1)
{'alpha': 1}

>最好的成绩和之前的一样。现在我们用测试数据来测试我们的模型。

```
# predict on the test data
y_pred = spam_classifier.predict(X_text_trans)
```

The model's performance will be evaluated by using the accuracy_score evaluation metric.

>该模型的性能将通过使用accuracy_score评估指标进行评估。

```
# check accuracy score
accuracy_score(y_test, y_pred)
```

output: 0.9760765550239234

The accuracy of our model is around 97.6%, which is good performance.

Another useful evaluation metric to use is f1_score when you have a class imbalance in your dataset.

>输出：0.9760765550239234

>我们模型的准确率约为97.6%，表现不错。

>另一个有用的评估指标是f1_score，当你的数据集中存在类不平衡时，可以使用这个指标。

```
# check f1_ score
f1_score(y_test, y_pred)
```
output: 0.908256880733945

The score is 0.91 which is closer to 1. This means that our model has good performance and we can now deploy it to production.

The model will be saved in models directory.

>输出：0.908256880733945 0.908256880733945

>分数是0.91，接近于1，这意味着我们的模型有良好的性能，现在我们可以将它部署到生产中。

>该模型将保存在models目录下。

```
#save model 
import joblib 
joblib.dump(spam_classifier, '../models/spam-detection-model.pkl')
```
['../models/spam-detection-model.pkl']

Our  Count Vectorizer will also be saved in the preprocessing directory

>['./models/spam-detection-model.pkl']

>我们的计数向量器也将被保存在预处理目录中。

```
#save Vectorizer
joblib.dump(vectorizer,'../preprocessing/count_vectorizer.pkl')

```
['../preprocessing/count_vectorizer.pkl']

After creating our spam detection model, it's time to deploy it on the Algorithmia platform.

>['./preprocessing/count_vectorizer.pkl']

>在创建了我们的垃圾邮件检测模型后，就可以在Algorithmia平台上进行部署了。

###What is Algorithmia?

Algorithmia is a MLOps tool that provides a simple and faster way to deploy your machine learning model into production.

Algorithmia specializes in "algorithms as a service". It allows users to create code snippets that run the ML model and host them on Algorithmia. Then you can call your code as an API.

Now your model can be used for different applications of your choice, such as web apps, mobile apps, or e-commerce with a simple API call from Algorithmia.

>###什么是Algorithmia？

>Algorithmia是一个MLOps工具，它提供了一个简单和快速的方法来部署你的机器学习模型到生产中。

>Algorithmia专注于 "算法即服务"。它允许用户创建运行ML模型的代码片段，并将其托管在Algorithmia上。然后，你可以将你的代码作为API调用。

>现在，你的模型可以用于你所选择的不同应用，如网络应用、移动应用或电子商务，只需从Algorithmia调用一个简单的API即可。

![](https://static.leiphone.com/uploads/new/sns/article/202011/1605687621959936.png)

Algorithmia supports machine learning models developed by different programming languages such as R, Python, Java, and Scala. It also support popular machine and deep learning frameworks such as Pytorch, Tensorflow, scikit-learn, XGBoost, and Keras.

Algorithmia uses both CPUs and GPUs on its serverless Artificial Intelligence layer to optimize cost and maximize its performance to match your needs.
Currently, this platform has over 60,000 developers with 4,500 algorithms.

Here are six steps you need to follow to deploy your machine learning model on Algorithmia.

>Algorithmia支持不同编程语言开发的机器学习模型，如R、Python、Java和Scala。它还支持流行的机器和深度学习框架，如Pytorch、Tensorflow、scikit-learn、XGBoost和Keras。

>Algorithmia在其无服务器的人工智能层上同时使用CPU和GPU，以优化成本并最大限度地提高其性能，以满足您的需求。

>目前，这个平台有超过6万名开发者，拥有4500种算法。

>以下是你在Algorithmia上部署机器学习模型需要遵循的六个步骤。

Step 1: Create an Account on Algorithmia

The first step is to create an account in Algorithmia by visiting this page: https://algorithmia.com/signup.

>步骤一：在Algorithmia上建立账户

>第一步是访问这个网页：https://algorithmia.com/signup，在Algorithmia创建一个账户。

Step 2: Create a New Algorithm
After creating and confirming your account and email, the next step is to create a new algorithm by clicking the dropdown menu button named "Create New". Then you just select Algorithm at the top right corner of the page.

>第二步：创建一个新算法

>在创建并确认你的账户和邮箱后，下一步就是点击名为 "创建新算法 "的下拉菜单按钮，创建一个新的算法。然后选择页面右上角的 "算法 "即可。

![](https://static.leiphone.com/uploads/new/sns/article/202011/1605687639897745.png)

Then enter the algorithm's name, for example SMS SPAM DETECTION. In the source code section you can determines where your algorithm's source code will live.

By default source code will be on the Algorithmia platform. You can choose to keep it on GitHub, but for this article we will use the default option.

The next section specifies the environment. Algorithmia gives you different options to select different environment such as Python, R, JavaScript, Java and Scala. The default option is Python 3.X, and here we will continue with this option. Finally, click the "Create new Algorithm" button.

>然后输入算法的名称，例如SMS SPAM DETECTION。在源码部分，你可以决定你的算法的源码存放在哪里。

>默认情况下，源代码会放在Algorithmia平台上。你也可以选择把它放在GitHub上，但在本文中我们将使用默认选项。

>下一节是指定环境。Algorithmia为你提供了不同的选项来选择不同的环境，如Python、R、JavaScript、Java和Scala。默认选项是Python 3.X，这里我们将继续使用这个选项。最后，点击 "创建新算法 "按钮。

Step 3: Upload the pre-trained Model & CountVectorizer to Algorithmia

You can upload your selected model to the data section by clicking Data Sources on the left panel of the Algorithmia platform. Then click the My hosted data directory where you can create a new folder to keep all your uploaded pkl files for this specific algorithm.

Inside the "My hosted data" directory I created a new folder called sms_spam_detection. Then I uploaded our pre-trained model and trained CountVectorizer to convert text messages (sms) to a vector of term/token counts.

>第三步：将预先训练好的模型和CountVectorizer上传至Algorithmia

>你可以通过点击Algorithmia平台左侧面板的数据源，将你选择的模型上传到数据部分。然后点击 "我的托管数据 "目录，在这里你可以创建一个新的文件夹来保存你上传的这个特定算法的所有pkl文件。

>在 "我的托管数据 "目录下，我创建了一个新的文件夹，名为sms_spam_detection。然后，我上传了我们的预训练模型，并训练了CountVectorizer，以将文本信息（短信）转换为术语/标记计数的向量。

![](https://static.leiphone.com/uploads/new/sns/article/202011/1605687656741084.png)

Step 4: Add the source code

After uploading our pre-trained model, click on the Source Code tab. It will open an IDE where you add the source code to run the machine learning model we created. Here's how you add the source code:

(a) Import the packages
We first import important Python packages including Algorithmia that will call the algorithm we created.

>第四步：添加源码

>上传我们的预训练模型后，点击源代码标签。它将打开一个IDE，在这里你可以添加源代码来运行我们创建的机器学习模型。下面是你如何添加源代码。

>(a) 导入软件包

>我们首先导入重要的Python包，包括Algorithmia，它将调用我们创建的算法。

```
import sys
import joblib
import pickle
import numpy as np
import Algorithmia
from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
from string import punctuationimport re 
import nltk
#download dependency
nltk.download('stopwords')
nltk.download('wordnet')
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
```

NB: Algorithmia's platform will automatically import the Algorithmia Python package in the file.

(b) Create the Client
We then create a client from the Algorithmia package which provides a standardized way of calling any Algorithm.

>注：Algorithmia平台会自动导入文件中的Algorithmia Python包。

>(b) 创建客户端

>然后，我们从Algorithmia包中创建一个客户端，它提供了一种调用任何算法的标准化方法。

```
# we are creating the variable in global scope to use throughout our algorithm.
client = Algorithmia.client()
```

(c) Add functions to load the pre-trained model and countvectorizer pkl files
The load_model() function will load our pre-trained model from the data source directory and the load_preprocessing() function will load the countvectorizer file.

>(c) 增加加载预训练模型和计数矢量器pkl文件的功能。

>load_model()函数将从数据源目录中加载我们预训练的模型，load_preprocessing()函数将加载countvectorizer文件。

```
def load_preprocessing():
    # Get file by name
    # Open file and load model
    file_path = 'data://Davis/sms_spam_detection/count-vectorizer.pkl'
    object_path = client.file(file_path).getFile().name    # Open file and preprocessin object
    with open(object_path, 'rb') as f:
        object = joblib.load(f)
        return object    def load_model():
    # Get file by name
    # Open file and load model
    file_path = 'data://Davis/sms_spam_detection/spam-detection-model.pkl'
    model_path = client.file(file_path).getFile().name    # Open file and load model
    with open(model_path, 'rb') as f:
        model = joblib.load(f)
        return model
# Load model outside of the apply function so it only gets loaded once
model = load_model()
vectorizer = load_preprocessing()
```
(d) Add the function to clean the text input

Here we will use the same text_clean() function to clean the sms messages.

>(d) 增加清理文本输入的功能。

>这里我们将使用同样的text_clean()函数来清理短信。

```
#set stopwordsstop_words = stopwords.words('english')def text_cleaning(text, remove_stop_words=True, lemmatize_words=True):
    # Clean the text, with the option to remove stop_words and to lemmatize word

# Remove punctuation from text
    text = ''.join([c for c in text if c not in punctuation])

# Optionally, remove stop words
    if remove_stop_words:
        text = text.split()
        text = [w for w in text if not w in stop_words]
        text = " ".join(text)

# Optionally, shorten words to their stems
    if lemmatize_words:
        text = text.split()
        lemmatizer = WordNetLemmatizer()
        lemmatized_words = [lemmatizer.lemmatize(word) for word in text]
        text = " ".join(lemmatized_words)

# Return a list of words
    return (text)
	```

(e) Add the function to pre-process the input
The process_input() method will preprocess the input sms before making a prediction.

>(e) 增加预处理输入的功能。

>process_input()方法将对输入的短信进行预处理，然后再进行预测。

```
def process_input(input):
    # Preprocess and Create numpy array from the input
    
    message = str(input)
    clean_message = text_cleaning(message)
    
    #vectorize the message 
    vect_message = vectorizer.transform([clean_message])
    
    print(vect_message)
    
    return vect_message

```
(f) Add the function to apply the model for prediction
The last function called apply() will be responsible for performing predictions from the pre-processed text sms. It will return "normal message" if the message is legitimate and "spam message" if the message is spam.

>(f) 增加应用模型进行预测的功能。

>最后一个名为apply()的函数将负责从预处理的文本短信中执行预测。如果信息是合法的，它将返回 "正常信息"，如果信息是垃圾信息，则返回 "垃圾信息"。

```
def apply(input):
    # pefrom prediction from the input. 
   
    message = process_input(input)
    prediction = model.predict(message)
    if prediction[0] == 0:
        return "normal message"
    else:
        return "spam message"
		```

Finally we save the source code added in the file.

>最后我们保存文件中添加的源码。

![](https://static.leiphone.com/uploads/new/sns/article/202011/1605687759439504.png)

Step 5: Add Dependencies to Algorithmia

Click the dependencies tab from the UI and add the following packages that our model relies on:

numpy

joblib

scikit-learn == 0.22.2.post1

unidecode

nltk == 3.5

Then  click save dependencies in the bottom right corner:

>第5步：为算法添加依赖关系

>点击用户界面中的环境依赖，并添加以下我们的模型所依赖的包：

>- numpy

>- joblib

>- scikit-learn == 0.22.2.post1

>- unidecode

>- nltk == 3.5

>然后点击右下角的保存依赖关系：

![](https://static.leiphone.com/uploads/new/sns/article/202011/1605687776230873.png)

NB: The dependency file in Algorithmia is the same as a requirements.txt file which pulls the dependencies listed from PyPi.

On the editor page click build tab at the top right to install all dependencies listed in the dependency file. If all dependencies are successfully installed, you will see that a specific version of your algorithm is now online and ready to be published.

>注意：Algorithmia中的依赖文件和request.txt文件是一样的，它从PyPi中提取了所有的依赖文件。

>在编辑器页面上点击右上角的构建选项卡，安装依赖文件中列出的所有依赖文件。如果所有的依赖项都成功安装，你便会看到你的算法特定版本已经上线并准备发布了。

![](https://static.leiphone.com/uploads/new/sns/article/202011/1605687793897124.png)

Step 6: Publish the Algorithm

Our last step is to publish the algorithm. There are 3 steps to publishing an algorithm: documenting all changes, adding example input and output, and configuring the algorithm settings.

(a) Document Changes
You will see your commit history and you'll be able to add a release note.

>第六步：发布算法

>我们的最后一步是发布算法。发布算法有3个步骤：记录所有更改、添加示例输入和输出、配置算法设置。

>(a) 记录更改

>你会看到你的提交历史，并可以添加发布说明

![](https://static.leiphone.com/uploads/new/sns/article/202011/1605687825674491.png)

(b) Add an Example

In this section you create your sample input and output so that users can try your algorithm.

>(b) 增加一个例子

>在本节中，你要创建你的输入和输出样本，以便用户可以尝试你的算法。

![](https://static.leiphone.com/uploads/new/sns/article/202011/1605687847112529.png)

Lastly, you select if your algorithm will be public (which means anyone can call it) or private (which means only owners can call it). You also need to set the royalty and then click the publish button at the bottom of the page.Our NLP model has successfully been deployed on the Algorithmia platform. Let's see how we can use our deployed model in a Python application such as Flask or Django.

>(c) 配置算法设置

>最后，你选择你的算法是公开的（这意味着任何人都可以调用它）还是私有的（这意味着只有所有者可以调用它）。你还需要设置版权费，然后点击页面底部的发布按钮。我们的NLP模型已经成功部署在Algorithmia平台上。让我们看看如何在Python应用中使用我们部署的模型，比如Flask或Django。

![](https://static.leiphone.com/uploads/new/sns/article/202011/1605687892753675.png)

How to Install the Algorithmia Python Client

We first install the Algorithmia Python Client by using PIP. This will help us call our code that runs the NLP model.

>如何安装Algorithmia Python客户端？

>我们首先通过使用PIP安装Algorithmia Python客户端。这将帮助我们调用我们运行NLP模型的代码。

```
pip install algorithmia
```
Collect the API Key

Click the "API Key" tab on your algorithm panel to collect the API key that will help you call the code that runs the NLP model.

Create the Algorithmia Client

We first import the Algorithmia Python package and then create the algorithm client object

>收集API密钥

>在你的算法面板上点击 "API密钥 "标签，收集API密钥，这将帮助你调用运行NLP模型的代码。

>创建Algorithmia客户端

>我们首先导入Algorithmia Python包，然后创建算法客户端对象。

```
import Algorithmia
# Authenticate with your API key
apiKey = "YOUR_API_KEY"
# Create the Algorithmia client object
client = Algorithmia.client(apiKey)
```
Call the Algorithm

To call the algorithm, we need to add the name of the algorithm with its version to the client object we have created.

The name is "Davis/spam_detection/0.2.0", which includes the name of your account on Algorithmia followed by the name of the algorithm we created. The final bit is the version of the algorithm (0.2.0).

>调用算法

>为了调用算法，我们需要在我们创建的客户端对象中添加算法的名称及其版本。

>名称是 "Davis/spam_detection/0.2.0"，其中包括你在Algorithmia上的账户名称，后面是我们创建的算法名称。最后一位是算法的版本（0.2.0）。

```
# Create the algorithm object using the Summarizer algorithm
algo = client.algo('Davis/spam_detection/0.2.0')

# Pass in input required by algorithm
input_sms = "Win a Â£1000 cash prize or a prize worth Â£5000"

try:
    # Get the result
    print(algo.pipe(input_sms).result)
except Exception as error:
    # Algorithm error if, for example, the input is not correctly formatted
    print(error)
	```

For this example the model predicts that the sms is a "spam message". Cool, it's working!

###Wrapping Up

Congratulations, you have made it to the end of this article!

You can download the dataset and notebook used in this article here: https://github.com/Davisy/SMS-Spam-Text-Classification

>###总结

>恭喜你，你已经走到了这篇文章的结尾！你可以在这里下载本文使用的数据集和代码。

>您可以在这里下载本文中使用的数据集和代码：https://github.com/Davisy/SMS-Spam-Text-Classification

本作品采用知识共享署名-非商业性使用-相同方式共享 4.0 国际许可协议进行许可，转载请附上原文出处链接和本声明。
本文链接地址：https://www.flyai.com/article/786

讨论

500字

表情

每日优质讨论奖励 20FAI

发送

每日优质讨论奖励 20FAI

删除确认

是否删除该条评论？

取消删除