Intelligent machine learning TensorFlow Kam Wong
Foreword
Recently doing a drawing-down service, public concern small number of partners must remember that small qi has said, will be released over the weekend, I am sorry to put everyone pigeons. The reason has dared release, Wong Kam is because the interface has not been debugged, although I have a small number of male partners trust one hundred percent, but since the Internet was mixed, in case something indecent pictures upload, then unfortunately is a department found I include my domain name server may completely burned out!
Architecture
As shown, first talk about the architecture diagram bed:
-
Nginx
That agency is a necessary artifact. -
lua
Current limit is certain, though nothing pre-sub-flow, the latter may not have. -
Limit the size limit is necessary, or else the bandwidth can not stand.
-
Access Kam Wong, after all, we are filing legal website.
-
Multiple file backup, OSS, distributed file, local file copy each to prevent wandered off.
-
In order to facilitate inquiries, library last fall.
tool
-
SpringBoot, a simplified
Spring
framework for development. -
WebUploader, a simple order
HTML5
-based,FLASH
supplemented by modern file upload component. -
Python, blessing various open source third-party library with pictures.
-
nsfw_data_scraper, a near-
1w
starred collection of data resources. -
docker_nsfw_data_scraper, used to collect training data.
-
TensorFlow, open source machine learning library.
-
ResNet, pre-training model image classification.
-
TensorFlow-serving, deployment
tensorflow
models, and provide services.
Trainer
Before training, introduce a noun NSFW
, before I do not know What do you mean, after all, is the village come, what Takizawa Lola, Kaede Matsushima, Maria Ozawa, Akiho, Yui Hatano, Tianhai wing, Sakurai Leah, Iijima, Aoi, Nozomi Aso, orange Risa, Muto blue, Sawai bud clothes ..... is not and will not know.
NSFW:不适合在工作场合出现的内容(英语:Not Safe/Suitable For Work,缩写:NSFW)是一个网络用语,多指裸露、暴力、色情或冒犯等不适宜公众场合的内容。在给出含有上述内容的超链接旁标注 NSFW,用于警告观看者。
在 nsfw_data_scraper
上传存放了成千上万张图片地址,并对图片进行了分类,以供训练:
- 绘画(Drawing),无害的艺术,或艺术绘画;
- 变态(Hentai),色情艺术,不适合大多数工作环境;
- 中立(Neutral),一般,无害的内容;
- 色情(Porn),不雅的内容和行为,通常涉及生殖器;
- 性感(Sexy),不合时宜的挑衅内容。
同时,官方也提供了收集方法:
$ docker build . -t docker_nsfw_data_scraper
Sending build context to Docker daemon 426.3MB
Step 1/3 : FROM ubuntu:18.04
---> 775349758637
Step 2/3 : RUN apt update && apt upgrade -y && apt install wget rsync imagemagick default-jre -y
---> Using cache ---> b2129908e7e2 Step 3/3 : ENTRYPOINT ["/bin/bash"] ---> Using cache ---> d32c5ae5235b Successfully built d32c5ae5235b Successfully tagged docker_nsfw_data_scraper:latest $ # Next command might run for several hours. It is recommended to leave it overnight $ docker run -v $(pwd):/root docker_nsfw_data_scraper /root/scripts/runall.sh Getting images for class: neutral ... ... $ ls data test train $ ls data/train/ drawings hentai neutral porn sexy $ ls data/test/ drawings hentai neutral porn sexy
如何训练模型,后面也很贴心的附上了训练方法,不过这里借用了 TensorFlow
的 ResNet
的模型,稍作修改。训练过程太过煎熬、痛苦,已经被湮灭在有限的带宽和无尽的小黄图中。
鉴黄服务
模型数据训练好以后就是搭建服务了,这里我们直接使用TensorFlow
的 TensorFlow-serving
对外提供服务,为了安装方便,我们使用Docker
安装部署。
NSFWDATA="/home/nsfw"
docker run -d --rm -p 8501:8501 \
--name nsfw \
-v "$NSFWDATA/models:/models/nsfw" \ -e MODEL_NAME=nsfw \ tensorflow/serving
serving
镜像提供了两种调用方式:gRPC
和HTTP
请求。gRPC
默认端口是8500
,HTTP
请求的默认端口是8501
,serving镜像中的程序会自动加载镜像内/models
下的模型,通过MODEL_NAME
指定/models
下的哪个模型。
HTTP调用API
地址:http://ip:port/v1/models/nsfw:predict
接口返回参数:
{
"classes": "porn",
"probabilities": {
"drawings": 0.0000170060648, "hentai": 0.00108581863, "neutral": 0.000101140722, "porn": 0.816358209, "sexy": 0.182437778 } }
图床服务
完事具备,只欠图床,恰好,最近新域名也备案成功了,那就赶紧上线吧。麻溜的开始小范围内测,内测期间各位小伙伴可以多多踢出宝贵意见,2019年12月31日内测结束将清空所有数据,请悉知!
题外话
这篇案例酝酿了许久,还差点导致其难产,其实各种云上都有鉴黄服务,比如阿里云,50w
次请求,810RMB
,一年有效期,算下来也就不到2分钱
。但是肉疼啊,如果省钱的同时又能学习知识,何乐而不为呢?
参考
https://github.com/tensorflow/serving
https://www.tensorflow.org/serving/api_rest
https://www.tensorflow.org/tfx/serving/docker
https://github.com/alexkimxyz/nsfw_data_scraper
https://github.com/tensorflow/models/tree/master/official
https://www.cnblogs.com/weiyinfu/p/9928363.html
https://github.com/tensorflow/models/tree/master/research/slim
https://github.com/tensorflow/models/tree/master/official/vision/image_classification#resnet
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/java
前言
最近在做一款图床服务,关注公号的小伙伴一定记得小柒曾说过,会在周末放出的,不好意思放大家鸽子了。之所以一直没敢放出,是因为鉴黄接口一直没调试好,虽然我对公号的小伙伴百分之百信任,奈何互联网鱼龙混杂,万一上传了什么不雅的图片,然后不巧被某部门发现了,我包括我的服务器域名可就彻底玩完了!
架构
如图,先聊一下图床的架构:
-
Nginx
代理那是必备神器了。 -
lua
限流是一定的了,虽然前期没啥子流量,后期可能也没有。 -
限量限制大小也是必须的了,不然带宽受不了。
-
接入鉴黄,毕竟咱是合法备案网站。
-
文件多重备份,OSS、分布式文件、本地文件各一份,防止走丢。
-
为了查询方便,最后落库。
工具
-
SpringBoot,一个简化
Spring
开发的框架。 -
WebUploader,一个简单的以
HTML5
为主,FLASH
为辅的现代文件上传组件。 -
Python,加持各种开源第三方库处理图片。
-
nsfw_data_scraper,一个近
1w
星标的珍藏数据资源。 -
docker_nsfw_data_scraper,用于收集训练数据。
-
TensorFlow,开源机器学习库。
-
ResNet,图像分类的预训练模型。
-
TensorFlow-serving,部署
tensorflow
模型,并提供服务。
训练模型
训练之前,先介绍一个名词 NSFW
,之前我也不知道啥意思,毕竟是村里来的,什么泷泽萝拉、松岛枫、小泽玛利亚、吉泽明步、波多野结衣、天海翼、樱井莉亚、饭岛爱、苍井空、麻生希、橘梨纱、武藤兰、泽井芽衣.....是一概不知道的。
NSFW:不适合在工作场合出现的内容(英语:Not Safe/Suitable For Work,缩写:NSFW)是一个网络用语,多指裸露、暴力、色情或冒犯等不适宜公众场合的内容。在给出含有上述内容的超链接旁标注 NSFW,用于警告观看者。
在 nsfw_data_scraper
上传存放了成千上万张图片地址,并对图片进行了分类,以供训练:
- 绘画(Drawing),无害的艺术,或艺术绘画;
- 变态(Hentai),色情艺术,不适合大多数工作环境;
- 中立(Neutral),一般,无害的内容;
- 色情(Porn),不雅的内容和行为,通常涉及生殖器;
- 性感(Sexy),不合时宜的挑衅内容。
同时,官方也提供了收集方法:
$ docker build . -t docker_nsfw_data_scraper
Sending build context to Docker daemon 426.3MB
Step 1/3 : FROM ubuntu:18.04
---> 775349758637
Step 2/3 : RUN apt update && apt upgrade -y && apt install wget rsync imagemagick default-jre -y
---> Using cache ---> b2129908e7e2 Step 3/3 : ENTRYPOINT ["/bin/bash"] ---> Using cache ---> d32c5ae5235b Successfully built d32c5ae5235b Successfully tagged docker_nsfw_data_scraper:latest $ # Next command might run for several hours. It is recommended to leave it overnight $ docker run -v $(pwd):/root docker_nsfw_data_scraper /root/scripts/runall.sh Getting images for class: neutral ... ... $ ls data test train $ ls data/train/ drawings hentai neutral porn sexy $ ls data/test/ drawings hentai neutral porn sexy
如何训练模型,后面也很贴心的附上了训练方法,不过这里借用了 TensorFlow
的 ResNet
的模型,稍作修改。训练过程太过煎熬、痛苦,已经被湮灭在有限的带宽和无尽的小黄图中。
鉴黄服务
模型数据训练好以后就是搭建服务了,这里我们直接使用TensorFlow
的 TensorFlow-serving
对外提供服务,为了安装方便,我们使用Docker
安装部署。
NSFWDATA="/home/nsfw"
docker run -d --rm -p 8501:8501 \
--name nsfw \
-v "$NSFWDATA/models:/models/nsfw" \ -e MODEL_NAME=nsfw \ tensorflow/serving
serving
镜像提供了两种调用方式:gRPC
和HTTP
请求。gRPC
默认端口是8500
,HTTP
请求的默认端口是8501
,serving镜像中的程序会自动加载镜像内/models
下的模型,通过MODEL_NAME
指定/models
下的哪个模型。
HTTP调用API
地址:http://ip:port/v1/models/nsfw:predict
接口返回参数:
{
"classes": "porn",
"probabilities": {
"drawings": 0.0000170060648, "hentai": 0.00108581863, "neutral": 0.000101140722, "porn": 0.816358209, "sexy": 0.182437778 } }
图床服务
完事具备,只欠图床,恰好,最近新域名也备案成功了,那就赶紧上线吧。麻溜的开始小范围内测,内测期间各位小伙伴可以多多踢出宝贵意见,2019年12月31日内测结束将清空所有数据,请悉知!
题外话
这篇案例酝酿了许久,还差点导致其难产,其实各种云上都有鉴黄服务,比如阿里云,50w
次请求,810RMB
,一年有效期,算下来也就不到2分钱
。但是肉疼啊,如果省钱的同时又能学习知识,何乐而不为呢?
参考
https://github.com/tensorflow/serving
https://www.tensorflow.org/serving/api_rest
https://www.tensorflow.org/tfx/serving/docker
https://github.com/alexkimxyz/nsfw_data_scraper
https://github.com/tensorflow/models/tree/master/official
https://www.cnblogs.com/weiyinfu/p/9928363.html
https://github.com/tensorflow/models/tree/master/research/slim
https://github.com/tensorflow/models/tree/master/official/vision/image_classification#resnet
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/java