-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathatom.xml
180 lines (86 loc) · 161 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>AndrewYq</title>
<link href="https://yqstar.github.io/atom.xml" rel="self"/>
<link href="https://yqstar.github.io/"/>
<updated>2022-06-12T14:43:57.203Z</updated>
<id>https://yqstar.github.io/</id>
<author>
<name>AndrewYq</name>
</author>
<generator uri="https://hexo.io/">Hexo</generator>
<entry>
<title>Pytorch深度学习教程(4):序列模型</title>
<link href="https://yqstar.github.io/2022/06/12/Pytorch%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E6%95%99%E7%A8%8B-4-%E5%BA%8F%E5%88%97%E6%A8%A1%E5%9E%8B/"/>
<id>https://yqstar.github.io/2022/06/12/Pytorch%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E6%95%99%E7%A8%8B-4-%E5%BA%8F%E5%88%97%E6%A8%A1%E5%9E%8B/</id>
<published>2022-06-12T13:00:17.000Z</published>
<updated>2022-06-12T14:43:57.203Z</updated>
<content type="html"><![CDATA[<h2 id="序列模型介绍"><a href="#序列模型介绍" class="headerlink" title="序列模型介绍"></a>序列模型介绍</h2><p>循环神经网络（Recurrent Neural Network，RNN）是一种用于处理序列数据的神经网络。相比一般的神经网络来说，他能够处理序列变化的数据。比如某个单词的意思会因为上文提到的内容不同而有不同的含义，RNN就能够很好地解决这类问题。<br>RNN也可以堆叠在一起。需要注意的是，堆中的每个RNN都有自己的权重矩阵。因此，权重矩阵在水平轴（时间轴）上共享，而不是在垂直轴（RNN的数量）上共享。</p><p><img src="/2022/06/12/Pytorch%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E6%95%99%E7%A8%8B-4-%E5%BA%8F%E5%88%97%E6%A8%A1%E5%9E%8B/rnn.jpg" alt="rnn"></p><p>因RNN模型训练采用BPTT算法(BackPropagation Through Time)，导致RNN有一个主要的缺点，即所谓的梯度消失和梯度爆炸，而梯度爆炸问题可以通过梯度裁剪，去抑制，梯度消失的问题是随着反向传播，梯度越来越小，在这些网络中，随着时间步长的增加，反向传播使梯度变得越来越小，导致梯度消失。进而导致RNN长期依赖问题。<br>注意，裁剪仅限制梯度的大小，而不限制其方向。所以，学习仍然朝着正确的方向前进。</p><p><a class="link" href="https://www.zhihu.com/question/279046805/answer/1153623199">https://www.zhihu.com/question/279046805/answer/1153623199<i class="fas fa-external-link-alt"></i></a></p>]]></content>
<summary type="html"><h2 id="序列模型介绍"><a href="#序列模型介绍" class="headerlink" title="序列模型介绍"></a>序列模型介绍</h2><p>循环神经网络（Recurrent Neural Network，RNN）是一种用于处理序列数据的神经网络。相</summary>
<category term="Pytorch_Tutorial" scheme="https://yqstar.github.io/tags/Pytorch-Tutorial/"/>
</entry>
<entry>
<title>Pytorch深度学习教程(3):Mnist图像分类TorchServe部署</title>
<link href="https://yqstar.github.io/2022/06/06/Pytorch%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E6%95%99%E7%A8%8B-3-Mnist%E5%9B%BE%E5%83%8F%E5%88%86%E7%B1%BBTorchServe%E9%83%A8%E7%BD%B2/"/>
<id>https://yqstar.github.io/2022/06/06/Pytorch%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E6%95%99%E7%A8%8B-3-Mnist%E5%9B%BE%E5%83%8F%E5%88%86%E7%B1%BBTorchServe%E9%83%A8%E7%BD%B2/</id>
<published>2022-06-06T10:05:23.000Z</published>
<updated>2022-06-12T14:43:59.313Z</updated>
<content type="html"><![CDATA[<h2 id="简介"><a href="#简介" class="headerlink" title="简介"></a>简介</h2><p>经过前 <a href="https://yqstar.github.io/tags/Pytorch-Tutorial/">两篇博客</a> 学习，我们已可使用CNN模型完成Mnist手写数字分类模型，对于算法从数据处理、模型构建、模型训练和评估链路有初步认知。但工业可能需要部署离线在线模型用于提供模型推理服务，所谓模型推理服务是指在系统配置训练完成机器学习模型，以便其可接受新的输入并将推理结果返回给系统。</p><p>其次，虽然很多大厂都会有封装好部署平台供算法人员便捷配置，但是学习中对于完整的工程链路开发对于个人能力建设也是非常重要的，而不是仅仅作为一颗螺丝钉，如何实现从Demo模型转换成线上模型推理服务部署，对于个人的正向反馈也是非常有意义的。</p><p>那么针对 <em>PyTorch</em> 训练好 Demo 模型，如何部署到生产环境用于提供模型推理服务呢？部署形式非常多样，其中 TorchServe 是 <a class="link" href="https://pytorch.org/serve/index.html">PyTorch开源项目<i class="fas fa-external-link-alt"></i></a> 部分，是AWS和Facebook合作开发的用于部署Pytorch的模型，对于算法工程师是相当友好的。本章介绍如何使用TorchServe完成PyTorch模型的部署和调用。</p><h2 id="TorchServe简介"><a href="#TorchServe简介" class="headerlink" title="TorchServe简介"></a>TorchServe简介</h2><p>Torchserve是PyTorch的首选模型部署解决方案。它允许为模型公开一个可供直接访问或者应用程序访问的WebAPI，借助TorchServe，PyTorch用户可以更快地将其模型应用于生产，而无需编写自定义代码，此外，TorchServe将工程开发和算法开发进行解耦，算法工程师主要完成数据Process和模型构建这一擅长领域，其他的多模型服务、A/B测试的版本控制、监视指标以及应用程序集成RESTful都已封装好。</p><blockquote><p>TorchServe is a performant, flexible and easy to use tool for serving PyTorch eager mode and torschripted models.</p></blockquote><p>官网介绍可看出：TorchServe是一款性能好、灵活性好、易使用的工具，其次可部署模型类型是Pytorch的Eager模式和Script模式模型。</p><p>TorchServe框架主要分为四个部分：Frontend是TorchServe的请求和响应的处理部分；WorkerProcess 指的是一组运行的模型实例，可以由管理API设定运行的数量；Model Store是模型存储加载的地方；Backend用于管理Worker Process，具体可参考下图里。</p><p><img src="/2022/06/06/Pytorch%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E6%95%99%E7%A8%8B-3-Mnist%E5%9B%BE%E5%83%8F%E5%88%86%E7%B1%BBTorchServe%E9%83%A8%E7%BD%B2/ts_frame.png" alt="ts_frame"></p><h2 id="环境安装"><a href="#环境安装" class="headerlink" title="环境安装"></a>环境安装</h2><p>本人使用 Windows11+WSL2+Ubuntu 环境进行部署。</p><h3 id="Conda配置"><a href="#Conda配置" class="headerlink" title="Conda配置"></a>Conda配置</h3><p><a class="link" href="https://github.com/pytorch/serve">官网要求<i class="fas fa-external-link-alt"></i></a>：Python Version >= 3.8，本文使用Conda管理深度学习环境，具体使用可参考之前的博文：<a href="https://yqstar.github.io/2022/05/01/Windows%E7%B3%BB%E7%BB%9F%E4%BD%BF%E7%94%A8Conda%E9%85%8D%E7%BD%AE%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E7%8E%AF%E5%A2%83/">深度学习管理配置</a>。</p><p><img src="/2022/06/06/Pytorch%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E6%95%99%E7%A8%8B-3-Mnist%E5%9B%BE%E5%83%8F%E5%88%86%E7%B1%BBTorchServe%E9%83%A8%E7%BD%B2/python_version.png" alt="python_version"></p><p>可使用下述命令创建Conda的Python环境（python版本为3.8，环境名为ts_ENV）和激活指定环境(ts_env)。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">conda create --name ts_env python=3.8</span><br><span class="line">conda activate ts_env</span><br></pre></td></tr></table></figure><h3 id="TS源码安装"><a href="#TS源码安装" class="headerlink" title="TS源码安装"></a>TS源码安装</h3><p>可参考官网TS安装<a class="link" href="https://pytorch.org/serve/torchserve_on_wsl.html">文档<i class="fas fa-external-link-alt"></i></a>。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">git <span class="built_in">clone</span> https://github.com/pytorch/serve.git</span><br><span class="line"><span class="built_in">cd</span> serve</span><br><span class="line"></span><br><span class="line">./ts_scripts/setup_wsl_ubuntu</span><br><span class="line"><span class="built_in">export</span> PATH=<span class="variable">$HOME</span>/.local/bin:<span class="variable">$PATH</span></span><br><span class="line">python ./ts_scripts/install_from_src.py</span><br><span class="line"><span class="built_in">echo</span> <span class="string">'export PATH=$HOME/.local/bin:$PATH'</span> >> ~/.bashrc</span><br><span class="line"><span class="built_in">source</span> ~/.bashrc</span><br></pre></td></tr></table></figure><h2 id="模型打包"><a href="#模型打包" class="headerlink" title="模型打包"></a>模型打包</h2><p>TorchServe 的一个关键特性是能够将所有模型工件打包到单个模型存档文件中。它是一个单独的命令行界面 (CLI)，torch-model-archiver，可以使用 state_dict 获取模型检查点或模型定义文件，并将它们打包成 .mar 文件。 然后，任何使用 TorchServe 的人都可以重新分发和提供该文件。它包含以下模型工件：在 torchscript 或模型定义文件的情况下的模型检查点文件和在急切模式的情况下的 state_dict 文件，以及服务模型可能需要的其他可选资产。 CLI 创建一个 .mar 文件，TorchServe 的服务器 CLI 使用该文件为模型提供服务。</p><p>torch-model-archiver 命令来打包模型，需要提供以下三个文件。</p><p>第 1 步：创建一个新的模型架构文件，其中包含从 torch.nn.modules 扩展的模型类。在这个例子中，我们创建了mnist模型文件mnist_model.py文件。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> torch</span><br><span class="line"><span class="keyword">from</span> torch <span class="keyword">import</span> nn</span><br><span class="line"><span class="keyword">import</span> torch.nn.functional <span class="keyword">as</span> F</span><br><span class="line"></span><br><span class="line"><span class="comment"># 构建网络</span></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">MnistClassificationNet</span>(nn.Module):</span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">__init__</span>(<span class="params">self</span>):</span><br><span class="line"> <span class="built_in">super</span>().__init__()</span><br><span class="line"> self.conv1 = nn.Conv2d(in_channels=<span class="number">1</span>, out_channels=<span class="number">32</span>, kernel_size=<span class="number">3</span>)</span><br><span class="line"> self.conv2 = nn.Conv2d(in_channels=<span class="number">32</span>, out_channels=<span class="number">64</span>, kernel_size=<span class="number">3</span>)</span><br><span class="line"> self.max_pool2d = nn.MaxPool2d(kernel_size=<span class="number">2</span>)</span><br><span class="line"> self.dropout = nn.Dropout2d(p=<span class="number">0.25</span>)</span><br><span class="line"> <span class="comment"># self.relu = nn.ReLU()</span></span><br><span class="line"> self.fc1 = nn.Linear(in_features=<span class="number">9216</span>, out_features=<span class="number">128</span>)</span><br><span class="line"> self.fc2 = nn.Linear(in_features=<span class="number">128</span>, out_features=<span class="number">10</span>)</span><br><span class="line"> self.log_softmax = nn.LogSoftmax(dim=<span class="number">1</span>)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">forward</span>(<span class="params">self, x</span>):</span><br><span class="line"> x = self.conv1(x)</span><br><span class="line"> x = F.relu(x)</span><br><span class="line"> x = self.conv2(x)</span><br><span class="line"> x = F.relu(x)</span><br><span class="line"> x= self.max_pool2d(x)</span><br><span class="line"> x = self.dropout(x)</span><br><span class="line"> x = x.view(-<span class="number">1</span>, <span class="number">9216</span>)</span><br><span class="line"> x = self.fc1(x)</span><br><span class="line"> x = F.relu(x)</span><br><span class="line"> x = self.fc2(x)</span><br><span class="line"> x= self.log_softmax(x)</span><br><span class="line"> <span class="keyword">return</span> x</span><br></pre></td></tr></table></figure><p>第 2 步：使用 <a href="https://yqstar.github.io/2022/05/11/Pytorch系列自学教程-2-深度学习“Hello-World”之Mnist图像分类/">mnist_sd</a> 训练 MNIST 数字识别模型并保存模型的状态字典。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">torch.save(model.state_dict(), <span class="string">"./checkpoints/model_pth/mnist_sd.pt"</span>)</span><br></pre></td></tr></table></figure><p>第 3 步：编写自定义处理程序以在您的模型上运行推理。 在此示例中，我们添加了一个 mnist_handler.py 文件，它使用上述模型对输入灰度图像进行推理并识别图像中的数字。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> torchvision <span class="keyword">import</span> transforms</span><br><span class="line"><span class="keyword">from</span> ts.torch_handler.image_classifier <span class="keyword">import</span> ImageClassifier</span><br><span class="line"><span class="keyword">from</span> torch.profiler <span class="keyword">import</span> ProfilerActivity</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">MNISTDigitClassifier</span>(<span class="title class_ inherited__">ImageClassifier</span>):</span><br><span class="line"> <span class="string">"""</span></span><br><span class="line"><span class="string"> MNISTDigitClassifier handler class. This handler extends class ImageClassifier from image_classifier.py, a</span></span><br><span class="line"><span class="string"> default handler. This handler takes an image and returns the number in that image.</span></span><br><span class="line"><span class="string"> Here method postprocess() has been overridden while others are reused from parent class.</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"></span><br><span class="line"> image_processing = transforms.Compose([</span><br><span class="line"> transforms.ToTensor()</span><br><span class="line"> <span class="comment"># transforms.Normalize((0.1307,), (0.3081,))</span></span><br><span class="line"> ])</span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">__init__</span>(<span class="params">self</span>):</span><br><span class="line"> <span class="built_in">super</span>(MNISTDigitClassifier, self).__init__()</span><br><span class="line"> self.profiler_args = {</span><br><span class="line"> <span class="string">"activities"</span> : [ProfilerActivity.CPU],</span><br><span class="line"> <span class="string">"record_shapes"</span>: <span class="literal">True</span>,</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">postprocess</span>(<span class="params">self, data</span>):</span><br><span class="line"> <span class="string">"""The post process of MNIST converts the predicted output response to a label.</span></span><br><span class="line"><span class="string"> Args:</span></span><br><span class="line"><span class="string"> data (list): The predicted output from the Inference with probabilities is passed</span></span><br><span class="line"><span class="string"> to the post-process function</span></span><br><span class="line"><span class="string"> Returns:</span></span><br><span class="line"><span class="string"> list : A list of dictionaries with predictions and explanations is returned</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"> <span class="keyword">return</span> data.argmax(<span class="number">1</span>).tolist()</span><br><span class="line"> <span class="comment"># return data.tolist()</span></span><br></pre></td></tr></table></figure><p>第 4 步：使用 torch-model-archiver 程序创建一个 Torch 模型存档以存档上述文件。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br></pre></td><td class="code"><pre><span class="line">$ torch-model-archiver -h</span><br><span class="line">usage: torch-model-archiver [-h] --model-name MODEL_NAME --version MODEL_VERSION_NUMBER</span><br><span class="line"> --model-file MODEL_FILE_PATH --serialized-file MODEL_SERIALIZED_PATH</span><br><span class="line"> --handler HANDLER [--runtime {python,python2,python3}]</span><br><span class="line"> [--export-path EXPORT_PATH] [-f] [--requirements-file]</span><br><span class="line"></span><br><span class="line">Model Archiver Tool</span><br><span class="line"></span><br><span class="line">optional arguments:</span><br><span class="line"> -h, --<span class="built_in">help</span> show this <span class="built_in">help</span> message and <span class="built_in">exit</span></span><br><span class="line"> --model-name MODEL_NAME</span><br><span class="line"> Exported model name. Exported file will be named as</span><br><span class="line"> model-name.mar and saved <span class="keyword">in</span> current working directory</span><br><span class="line"> <span class="keyword">if</span> no --export-path is specified, <span class="keyword">else</span> it will be</span><br><span class="line"> saved under the <span class="built_in">export</span> path</span><br><span class="line"> --serialized-file SERIALIZED_FILE</span><br><span class="line"> Path to .pt or .pth file containing state_dict <span class="keyword">in</span></span><br><span class="line"> <span class="keyword">case</span> of eager mode or an executable ScriptModule</span><br><span class="line"> <span class="keyword">in</span> <span class="keyword">case</span> of TorchScript.</span><br><span class="line"> --model-file MODEL_FILE</span><br><span class="line"> Path to python file containing model architecture.</span><br><span class="line"> This parameter is mandatory <span class="keyword">for</span> eager mode models.</span><br><span class="line"> The model architecture file must contain only one</span><br><span class="line"> class definition extended from torch.nn.modules.</span><br><span class="line"> --handler HANDLER TorchServe<span class="string">'s default handler name or handler python</span></span><br><span class="line"><span class="string"> file path to handle custom TorchServe inference logic.</span></span><br><span class="line"><span class="string"> --extra-files EXTRA_FILES</span></span><br><span class="line"><span class="string"> Comma separated path to extra dependency files.</span></span><br><span class="line"><span class="string"> --runtime {python,python2,python3}</span></span><br><span class="line"><span class="string"> The runtime specifies which language to run your</span></span><br><span class="line"><span class="string"> inference code on. The default runtime is</span></span><br><span class="line"><span class="string"> RuntimeType.PYTHON. At the present moment we support</span></span><br><span class="line"><span class="string"> the following runtimes python, python2, python3</span></span><br><span class="line"><span class="string"> --export-path EXPORT_PATH</span></span><br><span class="line"><span class="string"> Path where the exported .mar file will be saved. This</span></span><br><span class="line"><span class="string"> is an optional parameter. If --export-path is not</span></span><br><span class="line"><span class="string"> specified, the file will be saved in the current</span></span><br><span class="line"><span class="string"> working directory.</span></span><br><span class="line"><span class="string"> --archive-format {tgz,default}</span></span><br><span class="line"><span class="string"> The format in which the model artifacts are archived.</span></span><br><span class="line"><span class="string"> "tgz": This creates the model-archive in <model-name>.tar.gz format.</span></span><br><span class="line"><span class="string"> If platform hosting requires model-artifacts to be in ".tar.gz"</span></span><br><span class="line"><span class="string"> use this option.</span></span><br><span class="line"><span class="string"> "no-archive": This option creates an non-archived version of model artifacts</span></span><br><span class="line"><span class="string"> at "export-path/{model-name}" location. As a result of this choice,</span></span><br><span class="line"><span class="string"> MANIFEST file will be created at "export-path/{model-name}" location</span></span><br><span class="line"><span class="string"> without archiving these model files</span></span><br><span class="line"><span class="string"> "default": This creates the model-archive in <model-name>.mar format.</span></span><br><span class="line"><span class="string"> This is the default archiving format. Models archived in this format</span></span><br><span class="line"><span class="string"> will be readily hostable on TorchServe.</span></span><br><span class="line"><span class="string"> -f, --force When the -f or --force flag is specified, an existing</span></span><br><span class="line"><span class="string"> .mar file with same name as that provided in --model-</span></span><br><span class="line"><span class="string"> name in the path specified by --export-path will</span></span><br><span class="line"><span class="string"> overwritten</span></span><br><span class="line"><span class="string"> -v, --version Model'</span>s version.</span><br><span class="line"> -r, --requirements-file</span><br><span class="line"> Path to requirements.txt file containing a list of model specific python</span><br><span class="line"> packages to be installed by TorchServe <span class="keyword">for</span> seamless model serving.</span><br></pre></td></tr></table></figure><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">torch-model-archiver --model-name mnist --version 1.0 --model-file mnist_model.py --serialized-file mnist_sd.pt --export-path ./model_store --handler mnist_handler.py -f</span><br></pre></td></tr></table></figure><h2 id="模型部署"><a href="#模型部署" class="headerlink" title="模型部署"></a>模型部署</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br></pre></td><td class="code"><pre><span class="line">$ torchserve --<span class="built_in">help</span></span><br><span class="line">usage: torchserve [-h] [-v | --version]</span><br><span class="line"> [--start]</span><br><span class="line"> [--stop]</span><br><span class="line"> [--ts-config TS_CONFIG]</span><br><span class="line"> [--model-store MODEL_STORE]</span><br><span class="line"> [--workflow-store WORKFLOW_STORE]</span><br><span class="line"> [--models MODEL_PATH1 MODEL_NAME=MODEL_PATH2... [MODEL_PATH1 MODEL_NAME=MODEL_PATH2... ...]]</span><br><span class="line"> [--log-config LOG_CONFIG]</span><br><span class="line"></span><br><span class="line">torchserve</span><br><span class="line"></span><br><span class="line">mandatory arguments:</span><br><span class="line"> --model-store MODEL_STORE</span><br><span class="line"> Model store location <span class="built_in">where</span> models can be loaded</span><br><span class="line"></span><br><span class="line"> </span><br><span class="line"></span><br><span class="line">optional arguments:</span><br><span class="line"> -h, --<span class="built_in">help</span> show this <span class="built_in">help</span> message and <span class="built_in">exit</span></span><br><span class="line"> -v, --version Return TorchServe Version</span><br><span class="line"> --start Start the model-server</span><br><span class="line"> --stop Stop the model-server</span><br><span class="line"> --ts-config TS_CONFIG</span><br><span class="line"> Configuration file <span class="keyword">for</span> TorchServe</span><br><span class="line"></span><br><span class="line"> --models MODEL_PATH1 MODEL_NAME=MODEL_PATH2... [MODEL_PATH1 MODEL_NAME=MODEL_PATH2... ...]</span><br><span class="line"> Models to be loaded using [model_name=]model_location</span><br><span class="line"> format. Location can be a HTTP URL, a model archive</span><br><span class="line"> file or directory contains model archive files <span class="keyword">in</span></span><br><span class="line"> MODEL_STORE.</span><br><span class="line"> --log-config LOG_CONFIG</span><br><span class="line"> Log4j configuration file <span class="keyword">for</span> TorchServe</span><br><span class="line"> --ncs, --no-config-snapshots </span><br><span class="line"> Disable snapshot feature</span><br><span class="line"> --workflow-store WORKFLOW_STORE</span><br><span class="line"> Workflow store location <span class="built_in">where</span> workflow can be loaded. Defaults to model-store</span><br></pre></td></tr></table></figure><h3 id="启动torchserve服务"><a href="#启动torchserve服务" class="headerlink" title="启动torchserve服务"></a>启动torchserve服务</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">torchserve --start --ncs --model-store model_store --models mnist.mar</span><br></pre></td></tr></table></figure><p>模型启动日志如下截图：</p><p><img src="/2022/06/06/Pytorch%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E6%95%99%E7%A8%8B-3-Mnist%E5%9B%BE%E5%83%8F%E5%88%86%E7%B1%BBTorchServe%E9%83%A8%E7%BD%B2/ts_start.png" alt="ts_start"></p><h3 id="推理健康检查API"><a href="#推理健康检查API" class="headerlink" title="推理健康检查API"></a>推理健康检查API</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">curl http://localhost:8080/ping</span><br></pre></td></tr></table></figure><p>如果server正常运行, 响应会如截图所示：</p><p><img src="/2022/06/06/Pytorch%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E6%95%99%E7%A8%8B-3-Mnist%E5%9B%BE%E5%83%8F%E5%88%86%E7%B1%BBTorchServe%E9%83%A8%E7%BD%B2/ts_ping.png" alt="ts_ping"></p><h3 id="推理"><a href="#推理" class="headerlink" title="推理"></a>推理</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">curl http://127.0.0.1:8080/predictions/mnist -T ./data/test.png</span><br></pre></td></tr></table></figure><p>test.png为数字为0的图片，通过上述的调用推理，可以看出结果是能正常返回的，是可以作为下游应用调用。</p><p><img src="/2022/06/06/Pytorch%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E6%95%99%E7%A8%8B-3-Mnist%E5%9B%BE%E5%83%8F%E5%88%86%E7%B1%BBTorchServe%E9%83%A8%E7%BD%B2/ts_infer.png" alt="ts_infer"></p><h3 id="停止torchserve服务"><a href="#停止torchserve服务" class="headerlink" title="停止torchserve服务"></a>停止torchserve服务</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">torchserve --start</span><br></pre></td></tr></table></figure><h2 id="探索"><a href="#探索" class="headerlink" title="探索"></a>探索</h2><p><a class="link" href="https://github.com/yqstar/Awesome_Pytorch_Tutorial/tree/master/Pytorch_Lesson3">完整代码<i class="fas fa-external-link-alt"></i></a>已上传Github，有需要的可以自行下载代码，如果对你有帮助，请Star，哈哈哈哈！</p><p>到此为止，已经可以使用自己数据玩耍各种Demo，快（苦）乐（逼）地进行炼丹之路。道路阻且长，行则将至，但行好事莫问前程。</p><ul><li>除了使用TorchServe部署模型，还有其他的解决方案吗？</li><li>除了使用提供这种Web API的形式，是否可以构建一个GUI的形式提供呢？例如 PYQT5 ？这里放一张PYQT5的图，后面会填坑。</li></ul><p><img src="/2022/06/06/Pytorch%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E6%95%99%E7%A8%8B-3-Mnist%E5%9B%BE%E5%83%8F%E5%88%86%E7%B1%BBTorchServe%E9%83%A8%E7%BD%B2/pyqt_demo.png" alt="pyqt_demo"></p><p>正如，人往往会对未知的事情产生恐惧，因为结局是未知的。所以当一切不再未知的时候，那么是不是就不会产生恐惧呢？</p><h2 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h2><p>More info: <a class="link" href="https://zhuanlan.zhihu.com/p/344364948">Chenglu：如何部署PyTorch模型<i class="fas fa-external-link-alt"></i></a><br>More info: <a class="link" href="https://github.com/pytorch/serve">TorchServe<i class="fas fa-external-link-alt"></i></a><br>More info: <a class="link" href="https://github.com/pytorch/serve/tree/master/examples/image_classifier/mnist">TorchServe_Mnist Example<i class="fas fa-external-link-alt"></i></a><br>More info: <a class="link" href="https://blog.csdn.net/weixin_41977938/article/details/122258595">随便写点笔记<i class="fas fa-external-link-alt"></i></a><br>More info: <a class="link" href="https://blog.csdn.net/Chris_zhangrx/article/details/117380516">PyTorch Eager mode and Script mode<i class="fas fa-external-link-alt"></i></a><br>More info: <a class="link" href="https://cceyda.github.io/blog/huggingface/torchserve/streamlit/ner/2020/10/09/huggingface_streamlit_serve.html">Self-host your 🤗HuggingFace Transformer NER model with Torchserve + Streamlit<i class="fas fa-external-link-alt"></i></a><br>More info: <a class="link" href="https://ceshiren.com/t/topic/20770">TorchServe搭建codeBERT分类模型服务<i class="fas fa-external-link-alt"></i></a><br>More info: <a class="link" href="https://blog.csdn.net/qq_15821487/article/details/122684773">torchserver模型本地部署和docker部署<i class="fas fa-external-link-alt"></i></a></p>]]></content>
<summary type="html"><h2 id="简介"><a href="#简介" class="headerlink" title="简介"></a>简介</h2><p>经过前 <a href="https://yqstar.github.io/tags/Pytorch-Tutorial/">两篇博客</a></summary>
<category term="Pytorch_Tutorial" scheme="https://yqstar.github.io/tags/Pytorch-Tutorial/"/>
</entry>
<entry>
<title>Pytorch深度学习教程(2):“Hello World”之Mnist图像分类</title>
<link href="https://yqstar.github.io/2022/05/29/Pytorch%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E6%95%99%E7%A8%8B-2-%E2%80%9CHello-World%E2%80%9D%E4%B9%8BMnist%E5%9B%BE%E5%83%8F%E5%88%86%E7%B1%BB/"/>
<id>https://yqstar.github.io/2022/05/29/Pytorch%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E6%95%99%E7%A8%8B-2-%E2%80%9CHello-World%E2%80%9D%E4%B9%8BMnist%E5%9B%BE%E5%83%8F%E5%88%86%E7%B1%BB/</id>
<published>2022-05-29T15:37:29.000Z</published>
<updated>2022-06-12T14:44:01.173Z</updated>
<content type="html"><![CDATA[<h2 id="简介"><a href="#简介" class="headerlink" title="简介"></a>简介</h2><p>本章节主要通过深度学习“Hello World”之Mnist图像分类，学会深度学习的基本链路，快速搭建个人的Baseline模型，包括数据加载、数据可视化、模型构建、模型训练评估和模型结果指标展示。</p><p>Mnist数据集是手写数字0~9的MNIST数据集，包含60,000个用于训练的训练集和10,000个用于测试的测试集。这些数字已被大小归一化，并以固定尺寸的图像为中心。对于尝试学习深度学习技术的人来说，这是一个非常棒的数据集，同时可以花费最少时间来进行预处理和格式化。详细：<a class="link" href="http://yann.lecun.com/exdb/mnist/">Mnist官网<i class="fas fa-external-link-alt"></i></a></p><h2 id="环境配置"><a href="#环境配置" class="headerlink" title="环境配置"></a>环境配置</h2><p>环境配置主要包括：操作系统，显卡，深度学习Pytorch等配置如下。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">系统：Windows <span class="number">11</span> - WSL2 系统</span><br><span class="line">显卡：NVIDIA GeForce RTX <span class="number">3060</span></span><br><span class="line">python: <span class="number">3.6</span><span class="number">.13</span></span><br><span class="line">pytorch: <span class="number">1.7</span><span class="number">.1</span></span><br><span class="line">cudatoolkit: <span class="number">11.0</span><span class="number">.221</span></span><br><span class="line">torchvision: <span class="number">0.8</span><span class="number">.2</span></span><br></pre></td></tr></table></figure><h3 id="导入包环境"><a href="#导入包环境" class="headerlink" title="导入包环境"></a>导入包环境</h3><p>本文使用的Python Package 主要有Pytorch的深度学习框架，torchvision用于加载Mnist数据集，matplotlib用于可视化展示数据集和展示训练结果等。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> torchvision <span class="keyword">import</span> datasets, transforms</span><br><span class="line"><span class="keyword">from</span> torch.utils.data <span class="keyword">import</span> DataLoader</span><br><span class="line"><span class="keyword">import</span> torch</span><br><span class="line"><span class="keyword">import</span> torch.optim <span class="keyword">as</span> optim</span><br><span class="line"><span class="keyword">import</span> torch.nn <span class="keyword">as</span> nn</span><br><span class="line"><span class="keyword">import</span> torch.nn.functional <span class="keyword">as</span> F</span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"><span class="keyword">import</span> random</span><br><span class="line"><span class="keyword">import</span> matplotlib.pyplot <span class="keyword">as</span> plt</span><br><span class="line"><span class="keyword">from</span> torchinfo <span class="keyword">import</span> summary</span><br></pre></td></tr></table></figure><p>构建baseline模型核心是快速跑通完整链路，同时在Baseline的基础上迭代更新有合适的对比标准，方便验证算法模型的稳定性和正确性。为保证训练模型的可重复性，我们需要进行一些配置。全部配置主要分为三部分，具体如下：</p><ul><li>CUDNN配置，CUDNN中对卷积操作进行优化，牺牲精度来换取计算效率。</li><li>Pytorch在运行中会有很多随机初始化操作，所以需要固定随机种子。</li><li>Python & Numpy也需要固定对应的随机种子。</li><li>注意，如果Dataloader采用了多线程(num_workers > 1), 那么由于读取数据的顺序不同，最终运行结果也会有差异。也就是说，改变num_workers参数，也会对实验结果产生影响。</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 保证试验结果的稳定性</span></span><br><span class="line">seed = <span class="number">0</span></span><br><span class="line">random.seed(seed)</span><br><span class="line">np.random.seed(seed)</span><br><span class="line">torch.manual_seed(seed)</span><br><span class="line">torch.cuda.manual_seed_all(seed)</span><br><span class="line">torch.backends.cudnn.benchmark = <span class="literal">False</span></span><br><span class="line">torch.backends.cudnn.deterministic = <span class="literal">True</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 配置Cuda参数</span></span><br><span class="line">is_cuda = <span class="literal">False</span></span><br><span class="line"><span class="keyword">if</span> torch.cuda.is_available():</span><br><span class="line"> is_cuda = <span class="literal">True</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 模型参数</span></span><br><span class="line">batch_size = <span class="number">32</span></span><br><span class="line">num_epochs = <span class="number">2</span></span><br></pre></td></tr></table></figure><h2 id="模型数据"><a href="#模型数据" class="headerlink" title="模型数据"></a>模型数据</h2><p>本章节就不详细讲解，大家可以查看前一章<a href="https://yqstar.github.io/2022/05/05/Pytorch%E7%B3%BB%E5%88%97%E8%87%AA%E5%AD%A6%E6%95%99%E7%A8%8B-1-%E6%95%B0%E6%8D%AE%E5%8A%A0%E8%BD%BD%E4%B9%8BDataset%E5%92%8CDataLoader%E4%BD%BF%E7%94%A8/">博客</a>讲解。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 构建Dataset</span></span><br><span class="line">mnist_train_dataset = datasets.MNIST(root=<span class="string">r'./data'</span>,</span><br><span class="line"> transform=transforms.ToTensor(),</span><br><span class="line"> train=<span class="literal">True</span>,</span><br><span class="line"> download=<span class="literal">True</span>)</span><br><span class="line"></span><br><span class="line">mnist_test_dataset = datasets.MNIST(root=<span class="string">r'./data'</span>,</span><br><span class="line"> transform=transforms.ToTensor(),</span><br><span class="line"> train=<span class="literal">False</span>,</span><br><span class="line"> download=<span class="literal">True</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 构建Dataloader</span></span><br><span class="line">mnist_train_loader = DataLoader(mnist_train_dataset,batch_size=<span class="number">32</span>,shuffle=<span class="literal">True</span>)</span><br><span class="line"></span><br><span class="line">mnist_test_loader = DataLoader(mnist_test_dataset,batch_size=<span class="number">32</span>,shuffle=<span class="literal">True</span>)</span><br></pre></td></tr></table></figure><p>为更好地看到我们需要处理什么样的数据，我们可以用一些可视化的手段展示我们的数据。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 图片查看</span></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">plot_image</span>(<span class="params">image,batch_size=<span class="number">32</span></span>):</span><br><span class="line"> x_batch,y_batch = image[<span class="number">0</span>],image[<span class="number">1</span>]</span><br><span class="line"> f = plt.figure(figsize=(<span class="number">300</span>,<span class="number">300</span>))</span><br><span class="line"> <span class="comment"># mean = 0.1307</span></span><br><span class="line"> <span class="comment"># std = 0.3081</span></span><br><span class="line"> <span class="comment"># image = ((image * mean) + std)</span></span><br><span class="line"> <span class="comment"># plt.imshow(image, cmap='gray')</span></span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(batch_size):</span><br><span class="line"> image_tx = x_batch.numpy()[i]</span><br><span class="line"> <span class="comment"># image_ty = y_batch.numpy()[i]</span></span><br><span class="line"> np.math.sqrt(<span class="number">32</span>)</span><br><span class="line"> <span class="comment"># Debug, plot figure</span></span><br><span class="line"> sub_size = <span class="built_in">int</span>(np.math.sqrt(<span class="number">32</span>))+<span class="number">1</span></span><br><span class="line"> f.add_subplot(sub_size,sub_size, i + <span class="number">1</span>)</span><br><span class="line"> <span class="comment"># plt.subplot_mosaic</span></span><br><span class="line"> plt.imshow(image_tx[<span class="number">0</span>], cmap=<span class="string">'gray'</span>)</span><br><span class="line"> plt.show()</span><br><span class="line"></span><br><span class="line">sample_image_batch = <span class="built_in">next</span>(<span class="built_in">iter</span>(mnist_train_loader))</span><br><span class="line">plot_image(sample_image_batch)</span><br><span class="line"></span><br></pre></td></tr></table></figure><p>图片展示结果如下：</p><p><img src="/2022/05/29/Pytorch%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E6%95%99%E7%A8%8B-2-%E2%80%9CHello-World%E2%80%9D%E4%B9%8BMnist%E5%9B%BE%E5%83%8F%E5%88%86%E7%B1%BB/sample_vis.png" alt="样本图片"></p><h2 id="模型构建"><a href="#模型构建" class="headerlink" title="模型构建"></a>模型构建</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 构建网络</span></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">MnistNet</span>(nn.Module):</span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">__init__</span>(<span class="params">self</span>):</span><br><span class="line"> <span class="built_in">super</span>().__init__()</span><br><span class="line"> self.conv1 = nn.Conv2d(in_channels=<span class="number">1</span>,out_channels=<span class="number">32</span>,kernel_size= <span class="number">3</span>)</span><br><span class="line"> self.conv2 = nn.Conv2d(in_channels= <span class="number">32</span>,out_channels= <span class="number">64</span>,kernel_size= <span class="number">3</span>)</span><br><span class="line"> self.dropout = nn.Dropout2d(<span class="number">0.25</span>)</span><br><span class="line"> self.fc1 = nn.Linear(<span class="number">9216</span>, <span class="number">128</span>)</span><br><span class="line"> self.fc2 = nn.Linear(<span class="number">128</span>, <span class="number">10</span>)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">forward</span>(<span class="params">self, x</span>):</span><br><span class="line"> x = self.conv1(x)</span><br><span class="line"> x = F.relu(x)</span><br><span class="line"> x = self.conv2(x)</span><br><span class="line"> x = F.relu(x)</span><br><span class="line"> x = F.max_pool2d(x, <span class="number">2</span>)</span><br><span class="line"> x = self.dropout(x)</span><br><span class="line"> x = x.view(-<span class="number">1</span>, <span class="number">9216</span>)</span><br><span class="line"> x = self.fc1(x)</span><br><span class="line"> x = F.relu(x)</span><br><span class="line"> x = self.fc2(x)</span><br><span class="line"> x = F.log_softmax(x, dim=<span class="number">1</span>)</span><br><span class="line"> <span class="keyword">return</span> x</span><br></pre></td></tr></table></figure><p>在进行模型训练过程中，我们可以使用以下代码，查看构建的模型结构，目前torchinfo仅能展示 <em>torch.nn</em> 模块下的模型结构，采用 <em>torch.nn.functional</em> 构造的函数是无法显示的。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">summary(model, input_size=(batch_size, <span class="number">1</span>, <span class="number">28</span>, <span class="number">28</span>),depth=<span class="number">4</span>)</span><br><span class="line"></span><br></pre></td></tr></table></figure><p><img src="/2022/05/29/Pytorch%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E6%95%99%E7%A8%8B-2-%E2%80%9CHello-World%E2%80%9D%E4%B9%8BMnist%E5%9B%BE%E5%83%8F%E5%88%86%E7%B1%BB/model_vis.png" alt="样本图片"></p><h2 id="模型训练"><a href="#模型训练" class="headerlink" title="模型训练"></a>模型训练</h2><h3 id="模型训练函数"><a href="#模型训练函数" class="headerlink" title="模型训练函数"></a>模型训练函数</h3><p>如果模型中有BN层(Batch Normalization）和Dropout，需要在训练时添加model.train()，在测试时添加model.eval()。其中model.train()是保证BN层用每一批数据的均值和方差，而model.eval()是保证BN用全部训练数据的均值和方差；而对于Dropout，model.train()是随机取一部分网络连接来训练更新参数，而model.eval()是利用到了所有网络连接。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 模型训练函数</span></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">fit_train</span>(<span class="params">model,data_loader</span>):</span><br><span class="line"> model.train()</span><br><span class="line"> running_loss = <span class="number">0</span></span><br><span class="line"> running_correct = <span class="number">0</span></span><br><span class="line"> <span class="keyword">for</span> batch_idx, (data, target) <span class="keyword">in</span> <span class="built_in">enumerate</span>(data_loader):</span><br><span class="line"> <span class="keyword">if</span> is_cuda:</span><br><span class="line"> data, target = data.cuda(), target.cuda()</span><br><span class="line"> optimizer.zero_grad()</span><br><span class="line"> output = model(data)</span><br><span class="line"> loss = F.nll_loss(output, target)</span><br><span class="line"> running_loss += F.nll_loss(output ,target ,reduction=<span class="string">'sum'</span>).item()</span><br><span class="line"> preds = output.data.<span class="built_in">max</span>(<span class="number">1</span>, keepdim=<span class="literal">True</span>)[<span class="number">1</span>]</span><br><span class="line"> running_correct += preds.eq(target.data.view_as(preds)).cpu().<span class="built_in">sum</span>()</span><br><span class="line"> loss.backward()</span><br><span class="line"> optimizer.step()</span><br><span class="line"> loss = running_loss/<span class="built_in">len</span>(data_loader.dataset)</span><br><span class="line"> accuracy = <span class="number">100.</span> * running_correct/<span class="built_in">len</span>(data_loader.dataset)</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">f"Train loss is <span class="subst">{loss:{<span class="number">5</span>}</span>.<span class="subst">{<span class="number">2</span>}</span>} and Train accuracy is <span class="subst">{accuracy:{<span class="number">10</span>}</span>.<span class="subst">{<span class="number">4</span>}</span>} %"</span>)</span><br><span class="line"> <span class="keyword">return</span> loss, accuracy</span><br><span class="line"></span><br><span class="line"><span class="comment"># 模型评估函数</span></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">fit_eval</span>(<span class="params">model,data_loader</span>):</span><br><span class="line"> model.<span class="built_in">eval</span>()</span><br><span class="line"> running_loss = <span class="number">0</span></span><br><span class="line"> running_correct = <span class="number">0</span></span><br><span class="line"> <span class="keyword">for</span> batch_idx, (data, target) <span class="keyword">in</span> <span class="built_in">enumerate</span>(data_loader):</span><br><span class="line"> <span class="keyword">if</span> is_cuda:</span><br><span class="line"> data, target = data.cuda(), target.cuda()</span><br><span class="line"> output = model(data)</span><br><span class="line"> loss = F.nll_loss(output, target)</span><br><span class="line"> running_loss += F.nll_loss(output ,target ,reduction=<span class="string">'sum'</span>).item()</span><br><span class="line"> preds = output.data.<span class="built_in">max</span>(<span class="number">1</span>, keepdim=<span class="literal">True</span>)[<span class="number">1</span>]</span><br><span class="line"> running_correct += preds.eq(target.data.view_as(preds)).cpu().<span class="built_in">sum</span>()</span><br><span class="line"> loss = running_loss/<span class="built_in">len</span>(data_loader.dataset)</span><br><span class="line"> accuracy = <span class="number">100.</span> * running_correct/<span class="built_in">len</span>(data_loader.dataset)</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">f"Eval loss is <span class="subst">{loss:{<span class="number">5</span>}</span>.<span class="subst">{<span class="number">2</span>}</span>} and Eval accuracy is <span class="subst">{accuracy:{<span class="number">10</span>}</span>.<span class="subst">{<span class="number">4</span>}</span>} %"</span>)</span><br><span class="line"> <span class="keyword">return</span> loss, accuracy</span><br></pre></td></tr></table></figure><p>初始化模型和配置优化函数。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">model = MnistNet()</span><br><span class="line"><span class="keyword">if</span> is_cuda:</span><br><span class="line"> model.cuda()</span><br><span class="line">optimizer = optim.SGD(model.parameters(),lr=<span class="number">0.01</span>)</span><br><span class="line"></span><br></pre></td></tr></table></figure><h2 id="模型评估"><a href="#模型评估" class="headerlink" title="模型评估"></a>模型评估</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">train_losses, train_accuracy = [], []</span><br><span class="line">val_losses, val_accuracy = [], []</span><br><span class="line"><span class="keyword">for</span> epoch <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">1</span>, <span class="number">20</span>):</span><br><span class="line"> epoch_loss, epoch_accuracy = fit_train(model, mnist_train_loader)</span><br><span class="line"> val_epoch_loss, val_epoch_accuracy = fit_eval(model, mnist_test_loader)</span><br><span class="line"> train_losses.append(epoch_loss)</span><br><span class="line"> train_accuracy.append(epoch_accuracy)</span><br><span class="line"> val_losses.append(val_epoch_loss)</span><br><span class="line"> val_accuracy.append(val_epoch_accuracy)</span><br></pre></td></tr></table></figure><p><img src="/2022/05/29/Pytorch%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E6%95%99%E7%A8%8B-2-%E2%80%9CHello-World%E2%80%9D%E4%B9%8BMnist%E5%9B%BE%E5%83%8F%E5%88%86%E7%B1%BB/train_eval_vis.png" alt="样本图片"></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">plt.plot(<span class="built_in">range</span>(<span class="number">1</span>,<span class="built_in">len</span>(train_losses)+<span class="number">1</span>),train_losses,<span class="string">'bo'</span>,label = <span class="string">'training loss'</span>)</span><br><span class="line">plt.plot(<span class="built_in">range</span>(<span class="number">1</span>,<span class="built_in">len</span>(val_losses)+<span class="number">1</span>),val_losses,<span class="string">'r'</span>,label = <span class="string">'validation loss'</span>)</span><br><span class="line">plt.legend()</span><br><span class="line">plt.show()</span><br><span class="line"></span><br></pre></td></tr></table></figure><p><img src="/2022/05/29/Pytorch%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E6%95%99%E7%A8%8B-2-%E2%80%9CHello-World%E2%80%9D%E4%B9%8BMnist%E5%9B%BE%E5%83%8F%E5%88%86%E7%B1%BB/loss_vis.png" alt="样本图片"></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">plt.plot(<span class="built_in">range</span>(<span class="number">1</span>,<span class="built_in">len</span>(train_accuracy)+<span class="number">1</span>),train_accuracy,<span class="string">'bo'</span>,label = <span class="string">'train accuracy'</span>)</span><br><span class="line">plt.plot(<span class="built_in">range</span>(<span class="number">1</span>,<span class="built_in">len</span>(val_accuracy)+<span class="number">1</span>),val_accuracy,<span class="string">'r'</span>,label = <span class="string">'val accuracy'</span>)</span><br><span class="line">plt.legend()</span><br><span class="line">plt.show()</span><br></pre></td></tr></table></figure><p><img src="/2022/05/29/Pytorch%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E6%95%99%E7%A8%8B-2-%E2%80%9CHello-World%E2%80%9D%E4%B9%8BMnist%E5%9B%BE%E5%83%8F%E5%88%86%E7%B1%BB/accuracy_vis.png" alt="样本图片"></p><h2 id="探索"><a href="#探索" class="headerlink" title="探索"></a>探索</h2><p><a class="link" href="https://github.com/yqstar/Awesome_Pytorch_Tutorial/tree/master/Pytorch_Lesson2">完整代码<i class="fas fa-external-link-alt"></i></a>已上传Github，有需要的可以自行下载代码，如果对你有帮助，请Star，哈哈哈哈！</p><p>到此为止，已经可以使用自己数据玩耍各种Demo，快（苦）乐（逼）地进行炼丹之路。道路阻且长，行则将至，但行好事莫问前程。</p><ul><li>Demo模型还可以修改Loss函数，优化函数和模型结构？</li><li>除了Torchinfo，是否可以使用Tensorboard等可视化工具管理模型数据的可视化数据呢？</li><li>Demo模型离生产上线还有多远距离？如何上线部署模型呢？如何实现从Mysql等数据库到线上模型调用呢？任何的代码是否都应该以能够终端使用为目的？</li></ul><p>正如，人往往会对未知的事情产生恐惧，因为结局是未知的。所以当一切不再未知的时候，那么是不是就不会产生恐惧呢？</p>]]></content>
<summary type="html"><h2 id="简介"><a href="#简介" class="headerlink" title="简介"></a>简介</h2><p>本章节主要通过深度学习“Hello World”之Mnist图像分类，学会深度学习的基本链路，快速搭建个人的Baseline模型，包括数据加</summary>
<category term="深度学习" scheme="https://yqstar.github.io/categories/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0/"/>
<category term="Pytorch_Tutorial" scheme="https://yqstar.github.io/tags/Pytorch-Tutorial/"/>
</entry>
<entry>
<title>Pytorch深度学习教程(1):数据加载之Dataset和DataLoader使用</title>
<link href="https://yqstar.github.io/2022/05/22/Pytorch%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E6%95%99%E7%A8%8B-1-%E6%95%B0%E6%8D%AE%E5%8A%A0%E8%BD%BD%E4%B9%8BDataset%E5%92%8CDataLoader%E4%BD%BF%E7%94%A8/"/>
<id>https://yqstar.github.io/2022/05/22/Pytorch%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E6%95%99%E7%A8%8B-1-%E6%95%B0%E6%8D%AE%E5%8A%A0%E8%BD%BD%E4%B9%8BDataset%E5%92%8CDataLoader%E4%BD%BF%E7%94%A8/</id>
<published>2022-05-22T15:03:19.000Z</published>
<updated>2022-06-12T14:44:03.393Z</updated>
<content type="html"><![CDATA[<p>深度学习模型，区别于其他的机器学习模型，一方面，模型训练所需的数据量通常是非常大的，是无法一次性加载到内存中；另一方面，模型训练多采用基于梯度下降的优化方法对模型的权重和偏置进行逐步调整的，不可能一次性地在模型中进行正向传播和反向传播。通常，我们需要进行数据加载和预处理处理，将其封装成适合迭代训练的形式，具体会对整个数据进行随机打乱，然后将原始数据处理成一个一个的Batch，然后送到模型中进行训练。</p><p>深度学习模型流程中一般都是先解决数据加载问题，包括数据的<strong>输入问题</strong>和<strong>预处理问题</strong>，数据加载处理在深度学习链路中起着非常重要的基础作用。这篇文章将介绍Pytorch对自定义数据集进行封装的方法。</p><h2 id="Dataset、Batch、Iteration和Epoch的关系"><a href="#Dataset、Batch、Iteration和Epoch的关系" class="headerlink" title="Dataset、Batch、Iteration和Epoch的关系"></a>Dataset、Batch、Iteration和Epoch的关系</h2><p>在介绍如何使用Pytorch加载数据前，简单介绍下，<strong>Dataset</strong>，<strong>Batch</strong>，<strong>Iteration</strong> 和 <strong>Epoch</strong> 的区别和关系。</p><div class="table-container"><table><thead><tr><th style="text-align:center">名词</th><th style="text-align:center">解释</th></tr></thead><tbody><tr><td style="text-align:center"> Dataset</td><td style="text-align:center">待训练的全量数据集</td></tr><tr><td style="text-align:center"> Batch</td><td style="text-align:center">待训练全量数据集的一小部分样本对模型进行一次反向传播参数更新，这一小部分样本称为“一个Batch”</td></tr><tr><td style="text-align:center"> Iteration</td><td style="text-align:center">使用一个Batch数据对模型进行一次参数更新的过程，称之为“一次Iteration”</td></tr><tr><td style="text-align:center"> Epoch</td><td style="text-align:center">待训练全量数据集对模型进行一次完整的参数更新，称之为“一个Epoch”</td></tr></tbody></table></div><p>假设DatasetSize=10，BatchSize=3，那么每个Epoch会执行4个Iteration，对应四个Batch，每个BatchSize大小分别包括3，3，3和1个样本。</p><p><img src="/2022/05/22/Pytorch%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E6%95%99%E7%A8%8B-1-%E6%95%B0%E6%8D%AE%E5%8A%A0%E8%BD%BD%E4%B9%8BDataset%E5%92%8CDataLoader%E4%BD%BF%E7%94%A8/dataset.jpg" alt="data"></p><h2 id="Pytoch数据处理：DataSet和DataLoader"><a href="#Pytoch数据处理：DataSet和DataLoader" class="headerlink" title="Pytoch数据处理：DataSet和DataLoader"></a>Pytoch数据处理：DataSet和DataLoader</h2><p>Pytorch提供了几个有用的工具：<strong>torch.utils.data.Dataset类</strong>和<strong>torch.utils.data.DataLoader类</strong>，用于数据读取和预处理。<br>基本流程是先把原始数据转变成torch.utils.data.Dataset类，随后把得到的torch.utils.data.Dataset类当作一个参数传递给torch.utils.data.DataLoader类，得到一个数据加载器，这个数据加载器每次可以返回一个Batch的数据供模型训练使用。</p><h3 id="torch-utils-data-Dataset类"><a href="#torch-utils-data-Dataset类" class="headerlink" title="torch.utils.data.Dataset类"></a>torch.utils.data.Dataset类</h3><p><em>torch.utils.data.Dataset</em>是代表这一数据的抽象类，你可以自己定义数据类，继承和重写这个抽象类，只需要定义<strong>init</strong>，<strong>len</strong>和<strong>getitem</strong>这三个魔法函数,其中：</p><ul><li>_<em>init_</em>()：用于初始化原始数据的路径和文件名等。</li><li>_<em>len_</em>()：用于返回数据集中的样本总个数。</li><li>_<em>getitem_</em>()：用于返回指定索引的样本所对应的输入变量与输出变量。</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># class CustomDataset(torch.utils.data.Dataset):#需要继承data.Dataset</span></span><br><span class="line"><span class="comment"># def __init__(self):</span></span><br><span class="line"><span class="comment"># # TODO</span></span><br><span class="line"><span class="comment"># # 1. Initialize file path or list of file names.</span></span><br><span class="line"><span class="comment"># pass</span></span><br><span class="line"><span class="comment"># def __getitem__(self, index):</span></span><br><span class="line"><span class="comment"># # TODO</span></span><br><span class="line"><span class="comment"># # 1. Read one data from file (e.g. using numpy.fromfile, PIL.Image.open).</span></span><br><span class="line"><span class="comment"># # 2. Preprocess the data (e.g. torchvision.Transform).</span></span><br><span class="line"><span class="comment"># # 3. Return a data pair (e.g. image and label).</span></span><br><span class="line"><span class="comment"># #这里需要注意的是，第一步：read one data，是一个data</span></span><br><span class="line"><span class="comment"># pass</span></span><br><span class="line"><span class="comment"># def __len__(self):</span></span><br><span class="line"><span class="comment"># # You should change 0 to the total size of your dataset.</span></span><br><span class="line"><span class="comment"># pass</span></span><br></pre></td></tr></table></figure><p>用原始数据构造出来的<em>Dataset子类</em>可以理解成一个静态数据池，这个数据池使得我们可以用<em>索引</em>得到某个样本数据，而想要该数据池流动起来，源源不断地输出<em>Batch</em>供给给模型训练，还需要下一个工具<em>DataLoader类</em>。所以我们把创建的<em>Dataset子类</em>当参数传入即将构建的<em>DataLoader类</em>才是使用<em>Dataset子类</em>最终目的。</p><h3 id="torch-utils-data-DataLoader类"><a href="#torch-utils-data-DataLoader类" class="headerlink" title="torch.utils.data.DataLoader类"></a>torch.utils.data.DataLoader类</h3><p>DataLoader(object)可用参数:</p><ul><li>dataset(Dataset): 传入的数据集。</li><li>batch_size(int, optional): 每个batch有多少样本。</li><li>shuffle(bool, optional): 在每个epoch开始时，对数据进行打乱排序。</li><li>sampler(Sampler, optional): 自定义从数据集中取样本的策略，如果指定这个参数，那么shuffle必须为False。</li><li>batch_sampler(Sampler, optional): 与sampler类似，但是一次只返回一个batch的indices（索引），需要注意的是，一旦指定了这个参数，那么batch_size,shuffle,sampler,drop_last就不能再配置（互斥）。</li><li>num_workers (int, optional): 决定了有几个进程来处理data loading。0意味着所有的数据都会被load进主进程。（默认为0）</li><li>collate_fn (callable, optional): 将一个list的sample组成一个mini-batch的函数。</li><li>pin_memory (bool, optional)： 如果设置为True，那么data loader将会在返回它们之前，将tensors拷贝到CUDA中的固定内存中。</li><li>drop_last (bool, optional):如果设置为True：这个是对最后的未完成的batch来说的，比如batch_size设置为64，而一个epoch只有100个样本，那么训练时后面36个样本会被丢弃。 如果为False（默认），那么会继续正常执行，只是最后的batch_size会小一点。</li><li>timeout(numeric, optional):如果是正数，表明等待从worker进程中收集一个batch等待时间，若超出设定时间还没有收集到，那就不收集这个内容。这个numeric应总是大于等于0。默认为0</li><li>worker_init_fn (callable, optional): 每个worker初始化函数。</li></ul><h2 id="实例"><a href="#实例" class="headerlink" title="实例"></a>实例</h2><h3 id="txt数据读取"><a href="#txt数据读取" class="headerlink" title="txt数据读取"></a>txt数据读取</h3><p>使用个人创建的txt文件数据，进行数据读取操作。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> torch</span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"><span class="keyword">from</span> torch.utils.data <span class="keyword">import</span> Dataset, DataLoader</span><br><span class="line"></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">SampleTxtDataset</span>(<span class="title class_ inherited__">Dataset</span>):</span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">__init__</span>(<span class="params">self</span>):</span><br><span class="line"> <span class="comment"># 数据文件地址</span></span><br><span class="line"> self.txt_file_path = <span class="string">"./sample_easy_data.txt"</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">__getitem__</span>(<span class="params">self, item</span>):</span><br><span class="line"> txt_data = np.loadtxt(self.txt_file_path, delimiter=<span class="string">","</span>)</span><br><span class="line"> self._x = torch.from_numpy(txt_data[:, :<span class="number">2</span>])</span><br><span class="line"> self._y = torch.from_numpy(txt_data[:, <span class="number">2</span>])</span><br><span class="line"> <span class="keyword">return</span> self._x[item], self._y[item]</span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">__len__</span>(<span class="params">self</span>):</span><br><span class="line"> txt_data = np.loadtxt(self.txt_file_path, delimiter=<span class="string">","</span>)</span><br><span class="line"> self._<span class="built_in">len</span> = <span class="built_in">len</span>(txt_data)</span><br><span class="line"> <span class="keyword">return</span> self._<span class="built_in">len</span></span><br><span class="line"></span><br><span class="line">sample_txt_dataset = SampleTxtDataset()</span><br><span class="line"></span><br><span class="line"><span class="built_in">print</span>(<span class="string">"Data Size:"</span>,<span class="built_in">len</span>(sample_txt_dataset))</span><br><span class="line"></span><br><span class="line"><span class="built_in">print</span>(<span class="string">"First Sample:"</span>,<span class="built_in">next</span>(<span class="built_in">iter</span>(sample_txt_dataset)))</span><br><span class="line"></span><br><span class="line"><span class="built_in">print</span>(<span class="string">"First Sample's Type:"</span>,<span class="built_in">type</span>(<span class="built_in">next</span>(<span class="built_in">iter</span>(sample_txt_dataset))[<span class="number">0</span>]))</span><br><span class="line"></span><br><span class="line">sample_dataloader = DataLoader(dataset=sample_txt_dataset, batch_size=<span class="number">3</span>, shuffle=<span class="literal">True</span>)</span><br><span class="line"></span><br><span class="line">num_epochs = <span class="number">4</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span> epoch <span class="keyword">in</span> <span class="built_in">range</span>(num_epochs):</span><br><span class="line"> <span class="keyword">for</span> iteration, (batch_x, batch_y) <span class="keyword">in</span> <span class="built_in">enumerate</span>(sample_dataloader):</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">'Epoch: '</span>, epoch, <span class="string">'| Iteration: '</span>, iteration, <span class="string">'| batch x: '</span>, batch_x.numpy(), <span class="string">'| batch y: '</span>, batch_y.numpy())</span><br><span class="line"></span><br></pre></td></tr></table></figure><p>Dataset的示例结果：</p><p><img src="/2022/05/22/Pytorch%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E6%95%99%E7%A8%8B-1-%E6%95%B0%E6%8D%AE%E5%8A%A0%E8%BD%BD%E4%B9%8BDataset%E5%92%8CDataLoader%E4%BD%BF%E7%94%A8/dataset_tutorial.jpg" alt="dataset"></p><p>DataLoader的示例结果：</p><p><img src="/2022/05/22/Pytorch%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E6%95%99%E7%A8%8B-1-%E6%95%B0%E6%8D%AE%E5%8A%A0%E8%BD%BD%E4%B9%8BDataset%E5%92%8CDataLoader%E4%BD%BF%E7%94%A8/dataloader_tutorial.jpg" alt="dataloader"></p><h3 id="csv文件读取"><a href="#csv文件读取" class="headerlink" title="csv文件读取"></a>csv文件读取</h3><p>使用常见离线数据csv文件进行数据加载和预处理。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> torch</span><br><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line"><span class="keyword">from</span> torch.utils.data <span class="keyword">import</span> Dataset, DataLoader</span><br><span class="line"></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">SampleCsvDataset</span>(<span class="title class_ inherited__">Dataset</span>):</span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">__init__</span>(<span class="params">self</span>):</span><br><span class="line"> self.csv_file_path = <span class="string">"./sample_boston.csv"</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">__getitem__</span>(<span class="params">self, item</span>):</span><br><span class="line"> raw_data = pd.read_csv(self.csv_file_path)</span><br><span class="line"> raw_data_shape = raw_data.shape</span><br><span class="line"> self._x = torch.from_numpy(raw_data.iloc[:,:raw_data_shape[<span class="number">1</span>]-<span class="number">1</span>].values)</span><br><span class="line"> self._y = torch.from_numpy(raw_data.iloc[:,raw_data_shape[<span class="number">1</span>]-<span class="number">1</span>].values)</span><br><span class="line"> <span class="keyword">return</span> self._x[item], self._y[item]</span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">__len__</span>(<span class="params">self</span>):</span><br><span class="line"> raw_data = pd.read_csv(self.csv_file_path)</span><br><span class="line"> raw_data_shape = raw_data.shape</span><br><span class="line"> self._<span class="built_in">len</span> = raw_data_shape[<span class="number">0</span>]</span><br><span class="line"> <span class="keyword">return</span> self._<span class="built_in">len</span></span><br><span class="line"></span><br><span class="line">sample_csv_dataset = SampleCsvDataset()</span><br><span class="line"></span><br><span class="line"><span class="built_in">print</span>(<span class="string">"Data Size:"</span>,<span class="built_in">len</span>(sample_csv_dataset))</span><br><span class="line"></span><br><span class="line"><span class="built_in">print</span>(<span class="string">"First Sample:"</span>,<span class="built_in">next</span>(<span class="built_in">iter</span>(sample_csv_dataset)))</span><br><span class="line"></span><br><span class="line"><span class="built_in">print</span>(<span class="string">"First Sample's Type:"</span>,<span class="built_in">type</span>(<span class="built_in">next</span>(<span class="built_in">iter</span>(sample_csv_dataset))[<span class="number">0</span>]))</span><br><span class="line"></span><br><span class="line">sample_dataloader = DataLoader(dataset=sample_csv_dataset, batch_size=<span class="number">3</span>, shuffle=<span class="literal">True</span>)</span><br><span class="line"></span><br><span class="line">num_epochs = <span class="number">4</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span> epoch <span class="keyword">in</span> <span class="built_in">range</span>(num_epochs):</span><br><span class="line"> <span class="keyword">for</span> iteration, (batch_x, batch_y) <span class="keyword">in</span> <span class="built_in">enumerate</span>(sample_dataloader):</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">'Epoch: '</span>, epoch, <span class="string">'| Iteration: '</span>, iteration, <span class="string">'| batch x: '</span>, batch_x.numpy(), <span class="string">'| batch y: '</span>, batch_y.numpy())</span><br><span class="line"></span><br></pre></td></tr></table></figure><h3 id="mysql数据读取"><a href="#mysql数据读取" class="headerlink" title="mysql数据读取"></a>mysql数据读取</h3><p>生产落地数据多为数据库，本文也针对常见Mysql数据库进行了数据加载，使用的是MYSQL8.0数据库的示例数据库sakila.payment表进行数据读取演示。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> torch</span><br><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line"><span class="keyword">import</span> pymysql</span><br><span class="line"><span class="keyword">from</span> torch.utils.data <span class="keyword">import</span> Dataset, DataLoader</span><br><span class="line"></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">SampleMysqlDataset</span>(<span class="title class_ inherited__">Dataset</span>):</span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">__init__</span>(<span class="params">self</span>):</span><br><span class="line"> <span class="comment"># 初始化MySQL数据库连接配置参数</span></span><br><span class="line"> self.mysql_host = <span class="string">"localhost"</span></span><br><span class="line"> self.mysql_port = <span class="number">3307</span></span><br><span class="line"> self.mysql_user = <span class="string">"utest"</span></span><br><span class="line"> self.mysql_password = <span class="string">"123456xyq"</span></span><br><span class="line"> self.mysql_db = <span class="string">"sakila"</span></span><br><span class="line"> self.mysql_table = <span class="string">"payment"</span></span><br><span class="line"> self.mysql_charset = <span class="string">"utf8"</span></span><br><span class="line"> self.mysql_sql_data = <span class="string">"select payment_id, customer_id, staff_id, rental_id, amount from sakila.payment"</span></span><br><span class="line"> self.mysql_sql_cnt = <span class="string">"select count(*) from sakila.payment"</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">__getitem__</span>(<span class="params">self, item</span>):</span><br><span class="line"> <span class="comment"># 创建数据库连接</span></span><br><span class="line"> conn = pymysql.connect(host=self.mysql_host,</span><br><span class="line"> port=self.mysql_port,</span><br><span class="line"> user=self.mysql_user,</span><br><span class="line"> password=self.mysql_password,</span><br><span class="line"> db=self.mysql_db,</span><br><span class="line"> charset=self.mysql_charset)</span><br><span class="line"> raw_dataframe = pd.read_sql(self.mysql_sql_data, conn)</span><br><span class="line"> raw_dataframe_shape = raw_dataframe.shape</span><br><span class="line"> self._x = torch.from_numpy(raw_dataframe.iloc[:,:raw_dataframe_shape[<span class="number">1</span>]-<span class="number">1</span>].values)</span><br><span class="line"> self._y = torch.from_numpy(raw_dataframe.iloc[:,raw_dataframe_shape[<span class="number">1</span>]-<span class="number">1</span>].values)</span><br><span class="line"> <span class="keyword">return</span> self._x[item], self._y[item]</span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">__len__</span>(<span class="params">self</span>):</span><br><span class="line"> <span class="comment"># 创建数据库连接</span></span><br><span class="line"> conn = pymysql.connect(host=self.mysql_host,</span><br><span class="line"> port=self.mysql_port,</span><br><span class="line"> user=self.mysql_user,</span><br><span class="line"> password=self.mysql_password,</span><br><span class="line"> db=self.mysql_db,</span><br><span class="line"> charset=self.mysql_charset)</span><br><span class="line"> raw_dataframe = pd.read_sql(self.mysql_sql_data, conn)</span><br><span class="line"> raw_dataframe_shape = raw_dataframe.shape</span><br><span class="line"> self._<span class="built_in">len</span> = raw_dataframe_shape[<span class="number">0</span>]</span><br><span class="line"> <span class="keyword">return</span> self._<span class="built_in">len</span></span><br><span class="line"></span><br><span class="line">sample_mysql_dataset = SampleMysqlDataset()</span><br><span class="line"></span><br><span class="line"><span class="built_in">print</span>(<span class="string">"Data Size:"</span>,<span class="built_in">len</span>(sample_mysql_dataset))</span><br><span class="line"></span><br><span class="line"><span class="built_in">print</span>(<span class="string">"First Sample:"</span>,<span class="built_in">next</span>(<span class="built_in">iter</span>(sample_mysql_dataset)))</span><br><span class="line"></span><br><span class="line"><span class="built_in">print</span>(<span class="string">"First Sample's Type:"</span>,<span class="built_in">type</span>(<span class="built_in">next</span>(<span class="built_in">iter</span>(sample_mysql_dataset))[<span class="number">0</span>]))</span><br><span class="line"></span><br><span class="line">sample_dataloader = DataLoader(dataset=sample_mysql_dataset, batch_size=<span class="number">3</span>, shuffle=<span class="literal">True</span>)</span><br><span class="line"></span><br><span class="line">num_epochs = <span class="number">4</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span> epoch <span class="keyword">in</span> <span class="built_in">range</span>(num_epochs):</span><br><span class="line"> <span class="keyword">for</span> iteration, (batch_x, batch_y) <span class="keyword">in</span> <span class="built_in">enumerate</span>(sample_dataloader):</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">'Epoch: '</span>, epoch, <span class="string">'| Iteration: '</span>, iteration, <span class="string">'| batch x: '</span>, batch_x.numpy(), <span class="string">'| batch y: '</span>, batch_y.numpy())</span><br><span class="line"></span><br></pre></td></tr></table></figure><h3 id="使用pytorch自带数据集"><a href="#使用pytorch自带数据集" class="headerlink" title="使用pytorch自带数据集"></a>使用pytorch自带数据集</h3><p>为方便快速试验，Pytorch也集成了常见的数据集在torchaudio，torchtext和torchvision中，本代码使用torchvision读取常用的图像算法数据集MNIST，具体代码如下。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> torchvision <span class="keyword">import</span> datasets, transforms</span><br><span class="line"><span class="keyword">from</span> torch.utils.data <span class="keyword">import</span> DataLoader</span><br><span class="line"></span><br><span class="line"><span class="comment"># 导入训练集</span></span><br><span class="line">sample_mnist_dataset = datasets.MNIST(root=<span class="string">r'./data'</span>,</span><br><span class="line"> transform=transforms.ToTensor(),</span><br><span class="line"> train=<span class="literal">True</span>,</span><br><span class="line"> download=<span class="literal">True</span>)</span><br><span class="line"></span><br><span class="line"><span class="built_in">print</span>(<span class="string">"Data Size:"</span>,<span class="built_in">len</span>(sample_mnist_dataset))</span><br><span class="line"></span><br><span class="line"><span class="built_in">print</span>(<span class="string">"First Sample:"</span>,<span class="built_in">next</span>(<span class="built_in">iter</span>(sample_mnist_dataset)))</span><br><span class="line"></span><br><span class="line"><span class="built_in">print</span>(<span class="string">"First Sample's Type:"</span>,<span class="built_in">type</span>(<span class="built_in">next</span>(<span class="built_in">iter</span>(sample_mnist_dataset))[<span class="number">0</span>]))</span><br><span class="line"></span><br><span class="line">sample_dataloader = DataLoader(dataset=sample_mnist_dataset, batch_size=<span class="number">3</span>, shuffle=<span class="literal">True</span>)</span><br><span class="line"></span><br><span class="line">num_epochs = <span class="number">4</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span> epoch <span class="keyword">in</span> <span class="built_in">range</span>(num_epochs):</span><br><span class="line"> <span class="keyword">for</span> <span class="built_in">iter</span>, (batch_x, batch_y) <span class="keyword">in</span> <span class="built_in">enumerate</span>(sample_dataloader):</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">'Epoch: '</span>, epoch, <span class="string">'| Iteration: '</span>, <span class="built_in">iter</span>, <span class="string">'| batch x: '</span>, batch_x.numpy(), <span class="string">'| batch y: '</span>, batch_y.numpy())</span><br><span class="line"></span><br></pre></td></tr></table></figure><h2 id="探索"><a href="#探索" class="headerlink" title="探索"></a>探索</h2><p><a class="link" href="https://github.com/yqstar/Awesome_Pytorch_Tutorial/tree/master/Pytorch_Lesson1">完整代码<i class="fas fa-external-link-alt"></i></a>已上传Github，有需要的可以自行下载代码，如果对你有帮助，请Star，哈哈哈哈！</p><ul><li><p>生产读取大量数据无法一次加载到内存该如何操作呢？</p></li><li><p>如何使用TorchData进行数据读取和预处理？</p></li></ul><h2 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h2><p>More info: <a class="link" href="https://blog.csdn.net/guyuealian/article/details/88343924">pan_jinquan：Dataset, DataLoader产生自定义的训练数据<i class="fas fa-external-link-alt"></i></a></p><p>More info: <a class="link" href="https://www.jianshu.com/p/4818a1a4b5bd">夜和大帝：Dataset类的使用<i class="fas fa-external-link-alt"></i></a></p><p>More info: <a class="link" href="https://github.com/setail/pytorch_tutorial">setail：pytorch_tutorial<i class="fas fa-external-link-alt"></i></a></p><p>More info: <a class="link" href="https://blog.csdn.net/xjm850552586/article/details/109137914">Ericam_：十分钟搞懂Pytorch如何读取MNIST数据集<i class="fas fa-external-link-alt"></i></a></p><p>More info: <a class="link" href="https://zhuanlan.zhihu.com/p/105507334">Chenllliang：两文读懂PyTorch中Dataset与DataLoader（一）打造自己的数据集<i class="fas fa-external-link-alt"></i></a></p><p>More Info: <a class="link" href="https://blog.csdn.net/weixin_37913042/article/details/122129030">cici_iii：大数据量下如何使用Dataset和IterDataset构建数据集<i class="fas fa-external-link-alt"></i></a></p><p>More Info: <a class="link" href="https://blog.csdn.net/sdnuwjw/article/details/111227327">csdn-WJW: 如何划分训练集，测试集和验证集<i class="fas fa-external-link-alt"></i></a></p>]]></content>
<summary type="html"><p>深度学习模型，区别于其他的机器学习模型，一方面，模型训练所需的数据量通常是非常大的，是无法一次性加载到内存中；另一方面，模型训练多采用基于梯度下降的优化方法对模型的权重和偏置进行逐步调整的，不可能一次性地在模型中进行正向传播和反向传播。通常，我们需要进行数据加载和预处理处理</summary>
<category term="深度学习" scheme="https://yqstar.github.io/categories/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0/"/>
<category term="Pytorch_Tutorial" scheme="https://yqstar.github.io/tags/Pytorch-Tutorial/"/>
</entry>
<entry>
<title>Windows系统使用Conda配置深度学习环境</title>
<link href="https://yqstar.github.io/2022/05/15/Windows%E7%B3%BB%E7%BB%9F%E4%BD%BF%E7%94%A8Conda%E9%85%8D%E7%BD%AE%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E7%8E%AF%E5%A2%83/"/>
<id>https://yqstar.github.io/2022/05/15/Windows%E7%B3%BB%E7%BB%9F%E4%BD%BF%E7%94%A8Conda%E9%85%8D%E7%BD%AE%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E7%8E%AF%E5%A2%83/</id>
<published>2022-05-15T11:48:46.000Z</published>
<updated>2022-06-12T13:40:30.083Z</updated>
<content type="html"><![CDATA[<h2 id="Conda环境管理"><a href="#Conda环境管理" class="headerlink" title="Conda环境管理"></a>Conda环境管理</h2><p>Conda是一个开源软件包管理系统和环境管理系统，可以管理不同Python版本环境，不同的环境之间是互相隔离，互不影响的。</p><h3 id="查看环境"><a href="#查看环境" class="headerlink" title="查看环境"></a>查看环境</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 查看当前环境</span></span><br><span class="line">conda info --<span class="built_in">env</span></span><br></pre></td></tr></table></figure><h3 id="克隆环境"><a href="#克隆环境" class="headerlink" title="克隆环境"></a>克隆环境</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 假设已有环境名为A，需要生成的环境名为B：</span></span><br><span class="line">conda create -n B --<span class="built_in">clone</span> A</span><br><span class="line"></span><br><span class="line"><span class="comment"># 如果特殊环境为Base，需要采用以下方式</span></span><br><span class="line">conda update conda</span><br><span class="line">conda create -n <my_env> --<span class="built_in">clone</span> root</span><br><span class="line">conda create -n torch_env --<span class="built_in">clone</span> root</span><br><span class="line">conda install pytorch=0.4.0 cuda90 -c pytorch</span><br><span class="line"></span><br><span class="line"><span class="comment"># 用于复制环境到新的机器</span></span><br><span class="line">conda list --explicit > spec-file.txt</span><br><span class="line">conda create --name <my_env> --file spec-file.txt</span><br></pre></td></tr></table></figure><h3 id="创建环境"><a href="#创建环境" class="headerlink" title="创建环境"></a>创建环境</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 创建一个环境名为py34，指定Python版本是3.4 </span></span><br><span class="line"><span class="comment">#（不用管是3.4.x，conda会为我们自动寻找3.4.x中的最新版本） </span></span><br><span class="line">conda create --name py34 python=3.4 </span><br><span class="line"></span><br><span class="line"><span class="comment"># 通过创建环境，我们可以使用不同版本的Python </span></span><br><span class="line">conda create --name py27 python=2.7</span><br></pre></td></tr></table></figure><h3 id="激活环境"><a href="#激活环境" class="headerlink" title="激活环境"></a>激活环境</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 在windows环境下使用activate激活 </span></span><br><span class="line">activate py34</span><br><span class="line"></span><br><span class="line"><span class="comment"># 在Linux & Mac中使用source activate激活 </span></span><br><span class="line"><span class="built_in">source</span> activate py34 </span><br></pre></td></tr></table></figure><h3 id="退出环境"><a href="#退出环境" class="headerlink" title="退出环境"></a>退出环境</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 在windows环境下使用</span></span><br><span class="line">deactivate <my_env></span><br><span class="line"></span><br><span class="line"><span class="comment"># 在Linux & Mac中使用</span></span><br><span class="line"><span class="built_in">source</span> deactivate <my_env></span><br></pre></td></tr></table></figure><h3 id="删除环境"><a href="#删除环境" class="headerlink" title="删除环境"></a>删除环境</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 如果你不想要这个名为py34的环境，可以通过以下命令删除这个环境。 </span></span><br><span class="line">conda remove -n py34 --all </span><br><span class="line"></span><br><span class="line"><span class="comment"># 可以通过以下命令查看已有的环境列表，现在py34已经不在这个列表里。 </span></span><br><span class="line">conda info -e</span><br></pre></td></tr></table></figure><h3 id="配置镜像"><a href="#配置镜像" class="headerlink" title="配置镜像"></a>配置镜像</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 显示目前的channels </span></span><br><span class="line">conda config --show channels </span><br><span class="line"><span class="comment"># 切换默认镜像源 </span></span><br><span class="line">conda config --remove-key channels</span><br><span class="line"></span><br><span class="line"><span class="comment"># 删除指定channel </span></span><br><span class="line">conda config --remove channels_URL </span><br><span class="line"></span><br><span class="line"><span class="comment"># Windows 用户无法直接创建名为 .condarc 的文件，使用以下命令 C:\Users\用户名\.condarc</span></span><br><span class="line">conda config --<span class="built_in">set</span> show_channel_urls <span class="built_in">yes</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 中科大镜像源 </span></span><br><span class="line">conda config --add channels https://mirrors.ustc.edu.cn/anaconda/pkgs/main/ </span><br><span class="line">conda config --add channels https://mirrors.ustc.edu.cn/anaconda/pkgs/free/</span><br><span class="line">conda config --add channels https://mirrors.ustc.edu.cn/anaconda/cloud/conda-forge/ </span><br><span class="line">conda config --add channels https://mirrors.ustc.edu.cn/anaconda/cloud/msys2/ </span><br><span class="line">conda config --add channels https://mirrors.ustc.edu.cn/anaconda/cloud/bioconda/ </span><br><span class="line">conda config --add channels https://mirrors.ustc.edu.cn/anaconda/cloud/menpo/ </span><br><span class="line">conda config --add channels https://mirrors.ustc.edu.cn/anaconda/cloud/ </span><br><span class="line"> </span><br><span class="line"><span class="comment"># 北京外国语大学源 </span></span><br><span class="line">conda config --add channels https://mirrors.bfsu.edu.cn/anaconda/pkgs/main </span><br><span class="line">conda config --add channels https://mirrors.bfsu.edu.cn/anaconda/pkgs/free </span><br><span class="line">conda config --add channels https://mirrors.bfsu.edu.cn/anaconda/pkgs/r </span><br><span class="line">conda config --add channels https://mirrors.bfsu.edu.cn/anaconda/pkgs/pro </span><br><span class="line">conda config --add channels https://mirrors.bfsu.edu.cn/anaconda/pkgs/msys2</span><br><span class="line"></span><br><span class="line"><span class="comment"># 清华源 </span></span><br><span class="line">conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main</span><br><span class="line">conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free </span><br><span class="line">conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r </span><br><span class="line">conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/pro </span><br><span class="line">conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2 </span><br></pre></td></tr></table></figure><h2 id="Tensorflow环境安装"><a href="#Tensorflow环境安装" class="headerlink" title="Tensorflow环境安装"></a>Tensorflow环境安装</h2><p>查看Tensorflow版本和安装指定版本。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 查询tensorflow-gpu的版本</span></span><br><span class="line">conda search tensorflow-gpu</span><br><span class="line"><span class="comment"># 指定版本进行安装</span></span><br><span class="line">conda install tensorflow-gpu==1.13.1</span><br></pre></td></tr></table></figure><p><img src="/2022/05/15/Windows%E7%B3%BB%E7%BB%9F%E4%BD%BF%E7%94%A8Conda%E9%85%8D%E7%BD%AE%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E7%8E%AF%E5%A2%83/tf_install.jpg" alt="tf_install"></p><p>安装过程中会安装cudatoolkit-10.0.130和cudnn-7.6.5。</p><p><img src="/2022/05/15/Windows%E7%B3%BB%E7%BB%9F%E4%BD%BF%E7%94%A8Conda%E9%85%8D%E7%BD%AE%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E7%8E%AF%E5%A2%83/tf_install_info.jpg" alt="tf_install"></p><p>执行下述命令查看tf版本和GPU是否生效。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">查看是否安装成功</span><br><span class="line"><span class="keyword">import</span> tensorflow <span class="keyword">as</span> tf</span><br><span class="line"></span><br><span class="line"><span class="comment"># 打印Tensorflow版本</span></span><br><span class="line"><span class="built_in">print</span>(tf.__version__)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 打印是否支持GPU</span></span><br><span class="line"><span class="built_in">print</span>(tf.test.is_gpu_available())</span><br></pre></td></tr></table></figure><p>根据图片打印结果，成功安装tf 1.13.1 版本，同时GPU安装生效。</p><p><img src="/2022/05/15/Windows%E7%B3%BB%E7%BB%9F%E4%BD%BF%E7%94%A8Conda%E9%85%8D%E7%BD%AE%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E7%8E%AF%E5%A2%83/tf_version.jpg" alt="tf_version"></p><h2 id="Pytorch环境安装"><a href="#Pytorch环境安装" class="headerlink" title="Pytorch环境安装"></a>Pytorch环境安装</h2><p>执行下述命令安装Pytorch。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">conda install pytorch==1.7.0 torchvision==0.8.0 torchaudio==0.7.0 cudatoolkit=10.1 -c pytorch</span><br></pre></td></tr></table></figure><p>执行下述命令查看torch版本和GPU是否生效。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> torch</span><br><span class="line"></span><br><span class="line"><span class="built_in">print</span>(torch.__version__)</span><br><span class="line"></span><br><span class="line"><span class="built_in">print</span>(torch.cuda.is_available())</span><br><span class="line"></span><br></pre></td></tr></table></figure><p>根据图片打印结果，成功安装Pytorch 1.7.0 版本，同时GPU安装生效。</p><p><img src="/2022/05/15/Windows%E7%B3%BB%E7%BB%9F%E4%BD%BF%E7%94%A8Conda%E9%85%8D%E7%BD%AE%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E7%8E%AF%E5%A2%83/torch_version.jpg" alt="torch_version"></p><h2 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h2><ul><li><p>More info: <a class="link" href="https://blog.csdn.net/zhayushui/article/details/80433768?utm_medium=distribute.pc_relevant_t0.none-task-blog-BlogCommendFromMachineLearnPai2-1.nonecase&depth_1-utm_source=distribute.pc_relevant_t0.none-task-blog-BlogCommendFromMachineLearnPai2-1.nonecase">阿尔发go：conda常用命令<i class="fas fa-external-link-alt"></i></a></p></li><li><p>More info: <a class="link" href="https://www.jianshu.com/p/edaa744ea47d">卖萌哥：conda的安装与使用<i class="fas fa-external-link-alt"></i></a></p></li><li><p>More info: <a class="link" href="https://www.cnblogs.com/liuhuacai/p/11684666.html">无聊就看书：Tensorflow-gpu1.13.1 和 Tensorflow-gpu2.0.0共存之安装教程<i class="fas fa-external-link-alt"></i></a></p></li></ul>]]></content>
<summary type="html"><h2 id="Conda环境管理"><a href="#Conda环境管理" class="headerlink" title="Conda环境管理"></a>Conda环境管理</h2><p>Conda是一个开源软件包管理系统和环境管理系统，可以管理不同Python版本环境，</summary>
<category term="Conda" scheme="https://yqstar.github.io/tags/Conda/"/>
</entry>
<entry>
<title>snippets</title>
<link href="https://yqstar.github.io/2022/05/08/snippets/"/>
<id>https://yqstar.github.io/2022/05/08/snippets/</id>
<published>2022-05-08T14:45:18.000Z</published>
<updated>2022-06-12T13:40:32.253Z</updated>
<content type="html"><![CDATA[<p>本博客的目的主要是为了收集一些常用代码，或者一些有意思的代码，方便后续的工作学习使用。</p><h2 id="Sklearn的dataset转为dataframe"><a href="#Sklearn的dataset转为dataframe" class="headerlink" title="Sklearn的dataset转为dataframe"></a>Sklearn的dataset转为dataframe</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line"><span class="keyword">from</span> sklearn <span class="keyword">import</span> datasets</span><br><span class="line"></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">sklearn_to_df</span>(<span class="params">sklearn_dataset</span>):</span><br><span class="line"> df = pd.DataFrame(sklearn_dataset.data, columns=sklearn_dataset.feature_names)</span><br><span class="line"> df[<span class="string">'TARGET'</span>] = pd.Series(sklearn_dataset.target)</span><br><span class="line"> <span class="keyword">return</span> df</span><br><span class="line"></span><br><span class="line">df_boston = sklearn_to_df(datasets.load_boston())</span><br><span class="line"></span><br><span class="line"><span class="built_in">print</span>(df_boston.head())</span><br></pre></td></tr></table></figure><h2 id="pymysql读取mysql数据转化为dataframe"><a href="#pymysql读取mysql数据转化为dataframe" class="headerlink" title="pymysql读取mysql数据转化为dataframe"></a>pymysql读取mysql数据转化为dataframe</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> pymysql</span><br><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line"><span class="keyword">def</span> <span class="title function_">load_data_frame_from_mysql</span>():</span><br><span class="line"> conn = pymysql.connect(host=<span class="string">"127.0.0.1"</span>,</span><br><span class="line"> port=<span class="number">3307</span>,</span><br><span class="line"> user=<span class="string">"utest"</span>,</span><br><span class="line"> password=<span class="string">"123456xyq"</span>,</span><br><span class="line"> db=<span class="string">"sakila"</span>,</span><br><span class="line"> charset=<span class="string">"utf8"</span>)</span><br><span class="line"> sql = <span class="string">"SELECT * FROM sakila.payment"</span></span><br><span class="line"> data_frame = pd.read_sql(sql, conn)</span><br><span class="line"> <span class="keyword">return</span> data_frame</span><br><span class="line">pdata = load_data_frame_from_mysql()</span><br><span class="line"><span class="built_in">print</span>(pdata.head())</span><br></pre></td></tr></table></figure><h2 id="Image处理代码"><a href="#Image处理代码" class="headerlink" title="Image处理代码"></a>Image处理代码</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br><span class="line">185</span><br><span class="line">186</span><br><span class="line">187</span><br><span class="line">188</span><br><span class="line">189</span><br><span class="line">190</span><br><span class="line">191</span><br><span class="line">192</span><br><span class="line">193</span><br><span class="line">194</span><br><span class="line">195</span><br><span class="line">196</span><br><span class="line">197</span><br><span class="line">198</span><br><span class="line">199</span><br><span class="line">200</span><br><span class="line">201</span><br><span class="line">202</span><br><span class="line">203</span><br><span class="line">204</span><br><span class="line">205</span><br><span class="line">206</span><br><span class="line">207</span><br><span class="line">208</span><br><span class="line">209</span><br><span class="line">210</span><br><span class="line">211</span><br><span class="line">212</span><br><span class="line">213</span><br><span class="line">214</span><br><span class="line">215</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># -*-coding: utf-8 -*-</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string"> @Project: IntelligentManufacture</span></span><br><span class="line"><span class="string"> @File : image_processing.py</span></span><br><span class="line"><span class="string"> @Author : panjq</span></span><br><span class="line"><span class="string"> @E-mail : pan_jinquan@163.com</span></span><br><span class="line"><span class="string"> @Date : 2019-02-14 15:34:50</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"> </span><br><span class="line"><span class="keyword">import</span> os</span><br><span class="line"><span class="keyword">import</span> glob</span><br><span class="line"><span class="keyword">import</span> cv2</span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"><span class="keyword">import</span> matplotlib.pyplot <span class="keyword">as</span> plt</span><br><span class="line"> </span><br><span class="line"><span class="keyword">def</span> <span class="title function_">show_image</span>(<span class="params">title, image</span>):</span><br><span class="line"> <span class="string">'''</span></span><br><span class="line"><span class="string"> 调用matplotlib显示RGB图片</span></span><br><span class="line"><span class="string"> :param title: 图像标题</span></span><br><span class="line"><span class="string"> :param image: 图像的数据</span></span><br><span class="line"><span class="string"> :return:</span></span><br><span class="line"><span class="string"> '''</span></span><br><span class="line"> <span class="comment"># plt.figure("show_image")</span></span><br><span class="line"> <span class="comment"># print(image.dtype)</span></span><br><span class="line"> plt.imshow(image)</span><br><span class="line"> plt.axis(<span class="string">'on'</span>) <span class="comment"># 关掉坐标轴为 off</span></span><br><span class="line"> plt.title(title) <span class="comment"># 图像题目</span></span><br><span class="line"> plt.show()</span><br><span class="line"> </span><br><span class="line"><span class="keyword">def</span> <span class="title function_">cv_show_image</span>(<span class="params">title, image</span>):</span><br><span class="line"> <span class="string">'''</span></span><br><span class="line"><span class="string"> 调用OpenCV显示RGB图片</span></span><br><span class="line"><span class="string"> :param title: 图像标题</span></span><br><span class="line"><span class="string"> :param image: 输入RGB图像</span></span><br><span class="line"><span class="string"> :return:</span></span><br><span class="line"><span class="string"> '''</span></span><br><span class="line"> channels=image.shape[-<span class="number">1</span>]</span><br><span class="line"> <span class="keyword">if</span> channels==<span class="number">3</span>:</span><br><span class="line"> image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR) <span class="comment"># 将BGR转为RGB</span></span><br><span class="line"> cv2.imshow(title,image)</span><br><span class="line"> cv2.waitKey(<span class="number">0</span>)</span><br><span class="line"> </span><br><span class="line"><span class="keyword">def</span> <span class="title function_">read_image</span>(<span class="params">filename, resize_height=<span class="literal">None</span>, resize_width=<span class="literal">None</span>, normalization=<span class="literal">False</span></span>):</span><br><span class="line"> <span class="string">'''</span></span><br><span class="line"><span class="string"> 读取图片数据,默认返回的是uint8,[0,255]</span></span><br><span class="line"><span class="string"> :param filename:</span></span><br><span class="line"><span class="string"> :param resize_height:</span></span><br><span class="line"><span class="string"> :param resize_width:</span></span><br><span class="line"><span class="string"> :param normalization:是否归一化到[0.,1.0]</span></span><br><span class="line"><span class="string"> :return: 返回的RGB图片数据</span></span><br><span class="line"><span class="string"> '''</span></span><br><span class="line"> </span><br><span class="line"> bgr_image = cv2.imread(filename)</span><br><span class="line"> <span class="comment"># bgr_image = cv2.imread(filename,cv2.IMREAD_IGNORE_ORIENTATION|cv2.IMREAD_COLOR)</span></span><br><span class="line"> <span class="keyword">if</span> bgr_image <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">"Warning:不存在:{}"</span>, filename)</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">None</span></span><br><span class="line"> <span class="keyword">if</span> <span class="built_in">len</span>(bgr_image.shape) == <span class="number">2</span>: <span class="comment"># 若是灰度图则转为三通道</span></span><br><span class="line"> <span class="built_in">print</span>(<span class="string">"Warning:gray image"</span>, filename)</span><br><span class="line"> bgr_image = cv2.cvtColor(bgr_image, cv2.COLOR_GRAY2BGR)</span><br><span class="line"> </span><br><span class="line"> rgb_image = cv2.cvtColor(bgr_image, cv2.COLOR_BGR2RGB) <span class="comment"># 将BGR转为RGB</span></span><br><span class="line"> <span class="comment"># show_image(filename,rgb_image)</span></span><br><span class="line"> <span class="comment"># rgb_image=Image.open(filename)</span></span><br><span class="line"> rgb_image = resize_image(rgb_image,resize_height,resize_width)</span><br><span class="line"> rgb_image = np.asanyarray(rgb_image)</span><br><span class="line"> <span class="keyword">if</span> normalization:</span><br><span class="line"> <span class="comment"># 不能写成:rgb_image=rgb_image/255</span></span><br><span class="line"> rgb_image = rgb_image / <span class="number">255.0</span></span><br><span class="line"> <span class="comment"># show_image("src resize image",image)</span></span><br><span class="line"> <span class="keyword">return</span> rgb_image</span><br><span class="line"> </span><br><span class="line"><span class="keyword">def</span> <span class="title function_">fast_read_image_roi</span>(<span class="params">filename, orig_rect, ImreadModes=cv2.IMREAD_COLOR, normalization=<span class="literal">False</span></span>):</span><br><span class="line"> <span class="string">'''</span></span><br><span class="line"><span class="string"> 快速读取图片的方法</span></span><br><span class="line"><span class="string"> :param filename: 图片路径</span></span><br><span class="line"><span class="string"> :param orig_rect:原始图片的感兴趣区域rect</span></span><br><span class="line"><span class="string"> :param ImreadModes: IMREAD_UNCHANGED</span></span><br><span class="line"><span class="string"> IMREAD_GRAYSCALE</span></span><br><span class="line"><span class="string"> IMREAD_COLOR</span></span><br><span class="line"><span class="string"> IMREAD_ANYDEPTH</span></span><br><span class="line"><span class="string"> IMREAD_ANYCOLOR</span></span><br><span class="line"><span class="string"> IMREAD_LOAD_GDAL</span></span><br><span class="line"><span class="string"> IMREAD_REDUCED_GRAYSCALE_2</span></span><br><span class="line"><span class="string"> IMREAD_REDUCED_COLOR_2</span></span><br><span class="line"><span class="string"> IMREAD_REDUCED_GRAYSCALE_4</span></span><br><span class="line"><span class="string"> IMREAD_REDUCED_COLOR_4</span></span><br><span class="line"><span class="string"> IMREAD_REDUCED_GRAYSCALE_8</span></span><br><span class="line"><span class="string"> IMREAD_REDUCED_COLOR_8</span></span><br><span class="line"><span class="string"> IMREAD_IGNORE_ORIENTATION</span></span><br><span class="line"><span class="string"> :param normalization: 是否归一化</span></span><br><span class="line"><span class="string"> :return: 返回感兴趣区域ROI</span></span><br><span class="line"><span class="string"> '''</span></span><br><span class="line"> <span class="comment"># 当采用IMREAD_REDUCED模式时，对应rect也需要缩放</span></span><br><span class="line"> scale=<span class="number">1</span></span><br><span class="line"> <span class="keyword">if</span> ImreadModes == cv2.IMREAD_REDUCED_COLOR_2 <span class="keyword">or</span> ImreadModes == cv2.IMREAD_REDUCED_COLOR_2:</span><br><span class="line"> scale=<span class="number">1</span>/<span class="number">2</span></span><br><span class="line"> <span class="keyword">elif</span> ImreadModes == cv2.IMREAD_REDUCED_GRAYSCALE_4 <span class="keyword">or</span> ImreadModes == cv2.IMREAD_REDUCED_COLOR_4:</span><br><span class="line"> scale=<span class="number">1</span>/<span class="number">4</span></span><br><span class="line"> <span class="keyword">elif</span> ImreadModes == cv2.IMREAD_REDUCED_GRAYSCALE_8 <span class="keyword">or</span> ImreadModes == cv2.IMREAD_REDUCED_COLOR_8:</span><br><span class="line"> scale=<span class="number">1</span>/<span class="number">8</span></span><br><span class="line"> rect = np.array(orig_rect)*scale</span><br><span class="line"> rect = rect.astype(<span class="built_in">int</span>).tolist()</span><br><span class="line"> bgr_image = cv2.imread(filename,flags=ImreadModes)</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">if</span> bgr_image <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">"Warning:不存在:{}"</span>, filename)</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">None</span></span><br><span class="line"> <span class="keyword">if</span> <span class="built_in">len</span>(bgr_image.shape) == <span class="number">3</span>: <span class="comment">#</span></span><br><span class="line"> rgb_image = cv2.cvtColor(bgr_image, cv2.COLOR_BGR2RGB) <span class="comment"># 将BGR转为RGB</span></span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> rgb_image=bgr_image <span class="comment">#若是灰度图</span></span><br><span class="line"> rgb_image = np.asanyarray(rgb_image)</span><br><span class="line"> <span class="keyword">if</span> normalization:</span><br><span class="line"> <span class="comment"># 不能写成:rgb_image=rgb_image/255</span></span><br><span class="line"> rgb_image = rgb_image / <span class="number">255.0</span></span><br><span class="line"> roi_image=get_rect_image(rgb_image , rect)</span><br><span class="line"> <span class="comment"># show_image_rect("src resize image",rgb_image,rect)</span></span><br><span class="line"> <span class="comment"># cv_show_image("reROI",roi_image)</span></span><br><span class="line"> <span class="keyword">return</span> roi_image</span><br><span class="line"> </span><br><span class="line"><span class="keyword">def</span> <span class="title function_">resize_image</span>(<span class="params">image,resize_height, resize_width</span>):</span><br><span class="line"> <span class="string">'''</span></span><br><span class="line"><span class="string"> :param image:</span></span><br><span class="line"><span class="string"> :param resize_height:</span></span><br><span class="line"><span class="string"> :param resize_width:</span></span><br><span class="line"><span class="string"> :return:</span></span><br><span class="line"><span class="string"> '''</span></span><br><span class="line"> image_shape=np.shape(image)</span><br><span class="line"> height=image_shape[<span class="number">0</span>]</span><br><span class="line"> width=image_shape[<span class="number">1</span>]</span><br><span class="line"> <span class="keyword">if</span> (resize_height <span class="keyword">is</span> <span class="literal">None</span>) <span class="keyword">and</span> (resize_width <span class="keyword">is</span> <span class="literal">None</span>):<span class="comment">#错误写法：resize_height and resize_width is None</span></span><br><span class="line"> <span class="keyword">return</span> image</span><br><span class="line"> <span class="keyword">if</span> resize_height <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> resize_height=<span class="built_in">int</span>(height*resize_width/width)</span><br><span class="line"> <span class="keyword">elif</span> resize_width <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> resize_width=<span class="built_in">int</span>(width*resize_height/height)</span><br><span class="line"> image = cv2.resize(image, dsize=(resize_width, resize_height))</span><br><span class="line"> <span class="keyword">return</span> image</span><br><span class="line"><span class="keyword">def</span> <span class="title function_">scale_image</span>(<span class="params">image,scale</span>):</span><br><span class="line"> <span class="string">'''</span></span><br><span class="line"><span class="string"> :param image:</span></span><br><span class="line"><span class="string"> :param scale: (scale_w,scale_h)</span></span><br><span class="line"><span class="string"> :return:</span></span><br><span class="line"><span class="string"> '''</span></span><br><span class="line"> image = cv2.resize(image,dsize=<span class="literal">None</span>, fx=scale[<span class="number">0</span>],fy=scale[<span class="number">1</span>])</span><br><span class="line"> <span class="keyword">return</span> image</span><br><span class="line"> </span><br><span class="line"> </span><br><span class="line"><span class="keyword">def</span> <span class="title function_">get_rect_image</span>(<span class="params">image,rect</span>):</span><br><span class="line"> <span class="string">'''</span></span><br><span class="line"><span class="string"> :param image:</span></span><br><span class="line"><span class="string"> :param rect: [x,y,w,h]</span></span><br><span class="line"><span class="string"> :return:</span></span><br><span class="line"><span class="string"> '''</span></span><br><span class="line"> x, y, w, h=rect</span><br><span class="line"> cut_img = image[y:(y+ h),x:(x+w)]</span><br><span class="line"> <span class="keyword">return</span> cut_img</span><br><span class="line"><span class="keyword">def</span> <span class="title function_">scale_rect</span>(<span class="params">orig_rect,orig_shape,dest_shape</span>):</span><br><span class="line"> <span class="string">'''</span></span><br><span class="line"><span class="string"> 对图像进行缩放时，对应的rectangle也要进行缩放</span></span><br><span class="line"><span class="string"> :param orig_rect: 原始图像的rect=[x,y,w,h]</span></span><br><span class="line"><span class="string"> :param orig_shape: 原始图像的维度shape=[h,w]</span></span><br><span class="line"><span class="string"> :param dest_shape: 缩放后图像的维度shape=[h,w]</span></span><br><span class="line"><span class="string"> :return: 经过缩放后的rectangle</span></span><br><span class="line"><span class="string"> '''</span></span><br><span class="line"> new_x=<span class="built_in">int</span>(orig_rect[<span class="number">0</span>]*dest_shape[<span class="number">1</span>]/orig_shape[<span class="number">1</span>])</span><br><span class="line"> new_y=<span class="built_in">int</span>(orig_rect[<span class="number">1</span>]*dest_shape[<span class="number">0</span>]/orig_shape[<span class="number">0</span>])</span><br><span class="line"> new_w=<span class="built_in">int</span>(orig_rect[<span class="number">2</span>]*dest_shape[<span class="number">1</span>]/orig_shape[<span class="number">1</span>])</span><br><span class="line"> new_h=<span class="built_in">int</span>(orig_rect[<span class="number">3</span>]*dest_shape[<span class="number">0</span>]/orig_shape[<span class="number">0</span>])</span><br><span class="line"> dest_rect=[new_x,new_y,new_w,new_h]</span><br><span class="line"> <span class="keyword">return</span> dest_rect</span><br><span class="line"> </span><br><span class="line"><span class="keyword">def</span> <span class="title function_">show_image_rect</span>(<span class="params">win_name,image,rect</span>):</span><br><span class="line"> <span class="string">'''</span></span><br><span class="line"><span class="string"> :param win_name:</span></span><br><span class="line"><span class="string"> :param image:</span></span><br><span class="line"><span class="string"> :param rect:</span></span><br><span class="line"><span class="string"> :return:</span></span><br><span class="line"><span class="string"> '''</span></span><br><span class="line"> x, y, w, h=rect</span><br><span class="line"> point1=(x,y)</span><br><span class="line"> point2=(x+w,y+h)</span><br><span class="line"> cv2.rectangle(image, point1, point2, (<span class="number">0</span>, <span class="number">0</span>, <span class="number">255</span>), thickness=<span class="number">2</span>)</span><br><span class="line"> cv_show_image(win_name, image)</span><br><span class="line"> </span><br><span class="line"><span class="keyword">def</span> <span class="title function_">rgb_to_gray</span>(<span class="params">image</span>):</span><br><span class="line"> image = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)</span><br><span class="line"> <span class="keyword">return</span> image</span><br><span class="line"> </span><br><span class="line"><span class="keyword">def</span> <span class="title function_">save_image</span>(<span class="params">image_path, rgb_image,toUINT8=<span class="literal">True</span></span>):</span><br><span class="line"> <span class="keyword">if</span> toUINT8:</span><br><span class="line"> rgb_image = np.asanyarray(rgb_image * <span class="number">255</span>, dtype=np.uint8)</span><br><span class="line"> <span class="keyword">if</span> <span class="built_in">len</span>(rgb_image.shape) == <span class="number">2</span>: <span class="comment"># 若是灰度图则转为三通道</span></span><br><span class="line"> bgr_image = cv2.cvtColor(rgb_image, cv2.COLOR_GRAY2BGR)</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> bgr_image = cv2.cvtColor(rgb_image, cv2.COLOR_RGB2BGR)</span><br><span class="line"> cv2.imwrite(image_path, bgr_image)</span><br><span class="line"> </span><br><span class="line"><span class="keyword">def</span> <span class="title function_">combime_save_image</span>(<span class="params">orig_image, dest_image, out_dir,name,prefix</span>):</span><br><span class="line"> <span class="string">'''</span></span><br><span class="line"><span class="string"> 命名标准：out_dir/name_prefix.jpg</span></span><br><span class="line"><span class="string"> :param orig_image:</span></span><br><span class="line"><span class="string"> :param dest_image:</span></span><br><span class="line"><span class="string"> :param image_path:</span></span><br><span class="line"><span class="string"> :param out_dir:</span></span><br><span class="line"><span class="string"> :param prefix:</span></span><br><span class="line"><span class="string"> :return:</span></span><br><span class="line"><span class="string"> '''</span></span><br><span class="line"> dest_path = os.path.join(out_dir, name + <span class="string">"_"</span>+prefix+<span class="string">".jpg"</span>)</span><br><span class="line"> save_image(dest_path, dest_image)</span><br><span class="line"> </span><br><span class="line"> dest_image = np.hstack((orig_image, dest_image))</span><br><span class="line"> save_image(os.path.join(out_dir, <span class="string">"{}_src_{}.jpg"</span>.<span class="built_in">format</span>(name,prefix)), dest_image)</span><br><span class="line"></span><br></pre></td></tr></table></figure><h2 id="apt-get卸载软件"><a href="#apt-get卸载软件" class="headerlink" title="apt-get卸载软件"></a>apt-get卸载软件</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 删除软件及其配置文件</span></span><br><span class="line">apt-get --purge remove openjdk-11-jdk</span><br><span class="line"><span class="comment"># 删除没用的依赖包</span></span><br><span class="line">sudo apt-get autoremove openjdk-11-jdk</span><br><span class="line"><span class="comment"># 此时dpkg的列表中有“rc”状态的软件包，可以执行如下命令做最后清理：</span></span><br><span class="line">dpkg -l |grep ^rc|awk <span class="string">'{print $2}'</span> |sudo xargs dpkg -P</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html"><p>本博客的目的主要是为了收集一些常用代码，或者一些有意思的代码，方便后续的工作学习使用。</p>
<h2 id="Sklearn的dataset转为dataframe"><a href="#Sklearn的dataset转为dataframe" class="headerli</summary>
<category term="Snippet" scheme="https://yqstar.github.io/categories/Snippet/"/>
<category term="Snippet" scheme="https://yqstar.github.io/tags/Snippet/"/>
<category term="Python" scheme="https://yqstar.github.io/tags/Python/"/>
</entry>
<entry>
<title>使用GithubPages搭建个人静态博客</title>
<link href="https://yqstar.github.io/2022/05/01/%E4%BD%BF%E7%94%A8GithubPages%E6%90%AD%E5%BB%BA%E4%B8%AA%E4%BA%BA%E9%9D%99%E6%80%81%E5%8D%9A%E5%AE%A2/"/>
<id>https://yqstar.github.io/2022/05/01/%E4%BD%BF%E7%94%A8GithubPages%E6%90%AD%E5%BB%BA%E4%B8%AA%E4%BA%BA%E9%9D%99%E6%80%81%E5%8D%9A%E5%AE%A2/</id>
<published>2022-05-01T02:14:29.000Z</published>
<updated>2022-06-12T13:40:35.973Z</updated>
<content type="html"><![CDATA[<h2 id="原理"><a href="#原理" class="headerlink" title="原理"></a>原理</h2><p><strong>GitHub Pages</strong> 是使用 GitHub 存储仓库托管静态网站，使用 YAML 和 Markdown 等标准技术，任何人都可以在几分钟内生成和维护网站，但它不仅仅是静态文件的集合。通过利用 Jekyll 和 Liquid 等网站生成技术，开发人员可定义完整静态网站的动态模板。每次将更改提交到与网站关联的源代码分支时，都会使用源代码分支的最新代码配置重新生成静态网页，Github 自动将其发布到目标 URL。欢迎关注：<a href="https://yqstar.github.io/">我的博客</a>。</p><h2 id="环境配置"><a href="#环境配置" class="headerlink" title="环境配置"></a>环境配置</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">操作系统：Windows11 -> WSL2 -> Ubuntu 22.04 LTS</span><br><span class="line">Node.js: v16.15.0</span><br><span class="line">Npm: 8.5.5</span><br></pre></td></tr></table></figure><h2 id="操作部署"><a href="#操作部署" class="headerlink" title="操作部署"></a>操作部署</h2><h3 id="安装NVM"><a href="#安装NVM" class="headerlink" title="安装NVM"></a>安装NVM</h3><p>在安装nvm之前简单介绍下nvm，nodejs和npm之间的关系。</p><ul><li>nvm：<em>nodejs</em> 的版本管理工具。</li><li>nodejs：项目开发所需要的代码库。</li><li>npm：<em>nodejs</em> 包管理工具，<em>npm</em> 管理<em>nodejs</em>中的第三方插件。</li></ul><p>参考 <a class="link" href="https://github.com/nvm-sh/nvm">NPM-GITHUB<i class="fas fa-external-link-alt"></i></a>，使用以下命令安装nvm。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">wget -qO- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash</span><br></pre></td></tr></table></figure><p>安装过程Info截图如下：</p><p><img src="/2022/05/01/%E4%BD%BF%E7%94%A8GithubPages%E6%90%AD%E5%BB%BA%E4%B8%AA%E4%BA%BA%E9%9D%99%E6%80%81%E5%8D%9A%E5%AE%A2/nvm_install.png" alt="NVM安装截图"></p><p>判断nvm安装是否成功,可以使用以下命令查看nvm的版本信息。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">nvm -v</span><br></pre></td></tr></table></figure><p><img src="/2022/05/01/%E4%BD%BF%E7%94%A8GithubPages%E6%90%AD%E5%BB%BA%E4%B8%AA%E4%BA%BA%E9%9D%99%E6%80%81%E5%8D%9A%E5%AE%A2/nvm_version.png" alt="NVM版本截图"></p><h3 id="安装NODEJS"><a href="#安装NODEJS" class="headerlink" title="安装NODEJS"></a>安装NODEJS</h3><p>使用 <em>nvm install</em> 命令可以安装指定版本的NodeJs，本次安装v16版本，执行以下命令。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">nvm install 16</span><br></pre></td></tr></table></figure><p>通过下述命令查看node和npm的版本。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">node -v</span><br><span class="line">npm -v</span><br></pre></td></tr></table></figure><p>可查看到Node和npm的版本。</p><p><img src="/2022/05/01/%E4%BD%BF%E7%94%A8GithubPages%E6%90%AD%E5%BB%BA%E4%B8%AA%E4%BA%BA%E9%9D%99%E6%80%81%E5%8D%9A%E5%AE%A2/npm_node_version.png" alt="NVM版本截图"></p><h2 id="安装Hexo和配置"><a href="#安装Hexo和配置" class="headerlink" title="安装Hexo和配置"></a>安装Hexo和配置</h2><h3 id="安装Hexo命令行"><a href="#安装Hexo命令行" class="headerlink" title="安装Hexo命令行"></a>安装Hexo命令行</h3><p>Hexo是一个快速、简洁且高效的博客框架，官方提供一个命令行工具，用于快速创建项目、页面、编译、部署Hexo博客，安装命令行工具。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">npm install -g hexo-cli</span><br></pre></td></tr></table></figure><h3 id="初始化本地运行"><a href="#初始化本地运行" class="headerlink" title="初始化本地运行"></a>初始化本地运行</h3><p>接下来我们使用 Hexo 命令行创建一个项目，并将其在本地跑起来，整体跑通看看。</p><p>首先使用如下命令创建项目：<strong>hexo init {blog_name}</strong>，这里的 blog_name 就是博客名，我这里要创建 <em>hexo_blog</em> 的博客，我就把项目取名为 hexo_blog ，命令如下：hexo init hexo_blog。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">hexo init {blog_name}</span><br><span class="line"><span class="built_in">cd</span> {blog_name}</span><br><span class="line">npm install</span><br><span class="line">hexo server</span><br></pre></td></tr></table></figure><p>hexo_blog 文件夹下就会出现 Hexo 的初始化文件，包括 themes、scaffolds、source 等文件夹。</p><p><img src="/2022/05/01/%E4%BD%BF%E7%94%A8GithubPages%E6%90%AD%E5%BB%BA%E4%B8%AA%E4%BA%BA%E9%9D%99%E6%80%81%E5%8D%9A%E5%AE%A2/hexo_init.png" alt="Hexo初始化"></p><p>接下来我们首先进入新生成的文件夹里面，然后调用 Hexo 的 generate 命令，将 Hexo 编译生成 HTML 代码，命令如下：hexo generate可以看到输出结果里面包含了 js、css、font 等内容，并发现他们都处在了项目根目录下的 public 文件夹下面了。</p><p>然后使用 Hexo 提供的 server 命令把博客在本地运行起来，命令如下：hexo server 运行之后命令行输出如下：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">INFO Start processing</span><br><span class="line">INFO Hexois running at http://localhost:4000 . Press Ctrl+C to stop.</span><br></pre></td></tr></table></figure><p>本地 4000 端口上就可以查看博客站点，如图所示：</p><p><img src="/2022/05/01/%E4%BD%BF%E7%94%A8GithubPages%E6%90%AD%E5%BB%BA%E4%B8%AA%E4%BA%BA%E9%9D%99%E6%80%81%E5%8D%9A%E5%AE%A2/hexo_server.png" alt="Hexo初始化"></p><h3 id="主题配置"><a href="#主题配置" class="headerlink" title="主题配置"></a>主题配置</h3><p>登录官网<a class="link" href="https://hexo.io/zh-cn/">Hexo官网<i class="fas fa-external-link-alt"></i></a>,然后点击主题页面<a class="link" href="https://hexo.io/themes/">Theme主题<i class="fas fa-external-link-alt"></i></a>，选择一个喜欢主题，我这边选择的主题是<br><a class="link" href="https://keep-docs.xpoet.cn/usage-tutorial/configuration-guide.html#base-info">keep主题<i class="fas fa-external-link-alt"></i></a>。<br>执行安装命令，会将主题文件安装在node_modules文件夹中。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">npm install hexo-theme-keep --save</span><br></pre></td></tr></table></figure><p><img src="/2022/05/01/%E4%BD%BF%E7%94%A8GithubPages%E6%90%AD%E5%BB%BA%E4%B8%AA%E4%BA%BA%E9%9D%99%E6%80%81%E5%8D%9A%E5%AE%A2/keep_theme_path.png" alt="主题存放路径"></p><p>执行hexo server命令执行本地查看。</p><p>可以看出此时数据已经更新成keep主题。</p><h2 id="部署到Github"><a href="#部署到Github" class="headerlink" title="部署到Github"></a>部署到Github</h2><p>执行部署到Github需要使用hexo-deployer-git插件</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">npm install hexo-deployer-git --save</span><br></pre></td></tr></table></figure><p>安装好hexo-deployer-git后可以进行文件的配置。<br>具体配置如下：</p><p><img src="/2022/05/01/%E4%BD%BF%E7%94%A8GithubPages%E6%90%AD%E5%BB%BA%E4%B8%AA%E4%BA%BA%E9%9D%99%E6%80%81%E5%8D%9A%E5%AE%A2/deploy_settings.png" alt="主题存放路径"></p><p>在这之前需要github配置相应的仓库，我配置的仓库名为：yqstar.github.io.</p><p><img src="/2022/05/01/%E4%BD%BF%E7%94%A8GithubPages%E6%90%AD%E5%BB%BA%E4%B8%AA%E4%BA%BA%E9%9D%99%E6%80%81%E5%8D%9A%E5%AE%A2/deploy_online.png" alt="线上部署"></p><p><img src="/2022/05/01/%E4%BD%BF%E7%94%A8GithubPages%E6%90%AD%E5%BB%BA%E4%B8%AA%E4%BA%BA%E9%9D%99%E6%80%81%E5%8D%9A%E5%AE%A2/deploy_github.png" alt="线上部署"></p><h2 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h2><p>MoreInfo:<a class="link" href="https://github.com/xcodebuild/hexo-asset-image">hexo-asset-image插件<i class="fas fa-external-link-alt"></i></a><br>MoreInfo:<a class="link" href="https://blog.csdn.net/fitnig/article/details/106522811">如何使用本地插入图片<i class="fas fa-external-link-alt"></i></a></p>]]></content>
<summary type="html"><h2 id="原理"><a href="#原理" class="headerlink" title="原理"></a>原理</h2><p><strong>GitHub Pages</strong> 是使用 GitHub 存储仓库托管静态网站，使用 YAML 和 Markdown</summary>
</entry>
</feed>