文章目录
写在前面:
commands
在terminal中为直接输入运行
- 如在python程序中运行,语句为:
os.system('command')
,示例如下(创建文件):
import os
import tempfile
def tmp_path(ext=''):
tf = tempfile.NamedTemporaryFile()
return tf.name + ext
file_tt = tmp_path('.abc') #在系统tmp文件夹下创建名字随机,文件类型为.abc的文件
file_tt = 'tt.abc' #在当前文件夹下创建文件tt.abc
os.system('touch ' + file_tt)
- If in MATLAB programming, then please use
system('command')
, e.g., copy files to another folder:
file_dir = '/tmp1/xatx_1234/';
filelist_wav = dir(fullfile(file_dir,'*.wav'));
for fileid = 1:length(filelist_wav)
filename_wav = [file_dir, filelist_wav(fileid).name]; % full name of file
% disp(filename_wav);
system_command = ['cp ', filename_wav, ' ', '/tmp2/xatx_test/', filelist_wav(fileid).name];
% disp(system_command);
system(system_command);
end
参考:
文档:链接: https://pan.baidu.com/s/1QwUGQpRUs57PvcP5EGpVlQ ,提取码: 6whh
查看CPU / GPU / 内存 / 读写
htop
watch cpu occupatation
Note that you need to install htop before using, i.e., $ sudo apt-get install htop
htop
or use: top
top
nvidia-smi
watch GPU occupatation
nvidia-smi
or watch GPU per 10 seconds (-n 10)
watch -n 10 nvidia-smi
watch free
watch memory of the ubuntu, e.g.,
watch free -h
where -h means human.
we can also watch memory/CPU by system-monitor:
Step 1: click search
Step 2: input system-monitor, select the system-monitor.
iostat
watch the io read and write information (per 5 seconds: -d 5)
iostat -d 5
echo
watch $PATH, e.g.,
echo $PATH
Download
git clone
download sources from github, e.g.,
git clone https://github.com/mozilla/DeepSpeech
other git commands
please refer: https://blog.csdn.net/u010637291/article/details/107660891
wget
download
e.g., install netron tool (http://www.electronjs.org/apps/netron) on Ubuntu system:
按照要求从https://github.com/lutzroeder/netron/releases/tag/v4.5.1下载.AppImage文件:
wget https://github.com/lutzroeder/netron/releases/tag/v4.5.1 -o Netron-4.5.1.AppImage
curl
wget -c …
curl …
文件创建 / 修改 / 压缩 / 解压 / 移动 / 重命名 / 复制 / 删除
touch
create a file (e.g., tmp.txt)
touch tmp.txt
vim
change the content of a file:
vim tmp.txt
Note that vim should be installed.
Some commands for vim:
i #insert
Est #return to command state
:q # quit
:wq # save and quit
sed
ref: shell sed 命令详解:选取、替换、删除、新増数据
: https://ld246.com/article/1587370777231
modify txt lines in a file, e.g., add “sudo rm” in front of each lines in a.txt
train_file=a.txt
sed -i 's/^/sudo rm /g' $train_file
or add additional content in the end of lines, e.g., add a blank and a number “6” at the end of each line
sed -i 's/$/ 6/g' $train_file
tar -xvf tar_name
uncompress, e.g.,
tar -xvf XX.tar.gz
or compress
...
unzip
unzip a zip file:
unzip xx.zip
mv
move a file (e.g., myfile.XX) from current folder to another folder (e.g., /another_folder/), e.g.,
mv myfile.XX /another_folder/
or rename a file, e.g.,
mv a.jpg tmp.jpg.bak
rename
批量修改文件名字,e.g., 去除名字中空格
find $path -name '* *' -exec rename 's/ //g' {
} \;
ref:https://blog.csdn.net/u013946444/article/details/97244011
https://forum.ubuntu.org.cn/viewtopic.php?p=3074564
cp
copy a file to another place, e.g.,
cp /tmp/afolder/a.wav /tmp/anotherfolder/b.wav
cp -r
copy a directory (could contains files) to other folder, e.g.,
cp -r /tmp/afolder /tmp/desfolder
rm
remove/delete a file (permanately, not to trash)
!!! Important !!! rm 是直接删除,无法删除到trash,请谨慎使用。
如想删除到trash,请参考:https://askubuntu.com/questions/213533/command-to-move-a-file-to-trash-via-terminal
rm tmp.txt
some parameters:
-f:强制删除,忽略不存在的文件,从不给出提示。
-i:交互模式删除文件,删除文件前给出提示。
-r:递归的删除目录下面文件以及子目录下文件。
-R:递归的删除目录下面文件以及子目录下文件。
-v:显示运行时详细信息
rm -r
remove a directory (permanately, not to trash), e.g., remove ‘tmp’ folder (contains files)
rm -r tmp
rm -rf
常见用法:删除某一文件夹内所有子文件夹和文件:
rm -rf myfolder/
ls -l
查看文件权限,如
ls -l tmp.txt
chmod
更改文件权限,参见:https://blog.csdn.net/u010429424/article/details/48498081
格式:chmod [ u / g / o / a ] [ + / - / = ] [ r / w / x ] file
其中 :
u表示User,是文件的所有者
g表示跟User同Group的用户
o表示Other,即其他用户
a表示ALL,所有用户
+表示增加权限
-表示取消权限
=表示取消之前的权限,并给予唯一的权限
r表示Read,即读文件
w表示Write,即写文件
x表示运行文件
file表示文件的路径,如ubandy-rest/job/views.py
如,给tmp.AppImage文件给所有用户添加运行权限:
chmod a+x tmp.AppImage
文件夹 进入 / 查看 / 创建 / 删除 / 查找
cd
cd tmp/tmp2
cd .\. # 上一级
cd # 直接进入主目录
ls
watch the existing files and folders in current folder
ls
ls /usr/bin/python*
watch installed python
ls /usr/bin/python*
ls /usr/local/bin/python*
ls -a
如需显示所有文件(包括隐藏文件):
-a
ls -a
ls -ld
ls还可查看文件/文件夹权限:
ls -l xxx.txt # 查看文件权限
ls -ld # 查看当前文件夹权限
ls -lf a_folder # 查看a_folder文件夹权限
mkdir
create a new directory under current dir, e.g.,
mkdir new_folder
rmdir
if you want to remove a directory: using rmdir, e.g., remove tmp directory
rmdir tmp
find
find files, of a certain type (e.g., wav), and create file lists, e.g.,
find XXX_pathfolder -name "*.wav" > filelist.txt
if add content to a filelist file, change > to >>
find XXX_pathfolder -name "*.wav" >> filelist.txt
if list all files and folders under a folder:
ls > filelist.txt
find certain files, and move these files to another folder, e.g.,
find /tmp/folder -name *.dat -exec mv {
} /tmp/another_folder \\;
exec
execute a command, such as move files to another folder, after these files are found
find /tmp/folder -name *.dat -exec mv {
} /tmp/another_folder \\;
Note that there is ‘\;’ at the end
exec 是一个非常有用的工具,和find命令可实现批处理:
path=/tmp/20200819
find $path -name '* *' -exec rename 's/ //g' {
} \; #remove blank
find $path -name "*.aac" -exec faad {
} -o {
}.wav \; # faad: acc 转wav
find $path -name "*.aac.wav" -exec sox {
} -r 16k {
}_16k.wav \; #sox 该采样率
find $path -name "*.aac.wav_16k.wav" -exec sox {
} {
}_right.wav remix 2 \; #sox提取第二通道
find $path -name "*.aac.wav_16k.wav_right.wav" > filelist_2.txt # 声明文件列表
find $path -name "*.aac.wav_16k.wav_right.wav" -exec cp {
} ./testdata/2{
} \; # 拷贝重命名
find $path -name "*.aac.wav_16b_16k.wav_right.wav" -exec rm {
} \; #批量删除
安装 / 更新 包
sudo apt-get install/remove
install or uninstall a package for the Ubuntu system, e.g.,
sudo apt-get install vim
sudo apt-get remove vim
对于常用的pip,也可直接通过这种方式进行安装(ref: https://m.linuxidc.com/Linux/2018-05/152390.htm):
# python2.x
sudo apt-get install python-pip
# python3.x
sudo apt-get install python3-pip
但更见的安装方式为通过get-pip.py安装(ref:https://blog.csdn.net/dqy910606/article/details/89791549):
# 通过get-pip.py安装:
# 创建或指定get-pip.py下载位置,e.g., mkdir path_to_getpip, cd path_to_getpip
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python2.7 get-pip.py
#也可指定pip版本号等:
python2.7 get-pip.py pip==9.0.2 wheel==0.30.0 setuptools==28.8.0
sudo apt-get update/upgrade
update or upgrade the Ubuntu system, e.g.,
sudo apt-get update
sudo apt-get upgrade
(install) pythonx.x
安装其它版本的python(此处以安装python2.7.10为例):
- 查看当前python安装版本:
# 查看当前python2版本:
$ python2 --version
# 查看当前python2版本:
$ python3 --version
# 或根据安装位置查看python版本:
$ ls /usr/bin/python*
$ ls /usr/local/bin/python*
- 安装编译依赖工具
# 安装编译依赖工具
sudo apt-get install libncurses5-dev
- 准备源码
# 手动源码安装python2.7.10过程如下:
sudo apt update && sudo apt upgrade -y
# 安装python依赖包:
sudo apt-get install build-essential checkinstall
sudo apt-get install libreadline-gplv2-dev libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev libffi-dev zlib1g-dev
# 选择python包下载位置:
cd /opt
# 下载python安装包:
sudo wget -c https://www.python.org/ftp/python/2.7.10/Python-2.7.10.tgz
# 解压python安装包:
sudo tar xzf Python-2.7.10.tgz
- 手动安装
# 开始手动安装:
cd Python-2.7.10
sudo ./configure --enable-optimizations --prefix=/usr/local/bin/python2.7
sudo make altinstall
- 检查是否安装成功
# 检验是否安装成功:
python2 --version
# 如安装成功,会显示Python 2.7.10。新安装的python2.7.10在/usr/local/bin/python2.7
虚拟环境 安装 / 配置
(install) pip/pip3
install pip3 and virtualenv, e,g, sudo apt-get install python3-pip/ python3-dev/ python-virtualenv
sudo apt-get install python3-pip python3-dev python-virtualenv
更安全的安装方式为(ref:官网 https://pypi.org/project/pip/)
参考文档:链接: https://pan.baidu.com/s/1mPRz2_MTSmKCazXKaaNA9Q,提取码: 4ag7
# 通过get-pip.py安装:
# 创建或指定get-pip.py下载位置,e.g., mkdir path_to_getpip, cd path_to_getpip
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python2.7 get-pip.py
#也可指定pip版本号等:
python2.7 get-pip.py pip==9.0.2 wheel==0.30.0 setuptools==28.8.0
(install) virtualenv
python2.7 -m pip install virtualenv
# or install under python3.5
python3.5 -m pip install virtualenv
virtualenv
create a virtualenv for a corresponding project, in venv_folder with venv_name, for python3.X, e.g., virtualenv -p python3.X ~/venv_folder/venv_name
virtualenv -p python3.5 ~/venv/deepspeech
or
virtualenv --python==python3.5 ~/venv/deepspeech
source
activate the created existing virtualenv, source ~/venv_folder/venv_name/bin/activate, e.g.,
source ~/venv/deepspeech/bin/activate
pythonX.X -m pip install --update pip
update pip to the latest version, e.g.,
python3.5 -m pip install --update pip
pythonX.X -m pip install
install a certain package, e.g., tensorflow
python3.5 -m pip install --upgrade link_or_path_to_tensorflow_whl_file
注:实际遇到问题:
系统默认的python3版本为3.5(可通过python3 --version查看),此时查看电脑已安装的python3其它版本(可通过ls /usr/bin/python* 或 ls /usr/local/bin/python*查看),如下所示:
当某一个安装需在python3.6及以上时,系统默认的python3版本为3.5,此时可指定python3位置进行安装:
/usr/local/bin/python3.6 -m pip install XXX
pip list
pip list 可查看当前虚拟环境下已安装的所有包
pip list
pip freeze
pip freeze 可将当前虚拟环境下已安装的包,生成txt文件
pip freeze > requirements.txt
系统: 进程查找和停止 / 重启 / 清空垃圾箱 / 配置环境变量 / 软件安装位置查找 / 运行指令 / U盘识别
dkpg --list
ref: https://blog.csdn.net/luckydog612/article/details/80877179
查看安装的包
dkpg --list
ps & grep
search the thread of a certain software, e.g., to search audacity
ps -A | grep audacity
kill
then we can kill the thread by its thread id, e.g.,
kill -9 59265
reboot
reboot the computer
reboot
trash-empty
直接清空垃圾箱
trash-empty
注:垃圾箱在.local/share/Trash 文件夹内,也可直接sudo rm -rf *
当不在Trash文件夹内时:sudo rm -rf .local/share/Trash/*
expert
add an execuable bin path to $PATH, e.g.,
expert PATH=/mypath/bin:$PATH
vim ~/.bashrc
add a new path permalently
vim ~/.bashrc
#在最后一行添上:
export PATH=/mypath/bin:$PATH
ps: used operations for vim
i
—edit vim
esc
—exit edit
:wq
—save and close
whereis
查找软件安装位置,可用ls
在安装位置直接查找,如python
ls /usr/bin/python*
ls /usr/local/bin/python*
当不知道软件安装位置,更通用的方式是whereis
whereis python2
whereis python3
whereis python
make
to generate .bin, e.g.,
make
sudo chmod +x XX.bin
sudo ./XX.bin
make clean
clear all *.o, *.bin
make clean
sh
sh ./run.sh
creating sh file for commands is convenient
train_file=filelist.txt
path=/tmp/test
find $path -name “*.wav” > $train_file
U盘识别
当遇到无法识别大容量U盘时, 如error mounting - exfat:
sudo apt-get install exfat-fuse
Vmware
管理内存
vmware 里安装Ubuntu,占用空间只会增加不会减小,可用如下方式进行压缩:
sudo vmware-toolbox-cmd disk shrink /
同时,可能有帮助的操作包括:
sudo apt-get autoremove
sudo apt-get remove
虚拟机选择设置,压缩空间。
迁移虚拟机(vmdk文件)
vmware 虚拟机会将虚拟机内的文件存成*.vmdk
文件,如下图所示:
每个vmdk文件均为2-4G?大小。如设置虚拟机磁盘空间为120G,共有32个vmdk文件。
其中后20G是扩展后的,新增加了Ubuntu 64 位-s027.vmdk至 Ubuntu 64 位-s032.vmdk 共6个文件。
迁移虚拟机即将这些vmdk文件拷贝到另一个磁盘下,如这里的F:\VMware。迁移虚拟机必须拷贝的文件,包括
- Ubuntu 64位.cmx
- Ubuntu 64位.vmdk
- Ubuntu 64位-s001.vmdk
- …
- Ubuntu 64位-s032.vmdk
所以我们可以通过迁移vmdk:
- 重新指定虚拟机的新磁盘(如原存储盘内存不够时,可将虚拟机的存储文件即vmdk迁移到另一个磁盘):
具体操作可参考:【vmware磁盘文件(vmdk)迁移】(https://blog.csdn.net/weixin_34365635/article/details/85910019),即:
1、首先将虚拟机下所有磁盘文件(vmdk),复制到其它存储设备(移动硬盘)。
2、将原来的磁盘删除掉
3、添加新硬盘。硬件添加向导:“使用现有的虚拟磁盘”,指定虚拟磁盘位置。(即打开迁移后的Ubuntu 64位.vmdk)
4、创建新磁盘成功,可以正常启动虚拟机了。
- 迁移虚拟机到新的磁盘、电脑等:
即在启动虚拟机后,选择【打开虚拟机】,选择迁移后的【Ubuntu 64位.vmx 】。
注:系统会自动载入vmdk文件,但*.vmdk 须与Ubuntu 64位.vmx 在同一文件夹下,否则报错无法找到*.vmdk。且需包括Ubuntu 64位.vmdk。
Install/update NVIDIA driver
It’s useful when the ubuntu system do not work due to the update of the nvidia-smi driver.
Preparation: download the NVIDIA driver according to your system and computer configuration:, e.g.,:NVIDIA driver
Step 1: Ctrl + Alt + F1
Step 2: reinstall nvidia-smi driver
sudo./NVIDIA-Linux-x86_64-430.64.run -no-x-check -no-nouveau-check -no-opengl-files
Some options are:
- The distribution-provided pre-install script failed:
Continue installation
- Would you like to run the nvidia-xconfig utility to automatically update your X configuration file so set the NVIDIA x driver will be used when you restart X:
NO
- Install 32-bit compatibility libraries:
NO
Step 3: start service
sudo service lightdm restart
Step 4: login in and check whether NVIDIA driver is successfully installed.
nvidia-smi
or
watch -n 10 nvidia-smi
音频文件处理
ref: https://blog.csdn.net/u010637291/article/details/107913143
sox
https://www.cnblogs.com/zhuminghui/p/11971311.html
http://manpages.ubuntu.com/manpages/bionic/man1/sox.1.html
faad
见上链接
ffmpeg
见上链接
Shell
常用字符
ref: https://www.cnblogs.com/pengxl/archive/2010/11/29/1891071.html
$ (取值):
path=/tmp/afolder
$path
num=1
$num
> (覆盖性生成文件)
path=/tmp/afolder
filelist=filelist.txt
find $path -name '*.wav' > $filelist
>> (向文件追加内容)
path=/tmp/afolder
filelist=filelist.txt
find $path -name '*.wav' >> $filelist
| (管道符号)
上一个的命令输出作为下一个命令的输入。典型应用:
ps -ef | grep root
&& (连接多条命令)
如果命令1执行成功,继续执行命令2;否则,不执行命令2。典型应用:
apt-get update && apt-get dist-upgrade
|| (两条命令或)
用法:命令1 || 命令2
特性:如果命令1执行成功,不执行命令2;否则,才执行命令2.
; (命令分隔符)
用法:命令1 ; 命令2。一行语句中,顺次执行各命令
数字:加减乘除
- 整数
$expr( XX + XX)
#!/bin/sh
a=8
b=4
c=$(expr $a \* $b) #乘法
c=$(expr $a + $b) #加法
c=$(expr $a - $b) #减法
c=$(expr $a / $b) #除法
- 浮点数
bash 不支持浮点运算,如果需要进行浮点运算,需要借助bc,awk 处理。
ref: https://blog.csdn.net/u010168781/article/details/87370053
#!/bin/bash
#加
f=$(echo "4.3+2.5"|bc)
echo "4.3+2.5=$f"
#减
f=$(echo "4.3-2.5"|bc)
echo "4.3-2.5=$f"
#乘
f=$(echo "4.30*2.50"|bc)
echo "4.3*2.5=$f"
#除
f=$(echo "4.3/2.5"|bc)
echo "4.3/2.5=$f"
#混合运算
f=$(echo "2.2/(2.2-1.1)*2+1.1"|bc)
echo "2.2/(2.2-1.1)*2+1.1=$f"
注:获取shell命令的执行结果方式为:result=$(shell commands)
,如:f=$(echo "4.3/2.5"|bc)
实际应用示例:
path=/tmp/model/work/recordings
filelist=filelist.txt
fileid=1
fileid_interval=1
fileid_max=700
while [ $fileid -lt $fileid_max ] #true
do
filename=${
path}/recording_$fileid.wav
filename_maxed=${
filename}_our_maxed.wav
if [ -f $filename ];then
echo $filename_maxed >> $filelist
else
echo "文件不存在:"${
filename}
fileid=$(expr $fileid + $fileid_interval)
continue
fi
# HERE !!!
sox $filename -n stat -v 2> vc
vol_max=$(cat vc)
vol_our=$(echo "$vol_max*0.707"|bc)
# HERE !!!
sox -v $vol_our $filename $filename_maxed
sudo ./main.bin $filename_maxed
fileid=$(expr $fileid + $fileid_interval)
done
数字:比较大小(条件判断)
-gt
是>
-lt
是<
-eq
是=
-ne
是不等于
-ge
是>=
-le
是<=
示例:
#!/bin/bash
if [ 1 -gt 2 ] ; then
echo "参数1大于参数2"
else
echo "参数1小于参数2"
fi
# 输出:
# 参数1小于参数2
fileid=1
fileid_interval=1
fileid_max=700
while [ $fileid -lt $fileid_max ]
do
# some shell commands here...
fileid=$(expr $fileid + $fileid_interval)
done
字符串:长度 / 拼接 / 裁剪
ref:https://www.cnblogs.com/chengmo/archive/2010/10/02/1841355.html
长度
${
#string}
拼接
#!/bin/bash
var1="/HOME"
var2="/use1"
var3=$var1$var2
var4=${
var1}${
var2}
var5=$var1/use1
# 输出
echo var3: $var3
echo var4: $var4
echo var5: $var5
如:
var3: /HOME/use1
var4: /HOME/use1
var5: /HOME/use1
截取
#在$string中, 从位置$position开始提取子串
${
string:position}
#在$string中, 从位置$position开始提取长度为$length的子串
${
string:position:length}
文件 / 文件夹 是否存在
ref: https://www.cnblogs.com/37yan/p/6962563.html
文件夹是否存在,不存在则:如创建
if [ ! -d "/data/" ];then
mkdir /data
else
echo "文件夹已经存在"
fi
文件是否存在,存在则:如删除
if [ ! -f "/data/filename" ];then
echo "文件不存在"
else
rm -f /data/filename
fi
continue / break
和普通用法一样。
示例:文件遍历,不存在则,如跳过:
path=/media/me/nvme2n1/SoundPlus/ML-KWS-for-MCU/ML-KWS-for-MCU_tf2/model/work/recordings
filelist=filelist.txt
fileid=1
fileid_interval=1
fileid_max=700
while [ $fileid -lt $fileid_max ] #true
do
filename=${
path}/recording_$fileid.wav
if [ -f $filename ] ; then
echo $filename >> $filelist
else
echo "文件不存在:"${
filename}
fileid=$(expr $fileid + $fileid_interval)
continue
fi
fileid=$(expr $fileid + $fileid_interval)
done
参考
安装pip
官网: https://pypi.org/project/pip/)
参考文档Installation pip
:链接: https://pan.baidu.com/s/1mPRz2_MTSmKCazXKaaNA9Q,提取码: 4ag7
安装python其它版本
参考文档Managing Multiple Versions of Python on Ubuntu
:
链接: https://pan.baidu.com/s/1fHCCwdGHbTGrePa9ge5aRg,提取码: is89
pythonx.x -m pip install
参考文档Why you should use ’python -m pip‘
:
链接: https://pan.baidu.com/s/157yed_WKWcV3Gfl_IhSvfA,提取码: nbjy