常用shell脚本

1. 启动jupyter(可配置开机自启)

#!/bin/bash
origin_status=ps -ef | grep -w jupyter | wc -l # grep -w must match whole words;otherwise there is a difference between script and shell window.
if [ $origin_status -eq 1 ]
then
        current_time=date "+%Y%m%d%H%M%S"
        nohup jupyter notebook >jupyter.log_$current_time 2>&1 &
        now_status=ps -ef | grep -w  jupyter | wc -l
        if [ $now_status -eq 2 ]
        then
                echo "succeed in opening jupyter notebook at $current_time!"
        else
                echo "fail to open jupyter notebook."
        fi
else
        current_run=expr $origin_status - 1
        echo "$current_run jupyter notebook is open."
fi
1.修改文件
sudo vim /etc/rc.local
增加一行:su ubuntu -c /home/ubuntu/open_jupyter.sh (以ubuntu用户执行)

2.ubuntu 18.04 参考 https://zhuanlan.zhihu.com/p/63507762

3.注意
3.1 脚本中的所有函数或者命令都给出绝对路径,因为开机自启脚本会以root用户来执行
比如在该脚本中
nohup /home/ubuntu/anaconda3/bin/jupyter notebook >/home/ubuntu/jupyter.log_$current_time 2>&1 &

3.2 修改~/.jupyter/jupyter_notebook_config.py
c.NotebookApp.allow_root = True
c.NotebookApp.notebook_dir = '/home/ubuntu'

2. 自动登陆服务器(需要安装expect: brew install expect)

#!/usr/bin/expect
# set timeout 10 设置响应间隔
spawn ssh hostname@host
expect "*password*"
send "host pwd\r"
interact
#!/usr/bin/expect
set is_first [lindex $argv 0] #只能这样取得输入的参数,这是第一个参数,和bash不一样
set which_model [lindex $argv 1]
set which_ckpt [lindex $argv 2]
if { $is_first  == 1 } {# if只能这么写
        spawn ssh -i ~/Downloads/gongel_new.pem ubuntu@ec2-54-224-75-110.compute-1.amazonaws.com
        #172.31.45.39
        expect "Last login"
        send "rm -rf private_bert;sudo mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-151b9f95.efs.us-east-1.amazonaws.com:\/   ~\/efs-mount-point;git clone -b adv_mask_bert https:\/\/github.com\/gongel\/private_bert.git\r"
        expect "Username"
        send "@qq.com\r"
        expect "Password"
        send "\r"
        expect "Resolving deltas: 100%"
        send "cd private_bert;nohup bash helper_finetune_script $which_model $which_ckpt \&\r"
        expect "*nohup.out*"
        send "\rexit\r"
        interact
} else {
        spawn ssh -i ~/Downloads/gongel_new.pem ubuntu@ec2-54-224-75-110.compute-1.amazonaws.com
        expect "Last login"
        send "cd  private_bert\r"
        interact
}

3. 时间统计

# 方法一
#!/bin/bash

startTime=date +%Y%m%d-%H:%M:%S
startTime_s=date +%s

endTime=date +%Y%m%d-%H:%M:%S
endTime_s=date +%s

sumTime=$[ $endTime_s - $startTime_s ]

echo "$startTime ---> $endTime" "Total:$sumTime seconds"
# 方法二
time bash xxx.sh
# 会返回3个时间数据
# real 该命令的总耗时, 包括user和sys及io等待, 时间片切换等待等等
# user 该命令在用户模式下的CPU耗时,也就是内核外的CPU耗时,不含IO等待这些时间
# sys  该命令在内核中的CPU耗时,不含IO,时间片切换耗时.

4. 查看集群jps(这里免密)

hosts=('master' 'slave1' 'slave2')
for host in ${hosts[@]}
do
        echo "###### ${host} jps #####"
        ssh ${host} jps  # no need to run 'exit' command
done

5. Linux终端命令太长不换行遮盖

修改服务器端~/.bashrc,增加

export PS1='\[\e[32;1m\]\u\[\e[0m\]@\h \W\$ '

6. 防止iTerm/其他客户端不掉线

    • 永久:在/etc/ssh/ssh_config增加ServerAliveInterval 60(每60s向服务器发一次心跳);
    • 暂时:ssh -o ServerAliveInterval=60 hostname@host
0