[experience sharing] the best practice of remote login server with Python

Keywords: Programming ssh Python sftp github

When using Python to write some scripts, in some cases, we need to log in to the remote service frequently to execute a command and return some results.

In a shell environment, this is what we do.

$ sshpass -p ${passwd} ssh -p ${port} -l ${user} -o StrictHostKeyChecking=no xx.xx.xx.xx "ls -l"

Then you will find that your output has a lot of information that you don't need, but you can't get rid of (maybe there is a way, please leave a message for communication), like this

What I don't know in the learning process can add to me
 python learning qun, 855408893
 There are good learning video tutorials, development tools and e-books in the group.  
Share with you the current talent needs of python enterprises and how to learn python from scratch, and what to learn

host: xx.xx.xx.xx, port: xx
Warning: Permanently added '[xx.xx.xx.xx]:xx' (RSA) to the list of known hosts.
Login failure: [Errno 1] This server is not registered to rmp platform, please confirm whether cdn server.
total 4
-rw-r--r-- 1 root root 239 Mar 30  2018 admin-openrc

For the direct use of shell commands to execute commands, you can directly use the pipeline, or redirect the standard output to a file to get the results returned by executing the command

1. Use subprocess

If we use Python to do this, we usually think of using os.popen, os.system, commands, subprocess and other command execution libraries to obtain indirectly.

But as far as I know, these libraries get not only standard output, but also standard errors (that is, the redundant information above)

So we need to clean the output data every time, and then organize and format it to get the data we want.

Take subprocess for example, just like this

import subprocess
ssh_cmd = "sshpass -p ${passwd} ssh -p 22 -l root -o StrictHostKeyChecking=no xx.xx.xx.xx  'ls -l'"
status, output = subprocess.getstatusoutput(ssh_cmd)

# Data cleaning, formatted data will not be displayed
<code...>

Through the above text + code display, you can feel the pain points of ssh login

  • Pain point 1: additional installation of sshpass is required (if necessary)
  • Pain point 2: too much interference information, data cleaning and formatting are quite troublesome
  • Pain point 3: code implementation is not elegant (a little bit earthy), readability is too poor
  • Pain point 4: ssh connection cannot be reused, and only once connection can be executed
  • Pain point 5: the code cannot be used on all platforms, only on Linux and OSX

In order to solve these problems, I searched the articles about Python ssh in the whole network, but I didn't see any complete introduction to this skill.

So I went through a popular Github project: awesome Python CN( github.com/BingmingWon...

I hope to find some useful libraries about remote connection here.

I actually found two

  • sh.ssh
  • Paramiko

2. Use sh.ssh

First of all, sh.ssh

sh is a library that allows you to complete Linxu/OSX system commands through function calls. It's very easy to use. I'd like to write an introduction about it.

$ python3 -m pip install sh

Only one of its functions is introduced today: ssh

Usually two machines visit each other. For convenience, you can set the password free login, so you don't need to enter a password.

This code can realize password free login and execute our command ls-l

from sh import ssh
output=ssh("root@xx.xx.xx.xx", "-p 22", "ls -l")
print(output)

But it's possible that we don't want to set up mutual trust and password free. In order to make this code more general, I assume that we don't set up password free and can only log in with a password.

Here comes the question. To enter a password, you have to use an interactive method. How to implement it in Python?

The original ssh method received a ﹐ out parameter, which can be a string, indicating the file path, a file object (or class file object), or a callback function, meaning that when there is standard output, it will call to pass the output content to this function.

That's easy to do.

As long as I recognize the word "password", I will write my password into the standard input.

The complete code is as follows:

import sys
from sh import ssh

aggregated = ""
def ssh_interact(char, stdin):
    global aggregated
    sys.stdout.write(char.encode())
    sys.stdout.flush()
    aggregated += char
    if aggregated.endswith("password: "):
        stdin.put("you_password\n")

output=ssh("root@xx.xx.xx.xx", "-p 22", "ls -l",_tty_in=True, _out_bufsize=0, _out=ssh_interact)
print(output)

This is the official document( amoffat.github.io/sh/tutorial...

After trying to run, the program will always be running, never return, never exit, and the callback function will never enter.

The above problem can only appear when a password is required. If the machine mutual trust is set, there is no problem.

In order to feel the effect of sh.ssh, I set up the machine mutual trust and password free, and then use the following code.

from sh import ssh

my_server=ssh.bake("root@xx.xx.xx.xx", "-p 22")

# It is equivalent to executing the login command once, and exiting the login after execution
print(my_server.ls())

# You can log in to the server manually during sleep, and use top to check how many terminals are currently connected
time.sleep(5)

# When executing this command again, the number of login terminals will be + 1, and after execution, it will be - 1 again
print(my_server.ifconfig())

It's amazing to find that in the way of bake, my_server.ls() and my_server.ifconfig() seem to be connected through the same ssh and execute commands twice. In fact, you can execute the top command on the remote machine to see the changes of the connected terminals, which will first + 1 and then - 1, indicating that the execution of the two commands is realized through two connections.

So, using sh.ssh can solve pain point one (if the above problems can be solved), pain point two and pain point three.

But it still can't reuse ssh connection, it's not convenient, and it's not the best solution I want.

The most important point is that the sh module only supports Linxu/OSX. In Windows, you have to use its brother library pbs. Then I went to pypi to have a look pbs , it has been "in disrepair for a long time". No one has ever been here. I'm only one straw away from "pawn".

3. Use paramiko

With a last glimmer of hope, I tried to use the paramiko library, and finally found the elegance that should belong to Python in paramiko.

You can install it with the following command

$ python3 -m pip install paramiko

Then, we will introduce several common methods of ssh login

Method 1: login by SSH client based on user name and password

Then you can refer to the following code to connect remotely under Linux/OSX system

import paramiko

ssh = paramiko.SSHClient()
# Allow connections to hosts that are not in the know? Hosts file
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

# Establish a connection
ssh.connect("xx.xx.xx.xx", username="root", port=22, password="you_password")

# Use this connection to execute commands
ssh_stdin, ssh_stdout, ssh_stderr = ssh.exec_command("ls -l")

# Get output
print(ssh_stdout.read())

# Close connection
ssh.close()

Method 2: transport login based on user name and password

Method 1 is a traditional operation of connecting to the server, executing commands, and shutting down. Multiple operations need to be connected multiple times and cannot reuse the connection [pain point 4].

Sometimes you need to log in to the server to perform multiple operations, such as executing commands, uploading / downloading files. Method 1 cannot be implemented, so you can use the transport method.

import paramiko

# Establish a connection
trans = paramiko.Transport(("xx.xx.xx.xx", 22))
trans.connect(username="root", password="you_passwd")

# Specify the transport of the object of sshclient as the above transport
ssh = paramiko.SSHClient()
ssh._transport = trans

# The rest is the same as above
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh_stdin, ssh_stdout, ssh_stderr = ssh.exec_command("ls -l")
print(ssh_stdout.read())

# Close connection
trans.close()

Method 3: SSH client login based on public key

import paramiko

# Specify the local RSA private key file
# If there is a password set when establishing the key pair, password is the set password. If there is no need to specify the password parameter
pkey = paramiko.RSAKey.from_private_key_file('/home/you_username/.ssh/id_rsa', password='12345')

# Establish a connection
ssh = paramiko.SSHClient()
ssh.connect(hostname='xx.xx.xx.xx',
            port=22,
            username='you_username',
            pkey=pkey)

# Execute command
stdin, stdout, stderr = ssh.exec_command('ls -l')

# The result is put into stdout, and if there is any error, it will be put into stderr
print(stdout.read())

# Close connection
ssh.close()

Method 4: key based Transport login

import paramiko

# Specify the local RSA private key file
# If there is a password set when establishing the key pair, password is the set password. If there is no need to specify the password parameter
pkey = paramiko.RSAKey.from_private_key_file('/home/you_username/.ssh/id_rsa', password='12345')

# Establish a connection
trans = paramiko.Transport(('xx.xx.xx.xx', 22))
trans.connect(username='you_username', pkey=pkey)

# Specify the transport of the object of sshclient as the above transport
ssh = paramiko.SSHClient()
ssh._transport = trans

# Execute commands, as with traditional methods
stdin, stdout, stderr = ssh.exec_command('df -hl')
print(stdout.read().decode())

# Close connection
trans.close()

The above four methods can help you to remotely log in to the server to execute commands. If you need to reuse connections, you can use method 2 and method 4 to execute multiple commands in one connection

When you are finished, remember to close the connection.

Realize sftp file transfer

At the same time, paramiko, as the perfect solution of ssh, is very professional and can also be used to realize sftp file transfer.

import paramiko

# item base  trans object# Instantiate a transport object
trans = paramiko.Transport(('xx.xx.xx.xx', 22))

# Establish a connection
trans.connect(username='you_username', password='you_passwd')

# Instantiate an sftp object and specify the connected channel
sftp = paramiko.SFTPClient.from_transport(trans)

# send files
sftp.put(localpath='/tmp/11.txt', remotepath='/tmp/22.txt')

# Download File
sftp.get(remotepath='/tmp/22.txt', localpath='/tmp/33.txt')
trans.close()

Here, Paramiko has won, but there is still a pain point that we haven't mentioned, that is, multi platform, that is, Windows, there is a good thing, a bad thing,.

Good thing: paramiko supports windows

The bad thing is: you need to do a lot of complicated preparation. You can google it, but I suggest you give up directly. The pit is too deep.

4. Write at the end

After a comparison and some examples, we can see that Paramiko is a professional and easy-to-use ssh tool. I think Paramiko module is one of the necessary modules for operation and maintenance personnel. If you need to implement ssh in Python code to get some information from the remote server, I recommend Paramiko to you.

In order to solve the learning difficulties of beginners, the specially established Python learning buckle QUN: ⑧ ⑤ ⑤ - ④ zero ⑧ - ⑧ ⑨ ③ from zero foundation to practical project tutorials, development tools and e-books in various fields of Python. Share with you the current needs of enterprises for Python talents and learn Python's efficient skills. Keep updating the latest tutorials!

Posted by wakenbake on Mon, 11 May 2020 00:56:20 -0700