Raspeberry Pi - Connect to Wifi and send ip information to email

There is problem with the current setting of my Raspeberry Pi. Every time when it is started, I need to log into my account and connect to wifi. In order to use ssh on my desktop, I also need the ip information of the Raspberry Pi.

To automate this part, we need to write some autostart program. Essentially, we want to write scripts that can be executed upon the start.

In ArchLinux, it can be doen through systemctl. The documentation can be found here:
https://wiki.archlinux.org/index.php/systemd

It takes a while to read through all the sections. Some Useful examples can be found here:
https://www.freedesktop.org/software/systemd/man/systemd.service.html

Here is what we need to do:

Step 1: Create the script that need to be executed upon the start.

In our case, we want to have something like

#!/bin/sh
ip link set wlan0 down
netctl start home_wifi
ifconfig | mail -v -s "IP information" user@gmail.com

After editing the script, make it executable: chmod + x your-script

Step 2: Create a new service

Go to the /etc/systemd/system folder and create a new file
my-autostart.service

[Unit]
Description=Send ip information through email

[Service]
ExecStart=/path/your-script

[Install]
WantedBy=multi-user.target

Save the edit and enable the service by
systemctl enable my-autostart.service

Caveate
Though it can work, the configuration of the service may not be completely correct. It is better to have more fine control on it.

Use Raspberry Pi as a server

With the help of gmail, we can use Raspberry Pi as a server. What we need are:

• mail: to send/reply email
• getmail: to get new email messages

It is possible to use plain python to send/receive email, but here it is more convenient to use the existing tools.

The request consists of

• subject which contains [request@me]. This is the request identifier
• request message included in the email message. Each request starts with @begin and ends with @end. Each line in between has the form of attributeName = attributeValue. For example, request = take-picture indicates that this request is asking raspberry-pi to take a picture.

 As mentioned in one of the previous post, we need the raspistill working on raspberry pi. Otherwise, the flow is:

1. Check new email messages by calling getmail -n
2. Parse the new mail messages if there is any and get the request object
3. Process the request

Here is the code:

import re
import subprocess
import os
from os import path
import hashlib
from datetime import datetime
import time
import subprocess
import shlex
import logging
import glob


_currDir = os.path.dirname( os.path.abspath( __file__ ) )

# setup logger
logging.basicConfig(level    = logging.DEBUG,
                    format   = '%(asctime)s %(name)s %(levelname)s %(message)s',
                    datefmt  = '%m-%d %H:%M',
                    filename = path.join(_currDir, "_send-emil.log"),
                    filemode ='w')

console = logging.StreamHandler()
console.setLevel(logging.INFO)
formatter = logging.Formatter('%(name)s: %(levelname)s %(message)s')
console.setFormatter(formatter)
logging.getLogger('').addHandler(console)

logging.info( "current directory: {}".format( _currDir ) )

# settings
REQUEST_IDENTIFIER = '[request@me]'
IDLE               = 'the parser is idle'
SUBJECT            = 'hitted subject'
BEGIN              = 'beginning-of-request'
IMG_FORMAT         = 'jpg'
RQ_TAKE_PICTURE    = 'take-picture'
ARG_NUM            = 'number'
MAIL_DIR           = '/home/meretciel/mail/new'
IMG_DIR            = '/home/meretciel/workspace/camera'
RECEIVERS          = [ 'meretciel.fr@gmail.com' ]


class EmailMessage( object ):
    _COMMAND_TEMPLATE_PLAIN      = r'mail -v -s "{subject}" {receiver}'
    _COMMAND_TEMPLATE_ATTACHMENT = r'mail -v -s "{subject}" {attachment} {receiver}'

    def __init__( self, subject = None, message = None, receiver = None, attachment = None ):
        self._subject    = subject
        self._message    = message
        self._receiver   = receiver
        self._attachment = attachment

    def _generateEmailCommand( self ):
        if isinstance( self._receiver, list ):
            receiver = ';'.join( self._receiver )
        else:
            receiver = self._receiver

        if not self._attachment:
            return EmailMessage._COMMAND_TEMPLATE_PLAIN.format( message = self._message, subject = self._subject, receiver = receiver )
            

        if isinstance( self._attachment, list):
            attachment = ' -a '.join( self._attachment )
            attachment = ' -a ' + attachment

        else:
            attachment = ' -a ' + self._attachment
        

        return EmailMessage._COMMAND_TEMPLATE_ATTACHMENT.format( message = self._message, subject = self._subject, receiver = receiver, attachment = attachment )

    
    def send( self ):
        ''' send email using mail command. We can also implement this in plain python '''
        p1 = subprocess.Popen( 
            shlex.split( r'echo {message}'.format( message = self._message ) ),
            stdout = subprocess.PIPE,
        )
        p2 = subprocess.Popen(
            shlex.split( self._generateEmailCommand() ),
            stdin = subprocess.PIPE
        )
        p2.communicate()

def constructCommand( s_command ):
    l = [ x for x in s_command.split(' ') if x != '' ]
    return l 

def executeCommand( s_command ):
    logging.info( "execute command: {}".format( s_command ) )
    subprocess.call( constructCommand( s_command ) )

def getEmail():
    executeCommand( "getmail -n" )




def checkNewEmail():
    ''' check if there is new ( unread ) emails. '''
    ls_output = subprocess.check_output( constructCommand( "ls -lt {}".format( MAIL_DIR ) ) )
    ls_output = str( ls_output, "utf-8" )
    logging.debug( "ls_output: {}".format( ls_output ) )
    ls_output = ls_output.split('\n')
    ls_output = ls_output[1:]   # the first line is total output
    newFiles  = [ x.split(' ')[-1] for x in ls_output ][::-1]
    newFiles  = [ path.join( MAIL_DIR, x ) for x in newFiles if x != '' ]

    return newFiles



def takePicture( number ):
    ''' ask raspiberry pi to take picture. '''
    time            = datetime.utcnow()
    baseFileName    = path.join( IMG_DIR, hashlib.sha224( str( time ).encode( "utf-8" ) ).hexdigest() )
    commandTemplate = r'/opt/vc/bin/raspistill -n -vf -w 640 -h 480 -e {IMG_FORMAT} {_} -o {baseFileName}%04d.{IMG_FORMAT}'.format( IMG_FORMAT = IMG_FORMAT, baseFileName = baseFileName, _ = "{time}")
    number          = min( 10, int( number ) )
    number          = max( number, 1 ) 

    if number == 1:
        command = commandTemplate.format( time = '' )
        imgFileNames = [ "{baseFileName}-001.{IMG_FORMAT}".format( baseFileName = baseFileName, IMG_FORMAT = IMG_FORMAT ) ]
    else:
        _tl           = 2000    # in milliseconds
        _totalTime    = _tl * ( number - 1)
        _time         = "-t {} -tl {}".format( _totalTime, _tl )
        command       = commandTemplate.format( time = _time )

        imgFileNames = ["{baseFileName}{num}.{IMG_FORMAT}".format( baseFileName = baseFileName, num=str(i).zfill(4), IMG_FORMAT=IMG_FORMAT ) for i in range(number)]

    try:
        executeCommand( command )
        return imgFileNames
    except Exception as e:
       logging.error("Error when taking picture") 
       return []


class Request( object ):

    def __init__( self ):
        self._requestName = None
        self._args = {}

    def load( self, attrName, value ):
        if attrName == 'request':
            self._requestName = value

        else:
            self._args.update( { attrName : value } )

    def process( self ):
        if self._requestName:
            logging.info( 'processing {}'.format( self ) )

            if self._requestName.lower() == RQ_TAKE_PICTURE:
                number = self._args.get( ARG_NUM, 1 )
                imgFiles = takePicture( number )
                logging.debug( '<2> image files : {}'.format( imgFiles ) )
                newEmail = EmailMessage( subject = 'New Images', message = '', receiver = RECEIVERS, attachment = imgFiles )
                newEmail.send()
                for fn in imgFiles:
                    logging.info( "removing the file: {}".format( fn ) )
                    os.remove( fn )


    def __repr__( self ):
        return "Request( name={}, args={} )".format( self._requestName, str( self._args ) )

def _parseEmail( f, state, requests ):
    if state == IDLE:
        line = f.readline()
        while line:
            if 'Subject' in line and REQUEST_IDENTIFIER in line:
                state = SUBJECT
                break
            line = f.readline()
        return line, f, state, requests

    if state == SUBJECT:
        line = f.readline()
        while line:
            if '@begin' in line:
                state = BEGIN
                break
            line = f.readline()
        return line, f, state, requests

    if state == BEGIN:
        pattern = r'(?P<attrName>\w+)\s*=\s*(?P<value>.+)'
        line = f.readline()
        request = Request()
        while line:
            if '@end' in line:
                requests.append( request )
                state = IDLE
                break

            res = re.search( pattern, line )
            if res:
                attrName  = res.group( 'attrName' )
                value     = res.group( 'value' )
                request.load( res.group( 'attrName' ), res.group( 'value' ) )

            line = f.readline()

        return line, f, state, requests
        



def parseEmail( msgFile ):
    logging.info("parsing email file {}".format( msgFile ) )
    with open( msgFile ) as f:
        state = IDLE
        line = '__start__'
        requests = []
        while line:
            logging.info( "processing line : {}".format( line ) )
            line, f, state, request = _parseEmail( f, state, requests )

        return requests

def removeNewMsgFiles( newMsgFiles ):

    for item in newMsgFiles:
        os.remove( item )


def getNewRequestFromEmail():
    getEmail()
    newMsgFiles = checkNewEmail()
    logging.debug(" <1> New messages : {}".format( str( newMsgFiles ) ) ) 
    requests = []
    for newMsgFile in newMsgFiles:
        requests.extend( parseEmail( newMsgFile ) )
    removeNewMsgFiles( newMsgFiles )
    return requests


if __name__ == '__main__':
    
    existingFiles = glob.glob( path.join( MAIL_DIR, r'*.alarmpi' ) )
    for f in existingFiles:
        os.remove( f )

    while True:
        logging.info( "waiting for request." )
        requests = getNewRequestFromEmail()
        for request in requests:
            request.process()

        time.sleep( 10. )

--END--

Raspberry Pi & Arch Linux - Day 18 - Noncanonical mode input and child process in Python

In this post, we will talk about two thing: (1) noncanonical mode of input and (2) child process in Python. The knowledge of the child process is not necessary for our project whereas the first is crucial. Before we can build algorithm to instruct the robot to move automatically, we first need to know how to control the robot manually. In other words, we are going to build a remote control robot car before anything else.

The idea is that we enter the command through the keyboard and once the program receives the command it controls the speed and the direction of the robot accordingly. For example, when we type the up arrow key, the robot will increase its speed or when we type the left arrow key, the robot will turn left.

There is one problem. Most of the time, the default setting of the user input is based on line. The user will enter whatever he wants and then press "return" to submit the command. However in our case, we want to enter the command char by char. We do not want to type 10 times the up arrow key plus a "return" key to tell the program that we want to quickly increase the speed of the robot.

A quick search on Google gives us the following solution.

def getchar():
   #Returns a single character from standard input
   import tty, termios, sys
   fd = sys.stdin.fileno()
   old_settings = termios.tcgetattr(fd)
   try:
      tty.setraw(sys.stdin.fileno())
      ch = sys.stdin.read(1)
   finally:
      termios.tcsetattr(fd, termios.TCSADRAIN, old_settings)
   return ch

Note that the three packages used in this function are quite low-level.

This solution has one drawback. When we call the getchar() function, it will block and wait for user to enter the character. This is not convenient when the user does not want to any command. To deal with this problem, we need to have timeout feature. We can find many solutions on this issue.  Some of the solutions uses a separate process to handle the inputs. Here we have another solution.

def getchar(timeout=3):
    fd = sys.stdin.fileno()
    old_setting = termios.tcgetattr(fd)

    try:
        # switch to noncanonical mode
        tty.setraw(sys.stdin.fileno())
        new_setting = termios.tcgetattr(fd)

        # set the timeout
        cc = new_setting[6]
        cc[termios.VTIME] = timeout # timeout = 0.3s
        cc[termios.VMIN]  = 0       # start the timer immediately

        # update termios struct
        termios.tcsetattr(fd, termios.TCSADRAIN, new_setting)

        # read
        ch = sys.stdin.read(1)
    finally:
        termios.tcsetattr(fd, termios.TCSADRAIN, old_setting)
    return ch

As we mentioned, termios is a quite low-level package. What we need is actually called the noncanonical mode of input. It is controlled in the struct termios and it does provides parameters to control the timeout behavior.

The second thing we want to cover in this post is the child process. The following code shows how we can send the interrupt signal to all the processes.

if key_press == 'q':
    print("Exiting system")
    pid = os.getpid()
    parent = psutil.Process(pid)
    children = parent.children(recursive=True)
    for process in children:
        os.kill(process.pid, signal.SIGINT)

    os.kill(pid, signal.SIGINT)

In Python, we use os package to get the current process id and send signal to a given process. The signals are defined in the signal package. To send the signals to the child processes, we need to get the child process id of a given process. This functionality is provided in the psutil package. More specifically, we use the psutil.Process(pid) to get a list of child process object of the given process id.


--END--

Raspberry Pi & Arch Linux - Day 17 - Visualization in terminal and KeyboardInterrupt handler

Happy Halloween!

Last time, we saw that the robot car could run without any direction and speed control. To make the robot more intelligent, we should first make sure that it can interact with the environment. All the communication between the robot car and the environment go through the radar in the front of the car. It consists of a stepper motor as the base and a ultrasonic distance sensor which measure the distance to the obstacles.

As we add more components to the robot car, the number of parameters in the system increase dramatically. (probably it is a good idea to have an overview of all the parameters currently used in the robot car). Before we code the direction and speed control algorithm, it is necessary to configure all the parameters so that the robot car can function properly. The parameters of the most interests are

• the range of the radar
• the angle speed of the radar
• the frequency of the sampling (radar base and distance sensor)
• the buffer size of the data

One of the problems is how we can visualize the distance map in the terminal. There are two approaches:

• gnuplot
• bashplotlib in python

If we use gnuplot, we need to start a subprocess in python and sometimes it is very difficult to deal with the subprocess, especially when we want to load the outputs of the subprocess. So for our testing purpose, we choose the bashplotlib package. 

The function we use is plot_hist because the distance map is essentially a histogram. The x-axis is the degree, which indicates the position of the radar base (stepper motor); the y-axis is the distance to the obstacles. To plot the histogram, we need to convert the floating distance to int. 

I do not know if I missing something about the bashplotlib package. It seems to me that the plot_hist function only accepts the raw data, which means we can not pass a dictionary or pd.Series to it. So if you have a pd.Series 

In[4]: ts

Out[4]: 

-1    3

 0    2

 1    5

dtype: int64

You need to manually convert it to [-1,-1,-1,0,0,1,1,1,1,1].

The figures below present the plot in terminal. It is not very fancy but it is much better than looking at an array of 100 numbers. :)

To make the testing more convenient, we want the radar to go back to its zero or initial position when the process is interrupt by the ctr-c. When we enter the ctr-c, the python interpreter will raise a KeyboardInterrupt error in the "main" process and all the subprocess as well. It means that we need to handle the keyboard interrupt error in both the main script and the scripts that use multiprocessing.Process. Fortunately, all components that are running concurrently are wrapped in the ContinuousComponentWrapper. So we just need to add a exception handler in the run method of the ContinuousComponentWrapper

    # ContinuousComponentWrapper

    def run(self):
        """
        Running the component in the infinite loop. To change the status of the component, one can send command to the command queue.
        #TODO: add a stop-pill
        """
        try:

            while True:
                while not self._cmd_Q.empty():
                    cmd = self._cmd_Q.get()
                    if cmd == CMD_EXIT or cmd == (CMD_EXIT,):
                        return 
                    self._component.parse_and_execute(cmd)


                self._component.run()
                self._component.send_msg(self._output_Q)

        except KeyboardInterrupt:
            if hasattr(self._component, 'KeyboardInterruptHandler'):
                self._component.KeyboardInterruptHandler()
            print('{} property exit after KeyboardInterrupt.'.format(self._component.name))

and create a KeyboardInterruptHandler method for the DistanceRadarBaseComponent.

    # DistanceRadarBaseComponent

    def KeyboardInterruptHandler(self):
        self._stepper_motor.back_to_zero_pos(delay=self._delay)

--END---