Raspberry Pi & Arch Linux - Day 2

Raspberry Pi & Arch Linux - Day 2

Today's objective is to set up the automatic wifi connection. Again, the Arch Linux Wiki has everything, just follow the instruction and we will be fine. (https://wiki.archlinux.org/index.php/Netctl)

We will manage the wifi connection with the netctl tool. Instead of typing all the commands mentioned in Day 1, we can store the wifi connection configuration in a profile file. The profile file should be in /etc/netctl. As mentioned in the wiki, we can copy the sample profiles in the /etc/netctl/example folder and make appropriate customization.

Here is an example of the profile file: 

[To be added]

To establish a connection, just enter the following the commend:

netctl start profile

If the command fails, type netctl status profile to see the error message. Note that before running netctl make sure that there is no other network service running in the background. In our case, we need to first disable the wifi connection

ip link set wlan0 down

Now everything should work!

At this point, I need to find a solution to copy the message from Raspberry Pi to Mac. It becomes painful to re-type the output messages of Raspberry Pi.  Obviously it is not the right time to start writing blog on Raspberry directly. I haven't even try google on Raspberry. The best way I can think of is to send email from Raspberry Pi to my personal email.

TODO:

1. learn how to send email.

Raspberry Pi & Arch Linux - Day 1

Raspberry Pi & Arch Linux - Day 1

Today is the start of a new journey: learn linux system.

Past experience has taught me that doing experiment with linux on PC is too risky. At first I was thinking of getting a cheap used computer and install a linux system but this is still expensive. After some researches on the internet, the Raspberry Pi caught my attention. It is a mini computer that costs only $50. I am not sure if it is a powerful machine but obviously it is enough for me to learn the linux system. (For more information, please refer to https://www.raspberrypi.org)

The next question is which linux distribution to use. I have Ubuntu on my personal computer. Some people recommend to start with Ubuntu but I want something more challenging. Ubuntu is user-friendly precisely because it has everything pre-installed. In order to learn the linux, I believe it is better to build something on my own. This thought makes me finally chose the Arch Linux distribution which is a lightweight and flexible linux distribution. (For more information about Arch Linux, please refer to https://www.archlinux.org)


The first thing to do is to start the Raspberry Pi. I bought my Raspberry Pi from Amazon:

(https://www.amazon.com/CanaKit-Raspberry-Clear-Power-Supply/dp/B01C6EQNNK/ref=sr_1_6?s=pc&ie=UTF8&qid=1473563202&sr=1-6&keywords=raspberry+pi+3)

This product does not include the SD card and unexpectedly this makes the very first trouble.

Problem 1 : SD Card

For some reasons, I wanted to have a large SD card. I was thinking the larger the SD card is the better because SD card is used as a disk for the Raspberry Pi. I bought a 64G SD card but it turns out to be the first "mistake".

64G SD card uses a special filesystem: extFAT. According to Wikipedia:

exFAT is a proprietary and patent-protected file system with certain advantages over NTFS with regard to file system overhead.
exFAT is not backward compatible with FAT file systems such as FAT12, FAT16 or FAT32. The file system is supported with newer Windows systems, such as Windows Server 2003, Windows Vista, Windows 2008, Windows 7, Windows 8, and more recently, support has been added for Windows XP.[27]
exFAT is supported in OS X starting with version 10.6.5 (Snow Leopard).[26] Support in other operating systems is sparse since Microsoft has not published the specifications of the file system and implementing support for exFAT requires a license. exFAT is the only file system that is fully supported on both OS X and Windows that can hold files bigger than 4 GB.[citation needed]

Having no idea about these details, I tried to write the Arch Linux image file to SD card on a Linux machine but always got a permission error. I cannot even format the SD card on my linux machine and now I kind of understand this is a filesystem format issue. I did try to format the SD card on Mac but I chose the MS-DOS(FAT) which seems to make the situation worse.

After several attempts to write Arch Linux to the SD card, I finally gave up. At this point it is too difficult for me to solve this format problem. Once I format the SD card on Mac, it seems impossible for me to re-format on a Linux machine not to mention writing the image file to the SD card. The only choice is to buy a new SD card. This time I bought a 16G SD card and hopefully it can work.

(SanDisk Ultra 16GB Ultra Micro SDHC UHS-I/Class 10 Card with Adapter (SDSQUNC-016G-GN6MA, https://www.amazon.com/gp/product/B010Q57SEE/ref=oh_aui_detailpage_o01_s00?ie=UTF8&psc=1)

Following the instruction again (https://archlinuxarm.org/platforms/armv8/broadcom/raspberry-pi-3) and now everything works smoothly.

Now I can start the Arch Linux on my little Raspberry Pi, which is good and makes me excited, but soon I realize one important thing: Arch Linux is so simple that it has nothing! There is no GUI and everything is in command line. Interesting...

To play with Arch Linux, I need a real computer so that I can search useful information. This is a little bit annoying because I need to switch between to keyboards. So the next step is to get internet connection.



Problem 2: How to connect to Wifi

It turns out the Arch Linux Wiki has everything I need if I do not skip any sections on the page and strictly follow the instruction.(https://wiki.archlinux.org/index.php/Wireless_network_configuration#Getting_an_IP_address)

Here are the steps to connect to wifi:

1. set up the wlan0 interface: ip link set wlan0 up
2. connect to wifi: wpa_supplicant -B -D nl80211,wext -i wlan0 -c <(wpa_passphrase "your_SSID" "your_key"). wpa_supplicatn can run in the background. After entering this command, press ctrl+c to exit.
3. getting an IP address: dhcpcd wlan0 
4. (done.)

TODO list:

1. how to automatically config the wireless connection?
2. can we save the configuration/profile?
3. [research] Filesystem
4. [research] SD card
5. [research] WPA/WPA2, WEP

LeetCode: 386. Lexicographical Numbers

386. Lexicographical Numbers  QuestionEditorial Solution  My Submissions
Total Accepted: 4021
Total Submissions: 12734
Difficulty: Medium
Given an integer n, return 1 - n in lexicographical order.

For example, given 13, return: [1,10,11,12,13,2,3,4,5,6,7,8,9].

Please optimize your algorithm to use less time and space. The input size may be as large as 5,000,000.

class Solution {
public:

    vector<int> lexicalOrder(int n) {
        vector<int> res;
        
        int candidate = 1;
        int count = 0;
        
        while (count < n) {
            res.push_back(candidate);
            ++count;
            if (10 * candidate <= n) {
                candidate *= 10;
            } else if (candidate + 1 <= n) {
                candidate++;
                while (candidate % 10 == 0) candidate = candidate / 10;
            } else {
                candidate = candidate / 10;
                while ((candidate + 1) % 10 == 0) candidate = candidate / 10;
                ++candidate;
            }
        }
        return res;
    }
};

LeetCode - 347.Top K Frequent Elements QuestionEditorial Solution My Submissions

347. Top K Frequent Elements  QuestionEditorial Solution  My Submissions
Total Accepted: 28149
Total Submissions: 63972
Difficulty: Medium
Given a non-empty array of integers, return the k most frequent elements.

For example,
Given [1,1,1,2,2,3] and k = 2, return [1,2].

Note:
You may assume k is always valid, 1 ≤ k ≤ number of unique elements.
Your algorithm's time complexity must be better than O(n log n), where n is the array's size.

    def topKFrequent(self, nums, k):
        """
        All elements in the nums array can be a candidate of the final outputs. We use the dictionary candidates to count the number of appearance of candidates.
        
        
        """
        count_qualified = 0     # number of qualified candidates
                                # the initial value is obviously 0.
        level = 0               # if candidates[n] > level the n is qualified
                    
        candidates = {}
        
        for n in nums:
            candidates[n] = candidates.get(n,0) + 1  # add 1 to the count
            
            if candidates[n] == level + 1:    

                # this is the only case we care. If candidates[n] <= offset, 

                # then n is not qualified and no impact on the count_effect
                # for candidates[n] > offset + 1, it means that n has been 

                # already counted as qualified. So no impact neither
                count_qualified += 1           # add 1 more qualified candidate 
                
                if count_qualified > k:        

                    # if the number of quanlified candidates is greater than 

                    # the target, we will increase the level of qualification 
                    level += 1
                    for item in candidates.values():# update the count_quanlified
                        if item == level:
                            count_qualified -= 1
        
        
        res = [key for key,val in candidates.items() if val > level]
                        
        return res

One Line Code Challenge: Special Merge

Suppose we have two data frames:

In[29]: df_data
Out[29]: 
   A  B  C  val_0
0  Y  M  1      1
1  Y  F  1      2
2  N  M  1      3
3  N  F  1      4
4  Y  M  2      5
5  Y  F  2      4
6  N  M  2      3
7  N  F  2      2

In[30]: df_rule
Out[30]: 
   A    B   C  val_1
0  Y  NaN NaN    100
1  N    F NaN    200

2  N    M NaN    300

In the context of an ordinary merge on ['A','B','C'], there is no matches because there are NaN in the df_rule.

In the context of a special merge, we allow NaN to be anything. That is to say
(Y,M,1) can match with (Y,M,NaN) (or (Y,NaN,1)etc)

The task is to achieve this special merge.

Solution:
df_rule.groupby(df_rule.apply(lambda x: '-'.join(x[x.notnull()].index.values), axis=1)).apply(lambda df: pd.merge(df, df_data,on=df.dropna(axis=1,how='all').columns.intersection(df_data.columns).tolist(), how='left',suffixes=('_to_remove','')))[df_rule.columns.union(df_data.columns)]

One Line Code Challenge: compute the list of prime factors

factors = lambda n, k=2: ([k] + factors(n / k) if n % k == 0 else factors(n, k+1)) if n > 1 else []


In[17]: factors(3)
Out[17]: [3]
In[18]: factors(4)
Out[18]: [2, 2]
In[19]: factors(9)
Out[19]: [3, 3]
In[20]: factors(18)
Out[20]: [2, 3, 3]
In[21]: factors(24)
Out[21]: [2, 2, 2, 3]
In[22]: factors(1)
Out[22]: []

Python -- Pandas -- Add a new level to the column in DataFrame

Here is the situation, we have a data frame df

          X         Y         Z
0  0.738596  0.852906  0.333422
1  0.820456  0.014704  0.233935
2  0.118291  0.714536  0.176275
3  0.032417  0.819386  0.949590
4  0.739559  0.923865  0.791574

And for some reason, we want to add a new level to the columns. Unlike adding new level to index, it seems that there is no available function in Pandas to handle this problem.

We can of course do it "manually", something like this:

df.columns = pd.MultiIndex.from_arrays(np.vstack((np.asarray(['test'] * df.shape[1]), df.columns.tolist())))

A more general function is

def add_new_level(index, new_level):
    arrays = np.asarray(index.get_values().tolist()).T
    new_arrays = np.vstack((new_level, arrays))
    return pd.MultiIndex.from_arrays(new_arrays)

(not sure if this is the simplest one, have some doubts)

and the data frame becomes:
       test                  
          X         Y         Z
0  0.738596  0.852906  0.333422
1  0.820456  0.014704  0.233935
2  0.118291  0.714536  0.176275
3  0.032417  0.819386  0.949590
4  0.739559  0.923865  0.791574

or we can play some trick like this one:

df = pd.concat([df], axis=1, keys=['test'])