Spark with Pi Cluster 1

Infrastructure

flowchart TD; A[Router] -->|Ethernet| B[Switch]; subgraph s1[Cluster]; direction LR; C[(Pi 1)] <-..-> D[(Pi 2)]; D <-..-> E[(Pi 3)]; C <-..-> E; end; B -->|Ethernet| s1; A -..->|Wifi| F[Laptop] F <-..-> s1;

Configure SSH for intercommunications

  1. Change hostname for each pi,
    1. For each pi, do the same in /etc/hosts, comment out 127.0.1.1 raspberrypi and add lines at the end, 
      2. 
#127.0.1.1 raspberrypi
192.168.0.14 pi41
192.168.0.15 pi42
192.168.0.16 pi43
    2. For each pi, change the hostname as pi4x in /etc/hostname, replace x with another number, like pi41 pi42 pi43,
      3. 
pi4x
    3. Reboot each machine now.
    4. Test: ssh pi@pi4x.
  2. Simplify ssh command via ssh alias,
    1. In my laptop, edit ~/.ssh/config, copy to all pis as well,
      2. 
Host pi41
    User pi
    Hostname 192.168.0.14   

Host pi42
    User pi
    Hostname 192.168.0.15

Host pi43
    User pi
    Hostname 192.168.0.16
    2. Test: ssh pi4x
  3. Allow public/private key pairs for ssh among pis and laptop,
    1. In each pi server, generate a public/private key pairs, 
      2. 
ssh-keygen -t ed25519
# all default is fine
    2. Copy public key from client to the target server means that you can ssh into the target without any password anymore.
      3. 
# all server (14)
cat .ssh/id_ed25519.pub >> .ssh/authorized_keys
# all clients (15,16)
cat ~/.ssh/id_ed25519.pub | ssh pi@192.168.0.14 'cat >> .ssh/authorized_keys'
OR
ssh-copy-id -i pi@pi41
ssh-copy-id -i pi@pi42
ssh-copy-id -i pi@pi43
    3. Copy this authorized_keys to every server,
      4. 
scp ~/.ssh/authorized_keys pi@pi4x:~/.ssh
    4. Test: ssh pi41, it should not require any password anymore.
  4. Make command shorter,
    1. Add extra lines in ~/.bashrc,
      2. 
alias p1="ssh pi41"
alias p2="ssh pi42"
alias p3="ssh pi43"
    2. Run the bash file. source ~/.bashrc
    3. Test: p1 should have the same effect as ssh pi41.

References