Linux Administration (alternative name is Computer Devices and Operating Systems) 5 ECTS
A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyze a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly. Specialization is for insects.
- Robert A. Heinlein
Using command line interface, understanding operating system fundamentals, and computer operations are fundamental skills for every IT engineer. Studies in this course are not about specialisation but learning generic and very useful IT skills.
Grading and how to pass the course
- Document a learning diary with answers, notes, comments, and screenshots
- Use Moodle to report your weekly progress
- Follow the course Discord server
- Course equals approximately 100-150 hours of research, solving weekly assignments, writing the learning diary and reviewing two peers
- Before the week #7 deadline: Save your work to a single PDF file and use Moodle to submit your answers for peer reviews. Use generic document template and follow these instructions how to submit the answers
- Peer reviews and reflective assessments: After the week #7 deadline, go again to Moodle and study, review, comment and grade two student submissions randomly assigned to you. Generic peer review instructions
Study groups and the Gilligan's Island Rule
Students are encouraged to study and learn together. Another student is often the best resource for working out a complex computation or understanding a difficult concept. However, in (this class) all documents turned in to fulfill assignments must be the exclusive work of the person submitting them, unless otherwise stated. In order to allow students to work together, yet submit assignments that represent their own thought, the Gilligan's Island rule is adopted.
The Gilligan's Island Rule: You are free to meet with fellow students(s) and discuss an assignment with them. Writing on a board or shared piece of paper during the meeting is acceptable; however, you should not take any written (electronic or otherwise) record away from the meeting. Everything that you derive from the collaboration should be in your head. After the meeting, engage in at least a half-hour of mind-numbing activity (like watching an episode of Gilligan's Island*), before starting to work on the assignment. This will assure that you are able to reconstruct what you learned from the meeting by yourself.
* Gilligan's Island was a 1960s sitcom that set the standard for dim-witted TV. Seinfeld or anything more intelligent, e.g. some PBS programming, also works for this purpose.
Source
Essential links and instructions
Example books
Some fundamentals about computers
Optional assignments. No need to document to the learning diary. Ignore for peer reviews:
Describe these CPU, computer architecture and computing related terms and concepts shortly:
- RISC vs CISC
- Single-board computer (SBC)
- GPU
- CPU/SoC
- x86
- MIPS
- ARM
- AVR
- MOS 6502
- CPU registers
- Opcode
- Illegal opcode
- Bytecode
- F00F bug
Use Linux file command to determine the details of /usr/bin/ls and /usr/lib/sudo/sudoers.so
- What is the CPU architecture it was compiled to?
- What is the executable file format (or Extensible Linking Format) of /usr/bin/ls?
- What kind of file is /usr/lib/sudo/sudoers.so?
CPUs and your smart phone:
- What is the CPU/SoC of your smart phone?
- What is the cache size(s) of the CPU/SoC?
- What is the maximum clock speed it operates?
- Who manufactured it?
- What is the CPU architecture used? How many bits?
Describe these operating system and computing related terms and concepts shortly:
- What is the difference between kernel and operating system?
- Operating system device driver
- Monolithic kernel
- Microkernel
- Linux kernel module
- x86 privilege rings (protection rings)
- BIOS
- Bootloader
- Library
- System call
- Object file
- Compiler
- Interpreter
- Linker
- Dynamic linker
- Emulator
- TempleOS
- Contiki
- FreeRTOS
- RIOT
- Zephyr
- OpenBSD
- FreeBSD
Study x86 instruction listing and answer:
- What is NOP instruction?
- What is the x86 opcode for NOP instruction?
- What is JMP instruction?
Read this blog post: https://embeddedbits.org/reverse-engineering-router-firmware-with-binwalk/ and answer:
- What is Binwalk?
- What is U-boot?
- What is BusyBox?
- What is the CPU architecture of Linux kernel which was extracted from the firmware?
Install Android-x86 to VirtualBox or VMware
Install and try WSL2 with Windows 10 or 11
Try this ASM code with Linux (source: https://tldp.org/HOWTO/Assembly-HOWTO/hello.html):
section .text ;section declaration
;we must export the entry point to the ELF linker or
global _start ;loader. They conventionally recognize _start as their
;entry point. Use ld -e foo to override the default.
_start:
;write our string to stdout
mov edx,len ;third argument: message length
mov ecx,msg ;second argument: pointer to message to write
mov ebx,1 ;first argument: file handle (stdout)
mov eax,4 ;system call number (sys_write)
int 0x80 ;call kernel
;and exit
mov ebx,0 ;first syscall argument: exit code
mov eax,1 ;system call number (sys_exit)
int 0x80 ;call kernel
section .data ;section declaration
msg db "Hello, world!",0xa ;our dear string
len equ $ - msg ;length of our dear string
Compiling and linking (Ubuntu/Debian):
$ sudo apt install nasm
$ nasm -f elf64 -g -F DWARF helloWorld.asm
$ ld -o helloWorld helloWorld.o
Raspberry Pi
This is not directly course related assignment but just a quick command list how to make Raspberry Pi to work from read-only file system (Rasbian Linux): raspberry_pi_read_only_filesystem_tuning.txt
Some nice defaults and packages for Ubuntu 22.04 and later servers
This Bash script can be used to install some useful packages and change some server default settings. Completely optional.
$ curl -s -L https://tl.oamk.fi/cdos/dl/install.bash | sudo bash
Course week 1
Linux server extra demos, not much explaining but just doing (in Finnish but it shouldn’t be too hard to follow events):
Assignments to the learning diary. (You can do these in small groups. Still, learning diaries are personal):
Week 1-3 assignments may be boring, but are absolutely critical when using and maintaining Linux servers. Without knowing these commands and concepts, it is close to impossible to learn more. Do not skip these basics!
- Install Ubuntu Linux (or whatever Linux distribution you prefer) to VMware, VirtualBox, Hyper-V or KVM (or use Windows 10/11 WSL/WSL2). You can skip this if you are already using personal Linux desktop(s) or server from Oamk
Note: Virtualbox has often compatibility issues with WSL2
- Browse this Linux Command Line Primer
- Describe following commands and concepts:
- man, apropos
- man date
- ls
- ls --help
- date
- date --help
- cd
- cd -
- cd ..
- ls -lat
- ls -s aaaa*
- pwd
- chown
- chmod
- chgrp
- chmod 644 file
- chmod g+x myfile
- which
- rm
- rm -r mydoc
- cp primary secondary
- mv file2 file1
- wc -l myfile
- mkdir mydata
- rmdir mydata
- more, less
- file
- stat
- df
- ln
- which, whereis
- find
- touch
- touch mynewfile
- cp /tmp/test.txt ~/temp/
- What is the difference between Linux kernel and GNU/Linux distribution?
- Name some very common Linux distributions
- What is GPLv2/v3 license? And BSD style license?
- What is (operating system) shell?
- What are case sensitive file names?
- Describe common purpose of files and directories in “/etc”, “/usr/bin” and “/var”
- What is shell PATH? What is the difference between absolute and relative path?
- What is the purpose of tilde character (~) for most Linux shells. For example ls ~/
- How do you recognise a hidden file in any common Unix/Linux file systems?
- What is the meaning of “piping data between commands”?
- What are seti-uid (suid) and set-gid (sgid) bits for file permissions?
- What is a “sticky-bit”?
- Use manual pages and explain what will command “uname -a” do?
- Use manual pages and explain what will command “wc -l” do?
- Linux file handling intro:
- List five largest files in /usr/lib -directory
- Find out what is the group for /bin/ls file?
- How do you change file or directory owner and group?
- How do you change file permissions so that file user has all rights (read, write and execute), group and others have none?
- How do you change file permissions so that file user has read and write access (no execute), group and others have read access?
- How do you change file permissions so that file user, group and others have only read and execute (no write) access?
- Describe following file permissions and ownership:
drwxr-x--- 2 teemu root 4096 Jul 2 2002 webalizer
- Create directory “exercise1” under you home directory
- Create empty file (length 0 bytes) “qwerty.txt” to that directory
- Change directory name “exercise1” to “exer2”?
- Change file qwerty.txt file permissions so that only you have just a read access to it and nothing else
- Create symbolic link to you home directory “this_is_my_link” and make it point to the exer2-directory
- How can you find out your current directory location and path? How far (in directories) are you from file system root?
- Remove files and directories which you created on this exercise
Extra assignments (No need to answer to the learning diary. Ignore for grading):
- Install Apache web server to your Linux server and code some very basic web page with text editor such as Nano, Vim or Emacs
Course week 2
Extra tools:
Assignments to the learning diary (You can do these in small groups. Learning diaries are personal):
Watch some live coding Twitch streams, select one and answer:
- What is the programming language/languages used?
- Which programming libraries and frameworks are being used?
- What is the code editor / IDE being used?
If you have a Windows host:
- Install the Cmder and Winscp
- Try common file commands with Cmder (commands like ls, cd, mkdir, Bash shell etc)
- Use Winscp to move some files between the Windows and Linux hosts
Study and explain shortly following commands and concepts:
- zip, unzip
- tar
- gzip
- xz
- zcat, zgrep
- compress
- bzip2
- 7z
- ldd
- gnu gcc / g++
Install build-essential meta package (containing development tools) to your server with: sudo apt install build-essential
Get the source code for curses-based (“text-graphics”) worm game nibbles-1.2.tar.gz
- Unpack the source package to a some temporary directory under your home directory
- Compile the game and try playing it. Note: Ubuntu does not have ncursed development libraries installed by default. Use apt install to install the missing library dependencies: sudo apt install libncurses-dev
Get the source code for another curses-based (“text-graphics”) Tetris game nct-1.4.tar.gz
- Unpack source package to temporary directory in your home directory.
- Use source package’s configure script to generate Makefile with installation prefix pointing to your home directory
- Compile source code and install compiled files
- Test if game works
- Remove temporary game directory
Download the file harj_zip_paketti.zip. Zip-package has following hierarchy:
paahakemisto
hakemisto_a
karate_kat.jpg
lazy.jpg
hakemisto_b
etherkill.jpg
jap-inv3.jpg
tekstia.txt
With the ZIP file:
- Unpack package and all subdirectories to a temporary directory in your home directory
- Create tar archive from unpacked files and directories and name it to a paketti.tar
- List contents of the paketti.tar. If everything is correct, delete paahakemisto directory and all subdirectories under it. Delete also the harj_zip_paketti.zip file. Don’t delete the paketti.tar -file you just created.
- Unpack only the etherkill.jpg file from tar archive.
- Compress paketti.tar archive with a gzip command.
- What is the size of paketti.tar.gz now?
- Uncompress paketti.tar.gz and compress it again, but now with bzip2. Check the size again. Any difference?
- Create some gzipped tar archive and use SSH (scp) to copy it to the students.oamk.fi (students.oamk.fi is a Linux server you can use with your Oamk user account credentials)
- Delete temporary files and directories created on this practice
Compile this C source code with gcc and check if it works. helloworld.c source code:
#include <stdio.h>
int main(void) {
printf("Hello, world!\n");
return 0;
}
Compile this C++ source code with g++ and test it. helloworld.cpp source code:
#include <iostream>
using namespace std;
int main()
{
cout << "Hello World!\n";
}
With previously compiled helloworld C++ binary:
- What are statically linked libraries? Why would you use them?
- Inspect the size of ready binary file (that compiled helloworld binary). Compile it again and use some different output filename. With g++, use now statically linked libraries (with compiler’s -static parameter). Compare the file sizes of statically and dynamically linked binaries
- Use strace to inspect interiors (system calls) of ls command: “strace ls” and compare the output to a “strace chmod”. Check _exit -values. Why chmod returns 1 and ls returns 0?
- Why and when Unix administrators and programmers use system call tracing programs and debuggers such as gdb and strace?
Solve these service management assignments (Note: most assignments will require root access):
- Check what network adapters your Linux host/server has with command: ip addr or ifconfig (ifconfig is not necessary installed by default)
- Listen inbound ICMP traffic in your server with tcpdump command line protocol analyzer and test if you can see the traffic when you ping your server: tcpdump -n -i YOUR_NETWORK_ADAPTER_NAME_HERE icmp
- Install apache web server with apt install apache2 and test that you can access your server with a web browser
- Listen TCP/80 (web) traffic in your server with tcpdump and test if you can see the inbound TCP SYN segments after you try to access your server with a web browser: tcpdump -n -i YOUR_NETWORK_ADAPTER_NAME_HERE tcp port 80
- Explain what is runlevel?
- Explain what is systemd?
- Explain what are the files in /etc/init.d/ directory?
- Study but don’t do: What is runlevel 6? What is the purpose of init 6 command? How would you do the same with systemd?
Test these service management commands with your web server and use web browser to verify the operation whether the server is running or not:
Somewhat bad or at least old way of some or older distros:
service apache2 stop
service apache2 start
service apache2 restart
Test these service management commands with your web server and use web browser to verify the operation whether the server is running or not:
Generic System V style, also old school:
/etc/init.d/apache2 stop
/etc/init.d/apache2 start
/etc/init.d/apache2 restart
Test these service management commands with your web server and use web browser to verify the operation whether the server is running or not. Try and explain:
Modern way for distros using systemd:
journalctl | tail -20
systemctl restart apache2
journalctl | tail -20
systemctl stop apache2
systemctl start apache2
systemctl
Check Apache access.log file contents in /var/log/apache2/ directory. Can you find your connections to the web server?
Course week 3
Assignments to the learning diary (You can do these in small groups. Learning diaries are personal):
Study and explain shortly following commands and concepts:
- sh, bash, zsh
- screen and tmux
- ps, pgrep, pstree, pidof
- jobs, disown
- fg, bg
- top, htop
- nice, renice
- su, sudo
- sleep
- xargs
- nohup
- kill
- pkill, killall
- w, who
- write, wall
- aliases
- source, .bashrc
- shell build-in variables, export
How and when you start new shells? How to exit a shell?
Add shell alias “diskusage” to your shell startup-files (example .bashrc). Alias should print only current disk usage of your home directory
Create shell alias “pp” which requires one parameter and will print all running processes including details with that name. Usage example:
tkorpela$ pp sleep
root 21109 0.0 0.1 4084 556 pts/8 S 20:02 0:00 sleep 100
root 21111 0.0 0.1 4084 556 pts/8 S 20:03 0:00 sleep 100
root 21113 0.0 0.1 3684 556 pts/8 S 20:03 0:00 grep sleep
Which directories are currently in you PATH variable?
How do you start process directly into background when entering a command?
Start few sleep 60 processes (one minute idle loop) to the background and:
- How can you find and terminate them all with one-liner? Try not to use pkill, killall or xargs -commands.
- How would you do the previous killing task with xargs?
- Start one 1000 second sleep to the foreground.
- How do you suspend it?
- How do you list current jobs?
- How do you get previous sleep process back to foreground?
- Suspend process again and send it to background.
- Kill previous sleep process from background.
What is the difference between kill -9 and kill -1?
Delete unnecessary files created in this practice.
Extra assignments (No need to answer to the learning diary. Ignore for grading):
- Firewall assignments as root user:
- Download this simple firewall example to your server and rename it to firewall.bash. Move that file to the /etc directory and set permissions to 700
- Study the contents and logic of the script
- Run the script as a root user and use command iptables -L -n -v (as root) to see the packet counters
- Use # to comment out the line allowing inbound TCP/80 traffic and rerun the firewall script
- Enable logging lines in your firewall script and try to access your web server again (you shouldn't be able to access the web service anymore with web browser) and use tail -f /var/log/kern.log to follow linux kernel log file
- Enable inbound TCP/80 again by removing the comment. Check that you can access your web server again with browser
- Try to ping (IPv4) your server. It should not answer when the host firewall is now denying the new and unknown traffic. Modify your firewall script to allow inbound ICMP protocol traffic from all IPv4 addresses by adding a command iptables -A INPUT -p icmp -j ACCEPT after the HTTP allowed line. Rerun your firewall script. Try to ping your server again.
- Optional task (to run your firewall script every time when the server reboots):
- Study what is /etc/rc.local file. Edit the /etc/rc.local file with a nano text editor and add your firewall script into the startup script
- Alternatively, try to write a systemd service file to automatically start the script during server startup
Course week 4
Assignments to the learning diary (You can do these in small groups. Learning diaries are personal):
Study and explain shortly following commands and concepts:
- cat, tac
- grep / egrep
- wc
- sort
- cut
- awk
- sed
- tr
- expand, unexpand
- uniq
- head
- tail
- echo
- column
- fold
- join
- paste
- tee
- nl
Use word counter and piping to count how many files or directories are in /usr/bin -directory?
Use grep and extended regular expression syntax to list all files from /etc directory recursively which have IPv4 addresses mentioned inside
Download and extract Tetris game source file nct-1.4.tar.gz and
- Use grep to find which files contain string ncurses.h
- Use wc command to list line counts of each file and sort the output from longest to shortest file. Use data filtering command to remove the total amount of lines line in the beginning of sort output. Final result should be something like this:
2139 configure
943 nct.c
392 Makefile.in
340 COPYING
251 install-sh
195 score.c
190 missing
182 INSTALL
127 aclocal.m4
48 README
44 config.h.in
43 Makefile.am
43 configure.in
42 score.h
40 mkinstalldirs
38 nct.spec
22 NEWS
21 nct.h
17 nct.lsm
3 ChangeLog
2 acconfig.h
1 stamp-h.in
Use wget to download this irclog.txt and answers to these questions:
- How many lines are in the file?
- How many characters are in the file?
- List only lines where the timestamp starts with 05 and save the output to a file called result.txt
- Print result.txt in reverse order
- Create numerical statistics from the irclog.txt file: How many lines each nickname wrote. Use only those lines where someone actually said something and ignore the all other lines. Output should be something like this:
44 ryan_
40 ryan||
34 ryann
15 AlexC_
12 scottymeuk
12 Ruchira
11 ryannn
11 kyhwana
11 gerryvdm_mbp
10 shmoon
7 chesty
6 Ruchira_
3 ssthormess
2 rww
2 mestri
2 drclawski
1 ryan|
1 mikegrb
1 gkmngrgn
1 akerl
List only 5 largest files from /usr/bin -directory.
- Print largest files first
- Try to not use the ls command's -S option but use use sort command (and related text processing commands if necessary)
Print only usernames, UID and GID numbers from /etc/passwd -file. Replace all colons with a whitespace. Redirect output to file a “users.txt” in your home directory. Tip: In this example line from /etc/passwd the UID = 101 and GID = 50:
username:x:101:50:Teemu Korpela:/home/tkorpela:/bin/bash
Use text editor nano to create a points.txt file to your home directory with following content. This list presents first names and some game scores. Who has most points, wins
- List contents of points.txt in alphabetic order to STDOUT
- List contens of file on to STDOUT, but now order is score based. List only best three players with most points
- How do you list only player names and filter all other data
- List only first three characters from the beginning of each line of points.txt
- List points.txt but translate all characters to upper-case
- List points.txt so that points are printed before names
- Sort points.txt in alphabetic order and add line numbers in front of lines
points.txt file:
Teemu:4
Matti:8
Juha-Pekka:6
Timo:1
Mika:3
Esko:2
Jaska:5
Erkki:7
How do you list last 5 lines from the /etc/passwd file?
How do you list first 5 lines from the /etc/passwd file?
What does tail -f filename command do?
Fetch current weather in Oulu with lynx (TIP: if there is no lynx, install it with: sudo apt install lynx). The command to download Oulu's weather data is: lynx -dump http://weather.willab.fi/weather.html
- Filter the output so that only temperature is displayed and nothing else
Use wget to download this stock market textfile
Example line and explanation from file:
Name code change buy sell lowest highest last
Fiskars Corporation :FISAS: -0,36% 8,35 8,39 8,44 8,37 8,37
- Use grep (or egrep) and regular expressions to list only companies with “I” anywhere in in code part.
- List (only) company names and stock values starting with character “M”.
Output should be:
Metso Corporation :MEO1V: -0,08% 11,77 11,79 11,80 11,73 11,79
M-real Corporation A :MRLAV: -0,42% 4,64 4,83 4,75 4,75 4,75
M-real Corporation B :MRLBV: -1,06% 4,65 4,67 4,75 4,64 4,67
- Print line only if the company name begins with a character “R” and last stock value is 8,xx
Output should be:
Rautaruukki Corporation :RTRKS: +0,24% 8,46 8,50 8,52 8,42
Rocla Oyj :ROC1V: -0,60% 8,20 8,25 8,25 8,20 8,25
- List all companies except the names starting with characters “R” or “W”
- List only those stocks which have positive change value (i.e. +xx,xx%) in the list
Get nimipaivat.txt (Finnish name days) textfile from here nimipaivat.txt
- From nimipaivat.txt, find out how many names start with a letter A and end to a letter i?
- How can you convert previous names to lower-case?
- From previous names, who are celebrating in December?
- From all names in nimipaivat.txt, search those who celebrate either 1st, 2nd or 3rd day in any month.
Use lynx -dump “url” to print webpage to STDOUT
HTTP access to XML:
- Use Gnu tools or Cmder’s Curl and Grep (and maybe other command line tools) to create a one-liner, which downloads the XML file and parses current temperature from VTT’s weather station. One-liner must print only the current temperature in Oulu and nothing else. Command line one-liner and output should look something like this:
curl -s -L http://weather.willab.fi/weather.xml | _replace_with_your_commands_options_and_code_
12.3
Combine these two files to a single file with command line Gnu text tools
- The first file has timestamps and the second file has IP addresses
- Use : as delimeter between columns. Output should look something like this:
...
Sat Apr 11 11:03:42 2020:185.176.27.26
Sat Apr 11 11:03:43 2020:188.26.0.66
Sat Apr 11 11:04:15 2020:185.176.27.34
Sat Apr 11 11:04:57 2020:87.251.74.250
Sat Apr 11 11:05:00 2020:94.102.52.57
...
Delete unnecessary files created in this practice
Course week 5
Assignments to the learning diary (You can do these in small groups. Learning diaries are personal):
Study this Telegram bot game version 0.1, this improved version 0.2 and even more improved version 0.3
- Compare all three scripts
- Download the 3rd script (version 0.3) and related word list files (Github addresses are listed in the beginning of the script) with wget and run the script
- Modify the script to output all characters UPPER CASE
- Modify the script to add zero padded line numbers into the beginning of each line
- Note: No need to setup a Telegram account or bot but just print the words to STDOUT. Telegram API is commented out in the script and no need to change that
Output should look like this (that 11 is the word length):
$ ./telegram_wordgame_v3.bash 11
01 _____M_____
02 ___O_M_____
03 ___ORM_____
04 A__ORM_____
05 A__ORM__I__
06 A__ORM__I_Y
07 A__ORM_LI_Y
08 A_NORM_LI_Y
09 A_NORM_LITY
10 ABNORM_LITY
11 ABNORMALITY
Study and try this simple incremental / full backup example script. Do automatic backups as a root user:
- Use wget to download the backup script example
- Move the downloaded file to /etc/cron.daily/ and rename it to backup (cron does not like if filename end to .bash. leave it out)
- Set permissions to 700 (and make sure root user is the file owner)
- Run the backup script from command line and check that it worked
- Uncomment initial delay lines from the script to create a random delay before the backup script executes all those heavy disk IO (find and tar) backup operations
- Create some temporary test directory to /tmp and copy one of those smaller incremental backup files there
- Use tar to list contents of the copied package
- Use tar to decompress package and check that you managed to extract all the files from the package
- Remember that your virtual server host server is not taking any backups. It is up to you to backup and transfer your files if servers go down!
- Test how to pipe tar archived and compressed file through ssh connection to the students.oamk.fi server (students.oamk.fi accepts your student.oamk.fi credentials for SSH)
Create a file size comparison Bash script:
- It will prompt user to enter two filenames and then compares if the first given file is smaller, greater or equal size to the second file
- Before comparison, the script must check if given files exist
- If the file does not exist, script will print an error message
- Use Bash if expressions to check that file exists and to test which file was larger
- Use “read” to ask filenames interactively
Example output:
$ comparemyfiles
Enter name of first file:
something
Enter name of second file:
thisfiletoo
File something is larger.
Change previous script to accept two filenames as a command line parameters. Example output should be something like this:
$ comparemyfiles something thisfiletoo
File something is larger.
- Also, script must print an error message and exit if it does not receive two parameters (the filenames) from command line
Think why second solution (to use parameters after the command instead of interactive input) is usually more practical solution?
Create “Rock Scissor Paper”-game with Bash
- Script will prompt user to pick either Rock, Scissors or Paper
- Script will randomise one option (computer player’s selection) and return results
- Rules are: Rock wins scissors. Paper wins rock. Scissors wins paper
- Tip: Bash build-in $RANDOM variable returns random numbers. For example, numbers 0-9 would be:
echo $[RANDOM%10]
More advanced PRNG using /dev/urandom:
echo $[$(od -vAn -N2 -tu4 < /dev/urandom)%10]
Or with shuf:
shuf -i 1-10000 -n 1
Create a welcoming script which will check current time and will echo welcome message if time is:
06:xx-11:59 = Good morning user's_realname_from_/etc/passwd-file
12:xx-17:xx = Good day user's_realname_from_/etc/passwd-file
18:xx-21:xx = Good evening user's_realname_from_/etc/passwd-file
22:xx-05:xx = Good night user's_realname_from_/etc/passwd-file
Example output (time 12:43):
$ welcomescript.bash
Good day Teemu Korpela
Create a script which will check current date and searches all persons having finnish name day at the moment. You can use nimipaivat.txt as a datafile
Example output:
$ date
Mon Dec 2 17:42:21 EET 2021
$ ./nameday.bash
Anelma 2.12.
Unelma 2.12.
Install SQLite3 database engine and tools to the Linux server (basic tutorial here) and:
- Download nimipaivat.txt text file
- Create a new SQLite database
- Create a Bash script (or use Bash one-liner) which inserts only first names but not dates from the nimipaivat.txt to the SQLite database as a single column data
- Create a Bash script which will fetch one single random name from the SQLite database
- Basic SQLite usage examples
Example output:
./fetch_random_name.bash
Liisa
./fetch_random_name.bash
Tiina
Extra assignments (No need to answer to the learning diary. Ignore for grading):
Study the operation and try this Bash script fetching electricity spot prices from Nordpool's website
Repeat the previous SQLite assignment. But this time, add names and dates to the SQLite database
Use separate database table columns for the name and date
Create a Bash script to fetch a name from the SQLite database for a date specified by command line argument
Try to customise your nano text editor settings either by creating ~/.nanorc or system wide configuration in /etc/nanorc. Example:
# See: https://www.nano-editor.org/dist/latest/nanorc.5.html
set titlecolor bold,white,blue
set minicolor bold,white,blue
set promptcolor lightwhite,red
set statuscolor bold,white,red
set errorcolor bold,white,red
set spotlightcolor black,lightyellow
set selectedcolor lightwhite,magenta
set stripecolor ,yellow
set scrollercolor cyan
set numbercolor cyan
set keycolor cyan
set functioncolor green
#set indicator
#set autoindent
#set nonewlines
#set cutfromcursor
#set linenumbers
#set wordchars "<_>."
set softwrap
set atblanks
set historylog
set tabsize 4
set tabstospaces
set regexp
set locking
set constantshow
set nohelp
set boldtext
set afterends
set minibar
set stateflags
set zap
set locking
set boldtext
bind ^Z suspend main
Course week 6
Assignments to the learning diary (You can do these in small groups. Learning diaries are personal):
Install Apache web server to your Linux server if it isn’t installed already
- Ignore/skip possible ufw (firewall) parts. Ufw has been disabled intentionally on student virtual servers
- Add PHP support to your Apache web server. See the MySQL example with PHP for required software packages
- Create following PHP script under the web server document root (/var/www/html) and test that your server is executing the PHP script when requesting it with a web browser (address is: http://server/scriptname.php):
<?php
date_default_timezone_set('UTC');
echo date("l");
echo ("<br>");
echo date('l jS \of F Y h:i:s A');
echo ("<br>");
echo date(DATE_RFC2822);
?>
Modify the Apache web server configuration to redirect HTTP GET requests to the directory /weather/ to https://wttr.in/. You will most likely need to enable the redirect module for Apache
- So, visiting your server http://IP_or_DNSname/weather/ takes the browser to wttr.in. Check with web browser that the redirect works as intended
Check last entries in Apache access and error log files in /var/log/apache2/
Install MySQL server and create some basic database there with one or more tables and insert some data into the table(s). See the basic MySQL example with PHP
- Note: Do not use that mysql_secure_installation command I used in the lecture recording. mysql_secure_installation command is now obsolete. You can access MySQL root shell with Linux root user privileged like this:
$ sudo mysql
Create a Bash script which will fetch and process data from marine traffic API
- Script must download the JSON-file from marine traffic port call API and print how many ships are currently there? (Search vesselName from the JSON)
- Command line example with curl to get things started:
curl --compressed -L https://meri.digitraffic.fi/api/v1/port-calls -o /tmp/result.json
- Filter the /tmp/result.json file data with jq or with GNU text utilities such as sed, awk, cut, grep etc. to only show how many ships are currently there? (Search vesselName from the JSON)
Create a Bash script which checks spaceX launch schedule API and tells how many days ago was the last launch. Result should be something like this:
ubuntu@linux100:/tmp$ ./whenwasthelaunch.bash
4 days ago
- Tip: Use unix epoch timestamp from the API reply and current Unix epoch time from date command to calculate and show the time difference. date +%s outputs the current unix epoch time
Create this Bash script to /usr/local/bin directory. Name it to justtimestamps.bash and set the file permissions to 700 (root user has full access, others none). Check that the root is the file owner:
#!/bin/bash
while true; do # this is endless loop
date >> /root/timestamps.txt # writing timestamp to a file
sleep 60 # 60 second delay
done
- Create a new systemd service configuration file timestampdemo.service to the /etc/systemd/system directory with contents:
[Unit]
Description=My automatic service writing timestamps to a file in /tmp
After=network.target
[Service]
Type=simple
User=root
Group=root
ExecStart=/usr/local/bin/justtimestamps.bash
WorkingDirectory=/tmp
Restart=on-failure
[Install]
WantedBy=multi-user.target
- Manage the service:
- Reload service files with: systemctl daemon-reload
- Start the service with: systemctl start timestampdemo
- Check that /root/timestamp.txt file was created with the timestamp content (that date command should be running once per minute)
- Check the process list to verify that the script is running
- Use command: systemctl status timestampdemo to check the service status
- Stop the script with: systemctl stop timestampdemo
- Verify from process list that service is not running anymore
- Use systemctl status timestampdemo to check the service status
- Try running systemctl enable timestampdemo and systemctl disable timestampdemo (enables/disables the service during the server startup)
Extra assignments (No need to answer to the learning diary. Ignore for grading):
- Do this Node.js and SQLite tutorial
- Install NodeJS and some basic “hello world” web app to your server. Configure Apache to reverse proxy the traffic to your NodeJS application TCP port
- Write a Telegram bot which uses some network API to fetch information and processes and posts that data to a Telegram channel or account via Telegram API
- Write a CLI tool to fetch few random movie names from TMDB API
Course week 7
- Lecture (13.11.2022): (English) About containers and IaC
- Lecture (11.10.2022): (Finnish) About containers and IaC
- Book: Ansible: Up and Running, 3rd Edition
- Book: Terraform: Up and Running, 3rd Edition
- Book: Core Kubernetes
- Book: Rancher Deep Dive
- Book: Container Security
- Article: Open Container Initiative
- Article: Serverless computing vs. containers
- Article: How To Install and Use Docker on Ubuntu 22.04
- Article: Chroot vs Docker
- Article: Docker Swarm
- Article: Rancher
- Article: Kubernetes
- Article: Apache Mesos
- Article: What even is a container: namespaces and cgroups
- Article: Notes on running containers with bubblewrap
- Article: With Earthly, all builds are containerized, repeatable, and language agnostic
- Article: ContainerSSH
- Article: Rook, Open-Source, Cloud-Native Storage for Kubernetes
- Article: Falco, the cloud-native runtime security project, is the de facto Kubernetes threat detection engine
- Article: Cilium is an open source software for providing, securing and observing network connectivity between container workloads
- Article: Kubernetes using minikube and Katacoda
- Article: WSL now has built in support for systemd (read: also easier Linux containers)
- Article: Robot Framework
- Article: Truly headless docker-Selenium-Robot Framework -combination
- Article: Create a Docker image running Robot Framework
Assignments to the learning diary (You can do these in small groups. Learning diaries are personal):
- What is a Linux container? (with any technology such as LXC/LXD, Docker, Podman). Generic description what is a Linux container is enough.
- What is the difference between chroot, containers and full operating system virtualization (like Virtualbox, VMware, Hyper-V etc.)?
- What is Kubernetes?
- What is Ansible?
- What is Terraform?
- Install Docker engine (docker.io) package to your server from standard package repository or from the Docker official repository
- Use docker CLI command to search package searx with most stars
- Use docker CLI to pull searx/searx Docker container image from Docker repository (Searx is a self-hosted metasearch engine, combining multiple web search engine results)
- List local Docker images with docker images command. What is the image id of Searx?
- Start the Searx container and redirect inbound TCP 8080 port traffic to the Searx container:
docker run --rm -d -v /opt/docker/searx/searx:/etc/searx -p 44444:8080 -e BASE_URL=http://localhost:44444/ searx/searx
- Use docker ps to verify that Searx container is running
- Verify that you can access your container with your web browser: http://your_server_ip:44444/ (open TCP/44444 port in your host firewall if you are filtering traffic with Netfilter)
- Check container stats with interactive docker stats command
- Check Searx default settings in /opt/docker/searx/searx/settings.yml file. See https://docs.docker.com/engine/swarm/configs/ and variables like BASE_URL
- Use docker exec to list and explore Searx container filesystem and files. Note: Searx container does not have bash shell but Busybox has SH shell (From CLI: docker exec -t -i your_containerID sh)
- Stop the container and verify that is has been stopped. Remove the container image if using it anymore
- Browse these container building tutorials: Create a Docker image running Robot Framework and Docker Build: A Beginner’s Guide to Building Docker Images
- Install Ansible to the Linux server and to your workstation (or some another Linux workstation/server) and:
- Note: Following assignments are most likely easier if you use two Linux servers (for example, together with someone from your study group) and use Ansible between the servers
- Explain shortly what are Ansible playbooks?
- Define passwordless login for SSH (from client to the Linux server). Do not generate and use now obsolete RSA SHA-1 but generate and use ed25519 keys. You can use OpenSSH ssh-keygen.exe command (if Windows client) or ssh-keygen command (if Linux client) to generate the ed25519 public and private key files. Command example: ssh-keygen -t ed25519
- Test that you can use OpenSSH CLI client to access the server without password from the client
- Generic Ansible installation instructions for Ubuntu -> Ubuntu control: https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-ansible-on-ubuntu-22-04
- Define the server IP and default username for SSH to your client Ansible configuration. If your Ansible client is Ubuntu server, the default configuration file is (you need to create it): /etc/ansible/hosts
- Test that you can ping your Ansible server from the client with Ansible build-in ping module
- Try to run some commands remotely from your workstation with Ansible CLI tool from your client host. For example: id, ls /, touch /tmp/mytestfile etc.
- Explain shortly what is Robot Framework
- Install pip3 Python installer to the Linux server and install Robot Framework with pip (Note: Virtual environments are usually a very good idea with Python libraries. This tutorial doesn’t use virtualenv)
- Do this Robot Framework tutorial demonstrating acceptance tests
Extra assignments (No need to answer to the learning diary. Ignore for grading):
- Configure your Apache web server to reverse proxy HTTP traffic to your Searx Docker container
- Try WSL2 with Windows OS. Ubuntu is popular option
- Study (or try) how to run Linux containers with Windows OS (somewhat unusual but possible)
Course week 8
Topics:
- Mandatory peer reviews with Moodle
- Finishing the course
This page was modified: 2023-02-05