Command Notes for Script

These are command notes for scripting in linux.

Removing Duplicate Lines

Removing duplicate lines can be done by sord and uniq command.

$ sort textfile | uniq

Above approach is sorting first and filter unique line. The problem is that uniq command only check next line is identical to current line and skip if so. Therefore, sorting first is required. What if sorting is not desired? In case of that, awk command can be used. The clever part is '!_[$0]++' in below command.

$ awk '!_[$0]++' textfile

Using Variable in awk and sed

Using variable in awk command line is different from others. In below command -v VAR=apple '$0 ~ VAR' is key.

$ awk -v VAR=apple '$0 ~ VAR' textfile

It will print all line including apple that matched to VAR by awk. The option -v means assigning variable and $0 means the whole line. You can print specific field like $3 or $1, etc.

But using variable in sed command is different. Using double quotes and curly brackets like "s/${VAR}/replace/" is key.

$ VAR=apple
$ sed "s/${VAR}/orange/" textfile

It will replace apple with orange. If it is to replace all apples in line use g at the end of substitution statement like "s/${VAR}/orange/g".

Removing Duplicate Blank Lines

Removing duplicate blank lines by using sed command is a little tricky. The key part is '/^$/N;/^\n$/D'.

$ sed '/^$/N;/^\n$/D' textfile

Explanation: /^$/ is regular expression to match empty line and N appends new line to the matched line. And /^\n$/ regular expression matches lines containing only a newline character, which can happen after combination lines with N when there are consecutive empty lines, then D deletes the first line in the pattern space. If more content remains in the pattern space, it starts processing again from the top. This looping behavior continues until the pattern space is empty.

Comments

Post a Comment

Popular posts from this blog

printf - Printing with format

Image
printf formats and prints its arguments according to format, a character string which contains three types of objects: plain characters, which are simply copied to standard output, character escape sequences which are converted and copied to the standard output, and format specifications, each of which causes printing of the next successive value.   To Get Zero Leading Number $ printf "%03d" 1 $ 001   To Print Line by Line Including Speces in Terminal Interactive script in terminal sometime requires to clear terminal's face and print line by line without flickering the terminal's face. In that case, using printf "%-*.*s%s\n" "$MINCOL" "$MAXCOL" "string of the line" is a trick. For breaking down, %-*.*s is for width i.e., minimum number of columns and maximum number of columns. In above example, "$MINCOL" and "$MAXCOL" are variables of minimum and maximum...
Read more

rebase or no-rebase for git pull

Image
In the following scenario, the safest option is using git pull --rebase or setting configuration by git config pull.rebase true . Analysis of the scenario Current state on Remote : File aaaaa.txt exits and file bbbbb.txt does not exist yet. Current state on Local : File aaaaa.txt does not exist and file bbbbb.txt has been created. In above scenario and local repository on the computer, git push will fail with the following message: To https://codeberg.org/minsoehan/ttmy.git ! [rejected] main -> main (fetch first) error: failed to push some refs to 'https://codeberg.org/minsoehan/ttmy.git' hint: Updates were rejected because the remote contains work that you do not hint: have locally. This is usually caused by another repository pushing to hint: the same ref. If you want to integrate the remote changes, use hint: 'git pull' before pushing again. hint: See the 'Note about fast-forwards' in 'git push -...
Read more

dmmsh-wlr for managing outputs in Wayland using wlr-randr

Managing outputs in Wayland is smooth and easy by using wlr-randr while some compositors such as sway and hyprland have their own tools to manage outputs. See: sway-output(5) man page and Monitors - Hyprland Wiki respectively. To automate managing the outputs, the following dash script is provided by mshii. #!/bin/dash # check if it is run as root and exit if [ $(($(id -u))) -eq 0 ]; then exit 0 fi # test if it is already running and exit SCRIPTNAME= " $(basename -- $0) " if pidof -x " $SCRIPTNAME " -o $$ > /dev/null 2>&1; then notify-send -t 2700 "clean exit" "the script is already running" exit 0 fi # chech if it is in terminal or exit isinterminal_func () { case " $(ps -o stat= -p $$) " in *+*) echo "in terminal, continue..." ;; *) notify-send -t 2700 "clean exit" "please run it in terminal." ; exit 0 ;; esac } # simp...
Read more