Trick grep / egrep to always display filename

If you use grep / egrep on a single file, it will display the matched patterns, but not the filename nor the line number.

This can be boring in case you encapsulate a grep / egrep call inside i.e a find:

find ./mydir -type f -name '*.msg' -exec grep 'mysearch' {} \;

In that case it will only print the match and you will not know in which file.

The trick is to add a dummy filename as a second file to search. Doing so will trick grep / egrep into thinking you are searching in multiples files, thus displaying the filename on each match.
/dev/null is the perfect target for the second dummy file as it will never match anything.

Example:

find ./mydir -type f -name ‘*.msg’ -exec grep ‘mysearch’ {} /dev/null \;

Bonus / reminder:

In grep / egrep use the -n option to also display the match line number

Example:

find ./mydir -type f -name ‘*.msg’ -exec grep -n ‘mysearch’ {} /dev/null \;

#sysadmin #tips #tricks #grep #egrep

KAFKA

KAFKA is in all mouths these days. So I share here, also as a reminder for me, a couple of #KAFKA ressources that looked helpful to me.

TLDR:

Do not use KAFKA for:

-“Little” Data flows (overkill)

-Streaming ETL (handling transformation is a hassle)

-Store and process large files (images, videos, proprietary files, etc.)

https://www.kai-waehner.de/blog/2020/08/07/apache-kafka-handling-large-messages-and-files-for-image-video-audio-processing/

https://www.upsolver.com/blog/apache-kafka-use-cases-when-to-use-not

https://kafka.apache.org/intro

https://docs.confluent.io/kafka-clients/librdkafka/current/overview.html

https://docs.confluent.io/platform/current/clients/librdkafka/html/md_INTRODUCTION.html

Compare two json in shell

I did it using the JQ command line tool from https://stedolan.github.io/jq/

Exemple: list all json files from current directory and print the difference with updated jsons from updated/ directory

for user in  `ls updated-users`
do
    # print file name
    echo $user
    # simple 
    diff <(jq -S . users/$user) <(jq -S . updated-users/$user)
    # or full on one side and the diff on the other side
    # diff -y --left-column <(jq -S . users/$user) <(jq -S . updated-users/$user)
    # or full on one side and the diff on the other side, colored
    # diff -y --left-column --color  <(jq -S . users/$user) <(jq -S . updated-users/$user)
done

#shell #diff #json #script

Adding proxy to MSYS2

If you ever need to use a proxy when updating / installing packages in msys2 you’ll have to set the following environnement variables, and put them i.e in your .bash_profile :

# .bash_profile example
# Note: username and password have to be url encoded in case they contain special chars
export http_proxy=http://USERNAME:PASSWORD@proxy:port
# or like this if not user/password required
# export http_proxy=http://proxy:port
export https_proxy=$http_proxy
export ftp_proxy=$http_proxy
export rsync_proxy=$http_proxy
# if you need a proxy ignore list
export no_proxy="localhost,127.0.0.1,localaddr,.yourlocaldomain.ext,.local"

CYGWIN: list all process arguments

A few days ago I had to list the arguments for specific process.
While finding it using “ps -aef | grep ‘processname'” and getting the information is trivial under linux, it is not the case using cygwin, which is only reporting process name and pid when calling “ps”.

A lot of solutions involve installing cygwin packet, like pstree.

Another solution is to search the /proc/pid/cmdline files, like here:

grep -a /proc/*/cmdline | xargs -0
/proc/1990/cmdline:/bin/sh /usr/bin/startxwin
/proc/2016/cmdline:xinit /etc/X11/xinit/startxwinrc — /usr/bin/XWin :0 -multiwindow -auth /home/user/.serverauth.1990
/proc/2017/cmdline:/usr/bin/XWin :0 -multiwindow -auth /home/user/.serverauth.1990
/proc/2023/cmdline:/usr/bin/xwin-xdg-menu
/proc/2035/cmdline:dbus-launch –sh-syntax –exit-with-session
/proc/2036/cmdline:/usr/bin/dbus-daemon –fork –print-pid 5 –print-address 7 –session
/proc/2046/cmdline:/usr/libexec/gam_server
/proc/2512/cmdline:mintty
/proc/2513/cmdline:bash
/proc/3441/cmdline:grep –color -a /proc/1990/cmdline /proc/2016/cmdline /proc/2017/cmdline /proc/2023/cmdline /proc/2035/cmdline /proc/2036/cmdline /proc/2046/cmdline /proc/2512/cmdline /proc/2513/cmdline /proc/3441/cmdline /proc/self/cmdline
/proc/self/cmdline:grep –color -a /proc/1990/cmdline /proc/2016/cmdline /proc/2017/cmdline /proc/2023/cmdline /proc/2035/cmdline /proc/2036/cmdline /proc/2046/cmdline /proc/2512/cmdline /proc/2513/cmdline /proc/3441/cmdline /proc/self/cmdline

To restrain it to a particular process i.e XWin, excluding grep command:

grep -a “XWin” /proc/*/cmdline |xargs -0 |grep -v grep
/proc/2016/cmdline:xinit /etc/X11/xinit/startxwinrc — /usr/bin/XWin :0 -multiwindow -auth /home/user/.serverauth.1990
/proc/2017/cmdline:/usr/bin/XWin :0 -multiwindow -auth /home/user/.serverauth.1990

Cargo Cult Sytem Administrator

And some other types too.

Fun article to read here: https://blog.lastinfirstout.net/2009/11/cargo-cult-system-administration.html

One particularly right: “Asserting that [Technology O] or [Platform L] or [Methodology A] is inherently superior to all others and blindly applying it to all problems. When you make such claims, are you applying science or religion?”

“It’s easy to fall into cargo cult mode.
Just re-boot it, it’ll be fine.” – Michael Janke