KAFKA

KAFKA is in all mouths these days. So I share here, also as a reminder for me, a couple of #KAFKA ressources that looked helpful to me.

TLDR:

Do not use KAFKA for:

-“Little” Data flows (overkill)

-Streaming ETL (handling transformation is a hassle)

-Store and process large files (images, videos, proprietary files, etc.)

https://www.kai-waehner.de/blog/2020/08/07/apache-kafka-handling-large-messages-and-files-for-image-video-audio-processing/

https://www.upsolver.com/blog/apache-kafka-use-cases-when-to-use-not

https://kafka.apache.org/intro

https://docs.confluent.io/kafka-clients/librdkafka/current/overview.html

https://docs.confluent.io/platform/current/clients/librdkafka/html/md_INTRODUCTION.html

Why gettimeofday() is bad at timing things

I was reading a changelog from a library I love (unrelated but here is the link: https://liballeg.org/ ) and I stumble upon that change:


“Use clock_gettime with CLOCK_MONOTONIC instead of gettimeofday (check-switch-26)”

Then I did a bit of research on the topic, and found a good explanation here:

https://blog.habets.se/2010/09/gettimeofday-should-never-be-used-to-measure-time.html

TLDR: if you use gettimeofday to time things then your program may be affected by time shift, because gettimeofday is not monotonic. if you do not care about the date and only about elapsed time, use clock_gettime.

HackRF Portapack Mayhem Firmware, Search/Freqman evolution branch

On top of the existing ‘search and stay X seconds after a match’ the SearchApp have been updated with a ‘search and stay’ mode: stay while matching, leave after X seconds of inactivity, reset counters on activity during the wait. mode,
To use it just specify a negative value in the ‘wait’ field using the rotary encoder.

As some have some problems getting the bin back from discord, here you are:

The documentation is on the wiki, here:

https://github.com/GullCode/portapack-mayhem/wiki/Search

Github page:

https://github.com/GullCode/portapack-mayhem

grep: retain colors when piping

By default grep -color is not producing color symbols if it detects that the output is not a terminal.

grep -color=auto is doing the same, that is not producing color symbols if it detects that the output is not a terminal.

The solution is to use “grep -color=always“ in the place that need it.

Do not put it in an alias of grep as it would break some code somewhere else. Color symbols are strings like “ESC[35m” and they will be inserted in the text.

Portapack firmware with Search App and updated Freqman files (based on latest next )

  • I’ve build an up to date firmware with the last updates from next and the last version from search app and freqman enhancements.
  • The Search app is in the ‘receive’ menu
  • The documentation is here : https://github.com/GullCode/portapack-mayhem/wiki/Search
  • -Freqman now do understand freq files with that format:
    f=468000000 f=468000000,d=Single Freq f=468000000,m=AM,d=Single Freq AM f=468000000,m=NFM,d=Single Freq NFM f=468000000,m=WFM,d=Single Freq WFM f=468000000,m=AM,b=DSB,d=Single Freq AM DSB f=468000000,m=AM,b=USB,d=Single Freq AM USB f=468000000,m=AM,b=LSB,d=Single Freq AM LSB a=87000000,b=110000000 a=87000000,b=110000000,m=AM,s=100KHz,d=AM radio search a=87000000,b=110000000,m=AM,b=DSB,s=250KHz,d=AM radio search LSB a=87000000,b=110000000,m=WFM,b=16k,s=50KHz,d=WFM radio search s=50KHz r=430150000,t=430550000 r=430150000,t=430550000,d=HAM radio r=430150000,t=430550000,m=AM,b=DSB,d=HAM radio
  • The freqman GUI have note been improved and may show partial results on new formatted lines
  • Description of the fields:
    f=freq for one frequency or a=start_frequency,b=end_frequency for a range
    m=modulation
    b=bandwidth
    s=step
    d=description
  • All fields except ‘f=freq’ or ‘a=freqA,b=freqB’ are mandatory. If nothing specified actual value is used.
  • As a reminder :
    -Most of the time if the Search app is not working as you expected it’s coming from a SDCARD problem.
    -You need a SDCARD for the Search app to save settings between runs and between settings menu / main gui, and you need a SEARCH folder at the root of it.
    -Don’t forget to check that by default the ‘input: load’ fields in ‘search app -> params -> more ‘ are all checked.
    -You HAVE to click save in ordre to save the settings.

GitLab: WARNING: gl-sast-report.json: no matching files

When enabling SAST in a gitlab project you can have that kind of error at the tests part:

The yaml provided is not containing the needed part:

artifacts:
   name: sast
   paths:
     - gl-sast-report.json
   reports:
     sast: gl-sast-report.json
   when: always

Source:

https://forum.gitlab.com/t/uploading-artifacts-warning-gl-sast-report-json-no-matching-files-error-no-files-to-upload/60887/5

https://gitlab.com/gitlab-org/gitlab/-/issues/345696#note_737150614

Compare two json in shell

I did it using the JQ command line tool from https://stedolan.github.io/jq/

Exemple: list all json files from current directory and print the difference with updated jsons from updated/ directory

for user in  `ls updated-users`
do
    # print file name
    echo $user
    # simple 
    diff <(jq -S . users/$user) <(jq -S . updated-users/$user)
    # or full on one side and the diff on the other side
    # diff -y --left-column <(jq -S . users/$user) <(jq -S . updated-users/$user)
    # or full on one side and the diff on the other side, colored
    # diff -y --left-column --color  <(jq -S . users/$user) <(jq -S . updated-users/$user)
done

#shell #diff #json #script