Truly Reproducible Research Papers

A slide from Prof. Barry Smyth's presentation

If you perform an experiment and get some interesting results which cannot be redone and get the same results by somebody else, something is wrong with your finding. This is called reproducibility of research. If it is not reproducible, it is not science. You might think that systematic research carried out by academics and professional scientists who publish papers in conferences and journals are doing reproducible research. Not really.

Majority of research papers I've come across in my own domain are bare descriptions and explanations of their results without proper support for reproduction of the results by anybody interested. Even though a paper with a good quality provides a lot of details of their experimental setups and settings, it difficult to truly recreate their results completely based on the details in the paper. It is often necessary to contact the authors and have a correspondence back and forth several times to get things clear. Similarly, if I ask myself whether I can reproduce a research work I had published few years ago solely based on the details I had put down on my own paper, I have to give a big 'No' unfortunately.

This is a bad way to do science.

It is unfair to computer scientist if I say they are not putting any effort to make their research reproducible. There are two important ways they try do it these days. The first is giving away data sets that they had collected. This allows third parties to verify their results and also to extend and build upon it. The second is to provide the source codes of the experimental implementations they have made. They usually put their codes into a Github repository and provide the link in their research papers so that readers can find the source code repository and reuse their code.

Another slide from Prof. Barry Smyth's presentation

Recently I attended to a talk delivered by Prof. Barry Smyth in UCD, Ireland where he suggested two interesting ways to make our research papers reproducible. The first is a practice which is much simpler and easier to do. That is to produce a Jupyter Notebooks along the scientific publication which has both software codes, data, descriptions and explanations in a well documented manner which a third party can quickly run and build upon. If you haven't used or read about Jupyter Notebooks, have a look at the first link in the references section. It's a way to produce well documented software codes where you have your software codes, their descriptions and their output in a report-like format.

There's even more powerful way of making reproducible research papers. Imagine you are producing a research paper where the paper talks about a 30% improvement in something. How to enable the reader to verify whether this number is truly 30% by using their own experimental data? If I'm giving away the source codes of my implementations, does the reader has sufficient information to locate the correct programs and execute them in the correct sequence in order to get the final result? This is where the tool "Kallysto" comes in. It is a tool developed by Prof. Barry in order to make scientific publications fully reproducible and traceable. Kallysto combines Latex with Jupyter Notebooks in such a way, your Latex manuscript is directly linked with the original data and the software codes which analyze them. While the typical workflow of writing a research paper is to (1) analyze data, (2) produce graphs as images or PDF files, and finally (3) create a Latex manuscript which explicitly include those graphs. When you compile your Latex source files, Kallysto will run the Jupyter Notebooks analyzing data and generates the results in real-time which will be used by Latex to produce the final PDF document.

The idea of Prof. Barry Smyth is to make scientific publications truly reproducible by scripting everything from the data to results and finally to latex documents.

References:

[1] Jupyter notebooks

https://jupyter.org/

[2] Netflix Papermil tool

https://medium.com/netflix-techblog/notebook-innovation-591ee3221233

[3] The tool made by Prof. Barry Smyth called Kallysto

https://github.com/barrysmyth/kallysto

Google Colaboratory Notebook with Data from Google Drive

Until recently, I was using individual Python scripts with data files here and there for data analysis in my research works. After understanding the power of having a better documentation of my experiments along with the codes and graphs, I started to use Jupyter Notebooks. However, I still had a limitation. Jupyter Notebook works on my local computer with the data files. Every time I do some analysis, I have to do it locally and upload the results to Google Drive as a backup. Whenever I want to work on it again, I have to download the Jupyter Notebook and the data files which is a big hassle.

Today I realized that Google provides an online tool to run Python Notebooks while the data and the Notebook file is still in the Google Drive. There's no requirement to download my data and Python scripts to local computer each time I want to do some analysis. Here's how we use Google Colaboratory for this purpose.

(1) Create a directory in the Google Drive where I want to create my Colab Notebook. Let's say I've created the directory "Google-Colab-Demo" in the following location.

My Drive > UCD > Asanka's PhD > Experiments > Google-Colab-Demo

(2) Right-click inside the created directory and select Colaboratory from the menu. It will open a new web browser tab with a new Notebook. Give a name to the notebook. I'll set it to plotting.ipynb. 

(3) In the local computer, create a text file with the name data.csv and add the following content. Then upload it into the above directory we created in Google Drive.

1,2

2,4

3,9

4,16

5,25 

6,36

7,49

8,64

9,81

10,100

(4) Add a text cell and provide some details about what we are going to do.

(5) Add a code cell and insert the following code into it. Note that the full path to the data.csv file can be extracted by right-clicking on the data file on the file browser in the left hand size pane and then selecting the menu option Copy path.

1 2 3 4 5 6 7 8 9 10

import matplotlib.pyplot as plt import numpy as np x, y = np.loadtxt("/content/drive/My Drive/UCD/Asanka's PhD/Experiments/Google-Colab-Demo/data.csv", delimiter=',', unpack=True) plt.plot(x,y, label='Loaded from file!') plt.xlabel('x') plt.ylabel('y') plt.title('Interesting Graph\nCheck it out') plt.legend() plt.show()

(6) Now, run the code segment by clicking on the Play icon on the left corner of the code cell. The resulting graph will appear like the following.

Some extra work...

Sometimes, you can have a Jupyter Notebook with data in the local computer which you have copied to Google Drive. Now you want to run the same Jupyter Notebook in Google Colab. In that case, first you need to right-click on the Jupyter Notebook on Google Drive and open it with Collaboratory. If your Google Drive does not appear to be mounted automatically in the File browser pane, follow the stesps given below.

(7) Mounting Google Drive into the Notebook by running following code. It will prompt for an authentication code which should be typed in. In the left hand side corner of the screen, a file browser should be available now with the access to the google drive files in the mounted location.

1 2	from google.colab import drive drive.mount('/content/drive')

(8) If we want to import another Python file in the Google Drive as a Python module within our Notebook, first give the path to the location of the Python file. Then import the Python module as in a normal Python program.

1 2 3

import sys sys.path.append("/content/drive/path-to-current-directory") import my-module

Now everything should be good to go like a normal Colab Notebook.

Cheers!

~********************~

Saving Selected GNURadio Data using an External Trigger

When using GNURadio Companion (GRC) software package with a software defined radio (SDR) to capture radio signals, the usual approach is to save the incoming data stream into a file so that they can be processed later other other tools. However, the fast sample rate of SDR devices such as HackRF causes GRC script to generate large IQ data files which are difficult to handle on computers unless there's a big memory and a fast processor. It would be a wiser decision to save IQ data only when there's some useful signal is coming through the data stream. This article describes a way to give an external trigger to a GRC script in order to save selected segments of the incoming IQ data stream into a file. The two key elements of this trick are Burst Tagger and Tagged File Sink.

The Burst Tagger is a block which tag the incoming data stream by inserting a special tag with a value. The tagging work is one based on an input given to it through a second channel. When the value of this second channel goes above 0, a tag is inserted into the data string and when the value of the second channel goes below 0 again, another tag is inserted into the data. These two tags specify the region of the data stream which we needs to be saved. This output of the Burst Tagger is sent to the Tagged File Sink block. This block identify the tagged regions of the data stream and save each of those regions into separate data files.

Let's start.

(1) First of all, we need a GNURadio Companion flowchart like the one shown in the following figure. Once the flowchart was created, generate the Python script by clicking the button to "Generate the script". The generated file is named as top_block.py by default.

(2) Write the following Python script in a separate file and name as test.py. This script updates the "trigger" variable which causes the GRC script to perform the tagging and saving data files.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

import top_block import time print("Creating top_block class...") tb=top_block.top_block() print("Starting top_block...") tb.start() print("Started...") ####################################### # Testing the trigger print("saving...") tb.set_trigger(1) time.sleep(2) tb.set_trigger(-1) print("stopping...") time.sleep(2) print("saving...") tb.set_trigger(1) time.sleep(0.5) tb.set_trigger(-1) print("stopping...")

(3) Now, the script can be run as follows. You should see that two IQ data files get's saved into the current working directory.

python test.py

That's it!

~*************~

3.5 inch LCD Display on Raspberry Pi

I received a 3.5 inch LCD display which can be attached to the Raspberry Pi B+ device. It can be attached to the GPIO port of the RPi and use it as a display and as a touch screen for input. Installing drivers for the device was straightforward. Following are the steps I took to get it done.

(1) Download the software from the Github from the following project into a computer. https://github.com/goodtft/LCD-show

(2) Connect the RPi from the computer through SSH and copy the downloaded software into it.

(3) Move into the directory from the terminal and run the following command to install the driver software.

sudo ./LCD35-show

When the commend completes running, it automatically reboots the RPi and get back with the LCD touch screen.

A Signature Image for Documents

While filling forms and preparing documents on the computer, I come across needs to put my handwritten signature on the document. In such cases, the printing of the document, signing it by hand and then scanning it to produce the soft copy is too much. It is useful to have an image of the signature which can be easily inserted into documents.

On Linux, we can use the following steps to produce an image file where the handwritten signature stays in a transparent background.

(1) Put the signature on a white paper and take a photograph.

(2) Use a suitable tool to crop the photograph file to remove the unnecessary edges and have only the area covering the signature (e.g., Shotwell Viewer). Let's say the resulting image file is original.jpg.

(3) Run the following command to push the whiter pixels to fully white and darker pixels to fully black.

convert original.jpg -colorspace gray -threshold 28% filtered.png

(4) Run the following command to set the while pixels as transparent.

convert filtered.png -transparent white signature.png

Now, the resulting signature.png file contains the handwritten signature in black color while the background stays as transparent. This file can be used for the purpose of inserting the signature into electronic documents.

Low Graphics Mode Problem on Ubuntu 16.04 LTS

My laptop, which runs Ubuntu 16.04 LTS, recently started to give me a strange error. When the computer is turned on, it boots into a screen where a message is shown as "Your system is running on low graphics mode". If I proceed to boot the system, the system boots up however the on-board WiFi is not detected. My computer suffered a physical damage lately which may have caused some hardware to malfunction. Anyway, being unable to fix this strange error, I found a somewhat workaround for it in the web.

Following the steps shown below to workaround each time when the "Low Graphics Mode" error occurs on Ubuntu 16.04.

(1) Turn on the computer and wait till it lands on the "Your system is running on low graphics mode" error.

(2) Press the key combination Ctrl + Alt + F1 and login to the virtual terminal it provides.

(3) On this terminal, type the following command.

sudo service lightdm restart

(4) Now, when the GUI login prompt appears, login there and the desktop environment should load now without any issues.

Cheers!

Installing SETI@home on Ubuntu 16.04 LTS

Searching for alien life is the goal of many space exploration projects for many years. Aliens can come in all shapes and sizes starting with single celled life. Among them, finding intelligent life would be exciting than anything else. Search for Extraterrestrial Intelligence (SETI) is a project which attempts to achieve that goal by listening to their radio transmissions. They have large radio telescopes which collect radio signals from the outer space and then process them to look for any sign of message from an intelligent beings.

Now, here's the issue. The scientists at SETI don't have enough processing power in their computers to process all these data acquired from the radio telescopes. This is obviously a computationally intensive task. That's where we can give a little hand as a good citizen of Earth. We can share a part of our personal computers processing power to process SETI data. I'm writing down the steps to install the required software on a computer running Ubuntu 16.04 LTS and setup everything.

(1) First of all, we need to have an account in the SETI@home website, in order to contribute. Go to the following website and create an account.

 

https://setiathome.berkeley.edu/

(2) Now, it's time to install the relevant software on our computer. For a computer running Ubuntu 16.04 LTS, we can easily install the software through the software repositories. Open a terminal and issue the following command for that.

sudo apt update

 

sudo apt install boinc

(3) Open the BOINC software GUI and log into your account. Once you are logged in, you should be able to search for the SETI@home project from the list of projects available and add it.

(4) After a while, the SETI@home client will start taking data from the sever and process them locally. You can view the ongoing tasks on your computer in the GUI client.

Cheers!

SSIM to Measure Image Similarity

There are situations where we need to measure the similarity between two images. For example, when we have an original image and few other images with lower quality, we might need to identify when one is mostly similar to the original image. Bit-wise comparison of the image pixels is not applicable in this kind of scenarios. We need something more sophisticated. Structural similarity index (SSIM) is such a method. It provides a value between 1 and 0 when comparing two images. If two images are exactly similar the SSIM becomes 1. If they are totally different, SSIM becomes 0.

Following Python program implements SSIM to compare between to images. Since we need some extra python libraries for this code to run, we should install following libraries on Ubuntu Linux.

sudo apt install python-skimage

sudo apt install python-opencv

SSIM calculation code is as follows.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

from skimage.measure import structural_similarity as ssim import cv2 print("Reading files...") first = cv2.imread("image1.png") second = cv2.imread("image2.png") print("Resizing files...") first = cv2.resize(first, (2576,1125)) second = cv2.resize(second, (2576,1125)) print("Converting files to grayscale...") first = cv2.cvtColor(first, cv2.COLOR_BGR2GRAY) second = cv2.cvtColor(second, cv2.COLOR_BGR2GRAY) print("Calculating SSIM value...") s = ssim(first, second) print("first vs second", s)

~*******************~

Merging Video Files on Linux Terminal

When we need to merge multiple video files into a single file, there's an interesting tool called mkvtoolnix we can us. We can install it using the Linux software repository as follows.

sudo apt install mkvtoolnix

Let's say we have three mp4 video files which we need to merge and build a single mkv file. We can do it in the following way.

mkvmerge -o full-video.mkv part-1.mp4 \+ part-2.mp \+ part-3.mp

Meanwhile, I learned that there's a GUI version of the tool called mkvtoolnix-gui which I should explore later. For the moment, we can install it using the following command.

sudo apt install mkvtoolnix-gui

Cheers!

Kazam Screen Recorder

Recently, when I wanted to create a video demonstration which involves recording my computer screen and the input from my microphone. As an Ubuntu GNU/Linux user, I've tried few different screen recorder software but, the results were not very encouraging. However, in my recent attempt, I found a nice piece of software which could meet my requirements. It's called Kazam screencaster program. I just thought to leave a note about it here so that I can find it when I need it the next time.

It is available on the Ubuntu software repositories and we can install it with the following command.

sudo apt update

sudo apt install kazam

Running TempestSDR on Windows 10

TempestSDR is a nice tool which can be used to eavesdrop on computer monitors using the electromagnetic (EM) emissions causes by them. While this concept is not a new thing, the use of cheap software defined radio (SDR) hardware has enabled the possibility of performing this attack a lot more easier. I was struggling to get the tool up and running on Ubuntu Linux for a while and ended up without a result. Finally, I moved into Windows platform and tried it. Luckily, things went so smoothly and I got TempestSDR tool running with both RTL-SDR and HackRF hardware.

In this post, I'm writing down the steps I followed to get TempestSDR running on Windows 10 operating system with both RTL-SDR and HackRF hardware.

Preparing RTL-SDR and HackRF hardware:

It is necessary to have the required drivers installed on Wondows 10 in order to use both RTL-SDR and HackRF devices. Therefore before everything, let's get the drivers installed. A previous post written by me describes the required steps for. Refer it and install the drivers described here: http://recolog.blogspot.ie/2018/02/installing-drivers-for-rtl-sdr-and.html

Identifying the EM emission frequency of a target monitor:

Before we prepare the TempestSDR tool to eavesdrop on a computer monitor, we need to identify the frequencies where EM emissions occur on the target. This is a little bit cumbersome task as we have to go through the spectrum and identify them. We'll use SDR# software for this purpose.

(1) Download SDR# from here.
https://airspy.com/download/

(2) Extract the ZIP archive, and then inside it, double-click on the install-rtlsdr.bat file. A CMD prompt will start and download some files. It will exit automatically.

(3) Now double click on the SDRSharp.exe tool and it will open the window. You can select the "RTL-SDR (USB)" option for the source.

(4) Keep scrolling while looking for a signal which varies the peaks when I make any change in the screen of the computer such as maximizing / minimizing windows, etc. If there's a strong signal which changes the amplitude when a window is maximized, there's a good chance that it is an emission from the monitor. Note down such frequencies.

Setting up TempestSDR software:

(1) Installed JDK 8 - 32-bit version. I downloaded it from here,
http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html The exact file I downloaded is jdk-8u151-windows-i586.exe

(2) Download and install MinGW and MSYS. We have to download a single installer and inside it, we can select the packages of MingGW and MSYS that we want to install.
https://sourceforge.net/projects/mingw/files/

(3) Set the bin folder paths to MinGW and MSYS in Windows PATH. The instructions to set PATH environmental variable on Windows can be found here: https://www.computerhope.com/issues/ch000549.htm

In my system, the paths to the bin folders of those tools after the installation were as follows.

C:\MinGW\bin
C:\MinGW\msys\1.0\bin

(4) Add JAVA_HOME path variable too. The instructions to do this can be found in this link: https://www.mkyong.com/java/how-to-set-java_home-on-windows-10/

 In my system, the path to the directory where Java was installed is as follows.

C:\Program Files (x86)\Java\jdk1.8.0_151

(5) Download TempestSDR from here. Then extract the files.

https://github.com/rtlsdrblog/TempestSDR

(6) In the very first makefile, remove the following line

@$(MAKE) -C TSDRPlugin_Mirics/ all MIRICS_HOME=$(MIRICS_HOME)

(7) Due to the fact that there are spaces in the path to Java installation directory, TempestSDR tool faces some difficulties while running the make file. Therefore, let's copy java installation folder to a new place which does not have spaces in the path.

I copied "C:\Program Files (x86)\Java" folder to "C:\Java" location.

(8) Now go into TempestSDR folder from CMD prompt and and run the following command.

make all JAVA_HOME=C:\Java\jdk1.8.0_151

If the compilation completes successfully, we are good to go.

Running the TempestSDR software:

(1) Connect either RTL-SDR dongle or HackRF device into a USB port of the computer.

(2) Go to the JavaGUI folder in the TempestSDR source code directory. There should be a jar file which we need to run.

java -jar JTempestSDR.jar

(2) From the File menu, select the "Load ExtIO source" option. Then browse to the installation directory of HDSDR software where you copied the ExtIO DLL driver for either RTL-SDR or HackRF. Select that DLL file.

(3) Select the resolution and refresh rate of the monitor being eavesdropped. Then, select the frequency of EM emanation which we manually found using SDR# software. Click "Start" and we are good to go.

Trouble Shooting:

Time to time, TempestSDR tool faced difficulties in detecting the RTL-SDR or HackRF device. In such situations, I used the following steps to resolve the issue.

(1) Restart the machine.

(2) Run SDR# with RTL-SDR/HackRF first to get the correct driver running.

(3) Then try running TempestSDR jar file from the beginning.

Following are some of the screenshots of my attempts.

A checker board image was placed on the target computer screen.

TempestSDR capturing data from a Dell monitor with RTL-SDR

 

TempestSDR capturing data from a Samsung monitor with HackRF

~*******~

Installing Drivers for RTL-SDR and HackRF on Windows 10

Since I have been using software defined radio (SDR) tools on Linux platform for a long time, it was a very new thing to me when I had to use some SDR tools on Windows. Anyway, the installation of the relevant drivers went smoothly and the devices were ready to use within a short while. In this post, I'm writing down the steps I followed to get my RTL-SDR dongle and HackRF device up and running on a Windows 10 machine. Here we go.

Instructions for RTL-SDR:

(1) Connected RTL-SDR dongle to the USB port and Windows automatically detected the device and installed some drivers. But, we need to manually install a special driver called ExtIO.

(2) Download the Zadig USB driver installer from here: http://zadig.akeo.ie/
No installation necessary. It can be run immediately.

(3) Run Zadig executable. With all the default settings, click "Install Driver" to install the WinUSB driver.

Zadig tool is used to install WinUSB driver

(4) Download and install HDSDR tool. Even though we install it, do not attempt to use RTL-SDR with HDSDR software yet.
http://hdsdr.de/download/HDSDR_install.exe

(5) Download the ExtIO driver DLL for RTL-SDR from here.
http://hdsdr.de/download/ExtIO/ExtIO_RTL2832.dll

(6) Copy the ExtIO driver DLL file to the installation directory of our HDSDR software which we installed a short while ago. In my system, this directory is,
C:\Program Files (x86)\HDSDR

(7) Now, start HDSDR. In my system, HDSDR automatically picked the RTL-SDR dongle as the input and sound card as the output and started picking signals. That means everything is working.

Instructions for HackRF:

(1) Connect the HackRF to the USB port and windows automatically detected it and installed some drivers.

(2) Download the Zadig USB driver installer from here: http://zadig.akeo.ie/
No installation necessary. It can be run immediately.

(3) Run Zadig executable. From the options menu, select "List All Devices". Then from the drop-down list, select "HackRF One".

(4) Since I have already installed the WINUSB driver for RTL-SDR, I don't have to do anything here. It shows that the driver is the latest already. In case you don't have that option, go ahead and click "Install Driver" to install the WinUSB driver.

(5) Download and install HDSDR tool. Even though we install it, do not attempt to use HackRF with HDSDR software yet.
http://hdsdr.de/download/HDSDR_install.exe

(6) Download the ExtIO driver DLL for HackRF from here.
https://github.com/jocover/ExtIO_HackRF/releases

(7) Copy the ExtIO driver DLL file to the installation directory of our HDSDR software which we have installed. In my system, this directory is,
C:\Program Files (x86)\HDSDR

(8) Now, start HDSDR. In my system, HDSDR automatically prompted asking to select which DLL to be used, either RTL-SDR or HackRF. Select the DLL file for HackRF and it starts picking signals. That means everything is working.

~**********~

Capturing ADS-B packets using HackRF

A few weeks ago, I managed to setup my computer to capture ADS-B beacons transmitted from aircrafts flying over UCD. That was a fantastic experience. Unfortunately, I was woking on trying another tool while working on this and therefore, I cannot remember the exact tools I installed for this particular work. Therefore, I will write the steps I remember which may include some steps which are not required for this work. I did this work on a Kali Linux machine.

(1) Install some required packages using the apt-get command as follows.

sudo apt-get install gqrx gr-air-modes cmake g++ libpython-dev python-numpy swig hackrf libhackrf-dev

(2) Install the tool called SoapySDR which is available on the Github.

git clone https://github.com/pothosware/SoapySDR.git
mkdir build
cd build
cmake ..
make -j4
sudo make install
sudo ldconfig #needed on debian systems
SoapySDRUtil --info

(3) Install the tool called SoapyHackRF which is again available on Github.

git clone https://github.com/pothosware/SoapyHackRF.git
cd SoapyHackRF
mkdir build
cd build
cmake ..
make
sudo make install
SoapySDRUtil --probe="driver=hackrf"

(4) As a part of the two tools installed on the previous steps, we need some extra stuff which we can get using apt-get command.

sudo apt-get install soapysdr-module-uhd libuhd003.010.002 libuhd-dev

(5) Now, we need to add some UDEV rules which is explained in the following link.

https://github.com/mossmann/hackrf/wiki/FAQ

(6) We are good to go now. Let's connet the HackRF to the computer and run the following command to start the ADS-B receiver.

modes_rx -s osmocom -r 10e6

It takes some time to pick some signals from an aircraft which is not very frequent around our building. Whenever an ADS-B transmission is picked from by our setup, it will be displayed on the terminal.

~*****************~