Thursday, January 26, 2017

Using PDFTK to Process PDF Documents

When dealing with PDF documents, we come across different requirements such as splitting a document in to parts and sometimes merging multiple documents together. I recently came across a great tool which we can use to do various tasks with PDF documents. I decided to list down few things I can do using pdftk tool and leave a link to a better resource.

(1) Install pdftk tool.

sudo apt-get update

sudo apt-get install pdftk


(2) Suppose I want to create a new pdf file extracted from the contents of the page numbers 31 to 37 in a larger pdf file. We can do that as follows.

pdftk input.pdf cat 31-37 output output.pdf

(3) Merging two documents one at the end of another.

pdftk input-file1.pdf input-file2.pdf cat output new-document.pdf

(4) Selecting few pages from multiple documents and putting them together into a single document.

pdftk A=input-file1.pdf B=input-file2.pdf cat A110-117 B2-3 output new-document.pdf

That's it!

References


Wednesday, January 25, 2017

Independent Component Analysis (ICA)

When we want to separate two signals which are mixed up, one interesting method we can use is Independent Component Analysis (ICA).  I think I'm not knowledgeable enough to explain how this whole thing works but there are plenty of explanations about it in the web. Take a look at the references listed at the end for any further details. But, the purpose of this article is to put the codes I used recently for an ICA job so that I will not forget how to use it in the future.

In order to perform ICA on Python we need to install an important package first. Let's do it first.

sudo pip install --upgrade pip
pip install -U scikit-learn

Now, it's time to write the Python script. The following script is taking two wav files as input which contains two mixed signals in different ways. Then it generates another two new wav files which contains the separated signals.


"""
=====================================
Blind source separation using FastICA
=====================================

An example of estimating sources from noisy data.

:ref:`ICA` is used to estimate sources given noisy measurements.
Imagine 3 instruments playing simultaneously and 3 microphones
recording the mixed signals. ICA is used to recover the sources
ie. what is played by each instrument. Importantly, PCA fails
at recovering our `instruments` since the related signals reflect
non-Gaussian processes.

"""
print(__doc__)

import os
import wave
import pylab
import matplotlib

import numpy as np
import matplotlib.pyplot as plt
from scipy import signal
from scipy.io import wavfile

from sklearn.decomposition import FastICA, PCA

###############################################################################

matplotlib.rcParams['ps.useafm'] = True
matplotlib.rcParams['pdf.use14corefonts'] = True
matplotlib.rcParams['text.usetex'] = True

# read data from wav files
sample_rate1, samples1 = wavfile.read('100000010mix1.wav')
sample_rate2, samples2 = wavfile.read('100000010mix2.wav')

print 'sample_rate1', sample_rate1
print 'sample_rate2', sample_rate2

S = np.c_[samples1, samples2]

ica = FastICA(n_components=2)
S_ = ica.fit_transform(S)  # Reconstruct signals

print 'original signal=', S
print 'recovered signal=', S_
print 'extracted signal1', S_[:,0]
print 'extracted signal2', S_[:,1]

# write data to wav files
scaled1 = np.int16(S_[:,0]/np.max(np.abs(S_[:,0])) * 32767)
wavfile.write('extracted-signal-1.wav', sample_rate1, scaled1)

scaled2 = np.int16(S_[:,1]/np.max(np.abs(S_[:,1])) * 32767)
wavfile.write('extracted-signal-2.wav', sample_rate2, scaled2)

###############################################################################
# Plot results

pylab.figure(num=None, figsize=(10, 10))

pylab.subplot(411)
pylab.title('(received signal 1)')
pylab.xlabel('Time (s)')
pylab.ylabel('Sound amplitude')
pylab.plot(samples1)


pylab.subplot(412)
pylab.title('(received signal 2)')
pylab.xlabel('Time (s)')
pylab.ylabel('Sound amplitude')
pylab.plot(samples2)


pylab.subplot(413)
pylab.title('(extracted signal 1)')
pylab.xlabel('Time (s)')
pylab.ylabel('Sound amplitude')
pylab.plot(S_[:,0])

pylab.subplot(414)
pylab.title('(extracted signal 2)')
pylab.xlabel('Time (s)')
pylab.ylabel('Sound amplitude')
pylab.plot(S_[:,1])

pylab.subplots_adjust(hspace=.5)
pylab.savefig('extracted-data.pdf')
pylab.show()

Run this Python script with the two wav files in the same directory and you will get the output signals as wav files in addition to seeing waveform plots of all these signals.



Thursday, December 22, 2016

Listening to the Giants, Once Again!

It's been a long day :)
We have been working on a long term project to minimize human-elephant conflict in Sri Lanka by applying various technologies from our expertise in computer science and embedded systems. One such application we are working on is building a smart electric fence which can notify the maintainers and people protected by the fence about breakages of the fence wire at different places. The second application is a system which can locate elephants in the wild based on the infrasonic waves (low frequency sounds) they emit. In order to do that, we have designed and built low-cost infrasonic detectors by ourselves in the lab and have performed various experiments.

Meeting in the village.
In order to perform evaluations of the applications we have built, we have visited different places outside the lab environment and performed experiments with real-world conditions. Once I went with my lab colleagues to visit a domesticated elephant in order to record its sound about which you can read here. As an extension to all these experiments, about a week ago, we visited Udawalawa wildlife park and Hambegamuwa village which is situated in the edge of the wildlife park. There were two important goals of this visit. The first was to meet the villagers of Hambegamuwa and have a meeting to get their help on building our own electric fence in their premises. The second was to visit Udawalawa park and perform elephant infrasonic localization experiments. The journey was planned to be 2 days long since we had a lot of works to do.

Phantom-4 flying over the electric fence.
(video footage from the drone camera is shown
at the end of this article.)

On the journey day early morning, me and Namal came to lab and loaded all the equipment to the van. Then we went to Dr. Deepani's home to pick her. She is an ecologist who has been working on elephant conservation in Udawalawa area for many years. She is the one who mediate our contacts with the villagers and wildlife department officials. After she joined with us to the journey, we directly went to Dr. Kasun's home in Ambalangoda to pick him and Chathura. Dr. Kasun's home offered us our breakfast since we came from Colombo early morning without stopping to have breakfast. Finally, we all got together and set sail to Udawalawa.

Rain and the darkness failed to stop our guys.
When we arrive to Hambegamuwa town, it was evening. We directly went to the small guest house which we had selected to stay. We left our bags there and directly went to the village in Hambegamuwa for the meeting with the villagers which is the first target of our journey. The village is located at the edge of the Udawalwa wildlife park. Therefore, they are continuously getting hit by crop raiding elephants. The people in this small village are growing crops in the surrounding area and they live inside small houses under the threat of roaming elephants. To protect their village, they have an electric fence surrounding an area of approximately 20 acres. Their fence just like all the other electric fences breaks down frequently mostly because of the elephants. When the fence is broken, the villagers have a hard time locating the breakage by walking along the fence wire.

Preparing infrasonic detectors for the experiment.
In this small village, we had a brief meeting with the people and we agreed to donate them an electric fence energizer in order to build a new electric fence for them. They will provide the labor force and other materials such as poles and wires to build the fence while we are providing the most expensive electric fence energizer. We received this energizer as a donation from the students who did Google Summer of Code (GSoC) with our SCoRe lab. As a community service, we are going to donate that energizer to the villagers. In return, we get the chance to use this fence as an experimental testbed to try our electric fence breakage detection system which we developed as a result of a research in the lab.

Deploying a pair of infrasonic detectors.
After the brief meeting, we went to the location of the fence around their village and tried the fence energizer we had bought for them. We noticed that the voltage of the pulses in their existing fence is significantly higher than the voltage of the pulses provided by our energizer. Their energizer is a locally built one while ours is a branded energizer from an international manufacturer according to the standards of electric fencing. Even the villagers accepted that their current energizer is too dangerous to elephants as it can kill an elephant due to the higher voltage instead of driving it away. Therefore, use of an energizer built up to the standards is necessary for them. There was a small rain fall by the time we were inspecting the fence which was a disturbance as we we were dealing with high-voltage. When the darkness came, it was challenging to test our breakage detection system but we kept working under the torch lights. When we return to the guest house in Hambegamuwa town, it was about 9.00pm. The dinner was served to us at the house of the owner of the guest house. His house was located next to the guest house and he lived there with his family. After the dinner, we all went to sleep as we all were tired.

Calibrating the angle of an infrasonic detector.
The next morning, we woke up to start another long day as our plan was to test our elephant infrasonic localization system in the field. We have built low-cost infrasonic microphones and a set of firmware running on an embedded system which can be used to locate elephants from long distances. We have performed experiments on them at different places but our objective of this journey was to use it for real wild elephants. The owner of the guest house offered our breakfast again at his house before we start the journey. Namal was so busy from the morning as he had to prepare the microphones properly for the experiments. He even worked while we were traveling in the van from the guest house to the Udawalawa wildlife park entrance area.

Giants have been here...
After arriving to the Udawalawa wildlife park area, we parked the van near a temple and set up our microphones in the vehicle park as a start. We stayed there recording data for few hours while Dr. Kasun and Dr. Deepani went to have a meeting with the Udawalawa wildlife park warden. After the meeting, he returned with the good news that we can carry our quad-copter to their place and fly it around to take some pictures. Therefore, Chathura went with them in the van since he is our quad-copter specialist and the pilot (read more about our drone adventures here). Meanwhile, we decided that recording infrasonic data from the vehicle park is not a very successful approach as it receives a lot of noise from the vehicles going in the nearby road. Therefore, we decided to carry our two pairs of infrasonic detectors into the jungle.

Our hiding place where we spent the whole evening.
In order to get closer to the Udawalawa reservoir, we hand to crawl under an electric fence which was difficult with the equipment we were carrying. After getting closer to the Udawalawa reservoir, we found a nice hiding place where we can perform the sound recordings. We placed our infrasonic detectors on the ground and calibrated their settings to capture data continuously. Then, we stayed there with a pair of binoculars to note the sightings of elephants and the location based on the visual observation. Our hope was to compare the results of the captured data using the infrasonic detectors against our visual observation. An old fellow who lived in the temple where we parked our van came with us to this place and helped us in various ways for a while and finally went back to the temple. From that point onwards, only Namal and me was left near the reservoir.

Elephants in the distance as seen from the binoculars.
When it was lunch time, Chathura brought two lunch packets and a water bottle to us and went back in the van to the place where they were flying the quad-copter. Namal and me spent the whole evening at this location near the reservoir and had lunch at about 3.30pm while watching elephant herds coming and going to the Udawalawa reservoir. When we finish our lunch there was no any elephant near the reservoir and therefore we both decided to walk to the water and come back immediately if we notice any danger. By the way, while the microphones are recording data, we had nothing else to do there other than watching elephants using the binoculars. It was about 5.30pm when our guys returned to the location we were after flying the drone. Since it was late and many people warned us that elephants can arrive to the place we were hiding after the darkness falls, we decided to grab our stuff and move out of the jungle. We crawled again under the electric fence, loaded our equipment to the van parked near the temple and started moving. It was closer to 6.00pm and the time was right for the Udawalawa elephant orphanage to feed baby elephants with milk. So, we quickly moved in to that place to see it. Finally, we started our journey back to Colombo.




It was about 1.30am when I returned to home in Colombo and was so tired. However, the hope that our recorded data must contain some proof that our elephant localization system works in the real field kept spinning in my mind.

~********~

Sunday, December 18, 2016

Usage of NMCLI command

nmcli is a useful commandline tool on linux based systems to deal with the Network Manager without using GUI. In this post, I'm leaving some of my short notes about using it for connecting to WiFi networks.

Quickly check whether we are connected to the network using any
available device

# nmcli device status

check all the currently available connections

# nmcli connection show

turn off and on wifi

# nmcli radio wifi off
# nmcli radio wifi on


rescan for the available wifi networks

# nmcli device wifi rescan

list all the available wifi networks

# nmcli device wifi list

connect to a specific wifi network identified by ESSID ('eduroam' in this example)

# nmcli connection up eduroam

delete an existing connection from the network manager

#nmcli connection delete 3Com

connect to a wifi network

# nmcli device wifi connect 3Com password '1234'

adding a new connection information

# nmcli connection add type wifi con-name Asanka-WiFi ifname wlp1s0 ssid eduroam

That's it!

Monday, December 5, 2016

Audio Recording from the Linux Shell

This just a quick note about something I tried and hope that this information will be useful to me someday. First of all, let's check the available recording devices on our machine using the following command.

arecord -l

I received an output like the following.


Using the details we received from the above command, we can now capture any audio device as follows. If you want to record multiple devices at the same time, you can do that using two terminal windows. 

Built-in stereo sound card
ffmpeg -f alsa -ac 2 -i hw:1,0 -t 30 ~/Desktop/card1-device0.wav

USB mono sound card

ffmpeg -f alsa -ac 1 -i hw:2,0 -t 30 ~/Desktop/card2-device0.wav

 That's it!

Saturday, November 26, 2016

Prof.Thiemo's Visit to SCoRe lab in 2016

It is not necessary to mention that Prof.Thiemo Voigt is a regular visitor here. He used to visit Sri Lanka and meet our SCoRe group once in every year. This time, he arrived Sri Lanka on Last Sunday night. Without taking a break, the next day morning which was a Monday, he visited UCSC and met us. On this first day, we had organized a journey to goto Galle, Sri Lanka in order to allow him to goto the beach and get socialized with us. Everybody of our SCoRe group visited there with him. We all went to Jungle Beach in Unawatuna and had a bath in the sea. Then we had lunch from a nearby restaurant. It was about 5.00pm in the evening when we return to Colombo. Therefore, we didn't schedule any other work for that day.

Namal demonstrating the electric fence.
The next day, Thiemo delivered a talk in UCSC and his topic was "A highly efficient, low-cost backscatter reader". There he described a new technology they have published recently about a backscatter communication. After his talk and the lunch, we dedicated the rest of the evening to show off the outcomes of project works in our lab. Our guys in the lab had done a lot of hard work to prepare the demonstrations. Every thing went smoothly on the demonstrations. We mainly demonstrated the components of the elephant project which includes smart electric fence, infrasonic localization system and elephant detection with drone footages. In addition to the guys in our lab, some 4th year students also joined with us on this. It was a great day with so much enthusiasm. Finally, few guys from the lab joined with Thiemo to go for a dinner to which I didn't join since I had some other works.

Discussion at the infrasonic localization demonstration.
The third day of Thiemo's stay in Sri Lanka was the last day. We had a meeting in the morning to discuss about the next moves we should take on our collaborative project. Most importantly, we discussed about the potential improvements we can make in the paper which got rejected recently from a conference. We went through the comments of the reviewers and noted various actions to make as a response. Finally, we selected another venue to submit the paper soon after making improvements. In the evening of that day, Thiemo delivered a lecture for the MSc students. A part of the lecture was same as the guest talk he delivered a day before however the second part was something new. I attended to this lecture too and learned some new things. After the lecture, he took a little more time to have a on-line meeting with some student before conclude the day. Some of our guys again joined with him to go for dinner after which he left for the airport.

Thursday, October 27, 2016

Emulating Networks using CORE Network Emulator

I've been using Mininet in practical sessions for undergraduate computer networking courses for a while. The tool has great features and provided the required capabilities for our course works. However, Python scrips required to model a network topology was a trouble at some points because some students who were not good at Python programming faced difficulties in understanding networking concepts using Mininet. Meanwhile, I had a requirement to setup an IPv6 based network in a recent demonstration and found that Mininet is not capable of building IPv6 based networks.

After searching for a while on other available options, I found another interesting network emulator called CORE network emulator. This tool provided me easy setup of different network topologies by visually dragging and dropping components on a canvas. Setting IP addresses for each host and router interface could be done by either filling information on a graphical window or by using command line. It seems the features of the emulator are very intuitive and I believe that a student can easily self-learn how to model a network and test different things he/she learned using the emulator.

On a Ubuntu GNU/Linux 16.04 LTS machine, I was able to install the emulator by using the apt repositories easily as follows.

sudo apt-get update
 
sudo apt-get install core-network

We can start the emulator from the command line by using the following command which will bring up the emulator window.

core-gui

There are nice tutorials and resources to learn about how to use this emulator in the web. I will list some of them in the references list below.

References: