A better way of collecting usage stats in mobile applications

Let me start by setting some context. As a part of my course requirements, I have performed a study on TCP connections of various mobile applications. An interesting phenomenon that caught my attention is the various usage measurement stats being collected by application publishers. Many apps were sending information about the application usage to remote servers periodically. In a research paper titled, “Profiling Resource Usage for Mobile Applications: A Cross-layer Approach”, I learnt that during the time that the handset takes to turn off the radio (IDLE state) after it detects silence for a threshold (this is called tail period) consumes equal amount of energy as to when the communication is active. Now, I see a problem with these usage stats collection services.

Services like AppFigures.com, Splunk MINT, Flurry, Count.ly, Localytics etc help app developers gain insights on how their app is being used. While it is a great idea to use these stats to optimize user experience, periodically sending out data can be bad to battery usage on your smartphones. Here is why –

Let’s consider a device has a tail time of around 5 seconds (most devices have a tail time of 12 seconds). Meaning, the device waits for 5 seconds before turning off the radio after it observes a silence period. Now, let’s consider an app which uses a measurement tracking service that collects user stats every 8 sec or whenever a user performs an action (say for example, a click).

Case 1: Let’s say the user didn’t click anything for 10 seconds. Now the stats collection service will send the stats report in a burst of data to the server after 8 seconds. So, the radio that was turned off after the 5th second, now got activated again due to the burst of data packets. A significant amount of power is wasted during the tail time + the power needed to re-activate the radio.

Dead Battery

Case 2: Let’s consider the case that the user performed a click on the app (a region where clicking doesn’t cause anything to happen on the front end, may be to ensure the screen doesn’t dim off) at the 4th second, the stats collection service sends the data to the servers which means again a burst of data and the count down of 5 seconds is stopped. The radio will not be turned off until 5 seconds after the data transfer to the stats collection server is complete. Meaning, the power consumption isn’t reduced even though there is apparently no change on the front-end.

Case 3: Let’s consider the scenario where despite of any action on the user’s end, the stats collection service send the metrics to the server every 4 seconds. This keeps the radio in transmission state through out the time the app is open. If this app is a gaming app, which you like and you play for a couple of hours every day, the battery is drained super fast not because of the game itself, but the background usage measurement collection service.

Here is what I think is a good way to collect stats while not draining the battery.

These services can collect the stats in the following ways –

1. If the app doesn’t get any data from the internet, say a static game application, collect the stats through out the time the app is open and then send all the collected data at once in a single burst when the user closes the app. Now, there is a chance that the app crashes and the close doesn’t happen in a proper way. In those cases, the app has to check for data collected but not transmitted upon re-opening and then send them right away. In this method, may be the stats are not real-time, but the developer can still get insights while not hurting the resource utilization on the handset.

2. If the app is, say, a music service, it periodically downloads some information like songs or playlist information etc from its server. The stats collection mechanism can use this window to relay the collected stats until that time from previous send. This way, the radio time used by the stats service will be during the radio time used by the actual app thereby not effecting the battery much. This is partially real-time depending upon the intervals in which the app uses the transmission radio.

The same mechanism can be used to ad networks that reload the ads after a certain period of time (say 120 sec). The ad-network module of the app can pre-load certain ads to rotate during the app open time when the app loads for the first time. Or it can also use the radio window of the original app to fetch new set of ads depending upon the need).

This is just my idea and I may be wrong. Comments are welcome.

Resource(s)

Feng Qian , Zhaoguang Wang , Alexandre Gerber , Zhuoqing Mao , Subhabrata Sen , Oliver Spatscheck, Profiling resource usage for mobile applications: a cross-layer approach, Proceedings of the 9th international conference on Mobile systems, applications, and services, June 28-July 01, 2011, Bethesda, Maryland, USA

How to setup OpenVPN Client on DD-WRT with VPN.sh

VPN.sh is one of the most affordable and reliable VPN services out in the world of VPNs. Plans start at about £1/year, that’s dirt cheap. Now, most of us wants to be connected to the Internet via a VPN for various reasons. However, in today’s connected world, each person owns about 2-3 devices on average that always stay connected to the Internet in which case, you will have to setup a VPN client on each one of those and handle connections individually. It is ofcourse a pain in the ass. Hence the only solution is to connect the router itself to the VPN so that all the devices behind the router are sending out and receiving the traffic via the VPN automatically. This tutorial helps people looking forward to setup their DD-WRT router to connect to VPN.sh service.

Why VPN.sh? Why not my own OpenVPN on a VPS?

I have been playing around with OpenVPN on Low End VPS boxes for quite some time. So, I know what I’m saying. Here are the reasons why I prefer VPN.sh to my own setup of OpenVPN –

I frequently experiment with my VPS. So, more scope for service disruption. Although I can rebuild the complete VPN service in under 5 min, the sheer number of times I experiment and fail, thereby resulting in unavailability of VPN service leads to the feeling that I should obtain VPN services from a provider for a cost.

Having VPS boxes in multiple locations is going to be expensive. Because it is only you who use your service (or may be some family – who expects money from family for providing a no guarantee service?) paying for those boxes across various locations on the globe is very expensive. And as there are numerous customers like you using services like VPN.sh, it is fairly profitable for them to procure new locations.

A single Low End VPS would cost about $2/year on the minimum side. When a service provider like VPN.sh offers a fairly good deal for £1/year ($1.52/year), with guarantee of reliability and multiple locations, I would definitely go for it.

Now, the downside to this is that, the bandwidth is fairly limited. VPN.sh gives you about 150-250GB for this plan, while you may get upto a TB of bandwidth on a VPS for $2/year. But, who cares when all you need is to connect via a VPS to do some browsing stuff or make VoIP calls?

Why DD-WRT?

You are here, means that you know what you are upto. Hence, I don’t think I need to explain you the powerful features of DD-WRT. For the sake of this tutorial, let me say that it has got a OpenVPN client built in and it is fairly easy to connect to the VPN at router level.

A quick guide on how to setup DD-WRT on your router can be found here.

Setting up OpenVPN Client on DD-WRT

Now is the time to set that VPN connection on your router.

  1. Log in to the administrative interface of DD-WRT (usually http://192.168.1.1; Default Username: root; Default Password: admin)
  2. Setup your internet connection normally as you would on a router
  3. Go to Administration -> Commands
  4. Modify this script as mentioned in-line
  5. Paste the modified script into the Command textarea
  6. Save it as a Startup Script
  7. Reboot the router

This script is run at the time of router booting and a VPN connection is established with VPN.sh. Let me know if you have any troubles.

Creating your own VPN Server using OpenVPN

We all face the need for a proxy at one time or the other during our day-to-day Internet lives. To bypass the college firewall, access that websites which is not available in your country, to stream Netflix or other region specific streaming services and the list goes on. So, the question comes now is why should we create our own VPN service when you could buy from a service provider. The simple answer is it is economical to do it on your own if you wish to spend an 30 minutes of your weekend. You can also use the entire port speed of the server to yourself. Also, it feels good to have your own VPN server without having to depend on anyone else!

Let’s start by looking at what we need to implement this project. We would need a VPS with Ubuntu installed from a provider that allows private VPNs on their servers and Nyr’s OpenVPN auto-installer script for Debian based systems. This will get you started.

I recommend NanoVZ for various reasons. They offer 128MB (this is more than enough for our VPN server), 3GB HDD and a Gigabit link with 500GB bandwidth a month for just $3.80 (3 EUR) a year. Pretty cheap, isn’t it? Now, the reason why I prefer this to LowEndSpirit is because of the Gigabit port.

Go to NanoVZ Website and signup for this server.

NanoVZ.png

After successful payment, you will receive your order confirmation, payment receipt and server information. Look into the email carefully. You will see something like –

IPInfo.png

This service doesn’t come with a dedicated IPv4 address, one of the cost-cutting strategy, thereby keeping their operating costs minimum and offer you a VPS at such a low price. However, you will share an IPv4 address with many other clients on their server. You will get 20 ports of the public IPv4 address NAT’ed to your internal IP address and a port specifically mapped to your SSH port on internal IP.

Connecting to your VPS via SSH

In order to configure your server to run OpenVPN server, you first need to login into the server via SSH. Here comes the need of the SSH port on external IP that is mapped to your VPS. Download a program like PuTTY and login into [Public_IPv4]:[Mapped_Port]. In the above case, I will login into [Masked_Public_IP]:15100.

Enabling TUN/TAP

TUN and TAP are virtual network kernel devices in computer networking. For the OpenVPN server to work, we need to enable these for our server. To do this, you will have to login into your SolusVM panel, again, whose details are given in the email that you received with your order.

SolusVM_Details.png

TUN_TAP.png

Installing OpenVPN Server

This is a fairly simple task. I would first recommend updating your server to the latest packages before beginning with configuring your server for VPN.

Go ahead and run the update command –

apt-get update

Once that is done, you just need to run the Nyr’s auto-install script for OpenVPN.

wget git.io/vpn --no-check-certificate -O openvpn-install.sh; bash openvpn-install.sh

The installer will ask you for the IP on which OpenVPN has to listen to. Give your internal IP address here. Check the email for this IP.

Local_IP.png

Then it will ask you to furnish the port number. Make sure the port number you give here is in the range of ports NAT’ed for you. In my case it is 15101 to 15120.

Port.png

The installer will now ask if you would like to enable port 53 also. I would recommend this as a no because you will anyway not be able to use this port as it is not NAT’ed directly from the outside world.

Port_53.png

Now enter the client certificate name that you want to store it as. This is like the username you want to access the server with. If you plan to share this with friends and family members, name the certificates accordingly to be able to identify each user.

client_name.png

That’s all it needs. Just hit Return key to proceed. It takes a while to finish the installation and certificate generation. Get yourself a small break!

It then asks for the external IP and since we are NAT’ed, we need to give the external IP provided by NanoVZ in the email.

External_IP.png

That’s it. Your certificate will be available at /root/[certificate_name].ovpn. Download this certificate using any SFTP program like WinSCP and use it with your favorite OpenVPN Client. I will do a write-up on comparison between various OpenVPN clients soon.

Let me know how it goes!

Hosting your WebSite on LowEndSpirit

For those of you who are not aware of LowEndSpirit (LES), I highly recommend that you visit LowEndSpirit Website. In short, LES offers Virtual Private Servers (VPS) for bare minimum prices. You can get a VPS in each of the locations (US, UK, NL, IT, JP) for  3 EUR each ($3.80) which is really very cheap. The configuration of these servers is about 128MB RAM (256MB in case of US location),  2GB or 3GB HDD (SSD drive in some locations), 100 Mb/s port in all locations, 50 – 500 GB data transfer cap (varies by location). The catch is you don’t get a dedicated IPv4 address with any of the servers. You get IPv6 addresses and a NAT’ed IPv4 address with 20 ports forwarded to your VPS.

Usually people spend tens to hundreds of dollars on shared hosting every year even when they have the basic skills to deploy a web server and maintain their own website/blog. When the website gains some public attention, the shared hosting provider suspends the hosting account for heavy usage of server resources. Usually, these providers throttle you when you hit 30% CPU usage continuously for 3 minutes. Also you are restricted to only 64MB of RAM. I, personally, feel that a LES VPS for $3.80/year is much better than a shared hosting that would cost at least $4/month, if you are willing to take that extra step of configuring your own server.

People who knew about LES would mostly use it for private VPN. For example, someone who lives in India uses a US LES VPS as an OpenVPN server and uses it to stream US only content streaming services like Netflix or Hulu. They usually don’t use it for hosting websites. One of the most prominent reason is because they LES doesn’t offer a dedicated IPv4 address. But, here is a solution to this problem. You can use your IPv6 address that is supplied to you. Here is how –

Deploy a webserver

You can deploy one of Nginx or Apache webserver software. There are numerous number of tutorials available to do this. Just do a Google search. The tricky part here is to make sure that the server listens to port 80 on IPv6 addresses. By default, these webserver software are configured to listen to IPv4 addresses only. You can do this in the following way:

For Nginx, open the file /etc/nginx/nginx.conf and uncomment the line that says –

listen [::]:80;

This will enable the server to listen on all IPv6 addresses available to that VPS. If you want the server to listen to a specific IPv6 addresses out of the many that are assigned, modify the line to look like –

listen [2607:f0d0:1004:2::2]:80;

Point your domain to CloudFlare DNS

Signup with CloudFlare if you don’t have an account already and point your domain to their DNS. Now, add an AAAA record to the DNS entries of your domain. Just like A records point to the server’s IPv4 address, AAAA records point to the IPv6 address of the server.

Enable automatic IPv6 in CloudFlare

This IPv6 gateway by CloudFlare enables users from non IPv4 networks also to access your website on IPv6. This option is available under CloudFlare Settings –> Settings Overview. Be sure to double check that CloudFlare is enabled for all the AAAA records (the grey clouds should turn flame red).

automatic-ipv6

Once this step is done, your website on IPv6 is good to go.

Let me know if things go wrong. May be I can suggest you some quick fixes.

Please note that LES doesn’t do backups or snapshots of your data. In the event that a server crashes due to hardware failure, your data is prone to be lost. Take proper measures to backup your data from time-to-time.

Why documenting?

I have been experimenting with many things over the past many years. In this process I've learned things which my school or my books didn't teach. And I know, for a reason, that there are many more people like me who are trying to learn things that I have learned already or trying to learn now. This learning process is time consuming. It's not straight enough like a question and an answer in the exam. Everything is an experiment. And experiments take a great deal of time and carry a risk factor with them. Even after spending numerous hours trying to do something, you still may end up not succeeding.

This is an effort to make things a bit straight for those of you who don't want to spend time experimenting, instead want to get things done. This comes with its own advantages and disadvantages. You will not be going through all those resources that I have been through to accomplish these tasks. Therefore you will not be gaining the same experience as I did through searching and learning all those things that are connected to that task that I want to accomplish. But then, the advantage is that you will not have to spend all the time that I have already spent researching about the task.

Anyway, one other motive behind this blog is that I will have documentation to stuff I did in the past or doing right now. In my experience, in a significant number of cases, I had to revisit my old stuff to get a current task done. However, my old stuff had no/little documentation and that too not in an organized manner. I, therefore, wanted a  way to document my learnings and revisit them whenever I want in the future.

With this, let me end the story behind my thoughts of starting a blog.

Get in touch if you have any questions on things that I have documented.