How to use Memcached with PyOWM

This is just a little demonstration on how you can quickly change the basic cache provider provided by the PyOWM library. For this purpose we’ll use Memcached, which – simply put – is a key/value in-memory data store: this turns it into a perfect caching mechanism. I’ve never used Memcached before writing this post: this shall be a good moment for me to get to know it. This demo requires that you work on a Linux env, as Memcached originally is shipped for Unix-like systems via packet distribution systems (but can nevertheless be compiled from source).

I’ll use Ubuntu, with Memcached 1.4.6 and PyOWM 0.4.0. Let’s dive into it.

First we install Memcached and the relative Python bindings:

sudo apt-get install memcached python-memcache

Then we install PyOWM library and check the installation:

sudo pip install pyowm
ls /usr/local/lib/python2.6/dist-packages # check installation

(in my distro, Python packages are installed by pip in the /usr/local/lib/python2.6/dist-packages folder: change accordingly to yours) In order to “plug” Memcached support into PyOWM we are going to leverage the installed Python bindings by creating an adapter class that can conform the SW interface that PyOWM expects into the Memcached API for getting/setting cache elements. Fortunately, the Memcached API is very close to the PyOWM expected interface (which is stated into the pyowm.abstractions.owmcache.OWMCache class), so we have chances that our adapter will be simple enough. Let’s name it “memcachedadapter.py“: you can put it anywhere, provided that this anywhere is “seen” by the Python intepreter: in example, you can put it into a folder listed into the PYTHONPATH variable or you can place it directly into the PyOWM install folder. I did the latter:

cd /usr/local/lib/python2.6/dist-packages/pyowm
sudo vim memcachedadapter.py

The module will contain the MemcachedAdapter class:

#!/usr/bin/env python

class MemcachedAdapter():
  """
  Realizes the pyowm.abstractions.owmcache.OWMCache interface
  adapting a memcache.Client object
  """
  __ITEM_LIFETIME_MILLISECONDS = 1000*60*10 # Ten minutes

  def __init__(self, hostname="127.0.0.1",
    port="11211", item_lifetime=__ITEM_LIFETIME_MILLISECONDS):
    from memcache import Client
    self._memcached = Client([hostname+":"+port])
    self._item_lifetime = item_lifetime

  def get(self, request_url):
    return self._memcached.get(request_url)

  def set(self, request_url, response_json):
    self._memcached.set(request_url, response_json, self._item_lifetime)

I wrote this adapter in 5 minutes, so please don’t blame me for errors ūüėČ It can surely be improved. Now what is left to do is to tell the PyOWM library how to use the adapter: this is done via configuration. The library uses OWMCache concrete instance which is created into a configuration file and injected into the code; a separate configuration file exist for the code supporting each Open Weather Map web API version. Currently only API version 2.5 is supported, so we’ll put our adapter into the¬†pyowm.webapi25.configuration25.py¬†file, commenting out the default cache object:

# Cache provider to be used
# cache = NullCache()  # <-- comment this line
from memcachedadapter import MemcachedAdapter
cache = MemcachedAdapter("127.0.0.1", "11211")

As you can see, we are adapting a local Memcached instance listening on the default 11211 port, but you can change this configuration as needed. Now let’s try it out – let’s start Memcached and use the PyOWM library:

memcached &
python

in example:

>>> from pyowm import OWM
>>> owm = OWM()
>>> f = owm.daily_forecast("London,uk")  # This first call to the API is not cached, obviously
>>> g = owm.daily_forecast("London,uk")  # This second call is cached

Time saving should be at a glance.

In a similar way it is possible to write adapters for plugging other cache/storage providers (Redis, MongoDB, etc..) into the PyOWM library.

EDIT: this post stimulated me to write more adapters, you can find them here.

How to convert VMWare virtual machines to Virtual Box

Where I started from

This is my situation: I’m on a Windows7 x86 host, and I have an old Ubuntu 10.04 virtual machine with VMWare Tools installed on it.

My need is to turn it into an OVF appliance, so that I can run it on Virtual Box, no matter where what architecture it will be run on.

Steps

This is what I’ve done:

  • I made sure (e.g: using VMWare Player or Workstation) that the virtual appliance is powered off;
  • Opened a Command Prompt, moved to the VMWare Player/Workstation installation dir and executed the OVF conversion tool. Be aware that this conversion may take some time, depending on how big is your VMX appliance. I did it¬†as follows (replace the paths as needed):
cd "C:\Program Files\VMware\VMware Workstation\OVFTool"
ovftool.exe "C:\Users\claudio\Documents\Virtual Machines\ubuntu1004\ubuntu1004.vmx" "C:\Users\claudio\Documents\Virtual Machines\converted-to-virtualbox\ubuntu1004.ovf"
  • When the conversion process was over, I imported the “ubuntu1004.ovf” appliance into Virtual Box by using the “File” > “Import virtual appliance ..” menu element ¬†and leaving all the defaults;
  • Then I booted up the “ubuntu1004.ovf” appliance and performed VMWare Tools uninstallation by opening up an SSH shell and executing:
sudo /usr/bin/vmware-uninstall-tools.pl
  • Then I finished the whole process by executing the “Device” > “Install Guest Additions” menu item: a virtual CD is then mounted and I launched the installation process from a shell
cd /media/VBOXADDITIONS_4.2.12_84980
sudo bash VBoxLinuxAdditions.run

 

Not as difficult as it may seem….¬†Hope this helps!

How to deploy Flask applications to Apache webserver

This is a simple guide explaining how I managed to configure Apache 2.2 httpd server on a Windows platform so that it can serve a Python webapplication I wrote using the Flask micro-framework. The guide is valid, with a very little modification, also on Linux environments (you geeks know how to do)

Why I needed to to this

I developed this application at work and ‘ve been serving it from the beginning via the Flask’s built-in minimal webserver: unfortunately this ¬†is not enough for production stage as I need a more robust server with SSL capabilities, which Flask’s has not. This was my first time in deploying a Python webapp…So, after googling a bit and reading the Flask deployment notes, I came up with the answer: what I needed was¬†a WSGI-compliant server running on my target platform, a Windows 2012 server. The natural choice to me was¬†to enable the WSGI module on the “good ole” Apache webserver, which I’m experienced with.

Steps

Flask app

We choose a folder in which we place the Python code. For instance,

D:\webapps\test

In this folder we create the real Flask webapplication that we want to¬†deploy¬†(file “test.py”):

from flask import Flask, request
app = Flask(__name__)

@app.route('/hello')
def hello_world():
    name = request.args.get('name','')
    return 'Hello ' + name + '!'
if __name__ == '__main__':
    app.run()

The Apache server won’t be aware of “test.py” at all. What you need to do now is to write in the same folder a Python file named “test.wsgi” that we will link into the webserver’s configuration: the code in this file will import the main Flask application object (built in our case as a singleton) and will be actually executed by the WSGI module of Apache. In the code, it is vital that you DON’T change the name of the “application” variable, as it is exactly what the server expects to find. Also please note that we are extending the Python classes path to include our own webapplication’s folder. This is “test.wsgi”:

import sys

#Expand Python classes path with your app's path
sys.path.insert(0, "d:/webapps/test")

from test import app

#Put logging code (and imports) here ...

#Initialize WSGI app object
application = app

As an additional remark, if you want to put any logging code (e.g: file/e-mail/console loggers) into your Flask app, you must put it before the if __name__ == ‘__main__’ block, otherwise it won’t log anything! Add your loggers to the app object.

Apache setup

Ok, what’s next? Now it’s all about installing and properly configuring Apache.

First: install Apache webserver. I downloaded and executed the .msi installer. Apache was installed at

"C:\Program Files (x86)\Apache Software Foundation\Apache2.2"

Second: install the WSGI Apache module. Pay attention to download the module compiled for your specific combination of platform and Python and Apache versions: I downloaded this module. Once downloaded, rename the .so file into “mod_wsgi.so” and put it under the “modules” subfolder of your Apache installation folder. Then you have to tell Apache to use it: open in a text editor the “httpd.config” file which is under the “conf” subfolder and add the following line at the bottom:

LoadModule wsgi_module modules/mod_wsgi.so

Third: restart Apache.

Now Apache is ready to serve WSGI webapplications. What is left is to tell about where our application is and match it to a URL alias. It’s child’s play: open in a text editor the “httpd.config” file we used before and add these lines to the bottom:

<Directory d:/webapps/test>
    Order allow,deny
    Allow from all
</Directory>
WSGIScriptAlias /flasktest d:/webapps/test/test.wsgi

(nevertheless, I prefer to place the per-virtual-host or per-alias configurations’ stuff into separate files and then use an Include directive into the main “httpd.conf”)

Now restart Apache again and if you open a browser and point it to:

http://localhost/flasktest/hello?name=claudio

and you should see the webapp’s greetings!

Further references

  • This guide helped me a lot in understanding how to setup Apache WSGI.
  • I also found this tutorial which is far more comprehensive than mine and covers Flask deployment on Apache on Debian/Ubuntu environments

How to install MongoDB on Windows as a service

I’m showing you a quick procedure to install MongoDB as a service on Windows platforms (I simply gathered the hints explained in the official documentation page¬†and adapted the whole stuff to my specific case).

In my example, I’m installing MongoDB version 2.2.0 on a Windows Server 2003 R2 machine and my goal is to have it available as a service.

The procedure is pretty straightforward: all you need is to setup the path in which MongoDB will physically store the data collections, to provide a logfile for the service we’re going to create and finally to tell the mongod daemon to run as a service.

Here we go (be sure to enclose all the paths in double quotes if they contain spaces):


:: 1. Download the installation package (a .zip archive) and decompress it into folder C:\mongodb-2.2.0\

:: 2. Create the data path folders (in my example, C:\mongodb-2.2.0\data\db):
md C:\mongodb-2.2.0\data
md C:\mongodb-2.2.0\data\db
set datapath = C:\mongodb-2.2.0\data\db

:: 3. Setup mongod configuration file path (my file is C:\mongodb-2.2.0\mongod.cfg):
set configpath = C:\mongodb-2.2.0\mongod.cfg

:: 4. Create the folder that will host the service's log file (my folder is C:\mongodb-2.2.0\log) and write its path into the config file:
md C:\mongodb-2.2.0\log
echo logpath = C:\mongodb-2.2.0\log\mongod.log > C:\mongodb-2.2.0\mongod.cfg

:: 5. install mongod as a service:
C:\mongodb-2.2.0\bin\mongod.exe --config %configpath% --dbpath %dbpath% --install

Now you can start/stop/remove the MongoDB service via the services administration graphical interface, or via the command line using the following commands:


:: Start service
net start MongoDB

:: Stop service
net stop MongoDB

:: Uninstall the service
C:\mongodb-2.2.0\bin\mongod.exe --remove

Wanna test out your installation? Just call the MongoDB Javascript shell:


C:\mongodb-2.2.0\bin\mongo.exe

and if no error message appears – have fun!

Installing Android 4.1 Jelly Bean on LG Optimus One (LGP500)

A Happy LG Optimus One owner’s story

I’m a happy owner of an LG Optimus One: not only because it proved many times to be a performing device, but also because – a few days after I bought it – I managed to replace the custom Android it had on board (themed by TIM: ugly!!!) with a simple and plain Android 2.2.1 Froyo stock ROM. This I did, despite many warnings I read about phone brick and license invalidation. About six-months after, I had the same fever and replaced Froyo with Gingebread: no problems this time, as well.

Last monday, after reading that someone (Android Open Kang Project team) built an unoffical Android 4.1 Jelly Bean port for LG Optimus One, I took a fancy to it…”Danger is my business” – I thought. So, let’s go replace my current Android with the one on the crest of waves!!!

I am reporting the steps I followed: they took about 3 hours, considering also the conclusive phone re-configuration and apps re-install.

Result: a fresh and fluid install on my Optimus One!

READ THIS: DEATH DANGER ūüėČ

Please, don’t take what you’re gonna read as a serious guide to modding (you can find tons of forum posts on sites such as xda): the following steps were just conceived to provide a big and fuzzy idea of what it takes to perform such an install, it’s up to you to search for details!

You’re warned: I won’t be responsible for any danger to your phone or parts of it in case you follow the procedure and come up with a brick or malfunctionings!

PROCEDURE

    1. Copied Android 4.1 ROM and gapps .zip archives into the root folder of the SD card
    2. Installed ROM Manager via Google Play store
    3. Backupped all applications using Titanium Backup and then copied backups on PC
    4. Backupped personal data and contacts on PC using LG PC Suite 4
    5. Backupped current ROM using ROM manager and then copied backup on PC
    6. Installed ClockworkMod using ROM manager
    7. Rebooted phone in recovery mode
    8. Made a factory reset/wipe
    9. Flashed the Android 4.1 ROM from the .zip archive on the SD card
    10. Flashed the gapps from the .zip archive on the SD card
    11. Rebooted phone

Hope this helps for all of you who whish to try out Jelly Bean on LG Optimus One!

How to install MongoDB on Ubuntu

This is a quick-guide to install MongoDB on Ubuntu 12.04 (if you want to know how to install it also on Fedora Core 11, please check out my old post.

Here is all you have to do:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
sudo touch /etc/apt/sources.list.d/10gen.list
line="$(echo "deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen")"
sudo echo "$line" > /etc/apt/sources.list.d/10gen.list
sudo apt-get update

And then you can install MongoDB with

sudo apt-get install mongodb-10gen

Enjoy!

How to install MongoDB on Fedora

During the last days, I’ve been delving into NoSQL datastores study and now I’ve got the chance to use MongoDB for a real-life need (of course, something relating to work issues). In order to reach my target, I needed a fresh installation of one of the most promising NoSQL technologies: the document-oriented datastore MongoDB.

Here is how I managed to install MongoDB 2.0.3 on a Fedora Core 11 host. You can easily adapt the steps I went through to your operational environment.

We’re about to issue every command as root user.

First, I downloaded, extracted and placed the Mongo stuff (I didn’t use 10gen repo, but just a tar.gz from MongoDB website):

cd /opt
wget http://fastdl.mongodb.org/linux/mongodb-linux-i686-2.0.3.tgz
tar xvf mongodb-linux-i686-2.0.3.tgz
mv mongodb-linux-i686-2.0.3.tg mongodb-2.0.3

I decided that my databases would go under /usr/data/mongodb and that MongoDB log file would be /var/log/mongodb.log

As I wanted my MongoDB server instance to be started/stopped as a demon, I prepared the following init.d script named “mongodb” and placed it under /etc/init.d/ :

#!/bin/bash
source /etc/rc.d/init.d/functions
prog="mongod"
mongod="/opt/mongodb-2.0.3/bin/mongod"
RETVAL=0

start() {
  echo -n $"Starting $prog: "
  #This is the fundamental call to start the MongoDB server instance
  daemon $mongod "--fork --journal --dbpath /usr/data/mongodb \
  --logpath /var/log/mongodb.log \
  --logappend 2&gt;&amp;1 &gt;&gt;/var/log/mongodb.log"
  RETVAL=$?
  echo
  [ $RETVAL -eq 0 ] &amp;&amp; touch /var/lock/subsys/$prog
  return $RETVAL
 }

stop() {
  echo -n $"Stopping $prog: "
  killproc $prog
  RETVAL=$?
  echo
  [ $RETVAL -eq 0 ] &amp;&amp; rm -f /var/lock/subsys/$prog
  return $RETVAL
}

case "$1" in
 start)
  start
  ;;
 stop)
  stop
  ;;
 restart)
  stop
  start
  ;;
 status)
  status $mongod
  RETVAL=$?
  ;;
 *)
  echo $"Usage: $0 {start|stop|restart|status}"
  RETVAL=1
 esac

exit $RETVAL

Then, I opened my Iptables firewall’s INPUT chain so that port 27017 (the port MongoDB server is listening to) is not blocked: I opened the file /etc/sysconfig/iptables and added the following rule before of the COMMIT statement:

-A INPUT -p tcp -m tcp -m multiport --ports 27017 -j ACCEPT

and restarted iptables with:

service iptables restart

That’s it. Finally, I started the server instance with:

service mongodb start

and tested the whole thing opening the Mongo Javascript shell like this:

cd /opt/mongodb-2.0.3/bin
./mongo

and everything was fine.