How to use Memcached with PyOWM

This is just a little demonstration on how you can quickly change the basic cache provider provided by the PyOWM library. For this purpose we’ll use Memcached, which – simply put – is a key/value in-memory data store: this turns it into a perfect caching mechanism. I’ve never used Memcached before writing this post: this shall be a good moment for me to get to know it. This demo requires that you work on a Linux env, as Memcached originally is shipped for Unix-like systems via packet distribution systems (but can nevertheless be compiled from source).

I’ll use Ubuntu, with Memcached 1.4.6 and PyOWM 0.4.0. Let’s dive into it.

First we install Memcached and the relative Python bindings:

sudo apt-get install memcached python-memcache

Then we install PyOWM library and check the installation:

sudo pip install pyowm
ls /usr/local/lib/python2.6/dist-packages # check installation

(in my distro, Python packages are installed by pip in the /usr/local/lib/python2.6/dist-packages folder: change accordingly to yours) In order to “plug” Memcached support into PyOWM we are going to leverage the installed Python bindings by creating an adapter class that can conform the SW interface that PyOWM expects into the Memcached API for getting/setting cache elements. Fortunately, the Memcached API is very close to the PyOWM expected interface (which is stated into the pyowm.abstractions.owmcache.OWMCache class), so we have chances that our adapter will be simple enough. Let’s name it “memcachedadapter.py“: you can put it anywhere, provided that this anywhere is “seen” by the Python intepreter: in example, you can put it into a folder listed into the PYTHONPATH variable or you can place it directly into the PyOWM install folder. I did the latter:

cd /usr/local/lib/python2.6/dist-packages/pyowm
sudo vim memcachedadapter.py

The module will contain the MemcachedAdapter class:

#!/usr/bin/env python

class MemcachedAdapter():
  """
  Realizes the pyowm.abstractions.owmcache.OWMCache interface
  adapting a memcache.Client object
  """
  __ITEM_LIFETIME_MILLISECONDS = 1000*60*10 # Ten minutes

  def __init__(self, hostname="127.0.0.1",
    port="11211", item_lifetime=__ITEM_LIFETIME_MILLISECONDS):
    from memcache import Client
    self._memcached = Client([hostname+":"+port])
    self._item_lifetime = item_lifetime

  def get(self, request_url):
    return self._memcached.get(request_url)

  def set(self, request_url, response_json):
    self._memcached.set(request_url, response_json, self._item_lifetime)

I wrote this adapter in 5 minutes, so please don’t blame me for errors 😉 It can surely be improved. Now what is left to do is to tell the PyOWM library how to use the adapter: this is done via configuration. The library uses OWMCache concrete instance which is created into a configuration file and injected into the code; a separate configuration file exist for the code supporting each Open Weather Map web API version. Currently only API version 2.5 is supported, so we’ll put our adapter into the pyowm.webapi25.configuration25.py file, commenting out the default cache object:

# Cache provider to be used
# cache = NullCache()  # <-- comment this line
from memcachedadapter import MemcachedAdapter
cache = MemcachedAdapter("127.0.0.1", "11211")

As you can see, we are adapting a local Memcached instance listening on the default 11211 port, but you can change this configuration as needed. Now let’s try it out – let’s start Memcached and use the PyOWM library:

memcached &
python

in example:

>>> from pyowm import OWM
>>> owm = OWM()
>>> f = owm.daily_forecast("London,uk")  # This first call to the API is not cached, obviously
>>> g = owm.daily_forecast("London,uk")  # This second call is cached

Time saving should be at a glance.

In a similar way it is possible to write adapters for plugging other cache/storage providers (Redis, MongoDB, etc..) into the PyOWM library.

EDIT: this post stimulated me to write more adapters, you can find them here.

Advertisements

How to convert VMWare virtual machines to Virtual Box

Where I started from

This is my situation: I’m on a Windows7 x86 host, and I have an old Ubuntu 10.04 virtual machine with VMWare Tools installed on it.

My need is to turn it into an OVF appliance, so that I can run it on Virtual Box, no matter where what architecture it will be run on.

Steps

This is what I’ve done:

  • I made sure (e.g: using VMWare Player or Workstation) that the virtual appliance is powered off;
  • Opened a Command Prompt, moved to the VMWare Player/Workstation installation dir and executed the OVF conversion tool. Be aware that this conversion may take some time, depending on how big is your VMX appliance. I did it as follows (replace the paths as needed):
cd "C:\Program Files\VMware\VMware Workstation\OVFTool"
ovftool.exe "C:\Users\claudio\Documents\Virtual Machines\ubuntu1004\ubuntu1004.vmx" "C:\Users\claudio\Documents\Virtual Machines\converted-to-virtualbox\ubuntu1004.ovf"
  • When the conversion process was over, I imported the “ubuntu1004.ovf” appliance into Virtual Box by using the “File” > “Import virtual appliance ..” menu element  and leaving all the defaults;
  • Then I booted up the “ubuntu1004.ovf” appliance and performed VMWare Tools uninstallation by opening up an SSH shell and executing:
sudo /usr/bin/vmware-uninstall-tools.pl
  • Then I finished the whole process by executing the “Device” > “Install Guest Additions” menu item: a virtual CD is then mounted and I launched the installation process from a shell
cd /media/VBOXADDITIONS_4.2.12_84980
sudo bash VBoxLinuxAdditions.run

 

Not as difficult as it may seem…. Hope this helps!

How to install MongoDB on Ubuntu

This is a quick-guide to install MongoDB on Ubuntu 12.04 (if you want to know how to install it also on Fedora Core 11, please check out my old post.

Here is all you have to do:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
sudo touch /etc/apt/sources.list.d/10gen.list
line="$(echo "deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen")"
sudo echo "$line" > /etc/apt/sources.list.d/10gen.list
sudo apt-get update

And then you can install MongoDB with

sudo apt-get install mongodb-10gen

Enjoy!

The yes command: “Aye, sir!”

The dumbest “yes-man” Linux command: yes!

It has only one aim: to indefinitely repeat what you tell it. Useful if pipelined with interactive cmd line installers (where you usually have to type “y” a lot of times in order to accept the default values) and similar borying operations.

So, for example if you go with:

yes test

it keeps on printing the cmd line arguments followed by a new line character (CTRL+C to stop execution):

test
test
test
[...]

…and if you do not specify any argument, it just says:

y
y
y
y
[...]

Yes, man! 🙂

How to install MongoDB on Fedora

During the last days, I’ve been delving into NoSQL datastores study and now I’ve got the chance to use MongoDB for a real-life need (of course, something relating to work issues). In order to reach my target, I needed a fresh installation of one of the most promising NoSQL technologies: the document-oriented datastore MongoDB.

Here is how I managed to install MongoDB 2.0.3 on a Fedora Core 11 host. You can easily adapt the steps I went through to your operational environment.

We’re about to issue every command as root user.

First, I downloaded, extracted and placed the Mongo stuff (I didn’t use 10gen repo, but just a tar.gz from MongoDB website):

cd /opt
wget http://fastdl.mongodb.org/linux/mongodb-linux-i686-2.0.3.tgz
tar xvf mongodb-linux-i686-2.0.3.tgz
mv mongodb-linux-i686-2.0.3.tg mongodb-2.0.3

I decided that my databases would go under /usr/data/mongodb and that MongoDB log file would be /var/log/mongodb.log

As I wanted my MongoDB server instance to be started/stopped as a demon, I prepared the following init.d script named “mongodb” and placed it under /etc/init.d/ :

#!/bin/bash
source /etc/rc.d/init.d/functions
prog="mongod"
mongod="/opt/mongodb-2.0.3/bin/mongod"
RETVAL=0

start() {
  echo -n $"Starting $prog: "
  #This is the fundamental call to start the MongoDB server instance
  daemon $mongod "--fork --journal --dbpath /usr/data/mongodb \
  --logpath /var/log/mongodb.log \
  --logappend 2&gt;&amp;1 &gt;&gt;/var/log/mongodb.log"
  RETVAL=$?
  echo
  [ $RETVAL -eq 0 ] &amp;&amp; touch /var/lock/subsys/$prog
  return $RETVAL
 }

stop() {
  echo -n $"Stopping $prog: "
  killproc $prog
  RETVAL=$?
  echo
  [ $RETVAL -eq 0 ] &amp;&amp; rm -f /var/lock/subsys/$prog
  return $RETVAL
}

case "$1" in
 start)
  start
  ;;
 stop)
  stop
  ;;
 restart)
  stop
  start
  ;;
 status)
  status $mongod
  RETVAL=$?
  ;;
 *)
  echo $"Usage: $0 {start|stop|restart|status}"
  RETVAL=1
 esac

exit $RETVAL

Then, I opened my Iptables firewall’s INPUT chain so that port 27017 (the port MongoDB server is listening to) is not blocked: I opened the file /etc/sysconfig/iptables and added the following rule before of the COMMIT statement:

-A INPUT -p tcp -m tcp -m multiport --ports 27017 -j ACCEPT

and restarted iptables with:

service iptables restart

That’s it. Finally, I started the server instance with:

service mongodb start

and tested the whole thing opening the Mongo Javascript shell like this:

cd /opt/mongodb-2.0.3/bin
./mongo

and everything was fine.