MongoDB @ KhojKhabar

These days NoSQL databases are getting very popular. These databases don’t have a SQL interface for querying data and a lot of times don’t store data as collections of rows and tables. MongoDB is one such database that stores data in form of BSON(a form of JSON) documents and collections. The data in MongoDB can be queried using a REST API. It has a .Net driver available and there are a couple of hosted MongoDB cloud hosted providers with a free plan so I thought I will try to convert KhojKhabar data store from SQLite to MongoDB.

Out of MongoHQ and MongoLab I decided to use MongoLab as it allows 240 MB (vs 16 MB by MongoHQ). MongoLab also provides an option to host your MongoDB instance on Amazon or Rackspace cloud. So if your app is hosted on one of these providers you can reduce latency by hosting MongoDB on the same provider. Moving data storage of khojkhabar to MongoDB was not that difficult as the driver usage is like a combination of dataset style programming and fluid function calls (like used in subsonic 2), so someone with experience these can pick it up easily. There is also a good tutorial available at MongoDb.org, which was very helpful. Now the live khojkhabar site is using MongoLab as the data store.In khojkhabar right now I store the news data of last 10 days only so I think 240 MB storage quota should be fine for now.

AppHarbor – Heroku like service for ASP.Net

I recently came across AppHarbor. It looks like another interesting startup to come out of  YCombinator. Its a Heroku like service that builds, runs unit tests and deploys your ASP.Net code for you. It works with Git so to upload the code all you need to do is git push. They provide shared and dedicated databases with option of MS SQL server and MySQL.  Whats cool is that they provide a free application instance per applicaiton. So you pay only if your app grows to need another instance. Now if you need to test or host your next ASP.Net MVC app you know where to go.

Tonight I plan to play with it and see how it works for an ASP.Net MVC app I have been working on using Subsonic and SQLite. Details later.

jQuery ListCollapse

This plugin is meant to collapse/expand a list of items that grows beyond certain predefined number of items. It filters and collapses a list when data is bound, added or removed. If number of items are more than specified number n it displays first n items and displays an expand(customizable) link. When a user clicks on this link it displays the entire list and changes this link to collapse(customizable).

Download listcollapse

(Note: This plugin is based on Collapsorz 1.1 created by Aaron Kuzemchak. I have customized it to be able to be called whenever an item is added or removed and some minor modifications is think were required for dynamic list where items are added and removed on the fly)

Usage

$(listelement).listcollapse();

$(listelement).listcollapse(options);

Options

(All parameters are optional)

toggle

To select elements that you wish to collapse. By default it will collapse all direct children of the list selected.

maximum

Maximum number of elements to show. Collapses all items beyond this number. It defaults to 5.

showText

Text to be shown for the expand link. It defaults to Show.

hideText

Text to be shown for the collapse link. It defaults to Hide.

linkLocation

Whether to display expand/collapse link before or after the selected list. You can choose one of the following settings:

• after—Places the link after the list. This is the default setting.
• before—Places the link before the list.

defaultState

Whether the list should be displayed expanded or collapsed by default.

• collapsed—List will be collapsed by default. This is the default setting.
• expanded—List will be expanded by default.

wrapLink

HTML to wrap around the link.

Load balancing

I read an interesting article today about load balancing. Its a must read for anyone looking to scale a website or anyone just simple interested in knowing how these high traffic websites scale on the web server side. This documents explains different ways of doing it like hardware based approach and software based approach. It also explains various related factors like cookie persistence and using SSL certs in load balancing.

http://www.exceliance.fr/en/ART-2006-making%20applications%20scalable%20with%20LB.pdf

 

 

Moving postgres data folder

Our postgres server ran out of disk space on our EC2 server so we figured we should move the postgres data folder to a new bigger volume. We created an additional volume mounted to /disk2 and then moved the data folder as follows -

1. Login to shell as root. Stop the Postgres if running

$service postgresql-9.0 stop

2. Copy the data folder to new location

$cp -R /var/lib/pgsql/9.0/data  /disk2/pgdata/

3. Modify postgres startup script to point to new data directory

In  /etc/init.d/postgresql  file,  change the value of  PGDATA  variable to new location  which is /disk2/pgdata/data

4. Start the db server

$ service postgresql start

 

Adding a new volume to EC2 linux instance

Our postgres EC2 linux instance ran out of disk space so I had to add another volume to the server. Following is what I did -

1. Login to http://aws.amazon.com and go to EC2 tab.
2. Go to volumes section and create a new volume.
3. Enter the size of volume and select the availability zone. I generally skip creating a snapshot unless there is explicit need.
4. Once the volume is created and its status is available, select it and click attach button. Select instance id of the instance to which it needs to be associated and map it to /dev/sda2
5. Login to the server as root and execute following commands -

       mkfs -t ext3 /dev/sda2   #dont do it if creating from an existing snapshot
       echo "/dev/sda2  /disk2  ext3     noatime  0 0" >> /etc/fstab
       mkdir /disk2
       mount /disk2
       df -h #you should now see /disk2 as a new volume

Setting up replication in Postgres 9.0

For picksie I had to set up streaming replication in Postgres 9.0. We have a master server which is used for read/write and a hot standby where we want the data to updated using streaming replication. Following is a summary of what I did

1. Install postgres on both master and standby

2. Allow standby server to connect to the master by editing pg_hba.conf

host replication postgres 192.168.0.20/22 trust

3. Setup streaming on master by updating postgresql.conf

wal_level = hot_standby

max_wal_senders = 5

wal_keep_segments = 32

archive_mode = on

archive_command = ‘cp %p /path_to/archive/%f’

4. Start postgres on master

5. Make base backup on master

$ psql -c “SELECT pg_start_backup(‘label’, true)”

$ rsync -a ${PGDATA}/ standby:/srv/pgsql/standby/ –exclude postmaster.pid

$ psql -c “SELECT pg_stop_backup()”

6. Do authentication and streaming related configuration in postgresql.conf and pg_hba.conf in standby server so that when needed it can be promoted as master with less effort.

7. Set standby server as a hot standby by updating postgresql.conf

hot_standby = on

8. Create recovery.conf with following configuration on standby server in same folder as postgresql.conf

standby_mode = ‘on’

primary_conninfo = ‘host=192.168.0.10 port=5432 user=postgres’

trigger_file = ‘/path_to/trigger’

restore_command = ‘cp /path_to/archive/%f “%p”‘

Note: trigger file is the file, presence of which will make the standby server to stop replication and failover

9. Start postgres service on standby and it will start replication

 

Reference: http://wiki.postgresql.org/wiki/Streaming_Replication

$ $EDITOR postgresql.conf

listen_addresses = '192.168.0.10'

$ $EDITOR pg_hba.conf

# The standby server must have superuser access privileges.
host  replication  postgres  192.168.0.20/22  trust

Interesting projects at work

Cloud, iPad and SAAS are some of the most sought after buzz words of software industry right now.  Apart from looking after the IT infrastructure, my work involves projects bridge and picksie that together involve all of these.

Bridge is a product targeting enterprise collaboration space, built on my stack of choice for web apps these days ASP.Net/Subsonic/MySQL. We are looking into SAASifying this app and see if we can host it on the cloud using some knowledge that I have gained working on picksie.

Picksie is a web app for iPad, hosted on the cloud. My involvement is designing the infrastructure, making sure it is scalable and easily managable. Infrastructure of a consumer web app is really a completely different ballgame compared to enterprise IT infrastructure. So I am really getting to learn alot as picksie moves from demo to beta and from a simple 3 server setup to a more elaborate and load balanced and more elaborate setup.