2

Efficient Web programing with Erlang and Chicago Boss

This is the first post in a series to help spread the words erlang and Boss.

In the last 4 months I have been learning Erlang and I could start working with a web framework called Chicago Boss, Its simplicity and efficiency made me fall in love at first sight.

First I need to say that to develop simple web apps in Chicago Boss it is not necessary to know much about Erlang (just as ruby for rails). Chicago Boss was largely inspired by Rails,  so, many concepts will sound familiar if you come from Rails (MVC), the main features Chicago Boss provide are:

I have written simple applications with response times of 10-20 milliseconds without optimization. One of the things I like about Erlang and Boss is no magic, you never lose control of your application.

The key developer of Chicago Boss is Evan Miller, also key developer of the template engine erlydtl and other interesting projects, I had the opportunity of contribute recently with two core modules (Sessions and Flash), an excellent way to learn erlang.

Enough talk, if you need taste an example app, take an hour and follow this excelent tutorial (a bit outdated) .

I’m preparing a tutorial with a full application example, so, stay tunned.

2

Amazon EC2 Introduction – Useful info and tricks

I’m sure many people think the cloud is the future, and more or less have migration plannings. Also, some of these people point to Amazon as supplier for their price, stability and capabilities. This first post of EC2 are things you need to know from the standpoint of a sysadmin.

For a System Administrator, Amazon EC2 is a big challenge,  mostly because their ephemeral nature:

Storage:

  • When you starts and instance (from and image or AMI) EC2 attach a disk in /mnt, the size depends of the instance type (150 GB for the small instance type)
  • adminsis@ec2instance:~# df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda1             9.9G  2.6G  6.9G  27% /
    tmpfs                 854M     0  854M   0% /lib/init/rw
    varrun                854M   64K  854M   1% /var/run
    varlock               854M     0  854M   0% /var/lock
    udev                  854M   80K  854M   1% /dev
    tmpfs                 854M     0  854M   0% /dev/shm
    /dev/sda2             147G  188M  140G   1% /mnt
    
  • All the default storage is volatil, as the overall instance. you can reboot the instance but if you shutdown, all the modifications will be lost.
  • Use EBS (Elastic Block Store) to create persistent storage for your Instances (also you will gain with snapshoots of the file system)
  • ## Create Volume new EBS volume of 5GB in Europe)
    #ec2-create-volume [--size <code>size | --snapshot snapshot] --availability-zone zone
    ec2-create-volume -s 5 -z eu-west-1b
    ## Attach it on a device (/dev/sdf) on your instance:
    ec2-attach-volume vol-aaaaaa -i i-bbbbbb -d /dev/sdf
    ## Create the file system
    mkfs.ext3 /dev/sdf
    ## And mount
    mkdir /srv
    mount -t ext3 /dev/sdf /srv
    
  • Now you will have and persistent storage on /srv that will survive to instance live. If your instances goes down, launch other, associate the volume to this instance and mount it.
  • adminsis@ec2instance:~# df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda1             9.9G  2.6G  6.9G  27% /
    tmpfs                 854M     0  854M   0% /lib/init/rw
    varrun                854M   64K  854M   1% /var/run
    varlock               854M     0  854M   0% /var/lock
    udev                  854M   80K  854M   1% /dev
    tmpfs                 854M     0  854M   0% /dev/shm
    /dev/sda2             147G  188M  140G   1% /mnt
    /dev/sdf              5.0G  1.2G  3.6G  24% /srv
    

Networking

  • Each instance have only a network interface (eth0) configured by dhcp in the Amazon “LAN”.
  • This ethernet is NAT’ed to an public dhcp address (dns-based)
  • This NAT is firewalled with Security Groups, by default all traffic is closed and you open the necessary ports (tcp, udp, icmp) in groups of rules that can be reutilized.
  • The Amazon DNS assings a public DNS name to each intance, and:
    • Called outside the EC2 infrastructure will return the public ip
    • Called inside the EC2 infrastructure will return the private ip
  • Elastic ip’s are public fixed ip addresses that can be assigned to instances. The current IPv4 limitations
    make amazon decided to limit the number of Elastic ip’s to 5.

If your application needs a fixed configuration, ex. webservers database servers names or dhcp with a non-controlled dns server is not for you, you can install a dns server and autoregister the instances on boot.

#!/bin/bash
cat << EOF | /usr/bin/nsupdate -k Kyourdomain.com.+157+46088.private -v
server 127.0.0.1
zone yourdomain.com
update delete $1 A
update add $1 60 A $2
show
send
EOF

Parametrized Launches

When you creates an instance, you can pass parameters that you can programmatically retrieve on instance startup to make bootstraping, configuration, deploy, … task.

In this article from amazon you can follow this great step by step tutorial, by example you can:

Instance launch with an example parameter

ec2-run-instances $AMI_ID --region eu-west-1 -n 1 -g $SEC_GROUP -t $INSTANCE_TYPE -k $KEY -z eu-west-1b -d webserver

Note the -d parameter “webserver”, when the instance has started you can call the Amazon EC2 webservice as follows:

#curl http://169.254.169.254/1.0/user-data
--- webserver

You can think that this is good but not “impressive”, but without this the instance autoconfiguration is a lot harder to implement, think as the begin of the history:

  • Instance Launch (parameters, ex. instance role on your system, config files to execute, …)
  • Have an EC2 AMI that on startup (init.d/rc.local) read the parameters and starts the autoconfiguration
  • Automate everything

Also you allways can get the instance metadata from the EC2 webservices, from the instance you can by example:

LOCAL_IP=`curl http://169.254.169.254/latest/meta-data/local-ipv4`
echo "The local ip of this instance is "$LOCAL_IP
PUBLIC_IP=`curl http://169.254.169.254/latest/meta-data/public-ipv4`
echo "The public ip of this instance is "$PUBLIC_IP
echo "This is the list of metas"
curl http://169.254.169.254/latest/meta-data/
ami-id
ami-launch-index
ami-manifest-path
ancestor-ami-ids
block-device-mapping/
hostname
instance-action
instance-id
instance-type
kernel-id
local-hostname
local-ipv4
placement/
public-hostname
public-ipv4
public-keys/
ramdisk-id
reservation-id
security-groups

Soon I will write some posts on the automation arena and EC2.

0

Great news for the Spanish eID expansion

Short note:

Tractis in collaboration with Red.es are organizing a campaign to distribute more than 300,000 electronic ID readers to citizenship.

More info in ther blog.

5

Mysql triggers, a practical example


This post is intended to be an introduction and a real example of how triggers can be a powerfull tool for programers and integrators, with a real solution for a real problem (manage tags without changing the source code of an app).

If you has worked with Oracle likely to have used triggers and pl/sql, but in Mysql this feature available from version 5 (1) isn’t widely use, why?

When you are programing or integrating Mysql backed applications, you always have to keep it in mind; Triggers are your friends. Many times you have external applications to manage that source code can’t be touched (not own, hard to maintain if patched, different technologies, closed source, …) or simply you need integrated data between apps and don’t have api’s, webservices or when performance is a big deal.

Triggers are not sustitute to large developments, to much sql code can be a hell in order to have a well mantained/structured code and can difficult your testing strategy.

Problem:
We need manage tags in an application that we can’t touch the source code.

Solution:
We will use an existing text field in a form and transparently convert it in a tag field.

How:
Without change the code (at database level), we will split the comma separated text field, insert it in a tags table and link the tag to the record.

Background
Current example table, we are going to extend the field `other`as tag field, we name it products:

CREATE TABLE `products` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `name` varchar(50) NOT NULL DEFAULT '',
  `price` int(20) NOT NULL DEFAULT '0',
  `other` varchar(50) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `products_name_idx` (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

Tags Management tables: freetags and freetagged_objects:

CREATE TABLE `freetags` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `tag` varchar(50) NOT NULL DEFAULT '',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `freetagged_objects` (
  `tag_id` int(20) NOT NULL DEFAULT '0',
  `object_id` int(20) NOT NULL DEFAULT '0',
  `tagged_on` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `module` varchar(50) NOT NULL DEFAULT '',
  PRIMARY KEY (`tag_id`, `object_id`),
  KEY `freetagged_objects_tag_id_object_id_idx` (`tag_id`, `object_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

Now we will create two triggers (2), one for insert and other for updates, they will split the coma separated words in the `other` field on product table and manage the tags tables (insert the tag on the freetags table if not already, and link the product id to the tag on the freetagged_objects). You can use `module` field to have tags for differents tables/models.

INSERT_PRODUCTS_TAGS

DELIMITER ||
DROP TRIGGER IF EXISTS insert_products_tags;
||
DELIMITER @@
CREATE TRIGGER insert_products_tags AFTER INSERT ON products
  FOR EACH ROW
  BEGIN
    DECLARE current_id integer;
    DECLARE tag_id integer;
    DECLARE next integer;
    DECLARE tag_field varchar(255);
    DECLARE next_sep integer;
    DECLARE current_tag varchar(255);
    DECLARE right_tag varchar(255);

    -- We use the field other as comma-separated tag_field
    SET tag_field = NEW.other;

    -- Check for empty tags
    IF (CHAR_LENGTH(tag_field) <> 0) THEN
        -- Loop until no more ocurrencies
       set next = 1;
       WHILE next = 1 DO
         -- Find possition of the next ","
         SELECT INSTR(tag_field, ',') INTO next_sep;
         IF (next_sep > 0) THEN
            SELECT SUBSTR(tag_field, 1, next_sep - 1) INTO current_tag;
            SELECT SUBSTR(tag_field, next_sep + 1, CHAR_LENGTH(tag_field)) INTO right_tag;
            set tag_field = right_tag;
         ELSE
           set next = 0;
           set current_tag = tag_field;
         END IF;

         -- Drop spaces between comas
         SELECT TRIM(current_tag) INTO current_tag;

         -- Insert the tag if not already present
         IF (NOT EXISTS (SELECT tag FROM freetags WHERE tag = current_tag)) THEN
           -- Insert the tag
           INSERT INTO freetags (tag) values (current_tag);
           SELECT LAST_INSERT_ID() INTO tag_id;
         ELSE
           -- Or get the id
           SELECT id FROM freetags WHERE tag = current_tag INTO tag_id;
         END IF;

         -- Link the object tagged with the tag
         INSERT INTO freetagged_objects
           (tag_id, object_id, module)
            values
           (tag_id, NEW.id, 'products');
       END WHILE;
    END IF;
  END;
@@

Now If you execute an insert on products table:

INSERT INTO PRODUCTS
    (name, price, other)
    values
    ("product1", 2, "tag1, tag2,tag3 , tag 4");

You will see the tags inserted and linked on the generated product id :)

Now we need to ensure that if you update the “product1″ record and you modified the field `other` the tags gets updated. When programing I was thinking if review the new tags one by one, or delete all the links and parse again the tag field. Finally for eficiency and simplicity decided delete and regenerate.

UPDATE_PRODUCTS_TAGS

DELIMITER ||
DROP TRIGGER IF EXISTS update_products_tags;
||
DELIMITER @@
CREATE TRIGGER update_products_tags BEFORE UPDATE ON products
  FOR EACH ROW
  BEGIN
    DECLARE current_id integer;
    DECLARE tag_id integer;
    DECLARE next integer;
    DECLARE tag_field varchar(255);
    DECLARE next_sep integer;
    DECLARE current_tag varchar(255);
    DECLARE right_tag varchar(255);

    -- We use the field other as comma-separated tag_field
    SET tag_field = NEW.other;

    -- Only act if the field changes
    IF (tag_field <> OLD.other) THEN
      -- At the moment we regenerate the tags (not compare), needs some thinking because compare can be
      -- more performance killer
      DELETE FROM freetagged_objects WHERE object_id = OLD.id AND module = 'products';
      -- Insert again if not empty
      IF (CHAR_LENGTH(tag_field) <> 0) THEN
          -- Loop until no more ocurrencies
         set next = 1;
         WHILE next = 1 DO
           -- Find possition of the next ","
           SELECT INSTR(tag_field, ',') INTO next_sep;
           IF (next_sep > 0) THEN
              SELECT SUBSTR(tag_field, 1, next_sep - 1) INTO current_tag;
              SELECT SUBSTR(tag_field, next_sep + 1, CHAR_LENGTH(tag_field)) INTO right_tag;
              set tag_field = right_tag;
           ELSE
             set next = 0;
             set current_tag = tag_field;
           END IF;

           -- Drop spaces between comas
           SELECT TRIM(current_tag) INTO current_tag;

           -- Insert the tag if not already present
           IF (NOT EXISTS (SELECT tag FROM freetags WHERE tag = current_tag)) THEN
             -- Insert the tag
             INSERT INTO freetags (tag) values (current_tag);
             SELECT LAST_INSERT_ID() INTO tag_id;
           ELSE
             -- Or get the id
             SELECT id FROM freetags WHERE tag = current_tag INTO tag_id;
           END IF;

           -- Link the object tagged with the tag
           INSERT INTO freetagged_objects
             (tag_id, object_id, module)
              values
             (tag_id, NEW.id, 'products');
         END WHILE;
      END IF;
    END IF;
  END;
@@

This example has many common uses and shows a fairly wide variety of SQL syntax. I recommended Mysql 5.1 for triggers and stored procedures development, more info on Mysql Website.

1) Support for triggers is included beginning with MySQL 5.0.2.

2) For trigger development is really helpfull some database IDE development like (TOAD or SQL/Developer for Oracle). For Mysql you can use two exceptional tools: Tora (Open Source TOAD like clone for Oracle an Mysql) and Mysql Query Browser (Open Source by Mysql).

0

Monitoring anything with Zabbix

This is the first introduction post of a series of how-to monitoring with Zabbix.

I love Zabbix because is one of the most feature complete and easy to setup/mantain Open Source monitoring tool, some kill features:

  • Agent, snmp, web and database monitoring.
  • Windows, Linux & Unix monitoring.
  • Distributed Setups (you can have different installs on different locations and have one unique view of your systems and networks).
  • Configurable alerting systems (mail, jabber, sms, external scrips).
  • Autodiscovery of new Hosts.
  • Integrated SLA Calculator.
  • ZeroConf graphing with a great zooming tool in all graphs (Standard/Custom).
  • Custom graphs and views (screens & slideshows).
  • Great template systems in order to save time on configuration (all data is inside a db, Mysql, Oracle or PostgreeSQL)
  • Pro-active monitoring with remote commands when something happen.
  • Complete feature list on http://zabbix.com

There are many integrated monitoring solutions in the Open Source world, for mention the most important:

Also for more specific (process/system monitoring):

Network monitoring:

Before starting with Zabbix I was tested Nagios/Groundwork Open Source, also Zenoss and Cacti. All are excelent solutions, but none of them satisfied me completely.

0

The future of Mysql

Since the acquisition of SUN by Oracle on April 20, 2009,  many Mysql users around the globe (from enterprises, ISV’s to individuals) are a bit skeptics about the future of Database and engines, at least as we know it now.

Yes, is Open Source.

So, there are alternatives:

  • AskMonty MariaDB
  • Monty Program Ab was founded by Michael “Monty” Widenius, the founder and creator of MySQL. It consists of some of the original MySQL engineers, some new personnel, and Monty.

    MariaDB, a branch of the MySQL database which includes all major open source storage engines, including the Maria storage engine. Our ambition is to make this the best open and free MySQL branch for the MySQL user and developer communities.

  • Percona
  • Percona provides pay-as-you-go, prepaid and retainer-based consulting to create or scale applications built on the full LAMP stack (Linux, Apache, MySQL, PHP) and other open source technologies. Our customer list is large and diverse, including Fortune 50 firms, popular websites, and small startups.

  • Ourdelta
  • OurDelta produces enhanced builds for MySQL, with OurDelta and third-party patches, for common production platforms. All the builds are freely available for download.

  • The new Mysql-forked-enterprise Aliance
  • and many other initiatives and patched versions.

Yes, Oracle in October 7, 2005 bought Innobase (the enterprise behind the InnoDB engine) and the most popular engine has been improved since this.

But many of the best engineers (at least those remaining from the acquisition of SUN to MysqlAB) are leaving SUN/Oracle.

As you can read, all forks are based on the official source tree or are patches to improve performance, enhacements of features or stability. But what happens if Oracle stops development, are MariaDB and others really prepared to start big developments? What happens with the cluster solution? Is the community really interested on ndb engine?

My twisted conclusions:

  • Mysql isn’t on the same segment of the Oracle Database Family, Mysql+InnoDB has millions of users/clients and has a high market share of web based applications databases.
  • Oracle with Mysql can compete with the fastest growing Microsoft SQL Server on the Mysql niche, where the traditional Oracle stack is too expensive.
  • For ndb storage engine (the cluster based share nothing Mysql solution), is not clear the interest that Oracle can have. Isn’t a finish solution, has a lack of important features as foreign keys, complex querys performance, stability… and not have many clients requesting support (I like it). Also the list and forums is in mute mode (before purchase the SUN people was helping a lot).

In the lasts weeks, new versions of many products are released, is a good signal?

0

init

First post of this blog. I expect this as a how-to and known of things of my interest:

Systems, Networking, Programming, Databases, Open Source, Curiosities and more.

I live in Spain and my natural language is not English (as you will notice), my apologies in advance.