MySQL IF() statement holds 3 expressions, IF(expr1, expr2, expr3). expr1 used to check condition, if expr1 TRUE, then return expr2, else return expr3. The example above interpreted as get all the users records who register yesterday, where difference between today and register date equals to 1 day. When it performs CURDATE() function, it will checks if register date is NULL, if its NULL then return CURDATE(), else return register date. So, for those register date equals to NULL, it will return today, CURDATE() and hence results in return empty result or force the sql statement not be executed, as DATEDIFF(CURDATE(),CURDATE())<>1.

But here’s the rub: putting together a major BI landscape in a company can cost some serious cash. The better-bring-your-AMEX-goldcard attitude of a lot of data warehousing and BI vendors has caused some frustration on the part of many IT directors; in fact, a 2007 InformationWeek survey found that 39% of IT executives complained that software licensing costs prohibited them from rolling out the BI initiatives they want.

But enter open source! Just as open source has revolutionized many areas of software these days and opened the doors for nearly any company to craft a competitive IT environment of operating systems, development tools, and databases, it’s now doing the same thing in the area of data warehousing and BI. Witness the fact that data warehousing (as indicated on MySQL’s last major community and customer poll) is now the fifth most common use case for MySQL.

But here’s another statistic to digest: according to TDWI, the average growth rate of data warehouses ranges between 33 and 50% per year, and that’s conservative for some businesses. The most popular storage engine used for MySQL data warehousing today is MyISAM (second is InnoDB), and it does just fine performance-wise up to around 1TB of data. After that, a lot of users tend to partition their data warehouse across more than one server to help performance. Now, stats from the analyst group IDC say that the majority of data warehouses are 6TB and under (only 4% are over 25TB according to IDC), so that means most folks these days are looking to manage warehouses between hundreds of GB and 6TB.

If this is you, and you want to stay with a MySQL-based solution, then you owe it to yourself to check out the Infobright storage engine. Infobright, one of MySQL/Sun’s partners, supplies an engine that breaks through the limitations of MyISAM and other storage engines and delivers a very sophisticated piece of technology that (surprisingly…) doesn’t require heavy lifting when it comes to install, setup, and database design to get incredibly fast response times.

Let’s first take a look at the Infobright architecture to see how it accomplishes these things and then take it for a test drive to see how well it really performs on analytic-styled queries.

Columns are Cool

In a nutshell, the Infobright storage engine is a column-oriented architecture that combines a high-speed data loader, strong levels of data compression, and a smart, external optimizer and ‘knowledge grid’ to deliver impressive data warehousing capabilities. From a MySQL standpoint, Infobright presents just as any other storage engine would so there’s nothing really new to understand from an interface standpoint. Infobright is a separate install, however, from the general MySQL server, but it installs very simply (actually faster than a standard MySQL install I found), and is only a 17MB download, which is pretty remarkable for an engine that can scale up to managing ten’s of terabytes.

The first thing to note is that the Infobright engine is a column-oriented design. Column-oriented databases have been around for a while now (e.g. Sybase IQ), but are now coming on strong in popularity due to their strength in servicing data warehouse needs. In his March 2008 research report “What’s Cool About Columns”, Philip Howard of Bloor Research writes, “For much of the last decade the use of column-based approaches has been very much a niche activity. However … we believe that it is time for columns to step out of the shadows to become a major force in the data warehouse and associated markets.” He then adds, “Columns provide better performance at a lower cost with a smaller footprint: it is difficult to understand why any company seriously interested in query performance would not consider a column-based solution.”

Why would he make such statements? Because column-oriented designs like Infobright really do pack quite a punch in data warehousing situations. There are numerous reasons for this being the case, but here are just a few. Most data warehousing/analytic queries are only interested in a couple of columns in one or more tables rather than entire rows of columns. This being the case, storing data in the typical row-based format is inefficient for data warehouse purposes whereas storing data in a column-oriented fashion is much better. In a column-oriented design like Infobright, full table scans will never be performed (unless a query requests all or the majority of rows in a table); only full column scans may be. The end result is much less I/O and improved response times for column-oriented databases.

Infobright combines a column-oriented design with something else that everyone should like: high-levels of data compression. Because it’s column-oriented, when Infobright compresses data, it does so for each column, which is normally more efficient than standard row-based compression because the compression algorithms can be finely tuned for each column datatype. In normal row-based compression, you do well to see 2 or 3-to-1 compression; with Infobright, 10-to-1 compression is the norm (with some cases going much higher), so a 1TB database can be compressed down to 100GB. When some of the members of the Sun performance team saw this in action, they remarked to me how surprised they were at how well Infobright compressed data vs. many of the other databases (row-based and column-based) they had worked with. Of course, another by-product of compression is that not only does it help on the performance side, but it also helps on the storage cost side of the house, which should please most IT managers.

Other Infobright Technical Benefits

Although you can load data into a MySQL Infobright table via all the standard routes, Infobright also provides a special high-speed loader that really makes a difference in pumping data into a database. The loading algorithm is multi-threaded, which allows each column to be loaded in parallel. Single table loads with the Enterprise edition of Infobright run typically around 100GB an hour and multi-table loads can accomplish nearly 300GB an hour for binary data, which isn’t too shabby. The Community edition of Infobright has slower load speeds as it only handles text-based input (as opposed to binary), but you’ll still see loads around 40GB or so an hour.

But perhaps the major technical star in the Infobright architecture is its “knowledge grid” and accompanying optimizer. Infobright begins construction of the knowledge grid at the time data is loaded into the database (either initially or incrementally). The knowledge grid is essentially a statistical description of the data and demographics in the database. Here’s a key thing to remember about the knowledge grid: it serves as a substitute for indexes, which means you never create an index on an Infobright table. Never. And this is a good thing!

The Infobright knowledge grid does not have the maintenance disadvantages of indexes, which as we all know, cause insert and update response times to degrade over time as more and more modifications are made to a database. Another advantage is that Infobright does extremely well at servicing unpredictable queries, which are exactly what many data warehouses have to deal with. Such queries are a DBA’s nightmare because they can never design an efficient indexing or partitioning strategy, and performance is never that great. But with Infobright, such problems evaporate because it does all the work dynamically for you.

The actual data in Infobright is stored as columns as previously mentioned. The columns themselves are divided into groups of 64K values called data packs, whose metadata is stored in the knowledge grid. When a query is submitted to Infobright for execution, the optimizer consults the knowledge grid in order to generate a rough idea of which data packs contain data needed for the result set of the query. Queries run exceptionally fast when (a) there are relatively few data packs containing data in the result set, and (b) the optimizer is able to accurately identify the set of needed data packs.

One way to think of this data distribution is something akin to auto data partitioning. And again, this is a good thing. With Infobright, you don’t have to be a data warehouse design god to get an exceptionally performing data warehouse as it does nearly all the hard work for you – no indexing or partitioning strategies are necessary on your part.

In some cases a query can be executed without looking at any data packs at all; only the knowledge grid is consulted, and such queries will execute instantaneously. Since the knowledge grid contains many aggregate values, and because aggregates are a common aspect of queries in data warehousing applications, it is not unusual for many data warehousing type queries to execute with little or no required processing (examples follow below).

Application scenarios where Infobright tends to shine are star schemas (although true star design isn’t required) with deep, wide fact tables and shallow dimension tables, and designs where the tables in the databases are largely denormalized. Use cases where Infobright may not perform as well include applications with highly normalized schemas and more random data distributions. This is because such data doesn’t compress as well as data with clustered patterns and because the data in the query result sets are spread around the database so that large numbers of data packs need to be scanned.

So enough talk – let’s now exercise the Infobright engine across some standard data warehouse use cases and see how well it does. All the tests below were executed on a Dell PowerEdge 6850, with four Intel Xeon dual-core processors (3.4 GHz), 32GB of RAM, and five 300GB internal drives configured in RAID 10 style, running on 64-bit Red Hat Enterprise Linux 5 with Infobright Enterprise edition.

Kicking the Tires

Working with Infobright is pretty much the same as any other MySQL engine when it comes to creating tables – all you do is specify brighthouse as the engine type. For example:

mysql> create table t (c1 int) engine=brighthouse;
Query OK, 0 rows affected (0.02 sec)

mysql> insert into t values (1), (2), (3);
Query OK, 3 rows affected (0.16 sec)
Records: 3  Duplicates: 0  Warnings: 0

mysql> select * from t;
+------+
| c1   |
+------+
|    1 |
|    2 |
|    3 |
+------+
3 rows in set (0.00 sec)

So now onto the actual tests: the schema I used for the queries below was a standard data warehousing star schema (representing a car sales database) that can be depicted in a data model as follows:

The number of rows and overall size of the database (as calculated from the information_schema) is as follows:

+----------------------------+-------------+------------+--------------+
| table_name                 | engine      | table_rows | data_length  |
+----------------------------+-------------+------------+--------------+
| fact_sales5                | BRIGHTHOUSE | 8080000000 | 135581094511 |
| fact_sales                 | BRIGHTHOUSE | 1000000000 |  16789624027 |
| fact_sales1b               | BRIGHTHOUSE | 1000000000 |  17424704919 |
| mthly_sales_by_dealer_make | BRIGHTHOUSE |    4207788 |     43958330 |
| dim_vins                   | BRIGHTHOUSE |    2800013 |     15251819 |
| dim_sales_area             | BRIGHTHOUSE |      32765 |       302326 |
| dim_dates                  | BRIGHTHOUSE |       4017 |         9511 |
| dim_dealers                | BRIGHTHOUSE |       1000 |         9631 |
| dim_dealers2               | BRIGHTHOUSE |       1000 |        10222 |
| dim_cars                   | BRIGHTHOUSE |        400 |         4672 |
| dim_msa                    | BRIGHTHOUSE |        371 |         3527 |
| tt                         | BRIGHTHOUSE |          1 |          193 |
+----------------------------+-------------+------------+--------------+
12 rows in set (0.00 sec)

+------------------+
| sum(data_length) |
+------------------+
|     169854973688 |
+------------------+
1 row in set (0.01 sec)

The above shows a couple of fairly big fact tables at 1 billion rows each, a larger historical fact table at a little over 8 billion rows, one medium-sized summary table (4 million rows), and a number of dimension tables that are fairly small in size (except the 2.8 million row dim_vins table). The total physical size of the database is almost 170GB, but the actual raw size of the data was 1TB, so you can see the Infobright compression in action and that it delivers as promised.

Checking a few row counts prove the above numbers:

mysql> select count(*) from fact_sales5;
+------------+
| count(*)   |
+------------+
| 8080000000 |
+------------+
1 row in set (0.00 sec)

mysql> select count(*) from fact_sales;
+------------+
| count(*)   |
+------------+
| 1000000000 |
+------------+
1 row in set (0.00 sec)

Notice that Infobright responds just like MyISAM in full COUNT(*) queries; the knowledge grid knows how many rows are in each table so you won’t waste time figure-tapping in waiting for a response from such queries.

Now, let’s issue a few analytic queries and see what we get. First, let’s do a simple join to find out how much the dealership made over a certain time period for one make of car:

mysql> select sum(dlr_trans_amt)
    ->     from fact_sales a, dim_cars b
    ->     where a.make_id = b.make_id and
    ->     b.make_name = 'ACURA' and
    ->     b.model_name = 'MDX' and
    ->     trans_date between '2007-01-01' and '2007-01-31';
+--------------------+
| sum(dlr_trans_amt) |
+--------------------+
|        11264027726 |
+--------------------+
1 row in set (24.98 sec)

Not too bad at all. But now let’s put the knowledge grid / data packs to the test and see how big a dent in our response time we get by adding eight times more data to the mix:

mysql> select sum(dlr_trans_amt)
    ->     from fact_sales5 a, dim_cars b
    ->     where a.make_id = b.make_id and
    ->     b.make_name = 'ACURA' and
    ->     b.model_name = 'MDX' and
    ->     trans_date between '2007-01-01' and '2007-01-31';
+--------------------+
| sum(dlr_trans_amt) |
+--------------------+
|        11264027726 |
+--------------------+
1 row in set (27.20 sec)

Nice! Infobright was able to – again – only examine the needed data packs and exclude all the other data it didn’t need to look for to satisfy our query, with no real practical impact to the overall response time (running the same query again actually came in under the first query that used the smaller fact table).

Let’s now try something that the general MySQL Server will struggle with in some cases – nested subqueries:

mysql> select avg(dlr_trans_amt)
    -> from   fact_sales
    -> where  trans_date between '2007-01-01' and '2007-12-31' and
    ->        dlr_trans_type = 'SALE' and make_id =
    ->  (select make_id
    ->   from dim_cars
    ->   where make_name = 'ASTON MARTIN' and
    ->         model_name = 'DB7') and
    ->         sales_area_id in
    ->            (select sales_area_id
    ->             from dim_sales_area
    ->             where sales_state =
    ->               (select dealer_state
    ->                from dim_dealers
    ->                where dealer_name like 'BHUTANI%'));
+--------------------+
| avg(dlr_trans_amt) |
+--------------------+
|    45531.444471505 |
+--------------------+
1 row in set (50.78 sec)

Infobright plows through the data just fine. What about UNION statements – oftentimes these can cause response issues with MySQL. Let’s try both fact tables this time:

mysql> (select avg(dlr_trans_amt), avg(sales_commission), avg(sales_discount)
    -> from fact_sales
    -> where trans_date between '2007-01-01' and '2007-01-31')
    -> union all
    -> (select avg(dlr_trans_amt), avg(sales_commission), avg(sales_discount)
    -> from fact_sales
    -> where trans_date between '2007-02-01' and '2007-02-28');
+--------------------+-----------------------+---------------------+
| avg(dlr_trans_amt) | avg(sales_commission) | avg(sales_discount) |
+--------------------+-----------------------+---------------------+
|   45550.1568209903 |               5.39966 |     349.50289769532 |
|   45549.5774942714 |               5.39976 |    349.498835301098 |
+--------------------+-----------------------+---------------------+
2 rows in set (0.49 sec)

mysql> (select avg(dlr_trans_amt), avg(sales_commission), avg(sales_discount)
    -> from fact_sales5
    -> where trans_date between '2007-01-01' and '2007-01-31')
    -> union all
    -> (select avg(dlr_trans_amt), avg(sales_commission), avg(sales_discount)
    -> from fact_sales5
    -> where trans_date between '2007-02-01' and '2007-02-28');
+--------------------+-----------------------+---------------------+
| avg(dlr_trans_amt) | avg(sales_commission) | avg(sales_discount) |
+--------------------+-----------------------+---------------------+
|   45550.1568209903 |               5.39966 |     349.50289769532 |
|   45549.5774942714 |               5.39976 |    349.498835301098 |
+--------------------+-----------------------+---------------------+
2 rows in set (0.75 sec)

It appears the UNION’s were satisfied via knowledge grid access alone. Next, let’s try a few joins coupled with a having clause and ask for the average Ashton Martin dealer transaction amounts over one year for dealers in the state of Indiana:

mysql> select fact.dealer_id,
    ->        avg(fact.dlr_trans_amt)
    -> from   fact_sales fact
    -> inner  join dim_cars cars on (fact.make_id = cars.make_id)
    -> inner  join dim_sales_area sales on
    ->        (fact.sales_area_id = sales.sales_area_id)
    -> where  fact.trans_date between '2007-01-01' and '2007-12-31' and
    ->        fact.dlr_trans_type = 'SALE' and
    ->        cars.make_name = 'ASTON MARTIN' and
    ->        cars.model_name = 'DB7' and
    ->        sales.sales_state = 'IN'
    -> group  by fact.dealer_id
    -> having avg(fact.dlr_trans_amt) > 50000
    -> order  by fact.dealer_id desc;

.
.
.
|         2 |         51739.181818182 |
|         1 |                 57964.8 |
+-----------+-------------------------+
317 rows in set (50.66 sec)

Of course, there are plenty of other queries that could be tested, but the above will give you a feel for how Infobright performs for some typical analytic-styled queries. And again, one of the great things is you don’t have to spend time designing indexing or partitioning schemes to get performance results like those show above because none of that is necessary in Infobright. In fact, there are only around three tuning parameters for the engine and they’re all memory related.

Infobright Limitations

There are certain limitations you need to be aware of right now with Infobright. Not all queries can be satisfied with the Infobright optimizer; those that can’t end up being sent over to the MySQL optimizer. If that happens you’ll receive a warning after your query executes that states that such a thing occurred.

• Infobright can handle up to 32 concurrent queries at this time.
• A query is currently constrained to one CPU/Core.
• Correlated subqueries are supported, but typically aren’t run that efficiently.
• DML (insert, update, delete; only available in the Enterprise edition) only supports table-level locking, which could reduce concurrency if much DML occurs in an Infobright warehouse.
• Theoretically, an Infobright table can go up to 147 trillion rows, but practically, 50 billion is the limit today – a count which greatly depends on the data row size and datatypes used (i.e. the row limit could go higher depending on the row size).
• Internationalization support is lacking right now; UTF8 support planned first half of 2009.
• Infobright supports Windows for ICE and Solaris for IEE only right now. They also have pre-built VMs available for other environments (based on VMWare.).
• No ALTER TABLE support. You cannot switch from other tables to Infobright or vice versa.
• The knowledge grid can’t process non-Infobright tables well so if you have a mixture of storage engines in a query that involves Infobright tables too, expect your performance to take a hit

I also hit a small bug on two of the queries above where substituting the 8 billion row table for the one billion row table caused a performance hit on the query (everything else being equal). Apparently, there is a known bug on the sorting algorithm that relates to the source data and is only experienced with large tables. Infobright is in the process of correcting it.

Right now in terms of operating system and hardware support, the Infobright storage engine runs on 32-bit (Community edition only) and 64-bit (Community and Enterprise editions) Intel and AMD Red Hat Enterprise Linux, CentOS, Fedora (Community edition only) and Debian with standard commodity hardware. And all key BI tools (Business Objects, Cognos, Pentaho, Jaspersoft, etc.) support the MySQL and Infobright combination.

Reference:- http://dev.mysql.com/tech-resources/articles/datawarehousing_mysql_infobright.html

As usual there were two ways to split the buffer pool mutex, one way is to split it functionally into different mutexes protecting different parts. There have been experiments in particular on splitting out the buffer pool page hash table, the flush list. Other parts that have been broken out in experiments are the LRU list, the free list and other data structures internally in the buffer pool. Additionally it is as usual possible to split the buffer pool into multiple buffer pools. Interestingly one can also combine using multiple buffer pools with splitting the buffer pool mutex into smaller parts. The advantage of using multiple buffer pools is that it is very rare that it is necessary to grab multiple mutexes for the buffer pool operation which quickly becomes the case when splitting the buffer pool into multiple mutex protection areas.

After working on scalability improvements in MySQL and InnoDB I noted that all the discussion was around how to split the buffer pool mutex and no dicsussion centered around how to make multiple buffer pools out of the buffer pool. I decided to investigate how difficult it would be to make this change. I quickly realised that it needed a thorugh walk through of the code. It required a code check that required checking about 150 methods and their interaction. This sounds like a very big task, but fortunately the InnoDB code is well structured and have fairly simple dependencies between its methods. After this walk through of the buffer pool code one quickly found that there were 3 different ways of getting hold of the buffer pool, one method was to calculate it using the space id and page id. This is the normal method in most methods used in the external buffer pool interface. However there were numerous occasions where we only had access to the block or page data structure and it would be a bit useless to recalculate the hash value in every method that needed access to the buffer pool data structure. So it was decided to leave a reference to the buffer pool in every page data structure. There were also a few occasions where one needed to access all buffer pools.

The analysis proved that most of the accesses to the buffer pool was completely independent of other accesses to the buffer pool for other pages. InnoDB uses read-ahead and neighbour writes in the IO operations that are started from the buffer pool. These always operate on an extent of 64 pages. Thus it made sense to map the pages of 64 pages into one buffer pool to avoid having to operate on multiple buffer pools on every IO operation.

With these design ideas there were only a few occasions where it was necessary to operate on all buffer pools. One such operation was when the log required knowledge of the page with the oldest LSN of the buffer pool. Now this operation requires looping over all buffer pools and checking the minimum LSN of each buffer pool instance. This is a fairly rare operation so isn’t a scalability issue.

The other operation with requirement to loop over all pages needed a bit more care, this operation is the background operation flushing buffer pool pages to disk. A couple of problems needs consideration here. First it is necessary to flush pages regularly from all buffer pool instances, secondly it’s still important to flush neighbours. Given that many disks are fairly slow, it can be problematic to spread the load in this manner to many buffer pools. This is an important consideration when deciding how many buffer pool instances to configure.

The default number of buffer pool is one and for most small configurations with less than 8 cores it’s mostly a good idea not to increase this value. If you have an installation that uses 8 cores or more one should also pay attention to the disk subsystem that is used. Given that InnoDB often writes up to 64 neighbours in each operation and that the flushing should happen each second, it makes sense to have a disk subsystem capable of having 500 IO operations per second to use 8 buffer pool instances. This can be set in the innodb_io_capacity configuration variable. One SSD drive should be capable of handling this, two fast hard drives or 3 slow ones.

In our experiments we have mostly used 8 buffer pools, more buffer pools can be useful at times. The main problem with many buffer pools is related to the IO operations. It is important to have a balanced IO load in the MySQL server.

Our analysis of using multiple buffer pool instances have shown some interesting facts. First the accesses to the buffer pools is in no way evenly spread out. This is not surprising given that e.g. the root page of an index is a very frequently accessed page. So using sysbench with only one table, there will obviously be much more accesses to certain buffer pool instances. Our experiments shows that in sysbench using 8 buffer pools, the hottest buffer pool receives about one third of all accesses. Given that sysbench is a worst case scenario for the multiple buffer pool case, this means that most applications that tend to use more tables and more indexes should have a much more even load on the buffer pools.

So how much does multiple buffer pools improve the scalability of the MySQL Server. The answer is as usual dependent on application, OS, HW and so forth. But some general ideas can be found from our experiments. In sysbench using a load which is entirely held in main memory, so the disk is only used for flushing data pages and logging, in this system the multiple buffer pools can provide up to 10% improvement of the throughput in the system. In dbStress, the benchmark Dimitri uses, we have seen all the way up to 30% improvement. The reason here is most likely that dbStress uses more tables and have avoided many other bottlenecks in the MySQL Server and thus the buffer pool was a worse bottleneck in dbStress compared to sysbench. From the code it is also easy to see that the more IO operations the buffer pool performs, the more the buffer pool mutex will be acquired and also often held for a longer time. One such example is the search for a free page on the LRU list every time a read is performed into the buffer pool from the disk.

Furthermore the use of multiple buffer pool opens up for many more improvements and also it doesn’t remove the possibility to split the buffer pool mutex even more.

Another manner of displaying the importance of using multiple buffer pools is the mutex statistics on the buffer pool mutex. With one buffer pool the buffer pool had about 750k accesses per second in a sysbench test where the MySQL Server had access to 16 cores. 50% of those accesses met a mutex already held, so it’s obvious that the InnoDB mutex subsystem is very well aligned with the buffer pool mutex which have very short duration which makes spinning waiting for it very fruitful. Anyways a mutex which is held 50% of the time makes the buffer pool mutex a limiting factor of the MySQL Server. Quite a few threads will often spend time in the queue waiting for the buffer pool mutex. So splitting the buffer pool into 8 instances even in sysbench means that the hottest buffer pool receives about one third of the 750k accesses so should be held about 17% of the time. Our later experiments shows that the hottest buffer pool mutexes are now held up to about 14-15% of the time. So the theory matches the real world fairly well. This means that the buffer pool is still a major factor in the MySQL Scalability equation but is now more on par with the other bottlenecks in the MySQL Server.

The development project of multiple buffer pools happened at a time when the MySQL and InnoDB teams could start working together. I was impressed by the willingness to cooperate and the competence in the InnoDB team that made it possible to introduce multiple buffer pools into MySQL 5.5. Our cooperation has continued since then and this has led to improvements in productivity on both parts. So for you as a MySQL user this spells good times going forward.

If you do not know Upstart, this is the way you start and stop services in Ubuntu, and it serves the same purpose as the old /etc/init.dscripts, but are a bit more structured and powerful. That said, Upstart is regrettably far from complete, although the functionality is much better and Upstart has some cool features, some things do not work that well. For one thing, documentation, where it exists, is useless, at best. Secondly, there is very limited ability to test and develop Upstart scripts. And this is made worse by the fact that the documentation is so bad. Another thing is that Upstart insist on stopping services, by default, by sending a brutal kill signal. Not good for databases, mostly.

In the /etc/init directory are the Upstart scripts you have. In difference to the old init.d scripts, you cannot disable a service in Upstart curenntly. If it is in /etc/init it will be started at system start. That’s it. And this is something that I am sure will be fixed, but for now, again, is something we have to live with. Upstart scripts have the suffix .conf (don’t ask me why), so the default MySQL Upstart script, for example, is called /etc/init/mysql.conf.

In an Upstart script, there are Stanzas that determine what to do. Like the exec Stanza that runs a program for example. And you may then ask, when is it run? Startup? Shutdown? And the answer is startup. For shutting things down, as I said before, Upstart will by default just send a kill -9 signal.

The minimal startup script you can have, and this actually works in a reasonable way, is to just have one line with an exec stanza, like this:
exec /usr/bin/mydaemon
Which will start the daemon. For stopping the daemon, Upstart will send a -9 signal to the started process by default, and nothing more is needed in the Upstart script.

For MySQL, we need to make things a bit more complicated. The default mysql.conf Upstart script really is not good. For one thing, it will not do a controlled shutdown of MySQL (this is possible even if Upstart will eventually send a kill -9 anyway). Secondly, this script assumes that what we use is a standard Ubunty installed MySQL distribution, so if you have installed MySQL in /usr/bin/mysql5147 or somethings like that, you are out of luck.

So what I wanted to create was an Upstart script for MySQL that fullfilled these requirements:

1. Starts MySQL automatically.
2. Waits for MySQL to be available before exiting.
3. Be configurable to support different MySQL install locations, data directories etc.
4. Do a clean shutdown of MySQL when stopping the MySQL services.

Before I show you what I ended up with, I want to comment on the points 2 and 4 above. With Upstart, you can define a script or command to run just before or after a services has been started or stopped, and this is what I use to wait for MySQL to become available, and to send a SIGTERM to the MySQL Server when stopping (which will do a clean MySQL shutdown).

So here we go, a complete MySQL Upstart script, the way I want it to work:

#
# MySQL Service for Recorded Future
#
description     "MySQL Server"
author          "Anders Karlsson, Recorded Future"

start on (net-device-up
          and local-filesystems
          and runlevel [2345])
stop on runlevel [016]

expect fork
kill timeout 2

# Set variables.
env MYSQL_ETC=/etc/mysql
env MYSQL_PIDFILE=/var/run/mysql.pid
env MYSQL_HOME=/usr/local/mysql5.5
env MYSQL_INSTANCE=my
umask 007

exec $MYSQL_HOME/bin/mysqld_safe --defaults-file=$MYSQL_ETC/$MYSQL_INSTANCE.cnf >> /tmp/x.out &

post-start script
    loop=600
# Wait for MySQL to start.
    while [ $loop -gt 0 ]; do
        if $MYSQL_HOME/bin/mysqladmin --defaults-file=$MYSQL_ETC/$MYSQL_INSTANCE.cnf ping; then
            break
        fi
        loop=$(($loop - 1))
        sleep 1
    done
    exit 0
end script

# Send a soft SIGTERM to MySQL before Upstart will kill it.
# A Sigterm to mysqld will cause a controlled shutdown.
pre-stop script
    exec kill -SIGTERM `cat $MYSQL_PIDFILE`

# Wait for MySQL to end. Flushing buffers and all.
    loop=600
    while [ $loop -gt 0 ]; do
# If the pidfile is found, then continue waiting.
        if [ -e $MYSQL_PIDFILE ] ; then
            loop=$((loop - 1))
            sleep 1
            continue
        fi
        break
    done
end script

To be honest, this is not what I create for all our MySQL servers. Instead I used this to create a chef template, chef is what we use for configuration management here (see http://www.opscode.com/ for more on chef), and here it is put to good ude to generate an Upstart script for MySQL. The above is just an example.

In short the problem is that the Optimizer in the MySQL server does not get adequate statistics from the data nodes about table sizes, indexes etc. This makes the Optimizer clueless in some cases how to order tables in joins, and also in some cases which is the best indexes to use.

So here is the problem that Nick highlighted:

When the tables are stored in MyISAM:
mysql> SELECT v.vid_id, v.vid_title, u.user FROM phpfox_user AS u JOIN phpfox_videos AS v ON(v.vid_userid = u.id) limit 3;
3 rows in set (0.32 sec)

phpfox_user is about 100000 rows and phpfox_videos is 170000 rows large

Trying to run the same query on my cluster machine, i get
3 rows in set (20.47 sec)

Why is this? Let’s look at the EXPLAIN of the queries:

MyISAM explain:

mysql> explain SELECT v.vid_id, v.vid_title, u.user FROM phpfox_user AS u JOIN phpfox_videos AS v ON(v.vid_userid = u.id) ORDER BY v.vid_time DESC LIMIT 3;
+—-+————-+——-+——–+—————+———
+———+———————+——–+—————-+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+—-+————-+——-+——–+—————+———
+———+———————+——–+—————-+
| 1 | SIMPLE | v | ALL | vid_userid | NULL | NULL | NULL | 135025 | Using filesort |
| 1 | SIMPLE | u | eq_ref | PRIMARY | PRIMARY | 4 | nktest.v.vid_userid | 1 | |
+—-+————-+——-+——–+—————+———
+———+———————+——–+—————-+

NDB explain:

mysql> explain SELECT v.vid_id, v.vid_title, u.user FROM phpfox_user AS u JOIN phpfox_videos AS v ON(v.vid_userid = u.id) ORDER BY v.vid_time DESC LIMIT 3;
+—-+————-+——-+——+—————+———–
-+———+—————–+——-+————————
———+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+—-+————-+——-+——+—————+———–
-+———+—————–+——-+————————
———+
| 1 | SIMPLE | u | ALL | PRIMARY | NULL | NULL | NULL | 82124 | Using temporary; Using filesort |
| 1 | SIMPLE | v | ref | vid_userid | vid_userid | 4 | bb2_phpfox.u.id | 1 | |
+—-+————-+——-+——+—————+———–
-+———+—————–+——-+————————
———+

Note the additional ‘Using temporary’ clause for NDB. Nearly all of the 20 seconds taken by the query fall at ‘copying to temporary table’.

The problem is that the Optimizer has changed the order (in case of NDB) how tables should be joined!

Jeff Sturm replied to Nick with a remedy:

Looks like the query optimizer is putting the tables in the wrong order.
(This isn’t uncommon in my experience, as NDB seems to have less
information than other engines to optimize queries at runtime.)

Try modifying your query with STRAIGHT_JOIN syntax, i.e.:

SELECT v.vid_id, v.vid_title, u.user FROM phpfox_videos AS v
STRAIGHT_JOIN phpfox_user AS u ON v.vid_userid = u.id ORDER BY
v.vid_time DESC LIMIT 3;

Data/Index partitioning (range, hash, key, list, composite) in MySQL 5.1 and above
No practical storage limits (1 tablespace=110TB) with automatic storage management
Built-in Replication
Strong indexing support (B-tree, fulltext, clustered, hash, GIS)
Multiple, configurable data/index caches
Pre-loading of data into caches
Unique query cache (caches result set + query; not just data)
Parallel data load
Multi-insert DML
Read-only tables
Cost-based optimizer
Wide platform support

To mitigate these issues, leading modern businesses such as Los Alamos National Labs, MIT Lincoln Lab, Cox Communications, and others have selected MySQL to power their growing data warehouse infrastructure. The growth of MySQL in the area of data warehousing recently prompted Gartner Group to include MySQL in their 2006 Magic Quadrant for Data Warehouse DBMS Servers.

MySQL is uniquely designed to easily handle the most common data warehousing use cases:

Data Marts
Traditional Data Warehouses
Large Historical/Archive Data Warehouses
Real Time Data Warehouses
MySQL offers other storage engines that can also be used for data warehousing as well. MySQL supports these key data warehousing features:

Data/Index partitioning (range, hash, key, list, composite) in MySQL 5.1 and above
No practical storage limits with automatic storage management
Built-in Replication
Strong indexing support (B-tree, fulltext, clustered, hash, GIS)
Multiple, configurable data/index caches
Pre-loading of data into caches
Unique query cache (caches result set + query; not just data)
Parallel data load
Multi-insert DML
Read-only tables
Cost-based optimizer
Wide platform support
Native Storage Engines
MySQL currently offers a number of its own native Storage Engines, including:

InnoDB
MyISAM
Cluster
Federated
Archive
Merge
Memory
CSV
Blackhole

InnoDB I/O subsystem changes enable more effective use of available I/O capacity.

Several modifications improve operation of MySQL Server on Solaris.

There is better access to execution and performance information. Diagnostic improvements include DTrace probes, expanded SHOW ENGINE INNODB STATUS output, and a new status variable.

Support for an interface for semisynchronous replication: A commit performed on the master side blocks before returning to the session that performed the transaction until at least one slave acknowledges that it has received and logged the events for the transaction. Semisynchronous replication is implemented through an optional plugin component. See Section 16.3.8, “Semisynchronous Replication”

Support for the SQL standard SIGNAL and RESIGNAL statements. See Section 12.7.8, “SIGNAL and RESIGNAL”.

Support for Performance Schema, a feature for monitoring MySQL Server execution at a low level. See Chapter 20, MySQL Performance Schema.

Support for additional Unicode character sets: utf16, utf32, and utf8mb4. These character sets support supplementary Unicode characters; that is, characters outside the Basic Multilingual Plane (BMP). See Section 9.1.10, “Unicode Support”.

Enhancements to XML functionality, including a new LOAD XML statement.

Two new types of user-defined partitioning are supported: RANGE COLUMNS partitioning is an extension to RANGE partitioning; LIST COLUMNS partitioning is an extension to LIST partitioning. Each of these extensions provides two enhancements to MySQL partitioning capabilities:

It is possible to define partitioning ranges or lists based on DATE, DATETIME, or string values (such as CHAR or VARCHAR).

You can also define ranges or lists based on multiple column values when partitioning tables by RANGE COLUMNS or LIST COLUMNS, respectively. Such a range or list may refer to up to 16 columns.

For tables defined using these partitioning types, partition pruning can now optimize queries with WHERE conditions that use multiple comparisons between (different) column values and constants, such as a = 10 AND b > 5 or a < “2005-11-25″ AND b = 10 AND c = 50.

For more information, see Section 17.2.1, “RANGE Partitioning”, and Section 17.2.2, “LIST Partitioning”.

It is now possible to delete all rows from one or more partitions of a partitioned table using the ALTER TABLE … TRUNCATE PARTITION statement. Executing the statement deletes rows without affecting the structure of the table. The partitions named in the TRUNCATE PARTITION clause do not have to be contiguous.

Key caches are now supported for indexes on partitioned MyISAM tables, using the CACHE INDEX and LOAD INDEX INTO CACHE statements. In addition, a key cache can be defined for and loaded with indexes from an entire partitioned table, or for one or more partitions. In the latter case, the partitions are not required to be contiguous.

The TO_SECONDS() function is added. This function converts a date or datetime expression to a number of seconds since the year 0. You may use this function in partitioning expressions, and partition pruning is supported for tables defined using such expressions.

The following constructs are obsolete and have been removed in MySQL 5.5. Where alternatives are shown, applications should be updated to use them.

The log_bin_trust_routine_creators system variable (use log_bin_trust_function_creators).

The myisam_max_extra_sort_file_size system variable.

The record_buffer system variable (use read_buffer_size).

The sql_log_update system variable.

The table_type system variable (use storage_engine).

The FRAC_SECOND modifier for the TIMESTAMPADD() function.

The TYPE table option to specify the storage engine for CREATE TABLE or ALTER TABLE (use ENGINE).

The SHOW TABLE TYPES SQL statement (use SHOW ENGINES).

The SHOW INNODB STATUS and SHOW MUTEX STATUS SQL statements (use SHOW ENGINE INNODB STATUS SHOW ENGINE INNODB MUTEX).

The SHOW PLUGIN SQL statement (use SHOW PLUGINS).

The LOAD TABLE … FROM MASTER and LOAD DATA FROM MASTER SQL statements (use mysqldump or mysqlhotcopy to dump tables and mysql to reload dump files).

The BACKUP TABLE and RESTORE TABLE SQL statements (use mysqldump or mysqlhotcopy to dump tables and mysql to reload dump files).

TIMESTAMP(N) data type: The ability to specify a display width of N (use without N).

The –default-character-set and –default-collation server options (use –character-set-server and –collation-server).

The –delay-key-write-for-all-tables server option (use –delay-key-write=ALL).

The –enable-locking and –skip-locking server options (use –external-locking and –skip-external-locking).

The –log-bin-trust-routine-creators server option (use –log-bin-trust-function-creators).

The –log-long-format server option.

The –log-update server option.

The –master-xxx server options to set replication parameters (use the CHANGE MASTER TO statement instead): –master-host, –master-user, –master-password, –master-port, –master-connect-retry, –master-ssl, –master-ssl-ca, –master-ssl-capath, –master-ssl-cert, –master-ssl-cipher, –master-ssl-key.

The –safe-show-database server option.

The –skip-symlink and –use-symbolic-links server options (use –skip-symbolic-links and –symbolic-links).

The –sql-bin-update-same server option.

The –warnings server option (use –log-warnings).

The –no-named-commands option for mysql (use –skip-named-commands

The –no-pager option for mysql (use –skip-pager).

The –no-tee option for mysql (use –skip-tee).

The –position option for mysqlbinlog (use –start-position).

The –all option for mysqldump (use –create-options).

The –first-slave option for mysqldump (use –lock-all-tables).

The –config-file option for mysqld_multi (use –defaults-extra-file).

The –set-variable=var_name=value and -O var_name=value general-purpose options for setting program variables (use –var_name=value).

Syntax: concat_ws(‘separator’, ‘string 1’, ‘string 2’, ‘string 3’, ‘string N’, ……)

1) SELECT concat_ws(‘ ‘, ‘Firstname’, ‘Lastname’)

2) SELECT concat_ws(‘<br>’, ‘Address Line 1′, ‘Address Line 2′, ‘City’, ‘State’, ‘Zipcode’, ‘Country’)

3) SELECT concat_ws(‘<br>’, ‘Address Line 1′, NULL, ‘City’, ‘State’, ‘Zipcode’, ‘Country’)

Note: in the above examples, instead of the string values you can use the table fields.

IMPORTANT POINT to be noted in Query #3 is that if any of the values in the list is having NULL value then this function simply ignores that value and returns the concatenated values of the remaining values with the separator specified; in the above example the separator is a string ‘<br>’ it can be any string.

Where and how to use:

SELECT concat_ws(‘ ‘, u.firstname, u.lastname) as fullname, concat_ws(‘<br>’, u.address1, u.address2, u.city_name, s.state_name, u.zipcode, c.country_name) as fulladdress

FROM user_master u

LEFT JOIN state_master s ON s. state_id =u. state_id

LEFT JOIN country_master c ON c. countries_id =u.countries_id

WHERE user_id = ‘123’

Isn’t it helpful???

2. Scalability and Flexibility
The MySQL database server provides the ultimate in scalability, sporting the capacity to handle deeply embedded applications with a footprint of only 1MB to running massive data warehouses holding terabytes of information. Platform flexibility is a stalwart feature of MySQL with all flavors of Linux, UNIX, and Windows being supported. And, of course, the open source nature of MySQL allows complete customization for those wanting to add unique requirements to the database server.

3. Robust Transactional Support
MySQL offers one of the most powerful transactional database engines on the market. Features include complete ACID (atomic, consistent, isolated, durable) transaction support, unlimited row-level locking, distributed transaction capability, and multi-version transaction support where readers never block writers and vice-versa. Full data integrity is also assured through server-enforced referential integrity, specialized transaction isolation levels, and instant deadlock detection.

4. High Availability
Rock-solid reliability and constant availability are hallmarks of MySQL, with customers relying on MySQL to guarantee around-the-clock uptime. MySQL offers a variety of high-availability options from high-speed master/slave replication configurations, to specialized Cluster servers offering instant failover, to third party vendors offering unique high-availability solutions for the MySQL database server.

5. Web and Data Warehouse Strengths
MySQL is the de-facto standard for high-traffic web sites because of its high-performance query engine, tremendously fast data insert capability, and strong support for specialized web functions like fast full text searches. These same strengths also apply to data warehousing environments where MySQL scales up into the terabyte range for either single servers or scale-out architectures. Other features like main memory tables, B-tree and hash indexes, and compressed archive tables that reduce storage requirements by up to eighty-percent make MySQL a strong standout for both web and business intelligence applications.

6. Strong Data Protection
Because guarding the data assets of corporations is the number one job of database professionals, MySQL offers exceptional security features that ensure absolute data protection. In terms of database authentication, MySQL provides powerful mechanisms for ensuring only authorized users have entry to the database server, with the ability to block users down to the client machine level being possible. SSH and SSL support are also provided to ensure safe and secure connections. A granular object privilege framework is present so that users only see the data they should, and powerful data encryption and decryption functions ensure that sensitive data is protected from unauthorized viewing. Finally, backup and recovery utilities provided through MySQL and third party software vendors allow for complete logical and physical backup as well as full and point-in-time recovery.

7. Comprehensive Application Development
One of the reasons MySQL is the world’s most popular open source database is that it provides comprehensive support for every application development need. Within the database, support can be found for stored procedures, triggers, functions, views, cursors, ANSI-standard SQL, and more. For embedded applications, plug-in libraries are available to embed MySQL database support into nearly any application. MySQL also provides connectors and drivers (ODBC, JDBC, etc.) that allow all forms of applications to make use of MySQL as a preferred data management server. It doesn’t matter if it’s PHP, Perl, Java, Visual Basic, or .NET, MySQL offers application developers everything they need to be successful in building database-driven information systems.

8. Management Ease
MySQL offers exceptional quick-start capability with the average time from software download to installation completion being less than fifteen minutes. This rule holds true whether the platform is Microsoft Windows, Linux, Macintosh, or UNIX. Once installed, self-management features like automatic space expansion, auto-restart, and dynamic configuration changes take much of the burden off already overworked database administrators. MySQL also provides a complete suite of graphical management and migration tools that allow a DBA to manage, troubleshoot, and control the operation of many MySQL servers from a single workstation. Many third party software vendor tools are also available for MySQL that handle tasks ranging from data design and ETL, to complete database administration, job management, and performance monitoring.

9. Lowest Total Cost of Ownership
By migrating current database-drive applications to MySQL, or using MySQL for new development projects, corporations are realizing cost savings that many times stretch into seven figures. Accomplished through the use of the MySQL database server and scale-out architectures that utilize low-cost commodity hardware, corporations are finding that they can achieve amazing levels of scalability and performance, all at a cost that is far less than those offered by proprietary and scale-up software vendors. In addition, the reliability and easy maintainability of MySQL means that database administrators don’t waste time troubleshooting performance or downtime issues, but instead can concentrate on making a positive impact on higher level tasks that involve the business side of data.

10.Easy support.

You can get easy support from thousands of mysql administrators worldwide. Sometimes for free and sometimes for very negligible charges. You can contact me anytime for mysql support at nilesh.pawar@kaizenz.com or visit at http://www.kaizenz.com