There are a couple of different cases when you can get this error:
You are using an older MySQL version (before 3.23.0) when an in-memory temporary table becomes larger than tmp_table_size bytes. To avoid this problem, you can use the -O tmp_table_size=# option to make mysqld increase the temporary table size or use the SQL option SQL_BIG_TABLES before you issue the problematic query. See SET.
You can also start mysqld with the --big-tables option. This is exactly the same as using SQL_BIG_TABLES for all queries.
In MySQL Version 3.23, if an in-memory temporary table becomes larger than tmp_table_size, the server automatically converted it to a disk-based MyISAM table.
You are using InnoDB tables and run out of room in the InnoDB tablespace. In this case the solution is to extend the InnoDB tablespace.
You are using ISAM or MyISAM tables on an OS that only supports files of 2G in size and you have hit this limit for the datafile or index file.
You are using MyISAM tables and the needed data or index size is bigger than what MySQL has allocated pointers for. (If you don't specify MAX_ROWS to CREATE TABLE MySQL will only allocate pointers to hold 4G of data).
You can check the maximum data/index sizes by doing
SHOW TABLE STATUS FROM database LIKE 'table_name';
or using myisamchk -dv database/table_name.
If this is the problem, you can fix it by doing something like:
ALTER TABLE table_name MAX_ROWS=1000000000 AVG_ROW_LENGTH=nnn;
You only have to specify AVG_ROW_LENGTH for tables with BLOB/TEXT fields as in this case MySQL can't optimize the space required based only on the number of rows.