I can't think of an off the shelf RDBMS which can't handle queries on 3 billion rows.
SQL Server can
Oracle can
Postgres can
Even MySQL can (!)
The limitations are almost always in the hardware, not the software.
If you're looking at column based systems, you can look at Greenplum (does both row and column-based storage), InfiniDB (MySQL based), and all sorts of expensive but very fast appliance options like Netezza, Teradata, etc.
SQL Server can
Oracle can
Postgres can
Even MySQL can (!)
The limitations are almost always in the hardware, not the software.
If you're looking at column based systems, you can look at Greenplum (does both row and column-based storage), InfiniDB (MySQL based), and all sorts of expensive but very fast appliance options like Netezza, Teradata, etc.