A method, system and computer program product for reducing the execution cost of sampled queries of tables in a database. A subset of entries comprising row identifiers or key values based on an input query of data in the tables of the database is generated. A sampling predicate to reduce the number of entries in the subset is generated. The sampling predicate is applied to the subset to reduce the number of entries in the subset. The data in the database corresponding to the reduced subset is retrieved from the database.