large-data-volumes

I'm trying to get 4 random results from a table that holds approx 7 million records. Additionally, I also want to get 4 random records from the same table that are filtered by category. Now, as you would imagine doing random sorting on a table this large causes the queries to take a few seconds, which is not ideal. One other method I thought of for the non-filtered result set would be to just get PHP to select some random numbers between 1 - 7,000,000 or so and then do an IN(...) with the query to only grab those rows - and yes, I know that this method has a caveat in that you may get less

In my admin section, when I edit items, I have to attach each item to a parent item. I have a list of over 24,000 parent items, which are listed alphabetically in a drop down list (a list of music artists). The edit page that lists all these items in a drop down menu is 2MB, and it lags like crazy for people with old machines, especially in Internet Explorer. Whats a good alternative to replicate the same function, where I would need to select 1 of these 24,000 artists, without actually having them all pre-loaded into a drop down menu. Instead of filling a drop list with so many names you

I have a process that's going to initially generate 3-4 million PDF files, and continue at the rate of 80K/day. They'll be pretty small (50K) each, but what I'm worried about is how to manage the total mass of files I'm generating for easy lookup. Some details: I'll have some other steps to run once a file have been generated, and there will be a few servers participating, so I'll need to watch for files as they're generated. Once generated, the files will be available though a lookup process I've written. Essentially, I'll need to pull them based on an order number, which is unique per file.

Consider the following code in Python, using psycopg2 cursor object (Some column names were changed or omitted for clarity): filename='data.csv' file_columns=('id', 'node_id', 'segment_id', 'elevated', 'approximation', 'the_geom', 'azimuth') self._cur.copy_from(file=open(filename), table=self.new_table_name, columns=file_columns) The database is located on a remote machine on a fast LAN. Using \COPY from bash works very fast, even for large (~1,000,000 lines) files. This code is ultra-fast for 5,000 lines, but when data.csv grows beyond 10,000 lines, the program freezes completely. Any

For a web application I'm developing, I need to store a large number of records. Each record will consist of a primary key and a single (short-ish) string value. I expect to have about 100GB storage available and would like to be able to use it all. The records will be inserted, deleted and read frequently and I must use a MySQL database. Data integrity is not crucial, but performance is. What issues and pitfalls am I likely to encounter and which storage engine would be best suited to the task? Many thanks, J Whatever solution you use, since you say your database will be write-heavy you need