Importing .gzfiles

New Member Joined: Apr 15, 2006 Posts: 9

Can anyone tell me how to import unformatted .gz files into MySQL?

here is a link to the directory were the files are stored

http://setiathome.berkeley.edu/stats/

registered · Posted: Sat Apr 15, 2006 4:09 pm

Can you tell us a bit more about what you are trying to accomplish? Where did these files come from and what are you intending on doing with them?

Posted: Sat Apr 15, 2006 4:28 pm

I am not even sure that directory is supposed to be public but in any event that gz file is huge, over 128Meg and thats compressed.

Posted: Sat Apr 15, 2006 4:35 pm

Yeah, I made the mistake of clicking on it... pretty much brought my system to its knees. Thankfully I had everything saved... all 20+ windows... ROTFL

I can get a little carried away at times.

Posted: Sat Apr 15, 2006 5:14 pm

No kidding lol Wink

registered · Posted: Sat Apr 15, 2006 5:51 pm

I'm not sure if Big Dump reads .gz - if not, you will have to unzip it first. But it does great with big .sql files

Posted: Sat Apr 15, 2006 6:21 pm

Of course the directory and files are public. It's compressed XML stats released by SETI.

The problem is that they don't give instructions for those of us who aren't coding geniuses....and the guys who DO know how to import the files act like their code is tops secret. So I'm searching for answers outside the community.

As far as what I'm trying to accomplish, take a look at these sites

http://boincstats.com/
http://www.boincsynergy.com/

Except I'm not interested in showing stats for all BOINC projects, only SETI.

I'll check out Big dump....

Posted: Sat Apr 15, 2006 8:36 pm

Well that certainly helps me understand a little better what you are trying to achieve.
As this data will pretty quickly be out of date, had you considered just pulling the data from their site into an iframe?

Posted: Sat Apr 15, 2006 9:22 pm

The .gz files from SETI are updated every 24 hours (under normal circumstances)

so it's something I would have to update regularly. It's my understanding that I could set up a cron to do this automatically....but one step at a time.

As for the iframe...don't really want to do that. I'd rather learn how to do this on my own rather than rape another site's content......plus once I have the data imported to my database, I can make custom signatures for my team mates with their individual stats....something I couldn't achieve with the iframe method.

Posted: Tue Apr 25, 2006 8:16 pm

Any other ideas??

Posted: Wed Apr 26, 2006 7:10 am

When I originally looked at one of your files (uncompressed), it looked like it was in XML format. How are these files generated? I personally have never used the XML export from mySQL, so not sure if this is the format. If it came directly from a mySQL XML dump, it should be fairly easy to do what you are looking to do. However, if this XML format has nothing to do with mySQL, you will need to write a translation program to get into a format that mySQL can import OR a program to read the XML and insert the records.

There is no easy answer on this one, especially with so few details.

Posted: Fri Apr 28, 2006 9:25 pm

I'm not sure what other details you need. These are compressed XML files with no type of formating. Here is an example of what the team.gz file looks like uncompressed. obviously there are hundreds of teams, but this is just the first 4 or 5 from an older file.

Code:




<?xml version="1.0" encoding="iso-8859-1"?>


<teams>


<team>


 <id>2</id>


 <type>2</type>


 <name>Master Strategy Group</name>


 <userid>3747</userid>


 <total_credit>4101690.129301</total_credit>


 <expavg_credit>6865.881056</expavg_credit>


 <expavg_time>1145050820.247990</expavg_time>


 <nusers>158</nusers>


  <founder_name>James R. Davis</founder_name>


 <create_time>0</create_time>


 <url>www.msgroup.org</url>


<name_html>Master Strategy Group</name_html>


<description>Eclectic partnership vastly committed to the development of world distributed system functionality and the utilization of otherwise idle computer resources in behalf of scientific efforts.</description>


 <country>United States</country>


</team>


<team>


 <id>3</id>


 <type>2</type>


 <name>El Reg</name>


 <userid>2389603</userid>


 <total_credit>3417027.706709</total_credit>


 <expavg_credit>9044.901251</expavg_credit>


 <expavg_time>1145050736.425640</expavg_time>


 <nusers>669</nusers>


  <founder_name>Drew</founder_name>


 <create_time>0</create_time>


 <url>www.theregister.co.uk</url>


<name_html>El Reg</name_html>


<description>Readers of The Register, banding together for the common good</description>


 <country></country>


</team>


<team>


 <id>4</id>


 <type>2</type>


 <name>Taldren</name>


 <userid>21154</userid>


 <total_credit>2238469.717337</total_credit>


 <expavg_credit>14321.508464</expavg_credit>


 <expavg_time>1145050873.997690</expavg_time>


 <nusers>162</nusers>


  <founder_name>Kenneth Yeast</founder_name>


 <create_time>0</create_time>


 <url>www.taldren.com</url>


<name_html>Taldren</name_html>


<description>Taldren, "the Art and Science of Gaming".  Creators of "StarFleet Command II : Empires at War" and "Orion Pirates".


</description>


 <country></country>


</team>


<team>


 <id>5</id>


 <type>2</type>


 <name>Spot_E.T.</name>


 <userid>4112</userid>


 <total_credit>6145932.488896</total_credit>


 <expavg_credit>25649.588739</expavg_credit>


 <expavg_time>1145050862.111910</expavg_time>


 <nusers>213</nusers>


  <founder_name>Mike Bader</founder_name>


 <create_time>0</create_time>


 <url>www.boinc.us</url>


<name_html>&lt;img src='http://www.myurl.net/seti/spotet.gif'> Spot_E.T.</name_html>


<description>Open to all individuals/companies/schools/churches etc.


Join our International team.</description>


 <country>United States</country>


</team>


<teams>

What I'm thinking is the <team> tag should indicate a new line with the tags in between( <id>, <type>, <name>, etc) indicating fields within that line

Posted: Sun Apr 30, 2006 9:55 am

megabyte2001, the point I was trying to make is that you need to understand the differences between these files and a mySQL XML table dump (which you can do using phpMyAdmin) to see how it needs to be formatting so that it can be imported in.

Understanding this, you may be able to write something to do the translation.

Posted: Sat May 06, 2006 2:56 pm

Quote:

Understanding this, you may be able to write something to do the translation.

If I possessed the ability to "write" code then I wouldn't have asked for assistance.

Thanks for your help, but I really need someone to hold my hand and guide me through the proceedure and it doesn't look like that is the type of assistance I'm going to get.

Posted: Sat May 06, 2006 3:04 pm

Might I suggest you offer to hire someone for a small fee to help you. There is a "For Hire" forum here as well. We help tons of people here with their questions and issues on a daily basis, but we just do not have time to write complete sets of code (at least I do not). Sorry if your expectation was higher than this.