[TriLUG] Awk question

[Mark Freeze]

>Sounds like you desperately need some indexes on your tables.
> No way should that work be taking that long.  Have you
>analyzed what you are doing and applied appropriate
>indexes to the mysql tables?

You know, we believe that we created the database in the proper manner
and are using multiple index configurations. But, even after
re-thinking the process, re-indexing the columns, and checking the
logic of our code, MySQL seems to take forever to complete the given
task.

Now I make no profession to be a MySQL guru so I'll add a sub-query to
my original question because like Tanner, I'd like to test the speed
of several operations to see which is the quickest.

So, what kind of SQL code would you (or anyone) write to: (given my
previous data example)

Delete all records with the same LCode, AccountNum, and CustID for a
set of records. (Not the whole database, just the set of text records
that you are importing.) Would you do it as a query after import or
upon import.

What would your SQL query look like to perform the summarization as
described in my previous email.

I'd really be interested in testing any solutions provided.  We are
going to test the Perl solutions provided by Greg and James between
now and Friday and I'll post the results on Monday.

Thanks,
Mark.



On 8/7/07, Lance A. Brown <lance at bearcircle.net> wrote:
> Mark Freeze wrote:
> > When 'sort' gave us this type of performance we started looking for a
> > way to speed up the packet summarization (Did I just create a word?)
> > step.  MySQL is currently taking around 5 hours to analyze and
> > summarize around 40,000 records into 15,000 records.  A friend of ours
> > said that he thought that a combination of sort and awk would perform
> > this task without a hitch.  (The only hitch being that I don't know
> > anything about awk.)
>
> Gracious.  Sounds like you desperately need some indexes on your tables.
>  No way should that work be taking that long.  Have you analyzed what
> you are doing and applied appropriate indexes to the mysql tables?
>
> --[Lance]
> --
>  Celebrate The Circle   http://www.celebratethecircle.org/
>  Carolina Spirit Quest  http://www.carolinaspiritquest.org/
>  My LiveJournal  http://www.livejournal.com/users/labrown/
>  GPG Fingerprint: 409B A409 A38D 92BF 15D9 6EEE 9A82 F2AC 69AC 07B9
>  CACert.org Assurer
> --
> TriLUG mailing list        : http://www.trilug.org/mailman/listinfo/trilug
> TriLUG Organizational FAQ  : http://trilug.org/faq/
> TriLUG Member Services FAQ : http://members.trilug.org/services_faq/
>