I finally finished an initial draft of an online search tool for Arabic news sites. This work is part of the project Cactus: Computational Analysis of Cyber Terrorism against U.S., sponsored by the US Army Development Test Command, White Sand Missile Range. We (our team at New Mexico State University) are spidering a number of Arabic news sites daily. We index the content of these sites using the Indri Indexing and Search tool. We modified the tool slightly so it will perform stemming (Arabic Light 10 stemming) on UTF8 text. Right now we have over 10,000 Arabic documents in our collection and it is growing daily. We still have a problem with detecting duplicate documents and are investigating various methods for detecting duplicate documents in a reasonable amount of time. I currently detect exact duplicates by using the sha1 hash function.
Monthly Archives: June 2008
Identifying the source of Arabic documents
For the last few months I’ve been working on methods to identify the source of Arabic documents. For example, given a document I would like to identify where it was written (Syria, Libya, Sudan, etc). This task is part of a larger project to identify cyberterrorist threats involving New Mexico Tech, New Mexico State University, and the University of Mary Washington. I have over 4000 Arabic documents from 5 different newspapers. Most of the documents are around 15-25k in size. My method uses the sequential minimal optimization algorithm to train a support vector machine. I have been evaluating the approach using 10 fold cross validation and have been getting over 99% classification accuracy. I am currently working on writing several papers on this. As soon as I have a paper accepted I will post it here.
Praire Home Companion – Pan Am Center
This week I went to hear a Praire Home Companion at the Pan American Center in Las Cruces, New Mexico (you can listen to that show here). I can say that radio does not do justice to the quality of the regular musicians on the show. Maybe its how they mix or compress the show for radio, but the live performance is much more vibrant. Guest artists included the phenomenal vocal duo Sorela consisting of two sisters, D’ette and D’anna from Albuquerque. They sung songs both in Spanish and English and the audience at the Pan Am Center loved them for good reason. Also appearing were Joe Elyand Joel Guzman both from Austin. Joe Ely is a singer/guitarist and Joel is an extraordinary accordionist–totally mind-blowing. The biggest crowd pleaser of the evening was Chris Thile singing “El Paso”.