Voyaging Mind

As I mentioned in my previous post I am doing small targeted iterations and to my surprise I am actually managing to keep to this. Over the past few iterations I have added support for robots.txt file to my crawler so I am slowly becoming a good web citizen. The rate of sites downloaded is still pretty slow. This may be down to the time.sleep(1) I have put between each download. Having more than a few thousand websites download it not that useful to me at the moment. Having a large dataset of pages in my database would mean I would attach value to it and feeling I should attempt to correct mistakes with just simple blowing away the data. Really it still feels like I am just try to arrange the bones and am nowhere near trying to flesh it out. It has also dawned on me that I should limit the engine to just sites written in English.  The rational being is it is the only language I understand which will make attempting to create scoring algorithms a little easier. Of course detec...

I love this title as is it really defines the way I am working on my little search engine. I am making a series of decision that amount to getting each iteration completed as fast as possible. This means making a series of decisions that are at best maximally simple and at worst dumb, based on the idea I will fix them in a later iteration. When faced with a problem the first question I ask is "Is there a dumb solution that lets me progress?". If there is a dumb solution I use it with the expectation I will revisit it in the future. Yes, I have already started to revisit some of these decisions and usually it is because the performance was not great once the data got bigger. Making things fast is an enjoyable process so if revisiting means making things fast then I am probably going to enjoy this approach to coding. One feature of this simple as can be approach is the code can get a bit messy. So each iteration I make sure it includes a task to clean up the code in some way. J...

In my last post I talked a little bit a small web crawler I have been coding in python but I didn't really mention why I was coding other than it would be a fun little hack. Well I have sorted of decided to explore the world of Internet search engines by writing my own one. That is one that will be tiny and simple and generally not very useful. I will probably only ever index a couple of thousand sites but it will search non the less. The current status is I have a simple web crawler the pushes websites into a sqlite database. Another script then converts the HTML to text and splits this text into words. Each word is pushed into another sqlite database along with the url it came from. The second database effectively contains for every word encountered all the web pages it appears on. I then query this database with a single word and it is returns a list of the first 10 entries. I do the query in python code and just output the urls to the shell. I think you could say that is pretty...

I was previously taking a look at the Go programming language and finding it quite an enjoyable experience but I drifted away from it a little bit over the past week or two. I have sort of got interested in writing an Internet crawler and the first thing that became apparent was this was much easier to do using python. It is almost what python was designed for, a network bound task. Now I am sure there are plenty of crawlers out there but I wanted the experience of writing my own, even if it is a little simple. This is the first project where I have decided to use a database, to store the downloaded html obviously. The zero set up of sqlite won me over instantly, while setting up a different database would not be that hard using sqlite let me instantly get to doing the fun stuff like pushing data into it. After a bit of experimentation I decided that I would compress the data before pushing it into the database. With my rather small data set of 10 websites the compression made my datab...

I have completed watching the first two seasons of Black Sails and I have to say I really enjoyed it. I admit to having a liking to that era in part because of the many hours I spent playing Pirates! on an Amiga many years ago but it is really driven by a slight fascination with the culture of the time. Although the reality of the harshness that people suffered seems to not be well represented in the popular history reporting. Not wanting to reveal any spoilers I will keep things vague. Black Sails has a mix of pirates some fictional and some using the names of real pirates from the time. The story arcs are good and the writing/acting is well above average - in my opinion. It has all the hall marks to make it a set of above average TV series. What sets it apart for me is a number of things. None of the characters are particularly good or moral, they are all capable of horrific deeds and loyalty is often fleeting, at the same time many do show compassion. I really enjoyed the fact that ...

It is pretty clear to me that Clojure is not going to be a language I will get good at in the space of a couple of weeks. Partly because I am taking a relaxed approach to it, sort of enjoying the journey you might say, and partly there is a reasonable amount to learn, not just the language but the tooling. So after reading through "The Joy of Clojure" I am sure I have not understood even half of it, a book to return in at some point in the future. The how to progress next question hit me. I could just start writing code, like the little text adventure I created. This would be immensely enjoyable but I would rapidly fall out of learn Clojure mindset to how can I make this game better and that is not somewhere I want to be at the moment. So I have decided to take a look at the book "Living Clojure". It is simpler than the previous book and presents a series of exercises to get you up to a reasonable level over the period of a number of weeks. Seems like a fun thing to...

Yes it has been a long time and yes I have been pretty distracted and I have not been coding much. I am slowly getting the rhythm back. The little bit of coding I am doing using clojure. A long time ago I took a look at it bit but it never really pulled me in. Now the tool set looks a lot more mature. That does mean I have jumped back into the world of emacs. Yes other editors exist but I am trying to not explore all of them. My attitude at the moment is pretty laid back, I have no expectations of becoming a master at clojure or for that matter emacs. The plan is to just enjoy the coding and anything I learn is just a bonus. So far I have written a tiny text adventure and read a couple of books. Reading the books has given me a good overview but none of the sections really sunk in. Why the laid back attitude? Well apart from the usual Summer business I also was training for a black belt. So the past few months have been pretty intense. Now I would like a few months to decompress...