Voyaging Mind

As I mentioned in my previous post I am doing small targeted iterations and to my surprise I am actually managing to keep to this. Over the past few iterations I have added support for robots.txt file to my crawler so I am slowly becoming a good web citizen. The rate of sites downloaded is still pretty slow. This may be down to the time.sleep(1) I have put between each download. Having more than a few thousand websites download it not that useful to me at the moment. Having a large dataset of pages in my database would mean I would attach value to it and feeling I should attempt to correct mistakes with just simple blowing away the data. Really it still feels like I am just try to arrange the bones and am nowhere near trying to flesh it out. It has also dawned on me that I should limit the engine to just sites written in English.  The rational being is it is the only language I understand which will make attempting to create scoring algorithms a little easier. Of course detecting

I love this title as is it really defines the way I am working on my little search engine. I am making a series of decision that amount to getting each iteration completed as fast as possible. This means making a series of decisions that are at best maximally simple and at worst dumb, based on the idea I will fix them in a later iteration. When faced with a problem the first question I ask is "Is there a dumb solution that lets me progress?". If there is a dumb solution I use it with the expectation I will revisit it in the future. Yes, I have already started to revisit some of these decisions and usually it is because the performance was not great once the data got bigger. Making things fast is an enjoyable process so if revisiting means making things fast then I am probably going to enjoy this approach to coding. One feature of this simple as can be approach is the code can get a bit messy. So each iteration I make sure it includes a task to clean up the code in some way. J

In my last post I talked a little bit a small web crawler I have been coding in python but I didn't really mention why I was coding other than it would be a fun little hack. Well I have sorted of decided to explore the world of Internet search engines by writing my own one. That is one that will be tiny and simple and generally not very useful. I will probably only ever index a couple of thousand sites but it will search non the less. The current status is I have a simple web crawler the pushes websites into a sqlite database. Another script then converts the HTML to text and splits this text into words. Each word is pushed into another sqlite database along with the url it came from. The second database effectively contains for every word encountered all the web pages it appears on. I then query this database with a single word and it is returns a list of the first 10 entries. I do the query in python code and just output the urls to the shell. I think you could say that is pretty

I was previously taking a look at the Go programming language and finding it quite an enjoyable experience but I drifted away from it a little bit over the past week or two. I have sort of got interested in writing an Internet crawler and the first thing that became apparent was this was much easier to do using python. It is almost what python was designed for, a network bound task. Now I am sure there are plenty of crawlers out there but I wanted the experience of writing my own, even if it is a little simple. This is the first project where I have decided to use a database, to store the downloaded html obviously. The zero set up of sqlite won me over instantly, while setting up a different database would not be that hard using sqlite let me instantly get to doing the fun stuff like pushing data into it. After a bit of experimentation I decided that I would compress the data before pushing it into the database. With my rather small data set of 10 websites the compression made my datab

I have completed watching the first two seasons of Black Sails and I have to say I really enjoyed it. I admit to having a liking to that era in part because of the many hours I spent playing Pirates! on an Amiga many years ago but it is really driven by a slight fascination with the culture of the time. Although the reality of the harshness that people suffered seems to not be well represented in the popular history reporting. Not wanting to reveal any spoilers I will keep things vague. Black Sails has a mix of pirates some fictional and some using the names of real pirates from the time. The story arcs are good and the writing/acting is well above average - in my opinion. It has all the hall marks to make it a set of above average TV series. What sets it apart for me is a number of things. None of the characters are particularly good or moral, they are all capable of horrific deeds and loyalty is often fleeting, at the same time many do show compassion. I really enjoyed the fact that

It is pretty clear to me that Clojure is not going to be a language I will get good at in the space of a couple of weeks. Partly because I am taking a relaxed approach to it, sort of enjoying the journey you might say, and partly there is a reasonable amount to learn, not just the language but the tooling. So after reading through "The Joy of Clojure" I am sure I have not understood even half of it, a book to return in at some point in the future. The how to progress next question hit me. I could just start writing code, like the little text adventure I created. This would be immensely enjoyable but I would rapidly fall out of learn Clojure mindset to how can I make this game better and that is not somewhere I want to be at the moment. So I have decided to take a look at the book "Living Clojure". It is simpler than the previous book and presents a series of exercises to get you up to a reasonable level over the period of a number of weeks. Seems like a fun thing to

Yes it has been a long time and yes I have been pretty distracted and I have not been coding much. I am slowly getting the rhythm back. The little bit of coding I am doing using clojure. A long time ago I took a look at it bit but it never really pulled me in. Now the tool set looks a lot more mature. That does mean I have jumped back into the world of emacs. Yes other editors exist but I am trying to not explore all of them. My attitude at the moment is pretty laid back, I have no expectations of becoming a master at clojure or for that matter emacs. The plan is to just enjoy the coding and anything I learn is just a bonus. So far I have written a tiny text adventure and read a couple of books. Reading the books has given me a good overview but none of the sections really sunk in. Why the laid back attitude? Well apart from the usual Summer business I also was training for a black belt. So the past few months have been pretty intense. Now I would like a few months to decompress

Oh wow! I have actually created a new article over on the main voyagingmind site. It is about downloading stockmarket data from yahoo and you can find it here . I will probably do a couple more runs to proof read it, as I hammered it out fairly quickly and my brain is not in the mood for doing much in the way of proof reading tonight. Still I wanted to push it out tonight just so I don't procrastinate on it. I think I have finally settled on using pycharm as my python editor, much as I like both vim and emacs the zero set up and maintenance that pycharm offers is just too compelling. I am also thinking about doing a reasonable amount of Java soon (starting another game) so having a single editor (Intellij) that I spend most of my  at home coding means less load on the brain. I press vim commands a fair bit when coding in Visual Studio when at work... Having said that I have been playing with ipython and the notebook functionality it provides. I am very impressed.

Things stay roughly the same but also change a fair bit would be a fair way of summing up the past few weeks. First up python has really pulled me in. At the moment I just really enjoy it. I have read though the book Fluent Python and that really help for me to see that I needed to more past the "Yep python, I know it and code in it just like I code C++" stage that many have when learning python, well replace C++ with what ever language you know quite well. I have not put a lot of what is in the book to practice yet but am looking forward to doing so. I suppose I should explain that the goal of the book is to take someone who has a bit of python experience and explain the more advance features of python. It is about 700 pages (I got the ebook) and does a really good job in what it sets out to do. Well worth a read. The stock market is still attracting some of my attention. Just messing with little algorithms is such good fun. I should write an article or two about it.

Do excuse the slight break in blogging. It was sort of deliberate. I have been a little busy with a few hobby projects. I took a good dive into the Go programming language and I think I like it. No in a wow this is amazing more in a yep this can feel a little restrictive compared with C++ or others but I get what you are trying to achieve here and can like it. Probably the only thing I didn't like about it was the use of Capital letters to mean publicly accessible. It is weird that I don't like it  - it just feels well wrong. Given a few thousand lines of code I would get past this and if that is the only thing I have to complain about. In terms of development Go is moving pretty quick at the moment, lots of libraries appearing and generally a good vibe about it. I am not planning on committing any more time to it but overall I was left with a good feeling about it. One to watch in the future My other little project involved stocks and shares. After reading a couple of ar

It would be fair to say I have not been productive during this year. I have done a great deal of learning rather than actually getting on with the project I had in mind for much of this year. We are now gradually heading towards to the end of the third month and all I have is a few abortive attempts at starting something. Yes I have learned a great deal about servers and some about more modern ways of doing client side web apps. I do like that style of creating apps perhaps because servers are still a bit of an unknown to me at least in real use so pushing as much to the client is my safe place. But so far the results are telling, I have produced nothing of note. So I decided to kill it. If a project is not gaining some development momentum after two and a half months then it is probably a good idea to move on. Perhaps I will return to it at a later date but for now it is dead.  It is funny how the decision to kill it came about, I was having a frustrating time trying to figure somethi

It is fair to say I have written this entry about 4 times and that is kind of funny as it is just one of those what I had for dinner entries. As my previous posts indicate I am taking a bit of a dive into server side web development as it is area I know little about. The Udacity web course taught me a fair bit and made me think seriously about using the google app engine. Something just didn't feel right about it. The lack of control part where it seems very hard to get you data out if you decide to migrate away has just about scared me off. I however liked python and took a look at using Flask. It seem pretty nice but again I didn't really progress with it. Last night I realised that I don't yet know enough about server side webdev to get going on the project I want to create and decided to attack the Node JS book I have again. This keeps in with my only use Javascript of C++ for projects this year, plus it is self contained and should give me the knowledge I need to a

I have had a presence on the internet for a very long time, having built static sites, run sites on various content management systems as well having written a lot of text in various blogging sites. One thing I have never really is done is getting to grips with server side programming. Sure I wrote a couple of app-engine apps  but I am not sure I understood the concepts. I however had an idea for a site I am thinking of creating that would require a reasonable amount of server side coding. So I decided to learn a bit more about the topic. I signed up the the Web Development course at udacity, the free one as I don't need a certificate at the end. So far it has been pretty interesting. While I sort of recall some bit from previous adventures, taking it from the ground up has been pretty interesting and I am certainly learning a fair bit. I have also brought a book on node js and express. While I am only slowly progressing with this it is touching on similar topics the udacity

Well I would not say I have been productive over the past few days. I have been busy but in a distracted sort of way. No simple game has hooked me yet. I like the idea of producing some simple, small games but my drawing abilities let me down and I get disappointed. I am learning towards creating games that are a little bit more off the beaten path so my, lack of, drawing skills are not compared with full time artists. My art skill at art varies a great deal over a period of a year. It is all down to practice and knowing the tools. It would be fair to say I am at a low point at the moment and the only way out of this is to do more art.  The tool I am most familiar with is Inkscape and to be honest it is not ideal for the creation of art I would like in my games, that means I have to leave my comfort zone and learn some more tools. Usually at this point I would head towards something like gimp but this time I am taking a look around as all I really need in a sprite editor. It would be n

I call it v0.01 and it is ready to ship. I am of course talking about my little play around game engine I am creating. It is really just about 900 lines of code to wrap SDL a little bit. Not exactly breathe taking in terms of quality. Heck I have not bothered to put it in its own name space. Nor does it really have a name, well it does it compiles to a static library called gamelib. I just call it v0001 for short. The point is not to create a great library more to have fun while experimenting with C++ 11, along side SDL and cross compiling to HTML5 and android. Mmmm when has cross compiling been fun? Still I am sure it will be this time. So I made a small list as I said I would and set about completing the list. Now as promised I will create a simple game. Yay! Although I am still not entirely decided on what it will be. I may just go for the obvious clone and remake space invaders. It is simple to make so I can play with C++ 11 and will force me to draw a bit (a good thing). Dra

I have to admit I am quite enjoying SDL at the moment. About 10 years ago I used it a little bit and found it to be a bit low level for what I was looking for. Sure it was very cross platform but I was looking for something slightly higher level to knock out a few prototypes. I think I ended up using python and pygame (which uses SDL internally). Fast forward to now and with increased experience in C++ plus SDL having improved somewhat as well I still find it low level but am finding it fairly easy to wrap it into a slightly higher level api to use. It means so far this year I have been writing a game engine, I actually want to be writing a game rather than an engine. Game engines can and will steal your time in part because they are so interesting to create and in part because there is always something else to add. Thankfully I have avoided abstracting away SDL. The code I write is tightly coupled to SDL but then SDL is on every possible platform I can reasonably target making this is

I have spent the past couple of days knocking together the start of a small 2D gaming library in C++. Pretty much wrapping parts of SDL to make them more C++ like. I have to admit it has been great fun. It has been a while since I said that about C++ I have probably said it many times but C++11 has been a huge change and the alterations C++14 makes look like they will make things even easier. Sure there are still things needed such as a module systems but the feel of it is a lot more modern. If you have not spent time with C++ in the past couple of years then you will probably be presently surprised with the changes. Thanks to clang we have competition in how fast things compile and that is benefiting us all plus the relentless drive in performance increases of hardware, yes I am singling you out SSDs, is slowly making programming in C++ a little less of a chore. Sure I doubt it will ever feel as fast as Java but it is getting better. Finally I have been trying out Clion a new ID

Is there a better way to see in the new year than a really heavy cold that has you floored. Perhaps there is but the heavy cold options was the one dealt to me. Christmas resulted in a number of Hudl 2 tablets entering the house. They are pretty nice tablets that may be unique to the UK as they are produced by Tesco (a large supermarket chain in the UK). One fun thing about them was on the play store the game Terraria did not support these tablets. My children quite like this game. Looking on the Amazon store not only do I find it is happy to install it on the tablets but was for a limited period free. Kind of a win for amazon and a loss for google play as we now have the amazon app store installed and am now only one step from buying from them. Christmas also brought in the first chromebook into our house. It is an ARM based one and is really nice to use. I have not put it to any heavy duty tasks but it handles all the light browser based tasks like blogging, emails and facebook