SICP Initial Thoughts

After installing Racket I opened my browser to MIT's excellent book, Structure and Interpretation of Computer Programs. I have been through a four year engineering course, but never in all my time have I come across a textbook quite as impressive as this. I only had to get as far as the foreword to be amazed by the level of thought and clarity of writing observed in this text.

It opens with a strong and declarative first paragraph.

Educators, generals, dieticians, psychologists, and parents program. Armies, students, and some societies are programmed. An assault on large problems employs a succession of programs, most of which spring into existence en route. These programs are rife with issues that appear to be particular to the problem at hand. To appreciate programming as an intellectual activity in its own right you must turn to computer programming; you must read and write computer programs -- many of them. It doesn't matter much what the programs are about or what applications they serve. What does matter is how well they perform and how smoothly they fit with other programs in the creation of still greater programs. The programmer must seek both perfection of part and adequacy of collection.

To me, this first paragraph for the entire book gets to the real heart of programming. What it is, why we do it, and how. It invokes the mystical and powerful nature of programming that has entrapped many a young technologist.  It describes the difficulties faced on the road to learning this dark art; and hints that the road will be a long one. Programming is an art; perfected through practice and discipline.

Every computer program is a model, hatched in the mind, of a real or mental process. These processes, arising from human experience and thought, are huge in number, intricate in detail, and at any time only partially understood. They are modeled to our permanent satisfaction rarely by our computer programs. Thus even though our programs are carefully handcrafted discrete collections of symbols, mosaics of interlocking functions, they continually evolve: we change them as our perception of the model deepens, enlarges, generalizes until the model ultimately attains a metastable place within still another model with which we struggle. The source of the exhilaration associated with computer programming is the continual unfolding within the mind and on the computer of mechanisms expressed as programs and the explosion of perception they generate. If art interprets our dreams, the computer executes them in the guise of programs!

This is such a beautiful description of our art form. It lays plain that the real product of a programmers' work is not code; it is a model. This model dictates how we interface between the real world and the logic of a computer. This model may be but a part of a larger model, an actor in a greater world. As programmers we wrestle with these models; we seek to understand and optimise the model, until we can finally fit it within the constraints of a computer program.

Unfortunately, as programs get large and complicated, as they almost always do, the adequacy, consistency, and correctness of the specifications themselves become open to doubt, so that complete formal arguments of correctness seldom accompany large programs. Since large programs grow from small ones, it is crucial that we develop an arsenal of standard program structures of whose correctness we have become sure -- we call them idioms -- and learn to combine them into larger structures using organizational techniques of proven value.

Now the writer speaks about the content of this book, and its purpose. Computer programs always start small. Large computer programs are often made of many smaller programs. Unfortunately, as larger programs form, it becomes harder to manage the agreement of specifications and implementations. This is where computer science can help. By forming standard techniques in the form of idioms and algorithms, we can ensure consistency and quality in the approach taken no matter what problem is at hand.

Next, the language used in this text is considered.

Pascal is for building pyramids -- imposing, breathtaking, static structures built by armies pushing heavy blocks into place. Lisp is for building organisms -- imposing, breathtaking, dynamic structures built by squads fitting fluctuating myriads of simpler organisms into place. The organizing principles used are the same in both cases, except for one extraordinarily important difference: The discretionary exportable functionality entrusted to the individual Lisp programmer is more than an order of magnitude greater than that to be found within Pascal enterprises.

I think this is actually a reasonable approximation of the difference between the widespread modern languages such as C# and Java in comparison to some more academic or experimental languages. The former often strive to build structures and machines that may be very static and uninteresting in nature, but will stand the test of time. The latter type of languages seem to be very much more dynamic and unstable in nature, often trying new and outlandish ideas in particular areas of the language for a potentially positive gain. The writer finishes by adding:

As a result the pyramid must stand unchanged for a millennium; the organism must evolve or perish.

This is also a great description of modern businesses; the current hive of innovation in silicon valley is all about this "fail fast, evolve or perish" type of environment; meanwhile many much larger companies (and industries) that have long stood the test of time are starting to crumble in the face of such opposition.

To me, this book has already stirred deep thoughts on the nature of computer programming, and of the technology industry at large, and I have yet to write a line of code. It will be interesting to see whether the rest of the book is as enlightening.

All quotes are sourced from the Foreword of the book Structure and Interpretation of Computer Programs by Alan J. Perlis.

A Journey into Functional Programming/Lisp

I've officially been a "Developer" for about 2 years now, but still I can't shake this feeling that ever since I left university my skills have been stagnating. Maybe it's imposter syndrome or something; I just don't feel like the reality of the working environment has lived up to the ideals I conjured up while I was still undergoing tertiary education, or that I have lived up to what I thought I was capable of. Anyway, driven by this feeling of insecurity in my current skillset, I've been doing a bit of research on Functional Programming (FP), and in particular the Lisp-like languages which seem geared mainly towards FP.

My interest in functional programming was spurred by this blog post which turned up on Hacker News. It doesn't completely explain FP, but it does a good job of summarizing a few key points which definitely got me interested enough in Functional Programming to try and learn a little more. After searching around the web for information on Functional Programming there seemed to be at least 3 main languages used for functional programming; Haskell, Erlang, and Lisp. Yes, I know this glosses over many other Functional languages. These were the ones that I identified as the most widely used functional languages. That sentence seems to be a bit of an oxymoron: my research also revealed that Functional languages are only used sporadically outside of a few specific domains such as academia (Haskell) or telecommunications (Erlang).

I settled on Lisp for multiple reasons:

• Lisp is one of the oldest high-level Programming languages around, only rivalled by Fortran. It's been around for 50 years, and there's a good chance it'll still be around for another 50.
• Lisp has a range of different implementations which apply different tweaks and changes to the language, but all keep most of the same underlying syntax. This means Lisp is actually an ecosystem of many different languages, from Common Lisp, to Scheme (a whole other sub-category of Lisp-like languages), to Clojure.
• The language itself isn't that much of a jump from imperative programming. It is actually simpler than some mainstream programming languages; most Lisps break down into 3 main types of syntactical elements: lists, labels and atoms/values. It made sense immediately when I saw examples of it. I couldn't say the same for the first Haskell or Erlang code I looked at.
• There's some great material available from MIT on learning Lisp, since they used to use it for their introductory computer programming courses, as well as artificial intelligence work.

As I mentioned, Lisp is technically a collection of languages which have been derived from the original Lisp implementation over time. Scheme is a sub-category of these derivatives that focuses on a very small and stable core that can be easily extended. This makes it an excellent choice for an educational setting. In my case I chose to go with one of the (currently) most popular Scheme implementations: Racket (formerly PLT Scheme). It comes with an Excellent REPL (Read-Eval-Print-Loop) environment and plenty of built-in functionality so programmers can get up-and-running very quickly.

Now my plan is to work my way through How to Design Programs and Structure and Implementation of Computer Programs; hopefully I'll learn something in the process. Either way, I'm sure it will be an interesting journey into the world of functional programming and Lisp-like languages. Hopefully this will prepare me for a closer look at more advanced languages such as Haskell or Erlang in the future.

Free market ticket sales, the only real way to kill scalpers?

It seems to be all over the news at the moment; new legislation and ticket systems have done nothing to abate scalping of tickets for concerts and events in New Zealand and all over the world.

Scalpers depend on the supply/demand characteristics of a market being tipped far in favour of demand. What if the demand is removed? Ticket vendors and event organisers have tried to implement this with threats that tickets found to be onsold will not be honoured. The logic behind this is that if people don't buy tickets from scalpers, there will be no market for the scalpers to exploit. The failure on the part of the organisers and vendors is that this implementation of that logic is fatally flawed; good honest people still want to buy tickets; and because there is a finite supply of said tickets, they have to buy from the only source available: scalpers. This has been somewhat successful in decreasing the demand in the scalping market, but it has also meant that some people have had to make the conscious decision not to buy tickets for an event that they want to go to.

My theory is that the only way to truly reduce the demand for scalped tickets to zero (or anywhere near it) is to allow market demand to dictate ticket prices. Yes, I can hear the horrified screams of bands and music fans everywhere as I say this. What do you mean we should have to pay hundreds of dollars to go to the concert by that band I love?!?! WTFBBQ?!?!

Sure, ticket prices would probably skyrocket immediately. But consider this; instead of the hard-earned money you spend on tickets going to scalpers (if you're unlucky enough to not get in early), the money goes to the promoter, the venue, and of course the performer. For big name bands with huge fan bases this would be a massive win in terms of revenue from concerts. But what about the social implications?

Many bands are known to impose limits on ticket prices on purpose, just so that they can allow their most loyal fans and followers to attend concerts. Allowing only the richest fans to attend a concert will mean the overall culture suffers, right? Suddenly who can go and who cannot is dictated purely by how much money that person is willing to pay. Is this fair? That could be an entire other discussion in itself, and I don't particularly want to get into it here.

My thoughts are less on the implications of this on the followers and attendees of events, and more on the effect it will have for scalpers and bands, and how such a system could be implemented to maximise success for both bands and fans. I want to see less money going to scalpers and more money going to the performers and musicians that people are paying to see.

So how would we implement such a system? The only way scalpers can be defeated completely is if there is no market for the tickets they buy. So how can we do this?

• Decrease the demand for second-hand tickets - this is the approach currently being tried by governments and organisers.
• Increase the supply of tickets - this simply can't be done for most shows, especially if you already have the biggest venue in town.

Obviously, the only real option is to reduce the demand for scalped tickets. How can we do this? The approach already being taken is to reduce the demand by increasing the risk to potential buyers. This results in a loosing situation for buyers who just want to see their favourite band when their tickets are invalidated or they are barred from entering a concert.

What if ticket sales are turned into a priority list, ordered by the amount paid, which is completely fluid until tickets are printed at some time before the concert? Suddenly legitimate buyers can simply outbid scalpers. If scalpers try and outbid the legitimate buyers, they will find themselves with tickets on the night of the concert that no one wants to pay more for (because those people have already outbid the scalpers). Of course this system would not eliminate scalpers altogether; there would still be a market for scalped tickets among latecomers who wanted to buy tickets at the last minute. The implementation of such a system would be complex because of all the stakeholders involved. And as I have said, there are significant social issues and business relationships that would have to be sorted through before such a system could be put in place. Not impossible, just the sort of operation that would be suited to a committed and skilled entrepreneur.

The other option of course is to put up with scalpers, and accept them as a natural evolution of the supply/demand market. Either way, people will always have to pay market rates for tickets, whether that market is set by the organizers or the scalpers. There is no escaping the supply/demand curve it seems. That may sound cruel, but it seems that it is the honest truth of selling such limited quantity goods. 

I'm only an engineer, not a marketing or sales person. So by all means, feel free to discuss any flaws or shortcomings in the theories and opinions I have expressed.

SVN vs. DVCS

First post for 2012.

I was going to make this a SVN vs. Mercurial post, due to my recent efforts to push forward Mercurial as a replacement for the former at my current workplace. But such a post would miss the fact that what I'm really pushing is a migration away from SVN to a Distributed Version Control System (DVCS). Mercurial is just the choice of DVCS the company I work for has settled on, but there are plenty of other options, each more appropriate for a particular community or purpose. So SVN vs. DVCS it is.

I'll preface this post with a couple of disclaimers. First, yes, some of this is just my opinion. Secondly, SVN isn't terrible; there are still millions of software developers all over the world that use it daily as their version control system of choice (even if most of the hacker/open-source world has migrated away from it already). My argument is just that a DVCS and the workflow associated with it is better.

First, let's start with the word Distributed. Distributed Version Control Systems contain this word as part of their name because of the way they spread the repository out to all users. In SVN you would check out a particular revision and branch/tag of a repository (and that is all that is stored on your local machine); with a DVCS system, each user checks out a full copy of the repository. This means every user has all branches, all tags, and all revisions of all files.

You might think that checking out every tag, branch and revision would be very slow. DVCS systems mostly get around this by being smart about how they store and transmit changes; Git has a fairly complex database and garbage collection/compression system to keep on top of the worst of the repository bloat, while Mercurial only stores file deltas, instead of whole files.

Because of the way the repository is spread out to all users, DVCS systems allow users to commit to the local repository, as well as pushing their changes to a remote host. The biggest change in workflow this creates between SVN and DVCS systems is that it is now possible to commit any change to source control without affecting other users. I feel that this is the most significant benefit of a DVCS system on a day-to-day basis; the ability to commit early and often proves invaluable when it comes to experimentation and large chunks of development work. This has flow-on effects such as improving merge operations: changes are divided up into smaller chunks, and DVCS systems focus on the history of changes between revisions rather than the absolute state of a file at the two revisions being merged.

The workflow in DVCS systems is generally divided up into at least 3 steps:

• Pull changes from the central server or from another user.
• Commit changes to the local copy of the repository.
• Push the changes in the current repository state to the central server or another user.

One of my colleagues at work pointed out a very good downside to this pull-commit-push workflow; what if people simply work the their local version of the repository for a week and something happens to their computer, or the office burns down? The central server is backed up, but individual developer machines aren't. I mumbled something about no one being silly enough to do such a thing, but it raised an interesting point. Personally I think this would be the equivalent of someone working in an SVN respository for a week and not comitting; but DVCS lulls the user into a false sense of security on this front because when files are locally comitted, they're in version control, right? Sure, but this doesn't guarantee data replication and integrity. Logic should dictate that people always make sure they push DVCS code to a server or other user at least once every day or two.

But how do we deal with large chunks of isolated development in a DVCS repository? In SVN we would create a branch, commit to the branch, and then merge the branch back into the main trunk of the repository once we are finished. This process has never been as fluid or seamless as was promised in the early days of SVN, especially on large repositories and changes. In DVCS systems branches and tags are replaced by the concept of labels or tags. Because the repository history is encoded as a string of changes, the head/trunk of the repository is just a pointer to the most recent change in that string of changes. A branch is just a pointer to a different string of changes, which may share those changes with many other branches or tags. This means to create a branch, you simply create a new tag and assign it to the changeset you want to base the branch on, preserving the current head label/tag as it stands. If you need to merge the changes from a branch back into the trunk of a DVCS repository, the repository simply applies file changes in the branch on top of the file changes in the trunk.

Overall I think a DVCS system has huge advantages over a standard SVN workflow. The focus on changes, rather than the instantaneous state of a respository more closely aligns with a software development workflow. The decentralization of the version control systems allows for finer-grained control over commit points and allows users to commit code to source control even if it would normally have a negative affect on other users in an SVN-based version control system. SVN chains users to a repository server, while DVCS systems allow users to be the drivers of information flow.

Don't take my word for it, check out Mercurial, Git, or any of the other awesome DVCS systems out there!

Game development vs. Web development

As a reader of Hacker News, I have to say that the substantial majority of the content seems to be aimed towards web developers. I don't think this is on purpose; I think it just reflects the current state of technology.

For the last few years, even the last decade, we have been in a boom as far as web technology goes. The advancement of the state-of-the-art in software development, combined with the browser wars and subsequent arms race, has created an amazing array of technologies so advanced and comprehensive that Microsoft has chosen to base the GUI toolkit for Windows 8 on it. We have a range of programming languages and servers to choose from, combined with a huge amount of functionality on the client end thanks to innovations like HTML5, WebGL, and CSS3. This means we can now create real solutions to real problems using both client and server side processing in the browser. Where once there would have been dedicated terminals and a centralized mainframe, companies now use a web server and web-capable devices.

Games continue to push client hardware to the very edge, but games have also become a significant part of the web. Online game companies such as Popcap and Zynga command huge networks of players in online browser-based or flash-based games. Despite this, client-side games are still very popular, and when it comes to squeezing the very best out of hardware, native games still beat out browser-based games.

Game development as a profession is traditionally very desirable to young programmers. I may be over-generalizing, but the dream of most young programmers is to make it big creating games for studios like Epic, Bungie, and Blizzard. The reality is somewhat different. The industry has been rocked by reports of substandard work conditions and huge studio break-ups.  Because game development is such a desirable job, it is widely seen as an easy market to recruit new programmers in. This leads to problems like high burnout rates, high stress levels, and high staff turnover in some companies. I myself am at this stage as far as game development goes; the reality of long days and low wages has left me feeling burnt out and tired of game programming.

Web development is also a bit of a rock-star profession at the moment, because of the success of consumer-focused online websites, especially those in the social web space. Companies like Facebook, Google, and Microsoft are ready to pay top dollar to top-end web developers so they can compete head-to-head. At the same time, small scrappy startups are trying to offer developers a share of the pie and the chance to make it big, in exchange for long hours and low wages. At this point in my blog post, the two types of programming are really starting to sound very similar.

I guess in the end they are both similar professions in terms of the pressures applied to individual programmers. In reality to be doing anything at the extreme (and interesting) end of the technology spectrum, you have to be willing to compete with others by taking on risks like pay cuts and high-stress positions. In the end both high-end game developers and high-end web developers working with the latest technology both pay the price for their passion.

I sort of started this post aiming to let off some steam about how game development sucks because of the pressure and conditions (not that i'm really in "game" development), but i guess in reality, anything that pushes the boundary is going to come with high pressure. Competition is created by the mass of programmers who want to be challenged, who want to push themselves further. I suppose in reality, there are highs and lows in all sorts of development and programming, and in end end no matter what sub-field you choose, you will still get the full range of experiences, both the highs and the lows.

In then end, it's all just part of being a developer.

The Next Console Generation

After reading through the news today and seeing that Sony has outed active development of a next generation console, I can't help but feel like we could actually be seeing a new console generation beginning next year (2012). Sony is the last console company from the current generation to publicly acknowledge that it is working on a new console, after Nintendo revealed Project Cafe and Microsoft put out an ad looking for hardware architects for a new console graphics system.

After thinking about this, I wondered what the lead time was between the Xbox 360 reveal and the actual release of hardware to the public. While researching, I found an article trumpeting the launch of the 360 and had to have a laugh at what had changed:

Every Xbox 360 game is designed for high-definition, wide-screen televisions. Regardless of the television Xbox 360 connects to, gamers will experience smooth, cinematic experiences that far exceed anything they’ve seen or felt in games before.

I think we can all agree that the current generation of consoles does not push our 1080p HDTVs to their limits; most high-performance games still run at 720p, or lower.

Xbox 360 also features software so smart it remembers what gamers have achieved, continually evolving and enhancing the game experience.

Achievements must have been such a huge thing at the time (I certainly don't remember), but these days such features are expected, thanks to their success as part of the Xbox platform. Since it's release the software on the Xbox 360 has continued to evolve; by the current standards, the original Xbox dashboard seems so quaint.

Players can express their digital identity through their Gamer Profile; connect with friends anytime, anywhere through Xbox Live voice chat; send and receive text and voice message; and access Xbox Live Marketplace to download demos and trailers along with new game levels, maps, weapons, vehicles, skins, classic arcade and card and board games, community-created content, and more to the detachable Xbox 360 hard drive — all right out of the box at no extra cost.

I don't think anyone could have predicted how much of a success Xbox Live would be with it's tiered pricing plan. The value proposition for gamers seems to have paid off, with the platform receiving regular updates and additions such as party chat and the various dashboard redesigns.

Overall I think Microsoft can chalk the 360 up as a success; they didn't beat out Nintendo on pure sales or profit this generation, but I believe they delivered value and quality through experience on their platform. The platform features that are still recognizable on the platform to this day, such as voice chat and the Live Marketplace, are a testament to the decisions that were made correctly on day 1. Progress since then has been supported through continual investment into the platform and community.

Back to my original point. the official announcement for the Xbox 360 was made in May 2005, and the console was released about 6 months later, in November. This gives a lead-time of about 6 months over their official announcement. Given the stage of rumors around development of the next generation console, it seems unlikely that Microsoft would announce and release a console this year. My bet would be on a possible hint this year at E3, even if just to give people something to look forward to, with a reveal at E3 2012 and possible launch before Christmas 2012.

Of course this is all pure speculation! As much as would all like to know when new hardware will be arriving, it seems for now we will simply have to wait.

Rooting/Flashing Samsung Galaxy 3

I have just finished flashing my brand new Samsung Galaxy 3 (New Zealand model) to the latest 2.2 firmware found on the xda developers forum.

Here's the complete guide from xda-developers:

http://forum.xda-developers.com/showthread.php?t=778880

It still took me quite a while to figure out how to flash the phone, there are quite a few little quirks. Here is the process I followed:

1. Install the Samsung Kies application (comes with the phone on a CD), back up all your contacts, calendar, etc.
2. Use the z4root easy root app.
3. Install Titanium Backup for android, requires root access.
4. Run Titanium Backup and back up all applications and settings.
5. Shut down your phone.
6. Hold the Home, Menu, Volume Down and Power keys until a "download" screen appears.
7. Connect the phone to your PC via USB.
8. Follow the instructions from xda-developers on updating the firmware for the I5800 (Europe firmware seems to work fine for New Zealand phones). 

Note: when using the ODIN downloader to update your firmware, wait approximately 10 seconds after loading each firmware file. If these files are loaded too quickly in succession (before ODIN has finished checking the previous one), ODIN crashes with an "ODIN has stopped responding" hang.

Once you've completed the updating process you should be able to restore any apps and settings from the titanium backups.

UPDATE:
The latest firmware at time of writing (I5800XXJPF) turned out to be a bit flaky when trying to sync with the Samsung Kies utility, and in a few other places. Since I wanted to port my contacts over, this was less than ideal. The next most recent firmware does not seem to share this problem (I5800XXJPB).