Ones and Zeros

Cory Doctorow and Joel Johnson are debating the closed vs open source issue with respect to the Apple iPad and the Apple software ecosystem.  Cory says he’s not buying an iPad and he doesn’t think anyone should.  That we shouldn’t buy devices that force us only to be consumers, we need to buy products that let us take them apart, customize them, and fix them.  For computers like the iPad, this means to write software outside the DRM box as well.

Joel, on the other hand, points out here that computers becoming consumer appliances is a good thing.  That we don’t want to have to tinker with everything we buy and use, that just being able to use something efficiently and cleanly is a wonderful thing to do.  And that for complex devices, locking the design and internal access down is one way to do that.  Cory and Joel both are making the issues clear so people have a better chance at choosing the techno-ecosystem they want to play with.

They’re both right in a way.  Cory and Joel both are making the issues clear so people have a better chance at choosing the techno-ecosystem they want to play with. We have to make our choices as to what we’re willing to live with.  Apple’s created an ecosystem that’s much larger than the device.  That they’re able to do this in a closed way is impressive. I’ve discussed that before.  When you buy an iPad, iPhone, or iPod you’re buying into that ecosystem.  As a consumer that may be worth it to you, it is to Joel.  It’s not worth it to Cory.  And as a developer there are rules that I have to follow to play in that system too.

But there are rules for consumers and developers in the open source ecosystem too.  Those rules are  unwritten social rules perhaps, or ways of working dictated by the nature of open source projects: installation, updates, system management.  In the end the finished open source product typically requires more tinkering than many would like to see simply because there’s no overriding corporate authority driving a more unified approach.  (I’ve used Linux since the mid-1990s, written software for it, and written a kernel mod (a driver).  I like it and it’s great for my purposes, but requires tinkering.) Those open source rules more reached more collectively than the Apple ones, but as an independent developer I have as much effect on them as I do on Apple’s rules generally.

Basically, you have to decide what you want to deal with.  Sometimes I choose ease-of-use and pay the DRM penalty, sometimes I choose open-source and pay the tinkering penalty.  Trying to use either one for everything would be like trying to use a hammer in place of every tool in my tool box.

In this post Clay Shirky discusses the collapse of complex business models and Tainter’s book The Collapse of Complex Societies.

…a group of people, through a combination of social organization and environmental luck, finds itself with a surplus of resources. Managing this surplus makes society more complex—agriculture rewards mathematical skill, granaries require new forms of construction, and so on.

Early on, the marginal value of this complexity is positive—each additional bit of complexity more than pays for itself in improved output—but over time, the law of diminishing returns reduces the marginal value, until it disappears completely. At this point, any additional complexity is pure cost.

….

When the value of complexity turns negative, a society plagued by an inability to react remains as complex as ever, right up to the moment where it becomes suddenly and dramatically simpler, which is to say right up to the moment of collapse. Collapse is simply the last remaining method of simplification.

As Shirky points out in his post, the same thing can happen in business.  I believe something similar can happen with any over complex structure such as some software projects.  The FAA famously killed their modernization project in the late 1980’s due to this.  Way back in 1975 Fred Brooks in his book The Mythical Man-Month observed, “Adding people to a late project only makes it later.”  In reference to theoretical physics Einstein said, “Everything should be made as simple as possible, but not simpler.”  Sounds like good advice in general.

The study of failed software projects and systems should be part of our standard training for a software engineer.  In aviation accident analysis is an important area of study and civil engineers also do this.  I’m not the only pilot who’s read years of NTSB reports.  However in software we have nothing similar.  Fred Brooks’s book is a good start; it’s a classic.   But due to corporate embarassement and proprietary property issues, there’s no similar study in software engineering.

This lack forces us individually to replicate all of our predecessors mistakes.  The progress of our field suffers.

Sometimes it’s just one thing after another.  It’s been like that in my coding lately.  I need a plotting/graphic library for an iPhone app I’m writing.  I could write one, but it’s faster if I found one that I could use.  I’d prefer to focus on my app instead of writing a library.  Lo and behold; there was Google’s core-plot for the iPhone.  Wonderful!  Looks like it’s probably quality code and appears well supported.

It was archived under Mercurial source code control system in Google’s code website area.  So I needed Mercurial.  No problem.  I found an install for my Mac, I don’t even need to build from its source.

Ah.  The install doesn’t work.  It hangs and has problems with my Python version.  (Mercurial is very Python dependent, much of it is written in Python with a few native extensions.)  So I’ll install the necessary Python version.  But it still doesn’t install.  Several experiments (and days) later I decide to build from source.

Hm.  That doesn’t work either.  I start tracing and fixing problems: wrong SDK, wrong gcc version.  Of course the build process isn’t in an old-fashioned Makefile that everyone knows, but a new (to me at least) Python build process.  That means two things to learn: the build process and the Mercurial build.

Remember core-plot?  That’s what the goal is.  Actually, the goal is my app.

I’ll spare you the details of the more days working out the problems, downloading the correction older SDK and gcc and working it all out.  I finally was able to execute the one command to get the core-plot source locally on my machine.  I think some inconsistencies in my system installation with Python and the SDK/gcc version issues threw off the Mercurial build/install process.  It’s designed for an older system release than I have.

This is one reason that software development can take longer than expected.  There’s always some “little” problem that becomes another problems and so on.  This one’s resolved, but there will be others. I try to be positive and consider it job security.

It’s curious.  In a world becoming more and more open-source and using more and more open-source software (OSS) Apple is thriving being proprietary.  They are using OSS in some areas, and are participating in open source projects as well.  WebKit in their Safari browser is one major example.  Yet much of their core system software is and will remain proprietary.

Certainly they’re successful.  Measured in name recognition, units sold, $20 billion plus of cash in the bank, or stock price they’re clearly a major force.  (I wish I still had my stock grant from my employee days!)

The argument in open source circles is that a proprietary effort like Apple’s will always fail in the long run when compared to open source.  IBM became a major driver of Linux, Microsoft has felt the pinch more and more, and even the federal government and DoD are proponents of Linux and open source software.  How is Apple not merely surviving, but thriving being proprietary?

I think it’s two fold.  First, while Apple is using open source software, they’re just using it where it’s not a key part of their company’s added value.  Second, they are using their proprietary software in an integrated ecosystem of software systems that all work together.  This provides a “user experience” that enough customers value over Apple’s competitors.

By leveraging open-source software (OSS) in non-key areas Apple frees up engineers and effort on their key proprietary areas.  This is a no-brainer.  Yet many companies fail to do this.  Apple’s using Mach UNIX as their base OS on the Mac, iPhone, and iPad avoiding rewriting the OS.  They spend their efforts writing more complex layers on top of the OS to provide a richer computing environment.  Better graphics, animation in the UI, richer audio, apps like iPhoto, Aperture, the iWork suite are all enabled by this as a random grab-bag of items.  By comparison Microsoft writes everything - they’re working twice as hard and getting rather less the same result.  Their last several releases have suffered by many measures.  Apple focuses only on writing the software that’s key to their system.

But Apple embraces the proprietary and customers (despite the grumbling about DRM) embrace the Apple products.  MP3 players existed before Apple, but the iPod massively expanded that market to the point where it is dominated by Apple.  Smart phones and smart phone apps existed before the iPhone,  but Apple’s remade that market as well.  Through these innovations they’ve fostered the development of things like podcasting, and created a whole new developer market for iPhone apps.

I believe they key to their success in proprietary software is that they’re creating a very well integrated software ecosystem.  Having an iPod is richer with iTunes, they enhance each other.  You get used to this, then find the iPhone works very well with iTunes too. It’s a natural upgrade path.  The lack of the involved setup and configuration necessary with many Windows software systems helps tremedously.  the Mac systems just feel right, things fit well.  The Windows seems clunky, a moved window isn’t refreshed smoothly - it’s the little things that make a system feel like it is working well.

It’s an ignored truism in software that customers just want to get something useful done.  It’s geeks like me that want to spend their time messing with software.  Yet even I want to focus on the things I want to play with - not the boring configuration & setup stuff on the way.  The Mac/iPhone software ecosystem allows me to do that.

This singular focus on how things feel and work smoothly comes, I believe, from Steve Jobs.  Certainly Apple is much larger that Jobs and all employees are important to Apple’s overall success.  But the leader stamps a personality on the company.  Jobs is famously driven a high standard of usability.  His equally famous control on information and access allows Apple to work on something till they’re happy with it and not be bound by a schedule to they are ready to release.

That’s a key point.  In engineering it’s said that there are three things: schedule, quality, and features.  You can control two, but the third must flex.  If Apple drives quality and features, then schedule must be flexible.  Their control on public information allows that.

So a way for companies to compete against OSS is to use their proprietary work to provide a smoothly integrated environment, a software ecosystem.  OSS efforts have a hard time with this since the level of coordination and interaction required is high.   Most OSS efforts (barring Linux itself perhaps) are smaller and more loose.  The weak point of current OSS projects is their democracy and openness.

(Note: I’m a proponent of OSS and have contributed to OSS projects.  I used OSS daily in my work. But I find it interesting that Apple’s been contrarian and successful in their proprietary efforts.  That probably isn’t a universal solution for companies facing OSS competition though.  Most companies can’t maintain the high standards of quality necessary the Apple solution.)