Well after being under NDA for so long, I’m glad to be able to say that Amazon SimpleDB has gone into limited beta. Congratulations to everyone on the SDS / SimpleDB team; their several years of work on SimpleDB (formerly called SDS) is a brilliant piece of engineering.

What’s cool about SimpleDB

• Really large data sets
• Really Fast
• Highly Available - It’s Amazon. Running Erlang. Whoa.
• On demand scaling - Like S3, EC2, with a sensible data metering pricing model
• Schemaless - major cool factor for me here; items are little hash tables containing sets of key, value pairs

Considerations you’ll want to think about

• Eventual Consistency - Data is not immediately propagated across all nodes… the latency is usually around a second, but for high data sets or loads, you may experience more latency. On the plus side, your data isn’t lost!
• Queries are lexigraphical - You’ll need to store data in lexicographical ordered form (zero-pad your integers, add positive offsets to negative integer sets, and convert dates into something like ISO 8601)
• Search Indexes - You’ll need to construct your own indexes for text search - The SimpleDB query expressions don’t support text search, so you’ll have to construct inverted indexes to properly do “text search”. This is actually a really great lightweight way to do this and I’m sure many interesting indexing schemes will be possible.

Under the hood

According to the SimpleDB team, SimpleDB is built on top of Erlang. One of the developers, Jim Larson and I worked together at Sendmail, and he was part of a team doing some amazing stuff with an Erlang message store way back in 2000.

While you don’t need to know Erlang to use SimpleDB, many people have visited here interested in its Erlang roots. If you are interested in learning Erlang, I can recommend Programming Erlang, written by Erlang’s creator - the best introduction you can find. I’ve associate-linked to it on Amazon; just for a little meta-fun.

The data model is simply:

• Large collections of items organized into domains.
• Items are little hash tables containing attributes of key, value pairs.
• Attributes can be searched with various lexicographical queries.

Now you can easily build:

• Search indexes
• Log databases / analysis tools -
• Data mining stores
• Tools for World Domination

Further Reading

• Amazon SimpleDB Home Page
• Developer’s Guide
• Werner Vogels on Eventual Consistency

I also wrote a very basic Python module for SimpleDB to handle the XML and REST stuff (too bad it’s not JSON, at least for now), which I’ll release as soon as I figure out how much of the NDA is now lifted. There are a few floating around, so it shouldn’t be too long before they appear publicly.

Updates:

• Added a link to Nick Christenson’s paper on Sendmail’s Erlang message store - A great read for those of you building large scale messaging systems or anything in Erlang.
• Added a link to Werner Vogels’ article on eventual consistency - a great background behind SimpleDB’s consistency design choice.
• Whether or not SimpleDB and Dynamo are the same underlying technology has never been confirmed by an authoritative source. That’s all I’m allowed to say.

Technorati Tags: Amazon SimpleDB, SimpleDB, Amazon, Erlang, Databases

Here’s what John Carmack said at the WWDC ‘07 Keynote during his demo:

… in fact we’re showing on the Mac platform, the PC, the PS3, and the 360, the same data running. We’re going to be demoing this at E3, as well as our QuakeCon in August and I expect actually to have another Mac related announcement to make at that time which we can’t quite go into right now…

What could that announcement be? All of id Software’s titles have been ported to OS X already … could this be a port to a new forthcoming Apple console? (might this be Apple TV?) Or perhaps an original mobile game title or port for the iPhone?

We shall see.

Technorati Tags: Games, id Software, John Carmack, iPhone, Mac, Apple TV, Apple

It’s official. Want to build applications for the iPhone? Safari is the solution.
Web Standards is the SDK.

As I wrote about earlier this month, and this year, this is how you do it. It’s standards based, it’s pretty, and it’s the future.

Apple announced their iPhone extensions, there’s lots of integration points to make web apps first class on iPhone. Here are a few things you can do:

• Dial phone numbers.
• Send e-mail.
• Link to built-in maps.

All of the web developers on the planet just became iPhone developers. Ladies and gentlemen, here comes the Mobile Internet Revolution.

Technorati Tags: iPhone, Safari, Apple, OS X, WebKit

Today, John Gruber wrote an article asking for an iPhone SDK. Well, it’s here, but if you want it, you need to see things differently. –

For developers, the biggest thing about iPhone is that Apple has reinvented the mobile application platform. This isn’t mobile Flash, mobile Java, or even the mobile Web. It’s the real Web, the real deal.

The iPhone truly puts the Internet in your pocket. That’s huge. The Internet, the craziest, most amazing, important platform of our time is now on your phone. With a first class Web platform.

iPhone’s first 3rd party applications will be Internet services with web application or Dashboard user experiences. Apple’s will probably publish a few simple extensions; stuff to work with Multi-Touch high resolution, perhaps some offline support.

But remember, this is just the beginning:

Steve: What I’m saying is, I think the marriage of some really great client apps with some really great cloud services is incredibly powerful and right now, can be way more powerful than just having a browser on the client.
– Steve Jobs at D5

So until WWDC, you can find an iPhone SDK Developer Preview here.

</fanboy>

Update: I think people are also realizing what the implications of all this are. Mike Arrington @ TechCrunch put it very succinctly in that mobiles and desktops will stop living separate lives when iPhone (and its clones) arrives and changes the game completely.

Update #2: It’s official.

Technorati Tags: iPhone, Apple, Browsers, Platform, WWDC, Internet, Web, D5

A long post. Enjoy. :-)

Today, my friend Rand Wacker revived a great post about browsers as the ultimate application platform. At a high level, I agree with him, but the reason why we’re not seeing more “multi-platform” applications is that right now, for many companies, the web browser application platform presents the best choice.

Desktop and mobile applications are very expensive to develop and are extremely difficult to build at an Internet pace on today’s platforms. Mobile frameworks like WebKit and desktop frameworks like Apollo and WPF are trying to correct this, but it’s not an easy problem to solve, and hasn’t been solved right yet.

In my career, I’ve had the pleasure to work on mobile apps, mobile browsers, mobile web apps, desktop web apps, desktop Windows + OS X apps, Flash widgets, RIA frameworks and RIAs at FilmLoop and Openwave. So based on those experiences, I present you with:

The 3 Key Ingredients of the Next Ultimate App Platform.

1. Insane visual / graphics / text layout performance. It’ll off-load the heavy lifting to the GPU, do all the game programming tricks out there, and likely talks directly to DirectX and/or OpenGL.
2. Spectacular type-setting and type layout. Macintosh, PostScript, TeX, PDF, HTML. Each of these successful projects either did text right, or text fast.
3. Easy animation. I’ll speak more about this in a future topic, but to pass the bar Apple has set for application user experience, you must have animation and it must be easy. Most folks realize the need for animated UI now, but very few toolkits have made it easy. Apple’s Core Animation is one such toolkit, and you can see what it’s done for OS X Leopard and iPhone. At my company, PixVerse, we may open source our own implicit web animation toolkit once we get some free time.
4. The basics (Okay, so there are really four … ) The obvious things that so many have hyped and written about already… fast dynamic scripting language; database and searching engines; persistent storage for offline use; encryption; desktop notifications; networking; sync, etc.

Now for a disclaimer. This isn’t the whole picture. It’s just one piece — the visual user experience. It seems to be the one that the hyped up platforms seems to be addressing first… whizzy graphics and video. Important things are being ignored: machine learning; data mining; decent search facilities; robust multi-directional sync (have you read about RSS-SSE and Unison?)

Written using TextMate. ;-)

Technorati Tags: Browsers, WPF, Apollo, Applications, Platform, iPhone, Graphics, Video

FaceBook released their new OS Platform this afternoon. It’s a big deal, but it’s
hard to get a grasp on what the new changes are for developers.

Briefly, the FaceBook Platform extends the existing FaceBook API with:

• A few new APIs, for setting FBML on a user’s profile.
• A new FaceBook markup language, FBML, a subset of HTML with some FaceBook extensions. JavaScript and AJAX is not allowed per se (except via a Mock AJAX proxy), but Flash and SilverLight are.
• An application directory for new FaceBook applications.

I’ll post later why you might think of this as “Windows 1.0 with deep social network integration”, and all the dangers and excitement that goes with it, but for now… you can read all the fun details and try out some FBML here: (until the main Developers page is updated).

FaceBook Platform Developers

More coverage:
Facebook Launches Facebook Platform; They are the Anti-MySpace

Technorati Tags: FaceBook, FBML, APIs, Development, Mash-ups, Operating Systems

… a complete fabrication. April Fools!

Didn’t see it? See here: Digg.

Little easter eggs were mixed in that people found, namely:

• LayerKit instead of Core Animation
• LLVM as the static compiler and part of the runtime
• Apple Computer, Inc. instead of Apple, Inc.
• Bertrand Serlet as the PDF author
• OS X 10.4.8 internal developer seed
• Onyx as a successor to Quartz
• % date -r 1143931338
• Pixo is the original iPod OS
• llvm-gcc instead of gcc-llvm

WTF? Why? How?

• I like subtle April Fools jokes, and hadn’t seen any truly convincing ones so far. I had to be careful not to get carried away and slip on something (like llvm-gcc).
• A lot of the components (Core Services, BootCache, etc.) come from the Apple TV dissections posted around the net. I grabbed the file dump from a homebrew site and cherry picked some frameworks that seemed likely.
• LLVM comes from my prior posts on the LLVM ARM backend contributions from Apple.
• Onyx was picked because it matches the Quartz theme, and it’s black, matching recently designed Apple UI, including iPhone.
• And finally the major features (and the fact that it’s OS X) are derived from the Macworld keynote.

Thanks for the great comments, I hope you had a good laugh and it wasn’t too annoying. ;-)

Technorati Tags: iPhone, Apple, April Fools

Wow. I was surfing Scribd this morning for some code snippets and I happened upon this thing. It looks like a high level block diagram of the OS X that’s running in the iPhone. I’m not sure how much you can tell from this picture about what’s going on, though.

Technorati Tags: iPhone, Apple, OS X, leakage

I have good friends who work on “secret” projects at Google (among other companies). Recently, news about Google’s acquisition of Mike Reed’s company, Skia, was leaked. Thus, I believe I can talk more freely about it now.

So here’s the thing. Last year, Google open sourced the Skia vector graphics engine and placed the code up on Google Code under an Apache license. I heard about it through the grapevine (not from Mike) and downloaded a copy to take a peek. Shortly thereafter, the code disappeared.

So here’s the general dilemma I pose to our virtual community:

• What is the legal standing of the Skia source code at this point? It was made available briefly under an Apache license, possibly for several weeks. Is it still Apache licensed code, even though the code has been removed?
• What do you think would happen if I were to post it again for posterity? What do you think Google would do?
• What are the implications of this to open source in general?

And remember, don’t be evil! :-)

[Update #1] It seems that not many people know what the Skia engine is capable of. Skia built a vector graphics renderer core very similar to the one that Openwave Mobile Phone Suite V7 uses. (Mike built both, and I worked for him at Openwave) Skia is capable of full Java2D or PostScript (group opacity, bezier curve paths and clipping, kickass type rendering, gradients, filters) on typical “feature phone” hardware (average clocked ARM9ish processor, decent memory bandwidth) with a ~300K footprint. It’s like OS X’s Core Graphics, without the GPU rendering.

[Update #2] Simeon Simeonov, Engadget and CrunchGear all have follow on stories.

[Update #3] An interesting twist / pickle emerges: When I looked through the source archive this morning, I found that the actual Apache license is not included with the source code. It was only published on the Google Code web site! Not only that, but no copyright information exists or even authorship is in the codebase. So what now? I definitely need to speak to a IP lawyer.

[Update #4] According to a few authoritative sources, source code with an Apache license cannot be revoked once published. Source code without an accompanying license isn’t as clear cut a case and is still being examined.

[Update #5] I’ve decided not to release the source code at present, not for legal issues, but as a favor. I’m confident that this source code situation will, however, be resolved at some point by Google. :-)

Technorati Tags: Google, Skia, vector graphics, open source, graphics, legal

I’d like to connect the dots between Apple’s iPhone Graphics, OS X, LLVM and the ARM architecture, and what it means for compilers, interpreters, high-performance embedded software systems, specifically in graphics and bytecode engines.

We’ll start with what the heck LLVM does, and go from there.

• LLVM is a fascinating compiler system that optimizes code at many stages, from compile-time to run-time optimizations; e.g., your code is optimized while it’s still running!
• Apple uses LLVM to optimize OS X’s OpenGL pipeline to emulate older video hardware features and accelerate modern hardware.
• Apple is integrating LLVM with GCC and supporting OS X’s executable file format. Chris Lattner posted about it here.
• Apple has also contributed ARM backend enhancements to LLVM that would allow a ARM version for portions of a OpenGL graphics pipeline, and subsequently, Core Animation.

It means that any type of performance critical code that’s “interpreted” or “pipelined”, like OpenGL pipelines and JavaScript engines, can be effectively JITed automatically through run-time optimizations via LLVM.

Let’s see: Simple JavaScript engine, with automatic JIT compiler vs. manually coded JIT VM and compiler… hmmm….

This also has interesting implications for high-level languages like Ruby. Could a LLVM-Ruby provide the performance of YARV without the complexity of a manual JIT compiler and VM? I know that Python’s looking at it…

Plus you’d get a JIT embedded ARM mobile processor port of Ruby for nearly free. And if that’s true for Ruby… why not for all the important bits of OS X?

Ahh, so that’s how iPhone runs Mac OS X …

Technorati Tags: iPhone, OS X, Apple, LLVM, Graphics, OpenGL, Ruby