Adventures in JVM Serialization - part 2

So I got the software from 4 feet to run, though I wish he included all the files so that I could just import the project rather than have to hack odds and ends together and create a new manifest.

I'm a bit disappointed, I would have liked a bigger data set, but this is a starting place.  On my Nexus One, the JSON code took from 410-156ms, with the higher numbers at the beginning of the run, the typical being about 178.  With Protocol Buffers 2.2.0, compiled in, we get from 210-116, typically about 122.  With Protocol Buffers 2.3.0, using the lite JAR, we get from 245-93, 117 typical.

The code is not quite androids vs. androids, so I'm not entirely happy, but it's enough to confirm the other findings, so I guess I"m happy enough to call it a night.

JVM Serializers for Android - Part 1

I've been thinking a lot about Android <==> Server communications lately.  Well actually for a long time now.  The issues that I care about, of course are:

1. Message Size
2. Size of Library
3. Memory usage
4. Overall speed
5. Compression Speed
6. Decompression speed
7. Ease of Integration

I started thinking I was going to use XML, but I hate XML. Then I thought I'd use some kind of simple JSON, and I found the JSON Schema which actually has a draft RFC and tools.  I actually wrote quite a bit of code around JSON. Then I remembered Google's Protocol Buffers, which has versioning as a feature, and produces very small messages.

Fortunatly, I'm not alone on my quest, I found Serializing Data - JSON vs. Protocol Buffers, I was planning to just rerun his tests on Froyo, review his code, and be done with it.  (Froyo has a different 'faster' Dalvik VM, and is suposed to have much better JSON parsing.) But no, I found JVM-Serializer Benchmarks,  it has lots of great data, but is still missing one of my key data points, how much memory does it use.  The benchmarks have unfortunately pointed me to two additional projects Kryo and ProtoStuff.

Kryo seems to be the overall winner, but it doesn't really support versioning.  That's probably ok for my needs, but I need to really think about that as there may be somewhere an issue.  The other question is with the faster JSON implementation, and it being in the ROM, do I wish to pay the cost for an additional library.

It's now time for the beach, so I'll write more on this tonight or tomorrow.  For that I'll have modified the android test program to add in Kryo, and possibly ProtoStuff.

java.lang.VerifyError on Android

For the, you learn something everyday file.

Google shows you in the Backwards Compatibility and Using the Contacts API articles the right ways to write code for two versions of an API, but they don't tell you why this is important.

It turns out that JVM and Dalvik do a byte code verify on class load, and if there's something that's not supposed to be there, like a new API call, it won't load the class. (Ya, well duh!) I've read about it over the years, but never needed to do it until recently. It's of course fairly easy to fix, once you realize it, you've just got to put your new API code in it's own class, and only load the class when you need to. Or you can do it via reflection. I.e. it's a pain, but useful if you want just one binary. (APK, JAR, WAR, etc.)

This time is Different - 800 years of research

One always hopes that Empiricism and Theory will match up. Otherwise, what's the point? We've currently got a lot of economists and politicians who are saying we must be austere. Well, as Paul Krugman points out, what's the evidence for that?

Economists Who Did Their Homework (800 Years of It) - NYTimes.com: "Like a pair of financial sleuths, Ms. Reinhart and her collaborator from Harvard, Kenneth S. Rogoff, have spent years investigating wreckage scattered across documents from nearly a millennium of economic crises and collapses. They have wandered the basements of rare-book libraries, riffled through monks’ yellowed journals and begged central banks worldwide for centuries-old debt records. And they have manually entered their findings, digit by digit, into one of the biggest spreadsheets you’ve ever seen.

Their handiwork is contained in their recent best seller, “This Time Is Different,” a quantitative reconstruction of hundreds of historical episodes in which perfectly smart people made perfectly disastrous decisions. It is a panoramic opus, both geographically and temporally, covering crises from 66 countries over the last 800 years."

History runs in cycles, as much as we want to deny it. Want to learn those cycles, then look at a real data set, not just 30-50 years.

I'm looking forward to reading the book.




I think Rogoff/Reinhart's paper Growth in a Time of Debt is worth reding.

We study economic growth and inflation at different levels of government and external debt. Our analysis is based on new data on forty-four countries spanning about two hundred years. The dataset incorporates over 3,700 annual observations covering a wide range of political systems, institutions, exchange rate arrangements, and historic circumstances. Our main findings are: First, the relationship between government debt and real GDP growth is weak for debt/GDP ratios below a threshold of 90 percent of GDP. Above 90 percent, median growth rates fall by one percent, and average growth falls considerably more. We find that the threshold for public debt is similar in advanced and emerging economies. Second, emerging markets face lower thresholds for external debt (public and private)—which is usually denominated in a foreign currency. When external debt reaches 60 percent of GDP, annual growth declines by about two percent; for higher levels, growth rates are roughly cut in half. Third, there is no apparent contemporaneous link between inflation and public debt levels for the advanced countries as a group (some countries, such as the United States, have experienced higher inflation when debt/GDP is high.) The story is entirely different for emerging markets, where inflation rises sharply as debt increases.

More of Ken Rogoff's Writings

Choices with GData, and the new API's

Well, I've been investigating using the new Java GData client libraries for Android to get Google Spreadsheet data.

Doing my experiments with CURL, I go through and get past GoogleLogin, the the auth token, and figuring out which feeds, entries I want. Just a note, make sure you login using the right Service Name, as you'll be able use the URL's in a browser, but not w/ CURL if you get it wrong.

The Picasa Android sample code works if you remove two @Override. Seems to work after that.

Table feeds seem lame for use w/ new API, it names all the fields with your titles, which is nice, but not helpful if your trying to generalize.

Spreadsheets also don't support JSONC alt type. So, I'm really inclined to write a python GAE helper to get the data, and reformat it for Android, it will make everything quite a bit simpler, and a lot smaller.

Spreadsheet Feed - a list of spreadsheets


ForEach SS
Worksheet Feed
ForEach WS Table Feed - Not nesc. defined for all spreadsheets


ForEach Table
RecordFeed


ForEach WorkSheet ListFeed - A bit nicer, labeled data, good for most applications, but not a general spreadsheet.


ForEach WorkSheet Cell Feed - The way we expect a spreadsheet, very wordy, but general. (I'd like this w/ JSONC & fields query)

The Doc's say that I should use ?updated-min=2010-06-27T00:59:30.487Z, but I'm finding that updatedMin=2010-06-27T00:59:30.487Z actually works. Unfortunately, like eTags, it's a on/off switch using the cell feed, ie, I either get nothing, or the whole spreadsheet. What I'd like is all that's changed since this time.

Cyrus

I've played Iago, but I get creeped out by people who manipulate. Cyrus starts out that way, but it becomes a battle of wills, much like Sleuth. Watching two players fight while keeping the fight from the third they both love.

Being a Hollywood picture, Love is what brings this picture to it's conclusion. Not what one expects from Ridley Scott. Great performances. It does make me wonder what of the story landed on the cutting room floor.

Cyrus

Little Miss Eyeflap

Little Miss Eyeflap, a short, played before the RockSteady Concert footage, well worth watching if your ever given the chance. The film maker deserved a bigger audience, she came all the way from Norway. It was her vision of life as a Pakistani transplant in Norway. Alas, in a sense, it was the same as the base of her film. She was screwed by her relatives, and the norwegians, but she came to see the beauty in all around her. I hope she at least sees Maui while she's here.

Little Miss Eyeflap at Sundance

Rock Steady - The 2009 Montreal Jazz Festival Concert

What if you had a chance to watch a film for the first time? One so special that you made it after you finished your original documentary. What if NO ONE CAME?

Well, I got to see a wonderful film this evening, the concert film of the 2009 Montreal Jazz festival inviting the characters from the documentary that was shown the other night on the Green. Many folks thought this was the same film, it wasn't. It was the follow-on concert. And the making of the Concert video. Quite a show, though, I'd have liked the sound to have been a bit louder. RockSteady - is what came before Regge. Really incredible uplifting music. A reminder that small music movements can produce great results. Now if someone would only make a film about Cape Breton Music.

Little Miss Eyeflap, a short, played before the film, well worth watching if your ever given the chance. The film maker deserved a bigger audience, she came all the way from Norway. It was her vision of life as a Pakistani transplant in Norway. Alas, in a sense, it was the same as the base of her film. She was screwed by her relatives, and the norwegians, but she came to see the beauty in all around her. I hope she at least sees Maui while she's here.

The RockSteady Documentary

This Way of Life

It's hard to believe it's real, but it is. The documentarians started documenting the live of the Karenas about six months before they are to be evicted from their home that burns down just as they are packing to move. It's a slow movie, but no bullshit, just an alternative way of living that most of us have forgotten. One that would be difficult to pick up in America. It's simple, living off the land. That way is dying in New Zealand, and the Karenas are the last of it. Very hearty, very self-sufficient. You just want to know all the principles.

That said, I have friends who try one tenth of what they do here on Maui. Living in the forest, rarely poking their heads out. Outlaws of a sort, or is it living life as it used to be in America? Somewhere I recall reading the phrase "Life, liberty, and the pursuit of happiness". I wonder what happened to that idea?

This Way of Life

Sympathy for Delicious

It's a very Catholic story told in a modern way. Destitute & crippled man gets power to heal most others - not himself, well meaning priest exploits man, man seduced to exploit power, power doesn't work when needed most, preist retaliates against man - leaves priesthood, man goes to jail & forgives ex-preist, man heals enemy w/o gain, man get's miracle.

If you think that I've given it all away, well, I haven't. It was a good film, well acted with powerful actors. I enjoyed the film, it just promotes a vision of the universe I disagree with.