The coming data crunch

Over the past few years, the carriers in the USA have been steadily changing their pricing plans, offering lower and lower data caps for a slightly lower monthly fees, and getting rid of unlimited and higher capped plans.

The majority response to these changes are favorable, as the majority of people look at their current monthly utilization, see that they are currently below the lower, cheaper cap, and lock in to that plan. They feel like they are saving money. Any they are locked in so they feel they will continue to save money.

The truth is that this majority is acting in the most blitheringly stupid idiotic way possible. They are allowing the carriers to bribe them now into smaller capped plans so that they can be royally fleeced by the same carriers later.

How? This majority has no idea how quickly their bandwidth use is growing and how this growth is accelerating. And they have no idea how much the carriers do charge when they go over. It’s like the the overage interest charged on credit cards. Get card, pay it off, save money. Go one day over and suddenly you are paying 26% compounding interest and getting hammered.

The majority assumes there will be a way out when they hit the carrier limits. They are wrong. Here’s how.

The myth of choice.

Back five or ten years ago, US households had a multitude of choices for broadband internet, these days, it’s no longer true. According to the FCC via the Washington Post, 78% of US households have a choice of only two providers, 13% can only use 1 provider. So really no choice.

With cable, its even worse. If the National Broadband Plan comes into play, then up to 15% of households will gain choice in cable providers (FIOS fiber is included as cable), the remaining 85% will still have no choice (see the NYT article here). Word is that currently only 2% of households currently have a choice in cable provider. 2%! I know! 98% have no choice!

Which means that the cable providers (who are the broadband providers) own a monopoly on 98% of US households. If you want cable TV, if you want fast internet, you can only use one provider and whatever plans they offer. Since there is no competition for your household, the carrier who has the monopoly over you has absolutely no interest in providing competitive services or pricing. If you don’t like the plans offered, you don’t get TV or internet. At all! No choice.

What about satellite? If you want TV and slow internet on sunny days, and only if your landlord allows dishes, it may be an option. Until you see their capped plans are even smaller. And ADSL? The ADSL providers are going out of business because the line sharing law, which they relied on, was reversed in 2005.

There exists an organization called the FCC. It’s part of the government and its job is protect consumers from monopolistic and unfair practices by communications carriers, like cable TV, phone and ISP's. Turns out, lobbying by these carriers over the past few years (and the hiring of ex FCC commissioners by them for fat salaries) has led to a situation where the mandated government organization, the FCC, no longer has the jurisdiction or claws to monitor or regulate this industry, and therefore protect us consumers.

It is a myth that the average consumer has a choice in how they access the internet and in the plans that can get.

The myth of unlimited

Riddle me this. What is the cap on your home broadband internet connection? Its unlimited right? Do you know? Can you even find out? I bet not. I signed up for an unlimited plan on my home internet (both providers), yet they both manage an unlimited plan as a plan which they arbitrarily throttle and cap. So its not unlimited! Its capped! The cap itself is whatever they want it to be, and they don’t say what that cap is. What happens when the consumer hits this unknown cap? They get cut off. And there is no recourse. No option. No one to switch services to, no plan to upgrade to. No choice.

Those who have the AT&T unlimited data plan on their iPhones know what I am talking about. AT&T throttles or cuts off people using what they say is too much bandwidth on unlimited plans. Again, no one agreed to the cap, no one knows what the cap really is, and AT&T does what it pleases. Don’t like it? Nothing you can do. All the other carriers do the same, at the same price point, at the same unknown and very variable cap.

I smell cahooting. It looks to me like cartel behavior.

It is a myth that unlimited plans are capped, or that they will even exist again in the future.

The myth of static use

If you have gotten this far in the rant, and I’d be surprised if you do, you may be thinking that all this does not matter, your usage of the internet is not even close to the unknown caps on broadband or mobile, so why worry?

Here’s why.

• Five years ago you subscribed to, erhem, magazines and newspapers, no bandwidth used. Today you read all you want on the internet, and your ‘special’ needs (nudge nudge wink wink) are handled via streaming video, lots of bandwidth used.
• Three years ago you phoned your friends to plan a party, no bandwidth used. Today we use Facebook and FourSquare not only to plan, but to post photos afterward, lots of bandwidth used.
• Two years ago, you only got text email on your phone, little bandwidth used. Today, you video chat, use apps, load maps, and run your life on an iPhone, lots of bandwidth used.
• A year ago, you rented DVD’s, no bandwidth used. Today you stream Netflix, lots of bandwidth used.
• Six months ago you watched programs on TV, no bandwidth used. Today you watch shows on YouTube, Hulu and other services, lots of bandwidth used.
• Three months ago you listened to music on an iPod, no bandwidth used. Today we have Pandora and Spotify, lots of bandwidth used.
• Soon (like next month), your phone will download its own updates, replicate into the cloud and update its apps using the internet, using even more bandwidth.

See the trend? Your bandwidth use has doubled or quadrupled year in year out.

The carriers know this.

Project forward, your bandwidth use will double or quadruple annually again. How quickly will you hit the mysterious cap and be forced to pay the insane overage fees, or cut back on services.

The carriers want to make money off this trend. They are relying on apathy and consumer short-sigtedness to lock in a beneficial cap and fee structure for them. And the majority is blitheringly stupidly going along with this.

It is a myth that you won't bump up against your cap soon, whatever the cap is.

Aside: it gets worse

The current plans for LTE on all carriers have an interesting feature. If you use your LTE phone at maximum LTE speed to download something, you will hit the bandwidth cap in about one hour. One! So what do you do for the remaining about 719 hours in the month? You pay overages, expensive overages, or turn off your LTE data. I see the carriers rubbing their hands and drooling over this.

And I bet none of them will offer an upgrade from 3G to LTE for unlimited plan holders.

Data crunch

So while we are all happily watching movies, posting photos, video conferencing and listening to music over the internet, the carriers have circled, and set the trap.

As soon as our usage hits these caps, probably sometime this year, the carriers will make a killing. They will not offer higher caps, they will not offer unlimited plans, they will not offer any choice. Pay them, or piss off.

A data crunch is coming, when need for more data gets crushed by the carriers cap and fee structures. Consumers will get hurt, companies that rely on the internet will get hurt, innovation will be stifled in the USA, and no-one seems to care.

Or maybe I am the only blithering idiot worrying about this.

The hiltmon has left the building.

On TextMate 2

TextMate, for the three people in the world who don’t know, is simply the best programmers editor on the Mac. It was released in 2004 by Alan Odgaard in its most basic form, and won the Apple Design Award in 2006. What makes it so unique is that it ‘knows’ the context in which you are working and provides language specific shortcuts and snippets. For example, when working in Ruby on Rails, you may have an HTML file open with embedded Ruby code. When the cursor is in the HTML bits, TextMate acts like a HTML editor, move the cursor into a Ruby bit in the same file and TextMate acts like a Ruby programmers editor.

I purchased my first license in April 2006, one of the first 10,000 users, and have used it almost every day since. The purple gear icon is as much part of my dock as the finder smiley face. Even today, five years later, I still start my work day by opening a command prompt in my current Rails project and type in mate .. Everyone I know who programs on the Mac uses it, and swears by it.

What you may not know is that it’s very rare for the same software program to be used this frequently and for this long a period of time by this many diverse users without a major update. As any product, it has had minor patches and fixes to make it even more stable and reliable. But over the same time, I have upgraded OS X 3 or 4 times, changed word processors, blog hosts, mail servers, programming platforms and jobs!

TextMate is an exceptionally successful product. I wish I had such a product, all indies do. It didn’t happen overnight, it took a while to get stable (2004 - 2006) and then build up it’s feature set. Yet it has remained the trustiest programmer’s editor out there. Success is hard, making a great product is very hard, maintaining this product given its diverse users and use patterns is even harder, and yet, MacroMates has done it, brilliantly. It’s still in my dock, in many programmers docks (even famous ones like DHH), and used every day by thousands.

But TextMate 1 has its problems too, as all version 1 products do. The ATSUI text editing model is old and limiting, the undo and search need work, paned editing would be nice, etc. Alan recognized this a long time ago and set out to rewrite TextMate as TextMate 2.

If writing a successful product is hard, rewriting a blazingly successful product like TextMate is a Sysyphean task. The first release took only 5 months, the new version could and should take 10 times that long. The new version has to be better than the old in every way, on a completely new technology platform. They have a ton of work to do to replace the existing features with the same, add new ones without breaking existing customizations and make it fast and reliable. And to make it an even tougher challenge, TextMate users have insanely high expectations and are super vocal about it.

I believe that TextMate 2, my friends, will not and should not be the same as TextMate 1, it will not be as stable to start with, will do things differently and may not start out with the same feature set. But I expect it will be a whole bunch better. And to make the job even harder on the developers, they know a storm of comments, vitriol and complaints will hit them no matter what they release, no matter how good it is.

Just take a look at the public discourse on TextMate 2 right now just by googling it. People, strangers really, discussing it in StackOverflow without any facts, web sites saying nope its not released today, calling it hopeless or vaporware or nukem. Even nobody’s like me writing this article. I don’t envy Alan and his team; if I was faced with all this derision, I’d probably just give up and go do something else.

But there is a silver lining. Amongst all the white noise is that fact that all new programmers editors are compared to TextMate, most new text editors use the idioms and bundles and themes that TextMate invented, and most new text editors still fall short of TextMate as it is right now.

We know Alan and Co are working on it, and we know its hard, very very hard, to make a successor to TextMate. So give them a break, they are attempting something most of us cannot even conceive of. I paid EUR40 for a text editor that I have used for over 5 years straight. What a bargain! I owe them for a great product and all the free updates, they owe me nothing. And I do not hold them to their promise of a free 2.0 upgrade, it was a nice gesture, but in retrospect a rather silly one.

So keep on using TextMate as I do, it’s still the best programmer’s editor in existence and there’s no sign of it getting replaced on my dock by anything other than TextMate 2 whenever it comes out. And give its developers a break, let them do what they have to do, give them the time to do it right, without extraneous pressure, it will help them get it done all that much easier, and we’ll all get a better product.

UPDATE

Textmate 2 is coming, Alan and co have announced a public alpha by the end of the year here.

The Hiltmon has left the building.

30% Off

I decided to run an experiment today. I was wondering why my Windows VM's seem to kill my CPU and disks. Why they were so slow when I demo on them. So I launched 2 different virtual machines running Windows XP, ran all software updates, virus scanned them, and clean rebooted them, then launched Task Manager and nothing else. And I left them alone. For a long time. No mouse movement, no app switching, no apps running, they just sat there.

I expected them to eventually do nothing, no CPU usage, no disk activity. My Mac does nothing when its idle. My Linux VMs do nothing when idle. Surely...

I was wrong.

Looks like Windows takes 30% off the top. And maybe a little more...

VMWare reported both CPU's running at 100% for the first few hours. I know, hours! Then they dropped to 40% each. I give that VMWare does create a CPU hit (lets generously give up 10%). But 90% for the first few hours? And then 30% CPU Usage on idle VM's? Idle!

VMWare also reported that the virtual hard drives were reporting full time reads. But what is being read? Why is it being read?

So I started to watch Task Manager, to see what was consuming the CPU's. Both Task Manager's reported an average of 60% CPU Usage, 60% of the time the CPU is doing something on Windows.

Half of that 60% was being taken up by MsMpEng.exe (Windows Defender), thats 30% of all available CPU cycles, and it turns out, it is also the culprit that is causing those excessive reads. But Windows Defender has already run its scans. And search indexing is complete (for those who think it's the search indexer causing this). Since no user applications are running, and therefore no new files being created, opened, written, etc, there is no need for the indexer to update nor for Windows Defender to be doing anything.

The other half of the 60% seems to be evenly split between System.exe, and a few others like TaskManager itself, explorer, VMToolsd.exe and a few bits and pieces that pop up and then disappear for a while. I don't begrudge System.exe, it maintains the operating system and provides services. I do wonder why TaskManager uses so much CPU when its effectively UNIX's top. And VMToolsd.exe needs to run to make the VM remain smooth. But they really are not that big.

After several hours of still leaving it alone, the second 30% (System, VMTools, ...) drops to almost nothing, just spikes occasionally. But MsMpEng.exe still uses between 25% and 30% CPU, and the hard disk reads are still full time.

30% off CPU. And who knows how much time and productivity off because of this. I have to interrupt Windows Defender to so something. I have to interrupt Windows to get my own work done.

In comparison, my Mac chews up less that 5% of CPU when idle, and disk reads and writes remain quiet. My Linux VM's are the same, after their launch flurries, they too idle at 5% CPU and no disk activity.

So why, Windows, do your users get 30% off? 30% off performance, 30% off productivity, 30% off the computer we paid for.

Update: Several hours later, I come back, having left both VMs alone, and the laptop is overheating, the fans are running hard and CPUs are again at 100%, disk reads at 100%. As soon as I woke the VM's, it dropped to 30% usage. Something is going on, but I have no idea what!

What your programming language says about you

There was a big internet brouhaha over .net programmers this week, kicked off by this article at expensify. Since he insults .net programmers, I thought I'd make sure the rest of us were not left out.

C - C programmers are nerds, the real deal nerds. Pasty faced, pimply, girlfriend-less, pocket-protector wearing, open-source, stay at home and code nerds. They also create the operating system and compilers all other programmers rely on, and look down upon the rest of us with appropriate scorn.

C# - C# programmers are corporate drones. They are part of a vicious circle - corporates hire C# programmers because they can, and people become C# programmers to get hired by corporates. C# programmers are bored, have boring jobs, drive boring cars, churn out boring programs and pay boring rent.

Java - Java programmers are the couch potatoes of programming. They remain behind closed doors, lounging on fancy desk sofas, eating potato chips and creating bloated, monster programs that are larger than Zeppelins which have eaten too many potato chips. Big companies need them for their heavy lifting, so they provide them the big iron they desire and more potato chips.

C++ - C++ programmers are mad professors. They wear those ugly jackets with the leather elbow patches, speak in multidimensional math, and are the only ones who can look down on C programmers. They create the simulations that run on thousands of cores to calculate weather patterns and determine the weather outside, while the rest of us just look out the window.

Objective-C - Objective-C programmers are nerds with style. They dress well, focus on design, user experience and ease of use, but deep down, they are still C programmers. Anal, pixel aligning, minimalist, black turtleneck wearing C nerds, with a life. Objective-C programmers use the ugliest language to make beautiful things that just work.

Ruby - Ruby programmers are Japanese zen masters of convention and consistency. They wear the same boiler-suits, eat the same food, and work to the same routine. In doing so, they create supermassive applications with 3 lines of code, then spend the next 5 years struggling to keep it running.

Python - Python programmers are our sociopath, axe murderer and artist programmers. They have serially killed off blocks, line endings and readability, relying on negative space as if it were a real thing. They take data and systems apart and glue them back together in new and interesting ways.

PHP - PHP programmers are our streetwalker programmers. They hang out in public spaces, showing off their wares and changing often. They ride the freight railways of the net, delivering blogs and content to a starving clientele.

Javascript - Javascript programmers are the new robot masters. They take a static web page mannequin and make its arms and legs move and flail about, but the face remains creepily still. We can all hope that they have not yet created Skynet.

Delphi - Turbo Pascal programmers are the hippies amongst us. They still live in the 60's, drive old VW beetles and have long hair. They think Windows 3.1 is still an adequate operating system and C is this cute newfangled programming child. Get off my lawn!

Perl - Perl programmers are either idiot savants or just plain idiots. Their code is unreadable, they speak to trees and furniture in voices, and are too lazy to explain it to the rest of us, even to C programmers. Yet somehow their gibberish programs hold it all together.

Visual Basic - VB programmers are the wannabees. They wannabee programmers, they wannabee cool, they wannabee treated with respect, they wannabee C programmers. Their only way in to the cool crowd is to learn a real programming language, but that's too hard. They make extremely complicated excel worksheets even more complicated.

hiltmon = {'Objective-C', 'C#', 'Ruby', 'Perl', 'Javascript', 'C++', 'PHP', 'C'};

If your programming language is not one of the above, you feel left out and feel you have not been properly insulted, post a comment. But ask yourself this - why are you not a C/C#/Java/Objective-C/Ruby/Python/PHP/Javascript/Delphi/Perl or VB programmer and therefore already properly insulted?

My first OS X Experience

With Apple's OS X turning 10 today, I though I'd share to my first experiences with it.

For me, it was back in 2002. I was living and working in Tokyo, Japan, creating interactive Web applications. I had an old Dell laptop that I'd used for about 5 years and it was on its last legs. Every few months or so, I'd reformat the hard disk on it, install some random distribution of linux on it, try to get some work done, get mad, and restore my old Windows 2000. I wanted off Windows, I wanted in on this cool Open Source caper, I wanted back to my happy place in UNIX, the OS I grew up with. But no matter how bad Windows 2000 was, the UNIX variants at the time were worse. I just could not be productive. And I could not work with the corporate file formats my company used.

We'd invited a quant in one day to show us some C++ code he had written under contract for us. He brought in a Titanium Powerbook to show us his work. There was much 'oohing' and 'aahing' at the laptop itself, but it was nothing that compared to what happened next. I knew all about OS 9, having used Macintoshes in the 90's, and expected to see OS 9's classic look when he booted up the screen.

Instead, after the obligatory awesome chime, up came this beautiful, shiny, bright interface. Jaws dropped, silence ensued, the dock popped up, and we marveled at the bright icons, clean lines and smooth fonts. Work stopped. A crowd formed. Questions were asked and answered. My first impression was that they had taken OS 9 and made it pretty. It was better than that.

I dispersed the crowd, and asked him to show me his work. And he did the most astounding thing. He opened a bash shell window and started typing in UNIX commands. UNIX! Yes, real UNIX. I was verklempt. This was the operating system I was hoping Linux would be. UNIX, with an easy to use UI. And it had Excel on it.  Real Excel.  I was sold.

That weekend, I went to Akihabara, spent 2 hours tracking down the top of the range Titanium Powerbook (1Ghz, 1GB RAM), and bought it. It had a weird Japanese keyboard, so they sold me a US keyboard replacement part. I went home, peeled the old keyboard off, installed the new keyboard, and first booted to OS X 10.1.

I never used the Dell after that. I still have that PowerBook, and it still works.

I have been an OS X user ever since.

Say what you mean

"Great is the power of steady misrepresentation." Charles Darwin

"More bars in more places" AT&T

What they want you to think: You get better service in more places.

What you think they mean: You get better service in more places.

Reality check: The bars on a phone are not a measure of signal strength, its a measure of signal-to-noise ratio. A crappy signal with low noise gives more bars. Oh, and AT&T forced Apple to change the bars calculation (proving its bullshit), see the WSJ on it. So more bars in more places means that the algorithm to calculate the bars gives more bars, you still get crappy service, dropped calls and slow data.

"Own it today!" Blu-Ray

What they want you to think: You own the movie and do what you like with it.

What you think they mean: You own the movie and do what you like with it.

Reality check: You license the right to watch the movie subject to all and any restrictions they have placed on it, including forcing you to watch the piracy bit, all ads and previews, and preventing you from copying it to other devices for more convenient viewing. So own it on blu-ray means you own a piece of useless plastic, the content belongs to them.

"You’re in good hands with Allstate" Allstate

What they want you to think: If you insure with us, we'll pay when things go bad.

What you think they mean: If you insure with us, we'll pay when things go bad.

Reality Check: "For example, Allstate's CCPR manual says claims adjusters should strive to settle as many cases within the company's historical base range — the 10th percentile of all payouts. In other words, Allstate encourages its adjusters to settle as many claims as possible for no higher than what the company historically paid out on the lowest 10 percent of its claims." See here. And this "In July 2008, the American Association for Justice ranked Allstate No. 1 among nation's worst insurers. This ranking was given because: “While Allstate publicly touts its ‘good hands’ approach, it has instead privately instructed its agents to employ a ‘boxing gloves’ strategy against its policyholders,” said American Association for Justice CEO Jon Haber. “Allstate ducks, bobs and weaves to avoid paying claims to increase its profits.”[12]" from Wikipedia, yes Wikipedia!

"Higher standards" Bank of America

What they want you to think: We're a great bank.

What you think they mean: They are a great bank.

Reality Check: Higher fees. Oh and we'll stop you for performing legitimate transactions, see here.

"Fosters?Australian for beer." Fosters Australian Beer

What they want you to think: Its a great beer because Aussies who know great beer drink it.

What you think they mean: Its a great beer because Aussies who know great beer drink it.

Reality Check: Fosters Australian Beer is a British beer, made in Canada for the US market. Foster's Lager is an Aussie beer, available only in - wait for it - Australia!.

"Fair and Balanced" Fox News

What they want you to think: They provide factual unbiased news reporting.

What you think they mean: They provide factual unbiased news reporting.

Reality Check: They are biased as hell, and admit to it, see Slate. See also the wikipedia page. Its a 24 hour channel, with news on only between 9AM and 4PM, then 6PM - 8PM, only 9 hours of news out of the 24 hours available.

"Eat Fresh" Subway

What they want you to think: Eat at subway to get fresh food that is healthy for you.

What you think they mean: Eat at subway to get fresh food that is healthy for you.

Reality Check: They make the sandwich on baked bread in front of you, using old ingredients, processed meats and cheeses and curdled dressings, with huge calorie counts, sodium levels and cholesterol. But the bread is reasonably fresh.

"Made from the best stuff on Earth." Snapple

What they want you to think: Its made from fresh ingredients so it must be healthy.

What you think they mean: Its made from fresh ingredients so it must be healthy.

Reality Check: So you still have teeth left after drinking all that sugar, coloring and preservatives (Ok, I'll admit they use the best sugar, coloring and preservatives.)

"We know money" AIG

Enough said.

It is time for "Are you sure?" to end

Friends, Romans, Countrymen and Developers, lend me your ears. For it is time that the "Are You Sure?" dialog meets its demise. It is time that it annoys its last user, it is time it gets ignored by its last viewer, it is time it gets deleted by its last developer.

The "Are You Sure?" dialog came into being in the early days of computing, when things were text, undo had not yet been invented and dialogs were vocal discussions between people who were too pretentious to use the word discussion. It was a time if limited CPU, memory and disk, a time that predated the trash can. It was a time when deleting a file actually deleted it, a time when changing data meant that the computer forgot what was there before, and a time when computer asked questions were uncommon and not ignored.

Along came the trash can on this thing called a Macintosh, and file deletes were no longer irreversible. Delete a file, no problem, its NOT deleted, its in the trash. Why does the system still ask if I am sure I want to delete a file?

Following came undo, around the same time as cut, copy and paste, and we all use it well. Change some data, undo, mess up a document, undo, delete some rows in a spreadsheet, undo. So why do applications still ask if I am sure? If I make a mistake, I should always be able to undo. Asking if user's are unsure, makes them unsure, makes them fear their software and their interactions with it.

There was a time when the modal popup dialog box provided valuable information. And in that time, people read these popups. But then these popups started to proliferate. "You have an error" looks just like "I saved your data OK" which looks just like "Click here to install malware" which looks just like "Are You Sure?". Average users just ignore popups, click OK and move on, so lets reduce their interruptions by getting rid of the "Are You Sure?" ones first.

There are those who argue that sometimes the "Are You Sure?" is necessary. I'd agree that fully destructive events (like emptying the trash) should require a confirmation, but I'd argue that there are few fully destructive events left in computing that cannot be undone. And there there is the case when the user hits cancel with unsaved changes, surely then the app should ask an "Are You Sure?". I'd argue that the concept of saving should also meet its demise, let the system save everything, and if you don't want your changes, make the cancel button simply undo them (and if the user realizes a mistake, a redo will fix it).

Software should allow users to do whatever they want, even destructive acts, without annoying them. Software should also forgive user actions and mistakes and allow them to undo everything.

So please, friends, help make "Are You Sure?" achieve a dignified end. And may we rarely see it again.

Are you Sure? Ye-haaa!