The comments below add significant thoughts to what I said – be sure to read them.
A lot of people are intrigued with using “cloud” applications and storage for personal health data. This week we’re seeing what I think is the final nail in the coffin of “cloud only” for anything important. You gotta have offline backups: two huge cloud vendors – Amazon and now Google – have demonstrated that even they can go down, leaving their users absolutely powerless.
Cloud computing (Wikipedia) is hugely attractive to software developers and businesses. As shown in this diagram from Wikipedia, the idea is that you do your computing using storage or tools that are on some computer somewhere out there “in the cloud.” You don’t know or care where, because somebody out there takes care of things. As your business or database grows, “they” take care of it.
And it’s real – it works.
But when “they” screw up, you could be screwed.
Last month Amazon Web Services went down for a couple of days. PC Magazine posted a good summary, and many of us learned that well known companies like Hootsuite and Foursquare don’t actually own the computers that deliver their product: they rent services from Amazon Web Services (AWS). So when AWS went down, there was nothing they could do to help their customers.
Now the same problem has happened with Google’s Blogger.com (“blogspot”) blogs. For at least 48 hours the back end of their blogging system has been dead. For instance you can read my old blog (patientdave.blogspot.com), but when I try to log in to create a new post, here’s what I get:
ZDNet reports that all posts and comments added since the problem started have “been removed,” a euphemism for “lost forever.” ZDNet asks, what keeps this from happening to other cloud products, like Google Docs? (What if you’d stored your business documents in Google Docs? What if your last two days of Gmail were lost?)
What’s the relevance of this to patient engagement? Well, a lot of people talk about cloud-based personal health records (PHR), and cloud-based medical tools. Here’s the lesson: For anything you can’t afford to be without, you gotta have non-cloud backups.
- You gotta have offline data backups.
- My personal website is built using WordPress; every day they email me a backup file of my complete database. I can’t lose more than a day’s work. It’s vital, it’s obviously not complicated software, and it’s free.
- At my old day job, I used to manage a pretty big cloud-based database for our sales and marketing needs: Salesforce.com. Every week they emailed us a link to download a zip file of our entire database, plus a daily update.
- For any “can’t afford to be down” situation, you gotta have ironclad availability.
- Here at e-patients.net, we don’t rely on the cloud. Our WordPress blog is hosted on the highly reliable servers of one of our board members.
- Salesforce.com has a huge number of redundant server farms, and they’re totally transparent about outages and even degradation. To earn the trust of the corporate world, they published trust.salesforce.com. It displays the uptime, performance degradation and outages of every system around the world. Click on any symbol and see the root cause, how it happened, and what they did about it. (And you can subscribe to any RSS feed for all updates! Hey Google and Amazon, you gonna offer that?)
People I talk to tend not to “get” this unless it’s expressed as their kids’ medical record. A couple of years ago I spotted this message on the Google Health user forums:
Help – leaving for hospital – data won’t print.
I do believe in the cloud – it makes sense for many situations. It makes innovation far faster and less risky. But for anything important – which health is – you gotta have offline backups and you gotta be sure you won’t go down.
Of course, there’s an ultimate trap for anything online: if the internet goes down, the whole cloud goes down. That happens sometimes, and it could very well happen if an enemy hacked the Web. For anything mission-critical you have to consider whether you could afford to be offline for hours or days, and think out what you’d do meanwhile. (Same for power outages, which is why hospitals etc. have backup generators.)
All this has been debated for years in the IT community, but once again the health IT world seems to be naive. As we consumer/patients (and health workers) start to acquire IT tools, we must insist that tools we rely on have sufficient reliability – even in a disaster.
Well stated.
btw, I forgot to mention – on the same day when AWS was down, my bank’s website became unavailable for about 12 hours. (The first of two login screens worked, but the second screen produced a “not available” message quite like the ones that other AWS customers displayed.)
This strongly suggests that my bank just might be using Web Services – not their own computers – for their customer’s banking data. Yikes.
And yeah, I’m looking to change banks.
Amazon is still better and more available than most private hosted infrastructure. Note that AWS isn’t going down worldwide, but rather regions. Within the Amazon cloud, you can make your infrastructure geographically disperse thus greatly reducing outage risk (at an increased cost of course). Amazon has facilities in VA, CA, Ireland and around the globe. Still having very important medical information on a thumb drive or on your phone makes sense.
From a pathologist, via Facebook:
Beverly wrote: “Pardon me, Dave, while I laugh hysterically. Non-cloud computer downtime is already a major, major problem in hospitals, in addition to incompetent installations which I am sure have killed patients. Frankly the cloud wouldn’t be much different than what we have experienced already….. (:”
Corrected per Matthew on FB:
Also via Facebook:
Matthew Holt commented on your link.
Matthew wrote: “Dont forget that BIDMC went down for 43
dayshours straight in 2003 despite the best efforts of John D. Halamka and the rest. And it’s still better than paper”Okay, you guys, so I get that the reality may not be nearly as professional, responsible & reliable as I imagined. So I’ll broaden my advice to ALL health IT:
1. You gotta have offline data backups!
2. For any “can’t afford to be down” situation, you gotta have ironclad availability!
Hello?
And even:
3. “…the health IT world seems to be naive. As we consumer/patients (and health workers) start to acquire IT tools, we must insist that tools we rely on have sufficient reliability – even in a disaster.”
As far as Blogger goes I recommend you back up all your posts to disk. You can use
http://bloggerbackup.codeplex.com/
Any AWS client could have avoided the downtime (and some I know did) by buying redundant services from more than a single location.
Cloud services properly implemented are the safest way to both retain online availability and have your data saved.
But it gets to be very expensive.
I believe you make a mistake by comparing the potential availability of health data in the cloud to a theoretical optimal availability. This will only slow down the adoption of systems that can bring a real benefit compared to *the reality* of todays systems.
The big difference with vendors like amazon is, that outages become more visible whereas many small outages in on premise software that is not properly maintained or just lost paper documents never receive the same attention.
That said – I also maintain a printed folder with the most important medical documents. But “have an offline copy” works the other way round as well – its just as important to have an online copy in case the folder gets lost, is not where you are or gets destroyed in an accident etc.
Hi, Valentin – from others’ comments, I guess I’m being corrected about today’s non-cloud systems. Gee whiz, it’s starting to sound like “reliable health IT” is as much of an oxymoron as “easy-to-use EMRs.” (I know some people won’t like hearing that, but I’m just saying what others have told me repeatedly.)
As an example, see our post last year about a system a hospital purchased, unhapppily, calling it the cream of the crap.
It kinda grosses me out that perhaps this post’s title should have been more generic. What the heck has made these systems SO costly to implement and SO hard to use?? I’ll tell you, the next time I need major medical services, I want my wizards using professional-strength HELPFUL systems!
Vendors, help! Policy people, please, mandate quality! PATIENTS NEED IT!
And in the meantime, while workers struggle with grim systems, please let the patient & family read too, so they/we can be a second set of eyes.
Excellent article Dave… you’ve put a convincing bow around something I’ve been watching and pondering for a good while. Well done.
Dave, great points but I would not include Amazon as a Healthcare mission critical cloud. Cloud is the same all technology, it is only as good as the weakest link. They are probably fine for most website but not healthcare. Healthcare sites need to be mission critical fully redundant systems managed by network engineers that know what they are doing. You have to do your due- diligence when selecting any systems.
As for backups, ya! you have to backup Google email.
Amazon provided a black eye to the cloud and it will happen again, it is HIT that must me prepared for these types of failures
Jeff
Hi Jeff – I don’t think I suggested that Amazon is a health vendor – the problem isn’t the cloud itself, the problem is the system that’s out there *in* the cloud.
Gilles said above, and others said on Facebook, there ARE cloud services that include enough redundancy to be sufficiently reliable (whatever that means). I posted this on FB, and Microsoft’s Grad Conn said, “HealthVault never goes down … and has an installed client as well (software + service).”
That’s a gutsy thing to say :) … sorta like an airline saying “We’ve never crashed.”
But I know from the newspaper industry, where I once worked (as a vendor), it IS possible to be dagnabbed near bulletproof, at some price.
Correction: I said AWS IS completely reliable if your methodology includes redundancy INTRA AWS, ie setting up redundant systems on more than 1 AWS node. Companies that designed their AWS implementation for this kind of failure remained online during the AWS US-East breakdown.
You can read more about designing for failure and achieving high availability with AWS here
Grad did set the bar for HealthVault and I am glad he did. It is possible to have a fail-safe system.
Jeff
Thanks, Dave. We are evaluating EMR’s including one that is totally cloud-based. Between your article and the comments, we need to consider the best way to approach a hybrid strategy. With EMR software, having an onsite backup might not help much if the cloud-based application seizes up. But given the known risks with local servers, especially where IT support is thin, may not be much of a trade-off.
Many local IT departments don’t have the expertise to handle complex routing, failover, hot- standby, network design… Tier-One cloud services have this expertise, but organizations must still perform their due diligence
As with any “mission critical system” one would be wise to have redundancy. Those that put full faith in any one thing will reep the consequences. Really doesn’t matter whether it is up in the cloud or on the local desktop or back office server, they are all vulnerable.
BTW, I use cloud services galore from HealthVault to Evernote to WordPress, Twitter, Me.com (Apple) and my favorite, Dropbox. Seriously couldn’t do the work I do without them. All have their “issues” but wise use of these services provides me unmatched flexibility that I could not even begin to envision a decade ago.
Long live the Cloud, even if it does bring a little rain sometimes.
John, Well put. Desktops are far from fail-safe. Mission Critical is as much about planning and execution as it is software and hardware.
Jeff
Those last two posts are spot on.
Dave, I think your real point here might better be that PHR services, if an when they’re truly available and usable by patients and physicians. should include an ability for the patient to make regular backups, and the system should encourage those backups. IT is never 100% reliable, but it can be close.
Fascinating stuff, people. I confess I’m amazed at the news that in-house systems (even at big medical centers, apparently) are often poorly managed. Thinking back to newspaper shops, it’s hard for me to imagine that large medical practices would be less fail-safe; but I hear the comments that say that.
John (at Chilmark), do papers or posts or surveys exist that document the proven update or failure rates of different systems, and total cost of ownership for a given service level?
Grad (at Microsoft), you still got your ears on? What can you say about how MS achieves the “never fail” availability you describe?
Great discussion, all. Thanks.
(To return to the participatory medicine aspect: it does seem we ought to be able to download our data (or our kids’ etc) as often as we want.)
A couple of follow-up comments from ZDNet:
Ed Bott wrote the same point I made:
Google’s Blogger outage makes the case against a cloud-only strategy:
Sam Diaz posted Google’s Blogger outage will bruise but not hurt cloud momentum. I’m commenting…
Overstated Dave, as others have pointed out. If you only put your vital data into one company’s hands, then that’s problem #1. If that company fails to have redundant data strategies, as Amazon did, then that’s problem #2. Problems can arise with any technology, offline backups or paper can go up in a fire/Earthquake/Tsunami/you name it. True disaster recovery plans for these types of things, but saying that an entire technology is invalid and “dead” for a particular use just because of particularly poor planning on some companies’ parts is overkill.
Perhaps what needs to happen is for all patients to need to think about these things, and for true ePatients to know what reliable disaster recovery means for their medical data, and follow through. Your comment on transparency of the data backup provider is a good one too, if you don’t know what the company is doing, you don’t know how worried to be. (which points out, if you don’t know what the company is doing, be VERY worried)
I continue to be amazed at how this discussion has gone way beyond what I said. :–)
Do you feel I overstated when I said…
1. “cloud only” is dead for health?
2. you gotta have offline backups?
3. for any “can’t afford to be down” situation, you gotta have ironclad availability?
(Anyone?)
Yes, but for #1 only, which unfortunately is in the headline, and coming from a respected source.
Thanks, Jon. So to be clear, your view is that “cloud only” is viable for health data. Understood; I don’t mind different views. :&ndash)
Yup, numero uno Dave as you way over-stepped with that title. Then that is always a good way to get a lot of readers and comments, even if the statement is completely irrational.
As to you previous question, on reliability of cloud services, like anything in life it all depends on much you want to spend. Top service providers typically guarantee “5 Nines” uptime – that’s 99.999%, that’s a little over 5min a yr. Average for consumer type services is 3-4 Nines or 18 hr/yr for 3 Nines and 52min/yr for 4 Nines.
btw, all –
I’ve become irritated about parts of this discussion because the irresponsible “editors” of The Health Care Blog took this post without asking, and without even notifying me, and thus without the note I added at the top (where I noted that the discussion in comments added a lot). As a result, a slew of discussion happened there that gave a distorted view of my THEN CURRENT view.
I know they do this with many of their posts. Once an author gives the okay, they go around harvesting what they want later, sometimes editing the text(!) or headlines, and not responding to authors’ repeated requests. I’ve heard this from others.
I think THCB should remove the Wall Street Journal’s testimonial quote at the top of the site that it’s a “must read” blog. Rather, they’ve declined to irresponsibility.
What does one do with those daily backups? WordPress is easy. Find another host, upload file, you’re up and running. Pretty painless.
But the Salesforce backup? What can you do with it? Open it and… do what? Sure you have the data, but without a useable interface what are you supposed to do?
I have reams of financial information downloaded from an on-line accounting system that I quit using. I suppose I could reconstruct the data into spreadsheets, if I took the hours necessary to figure it out. And this doesn’t apply only to on-line applications. Try moving “simple” text documents among Word, Pages, and Google Docs without losing formatting. Good luck.
Backups are useful if you hop back onto the same system (usually proprietary). But what if a company drops the product, or goes out of business? What if you migrate to another similar system for whatever reason? It always ends up with data loss.
There’s not an easy fix in sight. A backup is useful only if you can do something with it. Dave Winer (and others) are attempting to address this issue, but the critical mass don’t seem to be listening.
Well put, Dave. I’ve been worried about this myself for years, which is why I haven’t started using cloud-based apps much when I’ve been able to avoid it.
Hi, Wendy! Again, I do use cloud for almost everything- the wide access from many devices is useful and enabling, and I’ve lost FAR more data from un-backed-up non-cloud storage.
My underlying thought was that for any case where wwe MUST have access to the data and functionality, cloud-only is insufficient, because an outage leaves us with no way to help ourselves.
So I’ve been surprised but informed by the insider response, here and on sites that cross-posted this, which said today’s non-cloud EMR systems do go down, and do lose data, far more than I’d call tolerable. That too leaves us with no way to help ourselves.
And I’ve come to terms with the realization that in the case of a multi-day power outage or internet outage I could have no access at all- cloud or not. So if my life depended on it, I’d get a generator.
Hi Dave,
Good points all system have to be redundant, but cloud is far from dead for healthcare. It has just started. BTW, I would never recommend Amazon for any system that must be fail safe. It has one of the worst records.
We must always remember that the weakest link is where a system may fail, In most cases, good Cloud system is far better then a in-house supported system.
Hi Jeff – great to see you here!
I’ve reflected a lot on this post since it went up. A lot of people thought I said “Cloud is dead,” but the headline makes pretty clear, it was cloud only.
I still can’t imagine anything mission critical being cloud-only; any frickin’ power outage or internet outage and your mission is dead. (For that reason, is there anything cloud that’s “fail safe”?)
But then, a lot of healthcare isn’t mission critical. ICUs are, hospitals in general are, but not most of a doctor’s office, and certainly not a PHR.
Dave, I agree that the current state of the EHR market is not Cloud ready. Running a Microsoft Application over Citrix is not an Enterprise solution. There is also a lot of cost associated with “mission critical” software. Many organization do not want to spend what it cost to have a fully duplex, hot standby system. True mission critical system must include, full redundancy, hot standby with failover software, dual IP routing through separated pathways and carriers along with backup power.
Healthcare software producers must realize that if their system fails someone will die. These system need to operate under military standards, heat, stress, failover. It will be a while before that happens in healthcare and unfortunately patients will die before we change the way Healthcare systems are designed, built, tested, and regulated.
Jeff
Jeff,
> Healthcare software producers must realize
> that if their system fails someone will die.
Do they not??
(Now that you mention it, I recall hearing such horrible thoughts over the past two years – but boy, does the engaged/informed patient need to realize THAT.)
I’m wondering if SPM should create a commit to dictate minimal acceptable IT standards to HIMSS! (That’s the health IT industry society, for those who don’t know it.) “Minimum acceptable standards for a system to support my mother or baby’s treatment.” Why not?
This is super informative, I realize to have backups of all my files which I decided to save online. You really have super thoughts in this blog.
The moment I first heard of cloud computing, either for data storage or for running apps, these were exactly my concerns. Not that I haven’t lost data in crashes, but at least I know whose fault that is – and have the option to make sure it doesn’t happen catastrophically. If I can’t touch it, it doesn’t exist.
I still need to figure out ways to back some of this stuff up that only seems to live online, though – but I for darn sure have not switched to things like email only bank statements.
Wendy, my thinking on this has evolved a bit in the near-year since those events and this post.
There was a lot of flak about this post last year, and most was because some dim bulbs were not slick enough to notice the word “only” in the headline. :) So they thought I’d asserted “cloud is dead.” Jeeze.
Today:
— In almost EVERY area, I’ve become intolerant of anything that’s NOT cloud-compatible. I want to log in to EVERYTHING from any computer, or my phone, and I’m irked (and inhibited) when I can’t. I feel the same about my health info. I haven’t yet found a PHR to use (because I haven’t had time to research and I haven’t had a pressing need), but if I had a “health project” to manage for self or family, I’d 100% want it to be cloud.
(The RFP I published about my skin cancer, and the notes I’m keeping on responses, are all cloud.)
— At the same time, for anything stored anywhere, I want backups. I use Carbonite.com to back up my computer continuously, and I similarly want cloud data to be backed up off-cloud.
(I’ve tried many backup methods through the decades, and nothing has worked for me that required conscious intervention. Passive / automatic things get done; other things require behavior change, which is hard. We’re finding the same about data collection through Fitbits, blood pressure, etc: passive works, active is hard.)
— Since I need to function when I’m offline, there’s some data that I need to access off-cloud, not just to recover a backup but as live working storage. For that I use Dropbox, which stores locally and syncs to the cloud.
My gmail is a problem, though I may be missing something. It has an offline storage option, to save several days’ worth of mail offline, so I can for instance read & write on a plane. But it REALLY seems to slow things down.
And, I agree about paper backups. More backup = more secure. But paper ONLY is vulnerable too.
Most of the sites referred to in this blog are not mission critical Cloud services. There are plenty of great solution however Amazon and Google are not one that I would suggest.
Having a offsite backup system is a great idea however you can also replicate your data over multiple Clouds. You must also have redundant paths for your Internet connection, including making sure they travel over different pipes…
Je
Interesting response, Jeff. Do you have a link to an article that illustrates how people are doing things the way you suggest?
Do you think “cloud only” can be viable for mission critical work? I’m no whiz at this, but it seems natural that if the data lives at the other end of a pipe (even your CHOICE of pipes) then the data’s vulnerable. OTOH, DropBox keeps my stuff locally as well as on the cloud.
But then we run into … it’s not just the data, it’s any logic that also lives in the cloud. (Yes?)
As I said, I’m no whiz at this, but these seem like reasonable questions. I’d welcome a resolution to any of them.
Dave I do but not at the moment, I would have to look. I have been architecting Mission Critical cloud systems since college, Siemens telecom. We called them Client Server. System that provide dialtone have redundant line cards, processor card, memory, dual paths from the switch and battery with a diesel generator backup at every switching center,
Cloud Mission critical, Sure, ATM, all financial networks, Datacom, Telecom. That doesn’t mean you shouldn’t have a backup system in the event of failure. Class 5 Telecom switches don’t fail however, cars run into line polls and disrupt service.
When setting up a EHR for instance, select two different carriers for the fiber to the hosting site. SANS are reliable but not unfail-able. You can place the logic at different geo-location as a hot or warm stand-by.
The issue is not cloud technology but the cost associated with full redundant systems. True 99.9% uptime is costly, weather it is cloud or the EHR under the receptionist desk.
Jeff