Climb Mt Kilimanjaro
Run around Cuba
Wednesday, December 29, 2010
Tuesday, October 26, 2010
IB, PE and Hedge Funds
Best book I found for a broad and comprehensive introduction to Investment Banking, Private Equity and Hedge Funds. Very useful for my research work and for anyone who needs a good but no-need-to-be-deep sweep of the space.
Discovery through Data-mining
A visitor walking the halls of Microsoft Research’s campus in Redmond, Washington, today is likely to overhear discussions not only about computer science but about a surprising variety of other subjects, from which way a galaxy rotates, to a new AIDS vaccine, to strategies for managing the planet’s precious supply of fresh water.
What could these issues possibly have in common? And why would Microsoft—ostensibly a software company—be involved with them? The simple answer is data—vast amounts of data. So vast that when we run the programs that analyze some of the databases, the temperature of the building that houses 10,000 microprocessors shoots up several degrees. Today our computer scientists find themselves in partnership with leading scientists in a wide array of disciplines—astronomy, biology, chemistry, hydrology, oceanography, physics, and zoology, just to name a few—working on efforts such as drug development, alternative energy, and health care cost containment. And, yes, even commercial software projects. We believe that a new generation of powerful software tools, which support collaboration and data exploration on an unprecedented scale, are about to enable revolutionary discoveries in these fields.
For decades computer scientists have tried to teach computers to think like human experts by embedding in them complex rules of linguistics and reasoning. Up to now, most of those efforts have failed to come close to generating the creative insights and solutions that come naturally to the best scientists, physicians, engineers, and marketers. The most talented experts not only have a deep understanding of data but also are able to see the possibilities “between the columns”; they can find the nonobvious connections within or between disciplines that make all the difference.
We have reached a point, however, where even the experts are drowning in data. Digital information is streaming in from all sorts of sensors, instruments, and simulations, overwhelming our capacity to organize, analyze, and store it. Moore’s Law has for decades accurately predicted that the number of transistors that could be placed on an integrated circuit would double every two years, and until recently, this decrease in transistor size was accompanied by increased microprocessor performance. To increase performance today, we must program multiple processors on multicore chips and exploit parallelism. The multicore revolution has arrived just as we face an exponential increase in data. That increase is not a challenge we can address with patches and upgrades; we must rethink our whole approach to data-intensive science. Which is why, several years ago, our colleague and Turing Award winner, the late Jim Gray, proposed what he called “the fourth paradigm” for scientific exploration. Jim’s vision of powerful new tools to analyze, visualize, mine, and manipulate scientific data may represent the only systematic hope we have for solving some of our thorniest global challenges.
The Four Paradigms of Science
The first two paradigms for scientific exploration and discovery, experiment and theory, have a long history. The experimental method can be traced back to ancient Greece and China, when people tried to explain their observations through natural rather than supernatural causes. Modern theoretical science originated with Isaac Newton in the 17th century. After high-performance computers were developed in the latter half of the 20th century, Nobel Prize winner Ken Wilson identified computation and simulation as a third paradigm for scientific exploration. Detailed computer simulations capable of solving equations on a massive scale allowed scientists to explore fields of inquiry that were inaccessible to experiment and theory, such as climate modeling or galaxy formation.
By the Numbers
The fourth paradigm also involves powerful computers. But instead of developing programs based on known rules, scientists begin with the data. They direct programs to mine enormous databases looking for relationships and correlations, in essence using the programs to discover the rules. We consider big data part of the solution, not the problem. The fourth paradigm isn’t trying to replace scientists or the other three methodologies, but it does require a different set of skills. Without the ability to harness sophisticated computer tools that manipulate data, even the most highly trained expert would never manage to unearth the insights that are now starting to come into focus.
Saving Lives with “Machine Learning”
Let’s start with an example of the kind of thinking that drives this type of research. In the 1980s my colleague Eric Horvitz, while training at a Veterans Administration hospital as part of his medical education, observed a disturbing phenomenon. During the holiday season, the hospital experienced a surge in admissions for congestive heart failure. Each year, some patients who had otherwise successfully managed their health despite a weakened heart would reach a tipping point after a salty holiday meal. That extra salt caused their bodies to retain additional fluids, which would lead to lung congestion and labored breathing—and often to a visit to the emergency room.
Those post-turkey collapses were expensive in every sense of the word. They could be fatal for some patients—sometimes quite rapidly, sometimes by causing a downward spiral of failing physiological systems that took days to weeks. Other, luckier patients were effectively stabilized, but most still required a stay of a week or more that would typically cost the VA system $10,000 to $15,000 a patient. (Today those bills would be far higher.)
More than two decades later, Eric and his colleagues at Microsoft Research have developed analyses that can predict with impressive accuracy whether a patient with congestive heart failure who is released from the hospital will be readmitted within 30 days. This feat is not based on programming a computer to run through the queries a given diagnostician would ask or on an overall estimate of how many patients return. Rather, this insight comes from what we call “machine learning,” a process by which computer scientists direct a program to pore through a huge database—in this instance, hundreds of thousands of data points involving hundreds of evidential variables of some 300,000 patients. The machine is able to “learn” the profiles of those patients most likely to be readmitted by analyzing the differences between cases for which it knows the outcome. Using the program, doctors can then plug in a new patient’s data profile to determine the probability of his or her “bouncing back” to the hospital.
In one sense we owe this project to a human expert spotting a nonobvious connection: Eric not only earned his MD but also has a PhD in computer science, and he realized that machine-learning techniques similar to the ones he and his team had used to analyze Seattle traffic patterns could work for this important health care challenge. In 2003 they had developed methods of predicting traffic jams by analyzing massive quantities of data, which included information on the flow of traffic over highways, weather reports, accidents, local events, and other variables that had been gathered over several years. The team’s new program compared data about patients who were and were not readmitted, and unearthed relationships among subtle evidence in a patient’s clinical history, diagnostic tests, and even socioeconomic factors, such as whether the patient lived alone. This integration was not trivial: Information on a patient’s living situation, for example, may reside in a social worker’s report, not on a medical chart. It is unlikely that a single clinician involved in a patient’s care could ever process the volume of variables sufficient to make a prediction like this.
The economic impact of this prediction tool could be huge. If physicians or hospitals understand a patient’s likelihood of being readmitted, they can take the right preventive steps. As Eric explains: “For chronic conditions like congestive heart disease, we can design patient-specific discharge programs that provide an effective mix of education and monitoring, aimed at keeping the patients in stable, safe regimes. Such programs can include visits or calls from a nurse, or special scales that indicate dangerous changes in a patient’s fluid balance and communicate them to the doctor. If we can spend even $500 or $1,000 on postdischarge programs for patients who have the highest likelihood of being rehospitalized, we can minimize readmissions and actually save money while enhancing health outcomes.”
It’s no wonder that health insurers and hospital chains are lining up to talk about this. And it doesn’t take much imagination to list other types of businesses that could benefit from this kind of data intensive discovery as well.
On Wall Street, massive data-mining programs are already tracking “sympathetic movements,” or related trading patterns among different investment vehicles. Hedge funds and large money managers are placing millions of dollars in bets every day based on these data-discovered relationships.
On the operational side of business, the possibilities are endless. Companies will be able to do massive analyses of customers and business opportunities using programs that unearth patterns in price, buying habits, geographic region, household income, or myriad other data points. The large quantities of available data on advertising effectiveness, customer retention, employee retention, customer satisfaction, and supply chain management will allow firms to make meaningful predictions about the behavior of any given customer or employee and the likelihood of gaps in service or supply. And more and more, we find companies using data techniques to spot irregularities in payments and receivables. These programs can predict, for example, the revenues that should be collected for a given list of delivered services. One health care provider we have worked with in New Mexico discovered $10 million in underpayments within the first six months of using such data-mining tools.
The relevance of the old joke “only half of all advertising dollars are successful—we just don’t know which half” will be imperiled by the new analytical tools. An electronic entertainment company in the Philippines is using Microsoft data-mining technology to customize its sales pitches to individual customers, based on extensive analysis of such factors as past buying patterns, age, gender, financial profile, and location. Almost immediately after implementing this technique, the company saw its response rate for offers for ringtones and other products double.
With all those business opportunities, some ask why Microsoft Research is working on so many global health and environmental projects. After all, aren’t those projects that the Bill & Melinda Gates Foundation might fund? Yes, but the reason Microsoft Research has several dozen computer scientists working on them is that they involve some of the most enormous data stores imaginable and constitute an invaluable testing ground. We need to expand our own thinking and the capabilities of our tools by working on the biggest problems out there, which happen to be of immense importance to humanity. Tackling these problems also opens more opportunities for collaboration and experiments. When there is a compelling incentive for experts in different disciplines to work together and share data in a transparent environment, we’re likely to make the fastest progress. As Jim Gray used to say, astronomy data are valuable precisely because they have no commercial value.
Plug-and-Play Ocean Research
One such ambitious environmental project involves ocean science and is now under construction beneath the cool Pacific waters west of Washington State and British Columbia. It’s impossible to overstate the importance of the oceans, which cover 70% of the Earth’s surface and make up the planet’s largest ecosystem. The oceans drive weather systems; are the source of powerful, still largely unpredictable hazards such as tsunamis and hurricanes; store much more carbon than the atmosphere, vegetation, and soil; and are a critical food source.
And yet, in many ways we understand more about the surfaces of Mars and Venus than about the seafloors. Water is opaque to the electromagnetic radiation that allows us to explore the heavens; that’s why the mainstays of our oceanographic research have been submarines, ships, and satellites. That is about to change. On a patch of the Pacific’s floor, oceanographers involved with the U.S. National Science Foundation’s $600 million Ocean Observatories Initiative (OOI) have mapped out a network of nodes that is designed to offer what my colleague Roger Barga wryly calls “USB for the ocean.” OOI will lay 1,500 miles of cable to and around the patch, providing power, internet access, and the ability to record and time-stamp data on phenomena scientists will study with all sorts of devices, ranging from simple temperature sensors to remote-controlled robots to state-of-the-art gene sequencers.
The project aims to involve scientists from all over the world. The ability to measure and analyze natural processes—such as silt buildup or changes in the density of microscopic organisms—is unprecedented. But the amount of information OOI will generate could swamp the effort if the data aren’t cleverly organized and stored. That’s why Roger and his team are using work-flow technology to manage the data collected and are figuring out how to store data in the shared computing cloud, so they don’t overwhelm any one facility and so scientists, students, and interested citizens everywhere can access them. The team is working out the data standards that will allow analysis programs to combine findings from different experiments into one larger analysis. That’s called “interoperability,” and it’s crucial to making these scientific mashups work, because researchers will want to combine and compare data generated by predictive models in laboratories, as well as data from other sources, with data from the OOI network on the seafloor.
“This new era draws on the emergence, and convergence, of many rapidly evolving new technologies,” Roger observes. The exploration will be focused on finding correlations across ocean events that will enhance our understanding of—and perhaps our ability to predict—land, ocean, and atmospheric interactions. Scientists will be able to measure such previously inaccessible underwater phenomena as erupting volcanoes, major migration patterns of sea life, earthquakes, and giant storms. Real-time video and new data visualization tools will allow students, educators, and the public at large to watch these events unfold and, in some cases, even conduct their own experiments. “The internet will emerge as the most powerful oceanographic tool on the planet,” Roger predicts.
OOI is unleashing the creativity of oceanographers worldwide, who are developing new kinds of instruments to plug into this undersea lab. One is a washing-machine-size DNA sequencer designed to operate unmanned and underwater. It will filter in local creatures, capture and sample their DNA, and then send the results to scientists on shore. That ability alone is impressive. Layer on the ability to merge the DNA information gathered with data about pollution levels, acidity, ocean temperatures, or the presence of migratory species that may affect the food chain—all of which are collected by other researchers—and we have the birth of a new era of oceanographic science.
Is there a business dimension to all of this? Well, for starters, imagine what might happen if a chemist at an energy company who was developing spill amelioration technology could consult a database on these organisms’ DNA. He or she would be able to instantly call up genetic profiles of the microorganisms in the waters surrounding a spill and predict how they were likely to interact with the chemicals or solutions under consideration. Today’s scientists grappling with the aftereffects of the massive deepwater oil spill in the Gulf of Mexico do not have comprehensive baseline measures of the ocean’s health and are relying instead on “downstream” indicators, such as the health of fish. Other interoperability tools refined for OOI could offer more prosaic, but no less important, insights. For example, a retail marketing executive sitting at a desk might receive a daily report generated by a program that combs the data streaming in from point-of-sale terminals throughout the world in real time, flagging anomalous patterns of sales and returns, and make connections that most retailers would never think to look for.
Solutions for Disease and Droughts
One way the fourth paradigm achieves faster breakthroughs is by allowing the general population to interact with databases and contribute knowledge that will advance discoveries. In the Seattle traffic effort, for example, volunteers with GPS devices in their cars helped gather critical data about local traffic routes simply by driving them. These methods were later extended to the task of predicting flows on all streets in greater metropolitan areas and now enable traffic-sensitive routing for 72 cities in North America, available today in Bing Maps. (See the sidebar “Crowdsourcing in the Heavens” for a description of another effort that’s taking place in astronomy.) Soon all sorts of citizen-scientists in different fields will likely use devices as simple as cell phones or laptops to collect specialized information and analyze it.
Crowdsourcing in the Heavens
My research team has a project in India, for instance, that allows nonmedical personnel in remote areas to diagnose certain illnesses with the help of cell phones. Using them, people dial into an enormous database of medical information, fill in answers to a set of questions, and receive valuable diagnoses on the spot. This system could someday be used to track and study the spread of diseases, particularly infectious ones. With large numbers of people performing quick diagnostics that feed into a database, public officials and health care workers can see where outbreaks are occurring, how fast they’re moving, and what kind of symptoms are appearing. Machine learning can enter the loop in real time, constantly comparing every new case with every other case of this and other infectious outbreaks—and looking for patterns that might aid prevention efforts.
The stress this kind of ambitious project puts on every aspect of current technology—processing power; demand for parallel programmers; and data storage, curation, and publishing—is enormous. Unless curation of the data is actually built into a project’s design, for example, the scientists involved usually try to figure it out ad hoc, which tends to lead to brittle, local solutions that don’t scale up. Scientists and policy makers, however, do not have the luxury of waiting until everything is figured out before taking action on urgent problems such as climate change or water shortages or planning for hurricanes or tsunamis.
Consider the plight of California, where the population is projected to increase from about 38 million today to more than 50 million by 2040. Says Jeff Dozier, a professor in the School of Environmental Science and Management at the University of California, Santa Barbara: “The availability of water drives California’s economy. Historically, we’ve tried to manage the supply of water to meet demand. We may not be able to do that anymore. Everyone would love a reliable, uniform supply, but that’s not what nature gives us. We will need much better technology to predict the amount of water we will have in a given year.”
Predicting water stores from snowpack is a much more difficult problem than it might appear, Dozier explains. Satellites collect huge volumes of data on snowpack, but they are still insufficient because they mainly reveal the snow’s surface characteristics. To manage runoff, we need to know the “water equivalent,” or the amount of water that would result from snowmelt. We can estimate the water equivalent from the weight of the snow, but that is difficult to measure across large stretches of variable terrain. The challenge: How do scientists combine data from satellites and surface measurements with information on economics and governance to better estimate, calibrate, and manage water supplies? In California alone, there are at least 400 different agencies that manage water. Microsoft is working with scientists at the University of California, Berkeley, and the Lawrence Berkeley National Laboratory to acquire and curate historic hydrologic data so that they can be used more effectively with data from new sensor networks to create better prediction models.
In another urgent arena, Microsoft’s David Heckerman, another MD with a PhD in computer science, is using data-intensive scientific discovery in the fight against the human immunodeficiency virus. “In several years in a single patient, HIV mutates about as much as the influenza virus has mutated in its known history,” he explains. That’s why developing a vaccine to thwart it has been so difficult. Moreover, the mutations seen in one individual are quite different from those seen in another, thanks to variability in human immune systems. David and his team are analyzing data about individual viral mutations in thousands of subjects, trying to zero in on the elements of the virus that are vulnerable to attack by the immune system. By creating a vaccine that can trigger a person’s own immune system to attack those elements, they hope to stop the virus in its tracks. He and his Harvard collaborator Bruce Walker expect to begin testing the first vaccine based on this work soon.
Shifting Gears—and Standards
Endeavors like vaccine development or fields like human genomics involve a limited number of disciplines but absolutely enormous amounts of data unique to each individual. In efforts to better characterize an environmental phenomenon like ocean processes or climate change, it’s not only the volume of data about any one factor but the number of disciplines and data sources that is daunting. Comprehensive calculations of warming trends might require factoring in measurements of radiant heat reflected from polar ice sheets, wasting of floating ice shelves caused by small increases in ocean temperature, the health of mangrove forests in tropical climates, global insect-hatching trends, climate changes captured in tree rings, CO2 levels preserved in stored ice cores—and more. Creating standards for collecting, storing, and mashing together such data will become increasingly important as scientists deploy more and more sensors.
Critically, too, most of us believe scientific publishing will change dramatically in the future. We foresee the end product today—papers that discuss an experiment and its findings and just refer to data sets—morphing into a wrapper for the data themselves, which other researchers will be able to access directly over the internet, probe with their own questions, or even mash into their own data sets in creative ways that yield insights the first researcher might never have dreamed of. The goal, as Jim Gray put it so well, is “a world in which all of the science literature is online, all of the science data are online and they interoperate with each other. Lots of new tools are needed to make this happen.”
While the realization of this goal would mean positive changes for society and the planet, the fourth paradigm also will inevitably create great business opportunities. For example, David Heckerman’s genomic analysis of HIV is just one small piece of the much bigger agenda of personalized medicine. The pharmaceutical industry is betting that finding out which drugs are most effective for someone with a particular genetic profile will bring a whole new dimension to drug design. Microsoft’s Health Solutions Group is integrating medical records and images as a first step in providing a set of smart tools to help the pharmaceutical industry fulfill this vision.
All scientific disciplines, including computer science, need to collaborate to realize the power of the fourth paradigm and solve important problems for humanity. The answers are hiding amid vast mountains of numbers—and it’s within our reach to find them.
HBR.org > November 2010
What could these issues possibly have in common? And why would Microsoft—ostensibly a software company—be involved with them? The simple answer is data—vast amounts of data. So vast that when we run the programs that analyze some of the databases, the temperature of the building that houses 10,000 microprocessors shoots up several degrees. Today our computer scientists find themselves in partnership with leading scientists in a wide array of disciplines—astronomy, biology, chemistry, hydrology, oceanography, physics, and zoology, just to name a few—working on efforts such as drug development, alternative energy, and health care cost containment. And, yes, even commercial software projects. We believe that a new generation of powerful software tools, which support collaboration and data exploration on an unprecedented scale, are about to enable revolutionary discoveries in these fields.
For decades computer scientists have tried to teach computers to think like human experts by embedding in them complex rules of linguistics and reasoning. Up to now, most of those efforts have failed to come close to generating the creative insights and solutions that come naturally to the best scientists, physicians, engineers, and marketers. The most talented experts not only have a deep understanding of data but also are able to see the possibilities “between the columns”; they can find the nonobvious connections within or between disciplines that make all the difference.
We have reached a point, however, where even the experts are drowning in data. Digital information is streaming in from all sorts of sensors, instruments, and simulations, overwhelming our capacity to organize, analyze, and store it. Moore’s Law has for decades accurately predicted that the number of transistors that could be placed on an integrated circuit would double every two years, and until recently, this decrease in transistor size was accompanied by increased microprocessor performance. To increase performance today, we must program multiple processors on multicore chips and exploit parallelism. The multicore revolution has arrived just as we face an exponential increase in data. That increase is not a challenge we can address with patches and upgrades; we must rethink our whole approach to data-intensive science. Which is why, several years ago, our colleague and Turing Award winner, the late Jim Gray, proposed what he called “the fourth paradigm” for scientific exploration. Jim’s vision of powerful new tools to analyze, visualize, mine, and manipulate scientific data may represent the only systematic hope we have for solving some of our thorniest global challenges.
The Four Paradigms of Science
The first two paradigms for scientific exploration and discovery, experiment and theory, have a long history. The experimental method can be traced back to ancient Greece and China, when people tried to explain their observations through natural rather than supernatural causes. Modern theoretical science originated with Isaac Newton in the 17th century. After high-performance computers were developed in the latter half of the 20th century, Nobel Prize winner Ken Wilson identified computation and simulation as a third paradigm for scientific exploration. Detailed computer simulations capable of solving equations on a massive scale allowed scientists to explore fields of inquiry that were inaccessible to experiment and theory, such as climate modeling or galaxy formation.
By the Numbers
The fourth paradigm also involves powerful computers. But instead of developing programs based on known rules, scientists begin with the data. They direct programs to mine enormous databases looking for relationships and correlations, in essence using the programs to discover the rules. We consider big data part of the solution, not the problem. The fourth paradigm isn’t trying to replace scientists or the other three methodologies, but it does require a different set of skills. Without the ability to harness sophisticated computer tools that manipulate data, even the most highly trained expert would never manage to unearth the insights that are now starting to come into focus.
Saving Lives with “Machine Learning”
Let’s start with an example of the kind of thinking that drives this type of research. In the 1980s my colleague Eric Horvitz, while training at a Veterans Administration hospital as part of his medical education, observed a disturbing phenomenon. During the holiday season, the hospital experienced a surge in admissions for congestive heart failure. Each year, some patients who had otherwise successfully managed their health despite a weakened heart would reach a tipping point after a salty holiday meal. That extra salt caused their bodies to retain additional fluids, which would lead to lung congestion and labored breathing—and often to a visit to the emergency room.
Those post-turkey collapses were expensive in every sense of the word. They could be fatal for some patients—sometimes quite rapidly, sometimes by causing a downward spiral of failing physiological systems that took days to weeks. Other, luckier patients were effectively stabilized, but most still required a stay of a week or more that would typically cost the VA system $10,000 to $15,000 a patient. (Today those bills would be far higher.)
More than two decades later, Eric and his colleagues at Microsoft Research have developed analyses that can predict with impressive accuracy whether a patient with congestive heart failure who is released from the hospital will be readmitted within 30 days. This feat is not based on programming a computer to run through the queries a given diagnostician would ask or on an overall estimate of how many patients return. Rather, this insight comes from what we call “machine learning,” a process by which computer scientists direct a program to pore through a huge database—in this instance, hundreds of thousands of data points involving hundreds of evidential variables of some 300,000 patients. The machine is able to “learn” the profiles of those patients most likely to be readmitted by analyzing the differences between cases for which it knows the outcome. Using the program, doctors can then plug in a new patient’s data profile to determine the probability of his or her “bouncing back” to the hospital.
In one sense we owe this project to a human expert spotting a nonobvious connection: Eric not only earned his MD but also has a PhD in computer science, and he realized that machine-learning techniques similar to the ones he and his team had used to analyze Seattle traffic patterns could work for this important health care challenge. In 2003 they had developed methods of predicting traffic jams by analyzing massive quantities of data, which included information on the flow of traffic over highways, weather reports, accidents, local events, and other variables that had been gathered over several years. The team’s new program compared data about patients who were and were not readmitted, and unearthed relationships among subtle evidence in a patient’s clinical history, diagnostic tests, and even socioeconomic factors, such as whether the patient lived alone. This integration was not trivial: Information on a patient’s living situation, for example, may reside in a social worker’s report, not on a medical chart. It is unlikely that a single clinician involved in a patient’s care could ever process the volume of variables sufficient to make a prediction like this.
The economic impact of this prediction tool could be huge. If physicians or hospitals understand a patient’s likelihood of being readmitted, they can take the right preventive steps. As Eric explains: “For chronic conditions like congestive heart disease, we can design patient-specific discharge programs that provide an effective mix of education and monitoring, aimed at keeping the patients in stable, safe regimes. Such programs can include visits or calls from a nurse, or special scales that indicate dangerous changes in a patient’s fluid balance and communicate them to the doctor. If we can spend even $500 or $1,000 on postdischarge programs for patients who have the highest likelihood of being rehospitalized, we can minimize readmissions and actually save money while enhancing health outcomes.”
It’s no wonder that health insurers and hospital chains are lining up to talk about this. And it doesn’t take much imagination to list other types of businesses that could benefit from this kind of data intensive discovery as well.
On Wall Street, massive data-mining programs are already tracking “sympathetic movements,” or related trading patterns among different investment vehicles. Hedge funds and large money managers are placing millions of dollars in bets every day based on these data-discovered relationships.
On the operational side of business, the possibilities are endless. Companies will be able to do massive analyses of customers and business opportunities using programs that unearth patterns in price, buying habits, geographic region, household income, or myriad other data points. The large quantities of available data on advertising effectiveness, customer retention, employee retention, customer satisfaction, and supply chain management will allow firms to make meaningful predictions about the behavior of any given customer or employee and the likelihood of gaps in service or supply. And more and more, we find companies using data techniques to spot irregularities in payments and receivables. These programs can predict, for example, the revenues that should be collected for a given list of delivered services. One health care provider we have worked with in New Mexico discovered $10 million in underpayments within the first six months of using such data-mining tools.
The relevance of the old joke “only half of all advertising dollars are successful—we just don’t know which half” will be imperiled by the new analytical tools. An electronic entertainment company in the Philippines is using Microsoft data-mining technology to customize its sales pitches to individual customers, based on extensive analysis of such factors as past buying patterns, age, gender, financial profile, and location. Almost immediately after implementing this technique, the company saw its response rate for offers for ringtones and other products double.
With all those business opportunities, some ask why Microsoft Research is working on so many global health and environmental projects. After all, aren’t those projects that the Bill & Melinda Gates Foundation might fund? Yes, but the reason Microsoft Research has several dozen computer scientists working on them is that they involve some of the most enormous data stores imaginable and constitute an invaluable testing ground. We need to expand our own thinking and the capabilities of our tools by working on the biggest problems out there, which happen to be of immense importance to humanity. Tackling these problems also opens more opportunities for collaboration and experiments. When there is a compelling incentive for experts in different disciplines to work together and share data in a transparent environment, we’re likely to make the fastest progress. As Jim Gray used to say, astronomy data are valuable precisely because they have no commercial value.
Plug-and-Play Ocean Research
One such ambitious environmental project involves ocean science and is now under construction beneath the cool Pacific waters west of Washington State and British Columbia. It’s impossible to overstate the importance of the oceans, which cover 70% of the Earth’s surface and make up the planet’s largest ecosystem. The oceans drive weather systems; are the source of powerful, still largely unpredictable hazards such as tsunamis and hurricanes; store much more carbon than the atmosphere, vegetation, and soil; and are a critical food source.
And yet, in many ways we understand more about the surfaces of Mars and Venus than about the seafloors. Water is opaque to the electromagnetic radiation that allows us to explore the heavens; that’s why the mainstays of our oceanographic research have been submarines, ships, and satellites. That is about to change. On a patch of the Pacific’s floor, oceanographers involved with the U.S. National Science Foundation’s $600 million Ocean Observatories Initiative (OOI) have mapped out a network of nodes that is designed to offer what my colleague Roger Barga wryly calls “USB for the ocean.” OOI will lay 1,500 miles of cable to and around the patch, providing power, internet access, and the ability to record and time-stamp data on phenomena scientists will study with all sorts of devices, ranging from simple temperature sensors to remote-controlled robots to state-of-the-art gene sequencers.
The project aims to involve scientists from all over the world. The ability to measure and analyze natural processes—such as silt buildup or changes in the density of microscopic organisms—is unprecedented. But the amount of information OOI will generate could swamp the effort if the data aren’t cleverly organized and stored. That’s why Roger and his team are using work-flow technology to manage the data collected and are figuring out how to store data in the shared computing cloud, so they don’t overwhelm any one facility and so scientists, students, and interested citizens everywhere can access them. The team is working out the data standards that will allow analysis programs to combine findings from different experiments into one larger analysis. That’s called “interoperability,” and it’s crucial to making these scientific mashups work, because researchers will want to combine and compare data generated by predictive models in laboratories, as well as data from other sources, with data from the OOI network on the seafloor.
“This new era draws on the emergence, and convergence, of many rapidly evolving new technologies,” Roger observes. The exploration will be focused on finding correlations across ocean events that will enhance our understanding of—and perhaps our ability to predict—land, ocean, and atmospheric interactions. Scientists will be able to measure such previously inaccessible underwater phenomena as erupting volcanoes, major migration patterns of sea life, earthquakes, and giant storms. Real-time video and new data visualization tools will allow students, educators, and the public at large to watch these events unfold and, in some cases, even conduct their own experiments. “The internet will emerge as the most powerful oceanographic tool on the planet,” Roger predicts.
OOI is unleashing the creativity of oceanographers worldwide, who are developing new kinds of instruments to plug into this undersea lab. One is a washing-machine-size DNA sequencer designed to operate unmanned and underwater. It will filter in local creatures, capture and sample their DNA, and then send the results to scientists on shore. That ability alone is impressive. Layer on the ability to merge the DNA information gathered with data about pollution levels, acidity, ocean temperatures, or the presence of migratory species that may affect the food chain—all of which are collected by other researchers—and we have the birth of a new era of oceanographic science.
Is there a business dimension to all of this? Well, for starters, imagine what might happen if a chemist at an energy company who was developing spill amelioration technology could consult a database on these organisms’ DNA. He or she would be able to instantly call up genetic profiles of the microorganisms in the waters surrounding a spill and predict how they were likely to interact with the chemicals or solutions under consideration. Today’s scientists grappling with the aftereffects of the massive deepwater oil spill in the Gulf of Mexico do not have comprehensive baseline measures of the ocean’s health and are relying instead on “downstream” indicators, such as the health of fish. Other interoperability tools refined for OOI could offer more prosaic, but no less important, insights. For example, a retail marketing executive sitting at a desk might receive a daily report generated by a program that combs the data streaming in from point-of-sale terminals throughout the world in real time, flagging anomalous patterns of sales and returns, and make connections that most retailers would never think to look for.
Solutions for Disease and Droughts
One way the fourth paradigm achieves faster breakthroughs is by allowing the general population to interact with databases and contribute knowledge that will advance discoveries. In the Seattle traffic effort, for example, volunteers with GPS devices in their cars helped gather critical data about local traffic routes simply by driving them. These methods were later extended to the task of predicting flows on all streets in greater metropolitan areas and now enable traffic-sensitive routing for 72 cities in North America, available today in Bing Maps. (See the sidebar “Crowdsourcing in the Heavens” for a description of another effort that’s taking place in astronomy.) Soon all sorts of citizen-scientists in different fields will likely use devices as simple as cell phones or laptops to collect specialized information and analyze it.
Crowdsourcing in the Heavens
My research team has a project in India, for instance, that allows nonmedical personnel in remote areas to diagnose certain illnesses with the help of cell phones. Using them, people dial into an enormous database of medical information, fill in answers to a set of questions, and receive valuable diagnoses on the spot. This system could someday be used to track and study the spread of diseases, particularly infectious ones. With large numbers of people performing quick diagnostics that feed into a database, public officials and health care workers can see where outbreaks are occurring, how fast they’re moving, and what kind of symptoms are appearing. Machine learning can enter the loop in real time, constantly comparing every new case with every other case of this and other infectious outbreaks—and looking for patterns that might aid prevention efforts.
The stress this kind of ambitious project puts on every aspect of current technology—processing power; demand for parallel programmers; and data storage, curation, and publishing—is enormous. Unless curation of the data is actually built into a project’s design, for example, the scientists involved usually try to figure it out ad hoc, which tends to lead to brittle, local solutions that don’t scale up. Scientists and policy makers, however, do not have the luxury of waiting until everything is figured out before taking action on urgent problems such as climate change or water shortages or planning for hurricanes or tsunamis.
Consider the plight of California, where the population is projected to increase from about 38 million today to more than 50 million by 2040. Says Jeff Dozier, a professor in the School of Environmental Science and Management at the University of California, Santa Barbara: “The availability of water drives California’s economy. Historically, we’ve tried to manage the supply of water to meet demand. We may not be able to do that anymore. Everyone would love a reliable, uniform supply, but that’s not what nature gives us. We will need much better technology to predict the amount of water we will have in a given year.”
Predicting water stores from snowpack is a much more difficult problem than it might appear, Dozier explains. Satellites collect huge volumes of data on snowpack, but they are still insufficient because they mainly reveal the snow’s surface characteristics. To manage runoff, we need to know the “water equivalent,” or the amount of water that would result from snowmelt. We can estimate the water equivalent from the weight of the snow, but that is difficult to measure across large stretches of variable terrain. The challenge: How do scientists combine data from satellites and surface measurements with information on economics and governance to better estimate, calibrate, and manage water supplies? In California alone, there are at least 400 different agencies that manage water. Microsoft is working with scientists at the University of California, Berkeley, and the Lawrence Berkeley National Laboratory to acquire and curate historic hydrologic data so that they can be used more effectively with data from new sensor networks to create better prediction models.
In another urgent arena, Microsoft’s David Heckerman, another MD with a PhD in computer science, is using data-intensive scientific discovery in the fight against the human immunodeficiency virus. “In several years in a single patient, HIV mutates about as much as the influenza virus has mutated in its known history,” he explains. That’s why developing a vaccine to thwart it has been so difficult. Moreover, the mutations seen in one individual are quite different from those seen in another, thanks to variability in human immune systems. David and his team are analyzing data about individual viral mutations in thousands of subjects, trying to zero in on the elements of the virus that are vulnerable to attack by the immune system. By creating a vaccine that can trigger a person’s own immune system to attack those elements, they hope to stop the virus in its tracks. He and his Harvard collaborator Bruce Walker expect to begin testing the first vaccine based on this work soon.
Shifting Gears—and Standards
Endeavors like vaccine development or fields like human genomics involve a limited number of disciplines but absolutely enormous amounts of data unique to each individual. In efforts to better characterize an environmental phenomenon like ocean processes or climate change, it’s not only the volume of data about any one factor but the number of disciplines and data sources that is daunting. Comprehensive calculations of warming trends might require factoring in measurements of radiant heat reflected from polar ice sheets, wasting of floating ice shelves caused by small increases in ocean temperature, the health of mangrove forests in tropical climates, global insect-hatching trends, climate changes captured in tree rings, CO2 levels preserved in stored ice cores—and more. Creating standards for collecting, storing, and mashing together such data will become increasingly important as scientists deploy more and more sensors.
Critically, too, most of us believe scientific publishing will change dramatically in the future. We foresee the end product today—papers that discuss an experiment and its findings and just refer to data sets—morphing into a wrapper for the data themselves, which other researchers will be able to access directly over the internet, probe with their own questions, or even mash into their own data sets in creative ways that yield insights the first researcher might never have dreamed of. The goal, as Jim Gray put it so well, is “a world in which all of the science literature is online, all of the science data are online and they interoperate with each other. Lots of new tools are needed to make this happen.”
While the realization of this goal would mean positive changes for society and the planet, the fourth paradigm also will inevitably create great business opportunities. For example, David Heckerman’s genomic analysis of HIV is just one small piece of the much bigger agenda of personalized medicine. The pharmaceutical industry is betting that finding out which drugs are most effective for someone with a particular genetic profile will bring a whole new dimension to drug design. Microsoft’s Health Solutions Group is integrating medical records and images as a first step in providing a set of smart tools to help the pharmaceutical industry fulfill this vision.
All scientific disciplines, including computer science, need to collaborate to realize the power of the fourth paradigm and solve important problems for humanity. The answers are hiding amid vast mountains of numbers—and it’s within our reach to find them.
HBR.org > November 2010
Sunday, August 22, 2010
Corporate Sustainability Model
How can costs of meeting social goals and the risks of not meeting them be factored into capital investments and allocation decisions?
- How can performance evaluation and reward systems be reconstructed to reflect the broadened set of goals?
- How can organizational information systems be constructed to help managers achieve the high performance those personnel systems seek to reward?
- How can we construct organizational processes that will define, collect, track and analyse relevant data to provide managerial incentives, drive organizational learning and guide strategic action across the full integrated panoply of firm objectives?
- How can the standard corporate processes associated with important business decisions – budgeting, personnel assignments and career tracking etc – be modified to include the full array of consequences from financial to social, that the firm now seeks to manage?
- How should firms organize the development of and carry out the internal and external communications of its goals and accomplishments across the full domain of consequences for which it is now taking responsibility?
Sunday, August 15, 2010
Norwegian Crime Fiction
Not quite Mankell or Larsson quality, in terms of complex and in-depth characterisation and layered plotting, but still worth a read.
Sunday, August 8, 2010
Recent Reads, Recommended
Non-Fiction
History of money and the evolution of global finance.
History of the Rothschilds and the development of modern finance in Europe in 2 parts.
Fundamental political economic concerns that will continue to threaten the world economy if left unresolved.
Fiction - More Nordic crime stories
History of money and the evolution of global finance.
History of the Rothschilds and the development of modern finance in Europe in 2 parts.
Fundamental political economic concerns that will continue to threaten the world economy if left unresolved.Fiction - More Nordic crime stories
Monday, June 28, 2010
Managing Oneself
Managing Oneself
by Peter R Drucker
HARVARD BUSINESS REVIEW, JANUARY 2005, pág 100-109
by Peter R Drucker
HARVARD BUSINESS REVIEW, JANUARY 2005, pág 100-109
History's great achievers - a Napoleon, a da Vinci, a Mozart - have always managed themselves. That, in large measure, is what makes them great achievers. But they are rare exceptions, so un-usual both in their talents and their accomplishments as to be considered outside the boundaries of ordinary human existence. Now, most of us, even those of us with modest endowments, will have to learn to manage ourselves. We will have to learn to develop ourselves. We will have to place our-selves where we can make the greatest contribution. And we will have to stay mentally alert and engaged during a 50-year working life, which means knowing how and when to change the work we do.
What Are My Strengths?
Most people think they know what they are good at. They are usually wrong. More often, people know what they are not good at - and even then more people are wrong than right. And yet, a per-son can perform only from strength. One cannot build performance on weaknesses, let alone on something one cannot do at all.
Throughout history, people had little need to know their strengths. A person was born into a position and a line of work: The peasant's son would also be a peasant; the artisan's daughter, an artisan's wife; and so on. But now people have choices. We need to know our strengths in order to know where we belong.
The only way to discover your strengths is through feedback analysis.
Whenever you make a key decision or take a key action, write down what you expect will happen. Nine or 12 months later, compare the actual results with your expectations. I have been practicing this method for 15 to 20 years now, and every time I do it, I am surprised.
The feedback analysis showed me, for instance-and to my great surprise-that I have an intuitive understanding of technical people, whether they are engineers or accountants or market research-ers.
It also showed me that I don't really resonate with generalists.
Feedback analysis is by no means new. It was invented sometime in the fourteenth century by an otherwise totally obscure German theologian and picked up quite independently, some 150 years later, by John Calvin and Ignatius of Loyola, each of whom incorporated it into the practice of his followers.
In fact, the steadfast focus on performance and results that this habit produces explains why the institutions these two men founded, the Calvinist church and the Jesuit order, came to dominate Europe within 30 years.
Practiced consistently, this simple method will show you within a fairly short period of time, maybe two or three years, where your strengths lie - and this is the most important thing to know. The method will show you what you are doing or failing to do that deprives you of the full benefits of your strengths. It will show you where you are not particularly competent. And finally, it will show you where you have no strengths and cannot perform.
Several implications for action follow from feedback analysis. First and foremost, concentrate on your strengths. Put yourself where your strengths can produce results.
Second, work on improving your strengths. Analysis will rapidly show where you need to improve skills or acquire new ones. It will also show the gaps in your knowledge -and those can usually be filled. Mathematicians are born, but everyone can learn trigonometry.
Third, discover where your intellectual arrogance is causing disabling ignorance and overcome it. Far too many people - especially people with great expertise in one area-are contemptuous of knowledge in other areas or believe that being bright is a substitute for knowledge. First-rate engi-neers, for instance, tend to take pride in not knowing anything about people. Human beings, they believe, are much too disorderly for the good engineering mind.
Human resources professionals, by contrast, often pride themselves on their ignorance of elemen-tary accounting or of quantitative methods altogether. But taking pride in such ignorance is self de-feating.
Go to work on acquiring the skills and knowledge you need to fully realize your strengths.
It is equally essential to remedy your bad habits-the things you do or fail to do that inhibit your effec-tiveness and performance. Such habits will quickly show up in the feedback. For example, a plan-ner may find that his beautiful plans fail because he does not follow through on them. Like so many brilliant people, he believes that ideas move mountains. But bulldozers move mountains; ideas show where the bulldozers should go to work. This planner will have to learn that the work does not stop when the plan is completed. He must find people to carry out the plan and explain it to them. He must adapt and change it as he puts it into action. And finally, he must decide when to stop pushing the plan.
At the same time, feedback will also reveal when the problem is a lack of manners. Manners are the lubricating oil of an organization. It is a law of nature that two moving bodies in contact with each other create friction. This is as true for human beings as it is for inanimate objects. Manners- simple things like saying "please" and "thank you" and knowing a person's name or asking after her family-enable two people to work together whether they like each other or not. Bright people, especially bright young people, often do not understand this. If analysis shows that someone's brilliant work fails again and again as soon as cooperation from others is required, it probably indicates a lack of courtesy - that is, a lack of manners.
Comparing your expectations with your results also indicates what not to do. We all have a vast number of areas in which we have no talent or skill and little chance of becoming even mediocre.
In those areas a person - and especially a knowledge worker-should not take on work, jobs, and assignments.
One should waste as little effort as possible on improving areas of low competence.
It takes far more energy and work to improve from incompetence to mediocrity than it takes to improve from first-rate performance to excellence.
And yet most people-especially most teachers and most organizations concentrate on making in-competent performers into mediocre ones. Energy, resources, and time should go instead to mak-ing a competent person into a star performer.
How Do I Perform?
Amazingly few people know how they get things done. Indeed, most of us do not even know that different people work and perform differently. Too many people work in ways that are not their ways, and that almost guarantees non-performance.
For knowledge workers, How do I perform? may be an even more important question than What are my strengths? Like one's strengths, how one performs is unique. It is a matter of personality.
Whether personality be a matter of nature or nurture, it surely is formed long before a person goes to work. And how a person performs is a given, just as what a person is good at or not good at is a given. A person's way of performing can be slightly modified, but it is unlikely to be completely changed-and certainly not easily. Just as people achieve results by doing what they are good at, they also achieve results by working in ways that they best perform. A few common personality traits usually determine how a person performs.
Am I a reader or a listener? The first thing to know is whether you are a reader or a listener. Far too few people even know that there are readers and listeners and that people are rarely both. Even fewer know which of the two they themselves are. But some examples will show how damaging such ignorance can be.
When Dwight Eisenhower was Supreme Commander of the Allied forces in Europe, he was the darling of the press. His press conferences were famous for their style - General Eisenhower showed total command of whatever question he was asked, and he was able to describe a situation and explain a policy in two or three beautifully polished and elegant sentences. Ten years later, the same journalists who had been his admirers held President Eisenhower in open contempt. He never addressed the questions, they complained, but rambled on endlessly about something else.
And they constantly ridiculed him for butchering the King's English in incoherent and ungrammatical answers.
Eisenhower apparently did not know that he was a reader, not a listener.
When he was Supreme Commander in Europe, his aides made sure that every question from the press was presented in writing at least half an hour before a conference was to begin. And then Eisenhower was in total command. When he became president, he succeeded two listeners, Frank-lin D. Roosevelt and Harry Truman. Both men knew themselves to be listeners and both enjoyed free-for-all press conferences. Eisenhower may have felt that he had to do what his two predeces-sors had done. As a result, he never even heard the questions journalists asked. And Eisenhower is not even an extreme case of a non-listener.
A few years later, Lyndon Johnson destroyed his presidency, in large measure, by not knowing that he was a listener.
His predecessor, John Kennedy, was a reader who had assembled a brilliant group of writers as his assistants, making sure that they wrote to him before discussing their memos in person. Johnson kept these people on his staff-and they kept on writing. He never, apparently, understood one word of what they wrote. Yet as a senator, Johnson had been superb; for parliamentarians have to be, above all, listeners.
Few listeners can be made, or can make themselves, into competent readers - and vice versa. The listener who tries to be a reader will, therefore, suffer the fate of Lyndon Johnson, whereas the reader who tries to be a listener will suffer the fate of Dwight Eisenhower. They will not perform or achieve.
How do I learn? The second thing to know about how one performs is to know how one learns. Many first-class writers - Winston Churchill is but one example -do poorly in school. They tend to remember their schooling as pure torture. Yet few of their classmates remember it the same way. They may not have enjoyed the school very much, but the worst they suffered was boredom.
The explanation is that writers do not, as a rule, learn by listening and reading. They learn by writ-ing. Because schools do not allow them to learn this way, they get poor grades.
Schools everywhere are organized on the assumption that there is only one right way to learn and that it is the same way for everybody. But to be forced to learn the way a school teaches is sheer hell for students who learn differently.
Indeed, there are probably half a dozen different ways to learn.
There are people, like Churchill, who learn by writing. Some people learn by taking copious notes. Beethoven, for example, left behind an enormous number of sketchbooks, yet he said he never actually looked at them when he composed.
Asked why he kept them, he is reported to have replied, "If I don't write it down immediately, I forget it right away. If I put it into a sketchbook, I never forget it and I never have to look it up again." Some people learn by doing. Others learn by hearing themselves talk.
A chief executive I know who converted a small and mediocre family business into the leading com-pany in its industry was one of those people who learn by talking. He was in the habit of calling his entire senior staff into his office once a week and then talking at them for two or three hours. He would raise policy issues and argue three different positions on each one. He rarely asked his as-sociates for comments or questions; he simply needed an audience to hear himself talk. That's how he learned. And although he is a fairly extreme case, learning through talking is by no means an unusual method. Successful trial lawyers learn the same way, as do many medical diagnosticians (and so do I).
Of all the important pieces of self knowledge, understanding how you learn is the easiest to acquire. When I ask people, "How do you learn?" most of them know the answer. But when I ask, "Do you act on this knowledge?" few answer yes. And yet, acting on this knowledge is the key to perform-ance; or rather, not acting on this knowledge condemns one to non performance.
Am I a reader or a listener? and How do I learn? are the first questions to ask.
But they are by no means the only ones.
To manage yourself effectively, you also have to ask. Do I work well with people or am I a loner? And if you do work well with people, you then must ask. In what relationship? Some people work best as subordinates.
General George Patton, the great American military hero of World War II, is a prime example. Pat-ton was America's top troop commander. Yet when he was proposed for an independent command. General George Marshall, the U.S. chief of staff-and probably the most successful picker of men in U.S. history - said, "Patton is the best subordinate the American army has ever produced, but he would be the worst commander." Some people work best as team members.
Others work best alone. Some are exceptionally talented as coaches and mentors; others are sim-ply incompetent as mentors.
Another crucial question is: Do I produce results as a decision maker or as an adviser? A great many people perform best as advisers but cannot take the burden and pressure of making the deci-sion. A good many other people, by contrast, need an adviser to force themselves to think; then they can make decisions and act on them with speed, self confidence, and courage.
This is a reason, by the way, that the number two person in an organization often fails when pro-moted to the number one position. The top spot requires a decision maker. Strong decision makers often put somebody they trust into the number two spot as their adviser and in that position the person is outstanding.
But in the number one spot, the same person fails. He or she knows what the decision should be but cannot accept the responsibility of actually making it.
Other important questions to ask include:
Do I perform well under stress, or do I need a highly structured and predictable environment?
Do I work best in a big organization or a small one? Few people work well in all kinds of environments.
Again and again, I have seen people who were very successful in large organizations flounder mis-erably when they moved into smaller ones. And the reverse is equally true.
The conclusion bears repeating: Do not try to change yourself-you are unlikely to succeed. But work hard to improve the way you perform. And try not to take on work you cannot perform or will only perform poorly.
What Are My Values?
To be able to manage yourself, you finally have to ask. What are my values? This is not a question of ethics. With respect to ethics, the rules are the same for everybody, and the test is a simple one.
I call it the "mirror test." In the early years of this century, the most highly respected diplomat of all the great powers was the German ambassador in London. He was clearly destined for great things - to become his country's foreign minister, at least, if not its federal chancellor. Yet in 1906 he abruptly resigned rather than preside over a dinner given by the diplomatic corps for Edward VII. The king was a notorious womanizer and made it clear what kind of dinner he wanted. The ambas-sador is reported to have said, "I refuse to see a pimp in the mirror in the morning when I shave." That is the mirror test. Ethics requires that you ask yourself. What kind of person do I want to see in the mirror in the morning? What is ethical behaviour in one kind of organization or situation is ethical behaviour in another.
But ethics is only part of a value system - especially of an organization's value system.
To work in an organization whose value system is unacceptable or incompatible with one's own condemns a person both to frustration and to non performance.
Consider the experience of a highly successful human resources executive whose company was acquired by a bigger organization. After the acquisition, she was promoted to do the kind of work she did best, which included selecting people for important positions.
The executive deeply believed that a company should hire people for such positions from the out-side only after exhausting all the inside possibilities. But her new company believed in first looking outside "to bring in fresh blood." There is something to be said for both approaches in my experience, the proper one is to do some of both. They are, however, fundamentally incompatible not as policies but as values. They bespeak different views of the relationship between organizations and people; different views of the responsibility of an organization to its people and their development; and different views of a person's most important contribution to an enterprise. After several years of frustration, the executive quit at considerable financial loss. Her values and the values of the or-ganization simply were not compatible.
Similarly, whether a pharmaceutical company tries to obtain results by making constant, small im-provements or by achieving occasional, highly expensive, and risky "breakthroughs" is not primarily an economic question. The results of either strategy may be pretty much the same. At bottom, there is a conflict between a value system that sees the company's contribution in terms of helping physi-cians do better what they already do and a value system that is oriented toward making scientific discoveries.
Whether a business should be run for short-term results or with a focus on the long term is likewise a question of values. Financial analysts believe that businesses can be run for both simultaneously.
Successful businesspeople know better. To be sure, every company has to produce short-term results.
But in any conflict between short-term results and long-term growth, each company will determine Its own priority.
This is not primarily a disagreement about economics. It is fundamentally a value conflict regarding the function of a business and the responsibility of management.
Value conflicts are not limited to business organizations. One of the fastest growing pastoral churches in the United States measures success by the number of new parishioners. Its leadership believes that what matters is how many newcomers join the congregation. The Good Lord will then minister to their spiritual needs or at least to the needs of a sufficient percentage. Another pastoral, evangelical church believes that what matters is people's spiritual growth. The church eases out newcomers who join but do not enter into its spiritual life.
Again, this is not a matter of numbers.
At first glance, it appears that the second church grows more slowly. But it retains a far larger pro-portion of newcomers than the first one does. Its growth, in other words, is more solid.
This is also not a theological problem, or only secondarily so. It is a problem about values. In a pub-lic debate, one pastor argued, "Unless you first come to church, you will never find the gate to the Kingdom of Heaven." "No," answered the other. "Until you first look for the gate to the Kingdom of Heaven, you don't belong in church." Organizations, like people, have values.
To be effective in an organization, a person's values must be compatible with the organization's values. They do not need to be the same, but they must be close enough to coexist. Otherwise, the person will not only be frustrated but also will not produce results.
A person's strengths and the way that person performs rarely conflict; the two are complementary. But there is sometimes a conflict between a person's values and his or her strengths.
What one does well, even very well and successfully, may not fit with one's value system. In that case, the work may not appear to be worth devoting one's life to (or even a substantial portion thereof).
If I may, allow me to interject a personal note. Many years ago, I too had to decide between my values and what I was doing successfully. I was doing very well as a young investment banker in London in the mid-1930´s, and the work clearly fit my strengths. Yet I did not see myself making a contribution as an asset manager. People, I realized, were what I valued, and I saw no point in be-ing the richest man in the cemetery. I had no money and no other job prospects. Despite the continuing Depression, I quit and it was the right thing to do. Values, in other words, are and should be the ultimate test.
Where Do I Belong?
A small number of people know very early where they belong. Mathematicians, musicians, and cooks, for instance, are usually mathematicians, musicians, and cooks by the time they are four or five years old. Physicians usually decide on their careers in their teens, if not earlier.
But most people, especially highly gifted people, do not really know where they belong until they are well past their mid-twenties. By that time, however, they should know the answers to the three questions: What are my strengths? How do I perform? and. What are my values? And then they can and should decide where they belong.
Or rather, they should be able to decide where they do not belong. The person who has learned that he or she does not perform well in a big organization should have learned to say no to a posi-tion in one. The person who has learned that he or she is not a decision maker should have learned to say no to a decision-making assignment. A General Patton (who probably never learned this himself) should have learned to say no to an independent command.
Equally important, knowing the answer to these questions enables a person to say to an opportu-nity, an offer, or an assignment, "Yes, I will do that. But this is the way I should be doing it. This is the way it should be structured. This is the way the relationships should be.
These are the kind of results you should expect from me, and in this time frame, because this is who I am." Successful careers are not planned. They develop when people are prepared for opportunities because they know their strengths, their method of work, and their values. Knowing where one belongs can transform an ordinary person - hardworking and competent but otherwise mediocre-into an outstanding performer.
What Should I Contribute?
Throughout history, the great majority of people never had to ask the question. What should I contribute? They were told what to contribute, and their tasks were dictated either by the work itself as it was for the peasant or artisan - or by a master or a mistress - as it was for domestic servants. And until very recently, it was taken for granted that most people were subordinates who did as they were told. Even in the 1950s and 1960s, the new knowledge workers (the so-called organization men) looked to their company's personnel department to plan their careers.
Then in the late 1960s, no one wanted to be told what to do any longer. Young men and women began to ask. What do / want to do? And what they heard was that the way to contribute was to "do your own thing." But this solution was as wrong as the organization men's had been. Very few of the people who believed that doing one's own thing would lead to contribution, self-fulfilment, and suc-cess achieved any of the three.
But still, there is no return to the old answer of doing what you are told or assigned to do. Knowl-edge workers in particular have to learn to ask a question that has not been asked before: What should my contribution be? To answer it, they must address three distinct elements: What does the situation require? Given my strengths, my way of performing, and my values, how can I make the greatest contribution to what needs to be done? And finally, What results have to be achieved to make a difference? Consider the experience of a newly appointed hospital administrator. The hospital was big and prestigious, but it had been coasting on its reputation for 30 years. The new admin-istrator decided that his contribution should be to establish a standard of excellence in one impor-tant area within two years. He chose to focus on the emergency room, which was big, visible, and sloppy. He decided that every patient who came into the ER had to be seen by a qualified nurse within 60 seconds. Within 12 months, the hospital's emergency room had become a model for all hospitals in the United States, and within another two years, the whole hospital had been transformed.
As this example suggests, it is rarely possible or even particularly fruitful to look too far ahead. A plan can usually cover no more than 18 months and still be reasonably clear and specific. So the question in most cases should be. Where and how can I achieve results that will make a difference within the next year and a half? The answer must balance several things. First, the results should be hard to achieve they should require "stretching," to use the current buzzword.
But also, they should be within reach. To aim at results that cannot be achieved or that can be only under the most unlikely circumstances is not being ambitious; it is being foolish.
Second, the results should be meaningful. They should make a difference. Finally, results should be visible and, if at all possible, measurable. From this will come a course of action: what to do, where and how to start, and what goals and deadlines to set.
Responsibility for Relationships
Very few people work by themselves and achieve results by themselves - a few great artists, a few great scientists, a few great athletes. Most people work with others and are effective with other people. That is true whether they are members of an organization or independently employed. Managing yourself requires taking responsibility for relationships. This has two parts.
The first is to accept the fact that other people are as much individuals as you yourself are. They perversely insist on behaving like human beings.
This means that they too have their strengths; they too have their ways of getting things done; they too have their values. To be effective, therefore, you have to know the strengths, the performance modes, and the values of your co-workers.
That sounds obvious, but few people pay attention to it. Typical is the person who was trained to write reports in his or her first assignment because that boss was a reader. Even if the next boss is a listener, the person goes on writing reports that, invariably, produce no results.
Invariably the boss will think the employee is stupid, incompetent, and lazy, and he or she will fail. But that could have been avoided if the employee had only looked at the new boss and analyzed how this boss performs.
Bosses are neither a title on the organization chart nor a "function."They are individuals and are entitled to do their work in the way they do it best, it is incumbent on the people who work with them to observe them, to find out how they work, and to adapt themselves to what makes their bosses most effective.
This, in fact, is the secret of "managing" the boss.
The same holds true for all your co-workers. Each works his or her way, not your way. And each is entitled to work in his or her way. What matters is whether they perform and what their values are. As for how they perform - each is likely to do it differently. The first secret of effectiveness is to understand the people you work with and depend on so that you can make use of their strengths, their ways of working, and their values. Working relationships are as much based on the people as they are on the work.
The second part of relationship responsibility is taking responsibility for communication. Whenever I, or any other consultant, start to work with an organization, the first thing I hear about are all the personality conflicts. Most of these arise from the fact that people do not know what other people are doing and how they do their work, or what contribution the other people are concentrating on and what results they expect.
And the reason they do not know is that they have not asked and therefore have not been told.
This failure to ask reflects human stupidity less than it reflects human history. Until recently, it was unnecessary to tell any of these things to anybody. In the medieval city, every-one in a district plied the same trade. In the countryside, everyone in a valley planted the same crop as soon as the frost was out of the ground. Even those few people who did things that were not "common" worked alone, so they did not have to tell anyone what they were doing.
Today the great majority of people work with others who have different tasks and responsibilities. The marketing vice president may have come out of sales and know everything about sales, but she knows nothing about the things she has never done-pricing, advertising, packaging, and the like. So the people who do these things must make sure that the marketing vice president under-stands what they are trying to do, why they are trying to do it, how they are going to do it, and what results to expect.
If the marketing vice president does not understand what these high-grade knowledge specialists are doing, it is primarily their fault, not hers. They have not educated her. Conversely, it is the mar-keting vice president's responsibility to make sure that all of her co-workers understand how she looks at marketing: what her goals are, how she works, and what she expects of herself and of each one of them.
Even people who understand the importance of taking responsibility for relationships often do not communicate sufficiently with their associates. They are afraid of being thought presumptuous or inquisitive or stupid. They are wrong. Whenever someone goes to his or her associates and says, "This is what I am good at. This is how I work. These are my values. This is the contribution I plan to concentrate on and the results I should be expected to deliver," the response is always, "This is most helpful. But why didn't you tell me earlier?" And one gets the same reaction - without exception, in my ex-perience-if one continues by asking, "And what do I need to know about your strengths, how you perform, your values, and your proposed contribution?" In fact, knowledge workers should request this of everyone with whom they work, whether as subordinate, superior, colleague, or team mem-ber. And again, whenever this is done, the reaction is always, "Thanks for asking me. But why didn't you ask me earlier?" Organizations are no longer built on force but on trust. The existence of trust between people does not necessarily mean that they like one another. It means that they understand one another.
Taking responsibility for relationships is therefore an absolute necessity.
It is a duty. Whether one is a member of the organization, a consultant to it, a supplier, or a distribu-tor, one owes that responsibility to all one's co-workers: those whose work one depends on as well as those who depend on one's own work.
The Second Half of Your Life
When work for most people meant manual labor, there was no need to worry about the second half of your life.
You simply kept on doing what you had always done. And if you were lucky enough to survive 40 years of hard work in the mill or on the railroad, you were quite happy to spend the rest of your life doing nothing. Today, however, most work is knowledge work, and knowledge workers are not "fin-ished" after 40 years on the job, they are merely bored.
We hear a great deal of talk about the midlife crisis of the executive. It is mostly boredom. At 45, most executives have reached the peak of their business careers, and they know it. After 20 years of doing very much the same kind of work, they are very good at their jobs.
But they are not learning or contributing or deriving challenge and satisfaction from the job. And yet they are still likely to face another 20 if not 25 years of work. That is why managing oneself increas-ingly leads one to begin a second career.
There are three ways to develop a second career. The first is actually to start one. Often this takes nothing more than moving from one kind of organization to another: the divisional controller in a large corporation, for instance, becomes the controller of a medium-sized hospital.
But there are also growing numbers of people who move into different lines of work altogether: the business executive or government official who enters the ministry at 45, for instance; or the midlevel manager who leaves corporate life after 20 years to attend law school and become a small-town attorney.
We will see many more second careers undertaken by people who have achieved modest success in their first jobs. Such people have substantial skills, and they know how to work. They need a community-the house is empty with the children gone - and they need income as well. But above all, they need challenge.
The second way to prepare for the second half of your life is to develop a parallel career. Many people who are very successful in their first careers stay in the work they have been doing, either on a full-time or part-time or consulting basis. But in addition, they create a parallel job, usually in a non profit organization, that takes another ten hours of work a week. They might take over the ad-ministration of their church, for instance, or the presidency of the local Girl Scouts council. They might run the battered women's shelter, work as a children's librarian for the local public library, sit on the school board, and so on.
Finally, there are the social entrepreneurs.
These are usually people who have been very successful in their first careers. They love their work, but it no longer challenges them. In many cases they keep on doing what they have been doing all along but spend less and less of their time on it. They also start another activity, usually a non-profit. My friend Bob Buford, for example, built a very successful television company that he still runs. But he has also founded and built a successful non-profit organization that works with Protes-tant churches, and he is building another to teach social entrepreneurs how to manage their own non-profit ventures while still running their original businesses.
People who manage the second half of their lives may always be a minority.
The majority may "retire on the job" and count the years until their actual retirement. But it is this minority, the men and women who see a long working- life expectancy as an opportunity both for themselves and for society, who will become leaders and models.
There is one prerequisite for managing the second half of your life: You must begin long before you enter it.
When it first became clear 30 years ago that working-life expectancies were lengthening very fast, many observers (including myself) believed that retired people would increasingly become volun-teers for non-profit institutions. That has not happened. If one does not begin to volunteer before one is 40 or so, one will not volunteer once past 60.
Similarly, ail the social entrepreneurs I know began to work in their chosen second enterprise long before they reached their peak in their original business.
Consider the example of a successful lawyer, the legal counsel to a large corporation, who has started a venture to establish model schools in his state.
He began to do volunteer legal work for the schools when he was around 35. He was elected to the school board at age 40. At age 50, when he had amassed a fortune, he started his own enterprise to build and to run model schools. He is, however, still working nearly full-time as the lead counsel in the company he helped found as a young lawyer.
There is another reason to develop a second major interest, and to develop it early. No one can expect to live very long without experiencing a serious setback in his or her life or work. There is the competent engineer who is passed over for promotion at age 45- There is the competent college professor who realizes at age 42 that she will never get a professorship at a big university, even though she may be fully qualified for it.
There are tragedies in one's family life: the break-up of one's marriage or the loss of a child. At such times, a second major interest-not just a hobby-may make all the difference. The engineer, for ex-ample, now knows that he has not been very successful in his job. But in his outside activity-as church treasurer, for example - he is a success. One's family may break up, but in that outside ac-tivity there is still a community.
In a society in which success has become so terribly important, having options will become increas-ingly vital. Historically, there was no such thing as "success." The overwhelming majority of people did not expect anything but to stay in their "proper station," as an old English prayer has it. The only mobility was downward mobility.
In a knowledge society, however, we expect everyone to be a success. This is clearly an impossibil-ity. For a great many people, there is at best an absence of failure. Wherever there is success, there has to be failure. And then it is vitally important for the individual, and equally for the individ-ual's family, to have an area in which he or she can contribute, make a difference, and be some-body That means finding a second area-whether in a second career, a parallel career, or a social venture-that offers an opportunity for being a leader, for being respected, for being a success.
The challenges of managing oneself may seem obvious, if not elementary.
And the answers may seem self-evident to the point of appearing naive. But managing oneself re-quires new and unprecedented things from the individual, and especially from the knowledge worker. In effect, managing oneself demands that each knowledge worker think and behave like a chief executive officer. Further, the shift from manual workers who do as they are told to knowledge workers who have to manage themselves profoundly challenges social structure. Every existing society, even the most individualistic one, takes two things for granted, if only subconsciously: that organizations outlive workers, and that most people stay put.
But today the opposite is true. Knowledge workers outlive organizations, and they are mobile. The need to manage oneself is therefore creating a revolution in human affairs.
Thursday, June 17, 2010
Thursday, June 10, 2010
Meaning of Work
This is excerpted from a book excerpt published in BusinessWeek from Dave Ulrich's new book "The Why of Work". Full article here.
Guiding questions on the journey of discovering the meaning of work in organisations.
1. Who am I?
Abundance includes clarity about identity and signature strengths and ensures that employees will build on their strengths that strengthen others.
2. Where am I going?
Abundance emerges from a clear sense of what we are trying to accomplish and why, sustains both social and fiscal responsibility, and aligns individual motivation.
3. Whom do I travel with?
Abundance is enhanced by meaningful relationships. High-performing teams need to be high relating teams.
4. How do I build a positive work environment?
Abundance thrives on positive routines that help ground us in what matters most. While bad habits thrive on isolation and shame, positive routines help us connect with ourselves and others. Routines and patterns driven by our deepest values help us stay grounded in what matters most and available to those who matter most.
5. What challenges interest me?
Abundance occurs when companies can engage not only employees' skills (competence) and loyalty (commitment), but also their values (contribution). The most engaged employees are generally those whose work gives them the opportunity to stretch while doing work they love and solving problems they care about.
6. How do I change, learn, and grow?
Abundance acknowledges that failure can be a powerful impetus to growth and learning. When we face change and take risks to work outside our comfort zone, resist defensiveness about mistakes, learn from failure, and keep trying, we become not only more resilient, but more satisfied with life.
7. What delights me?
Abundance thrives on simple pleasures. Sources of delight might include laughing at ourselves, appreciating excellence, relishing beauty, being present in the moment, and having fun at work. These sources of delight are highly personal, depending on the personality of the leader and the requirements of the employees.
Guiding questions on the journey of discovering the meaning of work in organisations.
1. Who am I?
Abundance includes clarity about identity and signature strengths and ensures that employees will build on their strengths that strengthen others.
2. Where am I going?
Abundance emerges from a clear sense of what we are trying to accomplish and why, sustains both social and fiscal responsibility, and aligns individual motivation.
3. Whom do I travel with?
Abundance is enhanced by meaningful relationships. High-performing teams need to be high relating teams.
4. How do I build a positive work environment?
Abundance thrives on positive routines that help ground us in what matters most. While bad habits thrive on isolation and shame, positive routines help us connect with ourselves and others. Routines and patterns driven by our deepest values help us stay grounded in what matters most and available to those who matter most.
5. What challenges interest me?
Abundance occurs when companies can engage not only employees' skills (competence) and loyalty (commitment), but also their values (contribution). The most engaged employees are generally those whose work gives them the opportunity to stretch while doing work they love and solving problems they care about.
6. How do I change, learn, and grow?
Abundance acknowledges that failure can be a powerful impetus to growth and learning. When we face change and take risks to work outside our comfort zone, resist defensiveness about mistakes, learn from failure, and keep trying, we become not only more resilient, but more satisfied with life.
7. What delights me?
Abundance thrives on simple pleasures. Sources of delight might include laughing at ourselves, appreciating excellence, relishing beauty, being present in the moment, and having fun at work. These sources of delight are highly personal, depending on the personality of the leader and the requirements of the employees.
Tuesday, June 8, 2010
Organizational Consulting
Some guy once said "Consultants are those fellas who use powerpoint presentations to state the bleeding obvious." Funny and true, to some extent. Alan Weiss states the "bleeding obvious" in this book but that can be really helpful. I was prompted to pick Weiss' book because of the difficulties and resistance my consultants and I encountered in our effort to put in place changes to raise the current level of quality of our client deliverables here. Sort of like an internal change agent role, but a lot less fancy in reality and instead a lot of hard work. It's hard to shape attitudes and influence behaviours, and I was looking for some help along the way. Weiss has a lot of common sense and he has the ability to frame issues and express ideas in a practical and usable fashion, which if implemented properly, could be really effective. This is one of those books that is not just a one-off read, but rather a guide that I would go back to again and again to thumb through, jot down thoughts and probably spin off a couple of ideas of my own.Tuesday, June 1, 2010
Wild Grass
One of the best books I have read so far on China. This is one of the few books out there that takes a microscopic look on how individuals are navigating through the vast changes that are sweeping across China on every level from political, economic to social, and still coming out with their sanity intact. Wild Grass is a compilation of three accounts on the experiences of ordinary Chinese. The first involves a self-educated peasant lawyer who takes on the local political elite over the excessive and illegal taxation of impoverished farmers, and mobilizes thousands in the process. He ended up being illegally held and denied a trial for 5 years. (Incidentally, I just found out that the Chinese government taxes everyone with an income, big or small, and the minimum tax rate is 5%, for the monthly income bracket of RMB 1-500.) The second case pits owners of homes in the historic heart of old Beijing against city planners who want to bulldoze nearly everything old to make way for high-rise developments for the Olympics showcase and succeed spectacularly through sheer brute coercion. The third case exposes the persecution and determined persistence in her faith of one woman who joined Falun Gong protests, and ended up losing her life. Her daughter takes up her wronged death to the authorities but realises only fading hopes of a redress against the baffling bureaucracy and general indifference of the cadres.Ian Johnson is a Pulitzer Prize-winning journalist with the Wall Street Journal.
Monday, May 17, 2010
Bruges, Belgium
I took a day trip from Amsterdam to Bruges over the Queen's Day weekend, mainly to see this very popular tourist destination in Europe, and also to avoid the riot that engulfed the Dutch city. Bruges is small enough to go everywhere by foot as long as one is shod in sensible footwear. It is very well-preserved with architecture going back to the 14th and 15th centuries. Because of this, Bruges is well-visited, mainly by Europeans, which resulted in tourist prices everywhere. I had my best lunch during my 10-day stay in Amsterdam at well, not Amsterdam, but Bruges. I wanted to try the seasonal (spring only) and highly-prized white asparagus, which the locals call "white gold", and ordered it done in the classic Belgian way at a local restaurant. The thick white asparagus were steamed, drizzled in extra virgin olive oil, and paired with mashed hard-boiled egg tossed in melted butter and chopped basil leaves. That was the best asparagus ever - soft yet still slightly crunchy and juicily sweet. It cost me 14.95 euros for three sticks but they were worth every single cent. The mussels with white wine was also a dish I had been wanting to have, after my tasty experience with the French version. The mussels in white wine pot was good but overshadowed by that wonderful appetiser of white gold. On hindsight, I should have stayed one night at Bruges to thoroughly enjoy the small town, avoid having to rush for the unpredictable inter-city train, and of course wolf down a few more sticks of asparagus.One note about the inter-city rail was that the timetables were never always adhered to, causing some trouble for passengers, especially non-Dutch, non-Flemish and non-French speaking folks, like me. A 4-hour train ride could turn into a 6-hour train ride because the rail company decided to drop everyone off at border station on Dutch land, requesting passengers to wait for the next train at another platform, just so that repair works can be carried out on the tracks. And when confused passengers grabbed a train conductor to find out what was going on, the poor Belgian guy could only say "I don't know. I am Belgian. I don't know the Dutch system." I am not particularly impressed by the European way of running their train system.
Sunday, May 16, 2010
Thursday, May 6, 2010
I am lucky that I am able to interact frequently and work with very high quality talent in my profession. Some of these highly successful people have become a role model of sorts whom I often find myself referring back to, which in turn becomes a source of motivation for me to seek self-improvement constantly. Learning through experience and imitation has greater impact than reading a book and noting down what it is telling you to do. I think often about these people and analyse what is it about them that differentiates them from most others and why I simply can’t quite put them out of my mind. I identified the one star-like quality of each of the following four people (in pseudonyms), who have made the strongest impression on me so far.
Elephant – Positivism
- Enthusiasm is infectious
- Believes in the change for good and always strives for improvement
- Believes in the good of people => Build people’s capability = Building organizational capability
- Believes in the power of many as opposed to one = Empowers people
- Positive towards life and oneself = Exudes a can-do attitude
Maggot – Personable
- Easy to connect with and build a relationship
- Makes an impact on others = Memorable
- Hard to dislike
- Gives others a sense of comfort
- Non-threatening
Yellow – Discipline
- Stays focused day in day out = Cut straight to the core of issues.
- Is relentless = Putting in the effort consistently
- Stays engaged throughout all happenings
- Excellent time and self management ability
Jam – Serenity & Balance
- Exudes a calm demeanor = Reassuring presence
- Has a strong sense of self (“a strong core”)
- Ability to stay on an even keel when dealing with shocks/problems/conflict
- Imparts confidence to others (“There is nothing I can’t handle”)
Subscribe to:
Posts (Atom)
