INFORMS Annual Conference

The annual INFORMS conference is a destination for scientists, researchers, and industry experts focused on operations research & analytics. The conference was recently held in Anaheim, CA, October 24-27, 2021.

I had the privilege of chairing a session titled “Macro Energy Systems: Energy and Climate.” Researchers shared presentations from the cutting edge of energy system modeling. Their work explored possible trajectories towards a low-carbon energy transition and the policies and technologies that could enable such transition.

  • Lei Duan of Carnegie Science presented an analysis of the potential for nuclear power combined with thermal energy storage to contribute to a low-carbon future.
  • Kenji Shiraishi of UC Berkeley shared an analysis of the possible role of hydrogen and policy choices in shaping Japan’s long-term energy future.
  • Charalampos Avraam of New York University presented a detailed study of the North American natural gas market and how renewable energy policy may affect it.
  • Tom Brown of the Technical University of Berlin presented a method of incorporating public acceptance of different technologies into a least-cost energy system model.
  • My presentation built on the work I shared at the MIT A+B 2021 Symposium and studied the improvements modelers can find when incorporating more years of weather data into their models with large fractions of wind and solar power.

The INFORMS community is increasing interest in energy system modeling from a low-carbon transition perspective. This can only lead to positive outcomes for energy systems modelers as more researchers with backgrounds in optimization and operations research become interested in our field.

I look forward to this conference and the connections I will make next year and hope to actually attend in-person.

Reducing Electricity System Variability with Solar Power

Solar power is often thought to increase the variability of electricity systems. In a recent paper (open access), Ken Caldeira and I show that adding solar to electricity systems, where solar power correlates with electricity demand, can actually reduce the variability in peak residual electricity load.

Residual Load

Residual load is electricity load (demand) minus generation from variable resources, such as wind and solar. Residual load represents the load that must be supplied by more controllable resources: firm generation (gas, nuclear, etc.), energy storage, and demand response.

Depiction of residual load (dashed line) and the peak residual load value

For a system operator, peak residual load indicates a lower bound on the quantity of firm generation, stored energy, and demand response that must be available in their system to supply all electricity loads.

As wind and solar are added to our electricity systems, system planners will likely rely on estimates of future peak residual load and how the peak values vary from year to year as crucial planning metrics.


We estimate the peak residual load and how much it varies from year to year as wind and solar generation are added to four example electricity systems. From this, we find that the variability in the peak values changes as more wind and solar are added.

The variability (spread) in the peak residual load values from year to year is depicted for systems as wind and solar generation increase. In PJM, for example, progressing vertically in the figure indicates adding solar generation. As solar generation is added, the variability decreases as indicated by the darker colors.

Interestingly, for the three modeled systems that experience their peak electricity usage in the summer months (ERCOT in Texas, PJM in the mid-Atlantic, and NYISO in New York state), adding solar statistically reduces the spread in the peak values from year to year.

These three summer peaking systems show a strong correlation between peak electricity usage and the hottest days. The hottest days are indicated by the largest “daily degree day” values in the below figure.

Thus, by adding generation that correlates with the most extreme peak load hours, electricity systems can become more predictable even if that generation is from a variable renewable resource like solar.

Reducing the spread in the peak values from year to year could possibly make system planning simpler by having more predictable peak residual load values.


We used historical electricity load data from the four studied systems: ERCOT, PJM, NYISO, and France.

Ten years of historical electricity load (demand) for the studied regions.

We used historical weather data to derive plausible wind and solar generation profiles concurrent with the load data.

We incrementally increased the contributions of wind and solar generation from zero to generation equivalent in quantity to providing 100% of annual load. For each residual load profile, we assessed the spread in the peak values from year to year.

To calculate the spread in the peak values, called the inter-annual variability (IAV), we take the mean of the 10 peak residual load values from each year of data and calculated the standard deviation of these 10 mean values.

Other links

Twitter discussions:

Emerging opportunities for hydrogen production as a flexible electricity load

Wind and solar generation are powering more and more of our electricity systems. Along with their zero-carbon electricity comes their variability and uncontrollable power output.

Utilities are increasingly tackling the variable nature of wind and solar power by building energy storage to shift available power from when it can be produced by nature to when it is most needed by the grid.

There is growing interest and possibilities in tackling the variability issue not by shifting available power to meet electricity demands, but by shifting electricity demands to meet available power.

One potential candidate flexible load candidate is producing hydrogen gas by splitting water using electrolysis. Producing low-cost hydrogen with minimal carbon emissions is currently viewed as a cornerstone of an energy transition away from carbon emitting sources.

Our new paper

We recently published a paper in Advances in Applied Energy considering producing hydrogen as a flexible electricity load (demand) in future low-carbon electricity systems.

We asked how the operations of future electricity systems would change if we introduced a small, flexible hydrogen producing load. Is there essentially “free” electricity available to a business who can choose to operate only when the sun is shining and wind is blowing? How much “free” electricity will there be?

Study results

We find that in systems with substantial wind and solar power, zero cost electricity is available sometimes and low-cost power is available almost always. In fact, in modeled systems powered exclusively by wind and solar power, zero-cost, zero-carbon power was available more than 95% of the time.

One enticing thing about flexible loads is when other electricity uses are pushing the grid to its maximum extent and power costs are high, flexible loads can simply throttle back or even turn off.  This would save them considerable money and could save the grid from needing to expand generation capacity, a win-win situation.

However, if we really push the envelope with vast amounts of flexible loads like electric vehicles and by producing hydrogen, the grid’s generation capacity will eventually need to expand. After all, there is only so much zero-cost and low-cost power available in the original electricity system.

Many more interesting results and all the details can be found in the paper.

I am looking forward to continuing this line of work and further exploring the integration of hydrogen production with low-carbon electricity systems and how both can enable a low-carbon energy transition.

Wind and solar resource droughts in California

I am glad to say that a paper led by Katherine Z. Rinaldi, which I helped contribute to, was recently published in Environmental Science and Technology. Like most places, California is susceptible to multi-day low wind or cloudy events. These events, if not properly planned for could have dire consequences for the electricity grid.

The paper studies the frequency and duration of severe weather events, specifically wind and solar resource droughts, and their impact on a 100% wind and solar powered electricity grid. We studied how the frequency and duration of these events changes when wind and solar generation resources are spread over larger or smaller geographic regions.

In short, a more geographically dispersed system has fewer resource droughts and those that happen as shorter. This suggests that integrating electricity systems over larger distances will be increasingly beneficial as the fraction of power supplied by wind and solar increases.

We hope this paper will aid policy makers, utilities, and others whom are building California’s clean energy transition towards a net-zero carbon system.

MIT A+B Symposium

During the last two years, MIT and Harvard have co-hosted the MIT A+B symposium on rapidly decarbonizing our society. These conferences have a unique approach that I really appreciate. The organizers call for presentations on A) mature, cost-effective technologies that are ready to deploy at scale, and B) potentially breakthrough technologies that may enable achieving a near-zero carbon emission society.

Unique conference structure

Splitting the presentations into these two categories or timelines does two things. It supports the urgency of the situation by emphasizing and exploring details of the abundant cost-effective, existing technologies that can be deployed now to make immediate impacts. These are the technologies that businesses can rely upon in the near-term and build into their business plans. They are also the technologies that policymakers should be looking towards to craft achievable near-term climate targets and policy.

The “potentially breakthrough technologies” category engages the research community and pushes the question of what is possible. Many studies have shown that achieving 80% carbon emission reductions is relatively simple. The last 20% of emissions that takes us to carbon neutral is by far the most difficult piece to solve in the carbon neutral puzzle. We need to cast a wide net to explore many possible technologies that could be available in a few decades to meet our final climate targets.

Electricity generation in systems with substantial wind- and solar-power

I submitted an abstract to the 2020 MIT A+B symposium focused on category A, deploying existing, cost-effective technologies. The abstract asked two questions: 1) how much traditional electricity generation capacity is needed to reliably meet society’s electricity demands as wind- and solar-power rapidly scale up? And, 2) how does the required traditional electricity generation capacity change year-to-year? This is an interesting question because the answer varies based on local industry and electricity use patterns and climate. A more detailed discussion of my presentation will follow.

For now, suffice to say, the abstract was accepted. A prerecorded virtual presentations is available online. Lastly, a short paper extending and refining the material in the presentation is now available in the conference proceedings.

I was invited to submit an extended version of the conference paper to the Applied Energy journal. The deadline for submission is Feb 1, 2021. Time to get moving.

Comparing FERC and EIA electricity demand data

The United States government coordinates the collection of hourly electricity demand data from regional entities for use in planning and decision making processes.  The Federal Energy Regulatory Commission (FERC) provides easily accessible data records spanning 2006-2018 for a mix of Balancing Authorities (BAs) and Planning Areas with Form 714.

While the Energy Information Administration (EIA) began their collection of hourly electricity demand data in July of 2015 for all BAs with Form 930. The EIA data are updated in near real-time and bring other benefits such as including hourly generation by resource type: coal, hydropower, natural gas, nuclear, wind, solar, petroleum, and other.

An interesting question for the energy modeling community is, does the 2017 data gathered by FERC align with the 2017 data gathered by EIA?  Can these records be used almost interchangeably?  Additionally, benefits will be realized by stitching together the longer historical FERC data records with the EIA records that contain more details of the current system.

One of our collaborators, Zane Selvans (@ZaneSelvans) of the Catalyst Cooperative (@CatalystCoop), mapped the ~200 FERC respondents to the ~70 EIA BAs and arranged the FERC data into a more usable format.  With this, we compared the hourly demand values for the successfully mapped BAs for 2017.  Details of the comparison methods are at the end of this post.


We compare the ratio of FERC hourly values to EIA hourly values and calculate the ratio of mean, minimum, and maximum values for each region.

California Independent System Operator (CISO)

Midwest Independent System Operator

The two examples here show hourly comparisons for CISO, with most values nearly identical and nearly all within 10%, and MISO, with most values agreeing within 10% and overall agreement based on the ratio of mean values of 1.01.

ISO New England (ISNE)

PJM Interconnection (PJM)

Some regions show a mean value close to 1 yet have non-uniform features in their distributions, such as ISNE (ratio of mean values = 0.99) and PJM (ratio of mean values = 0.98).

Furthermore, other regions have substantial discrepancies in the ratio of their mean values.  A histogram of the ratios of the mean values for each compared BA shows agreement within a few percent for over 30 BAs (a csv file is attached at the bottom showing the ratio of their mean, minimum, and maximum values). Additionally, we compare the minimum and maximum values and see a distribution similar to the mean value comparison.

Ratio of the mean of demand values for each mapped BA (FERC mean value/EIA mean value)

Ratio of the minimum and maximum demand values for each mapped BA


There are a considerable number of Balancing Authorities that have reasonably similar FERC and EIA hourly demand records based on agreement within a few percent of the ratios of mean, minimum, and maximum values.  This indicates that the FERC and EIA records may be approximately interchangeable for these BAs if the exact hourly profile is not a concern (see excel file for list).  The fact that many histograms contain a spread about 1.0 is worth exploring for anyone considering using these profiles as replacements for each other while modeling. Are there biases in which hours are misaligned?

In the future, this could also allow analysts to stitch together the longer FERC records with the more current and detailed EIA records.  The Catalyst Cooperative and Zane are pursuing work along these lines.  We wish them the best of luck!


The FERC data contains records from both Balancing Authorities and Planning Areas, while the EIA records are only for Balancing Authorities.  Therefore, many of the FERC records do not have EIA equivalents.  We only compare records that we think should align.

Both the FERC and EIA data records are imperfect, containing zero values, missing values, and the occasional outlier value.  For the EIA data, we use the EIA records after removing outlier values based on the details in this paper.  For the FERC data, we use the FERC records arranged by Zane with all zero values removed.  Hours are only included in the comparison if the corresponding hourly value in each record was present and was not removed by these two cleaning methods.

  • Summary csv file: comparing the mean, minimum, and maximum values in the FERC 714 and EIA 930 hourly demand data for year 2017 for the matched BAs.
  • FERC to EIA mapping: the mapping of FERC respondents to their EIA codes and acronyms provided by Zane.

More realistic data leads to more realistic models

There are many quirks of being an ex-high energy particle physicist who completed their PhD with the CMS experiment. For one, waking up in the middle of the night for an upset child doesn’t seem too bad compared to the many nights when I was “on-call” and woken up at 3am to help debug data collection issues with our experiment. I would much rather be “on-call” for my son than for a 14,000 tonne inanimate object.

Another quirk is that I am a year into my Postdoc at Carnegie Science and only now am I publishing my first ever first author paper. It is hard, in fact nearly impossible, to get to the front of the 3,000 person author list for the papers published by the CMS experiment. Needless to say, I did not make it to the front while I was part of the CMS team.

Now, I have the pleasure of being the first of only four authors on a paper discussing data cleaning and preparation for use in our energy models. While not the most glorious of papers, we hope this paper and the data we cleaned can be used by the energy modeling community. After all, more realistic data leads to more realistic models.

See this friendly blog post about the paper.

The Renewables Cliché

Part III of an energy and research discussion for my parents (part I & part II)

Who hasn’t heard the cliché renewable energy complaint, “the sun doesn’t always shine and the wind doesn’t always blow”? Solar and wind energy operate in stark contrast to the mechanical predictability of a natural gas power plant. Grid operators can not request a solar power plant to produce more electricity, only the sun can do that.

The biggest challenge for carbon-free, renewable energy technologies, like solar and wind, are their variability and intermittency. What does this actually look like and why does it matter?

Variability refers to the predictable changes in renewable energy output throughout the day and seasons. For example, solar power very predictably drops to zero every night. Solar output is also higher during the summer months because we have more hours of daylight.

Intermittency refers to the less predictable changes in renewable energy output. These can result from clouds passing over solar panels or from storm fronts rolling through and increasing the power output of wind turbines. In general, intermittency is caused by weather events. With improved forecasting, we can more easily predict and plan for intermittency.

Renewable Electricity Output

An installations of solar panels or wind turbines will provide a changing amount of power to the grid throughout the day.

This chart shows the availability of solar and wind energy over three October days in 2017.

To estimate the power output of a solar installation at any moment, multiply the output rating of the system by the availability (a.k.a. capacity factor). For example, a 10 Megawatt (MW) solar installation would produce 7.5 MW of power at 10:00am, which is indicated by the arrow above. Predictably, the solar capacity factor plummets to zero overnight.

The wind energy availability fluctuates up and down. It lacks a simple pattern like solar and is a great example of intermittency.

Renewables on the Grid

Wind and solar power never perfectly align with electricity demand. Because of this, they add complexity to operating the grid. Let’s take an example from one of the demand curves in the previous post for a small utility in Florida. I will use a few fictitious scenarios to illustrate some interesting points without getting bogged down in the details.

  1. their only power source is a natural gas plant
  2. they have a solar installation rated at 20 MW and natural gas provides the rest of their power
  3. same as 2) except 40 MW of solar
  4. same as 2) except 100 MW of solar

How large must a natural gas power plant be to satisfy all of the demand?

In scenario 1, the natural gas plant must provide power to meet the demand peak of 80 MW. So, the utility needs to build at least an 80 MW natural gas plant.

How much demand is satisfied by solar power in scenario 2? The yellow shaded region answers this question. It is the product of the 20 MW solar rating times the solar capacity factor. The solar output is slightly less than 20 MW at its peak and zero at night.

The remaining demand in scenario 2 must be covered by the natural gas plant. To calculate this, subtract the generated solar power from the demand curve as is shown by the orange dashed line. Therefore, a smaller, 65 MW natural gas facility will suffice.

In scenario 3, one of the challenges of solar power is apparent. Despite adding more solar, the required natural gas plant is still approximately 65 MW. There is a new “peak” in the remaining demand. And, it can not be addressed by simply building more solar.

If we continue to build even more solar as in 4, then we arrive at a situation where the generated solar power is greater than demanded. In a real-life situation, overproduction could either be: sent to an adjacent region if the transmission lines are capable of this, stored in batteries for use later, or “curtailed” which essentially means it is wasted.

I’ll have more discussions on curtailment, energy storage, and ways researchers and utilities are approaching these issues soon.

What does your electricity use look like?

Part II of an energy and research discussion for my parents (part I)

We all expect that when we flip the light switch at night, the lights will turn on. We won’t have to stumble around in the dark feeling around for a glass of water or to let the dog out. There are people and algorithms working around the clock to make sure when you and I request power, it is available.

This is exactly what our electric utilities do. They focus on delivering reliable and safe power to meet our “demand”. Because most utilities do such a good job of delivering electricity, we never think about the details.

The chart below gives a good idea of my family’s electricity demand last Thursday, October 24th. You can see there are many spikes as we made coffee and ran the dishwasher in the morning and other larger spikes later when we returned from work. Your energy use probably looks just as spiky though the details will certainly differ.

Cartoon of my family’s energy demand on Thursday, October 24th.

Our household daily usage is fairly similar day-to-day, even if the exact timing of making Henry and myself breakfast can differ quite a bit.

Sharp spikes to smooth curves

If every household has spiky electricity demand, how can our utilities anticipate the amount of power they need to produce at any one moment? Utilities rely on my demand, the demand of all my neighbors, and your demand being similar day after day. This helps them figure out a daily quantity which will likely be requested.

What about the precise timing of our morning coffee, how do they get that right? Utilities rely on having many customers and the law of large numbers. Not everyone makes coffee at 6:00am. Some make coffee earlier, some make it later, some not at all. When the actions of thousands of electricity customers are added together, their small differences smooth out the jagged spikes you see from my household when viewed in isolation. This leads to a very predictable energy demand throughout the day for a utility territory.

The below charts show electricity demand over three October days in 2017. The first is for a small utility with only 26,000 customers. This demand curve is already much smoother than my single household’s usage. And, the total demand across the contiguous United States is even smoother. In both of these cases, the demand has a cyclic peak-and-trough pattern with the lowest demand late at night.

The left figure shows the electricity demand for a small utility with 26,000 customers while the right figure is the total demand for the contiguous U.S. 1 Megawatt = 1,000,000 Watts

Utilities can make accurate forecasts of their territory’s electricity usage 24 hours in advance. Most can predict 24 hours ahead within 3% of the real value. This makes the cyclic peak-and-trough structure of demand very approachable for utilities.

Providing Electricity the Traditional Way

Over the past century, utilities have traditionally built enough coal, gas, nuclear, and hydro plants to match the peak electricity demand for their territory.

When a utility forecasts demand will reach 5,000 Megawatts tomorrow at 5:00pm in their territory, they make sure 5,000 Megawatts of their power plants will be ready to produce at that time. Human errors and mechanical failures can happen, and when they do, they are addressed. But, overall, the traditional system is very predictable.

The large scale introduction of intermittent renewable energy is changing this and will be the topic of the next post. Let me know if you have any questions or would love more detail. Check out the current energy use in your region with this amazing map.

Why Study Energy?

Part I of an energy and research discussion for my parents

Abundant energy has shaped the modern world. It has enabled wonderful innovations such as rapid and affordable travel, vaccines produced on an industrial scale, fertilizer for our crops, an elevated standard of living for billions of people, and the Information Age just to name a few. Fossil fuel makes up the majority of the abundant, easily accessible energy we have consumed to get here. While our standards of living have been elevated, the aggressive burning of fossil fuels has positioned us on a path for severe climate change [1].

Current technologies exist that can significantly reduce our global energy use while delivering what energy is still needed via clean, carbon free sources. And, yes, this can be done while bringing power to the one billion people currently living in energy poverty.

A Brief History of Energy Use

Energy use has skyrocketed over the past two centuries. Over this same period, the composition of fuels and power sources we use changed significantly. Prior to the 1850s, wood was the main fuel source. For the first half of the 1900s, coal dominated. But coal was quickly outpaced by petroleum with the rise of the automobile. The 1970s saw the introduction of natural gas and nuclear power on a large scale.

From EIA “History of energy consumption in the United States, 1775-2009”

At the start of the 1970s, petroleum was set to continue its exponential climb. Instead, the global energy market was struck by the oil crisis of 1973. The U.S. Federal Government enacted sweeping programs to beat down energy use while keeping the economy humming along. This was the introduction of energy efficiency as a staple of the U.S. energy strategy [2].

Two versions of energy intensity: the cyan “Energy Intensity Index” as developed by the U.S. Office for Energy Efficiency and Renewable Energy and the green ratio of energy use to GPD “E/GDP” – Reference [3]

It is difficult to disentangle the effects of a growing population, an expanding economy, and an economy transitioning away from heavy industry in a single chart. The above chart shows us that the “Energy Intensity” or energy used to create economic value has been decreasing in the U.S. for many years. However, in the 1970s, after enacting aggressive efficiency policies, Energy Intensity fell faster than before.

Another way of viewing energy use is considering the amount of energy used per capita. Historically, the the total energy use per person in the U.S. increased every year until the 1970s. Since then, use per person has been steady or slightly declining.

It is worth noting that the U.S. has outsourced a significant portion of its heavy industry as it transitions towards a service based economy. Regardless, energy use per capita and energy intensity are both helpful indicators of the efficacy of coordinated energy efficiency policy at the national level.

A re-invigoration of coordinated energy efficiency policies would help further decrease Energy Intensity and reduce the energy which we need to supply with carbon free sources.

Carbon Free Energy

Nuclear energy and large scale hydroelectric power have been staples of the U.S. electric system for many decades (see first figure). Solar and wind power are relatively new to the U.S. energy portfolio. These four technologies, plus biofuels, make up two different categories of carbon free energy.

  • Predictable power sources who’s power output can be increased or decreased as needed (nuclear, hydro, biofuels)
  • Intermittent sources who’s output is controlled by weather, not customer needs (solar and wind).

The above chart from the EIA shows carbon free energy production by source and is part of their annual energy review. Solar and wind energy have begun a rapid rise since the turn of the millennium.

The rapid increase in solar and wind energy is on a collision course with the way electric utilities traditionally operate their grid. Intermittent solar and wind challenge operators to deliver continuous, reliable power despite their fluctuations. Batteries and other storage technologies are being researched, developed, and continuously improved to help smooth out these difficulties.

Currently, in places with lots of installed solar power, electricity is stored in batteries when it is sunny and discharged back into the grid when large clouds pass over, reducing solar panel output, or during the night. Many new wind power installations also include batteries to help smooth out fluctuations.

To enable a large scale energy transition away from carbon intense sources towards carbon free sources, we need to figure out the right mix of intermittent renewable energy, other clean sources, and storage technologies to create a reliable grid. This is the central focus of my current research working with Ken Calderia at Carnegie Science.

  • [1] IPCC Working Group 3: Fifth Assessment Report “Summary for Policy Makers”
  • [2] U.S. Office of Energy Efficiency and Renewable Energy, “Energy Intensity Indicators”, Accessed 14 October 2019,